RunicGateway/website
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Activity

feat(engagement): deliverability — suppression, bounces and the verification gate
All checks were successful
PR Checks / client-build (pull_request) Successful in 36s
Details
PR Checks / bot-tests (pull_request) Successful in 36s
Details
PR Checks / server-tests (pull_request) Successful in 5m12s
Details

Browse Source 
ENGAGEMENT.md Phase 9, closing gap G16. Two mechanisms decide that somebody in a
rule's audience does not get the mail, and they sit at deliberately different
points in the pipeline.

`engagement_suppressions` is checked at DELIVERY: an outbox row can sit through a
rule's `delay_seconds` grace window and an address can bounce inside it, so the
only correct check is the one taken immediately before the transport call — which
is also what produces the `status='suppressed'` row with no transport call at all.

The Phase 1b verification gate is applied at ENQUEUE, through a new optional
`registerDeliveryChannel({ eligible })` that only `email` declares. Filtering the
shared audience would have silenced the wrong sink: a rule spanning email and
in-app must still put an item in an unverified user's inbox. The excluded counts
reach `summary.ineligible` and the admin reach preview, which until now reported
an audience size that was never the number of people who would be mailed.

`bounceClassify.js` is the only thing that may write a `bounce` row, and it is
deliberately NOT `mailer.PERMANENT_CODES`. That set answers "is retrying
pointless?" and contains EAUTH and 554 — an auth failure and a relay-wide policy
refusal, neither of which is a fact about the recipient. Reusing it would mean one
stale SMTP password suppressing every address the worker touched, silently. The
classifier reads the RFC 3463 enhanced status first, falls back to a phrase match
only past a veto list and only for 550/551/553, and does not suppress anything it
is unsure about.

Scope is engagement rules only: resets, invites, verification and the contact form
still attempt, matching the posture passwordReset.controller.js already stated.

Found on the live rig, against a real MariaDB and a real SMTP conversation: a hard
bounce was being recorded as `failed`, so the Send Log's "Bounced" filter — a
status `engagement_sends` has carried since §4.5 — matched nothing and always
would have. It is now its own outcome; the outbox row stays `failed`, since that
ENUM has no `bounced` and a bounced row is one that finished unsuccessfully.

`address_masked` is this phase's one addition to §4.5's DDL. A hash-only table
cannot be operated — an operator cannot tell three typos from a whole domain
refusing mail — and the domain survives while the local part is destroyed, so the
column can never be read back as an address book.

- schema: `engagement_suppressions` (+ `address_masked`, `created_by`)
- `GET/POST/DELETE /api/v1/admin/engagement/suppressions`, and Admin → Engagement
  → Suppressions, the only way out of the list
- `sendNotification` returns `smtp: { code, responseCode, response }`
- 26 new tests; swagger, routes manifest and guards regenerated

Docs: RunicGateway/docs#191.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
claude
2026-08-31 10:47:34 -05:00
parent 87c4e71025
commit c208543044
22 changed files with 2210 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files

216
server/src/engagement/bounceClassify.js Normal file
View File

@@ -0,0 +1,216 @@
// ── Which send failures are facts about the RECIPIENT ───────────────────────
//
// ENGAGEMENT.md Phase 9. The suppression list's whole value is that an address on
// it is genuinely undeliverable; the moment it fills with addresses that were
// fine, an operator learns to ignore it and it may as well not exist. This file
// is the one place that judgement is made.
//
// **It is deliberately NOT `mailer.PERMANENT_CODES`, and reusing that set would
// have been a mass-suppression bug.** That set answers "is retrying pointless?"
// and holds `EAUTH` and `554` alongside `550` — an authentication failure and a
// relay-wide policy refusal. Both are permanent and neither says anything about
// the person: one wrong SMTP password would suppress every address the outbox
// worker touched before anybody noticed the mail had stopped. "Do not retry" and
// "this mailbox does not exist" are different questions, and this file only
// answers the second.
//
// **The primary signal is the enhanced status code (RFC 3463), not the reply
// code.** `550` alone is the catch-all every refusal arrives as; `5.1.1` means
// one specific thing — no such mailbox. Every relay worth configuring emits an
// enhanced code, so it is read first and, when present, decides on its own.
//
// **The fallback is narrow on purpose.** Without an enhanced code a phrase match
// is all that is left, and phrase matching is how a classifier quietly starts
// suppressing everything. So it applies only after the reply code has already
// narrowed the failure to the recipient address — 550, 551 and 553 are RFC 5321's
// recipient-address codes — and only for phrases that cannot mean anything else,
// with a veto list checked first. `552` (storage exceeded) and `554` (transaction
// failed) are excluded from even that: a full mailbox gets emptied, and a generic
// transaction failure is generic.
//
// Anything this file is unsure about is NOT suppressed. The cost of a false
// negative is mailing a dead address again next month; the cost of a false
// positive is a person who silently stops hearing from the deployment and has no
// way to find out.
// RFC 3463 subject.detail pairs that mean "this address will not accept mail,
// today or ever". Kept as strings because `5.1.10` and `5.1.1` are different
// codes and numeric parsing loses that.
const PERMANENT_RECIPIENT = new Set([
'1.1', // bad destination mailbox address — no such user
'1.2', // bad destination system address — the domain does not take mail
'1.3', // bad destination mailbox address syntax
'1.6', // mailbox has moved, no forwarding address
'1.10', // recipient address has a null MX (RFC 7505)
'2.1', // mailbox disabled, not accepting messages
])
// Enhanced subjects that are permanent but are NOT about the recipient. Listed
// rather than merely omitted, because each is a plausible-looking 5.x.y that a
// later edit would otherwise be tempted to add:
// 2.2 — mailbox full. Permanent-coded by some relays, emptied by every user.
// 7.x — policy. Our sending reputation, our SPF, our content; the recipient is
// the one party it is not about.
// 3.x — the destination MAIL SYSTEM is full or refusing. Not the mailbox.
// 5.x — protocol failure. A bug at one end or the other.
const NEVER_RECIPIENT_SUBJECTS = new Set(['3', '5', '7'])
// RFC 5321 reply codes that name the recipient address specifically. 554 is
// absent deliberately: "transaction failed" is what a relay reaches for when it
// does not want to say why, and it is the commonest shape of a content or policy
// rejection.
const RECIPIENT_REPLY_CODES = new Set([550, 551, 553])
// Phrases that only ever mean "no such mailbox", checked only once a reply code
// above has established the failure is about the address. Each is a substring of
// a real refusal from a widely deployed MTA (Postfix, Exim, Exchange, Google,
// Microsoft 365).
const NO_SUCH_MAILBOX = [
'user unknown',
'unknown user',
'no such user',
'no such recipient',
'unknown recipient',
'invalid recipient',
'recipient address rejected',
'recipient not found',
'address does not exist',
'does not exist',
'mailbox unavailable',
'mailbox not found',
'no mailbox',
'user does not exist',
'address rejected',
]
// Phrases that appear alongside the ones above and mean the opposite, checked
// FIRST. "Mailbox unavailable" is a substring of the sentence a relay sends when
// a mailbox is merely full, so a substring match with no veto list would read a
// temporary condition as a dead address.
const NOT_A_DEAD_MAILBOX = [
'full',
'quota',
'storage',
'temporar',
'try again',
'greylist',
'rate limit',
'too many',
'spam',
'blocked',
'blacklist',
'blocklist',
'reputation',
'policy',
'authentication',
'not authorized',
]
/**
* The enhanced status code in an SMTP response, as `{ class, subject, detail }`,
* or null.
*
* Anchored to the start of the line rather than searched for anywhere in it: a
* bounce that quotes another server's answer ("...said: 550 5.1.1...") contains
* two, and the one that matters is the one this relay just gave us. A free search
* finds whichever comes first, which is not the same thing.
*/
function parseEnhanced(response) {
if (!response) return null
const m = /^\s*(\d{3})[\s-]+(\d)\.(\d{1,3})\.(\d{1,3})\b/.exec(String(response))
if (!m) return null
return { class: m[2], subject: m[3], detail: m[4] }
}
/** The three-digit reply code, off the error object or out of the response text. */
function replyCode(err) {
const direct = Number(err && err.responseCode)
if (Number.isInteger(direct) && direct >= 400 && direct <= 599) return direct
const m = /^\s*(\d{3})\b/.exec(String((err && err.response) || ''))
return m ? Number(m[1]) : null
}
const lower = (s) => String(s || '').toLowerCase()
/**
* Should this send failure suppress the address?
*
* @param {object} err the error a transport's send threw, or an object carrying
* the `responseCode` / `response` / `code` lifted off one
* @returns {{ suppress: boolean, reason: string, evidence: string|null }}
*
* `reason` is populated on a refusal too, and that is not decoration: it becomes
* the send log's `detail`, so "not suppressed: 554 does not name the recipient
* address" is the line that stops somebody re-deriving this decision from an
* unexplained non-event six months from now.
*/
function classify(err) {
const e = err || {}
const response = e.response || e.message || ''
const enhanced = parseEnhanced(response)
const code = replyCode(e)
// No reply code at all means the failure happened before or outside the SMTP
// transaction: the connection, the credentials, the socket. Never the
// recipient. `EAUTH` lands here, which is the whole reason this file exists.
if (!code) {
return {
suppress: false,
reason: `no SMTP reply code (${e.code || 'transport failure'}); not a recipient failure`,
evidence: null,
}
}
if (code < 500) {
return { suppress: false, reason: `${code} is a temporary failure`, evidence: null }
}
if (enhanced) {
const pair = `${enhanced.subject}.${enhanced.detail}`
if (enhanced.class !== '5') {
return { suppress: false, reason: `enhanced status ${enhanced.class}.${pair} is not permanent`, evidence: null }
}
if (PERMANENT_RECIPIENT.has(pair)) {
return { suppress: true, reason: 'bounce', evidence: `5.${pair}` }
}
if (NEVER_RECIPIENT_SUBJECTS.has(enhanced.subject)) {
return {
suppress: false,
reason: `5.${pair} is about the server or our standing with it, not the address`,
evidence: `5.${pair}`,
}
}
// A permanent 5.x.y this file has no opinion on. Unknown means no.
return {
suppress: false,
reason: `5.${pair} is not a known recipient failure`,
evidence: `5.${pair}`,
}
}
// No enhanced code: the narrow fallback.
if (!RECIPIENT_REPLY_CODES.has(code)) {
return { suppress: false, reason: `${code} does not name the recipient address`, evidence: null }
}
const text = lower(response)
const veto = NOT_A_DEAD_MAILBOX.find((p) => text.includes(p))
if (veto) {
return { suppress: false, reason: `${code}, but the response says "${veto}"`, evidence: null }
}
const hit = NO_SUCH_MAILBOX.find((p) => text.includes(p))
if (hit) {
return { suppress: true, reason: 'bounce', evidence: `${code} "${hit}"` }
}
return { suppress: false, reason: `${code} with no enhanced status and no recognised reason`, evidence: null }
}
module.exports = {
classify,
parseEnhanced,
replyCode,
PERMANENT_RECIPIENT,
NEVER_RECIPIENT_SUBJECTS,
RECIPIENT_REPLY_CODES,
NO_SUCH_MAILBOX,
NOT_A_DEAD_MAILBOX,
}

52
server/src/engagement/channels.js
View File

@@ -53,6 +53,24 @@ const isMode = (value) => MODES.includes(value)
* without one is declared but not yet deliverable, which is exactly what
* `inapp` is until Phase 7; the worker finishes such a row `failed` and
* says so in the send log rather than pretending it was sent.
* @param {(userIds: number[]) => Promise<{userIds: number[], excluded: object}>} [def.eligible]
* Phase 9. Narrow an already-resolved audience to the users this channel
* may write an outbox row for, and say how many it dropped and why.
*
* **It exists so the engine can stay channel-agnostic.** The verification
* gate is an email fact — an unverified address is a reason not to mail
* somebody and no reason at all not to put an item in their inbox — and a
* rule may name both channels. Filtering the shared audience before the
* per-channel loop would have silenced the wrong sink; an `if (channel ===
* 'email')` in `engine.js` would have put a channel's rule inside the
* generic engine. This is the seam that is neither.
*
* Distinct from `deliver`'s refusals on purpose: this runs at ENQUEUE and
* is for standing properties of a user (is this address verified), which
* are stable across a delay window and are worth not writing a row for.
* A suppression is not one of those — it can appear between the enqueue
* and the send — so it is checked in `deliver`, where it produces a
* `suppressed` row in the send log the acceptance criterion asks for.
*/
function registerDeliveryChannel(def) {
if (!def || typeof def !== 'object') throw new Error('registerDeliveryChannel: definition required')
@@ -81,7 +99,7 @@ function registerDeliveryChannel(def) {
// silently never delivers — the failure Phase 3 deferred the whole behavioural
// half to avoid freezing, and the one the worker's "no delivery implementation
// yet" branch would report as if it were by design.
for (const fn of ['addressFor', 'deliver']) {
for (const fn of ['addressFor', 'deliver', 'eligible']) {
if (def[fn] !== undefined && typeof def[fn] !== 'function') {
throw new Error(`registerDeliveryChannel(${id}): ${fn} must be a function`)
}
@@ -96,6 +114,7 @@ function registerDeliveryChannel(def) {
supportsDigest,
addressFor: def.addressFor,
deliver: def.deliver,
eligible: def.eligible,
})
return id
}
@@ -103,13 +122,14 @@ function registerDeliveryChannel(def) {
/**
* Every channel, in registration order. The preferences screen's column set.
*
* **Declarative fields only** — `addressFor` and `deliver` are stripped. This is
* what a route serializes, and a function on an object bound for `res.json` is a
* key that silently disappears rather than an error; keeping the boundary here
* means the API shape is decided in one place instead of by JSON.stringify.
* **Declarative fields only** — `addressFor`, `deliver` and `eligible` are
* stripped. This is what a route serializes, and a function on an object bound
* for `res.json` is a key that silently disappears rather than an error; keeping
* the boundary here means the API shape is decided in one place instead of by
* JSON.stringify.
*/
const all = () =>
[...channels.values()].map(({ addressFor, deliver, ...declared }) => ({ ...declared }))
[...channels.values()].map(({ addressFor, deliver, eligible, ...declared }) => ({ ...declared }))
/** Just the ids. */
const ids = () => [...channels.keys()]
@@ -137,6 +157,25 @@ const modesFor = (id) => {
/** Is `mode` a mode this channel accepts? The gate on every preference write. */
const acceptsMode = (id, mode) => modesFor(id).includes(mode)
/**
* Narrow an audience to the users this channel may enqueue for (Phase 9).
*
* The default for a channel that declares no `eligible` is "everyone the
* audience resolved to", which is what every channel but email does. It lives
* here rather than at each call site so the two consumers — the engine and the
* admin reach preview — cannot answer the question differently, which is exactly
* how a preview comes to promise a number the engine will not deliver.
*/
async function eligibleFor(id, userIds) {
const c = channels.get(id)
if (!c || typeof c.eligible !== 'function') return { userIds: userIds.slice(), excluded: {} }
const result = await c.eligible(userIds)
return {
userIds: (result && result.userIds) || [],
excluded: (result && result.excluded) || {},
}
}
// Test-only: the registry is module-level state.
function _reset() {
channels.clear()
@@ -153,5 +192,6 @@ module.exports = {
defaultMode,
modesFor,
acceptsMode,
eligibleFor,
_reset,
}

4
server/src/engagement/coreChannels.js
View File

@@ -66,6 +66,10 @@ const CHANNELS = [
// time (§4.2b), which is a different delivery path rather than a batched one.
addressFor: emailChannel.addressFor,
deliver: emailChannel.deliver,
// Phase 9: the only channel that declares one. The Phase 1b verification
// gate is an email fact, and this is the seam that keeps it out of the
// generic engine - see channels.js's `eligible` docs.
eligible: emailChannel.eligible,
},
{
id: 'inapp',

116
server/src/engagement/emailChannel.js
View File

@@ -29,11 +29,12 @@
// `teamName` (a display string the cooldown keys on) and the scope is `team:12`.
// A Team renamed between the mail and the click must not orphan the link in it.
const crypto = require('crypto')
const rulesDb = require('../model/engagement/engagementRules.db')
const recipients = require('../model/engagement/engagementRecipients.db')
const templates = require('./templates')
const projection = require('./projection')
const suppressions = require('./suppressions')
const settings = require('../model/settings/settings.model')
const unsubscribeToken = require('../utils/unsubscribeToken')
const log = require('../utils/logger')('engagement')
@@ -54,12 +55,10 @@ const DEFAULT_TEMPLATE = 'notify.event'
const baseUrl = () => templates.baseUrl()
// Lower-cased first: a bounce reported for "Darrow@example.com" has to match the
// row written for "darrow@example.com", and a hash of two spellings is two
// different rows. The local part is technically case-sensitive per RFC 5321 and
// no relay anybody deploys treats it that way.
const hashAddress = (address) =>
crypto.createHash('sha256').update(String(address).trim().toLowerCase()).digest('hex')
// Re-exported rather than defined here since Phase 9: the send log's hash and
// the suppression list's key have to be the same function or a bounce never finds
// the row it belongs to. `suppressions.js` owns it, next to the masking.
const hashAddress = suppressions.hashAddress
/**
* The two unsubscribe URLs for one recipient of one scope, or nulls.
@@ -94,6 +93,52 @@ function unsubscribeUrls(userId, scopeKey) {
/** Where this channel would send to, or null. */
const addressFor = (userId) => recipients.addressFor(userId)
/**
* Narrow an audience to the users this channel may write an outbox row for
* (Phase 9, decision 4).
*
* **One gate, and it is the Phase 1b verification setting.** With
* `email_verification_required` on, a user whose address is unverified is
* excluded here rather than refused at delivery, and the org lead settled it that
* way for two reasons. It is a STANDING property — unlike a suppression, which
* can appear inside a `delay_seconds` window and therefore has to be re-checked
* at send time — so the outbox row would be written only to be thrown away. And a
* deployment that upgraded before verifying anybody has an audience that is
* almost entirely unverified: excluding at delivery would write a `suppressed`
* row per person per rule firing, which is a send log nobody can read.
*
* The count comes back so the admin reach preview can say "1,204 excluded:
* unverified" instead of quietly promising a number the engine will not deliver.
*
* **It fails OPEN, and the try/catch is load-bearing rather than defensive
* habit.** `settings.isEmailVerificationRequired` swallows its own errors and
* answers `off`, but `unverifiedAmong` does not, and an uncaught throw here does
* not fail one recipient — `applyRule` awaits this before the per-user loop, so
* it would abandon the whole rule for every channel it names. A database having
* a bad minute would become a rule that silently sent nothing, with a clean send
* log and nothing in the outbox to retry. Same direction as the suppression
* check, for the same reason: the recoverable mistake is mail going out, not mail
* silently stopping.
*/
async function eligible(userIds) {
const list = userIds || []
if (!list.length) return { userIds: [], excluded: {} }
try {
if (!(await settings.isEmailVerificationRequired())) {
return { userIds: list.slice(), excluded: {} }
}
const unverified = await recipients.unverifiedAmong(list)
if (!unverified.size) return { userIds: list.slice(), excluded: {} }
return {
userIds: list.filter((id) => !unverified.has(Number(id))),
excluded: { unverified: unverified.size },
}
} catch (err) {
log.error('verification gate could not be evaluated; not excluding anyone', { message: err.message })
return { userIds: list.slice(), excluded: {} }
}
}
/**
* Deliver one claimed outbox row.
*
@@ -130,11 +175,60 @@ async function deliver(row) {
log.debug('template variables had no value', { key, missing: rendered.missing })
}
// **The suppression check is HERE and not at enqueue** (Phase 9). An outbox
// row can sit through a `delay_seconds` grace window, and an address can hard
// bounce inside it — so the only check that can be correct is the one taken
// immediately before the transport call. It is also the check the acceptance
// criterion describes: a `suppressed` row in the send log, and no transport
// call at all.
const blocked = await suppressions.isSuppressed(to.address)
if (blocked) {
return {
ok: false,
suppressed: true,
detail: `address is suppressed (${blocked.reason})`,
addressHash: hashAddress(to.address),
}
}
const result = await mailer().sendNotification({ to: to.address, rendered, ...unsub })
// A failed send is where a hard bounce enters the system on SMTP, and the
// reason it is worth catching rather than waiting for an API provider: a
// single-recipient send refused at RCPT TO is a synchronous 5.1.1, which is
// the most valuable deliverability signal there is and it was already being
// thrown away. `considerFailure` is narrow — see bounceClassify.js — and its
// note goes into the log on BOTH outcomes, so "this failed and was not
// suppressed" says why.
if (result && !result.ok && result.smtp) {
const verdict = await suppressions.considerFailure({ address: to.address, error: result.smtp })
// A bounce is terminal by definition. Overriding `retry` matters because
// `PERMANENT_CODES` does not contain every code that can carry a 5.1.x, so
// without this a genuine dead mailbox could still be retried four more
// times — each one another refusal on our record with the relay.
if (verdict.suppressed) {
return {
ok: false,
retry: false,
// `engagement_sends.status` has carried 'bounced' since §4.5 and
// nothing wrote it until here, so the Send Log's "Bounced" filter
// matched nothing — the live rig is what showed that. It is a distinct
// status rather than a flavour of 'failed' because the two need
// different actions: a failure means look at the relay, and a bounce
// means that person's address is gone.
bounced: true,
transport: result.transport,
detail: `${result.detail} — ${verdict.note}`,
addressHash: hashAddress(to.address),
}
}
return { ...result, detail: `${result.detail} — ${verdict.note}`, addressHash: hashAddress(to.address) }
}
// The send log stores a sha256 of the address and never the address itself
// (schema.sql): enough to correlate a bounce in Phase 9, useless as a mailing
// list. Attached on every outcome, because a failure is exactly the row a
// bounce would need to be matched against.
// (schema.sql): enough to correlate a bounce, useless as a mailing list.
// Attached on every outcome, because a failure is exactly the row a bounce
// would need to be matched against.
return { ...result, addressHash: hashAddress(to.address) }
} catch (err) {
// See the header: a throw here would be retried as if it were the relay's
@@ -144,4 +238,4 @@ async function deliver(row) {
}
}
module.exports = { addressFor, deliver, unsubscribeUrls, hashAddress, DEFAULT_TEMPLATE }
module.exports = { addressFor, eligible, deliver, unsubscribeUrls, hashAddress, DEFAULT_TEMPLATE }

25
server/src/engagement/engine.js
View File

@@ -120,7 +120,18 @@ async function subscribedTo(userIds, streamId, channel, scopeKey = null) {
* for tests; it is not read by the caller for control flow.
*/
async function applyRule(rule, event, now) {
const summary = { ruleId: rule.id, enqueued: 0, deduped: 0, cooled: 0, capped: 0, skipped: null }
const summary = {
ruleId: rule.id,
enqueued: 0,
deduped: 0,
cooled: 0,
capped: 0,
// Phase 9: people a CHANNEL refused to enqueue for, by reason. Counted
// separately from `cooled` and `capped` because those are the engine holding
// a message back and this is a channel saying it cannot carry one at all.
ineligible: {},
skipped: null,
}
if (!conditions.evaluate(rule.conditions, event.data)) {
summary.skipped = 'conditions'
@@ -176,7 +187,17 @@ async function applyRule(rule, event, now) {
const dueAt = new Date(now.getTime() + Math.max(0, rule.delay_seconds) * 1000)
for (const channel of live) {
const eligible = await subscribedTo(resolved.userIds, event.triggerId, channel, event.scopeKey)
// Phase 9, and it runs BEFORE the preference filter rather than after. Both
// orders reach the same recipients; this one costs one query against the
// narrower set only when the channel declares an `eligible` at all, and it
// means `summary.ineligible` counts people the CHANNEL cannot reach rather
// than people who happened to also be opted in. Channels that declare none —
// push and in-app — pass straight through.
const gated = await channels.eligibleFor(channel, resolved.userIds)
for (const [why, n] of Object.entries(gated.excluded)) {
summary.ineligible[why] = (summary.ineligible[why] || 0) + n
}
const eligible = await subscribedTo(gated.userIds, event.triggerId, channel, event.scopeKey)
for (const userId of eligible) {
if (budget <= 0) {
summary.capped += 1

160
server/src/engagement/suppressions.js Normal file
View File

@@ -0,0 +1,160 @@
// ── The suppression list ───────────────────────────────────────────────────
//
// ENGAGEMENT.md G16, Phase 9. Addresses this deployment has stopped mailing,
// and the two questions asked of them: "may I send to this one?" at delivery
// time, and "why did this one stop?" on the admin screen.
//
// **Scope: the engagement email channel only** (Phase 9 decision 2). A password
// reset, an invite, a verification mail and the contact form are all
// user-INITIATED and still attempt, exactly as they still attempt to an
// unverified address (`passwordReset.controller.js`). The posture is the same one
// that file already states: a background system's opinion about an address must
// not be able to lock somebody out of their own account. One reset to a dead
// mailbox is not a reputation problem; a rule mailing three thousand people every
// week is, and that is what this list guards.
//
// **The table holds a hash and a mask, never an address.** The hash is what
// correlates a bounce back to an `engagement_sends` row (Phase 6 was already
// writing `address_hash` on every outcome for this). The mask —
// `d***@example.com` — is Phase 9's one addition to §4.5's DDL and exists because
// a screen of sha256 digests cannot be operated: an operator has to be able to
// see that a whole domain is refusing mail, and to find the person who fixed
// their mailbox and let them back in. The local part is DESTROYED rather than
// shortened, so the column cannot be turned back into an address book.
//
// **Nothing here writes a suppression from "the send failed".** What may write
// one is `bounceClassify.classify`, which is a much narrower question — see that
// file's header for why reusing `mailer.PERMANENT_CODES` would have suppressed
// every address the moment an SMTP password went stale.
const crypto = require('crypto')
const db = require('../model/engagement/engagementSuppressions.db')
const bounceClassify = require('./bounceClassify')
const log = require('../utils/logger')('engagement')
const REASONS = ['bounce', 'complaint', 'manual', 'unverified']
/**
* The key an address is stored under.
*
* Lower-cased first, and that matters more here than anywhere else in the
* subsystem: a bounce reported for `Darrow@example.com` has to find the row
* written for `darrow@example.com`, and a hash of two spellings is two rows that
* never meet. RFC 5321 says the local part is technically case-sensitive; no
* relay anybody deploys treats it that way.
*/
const hashAddress = (address) =>
crypto.createHash('sha256').update(String(address).trim().toLowerCase()).digest('hex')
/**
* `darrow@example.com` → `d***@example.com`. Null for anything that is not an
* address.
*
* The domain survives intact because domain-level patterns are the signal an
* operator is actually looking for — "everything to this company is bouncing" is
* a different problem from three people mistyping their own address, and only the
* domain distinguishes them.
*
* The first character of the local part survives only when there are at least
* three, which is not fussiness: for a two-letter local part, one revealed
* character plus the domain is most of the address.
*/
function maskAddress(address) {
const s = String(address || '').trim()
const at = s.lastIndexOf('@')
if (at <= 0 || at === s.length - 1) return null
const local = s.slice(0, at)
const domain = s.slice(at + 1).toLowerCase()
const head = local.length >= 3 ? local[0].toLowerCase() : ''
return `${head}***@${domain}`.slice(0, 190)
}
/** Is this address suppressed on this channel? */
async function isSuppressed(address, channel = 'email') {
if (!address) return null
try {
return await db.get(hashAddress(address), channel)
} catch (err) {
// Fail OPEN, and the direction is deliberate. A database that cannot answer
// "is this suppressed" must not stop the deployment's mail; the failure mode
// it would otherwise produce is total silence with a clean send log, which is
// exactly G22's shape. Mailing one dead address during an outage is the
// cheaper mistake.
log.error('suppression check failed; sending anyway', { message: err.message })
return null
}
}
/**
* Suppress an address. Returns true when this call created the row.
*
* `reason` is validated rather than trusted: it is an ENUM in the schema, so an
* unknown value is a 500 from the driver at the worst possible moment (inside a
* failure handler), and the callers include an admin route.
*/
async function suppress({ address, reason, detail = null, channel = 'email', createdBy = null }) {
if (!address) return false
if (!REASONS.includes(reason)) throw new Error(`suppress: unknown reason "${reason}"`)
const created = await db.add({
address_hash: hashAddress(address),
address_masked: maskAddress(address),
channel,
reason,
detail,
created_by: createdBy,
})
if (created) {
// Masked, never the address — the same rule every other log line in this
// subsystem follows. It is logged at all because an address dropping off the
// mailing list is the kind of change an operator finds out about weeks later
// otherwise.
log.info('address suppressed', { address: maskAddress(address), reason, channel })
}
return created
}
/** Un-suppress. Returns true when a row was removed. */
async function unsuppress(address, channel = 'email') {
if (!address) return false
const removed = await db.remove(hashAddress(address), channel)
if (removed) log.info('suppression lifted', { address: maskAddress(address), channel })
return removed
}
/**
* Consider a failed send for suppression, and say what was decided.
*
* The seam between a delivery failure and this list, and the only one — nothing
* else in the codebase writes a `bounce` row. Called from `emailChannel.deliver`
* with the error the transport threw.
*
* @returns {Promise<{suppressed: boolean, note: string}>} `note` goes into the
* send log's detail, on both outcomes.
*/
async function considerFailure({ address, error, channel = 'email' }) {
const verdict = bounceClassify.classify(error)
if (!verdict.suppress) {
return { suppressed: false, note: `not suppressed (${verdict.reason})` }
}
try {
const detail = verdict.evidence ? `hard bounce: ${verdict.evidence}` : 'hard bounce'
await suppress({ address, reason: 'bounce', detail, channel })
return { suppressed: true, note: detail }
} catch (err) {
// A failure to record the suppression must not change how the send itself is
// reported. The mail failed either way, and that is the row the log owes.
log.error('could not record a suppression', { message: err.message })
return { suppressed: false, note: `hard bounce, not recorded: ${err.message}` }
}
}
module.exports = {
hashAddress,
maskAddress,
isSuppressed,
suppress,
unsuppress,
considerFailure,
REASONS,
}

74
server/src/model/engagement/engagementRecipients.db.js
View File

@@ -132,11 +132,14 @@ const storedModes = async (userIds, streamId, channel) => {
* between the emit and the send is exactly the case this catches. The cost is one
* primary-key lookup on a path that is about to open an SMTP conversation.
*
* **It does not gate on `email_verified`.** Whether an unverified address may
* receive opt-in mail is §7.1 Q1's narrower half, and it is a Phase 9 decision
* with the suppression list in front of it; deciding it here by accident would
* mean every deployment that upgraded before verifying its users stopped mailing
* them.
* **It still does not gate on `email_verified`, and Phase 9 kept it that way.**
* §7.1 Q1's narrower half was settled by the org lead 2026-08-31: the gate
* excludes an unverified user at ENQUEUE, in `emailChannel.eligible`, so no
* outbox row is written and the admin reach preview can say how many were
* dropped. Adding the same condition here as well would look like defence in
* depth and would in fact be a second, invisible answer to the question — this
* function is also what a password reset would reach if it ever routed through
* the channel, and reset mail is deliberately ungated (`passwordReset.controller.js`).
*/
const addressFor = async (userId) => {
const rows = await query(
@@ -147,4 +150,63 @@ const addressFor = async (userId) => {
return rows.length ? { address: rows[0].email } : null
}
module.exports = { active, staff, subscribers, filterActive, storedModes, addressFor, MAX_AUDIENCE }
/**
* Which of these users hold an UNVERIFIED address - the set the Phase 1b gate
* excludes when it is on (Phase 9).
*
* A user with no address at all is in this set, and that is not incidental: the
* gate's question is "may we mail this person", and nowhere to send is a stronger
* no than an unconfirmed somewhere. `addressFor` refuses them at delivery either
* way; including them here is what stops an outbox row being written for a send
* that is already known to be impossible.
*
* One query for a whole audience. The engine calls it once per rule per event
* with up to MAX_AUDIENCE ids, so a per-user lookup would be five thousand round
* trips on the path that is supposed to be the cheap one.
*/
const unverifiedAmong = async (userIds) => {
const wanted = [...new Set(userIds.map(Number).filter((n) => Number.isInteger(n) && n > 0))]
if (!wanted.length) return new Set()
const capped = wanted.slice(0, MAX_AUDIENCE)
const rows = await query(
`SELECT id FROM users
WHERE id IN (${marks(capped)})
AND (email_verified = 0 OR email IS NULL OR email = '')`,
capped,
)
return new Set(ids(rows))
}
/**
* The addresses for a set of active users, as a Map - what the admin reach
* preview hashes to count how many of them are suppressed.
*
* It returns PLAINTEXT, which is the one thing this subsystem otherwise avoids,
* and there is no way around it: a suppression is keyed on the sha256 of an
* address, so answering "how many of these people are suppressed" requires
* hashing each one. The caller (`engagement.controller`) hashes immediately and
* returns only a count - no route ever serializes what this returns.
*/
const addressesFor = async (userIds) => {
const wanted = [...new Set(userIds.map(Number).filter((n) => Number.isInteger(n) && n > 0))]
if (!wanted.length) return new Map()
const capped = wanted.slice(0, MAX_AUDIENCE)
const rows = await query(
`SELECT id, email FROM users
WHERE id IN (${marks(capped)}) AND status = 'active' AND email IS NOT NULL AND email <> ''`,
capped,
)
return new Map(rows.map((r) => [Number(r.id), r.email]))
}
module.exports = {
active,
staff,
subscribers,
filterActive,
storedModes,
addressFor,
unverifiedAmong,
addressesFor,
MAX_AUDIENCE,
}

139
server/src/model/engagement/engagementSuppressions.db.js Normal file
View File

@@ -0,0 +1,139 @@
const { query } = require('../../utils/db')
/**
* The suppression list (ENGAGEMENT.md G16, Phase 9).
*
* Every function here takes an ALREADY HASHED address. Hashing lives in
* `engagement/suppressions.js` beside the masking, so the two can never disagree
* about what a row for one address looks like; this file only reads and writes.
*/
/** Is this address suppressed on this channel? The row, or null. */
const get = async (addressHash, channel = 'email') => {
const rows = await query(
'SELECT * FROM engagement_suppressions WHERE address_hash = ? AND channel = ?',
[addressHash, channel],
)
return rows.length ? rows[0] : null
}
/**
* Suppress an address, or leave an existing row exactly as it is.
*
* `INSERT IGNORE`, deliberately, rather than an upsert. The FIRST reason an
* address was suppressed is the true one and the one an operator needs: an
* address that hard-bounced in March and was then manually re-added in June
* should still read `bounce`, because that is the fact that explains the mail
* stopping. An upsert would let the most recent write overwrite the diagnosis.
*
* @returns {Promise<boolean>} true when this call created the row
*/
const add = async (entry) => {
const result = await query(
`INSERT IGNORE INTO engagement_suppressions
(address_hash, address_masked, channel, reason, detail, created_by)
VALUES (?, ?, ?, ?, ?, ?)`,
[
entry.address_hash,
entry.address_masked ?? null,
entry.channel || 'email',
entry.reason,
entry.detail ? String(entry.detail).slice(0, 500) : null,
entry.created_by ?? null,
],
)
return Number(result.affectedRows || 0) > 0
}
/**
* Un-suppress an address. The one way out of this table, which is why the admin
* screen exists at all (Phase 9 decision 3).
*
* @returns {Promise<boolean>} true when a row was removed
*/
const remove = async (addressHash, channel = 'email') => {
const result = await query(
'DELETE FROM engagement_suppressions WHERE address_hash = ? AND channel = ?',
[addressHash, channel],
)
return Number(result.affectedRows || 0) > 0
}
/** WHERE-clause builder shared by `list` and `count`, so the two cannot disagree. */
const filters = ({ reason = null, channel = null, search = null } = {}) => {
const where = []
const params = []
if (reason) {
where.push('reason = ?')
params.push(reason)
}
if (channel) {
where.push('channel = ?')
params.push(channel)
}
if (search) {
// Against the MASKED column only. Searching the hash would need the caller to
// hash first, which makes a partial search impossible, and there is no
// plaintext column to search — that is the point of the table. A domain
// ("example.com") is what an operator actually types here, and the mask keeps
// the domain intact precisely so this works.
where.push('address_masked LIKE ?')
params.push(`%${String(search).slice(0, 120)}%`)
}
return { clause: where.length ? `WHERE ${where.join(' AND ')}` : '', params }
}
/** The admin list, newest first. */
const list = (opts = {}) => {
const { clause, params } = filters(opts)
return query(
`SELECT * FROM engagement_suppressions ${clause} ORDER BY created_at DESC, address_hash LIMIT ? OFFSET ?`,
[...params, opts.limit || 50, opts.offset || 0],
)
}
/** How many rows match the same filters — the paged screen's total. */
const count = async (opts = {}) => {
const { clause, params } = filters(opts)
const [row] = await query(`SELECT COUNT(*) AS n FROM engagement_suppressions ${clause}`, params)
return Number(row?.n || 0)
}
/**
* How many suppressions per reason — the summary the screen leads with.
*
* A count by reason is the difference between "eleven addresses are suppressed"
* and "eleven addresses hard-bounced", and only the second tells an operator
* whether to go and look at their relay.
*/
const countsByReason = async () => {
const rows = await query(
'SELECT reason, COUNT(*) AS n FROM engagement_suppressions GROUP BY reason',
)
const out = {}
for (const r of rows) out[r.reason] = Number(r.n || 0)
return out
}
/**
* The subset of these hashes that is suppressed, as a Set.
*
* One query for a whole audience rather than one per recipient. Not used by the
* delivery path — that checks a single address at send time, after the delay
* window, which is the only check that can be correct — but by the admin reach
* preview, which is asked about thousands of users at once and must not become
* thousands of round trips.
*/
const suppressedAmong = async (addressHashes, channel = 'email') => {
const wanted = [...new Set((addressHashes || []).filter(Boolean))]
if (!wanted.length) return new Set()
const marks = wanted.map(() => '?').join(', ')
const rows = await query(
`SELECT address_hash FROM engagement_suppressions
WHERE channel = ? AND address_hash IN (${marks})`,
[channel, ...wanted],
)
return new Set(rows.map((r) => r.address_hash))
}
module.exports = { get, add, remove, list, count, countsByReason, suppressedAmong }

151
server/src/router/v1/admin/engagement.controller.js
View File

@@ -32,6 +32,8 @@ const segments = require('../../../model/engagement/engagementSegments.model')
const recipients = require('../../../model/engagement/engagementRecipients.db')
const templates = require('../../../model/engagement/engagementTemplates.model')
const sendsDb = require('../../../model/engagement/engagementSends.db')
const suppressionsDb = require('../../../model/engagement/engagementSuppressions.db')
const suppressions = require('../../../engagement/suppressions')
// The lattice, flattened for a client: for each ceiling, the ones a rule may
// choose under it. Served with the catalog rather than hardcoded in the admin
@@ -245,6 +247,37 @@ exports.deleteSegment = async (req, res, next) => {
// ── Reach preview ──────────────────────────────────────────────────────────
/**
* How many of a resolved audience would actually receive an email, and what
* removed the rest (Phase 9).
*
* The two mechanisms are asked in the order the engine applies them, and the
* order is what makes the numbers add up: the verification gate runs at enqueue,
* so a user it excludes is never checked for suppression, and counting both
* independently would double-count anybody who is unverified AND bounced.
*
* **It never returns an address.** Addresses are read only to hash them, and only
* the counts leave this function — the send log route already refuses to return
* `address_hash` for exactly this reason, and a reach preview that leaked a list
* would be the same hole through a different door.
*/
async function emailReach(userIds) {
const gated = await channels.eligibleFor('email', userIds)
const addresses = await recipients.addressesFor(gated.userIds)
const hashes = [...addresses.values()].map((a) => suppressions.hashAddress(a))
const blocked = await suppressionsDb.suppressedAmong(hashes)
return {
// Everyone the audience resolved to who is not excluded by the gate and is
// not suppressed. Users with no address at all are already out: the gate
// drops them when it is on, and `addressesFor` does not return them when it
// is off, so they never reach the count either way.
deliverable: Math.max(0, addresses.size - blocked.size),
excluded: gated.excluded,
suppressed: blocked.size,
}
}
/**
* GET /api/v1/admin/engagement/audience-preview
*
@@ -270,6 +303,16 @@ exports.deleteSegment = async (req, res, next) => {
* at all. Without it the editor shows a healthy count beside a save the server
* will refuse, which reads as a bug in the save rather than as the G24 ceiling
* doing its job.
*
* **A fourth arrived with Phase 9: `email`.** `count` is how many people the
* audience resolves to, and that has never been how many will get a mail — the
* verification gate drops unverified users at enqueue and the suppression list
* drops bounced addresses at send. An operator reading "3,000" beside a rule that
* will mail 1,796 people has been told something false by a screen whose only job
* is that number, so the breakdown is computed the same way the engine computes
* it: `channels.eligibleFor('email', ...)` is the identical call `applyRule`
* makes. It is reported for the email channel only because it is the only channel
* either mechanism applies to.
*/
exports.previewAudience = async (req, res, next) => {
try {
@@ -297,6 +340,7 @@ exports.previewAudience = async (req, res, next) => {
dormant: resolved.dormant,
reason: resolved.reason,
permitted: triggerId && resolved.ceiling ? audiences.permitted(triggerId, resolved.ceiling) : null,
email: await emailReach(resolved.userIds),
})
} catch (err) {
next(err)
@@ -444,3 +488,110 @@ exports.listSends = async (req, res, next) => {
next(err)
}
}
// ── Suppressions (Phase 9) ─────────────────────────────────────────────────
//
// The one surface that can take an address OUT of the suppression list, which is
// why it exists at all: a hard bounce is written by a background worker with no
// human in the loop, and without a way back a mistyped-then-corrected mailbox is
// silenced permanently.
//
// **The list returns `address_masked`, never `address_hash`.** The send log route
// above strips the hash for a stated reason — shipping a sha256 of every address
// on the deployment to a browser is an offline dictionary attack waiting to be
// run — and the same reasoning applies twice over here, where the rows are
// exactly the addresses somebody would most want to confirm. The mask is what an
// operator can act on and is not reversible.
/** GET /api/v1/admin/engagement/suppressions */
exports.listSuppressions = async (req, res, next) => {
try {
const limit = Math.min(Math.max(Number(req.query.limit) || 50, 1), 200)
const offset = Math.max(Number(req.query.offset) || 0, 0)
const reason = typeof req.query.reason === 'string' ? req.query.reason : null
if (reason && !suppressions.REASONS.includes(reason)) {
return res.status(400).json({ message: `reason must be one of ${suppressions.REASONS.join(', ')}` })
}
const filters = {
reason,
channel: typeof req.query.channel === 'string' ? req.query.channel : null,
search: typeof req.query.search === 'string' ? req.query.search.trim() || null : null,
}
const [rows, total, byReason] = await Promise.all([
suppressionsDb.list({ ...filters, limit, offset }),
suppressionsDb.count(filters),
// Unfiltered on purpose: it is the summary strip above the table, and a
// count that moved with the filter would say "0 bounces" while the operator
// was looking at the manual ones.
suppressionsDb.countsByReason(),
])
res.json({
suppressions: rows.map(({ address_hash: _hash, ...row }) => row),
total,
limit,
offset,
byReason,
reasons: suppressions.REASONS,
})
} catch (err) {
next(err)
}
}
/**
* POST /api/v1/admin/engagement/suppressions
*
* Suppress an address by hand — the operator-side half of a bounce they were
* told about out of band (a person emailing to say "stop", a relay's dashboard).
*
* The reason is forced to `manual` rather than taken from the body. An admin
* typing an address is not evidence of a bounce or a complaint, and a list where
* `reason` sometimes means "the relay said so" and sometimes means "somebody
* chose this word" cannot be used to diagnose anything.
*/
exports.createSuppression = async (req, res, next) => {
try {
const address = typeof req.body?.address === 'string' ? req.body.address.trim() : ''
// The same shape check the rest of the codebase uses for an address, and no
// more: this is not a deliverability test, it is a guard against storing a
// hash of a typo that can never be matched or found again.
if (!address || !/^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(address)) {
return res.status(400).json({ message: 'A valid email address is required' })
}
const detail = typeof req.body?.detail === 'string' ? req.body.detail.slice(0, 500) : null
const created = await suppressions.suppress({
address,
reason: 'manual',
detail,
createdBy: req.user?.id ?? null,
})
// 200 rather than 409 for an address already on the list: the operator asked
// for it to be suppressed and it is, which is the outcome they wanted. `created`
// says which of the two happened, so the screen can say "already suppressed"
// without it reading as a failure.
res.status(created ? 201 : 200).json({ created, address: suppressions.maskAddress(address) })
} catch (err) {
next(err)
}
}
/**
* DELETE /api/v1/admin/engagement/suppressions
*
* Un-suppress. The address goes in the BODY rather than the path, and that is
* not a REST preference: a path parameter lands in the access log, the browser's
* history and any proxy in front of the deployment, and this one is a real
* address belonging to a real person. The hash cannot be used instead — the
* screen never receives one.
*/
exports.deleteSuppression = async (req, res, next) => {
try {
const address = typeof req.body?.address === 'string' ? req.body.address.trim() : ''
if (!address) return res.status(400).json({ message: 'An email address is required' })
const removed = await suppressions.unsuppress(address, req.body?.channel || 'email')
if (!removed) return res.status(404).json({ message: 'That address is not suppressed' })
res.json({ removed: true })
} catch (err) {
next(err)
}
}

50
server/src/router/v1/admin/engagement.router.js
View File

@@ -331,4 +331,54 @@ engagementRouter.get(
controller.listSends,
)
// -- Suppressions (Phase 9) ------------------------------------------------
engagementRouter.get(
'/suppressions',
// #swagger.tags = ['Admin - Engagement']
// #swagger.summary = 'Addresses this deployment has stopped mailing'
// #swagger.description = 'G16. Rows carry `address_masked` (`d***@example.com`) and never `address_hash` - the same rule the send log follows, and for the same reason: a sha256 of every address on the deployment, handed to a browser, is an offline dictionary attack. The mask keeps the domain intact so a whole-domain delivery failure is visible, and destroys the local part so the list cannot be turned back into an address book. `byReason` is deliberately unfiltered - it is the summary strip above the table.'
// #swagger.security = [{ "cookieAuth": [] }, { "bearerAuth": [] }]
// #swagger.parameters['limit'] = { in: 'query', description: 'Page size, 1-200 (default 50)', required: false, schema: { type: 'integer' } }
// #swagger.parameters['offset'] = { in: 'query', description: 'Rows to skip', required: false, schema: { type: 'integer' } }
// #swagger.parameters['reason'] = { in: 'query', description: 'bounce, complaint, manual or unverified', required: false, schema: { type: 'string' } }
// #swagger.parameters['channel'] = { in: 'query', description: 'Only this channel (default: all)', required: false, schema: { type: 'string' } }
// #swagger.parameters['search'] = { in: 'query', description: 'Substring of the masked address - a domain is what this is for', required: false, schema: { type: 'string' } }
/* #swagger.responses[200] = { description: 'One page of the list, with per-reason totals', content: { "application/json": { schema: { type: "object", properties: { suppressions: { type: "array", items: { type: "object", additionalProperties: true } }, total: { type: "integer" }, limit: { type: "integer" }, offset: { type: "integer" }, byReason: { type: "object", additionalProperties: { type: "integer" } }, reasons: { type: "array", items: { type: "string" } } } } } } } */
/* #swagger.responses[400] = { description: 'Unknown reason', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
/* #swagger.responses[403] = { description: 'Not an admin', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
adminOnly,
controller.listSuppressions,
)
engagementRouter.post(
'/suppressions',
// #swagger.tags = ['Admin - Engagement']
// #swagger.summary = 'Suppress an address by hand'
// #swagger.description = 'For a bounce or a complaint reported out of band. The reason is forced to `manual` rather than read from the body: an admin typing an address is not evidence of a bounce, and a `reason` column that sometimes means "the relay said so" and sometimes means "somebody chose this word" cannot diagnose anything. An address already on the list answers 200 with `created: false` rather than 409 - the operator asked for it to be suppressed and it is.'
// #swagger.security = [{ "cookieAuth": [] }, { "bearerAuth": [] }]
/* #swagger.requestBody = { required: true, content: { "application/json": { schema: { type: "object", properties: { address: { type: "string" }, detail: { type: "string", nullable: true } }, required: ["address"] } } } } */
/* #swagger.responses[201] = { description: 'Suppressed', content: { "application/json": { schema: { type: "object", properties: { created: { type: "boolean" }, address: { type: "string", nullable: true } } } } } } */
/* #swagger.responses[200] = { description: 'Already suppressed; nothing changed', content: { "application/json": { schema: { type: "object", properties: { created: { type: "boolean" }, address: { type: "string", nullable: true } } } } } } */
/* #swagger.responses[400] = { description: 'Not a valid address', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
/* #swagger.responses[403] = { description: 'Not an admin', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
adminOnly,
controller.createSuppression,
)
engagementRouter.delete(
'/suppressions',
// #swagger.tags = ['Admin - Engagement']
// #swagger.summary = 'Lift a suppression'
// #swagger.description = 'The only way out of the list, and the reason the screen exists: a hard bounce is written by a background worker with no human in the loop, so a mistyped-then-corrected mailbox would otherwise be silenced permanently. The address goes in the BODY, not the path - a path parameter lands in the access log, the browser history and every proxy in front of the deployment, and this one belongs to a real person. The hash cannot be used instead because the screen is never given one.'
// #swagger.security = [{ "cookieAuth": [] }, { "bearerAuth": [] }]
/* #swagger.requestBody = { required: true, content: { "application/json": { schema: { type: "object", properties: { address: { type: "string" }, channel: { type: "string", nullable: true } }, required: ["address"] } } } } */
/* #swagger.responses[200] = { description: 'Lifted', content: { "application/json": { schema: { type: "object", properties: { removed: { type: "boolean" } } } } } } */
/* #swagger.responses[400] = { description: 'No address given', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
/* #swagger.responses[403] = { description: 'Not an admin', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
/* #swagger.responses[404] = { description: 'That address is not suppressed', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
adminOnly,
controller.deleteSuppression,
)
module.exports = engagementRouter

33
server/src/utils/engagementWorker.js
View File

@@ -56,7 +56,7 @@ const STALE_MS = 15 * 60 * 1000
/**
* Deliver one claimed row.
*
* @returns {{ outcome: 'sent'|'retry'|'terminal', detail?: string, transport?: string, addressHash?: string }}
* @returns {{ outcome: 'sent'|'retry'|'terminal'|'suppressed'|'bounced', detail?: string, transport?: string, addressHash?: string }}
*/
async function deliver(row) {
const channel = channels.get(row.channel)
@@ -78,6 +78,21 @@ async function deliver(row) {
if (result && result.ok) {
return { outcome: 'sent', transport: result.transport, detail: result.detail, addressHash: hash }
}
// Its own outcome rather than a flavour of 'terminal' (Phase 9). Both statuses
// the ENUMs already carried for it say something a 'failed' row cannot: the
// outbox row was not attempted, and the send log's `suppressed` is the
// difference between "we tried and the relay refused" and "we declined to
// try". An operator reading a screen of failures needs those separated, and
// so does anybody counting deliverability.
if (result && result.suppressed) {
return { outcome: 'suppressed', detail: result.detail || 'suppressed', addressHash: hash }
}
// A hard bounce. Terminal like any other refusal, but recorded under its own
// name: "the relay would not take this" and "this mailbox does not exist"
// send an operator to two different places.
if (result && result.bounced) {
return { outcome: 'bounced', detail: result.detail || 'hard bounce', addressHash: hash }
}
if (result && result.retry) {
return { outcome: 'retry', detail: result.detail || 'transient failure', addressHash: hash }
}
@@ -107,8 +122,18 @@ async function processRow(row, now = new Date(), deliverFn = deliver) {
return 'retry'
}
const status = result.outcome === 'sent' ? 'sent' : 'failed'
await outboxDb.finish(row.id, status, status === 'failed' ? result.detail : null)
// **The two tables diverge here, deliberately.** `engagement_outbox.status` is
// the ROW's lifecycle and its ENUM has no 'bounced' - from the queue's point of
// view a bounced message is a row that finished unsuccessfully, which is
// 'failed'. `engagement_sends.status` is what happened to the MESSAGE, and
// there 'bounced' is the whole point: it is the difference between "look at
// your relay" and "this person's mailbox is gone".
let status = 'failed'
if (result.outcome === 'sent') status = 'sent'
else if (result.outcome === 'suppressed') status = 'suppressed'
else if (result.outcome === 'bounced') status = 'bounced'
const outboxStatus = status === 'bounced' ? 'failed' : status
await outboxDb.finish(row.id, outboxStatus, status === 'sent' ? null : result.detail)
// The send log is written for every terminal outcome, not only success. G15's
// question is "did user X get the mail?", and "no, and here is why" is an
// answer that table has to be able to give.
@@ -142,7 +167,7 @@ async function tick(now = new Date()) {
}
if (!due || !due.length) return
const counts = { sent: 0, failed: 0, retry: 0, taken: 0 }
const counts = { sent: 0, failed: 0, suppressed: 0, bounced: 0, retry: 0, taken: 0 }
for (const row of due) {
try {
const outcome = await processRow(row, now)

27
server/src/utils/mailer.js
View File

@@ -411,7 +411,17 @@ const PERMANENT_CODES = new Set([550, 553, 554, 'EENVELOPE', 'EAUTH'])
* itself. A FAILURE is still recorded, because a relay that has started refusing
* mail is precisely what that screen exists to show.
*
* @returns {Promise<{ok: boolean, retry?: boolean, transport?: string, detail?: string}>}
* **`smtp` carries the refusal itself, and Phase 9 is why.** `retry` says whether
* to try again; `detail` is a sentence for a human. Neither can answer "was this
* the recipient's fault?", which is the question the suppression list turns on —
* `550 5.1.1` and `550 5.7.1` produce an identical `retry: false` and mean
* completely different things. So the reply code, the enhanced status and the
* error code ride back untouched for `bounceClassify` to read. Three scalar
* fields rather than the error object: an `Error` from nodemailer carries the
* whole failed message, envelope included, and this return value is logged.
*
* @returns {Promise<{ok: boolean, retry?: boolean, transport?: string, detail?: string,
* smtp?: {code: string|null, responseCode: number|null, response: string|null}}>}
*/
async function sendNotification({ to, rendered, unsubscribeUrl, unsubscribeApiUrl }) {
const built = await buildTransport()
@@ -445,7 +455,20 @@ async function sendNotification({ to, rendered, unsubscribeUrl, unsubscribeApiUr
const code = err && (err.responseCode || err.code)
log.warn('engagement send failed', { message: err.message })
await emailConfig.recordStatus({ status: 'error', statusDetail: detail }).catch(() => {})
return { ok: false, retry: !PERMANENT_CODES.has(code), transport: config.transport, detail }
return {
ok: false,
retry: !PERMANENT_CODES.has(code),
transport: config.transport,
detail,
smtp: {
code: err && err.code ? String(err.code) : null,
responseCode: err && Number.isInteger(err.responseCode) ? err.responseCode : null,
// Truncated: a relay may answer with a multi-line essay, and this string
// reaches a 500-character log column. The reply code and the enhanced
// status are both at the front, which is where the classifier reads them.
response: err && err.response ? String(err.response).slice(0, 400) : null,
},
}
}
}