website/BACKEND_DESIGN.md

@@ -575,6 +575,7 @@ cooldown passes, always. See `ENGAGEMENT.md` Phase 4a.
 | user_id | INT NOT NULL FK→users(id) ON DELETE CASCADE | |
 | channel | VARCHAR(32) NOT NULL | VARCHAR, never ENUM: the channel set is data, and a module must not require an ALTER |
 | subject_key | VARCHAR(190) NOT NULL DEFAULT '' | what a COOLDOWN counts, from the trigger's declared `subjectKey`. A display string is fine here: it is only ever compared with itself |
+
 | scope_key | VARCHAR(190) NULL | what a PREFERENCE and an UNSUBSCRIBE are keyed on (engagement phase 6), e.g. `team:12`. Deliberately **not** `subject_key`: an unsubscribe token is signed over this and sits in a mailbox for months, so it has to be a stable identifier — signing over a display name orphans every link the first time somebody renames a Team. NULL means an unscoped event; `''` is reserved for "deployment-wide" in `engagement_digest_state` |
 | payload | JSON NOT NULL | the declared variables, snapshotted at emit |
 | dedupe_key | VARCHAR(190) NULL | the emitter's replay guard; NULL never collides |
@@ -623,6 +624,35 @@ a rule's budget and mute it.
 appear here, and neither do they appear in the engagement log lines, which carry variable *names* and
 counts only.
 
+Phase 9 gave two of those statuses their first writers. `suppressed` means the address was on the
+suppression list and **no transport call was made**; `bounced` means one was, and the mailbox does
+not exist. `complained` still has none — it needs a provider feedback loop, which SMTP has not got.
+
+### engagement_suppressions  — addresses we have stopped mailing (engagement phase 9)
+| col | type | notes |
+|---|---|---|
+| address_hash | CHAR(64) NOT NULL PK | sha256 of the **lower-cased, trimmed** address |
+| address_masked | VARCHAR(190) NULL | `d***@example.com`. Phase 9's one addition to the planned DDL |
+| channel | VARCHAR(32) NOT NULL DEFAULT 'email' | |
+| reason | ENUM('bounce','complaint','manual','unverified') | |
+| detail | VARCHAR(500) NULL | e.g. `hard bounce: 5.1.1` |
+| created_by | INT NULL FK→users(id) ON DELETE SET NULL | the admin, for a manual row; **NULL for an automatic one**, which is what separates the two |
+| created_at | DATETIME | |
+
+`INDEX(created_at)`, `INDEX(reason, created_at)` — the screen's two orderings.
+
+G16. **Keyed on the address, not the user**, and after Phase 1b made addresses unique that is a
+choice rather than a workaround: a bounce arrives as an address, it does not know which account was
+behind it, and it stays true after that account changed its address or was deleted.
+
+Writes are `INSERT IGNORE`, so **the first reason an address was suppressed is the one that
+survives** — an address that hard-bounced in March and was manually re-added in June still reads
+`bounce`, because that is the fact explaining why the mail stopped. An upsert would let the most
+recent write overwrite the diagnosis.
+
+`address_masked` exists because a hash-only table cannot be operated; the reasoning and the routes
+are in §7's *Deliverability* subsection.
+
 ### engagement_digest_state  — how far each digest has got (engagement phase 6)
 | col | type | notes |
 |---|---|---|
@@ -1601,6 +1631,72 @@ credentials.
 stops**. The admin dashboard warns whenever the deprecated Gmail token is present and no replacement
 credential is; see [`UPGRADE_NOTES.md`](UPGRADE_NOTES.md).
 
+### Deliverability: suppression, bounces and the verification gate  *(engagement phase 9)*
+
+Engagement Phase 9 ([`ENGAGEMENT.md`](ENGAGEMENT.md) Phase 9). Two mechanisms decide that a person
+who is *in* a rule's audience does not get the mail, and they are deliberately at different points
+in the pipeline.
+
+**`engagement_suppressions` — checked at DELIVERY.** Keyed on `address_hash` (sha256 of the
+lower-cased address), because a bounce arrives as an address and stays true after the account behind
+it changed its address or was deleted. An outbox row can sit through a rule's `delay_seconds` grace
+window and an address can bounce inside it, so the only correct check is the one taken immediately
+before the transport call — which is also what produces the `status='suppressed'` row in
+`engagement_sends` with no transport call at all.
+
+**The verification gate — applied at ENQUEUE.** With the `email_verification_required` setting on
+(seeded in Phase 1b: `on` for a fresh install, `off` for an upgrade), an unverified address is
+excluded before an outbox row is written. It hangs off a channel's optional **`eligible(userIds)`**
+registration rather than living in the engine: being unverified is an *email* fact, and a rule
+spanning email and in-app must still reach that person's inbox. Only `email` declares one. The
+excluded count comes back so the admin reach preview reports it instead of quietly promising a
+number the engine will not deliver.
+
+**Scope: engagement rules only.** Password resets, invites, verification mails and the contact form
+still attempt to a suppressed or unverified address. This is the posture `passwordReset.controller.js`
+already took — user-initiated mail must not be blocked by a background system's opinion, and one
+reset to a dead mailbox is not a reputation problem, whereas a rule mailing thousands of people
+weekly is.
+
+**What may write a `bounce` row is narrower than "the send failed".** `src/engagement/bounceClassify.js`
+is the only judge, and it is deliberately **not** `mailer.PERMANENT_CODES` — that set answers "is
+retrying pointless?" and contains `EAUTH` and `554`, so reusing it would mean one stale SMTP password
+suppressing every address the worker touched, silently. The classifier reads the **RFC 3463 enhanced
+status** first (`5.1.1`, `5.1.2`, `5.1.3`, `5.1.6`, `5.1.10`, `5.2.1` suppress; `5.3.x`, `5.5.x` and
+`5.7.x` never do, being about the server or our standing with it), and falls back — only for `550`,
+`551` and `553`, and only past a veto list — to a phrase match. **Anything it is unsure about is not
+suppressed:** a false negative costs one retry next month, a false positive costs a person who
+silently stops hearing from the deployment.
+
+SMTP has no *asynchronous* bounce or complaint feed — that is where an API-based provider would earn
+its place — but a single-recipient send refused at `RCPT TO` throws synchronously with the reply
+code intact, which is the highest-value signal there is and is what this reads. `sendNotification`
+therefore returns an `smtp: { code, responseCode, response }` triple alongside its classification;
+`retry` and `detail` cannot answer "was this the recipient's fault", since `550 5.1.1` and
+`550 5.7.1` are an identical `retry: false`.
+
+**Statuses.** `engagement_sends.status` gains two real writers: `suppressed` (declined to try) and
+`bounced` (tried, the mailbox does not exist). `engagement_outbox.status` records `bounced` as
+`failed` — its ENUM has no such value and, from the queue's point of view, a bounced row is one that
+finished unsuccessfully. `complained` still has no writer: it needs a provider feedback loop.
+
+**Routes** (all `adminOnly`, under `/api/v1/admin/engagement`):
+
+| Route | Notes |
+| --- | --- |
+| `GET /suppressions` | Paged, filterable by `reason` / `channel` / `search`, plus unfiltered `byReason` totals |
+| `POST /suppressions` | `reason` is forced to `manual` — an admin typing an address is not evidence of a bounce. An address already listed answers 200 with `created: false`, not 409 |
+| `DELETE /suppressions` | The only way out of the list. The address goes in the **body**, not the path: a path parameter lands in the access log, the browser history and every proxy in front of the deployment |
+
+**Neither route ever returns `address_hash`**, the same rule `GET /sends` follows: a sha256 of every
+address on the deployment, handed to a browser, is an offline dictionary attack. What the list
+returns is `address_masked` — `d***@example.com` — which Phase 9 added to §4.5's DDL because a
+hash-only table cannot be operated: an operator has to be able to see a whole domain refusing mail
+and to let back in somebody who fixed their mailbox. The domain survives intact for the first; the
+local part is destroyed rather than shortened, so the column can never be read back as an address
+book. The consequence is that **lifting a suppression needs the full address typed in** — the screen
+genuinely does not have it, which is the privacy design working rather than a rough edge.
+
 ---
 
 ## 7.5 Logging & observability

website/ENGAGEMENT.md

@@ -1,12 +1,13 @@
 # The Engagement System — findings and plan
 
-**Status:** design of record. **Phases 1, 1a, 1b, 2, 3, 4a, 4b, 5a, 5b, 6, 7 and 8 are built**
+**Status:** design of record. **Phases 1, 1a, 1b, 2, 3, 4a, 4b, 5a, 5b, 6, 7, 8 and 9 are built**
 (Phase 1: website#165 + docs#178, with website#164 as its prerequisite; Phase 1a: website#166 +
 docs#179; Phase 1b: website#167 + docs#180; Phase 2: website#168 + docs#181; Phase 3: website#169 +
 docs#182; Phase 4a: website#170 + docs#183; Phase 4b: website#171 + docs#184; Phase 5a: website#172 +
 docs#185; Phase 5b: website#173 + docs#186 + runicgateway.com#22; Phase 6: website#174 + docs#187 +
 runicgateway.com#23; Phase 7: website#175 + docs#188 + runicgateway.com#24; Phase 8: the Android
-half, Android-app#42 + docs#190); everything from Phase 9 on is still design. The scope decisions
+half, Android-app#42 + docs#190; Phase 9: website#176 + docs#191); everything from Phase 10 on is
+still design. The scope decisions
 below are settled; **eight of the nine questions in §7.1 are answered** - Q1, Q3, Q5 and Q7 on
 2026-08-28, Q6 on 2026-08-29 at the start of Phase 2 (which also settled §7.2's namespace question),
 **Q2 and Q4 on 2026-08-29 at the start of Phase 4**, and **Q8 on 2026-08-31 at the start of Phase
@@ -945,13 +946,30 @@ CREATE TABLE IF NOT EXISTS engagement_sends (
   INDEX idx_engs_user (user_id, created_at)
 ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
 
--- G16. Keyed on the ADDRESS, not the user: users.email is not unique.
+-- G16. Keyed on the ADDRESS, not the user. That was written when users.email was
+-- not unique; after Phase 1b it stays, for a better reason: a bounce arrives as
+-- an ADDRESS, does not know which account was behind it, and stays true after
+-- that account changed its address or was deleted.
+--
+-- **`address_masked` and `created_by` are Phase 9's additions to this DDL.** The
+-- hash-only table cannot be operated: an operator reading sha256 digests cannot
+-- tell three typos from a whole domain refusing mail, and un-suppressing somebody
+-- who fixed their mailbox is the one action the table must support. The domain
+-- survives so a domain-wide failure is visible; the local part is DESTROYED
+-- rather than shortened, so the column can never be read back as an address book.
+-- `created_by` is what separates a row an admin typed from one the outbox worker
+-- wrote (NULL).
 CREATE TABLE IF NOT EXISTS engagement_suppressions (
-  address_hash CHAR(64)     NOT NULL PRIMARY KEY,   -- sha256 of the lowercased address
-  channel      VARCHAR(32)  NOT NULL DEFAULT 'email',
-  reason       ENUM('bounce','complaint','manual','unverified') NOT NULL,
-  detail       VARCHAR(500) NULL,
-  created_at   DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
+  address_hash   CHAR(64)     NOT NULL PRIMARY KEY,   -- sha256 of the lowercased address
+  address_masked VARCHAR(190) NULL,                   -- d***@example.com; never the local part
+  channel        VARCHAR(32)  NOT NULL DEFAULT 'email',
+  reason         ENUM('bounce','complaint','manual','unverified') NOT NULL,
+  detail         VARCHAR(500) NULL,
+  created_by     INT          NULL,                   -- the admin, for a manual row; NULL if automatic
+  created_at     DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
+  CONSTRAINT fk_engsup_user FOREIGN KEY (created_by) REFERENCES users(id) ON DELETE SET NULL,
+  INDEX idx_engsup_created (created_at),
+  INDEX idx_engsup_reason (reason, created_at)
 ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
 
 -- G17: the in-app inbox. Core, game-agnostic, content-carrying.
@@ -2883,6 +2901,103 @@ verification mechanism 1b already built, with bounces following.
 transport call; a hard bounce suppresses the address; with the Phase 1b gate `on`, an unverified address
 is excluded from engagement rules but still receives password resets; with it `off`, it receives both.
 
+#### As built (2026-08-31)
+
+**Four decisions, settled by the org lead before any code:**
+
+1. **Mechanism plus SMTP's real signal; no API transport.** The phase's own text says "SMTP has none —
+   this is where the API-based providers earn their place", and that is too strong. SMTP has no
+   *asynchronous* bounce or complaint feed, but a single-recipient send refused at `RCPT TO` throws
+   synchronously with the reply code intact — the highest-value deliverability signal there is, and
+   `mailer.js` was already catching it as a `PERMANENT_CODE` and throwing it away. So this phase reads
+   it. A transport MAY declare a bounce handler; none does, and **no webhook route ships** — a route
+   with no producer is §7.1 Q9's problem in a different costume.
+2. **Suppression scopes to engagement rules only.** Password resets, invites, verification mail and
+   the contact form still attempt. This is the posture `passwordReset.controller.js` already stated
+   for the verification gate, and the argument carries: user-initiated mail must not be blocked by a
+   background system's opinion, and one reset to a dead mailbox is not a reputation problem.
+3. **`address_masked` is added to §4.5's DDL.** The hash-only table cannot be operated — an operator
+   staring at sha256 digests cannot tell three typos from a whole domain refusing mail, and
+   un-suppressing somebody who fixed their mailbox is the one action the table must support.
+4. **The verification gate filters at ENQUEUE, not at delivery.**
+
+**The defect this phase exists to have avoided: `PERMANENT_CODES` is not a bounce classifier.**
+The obvious implementation is "the mailer already tells us a failure is terminal, so suppress on
+that". `mailer.PERMANENT_CODES` is `{550, 553, 554, EENVELOPE, EAUTH}`, and it answers a different
+question — *is retrying pointless?* `EAUTH` is the operator's password being wrong and `554` is a
+relay-wide policy refusal; neither says anything about the recipient. Under that implementation **one
+stale SMTP credential suppresses every address the outbox worker touches**, with a clean send log, no
+warning, and a mailing list that has to be rebuilt by hand. So `bounceClassify.js` is its own judge:
+
+- the **RFC 3463 enhanced status** decides on its own where there is one — `5.1.1`, `5.1.2`, `5.1.3`,
+  `5.1.6`, `5.1.10` and `5.2.1` suppress, and `5.3.x`, `5.5.x` and `5.7.x` explicitly never do, being
+  about the server or about our standing with it;
+- without one, a phrase match applies **only** after `550`/`551`/`553` has already narrowed the
+  failure to the recipient address, and only past a veto list — `552` and `554` are excluded from even
+  that, because a full mailbox gets emptied and "transaction failed" is what a relay says when it does
+  not want to say why;
+- **anything it is unsure about is not suppressed.** A false negative costs one retry next month; a
+  false positive costs a person who silently stops hearing from the deployment and cannot find out.
+
+`sendNotification` now returns an `smtp: { code, responseCode, response }` triple so this is
+answerable at all: `retry` and `detail` cannot distinguish `550 5.1.1` from `550 5.7.1`, which are an
+identical `retry: false` and mean completely different things.
+
+**The two mechanisms sit at different points, and the split is the design.** A suppression can appear
+inside a rule's `delay_seconds` grace window, so the only correct check is the one taken immediately
+before the transport call — which is also what produces the `status='suppressed'` row with no
+transport call that the acceptance line asks for. Being unverified is a *standing* property, stable
+across that window, so excluding at delivery would write an outbox row purely to throw it away — and
+on a deployment that upgraded before verifying anybody, one rule firing would write thousands of
+`suppressed` rows nobody can read.
+
+**The gate is a channel hook, not an engine branch.** It hangs off a new optional
+`registerDeliveryChannel({ eligible })`, and `email` is the only channel that declares one. Both
+alternatives were wrong in a way the build made obvious: filtering the shared audience before the
+per-channel loop silences the wrong sink — **a rule spanning email and in-app must still put an item
+in an unverified user's inbox**, since being unverified is a reason not to mail somebody and no reason
+at all to hide their notifications — and an `if (channel === 'email')` in `engine.js` puts one
+channel's rule inside the generic engine. Its excluded counts flow into `summary.ineligible` and into
+the admin reach preview, which until now reported an audience size that was never the number of people
+who would get a mail.
+
+**Both new checks fail OPEN**, and the `try/catch` in `eligible` is load-bearing rather than habit:
+`applyRule` awaits it *before* the per-user loop, so an uncaught throw abandons the whole rule for
+every channel it names — a rule that silently sent nothing, with a clean log and an empty outbox. That
+is G22's shape again, and the recoverable mistake is mail going out.
+
+**The live rig found the one defect the stubs could not.** Against a real MariaDB, a real SMTP
+conversation (mailpit) and the real engine and worker, a hard bounce was recorded as `status='failed'`
+— honest, but `engagement_sends.status` has carried **`bounced`** since §4.5 and nothing had ever
+written it, so the Send Log's "Bounced" filter matched nothing and always would have. It is now a
+distinct outcome, because "the relay would not take this" and "this mailbox does not exist" send an
+operator to two different places. `engagement_outbox.status` still records it as `failed`: that ENUM
+has no `bounced`, and from the queue's point of view a bounced row is one that finished
+unsuccessfully. `complained` still has no writer, and cannot have one without a provider feedback loop.
+
+**Verified on the live rig**, seven rungs, each one a real send or a real refusal:
+
+1. baseline — three recipients, three mails in mailpit, three `sent` rows;
+2. one address suppressed by hand — **two** mails, and a `suppressed` row naming the reason;
+3. suppression lifted — three mails again;
+4. gate `on` — two mails, `ineligible: { unverified: 1 }`, and **no send-log row at all** for the
+   excluded user, which is what enqueue-time exclusion means;
+5. a `550 5.1.1` from the relay — two mails, one `bounced` row, and `b***@example.test` written to the
+   suppression list with `detail: hard bounce: 5.1.1`;
+6. the same rule again — that address `suppressed`, with no transport call;
+7. **a password reset to the suppressed address still arrived**, which is decision 2 proved rather
+   than asserted.
+
+The three admin routes were then driven over real HTTP: `created_by` records the admin on a manual row
+and stays **NULL** on the worker's automatic one, which is what separates them in the list; a repeat
+POST answers `200 {created:false}` rather than a 409; DELETE matches case-insensitively, which is the
+whole point of hashing a folded address; no response contains `address_hash`; and all three refuse a
+signed-out caller.
+
+**Still later phases':** an API-based transport with a signed webhook receiver, which is what would
+bring asynchronous bounces and `complained` to life. Nothing here blocks it — a transport may declare
+a bounce handler today.
+
 ---
 
 ### Phase 10 — Protocol bump: `house.decay` enrichment *(parallel from day one)*
@@ -3073,6 +3188,13 @@ day it ships.
 
    *Consequence for ordering:* Phase 9 no longer blocks Phase 11 — the verification *mechanism* moves
    forward into 1b, and Phase 9 keeps only bounces and suppression.
+
+   **The narrower half is settled (2026-08-31, at the start of Phase 9): an unverified address is
+   excluded at ENQUEUE, and only from the email channel.** With the gate on, `emailChannel.eligible`
+   drops the user before an outbox row is written, so nothing is queued only to be thrown away and
+   the admin reach preview can report the exclusion. It is email-only because a rule spanning
+   channels must still put an item in that person's in-app inbox. Transactional mail — resets,
+   invites, verification itself — is unaffected either way.
 2. ✅ **ANSWERED — multi-instance.** Neither of the two the question offered, and the third is
    better than both: the outbox sweep **claims each row with a compare-and-set** —
    `UPDATE … SET status='sending' WHERE id=? AND status='scheduled'` — and the instance the server