RunicGateway/docs
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(engagement): Phase 9 as built — deliverability, suppression and bounces

Browse Source 
Records what Phase 9 shipped (website#176) and the four decisions the org lead
settled before any of it: mechanism plus SMTP's own synchronous refusal rather
than an API transport; suppression scoped to engagement rules only; an
`address_masked` column added to §4.5's DDL; and the verification gate applied at
enqueue rather than at delivery.

The correction the phase's own text needed: "SMTP has none" is too strong. SMTP
has no asynchronous bounce feed, but a single-recipient send refused at RCPT TO
throws synchronously with the reply code intact, and mailer.js was already
catching that and discarding it.

The defect worth not repeating: `PERMANENT_CODES` is not a bounce classifier. It
answers "is retrying pointless?" and contains EAUTH, so suppressing on it would
have emptied the mailing list the first time an SMTP password expired.

Also records the one thing only the live rig could find — `engagement_sends`
has carried a `bounced` status since §4.5 and nothing had ever written it, so the
Send Log's Bounced filter matched nothing — and answers §7.1 Q1's narrower half.

- ENGAGEMENT.md: Phase 9 "As built", the §4.5 DDL, Q1's narrower half, status header
- BACKEND_DESIGN.md: §3 `engagement_suppressions`, and §7's Deliverability section

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
claude
2026-08-31 10:47:50 -05:00
parent 400873b83a
commit 221c5a9c9e
2 changed files with 227 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

98
website/BACKEND_DESIGN.md
View File

@@ -574,7 +574,8 @@ cooldown passes, always. See `ENGAGEMENT.md` Phase 4a.
| trigger_id | VARCHAR(96) NOT NULL | denormalized; survives a rule edit |
| user_id | INT NOT NULL FK→users(id) ON DELETE CASCADE | |
| channel | VARCHAR(32) NOT NULL | VARCHAR, never ENUM: the channel set is data, and a module must not require an ALTER |
| subject_key | VARCHAR(190) NOT NULL DEFAULT '' | what a COOLDOWN counts, from the trigger's declared `subjectKey`. A display string is fine here: it is only ever compared with itself |
| subject_key | VARCHAR(190) NOT NULL DEFAULT '' | what a COOLDOWN counts, from the trigger's declared `subjectKey`. A display string is fine here: it is only ever compared with itself |
| scope_key | VARCHAR(190) NULL | what a PREFERENCE and an UNSUBSCRIBE are keyed on (engagement phase 6), e.g. `team:12`. Deliberately **not** `subject_key`: an unsubscribe token is signed over this and sits in a mailbox for months, so it has to be a stable identifier — signing over a display name orphans every link the first time somebody renames a Team. NULL means an unscoped event; `''` is reserved for "deployment-wide" in `engagement_digest_state` |
| payload | JSON NOT NULL | the declared variables, snapshotted at emit |
| dedupe_key | VARCHAR(190) NULL | the emitter's replay guard; NULL never collides |
@@ -623,6 +624,35 @@ a rule's budget and mute it.
appear here, and neither do they appear in the engagement log lines, which carry variable *names* and
counts only.
Phase 9 gave two of those statuses their first writers. `suppressed` means the address was on the
suppression list and **no transport call was made**; `bounced` means one was, and the mailbox does
not exist. `complained` still has none — it needs a provider feedback loop, which SMTP has not got.
### engagement_suppressions — addresses we have stopped mailing (engagement phase 9)
| col | type | notes |
|---|---|---|
| address_hash | CHAR(64) NOT NULL PK | sha256 of the **lower-cased, trimmed** address |
| address_masked | VARCHAR(190) NULL | `d***@example.com`. Phase 9's one addition to the planned DDL |
| channel | VARCHAR(32) NOT NULL DEFAULT 'email' | |
| reason | ENUM('bounce','complaint','manual','unverified') | |
| detail | VARCHAR(500) NULL | e.g. `hard bounce: 5.1.1` |
| created_by | INT NULL FK→users(id) ON DELETE SET NULL | the admin, for a manual row; **NULL for an automatic one**, which is what separates the two |
| created_at | DATETIME | |
`INDEX(created_at)`, `INDEX(reason, created_at)` — the screen's two orderings.
G16. **Keyed on the address, not the user**, and after Phase 1b made addresses unique that is a
choice rather than a workaround: a bounce arrives as an address, it does not know which account was
behind it, and it stays true after that account changed its address or was deleted.
Writes are `INSERT IGNORE`, so **the first reason an address was suppressed is the one that
survives** — an address that hard-bounced in March and was manually re-added in June still reads
`bounce`, because that is the fact explaining why the mail stopped. An upsert would let the most
recent write overwrite the diagnosis.
`address_masked` exists because a hash-only table cannot be operated; the reasoning and the routes
are in §7's *Deliverability* subsection.
### engagement_digest_state — how far each digest has got (engagement phase 6)
| col | type | notes |
|---|---|---|
@@ -1601,6 +1631,72 @@ credentials.
stops**. The admin dashboard warns whenever the deprecated Gmail token is present and no replacement
credential is; see [`UPGRADE_NOTES.md`](UPGRADE_NOTES.md).
### Deliverability: suppression, bounces and the verification gate *(engagement phase 9)*
Engagement Phase 9 ([`ENGAGEMENT.md`](ENGAGEMENT.md) Phase 9). Two mechanisms decide that a person
who is *in* a rule's audience does not get the mail, and they are deliberately at different points
in the pipeline.
**`engagement_suppressions` — checked at DELIVERY.** Keyed on `address_hash` (sha256 of the
lower-cased address), because a bounce arrives as an address and stays true after the account behind
it changed its address or was deleted. An outbox row can sit through a rule's `delay_seconds` grace
window and an address can bounce inside it, so the only correct check is the one taken immediately
before the transport call — which is also what produces the `status='suppressed'` row in
`engagement_sends` with no transport call at all.
**The verification gate — applied at ENQUEUE.** With the `email_verification_required` setting on
(seeded in Phase 1b: `on` for a fresh install, `off` for an upgrade), an unverified address is
excluded before an outbox row is written. It hangs off a channel's optional **`eligible(userIds)`**
registration rather than living in the engine: being unverified is an *email* fact, and a rule
spanning email and in-app must still reach that person's inbox. Only `email` declares one. The
excluded count comes back so the admin reach preview reports it instead of quietly promising a
number the engine will not deliver.
**Scope: engagement rules only.** Password resets, invites, verification mails and the contact form
still attempt to a suppressed or unverified address. This is the posture `passwordReset.controller.js`
already took — user-initiated mail must not be blocked by a background system's opinion, and one
reset to a dead mailbox is not a reputation problem, whereas a rule mailing thousands of people
weekly is.
**What may write a `bounce` row is narrower than "the send failed".** `src/engagement/bounceClassify.js`
is the only judge, and it is deliberately **not** `mailer.PERMANENT_CODES` — that set answers "is
retrying pointless?" and contains `EAUTH` and `554`, so reusing it would mean one stale SMTP password
suppressing every address the worker touched, silently. The classifier reads the **RFC 3463 enhanced
status** first (`5.1.1`, `5.1.2`, `5.1.3`, `5.1.6`, `5.1.10`, `5.2.1` suppress; `5.3.x`, `5.5.x` and
`5.7.x` never do, being about the server or our standing with it), and falls back — only for `550`,
`551` and `553`, and only past a veto list — to a phrase match. **Anything it is unsure about is not
suppressed:** a false negative costs one retry next month, a false positive costs a person who
silently stops hearing from the deployment.
SMTP has no *asynchronous* bounce or complaint feed — that is where an API-based provider would earn
its place — but a single-recipient send refused at `RCPT TO` throws synchronously with the reply
code intact, which is the highest-value signal there is and is what this reads. `sendNotification`
therefore returns an `smtp: { code, responseCode, response }` triple alongside its classification;
`retry` and `detail` cannot answer "was this the recipient's fault", since `550 5.1.1` and
`550 5.7.1` are an identical `retry: false`.
**Statuses.** `engagement_sends.status` gains two real writers: `suppressed` (declined to try) and
`bounced` (tried, the mailbox does not exist). `engagement_outbox.status` records `bounced` as
`failed` — its ENUM has no such value and, from the queue's point of view, a bounced row is one that
finished unsuccessfully. `complained` still has no writer: it needs a provider feedback loop.
**Routes** (all `adminOnly`, under `/api/v1/admin/engagement`):
| Route | Notes |
| --- | --- |
| `GET /suppressions` | Paged, filterable by `reason` / `channel` / `search`, plus unfiltered `byReason` totals |
| `POST /suppressions` | `reason` is forced to `manual` — an admin typing an address is not evidence of a bounce. An address already listed answers 200 with `created: false`, not 409 |
| `DELETE /suppressions` | The only way out of the list. The address goes in the **body**, not the path: a path parameter lands in the access log, the browser history and every proxy in front of the deployment |
**Neither route ever returns `address_hash`**, the same rule `GET /sends` follows: a sha256 of every
address on the deployment, handed to a browser, is an offline dictionary attack. What the list
returns is `address_masked` — `d***@example.com` — which Phase 9 added to §4.5's DDL because a
hash-only table cannot be operated: an operator has to be able to see a whole domain refusing mail
and to let back in somebody who fixed their mailbox. The domain survives intact for the first; the
local part is destroyed rather than shortened, so the column can never be read back as an address
book. The consequence is that **lifting a suppression needs the full address typed in** — the screen
genuinely does not have it, which is the privacy design working rather than a rough edge.
---
## 7.5 Logging & observability