RunicGateway/docs
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: the engagement workstream — cutover 1 of 7 (edge → main) #200

Merged
whitlocktech merged 50 commits from edge into main 2026-09-01 13:54:15 +00:00
Conversation 1 Commits 50 Files Changed 13 +227 -9
2 changed files with 227 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit 221c5a9c9e - Show all commits

98
website/BACKEND_DESIGN.md
View File

@@ -574,7 +574,8 @@ cooldown passes, always. See `ENGAGEMENT.md` Phase 4a.
| trigger_id | VARCHAR(96) NOT NULL | denormalized; survives a rule edit |
| user_id | INT NOT NULL FK→users(id) ON DELETE CASCADE | |
| channel | VARCHAR(32) NOT NULL | VARCHAR, never ENUM: the channel set is data, and a module must not require an ALTER |
| subject_key | VARCHAR(190) NOT NULL DEFAULT '' | what a COOLDOWN counts, from the trigger's declared `subjectKey`. A display string is fine here: it is only ever compared with itself |
| subject_key | VARCHAR(190) NOT NULL DEFAULT '' | what a COOLDOWN counts, from the trigger's declared `subjectKey`. A display string is fine here: it is only ever compared with itself |
| scope_key | VARCHAR(190) NULL | what a PREFERENCE and an UNSUBSCRIBE are keyed on (engagement phase 6), e.g. `team:12`. Deliberately **not** `subject_key`: an unsubscribe token is signed over this and sits in a mailbox for months, so it has to be a stable identifier — signing over a display name orphans every link the first time somebody renames a Team. NULL means an unscoped event; `''` is reserved for "deployment-wide" in `engagement_digest_state` |
| payload | JSON NOT NULL | the declared variables, snapshotted at emit |
| dedupe_key | VARCHAR(190) NULL | the emitter's replay guard; NULL never collides |
@@ -623,6 +624,35 @@ a rule's budget and mute it.
appear here, and neither do they appear in the engagement log lines, which carry variable *names* and
counts only.
Phase 9 gave two of those statuses their first writers. `suppressed` means the address was on the
suppression list and **no transport call was made**; `bounced` means one was, and the mailbox does
not exist. `complained` still has none — it needs a provider feedback loop, which SMTP has not got.
### engagement_suppressions — addresses we have stopped mailing (engagement phase 9)
| col | type | notes |
|---|---|---|
| address_hash | CHAR(64) NOT NULL PK | sha256 of the **lower-cased, trimmed** address |
| address_masked | VARCHAR(190) NULL | `d***@example.com`. Phase 9's one addition to the planned DDL |
| channel | VARCHAR(32) NOT NULL DEFAULT 'email' | |
| reason | ENUM('bounce','complaint','manual','unverified') | |
| detail | VARCHAR(500) NULL | e.g. `hard bounce: 5.1.1` |
| created_by | INT NULL FK→users(id) ON DELETE SET NULL | the admin, for a manual row; **NULL for an automatic one**, which is what separates the two |
| created_at | DATETIME | |
`INDEX(created_at)`, `INDEX(reason, created_at)` — the screen's two orderings.
G16. **Keyed on the address, not the user**, and after Phase 1b made addresses unique that is a
choice rather than a workaround: a bounce arrives as an address, it does not know which account was
behind it, and it stays true after that account changed its address or was deleted.
Writes are `INSERT IGNORE`, so **the first reason an address was suppressed is the one that
survives** — an address that hard-bounced in March and was manually re-added in June still reads
`bounce`, because that is the fact explaining why the mail stopped. An upsert would let the most
recent write overwrite the diagnosis.
`address_masked` exists because a hash-only table cannot be operated; the reasoning and the routes
are in §7's *Deliverability* subsection.
### engagement_digest_state — how far each digest has got (engagement phase 6)
| col | type | notes |
|---|---|---|
@@ -1601,6 +1631,72 @@ credentials.
stops**. The admin dashboard warns whenever the deprecated Gmail token is present and no replacement
credential is; see [`UPGRADE_NOTES.md`](UPGRADE_NOTES.md).
### Deliverability: suppression, bounces and the verification gate *(engagement phase 9)*
Engagement Phase 9 ([`ENGAGEMENT.md`](ENGAGEMENT.md) Phase 9). Two mechanisms decide that a person
who is *in* a rule's audience does not get the mail, and they are deliberately at different points
in the pipeline.
**`engagement_suppressions` — checked at DELIVERY.** Keyed on `address_hash` (sha256 of the
lower-cased address), because a bounce arrives as an address and stays true after the account behind
it changed its address or was deleted. An outbox row can sit through a rule's `delay_seconds` grace
window and an address can bounce inside it, so the only correct check is the one taken immediately
before the transport call — which is also what produces the `status='suppressed'` row in
`engagement_sends` with no transport call at all.
**The verification gate — applied at ENQUEUE.** With the `email_verification_required` setting on
(seeded in Phase 1b: `on` for a fresh install, `off` for an upgrade), an unverified address is
excluded before an outbox row is written. It hangs off a channel's optional **`eligible(userIds)`**
registration rather than living in the engine: being unverified is an *email* fact, and a rule
spanning email and in-app must still reach that person's inbox. Only `email` declares one. The
excluded count comes back so the admin reach preview reports it instead of quietly promising a
number the engine will not deliver.
**Scope: engagement rules only.** Password resets, invites, verification mails and the contact form
still attempt to a suppressed or unverified address. This is the posture `passwordReset.controller.js`
already took — user-initiated mail must not be blocked by a background system's opinion, and one
reset to a dead mailbox is not a reputation problem, whereas a rule mailing thousands of people
weekly is.
**What may write a `bounce` row is narrower than "the send failed".** `src/engagement/bounceClassify.js`
is the only judge, and it is deliberately **not** `mailer.PERMANENT_CODES` — that set answers "is
retrying pointless?" and contains `EAUTH` and `554`, so reusing it would mean one stale SMTP password
suppressing every address the worker touched, silently. The classifier reads the **RFC 3463 enhanced
status** first (`5.1.1`, `5.1.2`, `5.1.3`, `5.1.6`, `5.1.10`, `5.2.1` suppress; `5.3.x`, `5.5.x` and
`5.7.x` never do, being about the server or our standing with it), and falls back — only for `550`,
`551` and `553`, and only past a veto list — to a phrase match. **Anything it is unsure about is not
suppressed:** a false negative costs one retry next month, a false positive costs a person who
silently stops hearing from the deployment.
SMTP has no *asynchronous* bounce or complaint feed — that is where an API-based provider would earn
its place — but a single-recipient send refused at `RCPT TO` throws synchronously with the reply
code intact, which is the highest-value signal there is and is what this reads. `sendNotification`
therefore returns an `smtp: { code, responseCode, response }` triple alongside its classification;
`retry` and `detail` cannot answer "was this the recipient's fault", since `550 5.1.1` and
`550 5.7.1` are an identical `retry: false`.
**Statuses.** `engagement_sends.status` gains two real writers: `suppressed` (declined to try) and
`bounced` (tried, the mailbox does not exist). `engagement_outbox.status` records `bounced` as
`failed` — its ENUM has no such value and, from the queue's point of view, a bounced row is one that
finished unsuccessfully. `complained` still has no writer: it needs a provider feedback loop.
**Routes** (all `adminOnly`, under `/api/v1/admin/engagement`):
| Route | Notes |
| --- | --- |
| `GET /suppressions` | Paged, filterable by `reason` / `channel` / `search`, plus unfiltered `byReason` totals |
| `POST /suppressions` | `reason` is forced to `manual` — an admin typing an address is not evidence of a bounce. An address already listed answers 200 with `created: false`, not 409 |
| `DELETE /suppressions` | The only way out of the list. The address goes in the **body**, not the path: a path parameter lands in the access log, the browser history and every proxy in front of the deployment |
**Neither route ever returns `address_hash`**, the same rule `GET /sends` follows: a sha256 of every
address on the deployment, handed to a browser, is an offline dictionary attack. What the list
returns is `address_masked` — `d***@example.com` — which Phase 9 added to §4.5's DDL because a
hash-only table cannot be operated: an operator has to be able to see a whole domain refusing mail
and to let back in somebody who fixed their mailbox. The domain survives intact for the first; the
local part is destroyed rather than shortened, so the column can never be read back as an address
book. The consequence is that **lifting a suppression needs the full address typed in** — the screen
genuinely does not have it, which is the privacy design working rather than a rough edge.
---
## 7.5 Logging & observability

138
website/ENGAGEMENT.md
View File

@@ -1,12 +1,13 @@
# The Engagement System — findings and plan
**Status:** design of record. **Phases 1, 1a, 1b, 2, 3, 4a, 4b, 5a, 5b, 6, 7 and 8 are built**
**Status:** design of record. **Phases 1, 1a, 1b, 2, 3, 4a, 4b, 5a, 5b, 6, 7, 8 and 9 are built**
(Phase 1: website#165 + docs#178, with website#164 as its prerequisite; Phase 1a: website#166 +
docs#179; Phase 1b: website#167 + docs#180; Phase 2: website#168 + docs#181; Phase 3: website#169 +
docs#182; Phase 4a: website#170 + docs#183; Phase 4b: website#171 + docs#184; Phase 5a: website#172 +
docs#185; Phase 5b: website#173 + docs#186 + runicgateway.com#22; Phase 6: website#174 + docs#187 +
runicgateway.com#23; Phase 7: website#175 + docs#188 + runicgateway.com#24; Phase 8: the Android
half, Android-app#42 + docs#190); everything from Phase 9 on is still design. The scope decisions
half, Android-app#42 + docs#190; Phase 9: website#176 + docs#191); everything from Phase 10 on is
still design. The scope decisions
below are settled; **eight of the nine questions in §7.1 are answered** - Q1, Q3, Q5 and Q7 on
2026-08-28, Q6 on 2026-08-29 at the start of Phase 2 (which also settled §7.2's namespace question),
**Q2 and Q4 on 2026-08-29 at the start of Phase 4**, and **Q8 on 2026-08-31 at the start of Phase
@@ -945,13 +946,30 @@ CREATE TABLE IF NOT EXISTS engagement_sends (
INDEX idx_engs_user (user_id, created_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
-- G16. Keyed on the ADDRESS, not the user: users.email is not unique.
-- G16. Keyed on the ADDRESS, not the user. That was written when users.email was
-- not unique; after Phase 1b it stays, for a better reason: a bounce arrives as
-- an ADDRESS, does not know which account was behind it, and stays true after
-- that account changed its address or was deleted.
--
-- **`address_masked` and `created_by` are Phase 9's additions to this DDL.** The
-- hash-only table cannot be operated: an operator reading sha256 digests cannot
-- tell three typos from a whole domain refusing mail, and un-suppressing somebody
-- who fixed their mailbox is the one action the table must support. The domain
-- survives so a domain-wide failure is visible; the local part is DESTROYED
-- rather than shortened, so the column can never be read back as an address book.
-- `created_by` is what separates a row an admin typed from one the outbox worker
-- wrote (NULL).
CREATE TABLE IF NOT EXISTS engagement_suppressions (
address_hash CHAR(64) NOT NULL PRIMARY KEY, -- sha256 of the lowercased address
channel VARCHAR(32) NOT NULL DEFAULT 'email',
reason ENUM('bounce','complaint','manual','unverified') NOT NULL,
detail VARCHAR(500) NULL,
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
address_hash CHAR(64) NOT NULL PRIMARY KEY, -- sha256 of the lowercased address
address_masked VARCHAR(190) NULL, -- d***@example.com; never the local part
channel VARCHAR(32) NOT NULL DEFAULT 'email',
reason ENUM('bounce','complaint','manual','unverified') NOT NULL,
detail VARCHAR(500) NULL,
created_by INT NULL, -- the admin, for a manual row; NULL if automatic
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT fk_engsup_user FOREIGN KEY (created_by) REFERENCES users(id) ON DELETE SET NULL,
INDEX idx_engsup_created (created_at),
INDEX idx_engsup_reason (reason, created_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
-- G17: the in-app inbox. Core, game-agnostic, content-carrying.
@@ -2883,6 +2901,103 @@ verification mechanism 1b already built, with bounces following.
transport call; a hard bounce suppresses the address; with the Phase 1b gate `on`, an unverified address
is excluded from engagement rules but still receives password resets; with it `off`, it receives both.
#### As built (2026-08-31)
**Four decisions, settled by the org lead before any code:**
1. **Mechanism plus SMTP's real signal; no API transport.** The phase's own text says "SMTP has none —
this is where the API-based providers earn their place", and that is too strong. SMTP has no
*asynchronous* bounce or complaint feed, but a single-recipient send refused at `RCPT TO` throws
synchronously with the reply code intact — the highest-value deliverability signal there is, and
`mailer.js` was already catching it as a `PERMANENT_CODE` and throwing it away. So this phase reads
it. A transport MAY declare a bounce handler; none does, and **no webhook route ships** — a route
with no producer is §7.1 Q9's problem in a different costume.
2. **Suppression scopes to engagement rules only.** Password resets, invites, verification mail and
the contact form still attempt. This is the posture `passwordReset.controller.js` already stated
for the verification gate, and the argument carries: user-initiated mail must not be blocked by a
background system's opinion, and one reset to a dead mailbox is not a reputation problem.
3. **`address_masked` is added to §4.5's DDL.** The hash-only table cannot be operated — an operator
staring at sha256 digests cannot tell three typos from a whole domain refusing mail, and
un-suppressing somebody who fixed their mailbox is the one action the table must support.
4. **The verification gate filters at ENQUEUE, not at delivery.**
**The defect this phase exists to have avoided: `PERMANENT_CODES` is not a bounce classifier.**
The obvious implementation is "the mailer already tells us a failure is terminal, so suppress on
that". `mailer.PERMANENT_CODES` is `{550, 553, 554, EENVELOPE, EAUTH}`, and it answers a different
question — *is retrying pointless?* `EAUTH` is the operator's password being wrong and `554` is a
relay-wide policy refusal; neither says anything about the recipient. Under that implementation **one
stale SMTP credential suppresses every address the outbox worker touches**, with a clean send log, no
warning, and a mailing list that has to be rebuilt by hand. So `bounceClassify.js` is its own judge:
- the **RFC 3463 enhanced status** decides on its own where there is one — `5.1.1`, `5.1.2`, `5.1.3`,
`5.1.6`, `5.1.10` and `5.2.1` suppress, and `5.3.x`, `5.5.x` and `5.7.x` explicitly never do, being
about the server or about our standing with it;
- without one, a phrase match applies **only** after `550`/`551`/`553` has already narrowed the
failure to the recipient address, and only past a veto list — `552` and `554` are excluded from even
that, because a full mailbox gets emptied and "transaction failed" is what a relay says when it does
not want to say why;
- **anything it is unsure about is not suppressed.** A false negative costs one retry next month; a
false positive costs a person who silently stops hearing from the deployment and cannot find out.
`sendNotification` now returns an `smtp: { code, responseCode, response }` triple so this is
answerable at all: `retry` and `detail` cannot distinguish `550 5.1.1` from `550 5.7.1`, which are an
identical `retry: false` and mean completely different things.
**The two mechanisms sit at different points, and the split is the design.** A suppression can appear
inside a rule's `delay_seconds` grace window, so the only correct check is the one taken immediately
before the transport call — which is also what produces the `status='suppressed'` row with no
transport call that the acceptance line asks for. Being unverified is a *standing* property, stable
across that window, so excluding at delivery would write an outbox row purely to throw it away — and
on a deployment that upgraded before verifying anybody, one rule firing would write thousands of
`suppressed` rows nobody can read.
**The gate is a channel hook, not an engine branch.** It hangs off a new optional
`registerDeliveryChannel({ eligible })`, and `email` is the only channel that declares one. Both
alternatives were wrong in a way the build made obvious: filtering the shared audience before the
per-channel loop silences the wrong sink — **a rule spanning email and in-app must still put an item
in an unverified user's inbox**, since being unverified is a reason not to mail somebody and no reason
at all to hide their notifications — and an `if (channel === 'email')` in `engine.js` puts one
channel's rule inside the generic engine. Its excluded counts flow into `summary.ineligible` and into
the admin reach preview, which until now reported an audience size that was never the number of people
who would get a mail.
**Both new checks fail OPEN**, and the `try/catch` in `eligible` is load-bearing rather than habit:
`applyRule` awaits it *before* the per-user loop, so an uncaught throw abandons the whole rule for
every channel it names — a rule that silently sent nothing, with a clean log and an empty outbox. That
is G22's shape again, and the recoverable mistake is mail going out.
**The live rig found the one defect the stubs could not.** Against a real MariaDB, a real SMTP
conversation (mailpit) and the real engine and worker, a hard bounce was recorded as `status='failed'`
— honest, but `engagement_sends.status` has carried **`bounced`** since §4.5 and nothing had ever
written it, so the Send Log's "Bounced" filter matched nothing and always would have. It is now a
distinct outcome, because "the relay would not take this" and "this mailbox does not exist" send an
operator to two different places. `engagement_outbox.status` still records it as `failed`: that ENUM
has no `bounced`, and from the queue's point of view a bounced row is one that finished
unsuccessfully. `complained` still has no writer, and cannot have one without a provider feedback loop.
**Verified on the live rig**, seven rungs, each one a real send or a real refusal:
1. baseline — three recipients, three mails in mailpit, three `sent` rows;
2. one address suppressed by hand — **two** mails, and a `suppressed` row naming the reason;
3. suppression lifted — three mails again;
4. gate `on` — two mails, `ineligible: { unverified: 1 }`, and **no send-log row at all** for the
excluded user, which is what enqueue-time exclusion means;
5. a `550 5.1.1` from the relay — two mails, one `bounced` row, and `b***@example.test` written to the
suppression list with `detail: hard bounce: 5.1.1`;
6. the same rule again — that address `suppressed`, with no transport call;
7. **a password reset to the suppressed address still arrived**, which is decision 2 proved rather
than asserted.
The three admin routes were then driven over real HTTP: `created_by` records the admin on a manual row
and stays **NULL** on the worker's automatic one, which is what separates them in the list; a repeat
POST answers `200 {created:false}` rather than a 409; DELETE matches case-insensitively, which is the
whole point of hashing a folded address; no response contains `address_hash`; and all three refuse a
signed-out caller.
**Still later phases':** an API-based transport with a signed webhook receiver, which is what would
bring asynchronous bounces and `complained` to life. Nothing here blocks it — a transport may declare
a bounce handler today.
---
### Phase 10 — Protocol bump: `house.decay` enrichment *(parallel from day one)*
@@ -3073,6 +3188,13 @@ day it ships.
*Consequence for ordering:* Phase 9 no longer blocks Phase 11 — the verification *mechanism* moves
forward into 1b, and Phase 9 keeps only bounces and suppression.
**The narrower half is settled (2026-08-31, at the start of Phase 9): an unverified address is
excluded at ENQUEUE, and only from the email channel.** With the gate on, `emailChannel.eligible`
drops the user before an outbox row is written, so nothing is queued only to be thrown away and
the admin reach preview can report the exclusion. It is email-only because a rule spanning
channels must still put an item in that person's in-app inbox. Transactional mail — resets,
invites, verification itself — is unaffected either way.
2. ✅ **ANSWERED — multi-instance.** Neither of the two the question offered, and the third is
better than both: the outbox sweep **claims each row with a compare-and-set** —
`UPDATE … SET status='sending' WHERE id=? AND status='scheduled'` — and the instance the server