docs(engagement): Phase 14 as built — retention
The last phase of the workstream. ENGAGEMENT.md gets Phase 14's as-built
(the six org-lead decisions, three findings, the acceptance line discharged
and one rig trap), and its header now reads COMPLETE rather than "Phase 14
is scoped and not started".
BACKEND_DESIGN.md gets the reference: a new "Engagement retention" section
with the horizon table, why the outbox sweep is terminal-only, why each
floor is a correctness bound rather than a UI nicety, why suppressions do
not expire, and why this one got a screen when team_activity and
user_notifications did not. Plus the three new routes in the admin table and
the address_hash reversal on GET /suppressions.
Three findings worth not re-deriving:
* the per-row purge could not be built without reversing Phase 9's
decision to strip address_hash — the row had no handle, and the existing
lift asks for an address the screen has never shown.
* reclaimStale could not give up, so an interrupted send never became
terminal and no retention sweep could ever have bounded the outbox.
* the cooldown warning is unreachable through the UI by construction
(MAX_COOLDOWN_SECONDS is smaller than the horizon's floor). It is not
dead: it catches a hand-edited row, and a future raise of that ceiling.
And one the phase text got wrong in the other direction: the Play Data
Safety answers correctly did NOT move, because deploy-engagement is
deployment-scoped and PLAY_DATA_SAFETY.md is generated from the app-scoped
entries only.
Code: website#TBD.
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -596,7 +596,7 @@ cooldown passes, always. See `ENGAGEMENT.md` Phase 4a.
|
||||
| status | ENUM('scheduled','sending','sent','failed','cancelled','suppressed') | |
|
||||
| due_at | DATETIME NOT NULL | the grace window's clock, and the retry backoff's |
|
||||
| attempts / last_error / sent_at | | |
|
||||
| created_at / updated_at | DATETIME | `updated_at` is what a stale-claim reclaim measures |
|
||||
| created_at / updated_at | DATETIME | `updated_at` is what a stale-claim reclaim measures; `created_at` is what the retention sweep measures |
|
||||
|
||||
`UNIQUE(rule_id, user_id, channel, dedupe_key)`, `INDEX(status, due_at)`,
|
||||
`INDEX(rule_id, user_id, subject_key, status)`.
|
||||
@@ -763,6 +763,58 @@ drops unread items is one whose badge means nothing. The horizon is `settings.us
|
||||
(default 90), so an operator tightens a busy shard without a deploy — `team_activity`'s posture, in the
|
||||
worker that file is modelled on.
|
||||
|
||||
### Engagement retention — three sweeps and one recorded refusal (engagement phase 14)
|
||||
|
||||
Four engagement tables grow, and until Phase 14 nothing deleted from any of them.
|
||||
`utils/engagementRetentionPrune.js` is one nightly worker over three of them —
|
||||
`setInterval` + `unref` + `stop()`, wired into `server.js` beside `teamActivityPrune` and
|
||||
`inboxPrune`, batched 1000 × 50 per table, each table's failure caught on its own so a lock
|
||||
timeout on one does not leave the other two unbounded.
|
||||
|
||||
| table | horizon | setting | what is eligible |
|
||||
|---|---|---|---|
|
||||
| `engagement_sends` | 180 days | `engagement_sends_retain_days` (7–3650) | every row; they are all terminal |
|
||||
| `engagement_cooldowns` | 30 days | `engagement_cooldowns_retain_days` (2–3650) | every row, by `last_fired_at` |
|
||||
| `engagement_outbox` | 30 days | `engagement_outbox_retain_days` (2–3650) | **terminal rows only** — `sent`, `failed`, `cancelled`, `suppressed` — by `created_at` |
|
||||
| `engagement_suppressions` | **never** | — | nothing. See below |
|
||||
|
||||
**The outbox sweep is terminal-only, and that is a correctness rule, not a preference.** A
|
||||
`scheduled` row is a send this deployment still intends to make — `delay_seconds` can legitimately
|
||||
put one a day out — and a `sending` row may be a worker mid-flight. A sweep by age alone would
|
||||
cancel sends nobody cancelled, and the only symptom would be mail that never arrived.
|
||||
|
||||
**The floors are not UI niceties.** Below two days, a pruned cooldown row makes the next fire a
|
||||
FIRST fire — the rule sends twice; `MAX_COOLDOWN_SECONDS` is a validated 86 400, so two days is the
|
||||
smallest provably-safe value against any rule that can be saved. The send log's floor is a week
|
||||
because `engagement_sends` has two live readers: the per-rule hourly ceiling counts it
|
||||
(§7.1 Q3), and Admin → Engagement → Send Log is the operator's only answer to "was this person
|
||||
told".
|
||||
|
||||
**The cooldown horizon is checked, not assumed.** `engagementRules.db.maxEnabledCooldownSeconds()`
|
||||
(enabled rules only — a disabled rule writes no cooldown row) is compared against the horizon on
|
||||
every read of the policy and on every sweep. A horizon that does not clear it produces a warning on
|
||||
the screen and in the log, **and the sweep runs anyway**: refusing to prune would trade a bounded,
|
||||
describable fault for the unbounded one this phase exists to end.
|
||||
|
||||
**`engagement_suppressions` does not expire, and that is the recorded decision** (org lead,
|
||||
2026-09-01). A suppression is a standing decision, not a record of something that happened; ageing
|
||||
out a hard bounce re-mails an address that already bounced, which is how a sender loses a domain's
|
||||
reputation. The way out stays deliberate — see the per-row lift in the route table.
|
||||
|
||||
**Retention has a screen**, unlike `team_activity` and `user_notifications`, whose horizons are
|
||||
invisible settings rows. The send-log horizon changes what an operator-facing page is *able to show*,
|
||||
so it has to be visible and settable; having made one visible, hiding the other two would split one
|
||||
question ("what does this deployment keep") across two places. `EngagementSendLog.jsx` reads the
|
||||
policy and prints *"entries older than N days are removed automatically"* beneath its pager, so the
|
||||
total it shows stops being quietly wrong.
|
||||
|
||||
**One shipped defect this phase had to fix to be a bound at all.** `outboxDb.reclaimStale` returned
|
||||
every stale `sending` row to `scheduled`, and `MAX_ATTEMPTS` is consulted only on a graceful `retry`
|
||||
outcome — so a send that killed the process mid-flight cycled sending → scheduled → sending forever,
|
||||
never reached a terminal status, and was therefore never eligible for any sweep. It now fails a row
|
||||
that has burned its attempts **before** reclaiming the rest; the order is the fix, and reversing it
|
||||
hands the exhausted row straight back to `findDue`.
|
||||
|
||||
### The two block registries — pages and mail (engagement phase 5a)
|
||||
|
||||
`server/src/blocks/` (the CMS page family) and `server/src/emailBlocks/` (`email.heading`, `email.text`,
|
||||
@@ -1717,9 +1769,11 @@ finished unsuccessfully. `complained` still has no writer: it needs a provider f
|
||||
|
||||
| Route | Notes |
|
||||
| --- | --- |
|
||||
| `GET /suppressions` | Paged, filterable by `reason` / `channel` / `search`, plus unfiltered `byReason` totals |
|
||||
| `GET /suppressions` | Paged, filterable by `reason` / `channel` / `search`, plus unfiltered `byReason` totals. **Returns `address_hash` from Phase 14 on**, reversing Phase 9's decision to strip it: without a handle the only way out of the list was a `window.prompt` asking the operator to retype an address the screen has never shown them. The trade — this route is admin-only, and an admin can already suppress and unsuppress any address they can name, so the hash grants no capability they lack. `GET /sends` still strips its own hash: nothing there needs to act on a row |
|
||||
| `POST /suppressions` | `reason` is forced to `manual` — an admin typing an address is not evidence of a bounce. An address already listed answers 200 with `created: false`, not 409 |
|
||||
| `DELETE /suppressions` | The only way out of the list. The address goes in the **body**, not the path: a path parameter lands in the access log, the browser history and every proxy in front of the deployment |
|
||||
| `DELETE /suppressions` | The way out for an address the operator can type. It goes in the **body**, not the path: a path parameter lands in the access log, the browser history and every proxy in front of the deployment |
|
||||
| `DELETE /suppressions/by-hash/:hash` | The per-row Lift button (Phase 14). Same effect, different input — the operator is looking at a mask and knows only the row's handle. The handle **is** safe in the path where an address is not: a sha256 already served only to an admin session leaks nothing further by being logged. Shape-validated to 64 hex characters before it reaches a WHERE clause; `404` rather than `200` when nothing matched, so a stale screen says so instead of claiming success |
|
||||
| `GET · PUT /retention` | The three sweep horizons in days, with the bounds each is validated against and a `warnings` array carrying the one check that is not a static bound — a cooldown horizon shorter than the longest cooldown on an ENABLED rule. `engagement_suppressions` is deliberately absent: it does not expire. The `PUT` is **sparse** (saving one select cannot clobber another admin's concurrent change) and **refuses out-of-range rather than clamping**, because storing something other than what was typed would leave the screen describing a policy the deployment is not running |
|
||||
|
||||
**Neither route ever returns `address_hash`**, the same rule `GET /sends` follows: a sha256 of every
|
||||
address on the deployment, handed to a browser, is an offline dictionary attack. What the list
|
||||
|
||||
Reference in New Issue
Block a user