feat(engagement): retention — three sweeps and one recorded refusal (Phase 14) #182

wtclaude · 2026-09-01T20:42:06Z

wtclaude commented

2026-09-01 20:42:06 +00:00

docs/website/ENGAGEMENT.md Phase 14 — the last phase of the workstream. Four engagement tables grew on every fire and nothing had ever deleted from any of them.

Docs: RunicGateway/docs#205 · Site: RunicGateway/runicgateway.com#29

What it does

Three tables get a horizon, swept nightly by one worker (utils/engagementRetentionPrune.js — setInterval + unref + stop(), batched 1000 × 50, each table's failure caught on its own so a lock timeout on one does not leave the other two unbounded):

table	default	setting	eligible
`engagement_sends`	180 days	`engagement_sends_retain_days` (7–3650)	every row — all terminal
`engagement_cooldowns`	30 days	`engagement_cooldowns_retain_days` (2–3650)	every row, by `last_fired_at`
`engagement_outbox`	30 days	`engagement_outbox_retain_days` (2–3650)	terminal only, by `created_at`

The fourth — engagement_suppressions — does not expire, and that is the recorded decision rather than an omission: a suppression is a standing decision, and ageing out a hard bounce re-mails an address that already bounced.

The six decisions (org lead, 2026-09-01, before any code)

Two of the four horizon answers widened the phase past what was offered:

The send-log horizon is admin-configurable, so retention got a screen where team_activity and user_notifications keep theirs in invisible settings rows. The send-log horizon changes what an operator-facing page is able to show, so it has to be visible; the other two came with it, because "what does this deployment keep" is one question.
The suppression purge, which cost a Phase 9 decision — see finding 1.

Three findings

The per-row purge could not be built without reversing Phase 9. The list deliberately stripped address_hash from every row, and the consequence its own screen shows is that the only way out was a window.prompt asking the operator to retype an address the screen has never shown them — it stores a mask. The row had no handle at all. The hash is now returned: this route is admin-only and an admin can already suppress and unsuppress any address they can name, so it grants no capability they lack. GET /sends still strips its own.
reclaimStale could not give up, so the outbox sweep would not have been a bound. It returned every stale sending row to scheduled, and MAX_ATTEMPTS is consulted only on a graceful retry outcome — so a send that killed the process mid-flight cycled sending → scheduled → sending forever: never terminal, therefore never eligible for the terminal-only sweep this PR adds. Fixed by failing an exhausted row before reclaiming the rest. The test runs the wrong order deliberately to show what it avoids.
The cooldown warning is unreachable through the UI by construction — MAX_COOLDOWN_SECONDS (86 400) is smaller than the horizon's floor (2 days). Not dead: it fires for a hand-edited row, and it catches a future raise of that ceiling that forgets this floor.

Notes for review

The outbox sweep is terminal-only and that is a correctness rule. A scheduled row is a send this deployment still intends to make (delay_seconds can put one a day out); a sending row may be mid-flight. Sweeping by age alone would cancel sends nobody cancelled, and the only symptom would be mail that never arrived.
The floors are not UI niceties. Below 2 days a pruned cooldown row makes the next fire a first fire — a duplicate send. The send log's week floor exists because the per-rule hourly ceiling counts that table.
Two indexes (idx_engo_sweep, idx_engs_sweep): every existing index on those tables has created_at in second position, which serves a per-rule window and is useless to a whole-table horizon. Both added idempotently (ADD INDEX IF NOT EXISTS) and verified applied on a live boot.
PUT /retention is sparse and refuses out-of-range rather than clamping — storing something other than what was typed would leave the screen describing a policy the deployment is not running.
One layering fix in passing: the "longest enabled cooldown" query started life as raw SQL in a .model.js and moved to engagementRules.db.js where CLAUDE.md's .model.js/.db.js split puts it. That is also what made it stubbable.

Proof

engagementRetention.test.js — 17 tests (policy, bounds, the guard, the worker's control flow).
engagementRetentionSql.test.js — 7 tests against a real MariaDB, skipping when there is none, incl. the phase's acceptance case and the index plans.
Server suite 1572/1573 (the one failure is a pre-existing local CRLF artifact in engagementManifest.test.js, present on clean main). Client 327/327 and the build.
Live stack: a 90-day-old cancelled row swept, a 90-day-old scheduled row survived; the guard warned and swept anyway with a hand-set 5-day cooldown; all three routes 403 for a player and 401 anonymously; per-row Lift, the retention screen and the Send Log's new footer line all walked in a browser.

_{Rig trap worth not re-deriving: the first live run appeared to delete the scheduled row. It had not — the outbox worker was running in the same server and the planted row's due_at was 90 days past, so the worker legitimately claimed it first. Plant it with a future due_at, or you are measuring the worker, not the sweep.}

AI-assisted: written with Claude Code (Claude Opus), reviewed by the author.

`docs/website/ENGAGEMENT.md` **Phase 14 — the last phase of the workstream.** Four engagement tables grew on every fire and nothing had ever deleted from any of them. Docs: RunicGateway/docs#205 · Site: RunicGateway/runicgateway.com#29 ## What it does Three tables get a horizon, swept nightly by one worker (`utils/engagementRetentionPrune.js` — `setInterval` + `unref` + `stop()`, batched 1000 × 50, each table's failure caught on its own so a lock timeout on one does not leave the other two unbounded): | table | default | setting | eligible | |---|---|---|---| | `engagement_sends` | 180 days | `engagement_sends_retain_days` (7–3650) | every row — all terminal | | `engagement_cooldowns` | 30 days | `engagement_cooldowns_retain_days` (2–3650) | every row, by `last_fired_at` | | `engagement_outbox` | 30 days | `engagement_outbox_retain_days` (2–3650) | **terminal only**, by `created_at` | The fourth — `engagement_suppressions` — **does not expire**, and that is the recorded decision rather than an omission: a suppression is a standing decision, and ageing out a hard bounce re-mails an address that already bounced. ## The six decisions (org lead, 2026-09-01, before any code) Two of the four horizon answers widened the phase past what was offered: - **The send-log horizon is admin-configurable**, so retention got a **screen** where `team_activity` and `user_notifications` keep theirs in invisible `settings` rows. The send-log horizon changes what an operator-facing page is *able to show*, so it has to be visible; the other two came with it, because "what does this deployment keep" is one question. - **The suppression purge**, which cost a Phase 9 decision — see finding 1. ## Three findings 1. **The per-row purge could not be built without reversing Phase 9.** The list deliberately stripped `address_hash` from every row, and the consequence its own screen shows is that the only way out was a `window.prompt` asking the operator to retype an address **the screen has never shown them** — it stores a mask. The row had no handle at all. The hash is now returned: this route is admin-only and an admin can already suppress and unsuppress any address they can name, so it grants no capability they lack. `GET /sends` still strips its own. 2. **`reclaimStale` could not give up, so the outbox sweep would not have been a bound.** It returned *every* stale `sending` row to `scheduled`, and `MAX_ATTEMPTS` is consulted only on a graceful `retry` outcome — so a send that killed the process mid-flight cycled sending → scheduled → sending **forever**: never terminal, therefore never eligible for the terminal-only sweep this PR adds. Fixed by failing an exhausted row **before** reclaiming the rest. The test runs the wrong order deliberately to show what it avoids. 3. **The cooldown warning is unreachable through the UI by construction** — `MAX_COOLDOWN_SECONDS` (86 400) is smaller than the horizon's floor (2 days). Not dead: it fires for a hand-edited row, and it catches a future raise of that ceiling that forgets this floor. ## Notes for review - **The outbox sweep is terminal-only and that is a correctness rule.** A `scheduled` row is a send this deployment still intends to make (`delay_seconds` can put one a day out); a `sending` row may be mid-flight. Sweeping by age alone would cancel sends nobody cancelled, and the only symptom would be mail that never arrived. - **The floors are not UI niceties.** Below 2 days a pruned cooldown row makes the next fire a *first* fire — a duplicate send. The send log's week floor exists because the per-rule hourly ceiling counts that table. - **Two indexes** (`idx_engo_sweep`, `idx_engs_sweep`): every existing index on those tables has `created_at` in *second* position, which serves a per-rule window and is useless to a whole-table horizon. Both added idempotently (`ADD INDEX IF NOT EXISTS`) and verified applied on a live boot. - `PUT /retention` is **sparse** and **refuses out-of-range rather than clamping** — storing something other than what was typed would leave the screen describing a policy the deployment is not running. - One layering fix in passing: the "longest enabled cooldown" query started life as raw SQL in a `.model.js` and moved to `engagementRules.db.js` where CLAUDE.md's `.model.js`/`.db.js` split puts it. That is also what made it stubbable. ## Proof - `engagementRetention.test.js` — 17 tests (policy, bounds, the guard, the worker's control flow). - `engagementRetentionSql.test.js` — 7 tests against a **real MariaDB**, skipping when there is none, incl. the phase's acceptance case and the index plans. - Server suite **1572/1573** (the one failure is a pre-existing local CRLF artifact in `engagementManifest.test.js`, present on clean `main`). Client **327/327** and the build. - **Live stack**: a 90-day-old `cancelled` row swept, a 90-day-old `scheduled` row survived; the guard warned and swept anyway with a hand-set 5-day cooldown; all three routes 403 for a player and 401 anonymously; per-row Lift, the retention screen and the Send Log's new footer line all walked in a browser. <sub>Rig trap worth not re-deriving: the first live run appeared to delete the `scheduled` row. It had not — the outbox worker was running in the same server and the planted row's `due_at` was 90 days past, so the worker legitimately claimed it first. Plant it with a **future** `due_at`, or you are measuring the worker, not the sweep.</sub> --- - [x] AI-assisted: written with Claude Code (Claude Opus), reviewed by the author.

wtclaude added 1 commit 2026-09-01 20:42:08 +00:00

feat(engagement): retention — three sweeps and one recorded refusal

PR Checks / client-build (pull_request) Successful in 34s

Details

PR Checks / bot-tests (pull_request) Successful in 34s

Details

PR Checks / server-tests (pull_request) Successful in 13m23s

Details

5779d15150

ENGAGEMENT.md Phase 14, the last phase of the workstream. Four engagement
tables grew on every fire and nothing had ever deleted from any of them.

Three of them now have a horizon, swept nightly by one worker
(utils/engagementRetentionPrune.js — setInterval + unref + stop(), batched
1000 x 50, each table's failure caught on its own so a lock timeout on one
does not leave the other two unbounded):

  engagement_sends      180 days   engagement_sends_retain_days      (7-3650)
  engagement_cooldowns   30 days   engagement_cooldowns_retain_days  (2-3650)
  engagement_outbox      30 days   engagement_outbox_retain_days     (2-3650)

The fourth, engagement_suppressions, does not expire, and that is the
recorded decision rather than an omission: a suppression is a standing
decision, and ageing out a hard bounce re-mails an address that already
bounced. The way out stays deliberate, and is now reachable per row.

Six decisions were settled by the org lead before any code. Two of them
widened the phase past what was offered:

  * the send-log horizon is admin-configurable, so retention got a SCREEN
    (Admin -> Engagement -> Retention) where team_activity and
    user_notifications keep theirs in invisible settings rows. The send-log
    horizon changes what an operator-facing page is able to show, so it has
    to be visible; the other two came with it, because "what does this
    deployment keep" is one question.
  * the suppression purge, which cost a Phase 9 decision. The list
    deliberately stripped address_hash from every row, so the only way out
    was a window.prompt asking the operator to retype an address the screen
    has never shown them. The row had no handle at all. The hash is now
    returned: this route is admin-only and an admin can already suppress and
    unsuppress any address they can name, so it grants no capability they
    lack. GET /sends still strips its own.

The outbox sweep is TERMINAL-ONLY and that is a correctness rule: a
scheduled row is a send this deployment still intends to make (delay_seconds
can put one a day out) and a sending row may be mid-flight.

One shipped defect had to be fixed for the sweep to be a bound at all.
reclaimStale returned every stale sending row to scheduled, and MAX_ATTEMPTS
is consulted only on a graceful retry outcome — so a send that killed the
process mid-flight cycled sending -> scheduled -> sending forever, never
terminal, therefore never eligible for any sweep. It now fails an exhausted
row BEFORE reclaiming the rest; the order is the fix.

Two indexes (idx_engo_sweep, idx_engs_sweep): every existing index on those
tables has created_at in second position, which serves a per-rule window and
is useless to a whole-table horizon.

Proved twice: engagementRetentionSql.test.js against a real MariaDB (7
tests, incl. the acceptance case and the wrong reclaim order run
deliberately), and the live stack, where a 90-day-old cancelled row was
swept and a 90-day-old scheduled row survived.

Co-Authored-By: Claude <noreply@anthropic.com>

wtclaude referenced this pull request from RunicGateway/docs

2026-09-01 20:42:27 +00:00

docs(engagement): Phase 14 as built — retention #205

wtclaude referenced this pull request from RunicGateway/runicgateway.com

2026-09-01 20:42:45 +00:00

docs(site): /privacy tells the truth about engagement retention again #29

whitlocktech approved these changes 2026-09-01 20:56:03 +00:00

whitlocktech scheduled this pull request to auto merge when all checks succeed 2026-09-01 20:56:08 +00:00

whitlocktech merged commit 6331b36c45 into main

2026-09-01 20:57:09 +00:00

whitlocktech deleted branch feature/engagement-retention

2026-09-01 20:57:10 +00:00