feat(engagement): retention — three sweeps and one recorded refusal
All checks were successful
PR Checks / client-build (pull_request) Successful in 34s
PR Checks / bot-tests (pull_request) Successful in 34s
PR Checks / server-tests (pull_request) Successful in 13m23s

ENGAGEMENT.md Phase 14, the last phase of the workstream. Four engagement
tables grew on every fire and nothing had ever deleted from any of them.

Three of them now have a horizon, swept nightly by one worker
(utils/engagementRetentionPrune.js — setInterval + unref + stop(), batched
1000 x 50, each table's failure caught on its own so a lock timeout on one
does not leave the other two unbounded):

  engagement_sends      180 days   engagement_sends_retain_days      (7-3650)
  engagement_cooldowns   30 days   engagement_cooldowns_retain_days  (2-3650)
  engagement_outbox      30 days   engagement_outbox_retain_days     (2-3650)

The fourth, engagement_suppressions, does not expire, and that is the
recorded decision rather than an omission: a suppression is a standing
decision, and ageing out a hard bounce re-mails an address that already
bounced. The way out stays deliberate, and is now reachable per row.

Six decisions were settled by the org lead before any code. Two of them
widened the phase past what was offered:

  * the send-log horizon is admin-configurable, so retention got a SCREEN
    (Admin -> Engagement -> Retention) where team_activity and
    user_notifications keep theirs in invisible settings rows. The send-log
    horizon changes what an operator-facing page is able to show, so it has
    to be visible; the other two came with it, because "what does this
    deployment keep" is one question.
  * the suppression purge, which cost a Phase 9 decision. The list
    deliberately stripped address_hash from every row, so the only way out
    was a window.prompt asking the operator to retype an address the screen
    has never shown them. The row had no handle at all. The hash is now
    returned: this route is admin-only and an admin can already suppress and
    unsuppress any address they can name, so it grants no capability they
    lack. GET /sends still strips its own.

The outbox sweep is TERMINAL-ONLY and that is a correctness rule: a
scheduled row is a send this deployment still intends to make (delay_seconds
can put one a day out) and a sending row may be mid-flight.

One shipped defect had to be fixed for the sweep to be a bound at all.
reclaimStale returned every stale sending row to scheduled, and MAX_ATTEMPTS
is consulted only on a graceful retry outcome — so a send that killed the
process mid-flight cycled sending -> scheduled -> sending forever, never
terminal, therefore never eligible for any sweep. It now fails an exhausted
row BEFORE reclaiming the rest; the order is the fix.

Two indexes (idx_engo_sweep, idx_engs_sweep): every existing index on those
tables has created_at in second position, which serves a per-rule window and
is useless to a whole-table horizon.

Proved twice: engagementRetentionSql.test.js against a real MariaDB (7
tests, incl. the acceptance case and the wrong reclaim order run
deliberately), and the live stack, where a 90-day-old cancelled row was
swept and a 90-day-old scheduled row survived.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2026-09-01 15:40:53 -05:00
parent e59a68c152
commit 5779d15150
22 changed files with 1728 additions and 17 deletions

View File

@@ -381,4 +381,48 @@ engagementRouter.delete(
controller.deleteSuppression,
)
engagementRouter.delete(
'/suppressions/by-hash/:hash',
// #swagger.tags = ['Admin - Engagement']
// #swagger.summary = 'Lift a suppression by its row handle'
// #swagger.description = 'The per-row Lift button (Phase 14). Same effect as the route above, different input: the screen shows a mask, so the operator does not know the address and can only act on the row handle the list gives them. The handle IS safe in the path where an address is not - it is a sha256 already served only to an admin session, so an access log or proxy that captures it learns nothing new. 404 rather than 200 when nothing matched, so a stale screen (two admins, one list) says so instead of claiming success.'
// #swagger.security = [{ "cookieAuth": [] }, { "bearerAuth": [] }]
// #swagger.parameters['hash'] = { in: 'path', description: 'The address_hash the list returns for that row, 64 hex characters', required: true, schema: { type: 'string' } }
// #swagger.parameters['channel'] = { in: 'query', description: 'Defaults to email', required: false, schema: { type: 'string' } }
/* #swagger.responses[200] = { description: 'Lifted', content: { "application/json": { schema: { type: "object", properties: { removed: { type: "boolean" } } } } } } */
/* #swagger.responses[400] = { description: 'Not a suppression handle', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
/* #swagger.responses[403] = { description: 'Not an admin', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
/* #swagger.responses[404] = { description: 'That address is not suppressed', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
adminOnly,
controller.deleteSuppressionByHash,
)
// ── Retention (Phase 14) ───────────────────────────────────────────────────
engagementRouter.get(
'/retention',
// #swagger.tags = ['Admin - Engagement']
// #swagger.summary = 'Read the engagement retention policy'
// #swagger.description = 'The three horizons the nightly sweep uses, in days, with the bounds each is validated against. `engagement_suppressions` is deliberately absent: a suppression is a standing decision and does not expire, because ageing out a hard bounce re-mails an address that already bounced. `warnings` carries the one check that cannot be a static bound - a cooldown horizon shorter than the longest cooldown on an ENABLED rule, which would let that rule send twice.'
// #swagger.security = [{ "cookieAuth": [] }, { "bearerAuth": [] }]
/* #swagger.responses[200] = { description: 'The current policy', content: { "application/json": { schema: { type: "object", properties: { retention: { type: "object", properties: { sends: { type: "integer" }, cooldowns: { type: "integer" }, outbox: { type: "integer" } } }, limits: { type: "object", additionalProperties: true }, longestCooldownSeconds: { type: "integer" }, warnings: { type: "array", items: { type: "string" } } } } } } } */
/* #swagger.responses[403] = { description: 'Not an admin', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
adminOnly,
controller.getRetention,
)
engagementRouter.put(
'/retention',
// #swagger.tags = ['Admin - Engagement']
// #swagger.summary = 'Set the engagement retention policy'
// #swagger.description = 'Sparse: only the horizons named in the body are written, so saving one select cannot clobber a value another admin changed between load and save. Out of range is a 400 rather than a clamp - storing something other than what was typed would leave the screen describing a policy the deployment is not running. The floors are not UI niceties: below 2 days a pruned cooldown row makes the next fire a FIRST fire (a duplicate send), and the send log is counted by the per-rule hourly ceiling.'
// #swagger.security = [{ "cookieAuth": [] }, { "bearerAuth": [] }]
/* #swagger.requestBody = { required: true, content: { "application/json": { schema: { type: "object", properties: { sends: { type: "integer", nullable: true }, cooldowns: { type: "integer", nullable: true }, outbox: { type: "integer", nullable: true } } } } } } */
/* #swagger.responses[200] = { description: 'The policy as it now stands', content: { "application/json": { schema: { type: "object", properties: { retention: { type: "object", additionalProperties: { type: "integer" } }, limits: { type: "object", additionalProperties: true }, longestCooldownSeconds: { type: "integer" }, warnings: { type: "array", items: { type: "string" } } } } } } } */
/* #swagger.responses[400] = { description: 'A horizon was not a whole number of days, or was out of range', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
/* #swagger.responses[403] = { description: 'Not an admin', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
adminOnly,
controller.putRetention,
)
module.exports = engagementRouter