feat(engagement): retention — three sweeps and one recorded refusal

ENGAGEMENT.md Phase 14, the last phase of the workstream. Four engagement
tables grew on every fire and nothing had ever deleted from any of them.

Three of them now have a horizon, swept nightly by one worker
(utils/engagementRetentionPrune.js — setInterval + unref + stop(), batched
1000 x 50, each table's failure caught on its own so a lock timeout on one
does not leave the other two unbounded):

  engagement_sends      180 days   engagement_sends_retain_days      (7-3650)
  engagement_cooldowns   30 days   engagement_cooldowns_retain_days  (2-3650)
  engagement_outbox      30 days   engagement_outbox_retain_days     (2-3650)

The fourth, engagement_suppressions, does not expire, and that is the
recorded decision rather than an omission: a suppression is a standing
decision, and ageing out a hard bounce re-mails an address that already
bounced. The way out stays deliberate, and is now reachable per row.

Six decisions were settled by the org lead before any code. Two of them
widened the phase past what was offered:

  * the send-log horizon is admin-configurable, so retention got a SCREEN
    (Admin -> Engagement -> Retention) where team_activity and
    user_notifications keep theirs in invisible settings rows. The send-log
    horizon changes what an operator-facing page is able to show, so it has
    to be visible; the other two came with it, because "what does this
    deployment keep" is one question.
  * the suppression purge, which cost a Phase 9 decision. The list
    deliberately stripped address_hash from every row, so the only way out
    was a window.prompt asking the operator to retype an address the screen
    has never shown them. The row had no handle at all. The hash is now
    returned: this route is admin-only and an admin can already suppress and
    unsuppress any address they can name, so it grants no capability they
    lack. GET /sends still strips its own.

The outbox sweep is TERMINAL-ONLY and that is a correctness rule: a
scheduled row is a send this deployment still intends to make (delay_seconds
can put one a day out) and a sending row may be mid-flight.

One shipped defect had to be fixed for the sweep to be a bound at all.
reclaimStale returned every stale sending row to scheduled, and MAX_ATTEMPTS
is consulted only on a graceful retry outcome — so a send that killed the
process mid-flight cycled sending -> scheduled -> sending forever, never
terminal, therefore never eligible for any sweep. It now fails an exhausted
row BEFORE reclaiming the rest; the order is the fix.

Two indexes (idx_engo_sweep, idx_engs_sweep): every existing index on those
tables has created_at in second position, which serves a per-rule window and
is useless to a whole-table horizon.

Proved twice: engagementRetentionSql.test.js against a real MariaDB (7
tests, incl. the acceptance case and the wrong reclaim order run
deliberately), and the live stack, where a 90-day-old cancelled row was
swept and a 90-day-old scheduled row survived.

Co-Authored-By: Claude <noreply@anthropic.com>

server/test/engagementRetention.test.js

@@ -0,0 +1,234 @@
+// ── Retention: the engagement schema's sweep (ENGAGEMENT.md Phase 14) ──────
+//
+// The phase's acceptance line is that every one of the four tables has a stated
+// policy — a sweep with a horizon, or a recorded decision that it does not
+// expire — and that the cooldown horizon is CHECKED against the longest enabled
+// rule rather than picked. Both are pinned here, plus the three things building
+// it showed are silent when wrong:
+//
+//   • **the outbox sweep is terminal-only.** A `scheduled` row is a message this
+//     deployment still intends to send; `delay_seconds` can legitimately put one
+//     a day out. Sweeping by age alone would cancel sends nobody cancelled, and
+//     the operator would see only that the mail never arrived.
+//   • **`reclaimStale` has to give up.** `MAX_ATTEMPTS` is consulted only on a
+//     graceful `retry` outcome, so before Phase 14 a send that killed the
+//     process mid-flight cycled sending → scheduled → sending forever, never
+//     reached a terminal status, and was therefore never eligible for ANY
+//     retention sweep. The bound depends on this.
+//   • **an unreadable setting means the default, not an exception.** The sweep
+//     runs on a timer with nobody watching.
+//
+// Point the DB at a closed port before requiring anything: the registries reach
+// utils/discordAnnounce, which builds the pool at require time.
+process.env.DB_HOST = '127.0.0.1'
+process.env.DB_PORT = '59999'
+
+const { test, beforeEach, afterEach, after } = require('node:test')
+const assert = require('node:assert/strict')
+
+const retention = require('../src/model/engagement/engagementRetention.model')
+const prune = require('../src/utils/engagementRetentionPrune')
+const cooldownsDb = require('../src/model/engagement/engagementCooldowns.db')
+const outboxDb = require('../src/model/engagement/engagementOutbox.db')
+const sendsDb = require('../src/model/engagement/engagementSends.db')
+const settings = require('../src/model/settings/settings.model')
+const rulesDb = require('../src/model/engagement/engagementRules.db')
+const db = require('../src/utils/db')
+
+after(() => db.close())
+
+const saved = new Map()
+function patch(mod, name, fn) {
+  if (!saved.has(mod)) saved.set(mod, new Map())
+  if (!saved.get(mod).has(name)) saved.get(mod).set(name, mod[name])
+  mod[name] = fn
+}
+function restore() {
+  for (const [mod, names] of saved) for (const [name, fn] of names) mod[name] = fn
+  saved.clear()
+}
+
+let world
+
+beforeEach(() => {
+  world = {
+    settings: new Map(),
+    longestCooldown: 0,
+    // Every prune call, so a test can assert on the horizon it was given rather
+    // than only on the fact that something was deleted.
+    calls: { cooldowns: [], outbox: [], sends: [] },
+    deleted: { cooldowns: 0, outbox: 0, sends: 0 },
+  }
+  patch(settings, 'get', async (key) => {
+    if (!world.settings.has(key)) return null
+    return world.settings.get(key)
+  })
+  patch(settings, 'set', async (key, value) => { world.settings.set(key, value) })
+  patch(rulesDb, 'maxEnabledCooldownSeconds', async () => world.longestCooldown)
+  patch(cooldownsDb, 'prune', async (before, limit) => {
+    world.calls.cooldowns.push({ before, limit })
+    return world.deleted.cooldowns
+  })
+  patch(outboxDb, 'pruneTerminal', async (before, limit) => {
+    world.calls.outbox.push({ before, limit })
+    return world.deleted.outbox
+  })
+  patch(sendsDb, 'prune', async (before, limit) => {
+    world.calls.sends.push({ before, limit })
+    return world.deleted.sends
+  })
+})
+
+afterEach(() => restore())
+
+const daysBetween = (now, before) => Math.round((now.getTime() - before.getTime()) / 86400000)
+
+// ── The policy ─────────────────────────────────────────────────────────────
+
+test('defaults are the shipped policy when nothing is stored', async () => {
+  const policy = await retention.get()
+  assert.deepEqual(policy, { sends: 180, cooldowns: 30, outbox: 30 })
+})
+
+test('a stored value is used, and written back as a string', async () => {
+  await retention.set({ sends: 365 }, 7)
+  assert.equal(world.settings.get('engagement_sends_retain_days'), '365')
+  assert.equal((await retention.get()).sends, 365)
+})
+
+test('the PUT is sparse — an absent horizon is left alone', async () => {
+  await retention.set({ sends: 90 })
+  await retention.set({ cooldowns: 45 })
+  const policy = await retention.get()
+  assert.equal(policy.sends, 90, 'the earlier write survives the second call')
+  assert.equal(policy.cooldowns, 45)
+  assert.equal(policy.outbox, 30, 'and the untouched one is still the default')
+})
+
+test('out of range is refused rather than clamped', async () => {
+  // Clamping would leave the screen describing a policy the deployment is not
+  // running, which is worse than a visible error.
+  await assert.rejects(() => retention.set({ cooldowns: 1 }), /between 2 and 3650/)
+  await assert.rejects(() => retention.set({ sends: 5 }), /between 7 and 3650/)
+  await assert.rejects(() => retention.set({ outbox: 99999 }), /between 2 and 3650/)
+  await assert.rejects(() => retention.set({ sends: 12.5 }), /whole number/)
+  assert.equal(world.settings.size, 0, 'nothing was written')
+})
+
+test('an unknown key in the body is ignored, not rejected', async () => {
+  // So a client that posts the whole object back is not coupled to the list.
+  await retention.set({ sends: 200, suppressions: 30, nonsense: 1 })
+  assert.equal((await retention.get()).sends, 200)
+  assert.equal(world.settings.has('engagement_suppressions_retain_days'), false)
+})
+
+test('a stored value outside the bounds falls back to the default', async () => {
+  // A row written before the bounds existed, or by hand.
+  world.settings.set('engagement_cooldowns_retain_days', '0')
+  assert.equal((await retention.get()).cooldowns, 30)
+  world.settings.set('engagement_cooldowns_retain_days', 'soon')
+  assert.equal((await retention.get()).cooldowns, 30)
+})
+
+test('an unreadable settings table yields defaults rather than throwing', async () => {
+  patch(settings, 'get', async () => { throw new Error('pool is dead') })
+  assert.deepEqual(await retention.get(), { sends: 180, cooldowns: 30, outbox: 30 })
+})
+
+// ── The cooldown guard ─────────────────────────────────────────────────────
+
+test('the cooldown horizon is checked against the longest ENABLED rule', async () => {
+  world.longestCooldown = 86400 // the validated maximum, one day
+  const ok = await retention.checkCooldownHorizon(30)
+  assert.equal(ok.ok, true)
+  assert.equal(ok.message, null)
+
+  // A horizon shorter than a live cooldown means a pruned row makes the next
+  // fire a FIRST fire — the rule sends twice.
+  const bad = await retention.checkCooldownHorizon(0.5)
+  assert.equal(bad.ok, false)
+  assert.match(bad.message, /can send twice/)
+  assert.equal(bad.longestCooldownSeconds, 86400)
+})
+
+test('the horizon equal to the longest cooldown is refused, not accepted', async () => {
+  // Equality is the boundary where a row is pruned exactly as it stops being in
+  // force; the check has to be <=, not <.
+  world.longestCooldown = 2 * 86400
+  assert.equal((await retention.checkCooldownHorizon(2)).ok, false)
+  assert.equal((await retention.checkCooldownHorizon(3)).ok, true)
+})
+
+test('no enabled rule means no warning at any horizon', async () => {
+  world.longestCooldown = 0
+  assert.equal((await retention.checkCooldownHorizon(2)).ok, true)
+})
+
+// ── The sweep ──────────────────────────────────────────────────────────────
+
+test('one tick sweeps all three tables at their own horizons', async () => {
+  const now = new Date('2026-09-01T03:00:00Z')
+  const result = await prune.tick(now)
+
+  assert.equal(daysBetween(now, world.calls.sends[0].before), 180)
+  assert.equal(daysBetween(now, world.calls.cooldowns[0].before), 30)
+  assert.equal(daysBetween(now, world.calls.outbox[0].before), 30)
+  assert.deepEqual(result.warnings, [])
+})
+
+test('each statement is bounded', async () => {
+  await prune.tick(new Date())
+  for (const table of ['cooldowns', 'outbox', 'sends']) {
+    assert.equal(world.calls[table][0].limit, prune.BATCH, `${table} is batched`)
+  }
+})
+
+test('a sweep repeats while its batches come back full, and stops', async () => {
+  world.deleted.sends = prune.BATCH
+  const result = await prune.tick(new Date())
+  assert.equal(world.calls.sends.length, prune.MAX_BATCHES, 'it stops rather than looping forever')
+  assert.equal(result.sends, prune.BATCH * prune.MAX_BATCHES)
+})
+
+test('one failing table does not stop the other two', async () => {
+  patch(outboxDb, 'pruneTerminal', async () => { throw new Error('lock wait timeout') })
+  const result = await prune.tick(new Date())
+  assert.equal(result.outbox, 0)
+  assert.equal(world.calls.cooldowns.length, 1, 'cooldowns still swept')
+  assert.equal(world.calls.sends.length, 1, 'the send log still swept')
+})
+
+test('the sweep runs even when the cooldown horizon is too short, and warns', async () => {
+  // Deliberate: refusing to prune would trade a bounded, describable fault (one
+  // rule may re-fire early) for the unbounded one this phase exists to end.
+  world.longestCooldown = 86400
+  world.settings.set('engagement_cooldowns_retain_days', '2')
+  const result = await prune.tick(new Date())
+  assert.equal(result.warnings.length, 0, 'two days clears a one-day cooldown')
+
+  world.longestCooldown = 30 * 86400 // longer than any rule can actually save
+  const second = await prune.tick(new Date())
+  assert.equal(second.warnings.length, 1)
+  assert.equal(world.calls.cooldowns.length, 2, 'it swept anyway')
+})
+
+test('an unreadable policy skips the run rather than sweeping on a guess', async () => {
+  patch(retention, 'get', async () => { throw new Error('pool is dead') })
+  const result = await prune.tick(new Date())
+  assert.deepEqual(result, { cooldowns: 0, outbox: 0, sends: 0, warnings: [] })
+  assert.equal(world.calls.sends.length, 0, 'nothing was deleted')
+})
+
+test('start/stop is idempotent and leaves no live timer', () => {
+  prune.start()
+  prune.start()
+  prune.stop()
+  prune.stop()
+})
+
+// The SQL these depend on — terminal-only, the give-up-before-reclaim order and
+// the batch LIMIT — is proved against a real server in engagementRetentionSql.test.js.
+// It cannot be proved here: the db modules destructure `query` at require time,
+// so there is nothing left to stub, and a hand-rolled stand-in would prove only
+// that two readings of the manual agree (which is exactly the trap Phase 4a's
+// `foundRows` defect sprang).