Files
website/server/src/model/engagement/engagementRules.db.js
wtclaude 5779d15150
All checks were successful
PR Checks / client-build (pull_request) Successful in 34s
PR Checks / bot-tests (pull_request) Successful in 34s
PR Checks / server-tests (pull_request) Successful in 13m23s
feat(engagement): retention — three sweeps and one recorded refusal
ENGAGEMENT.md Phase 14, the last phase of the workstream. Four engagement
tables grew on every fire and nothing had ever deleted from any of them.

Three of them now have a horizon, swept nightly by one worker
(utils/engagementRetentionPrune.js — setInterval + unref + stop(), batched
1000 x 50, each table's failure caught on its own so a lock timeout on one
does not leave the other two unbounded):

  engagement_sends      180 days   engagement_sends_retain_days      (7-3650)
  engagement_cooldowns   30 days   engagement_cooldowns_retain_days  (2-3650)
  engagement_outbox      30 days   engagement_outbox_retain_days     (2-3650)

The fourth, engagement_suppressions, does not expire, and that is the
recorded decision rather than an omission: a suppression is a standing
decision, and ageing out a hard bounce re-mails an address that already
bounced. The way out stays deliberate, and is now reachable per row.

Six decisions were settled by the org lead before any code. Two of them
widened the phase past what was offered:

  * the send-log horizon is admin-configurable, so retention got a SCREEN
    (Admin -> Engagement -> Retention) where team_activity and
    user_notifications keep theirs in invisible settings rows. The send-log
    horizon changes what an operator-facing page is able to show, so it has
    to be visible; the other two came with it, because "what does this
    deployment keep" is one question.
  * the suppression purge, which cost a Phase 9 decision. The list
    deliberately stripped address_hash from every row, so the only way out
    was a window.prompt asking the operator to retype an address the screen
    has never shown them. The row had no handle at all. The hash is now
    returned: this route is admin-only and an admin can already suppress and
    unsuppress any address they can name, so it grants no capability they
    lack. GET /sends still strips its own.

The outbox sweep is TERMINAL-ONLY and that is a correctness rule: a
scheduled row is a send this deployment still intends to make (delay_seconds
can put one a day out) and a sending row may be mid-flight.

One shipped defect had to be fixed for the sweep to be a bound at all.
reclaimStale returned every stale sending row to scheduled, and MAX_ATTEMPTS
is consulted only on a graceful retry outcome — so a send that killed the
process mid-flight cycled sending -> scheduled -> sending forever, never
terminal, therefore never eligible for any sweep. It now fails an exhausted
row BEFORE reclaiming the rest; the order is the fix.

Two indexes (idx_engo_sweep, idx_engs_sweep): every existing index on those
tables has created_at in second position, which serves a per-rule window and
is useless to a whole-table horizon.

Proved twice: engagementRetentionSql.test.js against a real MariaDB (7
tests, incl. the acceptance case and the wrong reclaim order run
deliberately), and the live stack, where a 90-day-old cancelled row was
swept and a 90-day-old scheduled row survived.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-01 15:40:53 -05:00

166 lines
5.4 KiB
JavaScript

const { query } = require('../../utils/db')
// JSON columns come back from the driver already parsed on some MariaDB/driver
// combinations and as a string on others (it depends on whether the column is a
// real JSON type or the LONGTEXT + CHECK alias MariaDB implements it as). Every
// read below goes through this, so no caller has to know which it got.
function parseJson(value, fallback) {
if (value === null || value === undefined) return fallback
if (typeof value !== 'string') return value
try {
return JSON.parse(value)
} catch {
return fallback
}
}
const hydrate = (row) =>
row && {
...row,
enabled: Boolean(row.enabled),
channels: parseJson(row.channels, []),
template_keys: parseJson(row.template_keys, {}),
conditions: parseJson(row.conditions, null),
cancel_on: parseJson(row.cancel_on, []),
}
const list = async () =>
(await query('SELECT * FROM engagement_rules ORDER BY trigger_id, name, id')).map(hydrate)
const getById = async (id) => {
const [row] = await query('SELECT * FROM engagement_rules WHERE id = ?', [id])
return hydrate(row)
}
/**
* Every ENABLED rule for one trigger. The engine's hot path: one indexed read
* per emit, and `idx_engr_trigger (trigger_id, enabled)` is exactly this query.
*/
const enabledForTrigger = async (triggerId) =>
(await query('SELECT * FROM engagement_rules WHERE trigger_id = ? AND enabled = 1', [triggerId])).map(hydrate)
/**
* Every enabled rule that names `triggerId` in its `cancel_on`.
*
* A JSON_CONTAINS rather than a scan: `cancel_on` is a small array on a small
* table, but this runs on EVERY emit — including the overwhelming majority that
* cancel nothing — so it must not be a full table read of the rule set.
*/
const enabledCancelledBy = async (triggerId) =>
(
await query(
"SELECT * FROM engagement_rules WHERE enabled = 1 AND cancel_on IS NOT NULL AND JSON_CONTAINS(cancel_on, JSON_QUOTE(?))",
[triggerId],
)
).map(hydrate)
const insert = async (rule) => {
const result = await query(
`INSERT INTO engagement_rules
(trigger_id, name, enabled, audience, audience_segment_id, max_sends_per_hour,
channels, template_keys, conditions, cooldown_seconds, delay_seconds, cancel_on, updated_by)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
[
rule.trigger_id,
rule.name,
rule.enabled ? 1 : 0,
rule.audience,
rule.audience_segment_id,
rule.max_sends_per_hour,
JSON.stringify(rule.channels),
JSON.stringify(rule.template_keys),
rule.conditions === null ? null : JSON.stringify(rule.conditions),
rule.cooldown_seconds,
rule.delay_seconds,
JSON.stringify(rule.cancel_on || []),
rule.updated_by,
],
)
return result.insertId
}
const update = (id, rule) =>
query(
`UPDATE engagement_rules
SET name = ?, enabled = ?, audience = ?, audience_segment_id = ?, max_sends_per_hour = ?,
channels = ?, template_keys = ?, conditions = ?, cooldown_seconds = ?,
delay_seconds = ?, cancel_on = ?, updated_by = ?
WHERE id = ?`,
[
rule.name,
rule.enabled ? 1 : 0,
rule.audience,
rule.audience_segment_id,
rule.max_sends_per_hour,
JSON.stringify(rule.channels),
JSON.stringify(rule.template_keys),
rule.conditions === null ? null : JSON.stringify(rule.conditions),
rule.cooldown_seconds,
rule.delay_seconds,
JSON.stringify(rule.cancel_on || []),
rule.updated_by,
id,
],
)
/**
* Flip `enabled` and nothing else (Phase 4b).
*
* Deliberately NOT a call through `validate`: turning a rule OFF is the panic
* button, and it has to work on a rule the registries would now refuse — one
* whose module was uninstalled, or whose trigger has since narrowed its ceiling
* underneath a saved audience. Re-validating on the way to `enabled = 0` would
* make exactly the rules an operator most wants to stop the ones they cannot.
*
* Turning a rule ON is safe without re-validation for a different reason: the
* engine re-runs the ceiling check at send time (audiences.permitted), so an
* enabled-but-no-longer-permitted rule resolves to nobody rather than to the
* wrong people.
*/
const setEnabled = (id, enabled, updatedBy = null) =>
query('UPDATE engagement_rules SET enabled = ?, updated_by = ? WHERE id = ?', [
enabled ? 1 : 0,
updatedBy,
id,
])
const remove = (id) => query('DELETE FROM engagement_rules WHERE id = ?', [id])
/** Does any rule still point at this segment? The check before a segment delete. */
const countUsingSegment = async (segmentId) => {
const [row] = await query(
'SELECT COUNT(*) AS n FROM engagement_rules WHERE audience_segment_id = ?',
[segmentId],
)
return Number(row?.n || 0)
}
/**
* The longest cooldown any ENABLED rule configures, in seconds (Phase 14).
*
* Enabled only, deliberately: a disabled rule fires nothing, so it writes no
* cooldown row a retention sweep could destroy, and letting a forgotten disabled
* rule with a 24-hour cooldown veto a tighter horizon would make the warning
* advice nobody can act on.
*/
const maxEnabledCooldownSeconds = async () => {
const [row] = await query(
'SELECT MAX(cooldown_seconds) AS n FROM engagement_rules WHERE enabled = 1',
)
return Number(row?.n || 0)
}
module.exports = {
list,
getById,
maxEnabledCooldownSeconds,
enabledForTrigger,
enabledCancelledBy,
insert,
update,
setEnabled,
remove,
countUsingSegment,
parseJson,
}