feat(events): the resource ledger, leases and cleanup (Phase 8)

Event System Phase 8 (EVENTS_PLAN.md). Docs half: RunicGateway/docs#NNN.

One table, one core action, one route, one body field, and two members added to
MODULE_API 1.10.0 in place. The safety property the whole world-write half
depends on: core now remembers what a run changed in the world, and gives it
back on every terminal path.

Four decisions settled by the org lead on 2026-09-03, all as recommended:

- A lease is acquired by a new CORE action, `core.lease`. Section F puts the
  duration bound and the two-events-one-target conflict check on core's side of
  the seam, and a lease verb per module would be both re-implemented once per
  module, advisory everywhere.
- Record-before-confirm is a PLACEHOLDER keyed by the step's idempotency key. A
  spawn's ref does not exist until the module answers, so what core writes
  before the dispatch is `kind: '@step'`, `ref` = that key. If the answer never
  comes it stands, and cleanup calls revert() with the key and no resources --
  which is why section F's revert takes the key at all.
- Cleanup is one sweep over the ledger, not synthetic step rows. The
  step-shaped version costs a second retry counter beside `revert_attempts`.
- `reconcile` is declared here and TRIGGERED BY THE MODULE, through
  `ctx.events.reconcile()`. Core has no concept of the game being up, so it
  cannot decide when to ask; it asks once at its own boot.

MODULE_API stays 1.10.0. A protocol owes a bump once it has landed on `main`;
while it is on `edge` it is amended in place, so the whole module contract
reaches an author as one version they read once.

Verify

- `npm test` -- 2025 tests, 1935 pass, 89 skipped, 1 fail. That one is the
  pre-existing engagementManifest CRLF failure, in a file this branch does not
  touch (`edge` before: 1950/1876/73/1). +75 tests.
- The unique key was proved against a REAL MariaDB, because nothing else can
  prove it: whether multiple NULLs collide in a unique index, whether a STORED
  generated column is recomputed on UPDATE, and whether the SET NULL foreign key
  survives beside it are properties of the server. eventRunnerSql.test.js gained
  16 tests; 65 pass against the container. The real schema.sql was applied to a
  fresh database and to an existing one.
- Client: 362 pass, and it builds. routes:manifest and swagger -- one route
  added, none moved.

The live walk found three defects, and two of them are the phase's real finding

Driven by a throwaway `rig` module in website/modules/, deleted before commit.

1. A lease was never given back at all. `core.lease` reserves its own ledger
   row, so it never went through the ledger's dirty-marking, so a run holding
   only a lease kept `cleanup_status = 'not_required'` and the cleanup leg --
   which selected on `pending` -- never looked at it.
2. EVENT_REVERT_MAX_ATTEMPTS meant one attempt, not three. The first failing
   sweep moved the run to `incomplete`, which took it out of the leg's own scan
   for ever. The test covering the bound asserted `<= 3` and was satisfied by 1:
   a bound has two halves, and a test that only asserts the ceiling passes
   against a floor.
3. The first fix for (2) made the console lie. Spending every row's
   `revert_attempts` was a tidy way to take a `cleanup: false` run out of a
   counter-bounded scan, and the run page then rendered "3 attempts" beside
   resources nothing had ever tried. Found by opening the page.

Both (1) and (2) are the same mistake: deriving "is there anything to do" from a
summary column instead of from the rows. Neither was visible to a unit test,
because a test that calls the sweep directly never asks what would have selected
the run.

The two properties that need the process to die were walked as the plan asks.
With the module's perform() hanging, the placeholder existed while the dispatch
was in flight and nothing was named; after taskkill and a restart the reclaim
re-dispatched the same idempotency key, the retry re-used its own placeholder,
and everything was given back. Then, with the module reporting one of two
resources as no longer in force, the boot-time reconcile marked the other
`orphaned` -- never `reverted`.

This branch does NOT bump MODULE_API_VERSION, so the integration kit stays as
Phase 7 left it: red until the Phase 16 cutover re-pins ci/core-ref.json.

Co-Authored-By: Claude <noreply@anthropic.com>

server/src/model/events/eventRunControls.model.js

@@ -15,10 +15,13 @@
 // diagnosis panel: a screen that explains why a phase has not started, beside a
 // control that does something about it.
 //
-// **One of §I's controls is still not here.** `cleanup` needs Phase 8's resource
-// ledger; there is nothing to revert, so cancel takes `{ reason }` and gains
-// `cleanup` when there is something for it to do. Absent rather than inert,
-// which is the posture Phase 1 set and every phase since has kept.
+// **`cleanup` is the eighth, and Phase 8 is what gave it a ledger to work over.**
+// It re-runs the teardown across every resource a run has not given back, and it
+// is `admin` where the other seven are `admin` + `moderator`: it is not incident
+// response, it is asking core to write to the world again. Its partner is
+// cancel's new `cleanup: false`, which is §L's "cancelling WITHOUT cleanup is a
+// separate, logged, admin-only action" — deliberately the flag that has to be
+// asked for, because the safe default is to give back what the run took.
 //
 // **Every control is guarded on the status it may act from, and the guard is a
 // WHERE clause rather than a read-then-write.** A run console rendered thirty
@@ -37,6 +40,7 @@ const runsDb = require('./eventRuns.db')
 const stepsDb = require('./eventRunSteps.db')
 const logDb = require('./eventRunLog.db')
 const gatesDb = require('./eventPhaseGates.db')
+const resourcesDb = require('./eventRunResources.db')
 const gates = require('../../events/gates')
 
 const MAX_REASON = 500
@@ -152,11 +156,23 @@ async function resume(runId, options = {}, userId = null) {
  * and a second writer on that row would race the process that owns it. It
  * finishes into a cancelled run, which is honest.
  */
-async function cancel(runId, { reason } = {}, userId = null) {
+async function cancel(runId, { reason, cleanup = true } = {}, userId = null, { isAdmin = true } = {}) {
   const run = await loadRun(runId)
   if (!run) return { ok: false, status: 404, errors: ['no such run'] }
   if (runsDb.TERMINAL.includes(run.status)) return conflict(`this run is already ${run.status}`)
 
+  // §L: cancelling WITHOUT cleanup is a separate, logged, ADMIN-only action. The
+  // route itself is `admin` + `moderator`, so the narrower gate cannot live in
+  // middleware — which of the two you have to be depends on what is in the body,
+  // exactly as the authoring role floor does (§K).
+  if (cleanup === false && !isAdmin) {
+    return {
+      ok: false,
+      status: 403,
+      errors: ['leaving a run\'s world changes in place is an administrator\'s decision'],
+    }
+  }
+
   const note = clean(reason)
   const from = ['scheduled', 'starting', 'running', 'paused', 'ending']
   if (!(await runsDb.transition(run.id, from, 'cancelled', { error: note || 'cancelled by staff' }))) {
@@ -168,9 +184,83 @@ async function cancel(runId, { reason } = {}, userId = null) {
     runId: run.id,
     kind: 'run.status',
     phase: run.current_phase,
-    detail: { from: run.status, to: 'cancelled', control: 'cancel', by: userId, reason: note, cancelledSteps: closed },
+    detail: {
+      from: run.status,
+      to: 'cancelled',
+      control: 'cancel',
+      by: userId,
+      reason: note,
+      cancelledSteps: closed,
+      cleanup: cleanup !== false,
+    },
   })
-  return { ok: true, run: await runsDb.getById(run.id), cancelledSteps: closed }
+
+  // **The teardown is not done here, and the request does not wait for it.**
+  // Cleanup is one leg of the runner's tick over terminal runs (§L), which is
+  // what makes it survive a process that dies halfway through it — and a cancel
+  // pressed at two in the morning must answer at once rather than after a dozen
+  // round trips to a shard that may be the reason it is being cancelled. The run
+  // is terminal the moment this returns, so the very next tick picks its ledger
+  // up.
+  //
+  // `cleanup: false` is the operator saying leave it. The resources stay
+  // unresolved and the run carries `incomplete`, which is the truthful value: the
+  // world changes are still up, they are listed on the console, and the log line
+  // above records who decided that.
+  let cleanupStatus = run.cleanup_status
+  if (cleanup === false && (await resourcesDb.unresolvedCount(run.id)) > 0) {
+    // `incomplete` is what takes the run out of the cleanup leg's scan, and it is
+    // the truthful value: the world changes are still up, they are listed on the
+    // console, and the log line above records who decided that.
+    //
+    // **The first draft spent every row's `revert_attempts` instead**, to stop the
+    // sweep by the same mechanism a failed retry does. It worked and it made the
+    // console lie: the run page rendered "3 attempts" beside resources nothing had
+    // ever tried, which reads as "core tried three times and could not". Found by
+    // opening the page. A counter that means two things is a counter a screen
+    // cannot render.
+    await runsDb.setCleanupStatus(run.id, 'incomplete')
+    cleanupStatus = 'incomplete'
+  }
+
+  return {
+    ok: true,
+    run: await runsDb.getById(run.id),
+    cancelledSteps: closed,
+    cleanup: cleanup !== false,
+    cleanupStatus,
+  }
+}
+
+/**
+ * Re-run cleanup over everything a run has not given back.
+ *
+ * The manual retry §L promises, and the only thing that clears
+ * `revert_attempts`. That licence is the same one a human's step retry has, and
+ * it is deliberately not extended to the automatic sweep: Engagement Phase 14's
+ * defect was exactly a sweep that reset every stale row, which made the attempt
+ * ceiling unreachable and left the row cycling for ever.
+ *
+ * Legal on a TERMINAL run only. A run still in flight has a ledger that is still
+ * growing, and reverting a resource the next step is about to use would be core
+ * undoing an event while it is happening.
+ */
+async function cleanupRun(runId, userId = null) {
+  const run = await loadRun(runId)
+  if (!run) return { ok: false, status: 404, errors: ['no such run'] }
+  if (!runsDb.TERMINAL.includes(run.status)) {
+    return conflict(`this run is still ${run.status}; cancel it before cleaning up after it`)
+  }
+  if (run.cleanup_status === 'not_required') {
+    return conflict('this run recorded no resources, so there is nothing to give back')
+  }
+
+  // eslint-disable-next-line global-require
+  const summary = await require('../../events/cleanup').cleanupRun(run, {
+    resetAttempts: true,
+    actor: userId,
+  })
+  return { ok: true, run: await runsDb.getById(run.id), summary }
 }
 
 /**
@@ -359,4 +449,4 @@ async function retryStep(runId, stepId, options = {}, userId = null) {
   }
 }
 
-module.exports = { pause, resume, cancel, advancePhase, confirmStep, skipStep, retryStep }
+module.exports = { pause, resume, cancel, cleanupRun, advancePhase, confirmStep, skipStep, retryStep }

server/src/model/events/eventRunLog.db.js

@@ -47,6 +47,16 @@ const KINDS = [
   'run.budget',       // the caps this run was seeded with, and which switch set each
   'step.refused',     // a step was not permitted: disabled, or over a cap
   'version.verified', // a dry run passed against a version, unlocking scheduled starts
+  // Phase 8's six, and every one of them is an answer to "what did this event
+  // leave behind". `resource.recorded` is written at the ANSWER rather than at
+  // the placeholder, because a placeholder is a promise and the operator's
+  // question is about the world.
+  'resource.recorded',  // a step reported what it created or borrowed, and it is ledgered
+  'resource.orphaned',  // a module reports a ledgered resource is no longer in force
+  'cleanup.reverted',   // a group of resources came back
+  'cleanup.failed',     // a group did not, with the reason and how it was left
+  'cleanup.swept',      // one pass over a run's ledger, and what it found
+  'cleanup.retry',      // a human cleared the attempt counter and asked again
 ]
 
 const hydrate = (row) => row && { ...row, detail: parseJson(row.detail, null) }

server/src/model/events/eventRunResources.db.js

@@ -0,0 +1,353 @@
+// ── event_run_resources — SQL only ─────────────────────────────────────────
+//
+// EVENTS.md §D and §L ("The ledger's two rules"), and Phase 8 of EVENTS_PLAN.md.
+// Everything one run created or leased, and what became of it.
+//
+// **Rule 1 lives in `reserve()`.** A resource is recorded BEFORE it is
+// confirmed, so the placeholder this writes is the row that exists while the
+// dispatch is in flight — and the row that SURVIVES when the acknowledgement is
+// lost. Recording on the answer instead would make every object whose ack went
+// missing invisible to cleanup for ever.
+//
+// **Rule 2 lives in the status column and in `failRevert()`.** A revert that
+// never succeeds leaves its row unreverted, with the error on it, and the run
+// completes with `cleanup_status = 'incomplete'` rather than being held open.
+// Loud and sticky.
+//
+// **The unique key is enforced by the database, not by a read.** `reserve()`
+// answers `{ ok: false, code: 'held' }` on a duplicate key rather than checking
+// first and then inserting — two runs entering the same tick would both pass the
+// check. It is the argument `event_run_budget.spend()` makes about the cap and
+// `runsDb.transition` makes about a status, in the third place it applies.
+
+const { query } = require('../../utils/db')
+const { parseJson } = require('./eventJson')
+
+// The one `kind` core owns. A module's kinds are opaque and stored verbatim; this
+// one is core's own, and `registries` refuses a module resource that claims it.
+const STEP_KIND = '@step'
+
+// The statuses that mean "core still believes this resource is this run's". They
+// are exactly the ones the `live_marker` generated column keeps non-NULL, so the
+// unique target key holds while a row is in one of them and releases when it
+// leaves. Duplicated here as a JavaScript list because the sweeps read by it too,
+// and a second copy that can drift is better than a query that cannot express it.
+const HELD = ['pending', 'confirmed', 'reverting']
+
+// Every status that still wants a human or a retry: `HELD` plus the two that mean
+// "we let go, and not cleanly". This is what "unreverted" means everywhere in
+// this feature — the console's list, `cleanup_status`, and the manual retry.
+const UNRESOLVED = [...HELD, 'orphaned', 'drifted']
+
+const COLUMNS = `id, run_id, step_id, owner_module, kind, ref, payload, lease_until,
+                 status, revert_attempts, last_error, member_key, created_at, updated_at`
+
+// `payload` is opaque to core and stored verbatim, but it comes back as a string
+// from the driver and every caller wants the object — the cleanup sweep reads a
+// lease's baseline out of it, and the console renders it. Hydrated here for the
+// same reason a step's params are: one place rather than at each read.
+const hydrate = (row) => row && { ...row, payload: parseJson(row.payload, null) }
+
+/**
+ * Record a resource that does not exist yet.
+ *
+ * Answers `{ ok: true, id }`, or `{ ok: false, code: 'held', holder }` when the
+ * target is already someone's — which is the lease conflict, surfaced as a
+ * refusal rather than a failure because nothing is wrong with the system: another
+ * run has the thing.
+ *
+ * **`ER_DUP_ENTRY` is the check.** The holder is looked up only to name it in the
+ * refusal, and only after the insert has already lost the race.
+ */
+async function reserve({ runId, stepId = null, owner, kind, ref, payload = null, leaseUntil = null, memberKey = null }) {
+  try {
+    const result = await query(
+      `INSERT INTO event_run_resources
+         (run_id, step_id, owner_module, kind, ref, payload, lease_until, member_key, status)
+       VALUES (?, ?, ?, ?, ?, ?, ?, ?, 'pending')`,
+      [runId, stepId, owner, kind, ref, payload === null ? null : JSON.stringify(payload), leaseUntil, memberKey],
+    )
+    return { ok: true, id: Number(result.insertId) }
+  } catch (err) {
+    if (err && (err.code === 'ER_DUP_ENTRY' || err.errno === 1062)) {
+      const [holder] = await query(
+        `SELECT run_id, status FROM event_run_resources
+          WHERE owner_module = ? AND kind = ? AND ref = ? AND status IN (?, ?, ?)
+          LIMIT 1`,
+        [owner, kind, ref, ...HELD],
+      )
+      return { ok: false, code: 'held', holder: holder || null }
+    }
+    throw err
+  }
+}
+
+/**
+ * Promote a reserved row to `confirmed`, optionally attaching what the module
+ * finally said about it.
+ *
+ * Guarded on `pending` so a late answer cannot un-revert a row cleanup has
+ * already dealt with — the same reason every other write in this feature is a
+ * compare-and-set rather than a read-then-write.
+ */
+async function confirm(id, { payload, leaseUntil, memberKey } = {}) {
+  const sets = ["status = 'confirmed'"]
+  const params = []
+  if (payload !== undefined) {
+    sets.push('payload = ?')
+    params.push(payload === null ? null : JSON.stringify(payload))
+  }
+  if (leaseUntil !== undefined) {
+    sets.push('lease_until = ?')
+    params.push(leaseUntil)
+  }
+  if (memberKey !== undefined) {
+    sets.push('member_key = ?')
+    params.push(memberKey)
+  }
+  const result = await query(
+    `UPDATE event_run_resources SET ${sets.join(', ')} WHERE id = ? AND status = 'pending'`,
+    [...params, id],
+  )
+  return (result.affectedRows || 0) > 0
+}
+
+/**
+ * Resolve a step placeholder once the module has named what it actually made.
+ *
+ * The placeholder's whole job is over at this point: the real rows exist, so the
+ * `@step` row must stop being one of the things cleanup will try to revert.
+ * `reverted` is the honest terminal state for it — there is nothing left to undo
+ * that the rows it stood in for do not now cover — and it releases the
+ * idempotency key for a later run, which matters because keys are per step and a
+ * re-materialised step reuses its own.
+ */
+async function resolvePlaceholder(id) {
+  const result = await query(
+    `UPDATE event_run_resources
+        SET status = 'reverted', last_error = NULL
+      WHERE id = ? AND kind = ? AND status IN ('pending', 'confirmed')`,
+    [id, STEP_KIND],
+  )
+  return (result.affectedRows || 0) > 0
+}
+
+/**
+ * One row by its target, live or not — how a caller that lost the insert race
+ * finds the row it meant to write. Newest first, so a target that has been held
+ * and released several times answers with the current holder.
+ */
+async function findByTarget(owner, kind, ref) {
+  const [row] = await query(
+    `SELECT ${COLUMNS} FROM event_run_resources
+      WHERE owner_module = ? AND kind = ? AND ref = ?
+      ORDER BY id DESC LIMIT 1`,
+    [owner, kind, ref],
+  )
+  return hydrate(row) || null
+}
+
+/** One run's whole ledger, oldest first — the console's read. */
+async function forRun(runId) {
+  const rows = await query(
+    `SELECT ${COLUMNS} FROM event_run_resources WHERE run_id = ? ORDER BY id`,
+    [runId],
+  )
+  return rows.map(hydrate)
+}
+
+/** The rows of one run that still want something: the cleanup sweep's input. */
+async function unresolvedForRun(runId, { maxAttempts = null } = {}) {
+  const params = [runId, ...UNRESOLVED]
+  const attemptClause = maxAttempts === null ? '' : ' AND revert_attempts < ?'
+  if (maxAttempts !== null) params.push(maxAttempts)
+  const rows = await query(
+    `SELECT ${COLUMNS} FROM event_run_resources
+      WHERE run_id = ? AND status IN (?, ?, ?, ?, ?)${attemptClause}
+      ORDER BY id`,
+    params,
+  )
+  return rows.map(hydrate)
+}
+
+/** How many of one run's rows are still unresolved — what `cleanup_status` is derived from. */
+async function unresolvedCount(runId) {
+  const [row] = await query(
+    `SELECT COUNT(*) AS n FROM event_run_resources
+      WHERE run_id = ? AND status IN (?, ?, ?, ?, ?)`,
+    [runId, ...UNRESOLVED],
+  )
+  return Number(row?.n || 0)
+}
+
+/** Unresolved counts for several runs at once, keyed by run id — the run LIST's read. */
+async function unresolvedCounts(runIds) {
+  const ids = [...new Set(runIds || [])].filter(Boolean)
+  if (!ids.length) return new Map()
+  const rows = await query(
+    `SELECT run_id, COUNT(*) AS n FROM event_run_resources
+      WHERE run_id IN (${ids.map(() => '?').join(',')}) AND status IN (?, ?, ?, ?, ?)
+      GROUP BY run_id`,
+    [...ids, ...UNRESOLVED],
+  )
+  return new Map(rows.map((r) => [r.run_id, Number(r.n)]))
+}
+
+/**
+ * Claim one row for a revert: `pending | confirmed | orphaned | drifted → reverting`.
+ *
+ * The compare-and-set that keeps the cleanup leg and the manual cleanup route off
+ * each other's rows. `reverting` is deliberately not claimable — a row another
+ * pass is mid-revert on is left alone, exactly as a step with a live claim is.
+ */
+async function claimRevert(id) {
+  const result = await query(
+    `UPDATE event_run_resources
+        SET status = 'reverting'
+      WHERE id = ? AND status IN ('pending', 'confirmed', 'orphaned', 'drifted')`,
+    [id],
+  )
+  return (result.affectedRows || 0) > 0
+}
+
+/** The revert worked. `reverted` is terminal and releases the target. */
+async function markReverted(id) {
+  await query(
+    `UPDATE event_run_resources SET status = 'reverted', last_error = NULL WHERE id = ?`,
+    [id],
+  )
+}
+
+/**
+ * The revert did not work, and the row goes back to being unresolved.
+ *
+ * `revert_attempts` is incremented here and NOWHERE else, and it is never reset by
+ * a sweep — Engagement Phase 14's rule, whose defect was a reclaim that returned
+ * every stale row to its start state and made the attempt ceiling unreachable, so
+ * the row cycled for ever and was never eligible for any retention sweep. The one
+ * thing that may reset it is a human pressing cleanup, which is the same licence
+ * a human's step retry has.
+ *
+ * `restoreTo` is where the row lands: `drifted` when the module says somebody else
+ * moved the value, `orphaned` when it says the thing is gone, and `confirmed`
+ * otherwise — still ours, still out there, try again.
+ */
+async function failRevert(id, error, restoreTo = 'confirmed') {
+  await query(
+    `UPDATE event_run_resources
+        SET status = ?, revert_attempts = revert_attempts + 1, last_error = ?
+      WHERE id = ?`,
+    [restoreTo, String(error || 'the revert did not answer').slice(0, 500), id],
+  )
+}
+
+/**
+ * A human is trying again: clear the attempt counter on one run's unresolved rows.
+ *
+ * Only ever called from the cleanup route with an actor behind it. The automatic
+ * leg must never do this (see `failRevert`).
+ */
+async function resetAttempts(runId) {
+  const result = await query(
+    `UPDATE event_run_resources
+        SET revert_attempts = 0
+      WHERE run_id = ? AND status IN (?, ?, ?, ?, ?)`,
+    [runId, ...UNRESOLVED],
+  )
+  return result.affectedRows || 0
+}
+
+/** Every live row one module owns, for the reconcile sweep. */
+async function liveForModule(owner, { limit = 500 } = {}) {
+  const rows = await query(
+    `SELECT ${COLUMNS} FROM event_run_resources
+      WHERE owner_module = ? AND status IN ('pending', 'confirmed')
+      ORDER BY id LIMIT ?`,
+    [owner, Number(limit)],
+  )
+  return rows.map(hydrate)
+}
+
+/** Every module that currently owns a live row — who the reconcile sweep asks. */
+async function modulesWithLiveRows() {
+  const rows = await query(
+    `SELECT DISTINCT owner_module FROM event_run_resources
+      WHERE status IN ('pending', 'confirmed')`,
+  )
+  return rows.map((r) => r.owner_module)
+}
+
+/**
+ * The game no longer has it. Never reached by a revert — a revert that finds
+ * nothing there is a SUCCESS (§L, and it is what a Rust wipe needs) — only by
+ * reconcile, which is a different question: nobody asked for this to go.
+ */
+async function markOrphaned(id, detail = null) {
+  await query(
+    `UPDATE event_run_resources
+        SET status = 'orphaned', last_error = ?
+      WHERE id = ? AND status IN ('pending', 'confirmed', 'reverting')`,
+    [detail === null ? null : String(detail).slice(0, 500), id],
+  )
+}
+
+/**
+ * Terminal runs that still owe the world something — the cleanup leg's scan.
+ *
+ * **Both halves of the WHERE were live-walk findings, and they are opposite
+ * mistakes.**
+ *
+ * `cleanup_status = 'pending'` alone missed a run whose only resource was a
+ * LEASE: `core.lease` reserves its own row and never goes through the ledger's
+ * `markRunDirty`, so the flag stayed `not_required` and the lease was never given
+ * back at all. Hence `not_required` is in the list — a terminal run with an
+ * unresolved row has something to do whatever any summary column says, and
+ * treating that combination as work is the fail-safe direction.
+ *
+ * And the run status filter alone made `MAX_REVERT_ATTEMPTS` mean ONE attempt,
+ * because the first failing sweep set `incomplete` and nothing looked at the run
+ * again. That is fixed in `cleanupRun`, which now only writes `incomplete` once
+ * there is nothing left it will try — so `incomplete` genuinely means "finished
+ * with, and not finished", which is exactly what excludes both a run whose
+ * retries are spent and a run an admin cancelled without cleanup.
+ *
+ * The attempt bound is in the join for a different reason: without it a run whose
+ * rows are all spent would be selected, worked over and found to have nothing to
+ * do on every tick for the rest of its life.
+ */
+async function runsNeedingCleanup(limit = 25, maxAttempts = 3) {
+  return query(
+    `SELECT DISTINCT r.id, r.status, r.cleanup_status, r.version_id, r.definition_id, r.scope
+       FROM event_runs r
+       JOIN event_run_resources res ON res.run_id = r.id
+      WHERE r.status IN ('completed', 'cancelled', 'failed', 'missed')
+        AND r.cleanup_status IN ('pending', 'not_required')
+        AND res.status IN (?, ?, ?, ?, ?)
+        AND res.revert_attempts < ?
+      ORDER BY r.id
+      LIMIT ?`,
+    [...UNRESOLVED, Number(maxAttempts), Number(limit)],
+  )
+}
+
+module.exports = {
+  STEP_KIND,
+  HELD,
+  UNRESOLVED,
+  reserve,
+  confirm,
+  resolvePlaceholder,
+  findByTarget,
+  forRun,
+  unresolvedForRun,
+  unresolvedCount,
+  unresolvedCounts,
+  claimRevert,
+  markReverted,
+  failRevert,
+  resetAttempts,
+  liveForModule,
+  modulesWithLiveRows,
+  markOrphaned,
+  runsNeedingCleanup,
+}

server/src/model/events/eventRuns.db.js

@@ -386,6 +386,32 @@ async function setHealth(id, health) {
   return Number(result?.affectedRows || 0) === 1
 }
 
+/**
+ * Set `cleanup_status`, optionally guarded on where it is now (Phase 8).
+ *
+ * Four values and three writers, which is why the guard is a parameter rather
+ * than baked in. The ledger stamps `pending` the first time a run records
+ * anything, and it must do so only over `not_required` — a run already marked
+ * `complete` must not be walked back to `pending` by a late resource, and a
+ * `incomplete` one must not be silently tidied. The cleanup sweep sets `complete`
+ * or `incomplete` from what it found, unguarded, because the sweep IS the
+ * authority on that. A human's cleanup route re-opens `pending` deliberately, and
+ * says so in the log with the actor.
+ *
+ * **`pending` on a run that is still running is not a bug and reads correctly**:
+ * there is something to clean up and it has not happened yet. The alternative -
+ * a fifth value meaning "there will be something later" - is a state nothing
+ * would ever branch on.
+ */
+async function setCleanupStatus(id, to, from = null) {
+  const guard = from === null ? '' : ` AND cleanup_status IN (${from.map(() => '?').join(',')})`
+  const result = await query(
+    `UPDATE event_runs SET cleanup_status = ? WHERE id = ?${guard}`,
+    from === null ? [to, id] : [to, id, ...from],
+  )
+  return Number(result?.affectedRows || 0) === 1
+}
+
 /**
  * Runs whose start instant passed more than their own grace window ago (§E, §L).
  *
@@ -488,6 +514,7 @@ module.exports = {
   statusOf,
   transition,
   setHealth,
+  setCleanupStatus,
   concurrencyHolder,
   reclaimStale,
   terminalBefore,

server/src/model/events/eventRuns.model.js

@@ -27,6 +27,7 @@ const definitionsDb = require('./eventDefinitions.db')
 const versionsDb = require('./eventVersions.db')
 const settingsDb = require('./eventActionSettings.db')
 const budgetDb = require('./eventRunBudget.db')
+const resourcesDb = require('./eventRunResources.db')
 const authorize = require('../../events/authorize')
 
 const MAX_SCOPE = 190
@@ -204,11 +205,12 @@ async function create(
 async function detail(runId) {
   const run = await db.getById(runId)
   if (!run) return null
-  const [steps, counts, gateRows, budget] = await Promise.all([
+  const [steps, counts, gateRows, budget, resources] = await Promise.all([
     stepsDb.listForRun(runId),
     stepsDb.statusCounts(runId),
     gatesDb.listForRun(runId),
     budgetDb.forRun(runId),
+    resourcesDb.forRun(runId),
   ])
   const now = new Date()
   return {
@@ -226,6 +228,37 @@ async function detail(runId) {
       cap: b.cap,
       from: b.effective_from,
     })),
+    // What this run changed in the world, and what became of it (Phase 8). The
+    // WHOLE ledger, reverted rows included, because "what did last night's
+    // invasion actually spawn, and did all of it come back" is the question this
+    // panel exists for and a list of only the failures cannot answer the second
+    // half of it.
+    //
+    // **The `@step` placeholders are filtered out.** They are core's own
+    // bookkeeping — a row that says "a dispatch is in flight and may have made
+    // something" — and the console's list is of things in the world. One left in
+    // would read as a resource nobody can name, which is exactly the confusion it
+    // exists to prevent internally.
+    resources: resources
+      .filter((r) => r.kind !== resourcesDb.STEP_KIND)
+      .map((r) => ({
+        id: r.id,
+        stepId: r.step_id,
+        module: r.owner_module,
+        kind: r.kind,
+        ref: r.ref,
+        payload: r.payload,
+        leaseUntil: r.lease_until,
+        status: r.status,
+        revertAttempts: r.revert_attempts,
+        lastError: r.last_error,
+        memberKey: r.member_key,
+        createdAt: r.created_at,
+      })),
+    // How many rows are still unresolved, counted over the WHOLE ledger rather
+    // than over the list above — a placeholder left standing by a lost
+    // acknowledgement is exactly the case `cleanup_status` must not call clean.
+    unresolvedResources: resources.filter((r) => resourcesDb.UNRESOLVED.includes(r.status)).length,
   }
 }