feat(events): the resource ledger, leases and cleanup (Phase 8)

Event System Phase 8 (EVENTS_PLAN.md). Docs half: RunicGateway/docs#NNN.

One table, one core action, one route, one body field, and two members added to
MODULE_API 1.10.0 in place. The safety property the whole world-write half
depends on: core now remembers what a run changed in the world, and gives it
back on every terminal path.

Four decisions settled by the org lead on 2026-09-03, all as recommended:

- A lease is acquired by a new CORE action, `core.lease`. Section F puts the
  duration bound and the two-events-one-target conflict check on core's side of
  the seam, and a lease verb per module would be both re-implemented once per
  module, advisory everywhere.
- Record-before-confirm is a PLACEHOLDER keyed by the step's idempotency key. A
  spawn's ref does not exist until the module answers, so what core writes
  before the dispatch is `kind: '@step'`, `ref` = that key. If the answer never
  comes it stands, and cleanup calls revert() with the key and no resources --
  which is why section F's revert takes the key at all.
- Cleanup is one sweep over the ledger, not synthetic step rows. The
  step-shaped version costs a second retry counter beside `revert_attempts`.
- `reconcile` is declared here and TRIGGERED BY THE MODULE, through
  `ctx.events.reconcile()`. Core has no concept of the game being up, so it
  cannot decide when to ask; it asks once at its own boot.

MODULE_API stays 1.10.0. A protocol owes a bump once it has landed on `main`;
while it is on `edge` it is amended in place, so the whole module contract
reaches an author as one version they read once.

Verify

- `npm test` -- 2025 tests, 1935 pass, 89 skipped, 1 fail. That one is the
  pre-existing engagementManifest CRLF failure, in a file this branch does not
  touch (`edge` before: 1950/1876/73/1). +75 tests.
- The unique key was proved against a REAL MariaDB, because nothing else can
  prove it: whether multiple NULLs collide in a unique index, whether a STORED
  generated column is recomputed on UPDATE, and whether the SET NULL foreign key
  survives beside it are properties of the server. eventRunnerSql.test.js gained
  16 tests; 65 pass against the container. The real schema.sql was applied to a
  fresh database and to an existing one.
- Client: 362 pass, and it builds. routes:manifest and swagger -- one route
  added, none moved.

The live walk found three defects, and two of them are the phase's real finding

Driven by a throwaway `rig` module in website/modules/, deleted before commit.

1. A lease was never given back at all. `core.lease` reserves its own ledger
   row, so it never went through the ledger's dirty-marking, so a run holding
   only a lease kept `cleanup_status = 'not_required'` and the cleanup leg --
   which selected on `pending` -- never looked at it.
2. EVENT_REVERT_MAX_ATTEMPTS meant one attempt, not three. The first failing
   sweep moved the run to `incomplete`, which took it out of the leg's own scan
   for ever. The test covering the bound asserted `<= 3` and was satisfied by 1:
   a bound has two halves, and a test that only asserts the ceiling passes
   against a floor.
3. The first fix for (2) made the console lie. Spending every row's
   `revert_attempts` was a tidy way to take a `cleanup: false` run out of a
   counter-bounded scan, and the run page then rendered "3 attempts" beside
   resources nothing had ever tried. Found by opening the page.

Both (1) and (2) are the same mistake: deriving "is there anything to do" from a
summary column instead of from the rows. Neither was visible to a unit test,
because a test that calls the sweep directly never asks what would have selected
the run.

The two properties that need the process to die were walked as the plan asks.
With the module's perform() hanging, the placeholder existed while the dispatch
was in flight and nothing was named; after taskkill and a restart the reclaim
re-dispatched the same idempotency key, the retry re-used its own placeholder,
and everything was given back. Then, with the module reporting one of two
resources as no longer in force, the boot-time reconcile marked the other
`orphaned` -- never `reverted`.

This branch does NOT bump MODULE_API_VERSION, so the integration kit stays as
Phase 7 left it: red until the Phase 16 cutover re-pins ci/core-ref.json.

Co-Authored-By: Claude <noreply@anthropic.com>

server/test/eventLedger.test.js

@@ -0,0 +1,337 @@
+// ── The resource ledger's write half (EVENTS_PLAN.md Phase 8) ──────────────
+//
+// §D's two rules, and rule 1 is the one this file exists for: **a resource is
+// recorded BEFORE it is confirmed.** The obstacle it works around is that a
+// spawn's serial does not exist until the module answers, so what goes in before
+// the dispatch is a placeholder keyed by the step's idempotency key — and the
+// property worth a test is that the placeholder SURVIVES an answer that never
+// comes, because that is the case where recording afterwards would have lost the
+// object for ever.
+//
+// The other rules here are about what core will and will not write down on a
+// module's say-so. Every one of them is fail-closed in a specific direction:
+//
+//   • the reserved `@step` kind is core's and a module may not claim it
+//   • an `override` must name a lease core knows how to give back, or core would
+//     be recording something it has no way to restore
+//   • a duplicate is "already recorded", not an error — a retry re-sends the same
+//     idempotency key and a module may honestly report the same resources twice
+//   • a badly shaped resource is dropped and LOGGED, never a failed step: the
+//     step changed the world, and turning bookkeeping into a retry would re-run
+//     a world write that already happened
+//
+// The db layer is stubbed with a store that enforces `uq_evres_target`, because
+// that refusal is behaviour the callers branch on rather than an implementation
+// detail. The SQL itself is `eventRunnerSql.test.js`'s, against a real MariaDB.
+
+process.env.DB_HOST = '127.0.0.1'
+process.env.DB_PORT = '59999'
+
+const { test, beforeEach, afterEach, after } = require('node:test')
+const assert = require('node:assert/strict')
+
+const registries = require('../src/modules/registries')
+const ledger = require('../src/events/ledger')
+const resourcesDb = require('../src/model/events/eventRunResources.db')
+const runsDb = require('../src/model/events/eventRuns.db')
+const db = require('../src/utils/db')
+
+after(() => db.close())
+
+const HELD = ['pending', 'confirmed', 'reverting']
+
+let store
+const originals = { resourcesDb: { ...resourcesDb }, runsDb: { ...runsDb } }
+
+beforeEach(() => {
+  registries._reset()
+  store = { rows: new Map(), next: 1, cleanupStatus: 'not_required' }
+
+  resourcesDb.reserve = async ({ runId, stepId = null, owner, kind, ref, payload = null, leaseUntil = null, memberKey = null }) => {
+    const holder = [...store.rows.values()].find(
+      (r) => r.owner_module === owner && r.kind === kind && r.ref === ref && HELD.includes(r.status),
+    )
+    if (holder) return { ok: false, code: 'held', holder: { run_id: holder.run_id, status: holder.status } }
+    const id = store.next++
+    store.rows.set(id, {
+      id,
+      run_id: runId,
+      step_id: stepId,
+      owner_module: owner,
+      kind,
+      ref,
+      payload,
+      lease_until: leaseUntil,
+      status: 'pending',
+      revert_attempts: 0,
+      last_error: null,
+      member_key: memberKey,
+    })
+    return { ok: true, id }
+  }
+  resourcesDb.confirm = async (id) => {
+    const r = store.rows.get(id)
+    if (!r || r.status !== 'pending') return false
+    r.status = 'confirmed'
+    return true
+  }
+  resourcesDb.resolvePlaceholder = async (id) => {
+    const r = store.rows.get(id)
+    if (!r || r.kind !== resourcesDb.STEP_KIND) return false
+    r.status = 'reverted'
+    return true
+  }
+  resourcesDb.findByTarget = async (owner, kind, ref) =>
+    [...store.rows.values()].reverse().find((r) => r.owner_module === owner && r.kind === kind && r.ref === ref) || null
+
+  runsDb.setCleanupStatus = async (id, to, from = null) => {
+    if (from && !from.includes(store.cleanupStatus)) return false
+    store.cleanupStatus = to
+    return true
+  }
+})
+
+afterEach(() => {
+  Object.assign(resourcesDb, originals.resourcesDb)
+  Object.assign(runsDb, originals.runsDb)
+  registries._reset()
+})
+
+const RUN = { id: 7 }
+const STEP = { id: 42, phase: 'invasion', seq: 0, idempotency_key: 'a'.repeat(40) }
+
+const action = (over = {}) => ({
+  id: 'demo.spawn',
+  owner: 'demo',
+  label: 'Spawn',
+  risk: 'change',
+  reversible: 'ledger',
+  budgetMs: 1000,
+  ...over,
+})
+
+const rows = () => [...store.rows.values()]
+
+// ── Which actions ledger at all ────────────────────────────────────────────
+
+test('only the two reversible classes core has to come back for are ledgered', () => {
+  // `none` is gone once done, `self` undoes itself. Neither has anything core
+  // could revert, and giving one a placeholder would put a row in the ledger that
+  // teardown could never resolve — the exact reason `core.announce` is declared
+  // `none` rather than `ledger`.
+  assert.equal(ledger.ledgers(action({ reversible: 'ledger' })), true)
+  assert.equal(ledger.ledgers(action({ reversible: 'override' })), true)
+  assert.equal(ledger.ledgers(action({ reversible: 'none' })), false)
+  assert.equal(ledger.ledgers(action({ reversible: 'self' })), false)
+})
+
+test('only a ledger action gets a placeholder; an override reserves its own target', async () => {
+  // The asymmetry is the design. A spawn's ref is unknown until the module
+  // answers, so the placeholder stands in for it; a lease's target is the lease
+  // id the step already names, so `core.lease` writes the real row before it
+  // touches the world — which is rule 1 in a stronger form, and the only place
+  // the two-events-one-target refusal can happen before the world has changed.
+  assert.equal(typeof (await ledger.reserveStep(RUN, STEP, action())), 'number')
+  assert.equal(await ledger.reserveStep(RUN, STEP, action({ id: 'demo.b', reversible: 'override' })), null)
+  assert.equal(await ledger.reserveStep(RUN, STEP, action({ id: 'demo.c', reversible: 'none' })), null)
+  assert.equal(rows().length, 1)
+  assert.equal(rows()[0].kind, '@step')
+  assert.equal(rows()[0].ref, STEP.idempotency_key)
+})
+
+test('the first ledger row is what makes a run dirty, and only from not_required', async () => {
+  assert.equal(store.cleanupStatus, 'not_required')
+  await ledger.reserveStep(RUN, STEP, action())
+  assert.equal(store.cleanupStatus, 'pending')
+
+  // A run whose sweep has already finished must not be walked back to `pending`
+  // by a late row: only a human's cleanup re-opens it, and it does so
+  // deliberately and with an actor on the log line.
+  store.cleanupStatus = 'complete'
+  await ledger.recordAnswer({
+    run: RUN,
+    step: { ...STEP, id: 43, idempotency_key: 'b'.repeat(40) },
+    action: action(),
+    placeholderId: null,
+    resources: [{ kind: 'creature', ref: '0x1' }],
+  })
+  assert.equal(store.cleanupStatus, 'complete')
+})
+
+// ── Rule 1 ─────────────────────────────────────────────────────────────────
+
+test('a lost acknowledgement leaves the placeholder standing, which is the whole point', async () => {
+  const placeholderId = await ledger.reserveStep(RUN, STEP, action())
+  // The dispatch timed out: no answer, so `recordAnswer` is never reached. This
+  // is the case rule 1 exists for — record afterwards and the object the module
+  // may well have created is invisible to cleanup for ever.
+  assert.equal(rows()[0].status, 'pending')
+  assert.equal(rows()[0].payload.action, 'demo.spawn')
+  assert.ok(placeholderId)
+})
+
+test('a retry reuses its own placeholder rather than writing a second', async () => {
+  // An idempotency key is minted once per step and does not vary by attempt (§E),
+  // so the second attempt's insert collides with the first attempt's row. Finding
+  // it already there is the correct answer, and a second row would be a second
+  // thing for cleanup to revert.
+  const first = await ledger.reserveStep(RUN, STEP, action())
+  const second = await ledger.reserveStep(RUN, STEP, action())
+  assert.equal(first, second)
+  assert.equal(rows().length, 1)
+})
+
+test('the placeholder is resolved once the real rows exist', async () => {
+  const placeholderId = await ledger.reserveStep(RUN, STEP, action())
+  const out = await ledger.recordAnswer({
+    run: RUN,
+    step: STEP,
+    action: action(),
+    placeholderId,
+    resources: [
+      { kind: 'creature', ref: '0x40001234' },
+      { kind: 'creature', ref: '0x40001235' },
+    ],
+  })
+  assert.equal(out.recorded, 2)
+  assert.deepEqual(out.rejected, [])
+  assert.equal(store.rows.get(placeholderId).status, 'reverted')
+  assert.deepEqual(
+    rows().filter((r) => r.kind === 'creature').map((r) => [r.ref, r.status]),
+    [['0x40001234', 'confirmed'], ['0x40001235', 'confirmed']],
+  )
+})
+
+test('an action that ledgers and reports nothing still resolves its placeholder', async () => {
+  // "I made nothing" is a real answer. Holding the placeholder open for it would
+  // make cleanup call `revert()` on every terminal path, for ever, for a step that
+  // has nothing to give back.
+  const placeholderId = await ledger.reserveStep(RUN, STEP, action())
+  const out = await ledger.recordAnswer({ run: RUN, step: STEP, action: action(), placeholderId, resources: [] })
+  assert.equal(out.recorded, 0)
+  assert.equal(store.rows.get(placeholderId).status, 'reverted')
+})
+
+test('a module reporting the same resources twice produces one row', async () => {
+  // The database is what makes recording idempotent: `uq_evres_target` refuses
+  // the second insert and this file reads that as "already recorded". Without it
+  // a retry against a module that honestly re-reports its work would double every
+  // row cleanup then has to revert.
+  const args = { run: RUN, step: STEP, action: action(), placeholderId: null, resources: [{ kind: 'creature', ref: '0x1' }] }
+  await ledger.recordAnswer(args)
+  const again = await ledger.recordAnswer(args)
+  assert.equal(again.recorded, 0)
+  assert.deepEqual(again.rejected, [])
+  assert.equal(rows().filter((r) => r.kind === 'creature').length, 1)
+})
+
+test('a target another RUN holds is rejected by name rather than silently skipped', async () => {
+  await ledger.recordAnswer({
+    run: { id: 1 },
+    step: STEP,
+    action: action(),
+    placeholderId: null,
+    resources: [{ kind: 'creature', ref: '0x1' }],
+  })
+  const out = await ledger.recordAnswer({
+    run: { id: 2 },
+    step: { ...STEP, id: 99 },
+    action: action(),
+    placeholderId: null,
+    resources: [{ kind: 'creature', ref: '0x1' }],
+  })
+  assert.equal(out.recorded, 0)
+  assert.match(out.rejected.join('\n'), /already held by run 1/)
+})
+
+// ── What core will not write down ──────────────────────────────────────────
+
+test('a module may not claim core\'s reserved kind', () => {
+  // A module that could write a `@step` row could make its own step's placeholder
+  // look resolved — which is the one row whose survival is the safety property.
+  const bad = ledger.normalise({ kind: '@step', ref: 'x' }, 'demo.spawn')
+  assert.equal(bad.ok, false)
+  assert.match(bad.reason, /reserved kind/)
+})
+
+test('an override must name a lease core knows how to give back', () => {
+  // Core restores an `override` through the LEASE registry — that is the split §F
+  // draws — so a ref naming nothing registered is a resource core would be
+  // recording with no way to undo it. Refusing to record it is the fail-closed
+  // direction: rule 2 is a promise core must not make and then break.
+  assert.equal(ledger.normalise({ kind: 'override', ref: 'demo.rate' }, 'demo.x').ok, false)
+
+  const api = registries.stage('demo')
+  api.registerEventLeases([
+    {
+      id: 'demo.rate',
+      label: 'Rate',
+      type: 'float',
+      min: 0.5,
+      max: 5,
+      maxDurationMs: 3_600_000,
+      read: async () => ({ ok: true, value: 1 }),
+      apply: async () => ({ ok: true }),
+      restore: async () => ({ ok: true }),
+    },
+  ])
+  registries.apply(api.staged)
+  assert.equal(ledger.normalise({ kind: 'override', ref: 'demo.rate' }, 'demo.x').ok, true)
+})
+
+test('every bad shape is refused, and none of them is a retry', () => {
+  // A badly shaped resource is the module's mistake rather than the world's, and
+  // it will be just as badly shaped on the second attempt. They are dropped and
+  // reported; the STEP still counts as done, because it is — something happened
+  // in the world, and refusing to record it would be the one outcome worse than
+  // recording it imperfectly.
+  const bad = [
+    null,
+    'a string',
+    ['an array'],
+    { ref: 'x' }, // no kind
+    { kind: 'creature' }, // no ref
+    { kind: 'creature', ref: 'x'.repeat(200) },
+    { kind: 'k'.repeat(80), ref: 'x' },
+    { kind: 'creature', ref: 'x', memberKey: 'm'.repeat(200) },
+    { kind: 'creature', ref: 'x', until: 'not a date' },
+  ]
+  for (const entry of bad) {
+    assert.equal(ledger.normalise(entry, 'demo.spawn').ok, false, JSON.stringify(entry))
+  }
+})
+
+test('a bad resource never fails the step it came from', async () => {
+  const placeholderId = await ledger.reserveStep(RUN, STEP, action())
+  const out = await ledger.recordAnswer({
+    run: RUN,
+    step: STEP,
+    action: action(),
+    placeholderId,
+    resources: [{ kind: 'creature', ref: '0x1' }, { nonsense: true }],
+  })
+  assert.equal(out.recorded, 1)
+  assert.equal(out.rejected.length, 1)
+  // And the placeholder is still resolved: the good row exists, and leaving the
+  // placeholder open would ask the module to undo the step a second time.
+  assert.equal(store.rows.get(placeholderId).status, 'reverted')
+})
+
+test('a lease deadline and a member key ride through verbatim', async () => {
+  const until = new Date('2026-09-04T00:00:00Z')
+  await ledger.recordAnswer({
+    run: RUN,
+    step: STEP,
+    action: action(),
+    placeholderId: null,
+    resources: [{ kind: 'reward', ref: 'item-1', memberKey: 'Darrow', until, payload: { cliloc: 1234 } }],
+  })
+  const row = rows()[0]
+  assert.equal(row.member_key, 'Darrow')
+  assert.equal(row.lease_until.getTime(), until.getTime())
+  assert.deepEqual(row.payload, { cliloc: 1234 })
+  // Opaque: core stores what the module said and never interprets it, which is
+  // `ctx.teams.activity.push`'s exact treatment one registry along.
+  assert.equal(row.kind, 'reward')
+  assert.equal(row.owner_module, 'demo')
+})