feat(events): the resource ledger, leases and cleanup (Phase 8)
Some checks failed
PR Checks / client-build (pull_request) Successful in 3m15s
PR Checks / server-tests (pull_request) Failing after 8m21s
PR Checks / bot-tests (pull_request) Successful in 11m12s

Event System Phase 8 (EVENTS_PLAN.md). Docs half: RunicGateway/docs#NNN.

One table, one core action, one route, one body field, and two members added to
MODULE_API 1.10.0 in place. The safety property the whole world-write half
depends on: core now remembers what a run changed in the world, and gives it
back on every terminal path.

Four decisions settled by the org lead on 2026-09-03, all as recommended:

- A lease is acquired by a new CORE action, `core.lease`. Section F puts the
  duration bound and the two-events-one-target conflict check on core's side of
  the seam, and a lease verb per module would be both re-implemented once per
  module, advisory everywhere.
- Record-before-confirm is a PLACEHOLDER keyed by the step's idempotency key. A
  spawn's ref does not exist until the module answers, so what core writes
  before the dispatch is `kind: '@step'`, `ref` = that key. If the answer never
  comes it stands, and cleanup calls revert() with the key and no resources --
  which is why section F's revert takes the key at all.
- Cleanup is one sweep over the ledger, not synthetic step rows. The
  step-shaped version costs a second retry counter beside `revert_attempts`.
- `reconcile` is declared here and TRIGGERED BY THE MODULE, through
  `ctx.events.reconcile()`. Core has no concept of the game being up, so it
  cannot decide when to ask; it asks once at its own boot.

MODULE_API stays 1.10.0. A protocol owes a bump once it has landed on `main`;
while it is on `edge` it is amended in place, so the whole module contract
reaches an author as one version they read once.

Verify

- `npm test` -- 2025 tests, 1935 pass, 89 skipped, 1 fail. That one is the
  pre-existing engagementManifest CRLF failure, in a file this branch does not
  touch (`edge` before: 1950/1876/73/1). +75 tests.
- The unique key was proved against a REAL MariaDB, because nothing else can
  prove it: whether multiple NULLs collide in a unique index, whether a STORED
  generated column is recomputed on UPDATE, and whether the SET NULL foreign key
  survives beside it are properties of the server. eventRunnerSql.test.js gained
  16 tests; 65 pass against the container. The real schema.sql was applied to a
  fresh database and to an existing one.
- Client: 362 pass, and it builds. routes:manifest and swagger -- one route
  added, none moved.

The live walk found three defects, and two of them are the phase's real finding

Driven by a throwaway `rig` module in website/modules/, deleted before commit.

1. A lease was never given back at all. `core.lease` reserves its own ledger
   row, so it never went through the ledger's dirty-marking, so a run holding
   only a lease kept `cleanup_status = 'not_required'` and the cleanup leg --
   which selected on `pending` -- never looked at it.
2. EVENT_REVERT_MAX_ATTEMPTS meant one attempt, not three. The first failing
   sweep moved the run to `incomplete`, which took it out of the leg's own scan
   for ever. The test covering the bound asserted `<= 3` and was satisfied by 1:
   a bound has two halves, and a test that only asserts the ceiling passes
   against a floor.
3. The first fix for (2) made the console lie. Spending every row's
   `revert_attempts` was a tidy way to take a `cleanup: false` run out of a
   counter-bounded scan, and the run page then rendered "3 attempts" beside
   resources nothing had ever tried. Found by opening the page.

Both (1) and (2) are the same mistake: deriving "is there anything to do" from a
summary column instead of from the rows. Neither was visible to a unit test,
because a test that calls the sweep directly never asks what would have selected
the run.

The two properties that need the process to die were walked as the plan asks.
With the module's perform() hanging, the placeholder existed while the dispatch
was in flight and nothing was named; after taskkill and a restart the reclaim
re-dispatched the same idempotency key, the retry re-used its own placeholder,
and everything was given back. Then, with the module reporting one of two
resources as no longer in force, the boot-time reconcile marked the other
`orphaned` -- never `reverted`.

This branch does NOT bump MODULE_API_VERSION, so the integration kit stays as
Phase 7 left it: red until the Phase 16 cutover re-pins ci/core-ref.json.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2026-09-03 21:19:27 -05:00
parent 57d183e921
commit fdc118166c
29 changed files with 3928 additions and 72 deletions

View File

@@ -44,10 +44,15 @@ const register = (owner, entries) => {
registries.apply(api.staged)
}
test('core registers its three actions on every boot', () => {
test('core registers its four actions on every boot', () => {
registries.registerCore()
const ids = registries.allEventActions().map((a) => a.id)
assert.deepEqual(ids, ['core.announce', 'core.wait', 'core.cue'])
// `core.lease` joined the three in Phase 8, and it is the only one of the four
// that genuinely changes the world — which is why it is core's rather than each
// module's: §F puts the duration bound and the two-events-one-target conflict
// check on core's side of the seam, and a lease verb per module would be that
// bound re-implemented once per module and advisory everywhere.
assert.deepEqual(ids, ['core.announce', 'core.wait', 'core.cue', 'core.lease'])
assert.equal(ids.length, coreEventActions.ACTIONS.length)
})
@@ -56,6 +61,7 @@ test('the catalog carries no callable', () => {
for (const action of registries.allEventActions()) {
assert.equal(action.perform, undefined, `${action.id} leaked perform`)
assert.equal(action.revert, undefined, `${action.id} leaked revert`)
assert.equal(action.reconcile, undefined, `${action.id} leaked reconcile`)
assert.equal(action.cost, undefined, `${action.id} leaked cost`)
}
// …and the runner's own lookup still has it, which is the half that makes the
@@ -242,7 +248,7 @@ test('a whole batch is refused or taken, never half', () => {
test('_reset() hands the process back', () => {
registries.registerCore()
assert.equal(registries.allEventActions().length, 3)
assert.equal(registries.allEventActions().length, 4)
registries._reset()
assert.equal(registries.allEventActions().length, 0)
assert.equal(registries.isEventAction('core.wait'), false)
@@ -366,9 +372,80 @@ test('an option source needs a resolver, and core registers one of its own', ()
// Core through the same door (Phase 7): `core.announce`'s `leg` param names a
// source, and the announce legs are already a registry with labels in them.
registries.registerCore()
assert.deepEqual(registries.allEventOptionSources().map((s) => s.id), ['core.options.legs'])
assert.deepEqual(registries.allEventOptionSources().map((s) => s.id), [
'core.options.legs',
// Phase 8's, and it is the same argument one phase on: `core.lease`'s `lease`
// param would otherwise be a free-text box whose typo is caught at dispatch,
// mid-run — and the leases are already a registry with labels in them.
'core.options.leases',
])
const leg = registries.eventAction('core.announce').params.find((p) => p.name === 'leg')
assert.equal(leg.source, 'core.options.legs')
const which = registries.eventAction('core.lease').params.find((p) => p.name === 'lease')
assert.equal(which.source, 'core.options.leases')
})
// ── `reconcile`, the one member Phase 8 added to the action shape ──────────
//
// Optional where `revert` is required, and the asymmetry is the design: a module
// that cannot say what the game still has is not broken — core keeps believing
// its own ledger, which is the behaviour before this phase — whereas a module
// that created something and cannot undo it has made a promise core has no way
// to keep.
test('reconcile is optional, must be a function, and only on an action that ledgers', () => {
const ledgering = {
id: 'demo.spawn',
label: 'Spawn',
risk: 'change',
reversible: 'ledger',
perform: async () => ({ ok: true }),
revert: async () => ({ ok: true }),
}
// Absent is legal, and it lands as an explicit null rather than as a missing
// key — the same shape `revert` and `cost` take, so the catalog's strip list
// and the sweep's `typeof` check both have something to look at.
register('demo', [ledgering])
assert.equal(registries.eventAction('demo.spawn').reconcile, null)
registries._reset()
assert.throws(
() => register('demo', [{ ...ledgering, reconcile: 'yes please' }]),
/reconcile must be a function/,
)
// The mirror check `revert` already has. An action that ledgers nothing has no
// rows for core to ask about, so a `reconcile` on one is an author who believes
// something is being tracked and a sweep that will never call it.
assert.throws(
() =>
register('demo', [
{
id: 'demo.shout',
label: 'Shout',
risk: 'notify',
reversible: 'none',
perform: async () => ({ ok: true }),
reconcile: async () => ({ ok: true, inForce: [] }),
},
]),
/declares reconcile\(\) but is reversible: 'none' and ledgers nothing/,
)
})
test('an override action may reconcile, because a lease is ledgered too', () => {
register('demo', [
{
id: 'demo.borrow',
label: 'Borrow',
risk: 'change',
reversible: 'override',
perform: async () => ({ ok: true }),
reconcile: async () => ({ ok: true, inForce: [] }),
},
])
assert.equal(typeof registries.eventAction('demo.borrow').reconcile, 'function')
})
test('_reset() hands back the three new registries too', () => {

View File

@@ -0,0 +1,500 @@
// ── Giving back what a run took (EVENTS_PLAN.md Phase 8) ───────────────────
//
// The phase's shipped claim: **cleanup is generated from the ledger and runs on
// every terminal path.** An operator cannot be relied on to write the undo, and
// an aborted run never reaches the phase they wrote it in — so there is no
// cleanup phase in a spec, no `on_teardown` on an action, and one function that
// reads rows.
//
// The properties around it are §L's, and two of them are ones this codebase has
// already paid for once:
//
// • **a revert that never succeeds stays visible rather than cycling** — rule 2,
// and `MAX_REVERT_ATTEMPTS` is what stops the automatic retry. Only a human
// clears the counter, which is Engagement Phase 14's rule stated a third time
// • **reverting something that does not exist is a SUCCESS** — §L, and what a
// Rust wipe needs
// • **drift is not an error.** The module did exactly what it was asked and
// found somebody else's value in place. A restore that wrote anyway would
// silently revert an operator's manual fix
// • **a resource the module no longer has becomes `orphaned`, never `reverted`** —
// reverting it would be core recording that it put something back when what
// happened is that the thing vanished
// • **"I do not know" is never read as "it is gone".** Every unanswerable
// reconcile leaves the ledger alone
//
// The sweep is driven against a stubbed db layer, exactly as `eventRunner.test.js`
// drives the runner: what a stub cannot prove is the SQL, and the unique key that
// makes two events unable to lease one target runs against a real MariaDB in
// `eventRunnerSql.test.js`.
process.env.DB_HOST = '127.0.0.1'
process.env.DB_PORT = '59999'
const { test, beforeEach, afterEach, after } = require('node:test')
const assert = require('node:assert/strict')
const registries = require('../src/modules/registries')
const cleanup = require('../src/events/cleanup')
const resourcesDb = require('../src/model/events/eventRunResources.db')
const runsDb = require('../src/model/events/eventRuns.db')
const stepsDb = require('../src/model/events/eventRunSteps.db')
const logDb = require('../src/model/events/eventRunLog.db')
const db = require('../src/utils/db')
after(() => db.close())
const UNRESOLVED = ['pending', 'confirmed', 'reverting', 'orphaned', 'drifted']
let store
const originals = {
resourcesDb: { ...resourcesDb },
runsDb: { ...runsDb },
stepsDb: { ...stepsDb },
logDb: { ...logDb },
}
beforeEach(() => {
registries._reset()
store = { rows: new Map(), steps: new Map(), log: [], next: 1, cleanupStatus: 'pending' }
resourcesDb.unresolvedForRun = async (runId, { maxAttempts = null } = {}) =>
[...store.rows.values()]
.filter((r) => r.run_id === runId && UNRESOLVED.includes(r.status))
.filter((r) => maxAttempts === null || r.revert_attempts < maxAttempts)
.map((r) => ({ ...r }))
resourcesDb.unresolvedCount = async (runId) =>
[...store.rows.values()].filter((r) => r.run_id === runId && UNRESOLVED.includes(r.status)).length
resourcesDb.claimRevert = async (id) => {
const r = store.rows.get(id)
if (!r || !['pending', 'confirmed', 'orphaned', 'drifted'].includes(r.status)) return false
r.status = 'reverting'
return true
}
resourcesDb.markReverted = async (id) => {
const r = store.rows.get(id)
if (r) Object.assign(r, { status: 'reverted', last_error: null })
}
resourcesDb.failRevert = async (id, error, restoreTo = 'confirmed') => {
const r = store.rows.get(id)
if (r) Object.assign(r, { status: restoreTo, revert_attempts: r.revert_attempts + 1, last_error: String(error) })
}
resourcesDb.resetAttempts = async (runId) => {
let n = 0
for (const r of store.rows.values()) {
if (r.run_id === runId && UNRESOLVED.includes(r.status)) {
r.revert_attempts = 0
n += 1
}
}
return n
}
resourcesDb.markOrphaned = async (id, detail = null) => {
const r = store.rows.get(id)
if (r && ['pending', 'confirmed', 'reverting'].includes(r.status)) {
Object.assign(r, { status: 'orphaned', last_error: detail })
}
}
resourcesDb.liveForModule = async (owner) =>
[...store.rows.values()].filter((r) => r.owner_module === owner && ['pending', 'confirmed'].includes(r.status)).map((r) => ({ ...r }))
resourcesDb.modulesWithLiveRows = async () => [
...new Set([...store.rows.values()].filter((r) => ['pending', 'confirmed'].includes(r.status)).map((r) => r.owner_module)),
]
resourcesDb.runsNeedingCleanup = async () => store.candidates || []
runsDb.setCleanupStatus = async (id, to, from = null) => {
if (from && !from.includes(store.cleanupStatus)) return false
store.cleanupStatus = to
return true
}
stepsDb.getById = async (id) => store.steps.get(id) || null
logDb.write = async (entry) => {
store.log.push(entry)
}
})
afterEach(() => {
Object.assign(resourcesDb, originals.resourcesDb)
Object.assign(runsDb, originals.runsDb)
Object.assign(stepsDb, originals.stepsDb)
Object.assign(logDb, originals.logDb)
registries._reset()
})
const RUN = { id: 7, status: 'completed', cleanup_status: 'pending' }
function addStep(id, actionId, key = 'k'.repeat(40)) {
store.steps.set(id, { id, run_id: RUN.id, action_id: actionId, idempotency_key: key, phase: 'p', seq: 0 })
}
function addResource(over = {}) {
const id = store.next++
const row = {
id,
run_id: RUN.id,
step_id: 1,
owner_module: 'demo',
kind: 'creature',
ref: `0x${id}`,
payload: null,
lease_until: null,
status: 'confirmed',
revert_attempts: 0,
last_error: null,
member_key: null,
...over,
}
store.rows.set(id, row)
return row
}
function registerAction(over = {}) {
const api = registries.stage('demo')
api.registerEventActions([
{
id: 'demo.spawn',
label: 'Spawn',
risk: 'change',
reversible: 'ledger',
budgetMs: 500,
perform: async () => ({ ok: true }),
revert: async () => ({ ok: true }),
...over,
},
])
registries.apply(api.staged)
}
function registerLease(over = {}) {
const api = registries.stage('demo')
api.registerEventLeases([
{
id: 'demo.rate',
label: 'Gather rate',
type: 'float',
min: 0.5,
max: 5,
maxDurationMs: 3_600_000,
read: async () => ({ ok: true, value: 1 }),
apply: async () => ({ ok: true }),
restore: async () => ({ ok: true }),
...over,
},
])
registries.apply(api.staged)
}
const kinds = () => store.log.map((e) => e.kind)
// ── The shipped claim ──────────────────────────────────────────────────────
test('a run\'s ledger is given back in one call per step, and the run goes complete', async () => {
// `revert` takes a LIST because that is what makes twelve creatures one round
// trip rather than twelve. The grouping key is the STEP, because the resource
// row records the module and the opaque names while the step records the verb.
let seen = null
registerAction({ revert: async (arg) => { seen = arg; return { ok: true } } })
addStep(1, 'demo.spawn', 'key-one')
addResource({ ref: '0xA' })
addResource({ ref: '0xB' })
const summary = await cleanup.cleanupRun(RUN)
assert.deepEqual(summary, { attempted: 2, reverted: 2, drifted: 0, failed: 0, remaining: 0 })
assert.equal(seen.runId, RUN.id)
assert.equal(seen.idempotencyKey, 'key-one')
assert.deepEqual(seen.resources.map((r) => r.ref), ['0xA', '0xB'])
assert.equal(store.cleanupStatus, 'complete')
})
test('two steps are two calls, and one failing does not take the other down', async () => {
const calls = []
registerAction({
revert: async ({ idempotencyKey, resources }) => {
calls.push(idempotencyKey)
return idempotencyKey === 'bad' ? { ok: false, error: 'the shard did not answer' } : { ok: true, resources }
},
})
addStep(1, 'demo.spawn', 'good')
addStep(2, 'demo.spawn', 'bad')
addResource({ step_id: 1 })
addResource({ step_id: 2 })
const summary = await cleanup.cleanupRun(RUN)
assert.equal(calls.length, 2)
assert.equal(summary.reverted, 1)
assert.equal(summary.failed, 1)
assert.equal(summary.remaining, 1)
// **Still `pending`, because the failed row has retries left.** `incomplete`
// means "finished with, and not finished" — it is what takes a run out of the
// sweep's own scan, so writing it after the FIRST failure made
// `MAX_REVERT_ATTEMPTS` quietly mean one attempt. Found by watching
// `revert_attempts` sit at 1 through half a minute of live ticks.
assert.equal(store.cleanupStatus, 'pending')
})
test('reverting something that does not exist is a success', async () => {
// §L, and the Rust wipe: "gone, and that is fine". The module never has to
// distinguish "I deleted it" from "it was not there" — which is also what makes
// a placeholder for an object that may never have existed safe to write.
registerAction({ revert: async () => ({ ok: true, detail: 'resource no longer exists' }) })
addStep(1, 'demo.spawn')
addResource()
const summary = await cleanup.cleanupRun(RUN)
assert.equal(summary.reverted, 1)
assert.equal(store.cleanupStatus, 'complete')
})
test('a module may name the ones that did not come back', async () => {
// Partial cleanup is the ordinary case — eleven of twelve creatures deleted —
// and it is why the ledger is a row per object rather than a row per step.
registerAction({ revert: async () => ({ ok: true, failed: ['0x2'] }) })
addStep(1, 'demo.spawn')
addResource({ ref: '0x1' })
addResource({ ref: '0x2' })
const summary = await cleanup.cleanupRun(RUN)
assert.equal(summary.reverted, 1)
assert.equal(summary.failed, 1)
assert.match([...store.rows.values()].find((r) => r.ref === '0x2').last_error, /could not give "0x2" back/)
})
// ── Rule 2: loud and sticky ────────────────────────────────────────────────
test('a revert that never works stops retrying and stays visible', async () => {
let calls = 0
registerAction({ revert: async () => { calls += 1; return { ok: false, error: 'nope' } } })
addStep(1, 'demo.spawn')
addResource()
for (let i = 0; i < 6; i++) await cleanup.cleanupRun(RUN)
// Bounded at MAX_REVERT_ATTEMPTS, exactly like a step's attempts. A fourth ask
// of a shard that has answered the same way three times is not new information,
// and an unbounded counter is a row nothing can ever sweep.
assert.equal(calls, cleanup.MAX_REVERT_ATTEMPTS)
const row = [...store.rows.values()][0]
assert.equal(row.revert_attempts, cleanup.MAX_REVERT_ATTEMPTS)
assert.equal(row.status, 'confirmed')
assert.equal(row.last_error, 'nope')
// And ONLY now, with nothing left to try, does the run stop being the sweep's
// business. §L: it does not stay `running` — an event whose world changes are
// still up is a real state, and pretending the event is in progress hides it.
assert.equal(store.cleanupStatus, 'incomplete')
})
test('the run goes back to pending each time it still has an attempt left', async () => {
// The other half of the same rule, watched one pass at a time rather than at
// the end. Each of the first two sweeps leaves the run in the scan; the third
// takes it out. A test that only looked at the end state would pass against the
// defect this replaced.
registerAction({ revert: async () => ({ ok: false, error: 'nope' }) })
addStep(1, 'demo.spawn')
addResource()
const seen = []
for (let i = 0; i < 3; i++) {
await cleanup.cleanupRun(RUN)
seen.push(store.cleanupStatus)
}
assert.deepEqual(seen, ['pending', 'pending', 'incomplete'])
})
test('only a human clears the attempt counter', async () => {
// Engagement Phase 14's defect, stated a third time: a SWEEP that returned every
// stale row to its start state made the attempt ceiling unreachable, so the row
// cycled for ever and was never eligible for any retention sweep. The automatic
// leg must never do this; the cleanup route may, because a person asked.
let calls = 0
registerAction({ revert: async () => { calls += 1; return { ok: false, error: 'nope' } } })
addStep(1, 'demo.spawn')
addResource()
for (let i = 0; i < 5; i++) await cleanup.cleanupRun(RUN)
assert.equal(calls, cleanup.MAX_REVERT_ATTEMPTS)
await cleanup.cleanupRun(RUN, { resetAttempts: true, actor: 9 })
assert.equal(calls, cleanup.MAX_REVERT_ATTEMPTS + 1)
assert.ok(store.log.some((e) => e.kind === 'cleanup.retry' && e.detail.by === 9))
})
test('a module that throws from revert is a transient failure, not a crashed sweep', async () => {
registerAction({ revert: async () => { throw new Error('socket hung up') } })
addStep(1, 'demo.spawn')
addResource()
const summary = await cleanup.cleanupRun(RUN)
assert.equal(summary.failed, 1)
assert.match([...store.rows.values()][0].last_error, /socket hung up/)
})
test('no shape a revert failure can take reads as success', () => {
// `dispatch.classify`'s rule, applied to the other direction of the contract.
// The expensive mistake here is the mirror of the one there: recording that a
// world change was UNDONE when it was not.
for (const raw of [null, undefined, 'ok', [], {}, { ok: 'yes' }, { ok: 1 }]) {
assert.notEqual(cleanup.classifyRevert(raw, 'demo.spawn').outcome, 'done', JSON.stringify(raw))
}
assert.equal(cleanup.classifyRevert({ __timedOut: true, error: 'slow' }, 'x').outcome, 'retry')
assert.equal(cleanup.classifyRevert({ ok: false, retry: false, error: 'never' }, 'x').outcome, 'terminal')
assert.equal(cleanup.classifyRevert({ ok: true }, 'x').outcome, 'done')
})
test('an action whose module is gone leaves its rows unresolved with the reason', async () => {
// Not a retry — nothing will change until an operator reinstalls it — and not
// an orphan either, because core has no idea whether the thing is still there.
addStep(1, 'demo.spawn')
addResource()
const summary = await cleanup.cleanupRun(RUN)
assert.equal(summary.failed, 1)
assert.match([...store.rows.values()][0].last_error, /no module registers "demo.spawn"/)
// Retried like any other failure rather than given up on at once, and that is
// the right uniformity here: "nothing registers this" stops being true the
// moment an operator reinstalls the module, and three registry lookups cost
// nothing. So it is `pending` until the attempts are spent.
assert.equal(store.cleanupStatus, 'pending')
})
// ── Leases ─────────────────────────────────────────────────────────────────
test('a lease is restored through the LEASE registry, not through any action', async () => {
// The split §F draws: core owns the duration and the conflict check, the module
// owns reading and writing. It is why `core.lease` needs no `revert()` of its
// own, and why an `override` row routes here rather than to its step's action.
let seen = null
registerLease({ restore: async (baseline, opts) => { seen = { baseline, opts }; return { ok: true } } })
addResource({ kind: 'override', ref: 'demo.rate', step_id: null, payload: { baseline: 1, applied: 3 } })
const summary = await cleanup.cleanupRun(RUN)
assert.equal(summary.reverted, 1)
assert.equal(seen.baseline, 1)
// The drift check's input. `restore` MUST verify current === expected before
// writing, and a lease whose restore wrote blindly would silently revert an
// operator's manual fix.
assert.equal(seen.opts.expected, 3)
})
test('drift is not an error: the world is left alone and the row says so', async () => {
registerLease({ restore: async () => ({ ok: false, drifted: true, current: 4.5 }) })
addResource({ kind: 'override', ref: 'demo.rate', step_id: null, payload: { baseline: 1, applied: 3 } })
const summary = await cleanup.cleanupRun(RUN)
assert.equal(summary.drifted, 1)
assert.equal(summary.failed, 0)
const row = [...store.rows.values()][0]
assert.equal(row.status, 'drifted')
assert.match(row.last_error, /now 4\.5 rather than what this run applied/)
// Still surfaced. §L: "surfaced beside the unreverted ones" — the run does not
// get to call itself clean because somebody else took the value. It is `pending`
// rather than `incomplete` for one more reason worth keeping: drift is retried
// like any other failure, because a GM who puts the value back between two ticks
// should have the lease close cleanly.
assert.equal(store.cleanupStatus, 'pending')
})
test('a lease whose module is uninstalled is unresolved, never assumed restored', async () => {
addResource({ kind: 'override', ref: 'demo.rate', step_id: null, payload: { baseline: 1, applied: 3 } })
const summary = await cleanup.cleanupRun(RUN)
assert.equal(summary.failed, 1)
assert.match([...store.rows.values()][0].last_error, /no module registers the lease "demo.rate"/)
})
// ── The sweep ──────────────────────────────────────────────────────────────
test('the sweep only touches TERMINAL runs', async () => {
// A run still in flight has a ledger that is still growing, and reverting a
// resource the next step is about to use would be core undoing an event while
// it is happening.
registerAction()
addStep(1, 'demo.spawn')
addResource()
store.candidates = [{ id: RUN.id, status: 'running', cleanup_status: 'pending' }]
assert.equal(await cleanup.sweep(), 0)
assert.equal([...store.rows.values()][0].status, 'confirmed')
store.candidates = [{ id: RUN.id, status: 'cancelled', cleanup_status: 'pending' }]
assert.equal(await cleanup.sweep(), 1)
assert.equal([...store.rows.values()][0].status, 'reverted')
})
// ── Reconcile ──────────────────────────────────────────────────────────────
test('a resource the module no longer has becomes orphaned, never reverted', async () => {
// §L, and the distinction matters to the operator reading the console
// afterwards: `reverted` says core put something back, `orphaned` says the
// thing vanished while nobody was looking. Recording the second as the first
// would be core claiming credit for a shard restart.
registerAction({ reconcile: async () => ({ ok: true, inForce: ['0x1'] }) })
addStep(1, 'demo.spawn')
addResource({ ref: '0x1' })
addResource({ ref: '0x2' })
const summary = await cleanup.reconcileModule('demo')
assert.deepEqual(summary, { asked: 2, inForce: 1, orphaned: 1, unanswered: 0 })
assert.equal([...store.rows.values()].find((r) => r.ref === '0x2').status, 'orphaned')
assert.equal([...store.rows.values()].find((r) => r.ref === '0x1').status, 'confirmed')
assert.ok(store.log.some((e) => e.kind === 'resource.orphaned'))
})
test('"I do not know" is never read as "it is gone"', async () => {
// Every unanswerable shape leaves the ledger exactly as it was. A reconcile
// that read silence as absence would orphan a whole shard's worth of live
// spawns the first time a sidecar was slow.
for (const answer of [null, undefined, { ok: false }, { ok: true }, { ok: true, inForce: 'all' }, 'yes']) {
store.rows.clear()
store.log.length = 0
registries._reset()
registerAction({ reconcile: async () => answer })
addStep(1, 'demo.spawn')
addResource({ ref: '0x1' })
const summary = await cleanup.reconcileModule('demo')
assert.equal(summary.orphaned, 0, JSON.stringify(answer))
assert.equal(summary.unanswered, 1, JSON.stringify(answer))
assert.equal([...store.rows.values()][0].status, 'confirmed')
}
})
test('a module with no reconcile is not broken; core keeps believing its ledger', async () => {
// Optional where `revert` is required. A module that cannot answer leaves core
// exactly where it was before this phase, which is a capability its deployment
// does without rather than a boot it fails.
registerAction()
addStep(1, 'demo.spawn')
addResource()
const summary = await cleanup.reconcileModule('demo')
assert.deepEqual(summary, { asked: 0, inForce: 0, orphaned: 0, unanswered: 1 })
assert.equal([...store.rows.values()][0].status, 'confirmed')
})
test('a reconcile that throws orphans nothing', async () => {
registerAction({ reconcile: async () => { throw new Error('sidecar gone') } })
addStep(1, 'demo.spawn')
addResource()
const summary = await cleanup.reconcileModule('demo')
assert.equal(summary.unanswered, 1)
assert.equal([...store.rows.values()][0].status, 'confirmed')
})
test('placeholders are not asked about, because there is nothing to ask yet', async () => {
// A `@step` row names no object — it says "a dispatch was in flight and may
// have made something". Asking a module whether it is in force is a question
// with no answer, and reading a shrug as absence would resolve the one row whose
// survival is the safety property.
registerAction({ reconcile: async () => ({ ok: true, inForce: [] }) })
addStep(1, 'demo.spawn')
addResource({ kind: resourcesDb.STEP_KIND, ref: 'a'.repeat(40), status: 'pending' })
const summary = await cleanup.reconcileModule('demo')
assert.deepEqual(summary, { asked: 0, inForce: 0, orphaned: 0, unanswered: 0 })
assert.equal([...store.rows.values()][0].status, 'pending')
})
test('reconcileAll asks every module that owns a live row', async () => {
registerAction({ reconcile: async () => ({ ok: true, inForce: [] }) })
addStep(1, 'demo.spawn')
addStep(2, 'other.thing')
addResource({ owner_module: 'demo' })
addResource({ owner_module: 'other', step_id: 2 })
const out = await cleanup.reconcileAll()
assert.deepEqual(Object.keys(out).sort(), ['demo', 'other'])
assert.equal(out.demo.orphaned, 1)
// The other module registers nothing, so its row is left alone rather than
// orphaned by a module that is not there to be asked.
assert.equal(out.other.unanswered, 1)
})

View File

@@ -0,0 +1,337 @@
// ── The resource ledger's write half (EVENTS_PLAN.md Phase 8) ──────────────
//
// §D's two rules, and rule 1 is the one this file exists for: **a resource is
// recorded BEFORE it is confirmed.** The obstacle it works around is that a
// spawn's serial does not exist until the module answers, so what goes in before
// the dispatch is a placeholder keyed by the step's idempotency key — and the
// property worth a test is that the placeholder SURVIVES an answer that never
// comes, because that is the case where recording afterwards would have lost the
// object for ever.
//
// The other rules here are about what core will and will not write down on a
// module's say-so. Every one of them is fail-closed in a specific direction:
//
// • the reserved `@step` kind is core's and a module may not claim it
// • an `override` must name a lease core knows how to give back, or core would
// be recording something it has no way to restore
// • a duplicate is "already recorded", not an error — a retry re-sends the same
// idempotency key and a module may honestly report the same resources twice
// • a badly shaped resource is dropped and LOGGED, never a failed step: the
// step changed the world, and turning bookkeeping into a retry would re-run
// a world write that already happened
//
// The db layer is stubbed with a store that enforces `uq_evres_target`, because
// that refusal is behaviour the callers branch on rather than an implementation
// detail. The SQL itself is `eventRunnerSql.test.js`'s, against a real MariaDB.
process.env.DB_HOST = '127.0.0.1'
process.env.DB_PORT = '59999'
const { test, beforeEach, afterEach, after } = require('node:test')
const assert = require('node:assert/strict')
const registries = require('../src/modules/registries')
const ledger = require('../src/events/ledger')
const resourcesDb = require('../src/model/events/eventRunResources.db')
const runsDb = require('../src/model/events/eventRuns.db')
const db = require('../src/utils/db')
after(() => db.close())
const HELD = ['pending', 'confirmed', 'reverting']
let store
const originals = { resourcesDb: { ...resourcesDb }, runsDb: { ...runsDb } }
beforeEach(() => {
registries._reset()
store = { rows: new Map(), next: 1, cleanupStatus: 'not_required' }
resourcesDb.reserve = async ({ runId, stepId = null, owner, kind, ref, payload = null, leaseUntil = null, memberKey = null }) => {
const holder = [...store.rows.values()].find(
(r) => r.owner_module === owner && r.kind === kind && r.ref === ref && HELD.includes(r.status),
)
if (holder) return { ok: false, code: 'held', holder: { run_id: holder.run_id, status: holder.status } }
const id = store.next++
store.rows.set(id, {
id,
run_id: runId,
step_id: stepId,
owner_module: owner,
kind,
ref,
payload,
lease_until: leaseUntil,
status: 'pending',
revert_attempts: 0,
last_error: null,
member_key: memberKey,
})
return { ok: true, id }
}
resourcesDb.confirm = async (id) => {
const r = store.rows.get(id)
if (!r || r.status !== 'pending') return false
r.status = 'confirmed'
return true
}
resourcesDb.resolvePlaceholder = async (id) => {
const r = store.rows.get(id)
if (!r || r.kind !== resourcesDb.STEP_KIND) return false
r.status = 'reverted'
return true
}
resourcesDb.findByTarget = async (owner, kind, ref) =>
[...store.rows.values()].reverse().find((r) => r.owner_module === owner && r.kind === kind && r.ref === ref) || null
runsDb.setCleanupStatus = async (id, to, from = null) => {
if (from && !from.includes(store.cleanupStatus)) return false
store.cleanupStatus = to
return true
}
})
afterEach(() => {
Object.assign(resourcesDb, originals.resourcesDb)
Object.assign(runsDb, originals.runsDb)
registries._reset()
})
const RUN = { id: 7 }
const STEP = { id: 42, phase: 'invasion', seq: 0, idempotency_key: 'a'.repeat(40) }
const action = (over = {}) => ({
id: 'demo.spawn',
owner: 'demo',
label: 'Spawn',
risk: 'change',
reversible: 'ledger',
budgetMs: 1000,
...over,
})
const rows = () => [...store.rows.values()]
// ── Which actions ledger at all ────────────────────────────────────────────
test('only the two reversible classes core has to come back for are ledgered', () => {
// `none` is gone once done, `self` undoes itself. Neither has anything core
// could revert, and giving one a placeholder would put a row in the ledger that
// teardown could never resolve — the exact reason `core.announce` is declared
// `none` rather than `ledger`.
assert.equal(ledger.ledgers(action({ reversible: 'ledger' })), true)
assert.equal(ledger.ledgers(action({ reversible: 'override' })), true)
assert.equal(ledger.ledgers(action({ reversible: 'none' })), false)
assert.equal(ledger.ledgers(action({ reversible: 'self' })), false)
})
test('only a ledger action gets a placeholder; an override reserves its own target', async () => {
// The asymmetry is the design. A spawn's ref is unknown until the module
// answers, so the placeholder stands in for it; a lease's target is the lease
// id the step already names, so `core.lease` writes the real row before it
// touches the world — which is rule 1 in a stronger form, and the only place
// the two-events-one-target refusal can happen before the world has changed.
assert.equal(typeof (await ledger.reserveStep(RUN, STEP, action())), 'number')
assert.equal(await ledger.reserveStep(RUN, STEP, action({ id: 'demo.b', reversible: 'override' })), null)
assert.equal(await ledger.reserveStep(RUN, STEP, action({ id: 'demo.c', reversible: 'none' })), null)
assert.equal(rows().length, 1)
assert.equal(rows()[0].kind, '@step')
assert.equal(rows()[0].ref, STEP.idempotency_key)
})
test('the first ledger row is what makes a run dirty, and only from not_required', async () => {
assert.equal(store.cleanupStatus, 'not_required')
await ledger.reserveStep(RUN, STEP, action())
assert.equal(store.cleanupStatus, 'pending')
// A run whose sweep has already finished must not be walked back to `pending`
// by a late row: only a human's cleanup re-opens it, and it does so
// deliberately and with an actor on the log line.
store.cleanupStatus = 'complete'
await ledger.recordAnswer({
run: RUN,
step: { ...STEP, id: 43, idempotency_key: 'b'.repeat(40) },
action: action(),
placeholderId: null,
resources: [{ kind: 'creature', ref: '0x1' }],
})
assert.equal(store.cleanupStatus, 'complete')
})
// ── Rule 1 ─────────────────────────────────────────────────────────────────
test('a lost acknowledgement leaves the placeholder standing, which is the whole point', async () => {
const placeholderId = await ledger.reserveStep(RUN, STEP, action())
// The dispatch timed out: no answer, so `recordAnswer` is never reached. This
// is the case rule 1 exists for — record afterwards and the object the module
// may well have created is invisible to cleanup for ever.
assert.equal(rows()[0].status, 'pending')
assert.equal(rows()[0].payload.action, 'demo.spawn')
assert.ok(placeholderId)
})
test('a retry reuses its own placeholder rather than writing a second', async () => {
// An idempotency key is minted once per step and does not vary by attempt (§E),
// so the second attempt's insert collides with the first attempt's row. Finding
// it already there is the correct answer, and a second row would be a second
// thing for cleanup to revert.
const first = await ledger.reserveStep(RUN, STEP, action())
const second = await ledger.reserveStep(RUN, STEP, action())
assert.equal(first, second)
assert.equal(rows().length, 1)
})
test('the placeholder is resolved once the real rows exist', async () => {
const placeholderId = await ledger.reserveStep(RUN, STEP, action())
const out = await ledger.recordAnswer({
run: RUN,
step: STEP,
action: action(),
placeholderId,
resources: [
{ kind: 'creature', ref: '0x40001234' },
{ kind: 'creature', ref: '0x40001235' },
],
})
assert.equal(out.recorded, 2)
assert.deepEqual(out.rejected, [])
assert.equal(store.rows.get(placeholderId).status, 'reverted')
assert.deepEqual(
rows().filter((r) => r.kind === 'creature').map((r) => [r.ref, r.status]),
[['0x40001234', 'confirmed'], ['0x40001235', 'confirmed']],
)
})
test('an action that ledgers and reports nothing still resolves its placeholder', async () => {
// "I made nothing" is a real answer. Holding the placeholder open for it would
// make cleanup call `revert()` on every terminal path, for ever, for a step that
// has nothing to give back.
const placeholderId = await ledger.reserveStep(RUN, STEP, action())
const out = await ledger.recordAnswer({ run: RUN, step: STEP, action: action(), placeholderId, resources: [] })
assert.equal(out.recorded, 0)
assert.equal(store.rows.get(placeholderId).status, 'reverted')
})
test('a module reporting the same resources twice produces one row', async () => {
// The database is what makes recording idempotent: `uq_evres_target` refuses
// the second insert and this file reads that as "already recorded". Without it
// a retry against a module that honestly re-reports its work would double every
// row cleanup then has to revert.
const args = { run: RUN, step: STEP, action: action(), placeholderId: null, resources: [{ kind: 'creature', ref: '0x1' }] }
await ledger.recordAnswer(args)
const again = await ledger.recordAnswer(args)
assert.equal(again.recorded, 0)
assert.deepEqual(again.rejected, [])
assert.equal(rows().filter((r) => r.kind === 'creature').length, 1)
})
test('a target another RUN holds is rejected by name rather than silently skipped', async () => {
await ledger.recordAnswer({
run: { id: 1 },
step: STEP,
action: action(),
placeholderId: null,
resources: [{ kind: 'creature', ref: '0x1' }],
})
const out = await ledger.recordAnswer({
run: { id: 2 },
step: { ...STEP, id: 99 },
action: action(),
placeholderId: null,
resources: [{ kind: 'creature', ref: '0x1' }],
})
assert.equal(out.recorded, 0)
assert.match(out.rejected.join('\n'), /already held by run 1/)
})
// ── What core will not write down ──────────────────────────────────────────
test('a module may not claim core\'s reserved kind', () => {
// A module that could write a `@step` row could make its own step's placeholder
// look resolved — which is the one row whose survival is the safety property.
const bad = ledger.normalise({ kind: '@step', ref: 'x' }, 'demo.spawn')
assert.equal(bad.ok, false)
assert.match(bad.reason, /reserved kind/)
})
test('an override must name a lease core knows how to give back', () => {
// Core restores an `override` through the LEASE registry — that is the split §F
// draws — so a ref naming nothing registered is a resource core would be
// recording with no way to undo it. Refusing to record it is the fail-closed
// direction: rule 2 is a promise core must not make and then break.
assert.equal(ledger.normalise({ kind: 'override', ref: 'demo.rate' }, 'demo.x').ok, false)
const api = registries.stage('demo')
api.registerEventLeases([
{
id: 'demo.rate',
label: 'Rate',
type: 'float',
min: 0.5,
max: 5,
maxDurationMs: 3_600_000,
read: async () => ({ ok: true, value: 1 }),
apply: async () => ({ ok: true }),
restore: async () => ({ ok: true }),
},
])
registries.apply(api.staged)
assert.equal(ledger.normalise({ kind: 'override', ref: 'demo.rate' }, 'demo.x').ok, true)
})
test('every bad shape is refused, and none of them is a retry', () => {
// A badly shaped resource is the module's mistake rather than the world's, and
// it will be just as badly shaped on the second attempt. They are dropped and
// reported; the STEP still counts as done, because it is — something happened
// in the world, and refusing to record it would be the one outcome worse than
// recording it imperfectly.
const bad = [
null,
'a string',
['an array'],
{ ref: 'x' }, // no kind
{ kind: 'creature' }, // no ref
{ kind: 'creature', ref: 'x'.repeat(200) },
{ kind: 'k'.repeat(80), ref: 'x' },
{ kind: 'creature', ref: 'x', memberKey: 'm'.repeat(200) },
{ kind: 'creature', ref: 'x', until: 'not a date' },
]
for (const entry of bad) {
assert.equal(ledger.normalise(entry, 'demo.spawn').ok, false, JSON.stringify(entry))
}
})
test('a bad resource never fails the step it came from', async () => {
const placeholderId = await ledger.reserveStep(RUN, STEP, action())
const out = await ledger.recordAnswer({
run: RUN,
step: STEP,
action: action(),
placeholderId,
resources: [{ kind: 'creature', ref: '0x1' }, { nonsense: true }],
})
assert.equal(out.recorded, 1)
assert.equal(out.rejected.length, 1)
// And the placeholder is still resolved: the good row exists, and leaving the
// placeholder open would ask the module to undo the step a second time.
assert.equal(store.rows.get(placeholderId).status, 'reverted')
})
test('a lease deadline and a member key ride through verbatim', async () => {
const until = new Date('2026-09-04T00:00:00Z')
await ledger.recordAnswer({
run: RUN,
step: STEP,
action: action(),
placeholderId: null,
resources: [{ kind: 'reward', ref: 'item-1', memberKey: 'Darrow', until, payload: { cliloc: 1234 } }],
})
const row = rows()[0]
assert.equal(row.member_key, 'Darrow')
assert.equal(row.lease_until.getTime(), until.getTime())
assert.deepEqual(row.payload, { cliloc: 1234 })
// Opaque: core stores what the module said and never interprets it, which is
// `ctx.teams.activity.push`'s exact treatment one registry along.
assert.equal(row.kind, 'reward')
assert.equal(row.owner_module, 'demo')
})

View File

@@ -513,3 +513,242 @@ test('an action whose module is gone goes dormant, and a step naming it fails te
assert.equal(result.dormant, true)
assert.match(result.error, /no module registers "demo\.summon"/)
})
// ── Phase 8: the ledger's two callables, from a module ─────────────────────
//
// `revert` was already required at registration for `reversible: 'ledger'` —
// Phase 1 put that check in. What Phase 8 added is a caller for it, and
// `reconcile` beside it. Both are proved here through the REAL loader for the
// same reason the four registrations are: a `revert` a test called directly is a
// `revert` core might still have no way to reach.
test('a module\'s revert is reached by the cleanup sweep, resources and key in hand', async () => {
const record = loadModule('demo', `
let seen = null
module.exports = (ctx, api) => {
api.registerEventActions([{
id: 'demo.spawn',
label: 'Spawn',
risk: 'change',
reversible: 'ledger',
params: [],
async perform() { return { ok: true, resources: [{ kind: 'creature', ref: '0xA' }] } },
async revert(arg) { seen = arg; return { ok: true } },
async reconcile() { return { ok: true, inForce: [] } },
}])
api.registerEventOptionSources([
{ id: 'demo.options.seen', label: 'seen', async resolve() { return [{ value: JSON.stringify(seen), label: 'seen' }] } },
])
}
`)
assertRegistered(record)
const action = registries.eventAction('demo.spawn')
// Both callables survived the registration copy — which is not a given: that
// copy is explicit rather than a spread, precisely so nothing rides along, and
// a member added to the contract without being added to it is a member that
// silently does not exist.
assert.equal(typeof action.revert, 'function')
assert.equal(typeof action.reconcile, 'function')
assert.equal(action.owner, 'demo')
const answer = await action.revert({
runId: 3,
resources: [{ kind: 'creature', ref: '0xA', payload: null, memberKey: null }],
idempotencyKey: 'k-1',
})
assert.deepEqual(answer, { ok: true })
// Read back through the module's own option source rather than out of a
// closure this file holds: the point is that what core PASSED is what the
// module SAW, across the seam.
const seen = JSON.parse((await registries.resolveOptionSource('demo.options.seen')).options[0].value)
assert.equal(seen.runId, 3)
assert.equal(seen.idempotencyKey, 'k-1')
assert.deepEqual(seen.resources, [{ kind: 'creature', ref: '0xA', payload: null, memberKey: null }])
})
test('reconcile is optional, and a module without one still registers', () => {
// The asymmetry with `revert`, from the loader's side. A module that cannot say
// what the game still has is not broken — core keeps believing its own ledger,
// which is the behaviour before this phase — whereas one that creates something
// and cannot undo it has made a promise core has no way to keep.
const withNone = loadModule('quiet', `module.exports = (ctx, api) => {
api.registerEventActions([{
id: 'quiet.spawn',
label: 'Spawn',
risk: 'change',
reversible: 'ledger',
params: [],
async perform() { return { ok: true } },
async revert() { return { ok: true } },
}])
}`)
assertRegistered(withNone)
assert.equal(registries.eventAction('quiet.spawn').reconcile, null)
const withoutRevert = loadModule('broken', `module.exports = (ctx, api) => {
api.registerEventActions([{
id: 'broken.spawn',
label: 'Spawn',
risk: 'change',
reversible: 'ledger',
params: [],
async perform() { return { ok: true } },
}])
}`)
assert.equal(withoutRevert.state, 'startup_failed')
assert.match(withoutRevert.reason, /reversible: 'ledger' but has no revert\(\)/)
})
test('a module cannot claim core\'s reserved resource kind', async () => {
// `@step` is the placeholder's kind, and the placeholder is the row whose
// survival is the safety property: a module able to write one could make its
// own step look already accounted for. Refused at recording, and the STEP still
// succeeds — because it did.
const record = loadModule('sneaky', `module.exports = (ctx, api) => {
api.registerEventActions([{
id: 'sneaky.spawn',
label: 'Spawn',
risk: 'change',
reversible: 'ledger',
params: [],
async perform() { return { ok: true, resources: [{ kind: '@step', ref: 'anything' }] } },
async revert() { return { ok: true } },
}])
}`)
assertRegistered(record)
// eslint-disable-next-line global-require
const ledger = require('../src/events/ledger')
const result = await dispatch.dispatchStep(step('sneaky.spawn'), { run: RUN })
assert.equal(result.outcome, 'done')
const parsed = ledger.normalise(result.resources[0], 'sneaky.spawn')
assert.equal(parsed.ok, false)
assert.match(parsed.reason, /reserved kind/)
})
test('core registers the lease VERB and a module registers the lease', async () => {
// The seam working the way round it is meant to (Phase 8). A module ships the
// three callables; the verb an author puts in a step is `core.lease`, so the
// duration bound and the two-events-one-target conflict check live in one place
// rather than being re-implemented once per module and advisory everywhere.
const record = loadModule('demo', `module.exports = (ctx, api) => {
api.registerEventLeases([{
id: 'demo.rate.gain',
label: 'Gain rate',
type: 'float',
min: 0.5,
max: 5,
maxDurationMs: 3600000,
async read() { return { ok: true, value: 1 } },
async apply() { return { ok: true } },
async restore() { return { ok: true } },
}])
}`)
assertRegistered(record)
registries.registerCore()
// The module registers no ACTION at all, and its lease is still reachable.
assert.equal(registries.eventAction('demo.lease'), null)
assert.equal(registries.eventAction('core.lease').reversible, 'override')
// And the dropdown behind `core.lease`'s first param is answered by what the
// module declared — resolved per request, so a module that booted later is
// still in the list.
const options = await registries.resolveOptionSource('core.options.leases')
assert.equal(options.ok, true)
assert.deepEqual(options.options, [{ value: 'demo.rate.gain', label: 'Gain rate', group: 'demo' }])
})
test('core refuses a lease held longer than the module allows', async () => {
// The bound is the MODULE's number and the enforcement is CORE's, which is the
// §F split stated as one assertion. `retry: false` because a duration that is
// too long will still be too long in sixty seconds: it is an authoring error,
// not an outage.
const record = loadModule('demo', `module.exports = (ctx, api) => {
api.registerEventLeases([{
id: 'demo.rate.gain',
label: 'Gain rate',
type: 'float',
min: 0.5,
max: 5,
maxDurationMs: 3600000,
async read() { return { ok: true, value: 1 } },
async apply() { return { ok: true } },
async restore() { return { ok: true } },
}])
}`)
assertRegistered(record)
registries.registerCore()
const tooLong = await dispatch.dispatchStep(
step('core.lease', { lease: 'demo.rate.gain', value: '3', minutes: 120 }),
{ run: RUN },
)
assert.equal(tooLong.outcome, 'terminal')
assert.match(tooLong.error, /at most 60 minutes, not 120/)
// The same for a value outside the declared range. Unlike a cap, a bad lease
// value is in force the moment it is applied, which is why min/max are required
// on the numeric types rather than advisory.
const tooBig = await dispatch.dispatchStep(
step('core.lease', { lease: 'demo.rate.gain', value: '9', minutes: 10 }),
{ run: RUN },
)
assert.equal(tooBig.outcome, 'terminal')
assert.match(tooBig.error, /accepts 0\.5 to 5/)
// And a lease nobody registers, which is the dormancy rule one registry along.
const missing = await dispatch.dispatchStep(
step('core.lease', { lease: 'demo.nope', value: '3', minutes: 10 }),
{ run: RUN },
)
assert.equal(missing.outcome, 'terminal')
assert.match(missing.error, /no module registers the lease "demo\.nope"/)
})
test('a dry run of core.lease checks everything and takes nothing', async () => {
// `verify: true` must change nothing and must answer honestly (§F). A verify
// that reserved the target would be a dry run that changed something — and it
// would then refuse the real run that followed it, which is the worst of both.
let applied = 0
const record = loadModule('demo', `
let applied = 0
module.exports = (ctx, api) => {
api.registerEventLeases([{
id: 'demo.rate.gain',
label: 'Gain rate',
type: 'float',
min: 0.5,
max: 5,
maxDurationMs: 3600000,
async read() { return { ok: true, value: 1 } },
async apply() { applied += 1; return { ok: true } },
async restore() { return { ok: true } },
}])
api.registerEventOptionSources([
{ id: 'demo.options.applied', label: 'applied', async resolve() { return [{ value: String(applied), label: 'n' }] } },
])
}
`)
assertRegistered(record)
registries.registerCore()
void applied
const ok = await dispatch.dispatchStep(
step('core.lease', { lease: 'demo.rate.gain', value: '3', minutes: 10 }),
{ run: RUN, verify: true },
)
assert.equal(ok.outcome, 'done')
assert.equal((await registries.resolveOptionSource('demo.options.applied')).options[0].value, '0')
// A dry run that is still a real check: the bad duration is caught with
// `verify: true` as well, which is the whole value of the switchboard's
// "find out before you schedule it".
const bad = await dispatch.dispatchStep(
step('core.lease', { lease: 'demo.rate.gain', value: '3', minutes: 999 }),
{ run: RUN, verify: true },
)
assert.equal(bad.outcome, 'terminal')
})

View File

@@ -35,6 +35,11 @@ const runsDb = require('../src/model/events/eventRuns.db')
const stepsDb = require('../src/model/events/eventRunSteps.db')
const logDb = require('../src/model/events/eventRunLog.db')
const gatesDb = require('../src/model/events/eventPhaseGates.db')
// Phase 8: cancel now decides what happens to the run's world changes, and
// `cleanupRun` is the eighth control. Same rule as every phase since the fourth —
// a new leg under a model needs a stub in every file that stubs that layer.
const resourcesDb = require('../src/model/events/eventRunResources.db')
const eventCleanup = require('../src/events/cleanup')
const db = require('../src/utils/db')
after(() => db.close())
@@ -48,12 +53,35 @@ const originals = [
['steps', stepsDb, { ...stepsDb }],
['log', logDb, { ...logDb }],
['gates', gatesDb, { ...gatesDb }],
['resources', resourcesDb, { ...resourcesDb }],
['cleanup', eventCleanup, { ...eventCleanup }],
]
function installStubs() {
store = { runs: new Map(), steps: new Map(), log: [], gates: new Map(), nextStepId: 1, nextGateId: 1 }
store = { runs: new Map(), steps: new Map(), log: [], gates: new Map(), sweeps: [], unresolved: {}, nextStepId: 1, nextGateId: 1 }
const snap = (o) => ({ ...o })
runsDb.setCleanupStatus = async (id, to, from = null) => {
const r = store.runs.get(Number(id))
if (!r) return false
if (from && !from.includes(r.cleanup_status)) return false
r.cleanup_status = to
return true
}
// The sweep itself is `eventCleanup.test.js`'s subject. What this file is
// about is which control calls it, with what, and whether it is allowed to.
eventCleanup.cleanupRun = async (run, opts = {}) => {
store.sweeps.push({ runId: run.id, ...opts })
return { attempted: 1, reverted: 1, drifted: 0, failed: 0, remaining: 0 }
}
// Cancel asks the LEDGER whether the run owes the world anything, so that
// `cleanup: false` on a run with nothing recorded does not stamp `incomplete`
// over `not_required`. Unstubbed this is the ten-second dead-port wait, for the
// fifth time in this feature.
resourcesDb.unresolvedCount = async (runId) => store.unresolved[runId] ?? 0
runsDb.getById = async (id) => {
const r = store.runs.get(Number(id))
return r ? snap(r) : null
@@ -184,7 +212,7 @@ function seedGate(runId, phase, { kind = 'on', trigger = 'test.trigger', needed
return g
}
function seedRun({ status = 'running', phase = 'main', steps = [] } = {}) {
function seedRun({ status = 'running', phase = 'main', steps = [], cleanupStatus = 'not_required' } = {}) {
const id = nextRunId++
store.runs.set(id, {
id,
@@ -192,6 +220,7 @@ function seedRun({ status = 'running', phase = 'main', steps = [] } = {}) {
version_id: id,
status,
health: 'ok',
cleanup_status: cleanupStatus,
current_phase: phase,
claimed_by: null,
claim_expires_at: null,
@@ -559,3 +588,115 @@ test('an empty reason is stored as NULL rather than as an empty string', async (
await controls.pause(id, { reason: ' ' }, ACTOR)
assert.equal(lastLog().detail.reason, null)
})
// ── cancel decides what happens to the world (Phase 8) ─────────────────────
test('cancel gives back what the run took, by default and without waiting for it', async () => {
// The teardown is the runner cleanup leg over TERMINAL runs, not this request.
// Two reasons, and both are why the control answers at once: a cancel pressed
// at two in the morning must not block on a dozen round trips to the shard
// that may BE the reason it is being cancelled, and a process that dies
// halfway through a teardown has to resume rather than leave a world half
// restored with nothing scheduled to finish it.
const id = seedRun({ status: 'running', cleanupStatus: 'pending', steps: [{ status: 'pending' }] })
store.unresolved[id] = 3
const result = await controls.cancel(id, { reason: 'called off' }, ACTOR)
assert.equal(result.ok, true)
assert.equal(result.cleanup, true)
assert.deepEqual(store.sweeps, [], 'the request must not do the teardown itself')
// Still `pending`, which is what the leg looks for. The run is terminal the
// moment this returns, so the very next tick picks its ledger up.
assert.equal(runRow(id).cleanup_status, 'pending')
assert.equal(store.log.at(-1).detail.cleanup, true)
})
test('cancel WITHOUT cleanup is admin-only, even though the route is wider', async () => {
// §L: "cancelling without cleanup is a separate, logged, admin-only action."
// The route is `admin` + `moderator`, so the narrower gate cannot live in
// middleware — WHICH of the two you have to be depends on what is in the body,
// exactly as the authoring role floor does (§K).
const id = seedRun({ status: 'running', cleanupStatus: 'pending', steps: [{ status: 'pending' }] })
store.unresolved[id] = 3
const refused = await controls.cancel(id, { cleanup: false }, ACTOR, { isAdmin: false })
assert.equal(refused.ok, false)
assert.equal(refused.status, 403)
assert.equal(runRow(id).status, 'running', 'and the run is not cancelled either')
// A moderator asking for the ordinary cancel is fine: the safe direction is
// the default, so the widest gate keeps the button it exists for.
const allowed = await controls.cancel(id, {}, ACTOR, { isAdmin: false })
assert.equal(allowed.ok, true)
assert.equal(allowed.cleanup, true)
})
test('cancel without cleanup leaves the world changes up, and says so on the run', async () => {
// `incomplete` is the truthful value rather than a tidy one: the changes are
// still up, they are listed on the console, and the log line records who
// decided that. A `complete` here would be the "tidy completed row over a shard
// full of orphaned monsters" §L names as the failure that ends this feature's
// credibility.
const id = seedRun({ status: 'running', cleanupStatus: 'pending', steps: [{ status: 'pending' }] })
store.unresolved[id] = 3
const result = await controls.cancel(id, { cleanup: false, reason: 'leave it up' }, ACTOR)
assert.equal(result.ok, true)
assert.equal(result.cleanup, false)
assert.equal(runRow(id).cleanup_status, 'incomplete')
assert.equal(store.log.at(-1).detail.cleanup, false)
assert.equal(store.log.at(-1).detail.by, ACTOR)
})
test('a run that recorded nothing is unaffected by either flag', async () => {
// `not_required` is not walked to `incomplete` by a cancel that skipped a
// teardown there was nothing to do — and it is the LEDGER that says so, not the
// status column, because `not_required` is also what a run holding only a lease
// wrongly carried before the live walk found it.
const id = seedRun({ status: 'running', cleanupStatus: 'not_required', steps: [{ status: 'pending' }] })
store.unresolved[id] = 0
await controls.cancel(id, { cleanup: false }, ACTOR)
assert.equal(runRow(id).cleanup_status, 'not_required')
})
// ── cleanup, the eighth control ────────────────────────────────────────────
test('cleanup re-runs the teardown and clears the attempt counter', async () => {
// The manual retry §L promises. `resetAttempts` is the licence a human has and
// the automatic sweep does not — Engagement Phase 14's rule, whose defect was
// a sweep that reset every stale row and made the attempt ceiling unreachable.
const id = seedRun({ status: 'completed', cleanupStatus: 'incomplete' })
const result = await controls.cleanupRun(id, ACTOR)
assert.equal(result.ok, true)
assert.deepEqual(store.sweeps, [{ runId: id, resetAttempts: true, actor: ACTOR }])
assert.equal(result.summary.reverted, 1)
})
test('cleanup refuses a run that is still in flight', async () => {
// A run still going has a ledger that is still growing, and reverting a
// resource the next step is about to use would be core undoing an event while
// it is happening. Cancel is the control for a run that should stop.
for (const status of ['scheduled', 'starting', 'running', 'paused', 'ending']) {
const id = seedRun({ status, cleanupStatus: 'pending' })
const result = await controls.cleanupRun(id, ACTOR)
assert.equal(result.ok, false, status)
assert.match(result.errors[0], /cancel it before cleaning up after it/)
}
assert.deepEqual(store.sweeps, [])
})
test('cleanup refuses a run that recorded no resources', async () => {
const id = seedRun({ status: 'completed', cleanupStatus: 'not_required' })
const result = await controls.cleanupRun(id, ACTOR)
assert.equal(result.ok, false)
assert.match(result.errors[0], /nothing to give back/)
})
test('cleanup on an unknown run is a 404, not a 409', async () => {
const result = await controls.cleanupRun(9999, ACTOR)
assert.equal(result.status, 404)
})

View File

@@ -45,6 +45,11 @@ const gatesDb = require('../src/model/events/eventPhaseGates.db')
// every file that stubs the layer under it needs the stub.
const settingsDb = require('../src/model/events/eventActionSettings.db')
const budgetDb = require('../src/model/events/eventRunBudget.db')
// Phase 8 put a ledger write in front of every world-changing dispatch and a
// cleanup leg at the end of the tick. **Fourth time, same rule, and this file's
// own note is what caught it**: unstubbed, one of these is not a wrong answer,
// it is a ten-second wait on the dead port.
const resourcesDb = require('../src/model/events/eventRunResources.db')
const gates = require('../src/events/gates')
const db = require('../src/utils/db')
@@ -58,7 +63,7 @@ const later = (ms) => new Date(T0.getTime() + ms)
let store
const originals = {}
for (const [name, mod] of [['runsDb', runsDb], ['stepsDb', stepsDb], ['logDb', logDb], ['versionsDb', versionsDb], ['definitionsDb', definitionsDb], ['gatesDb', gatesDb], ['settingsDb', settingsDb], ['budgetDb', budgetDb]]) {
for (const [name, mod] of [['runsDb', runsDb], ['stepsDb', stepsDb], ['logDb', logDb], ['versionsDb', versionsDb], ['definitionsDb', definitionsDb], ['gatesDb', gatesDb], ['settingsDb', settingsDb], ['budgetDb', budgetDb], ['resourcesDb', resourcesDb]]) {
originals[name] = { mod, fns: { ...mod } }
}
@@ -79,8 +84,10 @@ function installStubs() {
gates: new Map(),
settings: new Map(),
budget: new Map(),
resources: new Map(),
nextStepId: 1,
nextGateId: 1,
nextResourceId: 1,
}
// Phase 4 put a schedule-expansion leg in front of the tick. This file is
@@ -91,6 +98,69 @@ function installStubs() {
// connection timeout.
Object.assign(definitionsDb, { findSchedulable: async () => [] })
// ── The resource ledger (Phase 8) ──
//
// `reserve` enforces `uq_evres_target` in the stub, because the refusal it
// produces is BEHAVIOUR the runner branches on rather than an implementation
// detail: a placeholder that collides is this step's own earlier attempt and is
// reused, and a lease that collides is another run holding the target. A stub
// that let both inserts through would make the retry path grow a second row and
// the conflict test pass for no reason.
const HELD_STATUSES = ['pending', 'confirmed', 'reverting']
resourcesDb.reserve = async ({ runId, stepId = null, owner, kind, ref, payload = null, leaseUntil = null, memberKey = null }) => {
const holder = [...store.resources.values()].find(
(r) => r.owner_module === owner && r.kind === kind && r.ref === ref && HELD_STATUSES.includes(r.status),
)
if (holder) return { ok: false, code: 'held', holder: { run_id: holder.run_id, status: holder.status } }
const id = store.nextResourceId++
store.resources.set(id, {
id,
run_id: runId,
step_id: stepId,
owner_module: owner,
kind,
ref,
payload,
lease_until: leaseUntil,
status: 'pending',
revert_attempts: 0,
last_error: null,
member_key: memberKey,
})
return { ok: true, id }
}
resourcesDb.confirm = async (id) => {
const r = store.resources.get(id)
if (!r || r.status !== 'pending') return false
r.status = 'confirmed'
return true
}
resourcesDb.resolvePlaceholder = async (id) => {
const r = store.resources.get(id)
if (!r || r.kind !== resourcesDb.STEP_KIND || !['pending', 'confirmed'].includes(r.status)) return false
r.status = 'reverted'
return true
}
resourcesDb.findByTarget = async (owner, kind, ref) =>
[...store.resources.values()].reverse().find((r) => r.owner_module === owner && r.kind === kind && r.ref === ref) || null
resourcesDb.forRun = async (runId) => [...store.resources.values()].filter((r) => r.run_id === runId)
resourcesDb.markReverted = async (id) => {
const r = store.resources.get(id)
if (r) r.status = 'reverted'
}
// The cleanup leg's scan. This file is about the runner's own legs, and the
// sweep has its own file — answering with nothing is what keeps every `tick()`
// here measuring the runner rather than a teardown.
resourcesDb.runsNeedingCleanup = async () => []
runsDb.setCleanupStatus = async (id, to, from = null) => {
const r = store.runs.get(id)
if (!r) return false
if (from && !from.includes(r.cleanup_status)) return false
r.cleanup_status = to
return true
}
// Snapshots, not live references. A SQL SELECT hands back a copy, and the
// runner reads `step.attempts` as the value BEFORE its own claim incremented
// it — returning references here would make the retry budget off by one in the
@@ -1319,3 +1389,163 @@ test('the runner never re-checks the role of whoever started the run', async ()
assert.equal(stepsOf(id)[0].status, 'done')
})
// ── The resource ledger, from the runner's side (Phase 8) ──────────────────
//
// `eventLedger.test.js` owns the recording RULES and `eventCleanup.test.js` owns
// the undo. What belongs here is the ORDER — that the placeholder is written
// before the module is reached and after the permission check, and that a step
// which never answers leaves it standing. Those are properties of `drainStep`,
// and nothing below the runner can observe them.
const ledgerRows = (id) => [...store.resources.values()].filter((r) => r.run_id === id)
test('a world-changing step is recorded BEFORE it is dispatched', async () => {
// §D rule 1. The assertion is made from INSIDE `perform()`, which is the only
// place that can tell "recorded first" from "recorded at all" — and the
// difference between the two is every object whose acknowledgement is lost.
let seenDuringDispatch = null
register([scriptedAction('test.spawn', { risk: 'change', reversible: 'ledger', revert: async () => ({ ok: true }) })])
setSwitch('test.spawn', true)
const id = seedRun([{ key: 'main', steps: [step('test.spawn')] }])
scripted['test.spawn'] = {
calls: [],
answer: () => {
seenDuringDispatch = ledgerRows(id).map((r) => [r.kind, r.status])
return { ok: true, resources: [{ kind: 'creature', ref: '0x40001234' }] }
},
}
await runner.tick(T0)
assert.deepEqual(seenDuringDispatch, [['@step', 'pending']])
// And on the answer the real row exists and the placeholder is done with.
assert.deepEqual(
ledgerRows(id).map((r) => [r.kind, r.ref, r.status]),
[
['@step', stepsOf(id)[0].idempotency_key, 'reverted'],
['creature', '0x40001234', 'confirmed'],
],
)
assert.equal(run(id).cleanup_status, 'pending')
assert.ok(kinds(id).includes('resource.recorded'))
})
test('a step that never answers leaves its placeholder pending', async () => {
// The whole reason the placeholder exists. The module timed out, so core does
// not know whether anything was created — and the row that survives is what
// lets cleanup ask it by idempotency key later. Record on the answer instead
// and this run finishes looking spotless over a shard full of orphans.
register([
scriptedAction('test.spawn', {
risk: 'change',
reversible: 'ledger',
budgetMs: 5,
revert: async () => ({ ok: true }),
perform: () => new Promise(() => {}),
}),
])
setSwitch('test.spawn', true)
const id = seedRun([{ key: 'main', steps: [step('test.spawn', {}, 'skip')] }])
await runner.tick(T0)
assert.deepEqual(ledgerRows(id).map((r) => [r.kind, r.status]), [['@step', 'pending']])
})
test('a refused step ledgers nothing, because it never reached the module', async () => {
// The placeholder is written AFTER the permission check, deliberately. A step
// refused by a cap or by a switch created nothing, and a ledger row for it
// would be core asking a module to undo something it was never asked to do.
register([scriptedAction('test.spawn', { risk: 'change', reversible: 'ledger', revert: async () => ({ ok: true }) })])
// No `setSwitch`, so it is default-off: §K's world-changing default.
const id = seedRun([{ key: 'main', steps: [step('test.spawn', {}, 'skip')] }])
await runner.tick(T0)
assert.equal(stepsOf(id)[0].status, 'refused')
assert.deepEqual(ledgerRows(id), [])
assert.equal(run(id).cleanup_status, 'not_required')
})
test('an announce step ledgers nothing at all', async () => {
// `reversible: 'none'`, so there is nothing core could come back for. A
// placeholder here would be a row teardown could never resolve — which is the
// reason `core.announce` is declared `none` rather than `ledger` even though a
// sent message cannot be unsent.
register([scriptedAction('test.say')])
const id = seedRun([{ key: 'main', steps: [step('test.say')] }])
await runner.tick(T0)
assert.equal(run(id).status, 'completed')
assert.deepEqual(ledgerRows(id), [])
assert.equal(run(id).cleanup_status, 'not_required')
})
test('a parked step records what it made, because its confirm never dispatches again', async () => {
// `await: 'human'` is a SUCCESS: the module did its part and something outside
// the system has to happen next. The cue's confirm finishes the step without a
// second dispatch, so this is the only moment its resources can be recorded.
register([
scriptedAction('test.stage', { risk: 'change', reversible: 'ledger', revert: async () => ({ ok: true }) }),
])
setSwitch('test.stage', true)
const id = seedRun([{ key: 'main', steps: [step('test.stage')] }])
scripted['test.stage'] = {
calls: [],
answer: { ok: true, await: 'human', resources: [{ kind: 'prop', ref: 'gate-1' }] },
}
await runner.tick(T0)
assert.equal(stepsOf(id)[0].status, 'running')
assert.deepEqual(
ledgerRows(id).map((r) => [r.kind, r.status]),
[['@step', 'reverted'], ['prop', 'confirmed']],
)
})
test('a retry re-uses its placeholder and records the resources once', async () => {
// An idempotency key does not vary by attempt (§E), so the second attempt's
// placeholder insert collides with the first attempt's row — and a module that
// honestly re-reports the same creature must not produce a second thing for
// cleanup to revert.
register([
scriptedAction('test.spawn', { risk: 'change', reversible: 'ledger', revert: async () => ({ ok: true }) }),
])
setSwitch('test.spawn', true)
const id = seedRun([{ key: 'main', steps: [step('test.spawn', {}, 'skip')] }])
scripted['test.spawn'] = {
calls: [],
answers: [
{ ok: false, error: 'the shard did not answer' },
{ ok: true, resources: [{ kind: 'creature', ref: '0xFEED' }] },
],
}
await runner.tick(T0)
await runner.tick(later(runner.RETRY_MS + 1000))
assert.equal(stepsOf(id)[0].status, 'done')
assert.equal(ledgerRows(id).filter((r) => r.kind === '@step').length, 1)
assert.equal(ledgerRows(id).filter((r) => r.kind === 'creature').length, 1)
})
test('a ledger write that fails stops the dispatch rather than losing the record', async () => {
// The ledger is what makes a world write recoverable, so a step that cannot be
// recorded must not be sent. Transient, because the alternative is an
// unrecorded world change — the one outcome §D rule 1 exists to make impossible.
register([
scriptedAction('test.spawn', { risk: 'change', reversible: 'ledger', revert: async () => ({ ok: true }) }),
])
setSwitch('test.spawn', true)
const id = seedRun([{ key: 'main', steps: [step('test.spawn', {}, 'skip')] }])
scripted['test.spawn'] = { calls: [], answer: { ok: true } }
resourcesDb.reserve = async () => {
throw new Error('the ledger is unreachable')
}
await runner.tick(T0)
assert.equal(scripted['test.spawn'].calls.length, 0, 'the module must not have been reached')
assert.match(stepsOf(id)[0].last_error, /the resource ledger could not record this step/)
})

View File

@@ -80,6 +80,18 @@
// that died between entering a phase and opening its gate opens no second
// one on the next tick.
//
// **Phase 8 added the resource ledger**, and its unique key is the single most
// server-dependent thing in this feature. "Two events cannot hold a lease on one
// target" has to hold among LIVE rows only — last week's finished event must not
// keep this week's from leasing the same rate — and MariaDB has no partial index,
// so the encoding is a STORED generated column that goes NULL once the row is no
// longer ours. Whether multiple NULLs collide in a unique index is a property of
// the server and of nothing else, and TEAMS.md §2.5 already had to be corrected
// once on this exact shape: MariaDB refuses ON DELETE SET NULL on a foreign key
// whose column is a base column of a stored generated column (error 1901), which
// is why the expression reads `status` alone and `step_id` stays a SET NULL FK.
// Both halves of that are proved below rather than believed.
//
// Plus the two unique indexes that are load-bearing rather than tidy:
// `uq_evrun_occurrence` (which, not the claim, is what stops two runs of one
// occurrence existing) and `uq_evstep_slot` (which is what makes re-materialising
@@ -208,6 +220,29 @@ CREATE TABLE event_run_budget (
CONSTRAINT fk_evbud_run FOREIGN KEY (run_id) REFERENCES event_runs(id) ON DELETE CASCADE,
UNIQUE KEY uq_evbud_dim (run_id, dimension)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
CREATE TABLE event_run_resources (
id BIGINT AUTO_INCREMENT PRIMARY KEY,
run_id BIGINT NOT NULL,
step_id BIGINT NULL,
owner_module VARCHAR(64) NOT NULL,
kind VARCHAR(64) NOT NULL,
ref VARCHAR(190) NOT NULL,
payload JSON NULL,
lease_until DATETIME NULL,
status ENUM('pending','confirmed','reverting','reverted','orphaned','drifted')
NOT NULL DEFAULT 'pending',
revert_attempts INT NOT NULL DEFAULT 0,
last_error VARCHAR(500) NULL,
member_key VARCHAR(190) NULL,
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
live_marker TINYINT AS (IF(status IN ('pending','confirmed','reverting'), 1, NULL)) STORED,
CONSTRAINT fk_evres_run FOREIGN KEY (run_id) REFERENCES event_runs(id) ON DELETE CASCADE,
CONSTRAINT fk_evres_step FOREIGN KEY (step_id) REFERENCES event_run_steps(id) ON DELETE SET NULL,
UNIQUE KEY uq_evres_target (owner_module, kind, ref, live_marker),
INDEX idx_evres_run (run_id, status),
INDEX idx_evres_live (status, lease_until)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
CREATE TABLE event_action_settings (
action_id VARCHAR(96) NOT NULL PRIMARY KEY,
enabled TINYINT(1) NOT NULL DEFAULT 0,
@@ -438,6 +473,7 @@ const rows = (r) => Number(r.affectedRows)
beforeEach(async () => {
if (!available) return
await pool.query('DELETE FROM event_run_phase_gates')
await pool.query('DELETE FROM event_run_resources')
await pool.query('DELETE FROM event_run_steps')
await pool.query('DELETE FROM event_runs')
await pool.query('DELETE FROM event_definitions')
@@ -1139,7 +1175,21 @@ test('a gate goes with its run', async (t) => {
// live server, so the pool holds open connections and the process never exits —
// 49 green tests and a file that hangs until the harness kills it. Every other
// event test file already closes it in an `after`; this one now has a reason to.
//
// **And that second pool has to be aimed at the throwaway database**, which it
// was not before Phase 8 noticed. `utils/db` builds its pool at REQUIRE time from
// `DB_NAME`, and its own `dotenv.config()` reads `server/.env` — so a model test
// run on a developer's machine was reaching that developer's real schema while
// the fixtures it was asserting against were being written next door. It passed
// only because the two tables happened to exist in both. `dotenv` does not
// overwrite a variable that already exists, so setting it here, before the
// require below, is what beats the file. This file drops its database in `after`,
// so aiming the model pool at it is also what keeps the whole run disposable.
process.env.DB_NAME = DB
const budgetDb = require('../src/model/events/eventRunBudget.db')
const resourcesDb = require('../src/model/events/eventRunResources.db')
const runsDb = require('../src/model/events/eventRuns.db')
const appDb = require('../src/utils/db')
after(() => appDb.close())
@@ -1303,3 +1353,324 @@ test('a non-positive spend never reaches the database', async (t) => {
assert.equal(await budgetDb.spend(runId, 'uo.creatures', 0), true)
assert.equal(await consumedOf(runId), 0)
})
// ── The resource ledger's unique key (Phase 8) ─────────────────────────────
//
// Every test here is about a property of the SERVER. A stub can enforce whatever
// rule its author had in mind; only MariaDB can say whether this encoding of
// "unique among live rows" actually is one.
const insertResource = async (runId, over = {}) =>
(
await pool.query(
`INSERT INTO event_run_resources (run_id, step_id, owner_module, kind, ref, payload, lease_until, member_key, status)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
[
runId,
over.stepId ?? null,
over.owner ?? 'demo',
over.kind ?? 'creature',
over.ref ?? '0xA',
over.payload ?? null,
over.leaseUntil ?? null,
over.memberKey ?? null,
over.status ?? 'pending',
],
)
).insertId
const dup = async (fn) => {
try {
await fn()
return null
} catch (err) {
return err.code || String(err.errno)
}
}
test('two runs cannot hold one target, and the refusal comes from the server', async (t) => {
if (needDb(t)) return
// §D: "the unique index is what stops two events leasing one target." Not a
// read-then-insert — two runs entering the same tick would both pass the check
// — so the whole conflict story is this one constraint answering.
const a = await seedRun()
const b = await seedRun()
await insertResource(a.runId, { kind: 'override', ref: 'demo.rate', status: 'confirmed' })
const code = await dup(() => insertResource(b.runId, { kind: 'override', ref: 'demo.rate' }))
assert.equal(code, 'ER_DUP_ENTRY')
})
test('the key is released by the three statuses that mean it is no longer ours', async (t) => {
if (needDb(t)) return
// Amended 2026-09-03. §D says "among non-reverted rows", which was written
// before the six statuses had their meanings; taken literally it makes
// `drifted` and `orphaned` hold a target for ever, so one bad night would
// disable a lease permanently with no control able to clear it. `drifted` means
// somebody else has hold of the value and this run has let go; `orphaned` means
// it vanished. Neither is a claim on the target.
for (const status of ['reverted', 'drifted', 'orphaned']) {
await pool.query('DELETE FROM event_run_resources')
const a = await seedRun()
const b = await seedRun()
await insertResource(a.runId, { kind: 'override', ref: 'demo.rate', status })
const code = await dup(() => insertResource(b.runId, { kind: 'override', ref: 'demo.rate' }))
assert.equal(code, null, `a ${status} row must not hold the target`)
}
})
test('the key is HELD by the three that mean core still believes it is ours', async (t) => {
if (needDb(t)) return
for (const status of ['pending', 'confirmed', 'reverting']) {
await pool.query('DELETE FROM event_run_resources')
const a = await seedRun()
const b = await seedRun()
await insertResource(a.runId, { kind: 'override', ref: 'demo.rate', status })
const code = await dup(() => insertResource(b.runId, { kind: 'override', ref: 'demo.rate' }))
assert.equal(code, 'ER_DUP_ENTRY', `a ${status} row must hold the target`)
}
})
test('many released rows on one target coexist, which is the whole encoding', async (t) => {
if (needDb(t)) return
// The property the NULL depends on: multiple NULLs do not collide in a unique
// index. A weekly event that leases the same rate every Saturday accumulates one
// released row per week, and the fiftieth must not fail to insert.
const a = await seedRun()
for (let i = 0; i < 5; i++) {
await insertResource(a.runId, { kind: 'override', ref: 'demo.rate', status: 'reverted' })
}
const held = await pool.query(
'SELECT COUNT(*) AS n FROM event_run_resources WHERE ref = ? AND live_marker IS NULL',
['demo.rate'],
)
assert.equal(Number(held[0].n), 5)
// And a live one still goes on top of them.
assert.equal(await dup(() => insertResource(a.runId, { kind: 'override', ref: 'demo.rate' })), null)
})
test('an UPDATE that releases a row frees the target at once', async (t) => {
if (needDb(t)) return
// The generated column is STORED, so this is really asking whether MariaDB
// recomputes it on UPDATE and re-indexes. Cleanup depends on it entirely: the
// moment a lease is restored, the next event may take it.
const a = await seedRun()
const b = await seedRun()
const id = await insertResource(a.runId, { kind: 'override', ref: 'demo.rate', status: 'confirmed' })
assert.equal(await dup(() => insertResource(b.runId, { kind: 'override', ref: 'demo.rate' })), 'ER_DUP_ENTRY')
await pool.query("UPDATE event_run_resources SET status = 'reverted' WHERE id = ?", [id])
assert.equal(await dup(() => insertResource(b.runId, { kind: 'override', ref: 'demo.rate' })), null)
})
test('the key is per owner and per kind, not per ref', async (t) => {
if (needDb(t)) return
// `kind` and `ref` are module-opaque strings core stores verbatim, so two
// modules using the same word for different things must not collide — and one
// module's `creature:0xA` and `item:0xA` are two objects.
const a = await seedRun()
await insertResource(a.runId, { owner: 'demo', kind: 'creature', ref: '0xA', status: 'confirmed' })
assert.equal(await dup(() => insertResource(a.runId, { owner: 'other', kind: 'creature', ref: '0xA' })), null)
assert.equal(await dup(() => insertResource(a.runId, { owner: 'demo', kind: 'item', ref: '0xA' })), null)
assert.equal(await dup(() => insertResource(a.runId, { owner: 'demo', kind: 'creature', ref: '0xB' })), null)
})
test('a deleted step leaves its resources behind, and a deleted run does not', async (t) => {
if (needDb(t)) return
// TEAMS.md §2.5's correction, held as a test: `step_id` is SET NULL and it only
// works because the generated column reads `status` alone. If `step_id` were in
// that expression MariaDB would refuse the constraint outright (error 1901), and
// the migration would fail on a fresh install rather than here.
//
// The two directions are different on purpose. A record of what was changed in
// the WORLD must outlive the row that scheduled it — `engagement_sends`'
// argument — while a deleted RUN takes its ledger with it, because the ledger
// exists to answer questions about a run.
const a = await seedRun()
const stepId = await seedStep(a.runId)
const id = await insertResource(a.runId, { stepId, status: 'confirmed' })
await pool.query('DELETE FROM event_run_steps WHERE id = ?', [stepId])
const orphaned = (await pool.query('SELECT step_id, status FROM event_run_resources WHERE id = ?', [id]))[0]
assert.equal(orphaned.step_id, null)
assert.equal(orphaned.status, 'confirmed')
await pool.query('DELETE FROM event_runs WHERE id = ?', [a.runId])
const gone = await pool.query('SELECT id FROM event_run_resources WHERE id = ?', [id])
assert.equal(gone.length, 0)
})
test('the ledger model reserves, confirms, reverts and refuses for real', async (t) => {
if (needDb(t)) return
// Through the shipping module rather than a copy of its statements, like the
// budget tests above: `reserve` reads ER_DUP_ENTRY as a refusal and looks the
// holder up to name it, and both halves of that are the connector's behaviour
// rather than this file's.
const a = await seedRun()
const b = await seedRun()
const first = await resourcesDb.reserve({ runId: a.runId, owner: 'demo', kind: 'override', ref: 'demo.rate' })
assert.equal(first.ok, true)
await resourcesDb.confirm(first.id)
const second = await resourcesDb.reserve({ runId: b.runId, owner: 'demo', kind: 'override', ref: 'demo.rate' })
assert.equal(second.ok, false)
assert.equal(second.code, 'held')
assert.equal(Number(second.holder.run_id), Number(a.runId))
assert.equal(second.holder.status, 'confirmed')
// And once it is given back the second run gets it.
await resourcesDb.markReverted(first.id)
const third = await resourcesDb.reserve({ runId: b.runId, owner: 'demo', kind: 'override', ref: 'demo.rate' })
assert.equal(third.ok, true)
})
test('failRevert increments and NEVER resets, and only a human clears it', async (t) => {
if (needDb(t)) return
// Engagement Phase 14's rule at the statement level: `revert_attempts =
// revert_attempts + 1` is written in one place, and the reset is a separate
// statement with an actor behind it. A `SET revert_attempts = ?` anywhere in
// the sweep would make the ceiling unreachable and the row cycle for ever.
const a = await seedRun()
const id = await resourcesDb
.reserve({ runId: a.runId, owner: 'demo', kind: 'creature', ref: '0xA' })
.then((r) => r.id)
await resourcesDb.confirm(id)
await resourcesDb.failRevert(id, 'the shard did not answer')
await resourcesDb.failRevert(id, 'still nothing')
let row = (await pool.query('SELECT * FROM event_run_resources WHERE id = ?', [id]))[0]
assert.equal(Number(row.revert_attempts), 2)
assert.equal(row.status, 'confirmed')
assert.equal(row.last_error, 'still nothing')
assert.equal(await resourcesDb.resetAttempts(a.runId), 1)
row = (await pool.query('SELECT * FROM event_run_resources WHERE id = ?', [id]))[0]
assert.equal(Number(row.revert_attempts), 0)
})
test('claimRevert is a compare-and-set, and reverting is not re-claimable', async (t) => {
if (needDb(t)) return
// The cleanup leg and the manual cleanup route can both be working one run at
// once. `reverting` is deliberately not claimable — a row another pass is
// mid-revert on is left alone, exactly as a step with a live claim is.
const a = await seedRun()
const id = await resourcesDb
.reserve({ runId: a.runId, owner: 'demo', kind: 'creature', ref: '0xA' })
.then((r) => r.id)
await resourcesDb.confirm(id)
assert.equal(await resourcesDb.claimRevert(id), true)
assert.equal(await resourcesDb.claimRevert(id), false, 'a second pass must not take a row mid-revert')
await resourcesDb.markReverted(id)
assert.equal(await resourcesDb.claimRevert(id), false, 'and a reverted row is finished')
})
test('the placeholder is resolvable exactly once, and only if it is a placeholder', async (t) => {
if (needDb(t)) return
// The `kind = '@step'` guard in the statement, not in JavaScript. It is what
// stops a bug elsewhere resolving a real resource — which would be core marking
// a live creature as given back without asking anyone.
const a = await seedRun()
const placeholder = await resourcesDb
.reserve({ runId: a.runId, owner: 'demo', kind: resourcesDb.STEP_KIND, ref: 'k'.repeat(40) })
.then((r) => r.id)
const real = await resourcesDb
.reserve({ runId: a.runId, owner: 'demo', kind: 'creature', ref: '0xA' })
.then((r) => r.id)
assert.equal(await resourcesDb.resolvePlaceholder(placeholder), true)
assert.equal(await resourcesDb.resolvePlaceholder(placeholder), false)
assert.equal(await resourcesDb.resolvePlaceholder(real), false)
const stillThere = (await pool.query('SELECT status FROM event_run_resources WHERE id = ?', [real]))[0]
assert.equal(stillThere.status, 'pending')
})
test('the unresolved reads see the five statuses that still want something', async (t) => {
if (needDb(t)) return
// `cleanup_status` is derived from this count, so what it includes IS the
// definition of "clean". A `drifted` row left out of it would let a run whose
// lease somebody else took call itself complete.
const a = await seedRun()
for (const status of ['pending', 'confirmed', 'reverting', 'reverted', 'orphaned', 'drifted']) {
await insertResource(a.runId, { ref: `ref-${status}`, status })
}
assert.equal(await resourcesDb.unresolvedCount(a.runId), 5)
const rows = await resourcesDb.unresolvedForRun(a.runId)
assert.deepEqual(
rows.map((r) => r.status).sort(),
['confirmed', 'drifted', 'orphaned', 'pending', 'reverting'],
)
const counts = await resourcesDb.unresolvedCounts([a.runId])
assert.equal(counts.get(a.runId) ?? counts.get(String(a.runId)), 5)
})
test('payload comes back as an object rather than as a string', async (t) => {
if (needDb(t)) return
// A lease's baseline lives in here and the cleanup sweep reads it out to pass
// to `restore`. The connector hands JSON back as text, so a missing hydration
// is a `restore(undefined)` — a lease put back to nothing, silently.
const a = await seedRun()
const id = await resourcesDb
.reserve({
runId: a.runId,
owner: 'demo',
kind: 'override',
ref: 'demo.rate',
payload: { baseline: 1, applied: 3 },
})
.then((r) => r.id)
void id
const [row] = await resourcesDb.forRun(a.runId)
assert.deepEqual(row.payload, { baseline: 1, applied: 3 })
})
test('the cleanup scan finds a run that owes something, whatever its status column says', async (t) => {
if (needDb(t)) return
await pool.query('ALTER TABLE event_runs ADD COLUMN IF NOT EXISTS cleanup_status ' +
"ENUM('not_required','pending','complete','incomplete') NOT NULL DEFAULT 'not_required'")
// **`not_required` is in the scan because of a live-walk defect**, not for
// tidiness. A run whose only resource was a LEASE never went through the
// ledger's `markRunDirty` — `core.lease` reserves its own row — so its column
// stayed `not_required` and the lease was never given back at all. A terminal
// run with an unresolved row has work to do whatever any summary column says.
const lease = await seedRun({ status: 'completed' })
await insertResource(lease.runId, { kind: 'override', ref: 'demo.rate', status: 'confirmed' })
const marked = await seedRun({ status: 'cancelled' })
await insertResource(marked.runId, { ref: '0xB', status: 'pending' })
await pool.query("UPDATE event_runs SET cleanup_status = 'pending' WHERE id = ?", [marked.runId])
// Three that must NOT be selected, one per reason.
const running = await seedRun({ status: 'running' })
await insertResource(running.runId, { ref: '0xC', status: 'confirmed' })
const done = await seedRun({ status: 'completed' })
await insertResource(done.runId, { ref: '0xD', status: 'reverted' })
const givenUp = await seedRun({ status: 'completed' })
await insertResource(givenUp.runId, { ref: '0xE', status: 'confirmed' })
await pool.query("UPDATE event_runs SET cleanup_status = 'incomplete' WHERE id = ?", [givenUp.runId])
const found = (await resourcesDb.runsNeedingCleanup(10, 3)).map((r) => Number(r.id))
assert.deepEqual(found.sort(), [Number(lease.runId), Number(marked.runId)].sort())
})
test('the scan stops selecting a run whose attempts are spent', async (t) => {
if (needDb(t)) return
// Without the bound in the join, a run whose rows are all spent would be
// selected, worked over and found to have nothing to do on every tick for the
// rest of its life.
const a = await seedRun({ status: 'completed' })
const id = await insertResource(a.runId, { status: 'confirmed' })
await pool.query("UPDATE event_runs SET cleanup_status = 'pending' WHERE id = ?", [a.runId])
assert.equal((await resourcesDb.runsNeedingCleanup(10, 3)).length, 1)
await pool.query('UPDATE event_run_resources SET revert_attempts = 3 WHERE id = ?', [id])
assert.equal((await resourcesDb.runsNeedingCleanup(10, 3)).length, 0)
})
test('setCleanupStatus is guarded, which is what stops a late row re-opening a swept run', async (t) => {
if (needDb(t)) return
const a = await seedRun({ status: 'completed' })
assert.equal(await runsDb.setCleanupStatus(a.runId, 'complete'), true)
assert.equal(await runsDb.setCleanupStatus(a.runId, 'pending', ['not_required']), false)
assert.equal(await runsDb.setCleanupStatus(a.runId, 'pending'), true)
})

View File

@@ -42,6 +42,13 @@ const logDb = require('../src/model/events/eventRunLog.db')
// model needs a stub in every file that stubs that layer.
const settingsDb = require('../src/model/events/eventActionSettings.db')
const budgetDb = require('../src/model/events/eventRunBudget.db')
// Phase 8: the run console reads the resource ledger. The SAME rule, for the
// fourth time in this feature -- Phase 4's expansion leg, Phase 5's gate read,
// Phase 6's settings read and now this one. **Unstubbed it is not a wrong
// answer, it is a ten-second ECONNREFUSED against the dead port**, which is why
// one missing stub here cost the run detail test ten seconds and said nothing
// about the route it was testing.
const resourcesDb = require('../src/model/events/eventRunResources.db')
const seriesDb = require('../src/model/events/eventSeries.db')
const gatesDb = require('../src/model/events/eventPhaseGates.db')
const activity = require('../src/model/activity/activity.model')
@@ -63,6 +70,7 @@ for (const [name, mod] of [
['gatesDb', gatesDb],
['settingsDb', settingsDb],
['budgetDb', budgetDb],
['resourcesDb', resourcesDb],
['activity', activity],
]) {
originals[name] = { mod, fns: { ...mod } }
@@ -88,6 +96,7 @@ function installStubs() {
gates: [],
settings: new Map(),
budget: new Map(),
resources: [],
occurrences: new Set(),
nextDefinition: 1,
nextVersion: 1,
@@ -341,6 +350,9 @@ function installStubs() {
[...store.budget.values()]
.filter((b) => Number(b.run_id) === Number(runId))
.sort((a, b) => a.dimension.localeCompare(b.dimension))
resourcesDb.forRun = async (runId) =>
store.resources.filter((r) => Number(r.run_id) === Number(runId))
}
// ── Fixtures ───────────────────────────────────────────────────────────────
@@ -415,23 +427,26 @@ test('the catalog serves the registry, callables stripped, with its vocabularies
assert.equal(res.statusCode, 200)
assert.deepEqual(
res.body.actions.map((a) => a.id),
['core.announce', 'core.wait', 'core.cue'],
['core.announce', 'core.wait', 'core.cue', 'core.lease'],
)
for (const action of res.body.actions) assert.equal(action.perform, undefined)
assert.deepEqual(res.body.risks, ['notify', 'inspect', 'change', 'irreversible'])
assert.deepEqual(res.body.onFailure, ['skip', 'pause', 'abort_run'])
// The other three registrations of the module contract arrived in Phase 7, and
// they are served BESIDE the actions because the editor needs all four to draw
// one step. Core declares no budgets and no leases of its own — its three
// actions cost nothing and hold nothing — so those are empty here, and that is
// the fact worth asserting: present and empty, not absent.
// one step. Core declares no budgets and no leases of its own — its four
// actions cost nothing, and `core.lease` BORROWS a lease rather than declaring
// one, which is the seam working the way round it is meant to: core owns the
// verb, a module owns the value. So both are empty here, and that is the fact
// worth asserting: present and empty, not absent.
assert.deepEqual(res.body.budgets, [])
assert.deepEqual(res.body.leases, [])
// One option source, and it is core's: `core.announce`'s leg param. It is here
// WITHOUT its resolver — the values are a request of their own.
// Two option sources, both core's: `core.announce`'s leg and `core.lease`'s
// lease. They are here WITHOUT their resolvers — the values are a request of
// their own.
assert.deepEqual(
res.body.optionSources.map((s) => s.id),
['core.options.legs'],
['core.options.legs', 'core.options.leases'],
)
for (const s of res.body.optionSources) assert.equal(s.resolve, undefined)
})
@@ -824,7 +839,12 @@ test('the board serves every registered action with its risk-class default, and
assert.equal(res.statusCode, 200)
const byId = Object.fromEntries(res.body.actions.map((a) => [a.id, a]))
assert.deepEqual(Object.keys(byId).sort(), ['core.announce', 'core.cue', 'core.wait'])
assert.deepEqual(Object.keys(byId).sort(), ['core.announce', 'core.cue', 'core.lease', 'core.wait'])
// And `core.lease` is the one core action the default-off rule bites: it is
// `change`, so a fresh deployment cannot borrow a value until an admin says so.
// §K's sentence, applied to core's own verb rather than only to a module's.
assert.equal(byId['core.lease'].enabled, false)
assert.equal(byId['core.lease'].changesWorld, true)
// core.wait is `inspect`, and it arrives ENABLED. Read §K's sentence literally
// and it would not, and every published event that waits would break on a fresh
// deployment (org lead, 2026-09-03).
@@ -843,6 +863,7 @@ test('the board never serves a callable', async () => {
for (const a of res.body.actions) {
assert.equal(a.perform, undefined)
assert.equal(a.revert, undefined)
assert.equal(a.reconcile, undefined)
assert.equal(a.cost, undefined)
}
})

View File

@@ -162,6 +162,11 @@ const SURFACE = [
// Phase 6's switchboard — configuration that can break things.
['GET', '/events/actions', ['admin']],
['PUT', '/events/actions', ['admin']],
// Phase 8's cleanup, and it sits in the ADMIN column rather than with the live
// controls it is rendered beside. Re-running a teardown is not incident
// response — it asks core to write to the world again, which §K puts in the
// same row as the world-changing actions themselves.
['POST', '/events/runs/1/cleanup', ['admin']],
// Live control of a run in flight: admin and moderator, deliberately WIDER
// than start.
@@ -214,6 +219,19 @@ test('an editor may price an event but not publish or start it', async () => {
assert.equal(await forbidden('POST', '/events/1/runs', 'editor'), true)
})
test('a moderator may stop a run but not re-run its cleanup', async () => {
// The same shape as start-and-stop above, one row further on, and held as its
// own claim for the same reason: the two controls sit next to each other on the
// run console and a later tidying pass that gave them one gate would have to
// delete an assertion that says why they do not share one.
//
// Cancelling is the 2am incident. Cleanup asks core to delete things in a live
// world, which is the narrower decision even though it is the tidier-sounding
// button.
assert.equal(await forbidden('POST', '/events/runs/1/cancel', 'moderator'), false)
assert.equal(await forbidden('POST', '/events/runs/1/cleanup', 'moderator'), true)
})
test('the switchboard is admin only in both directions', async () => {
// Reading which actions are enabled is as much `admin` as writing it: the board
// is the deployment's posture, and §K puts it in the same row as the actions it