feat(events): the resource ledger, leases and cleanup (Phase 8)
Event System Phase 8 (EVENTS_PLAN.md). Docs half: RunicGateway/docs#NNN. One table, one core action, one route, one body field, and two members added to MODULE_API 1.10.0 in place. The safety property the whole world-write half depends on: core now remembers what a run changed in the world, and gives it back on every terminal path. Four decisions settled by the org lead on 2026-09-03, all as recommended: - A lease is acquired by a new CORE action, `core.lease`. Section F puts the duration bound and the two-events-one-target conflict check on core's side of the seam, and a lease verb per module would be both re-implemented once per module, advisory everywhere. - Record-before-confirm is a PLACEHOLDER keyed by the step's idempotency key. A spawn's ref does not exist until the module answers, so what core writes before the dispatch is `kind: '@step'`, `ref` = that key. If the answer never comes it stands, and cleanup calls revert() with the key and no resources -- which is why section F's revert takes the key at all. - Cleanup is one sweep over the ledger, not synthetic step rows. The step-shaped version costs a second retry counter beside `revert_attempts`. - `reconcile` is declared here and TRIGGERED BY THE MODULE, through `ctx.events.reconcile()`. Core has no concept of the game being up, so it cannot decide when to ask; it asks once at its own boot. MODULE_API stays 1.10.0. A protocol owes a bump once it has landed on `main`; while it is on `edge` it is amended in place, so the whole module contract reaches an author as one version they read once. Verify - `npm test` -- 2025 tests, 1935 pass, 89 skipped, 1 fail. That one is the pre-existing engagementManifest CRLF failure, in a file this branch does not touch (`edge` before: 1950/1876/73/1). +75 tests. - The unique key was proved against a REAL MariaDB, because nothing else can prove it: whether multiple NULLs collide in a unique index, whether a STORED generated column is recomputed on UPDATE, and whether the SET NULL foreign key survives beside it are properties of the server. eventRunnerSql.test.js gained 16 tests; 65 pass against the container. The real schema.sql was applied to a fresh database and to an existing one. - Client: 362 pass, and it builds. routes:manifest and swagger -- one route added, none moved. The live walk found three defects, and two of them are the phase's real finding Driven by a throwaway `rig` module in website/modules/, deleted before commit. 1. A lease was never given back at all. `core.lease` reserves its own ledger row, so it never went through the ledger's dirty-marking, so a run holding only a lease kept `cleanup_status = 'not_required'` and the cleanup leg -- which selected on `pending` -- never looked at it. 2. EVENT_REVERT_MAX_ATTEMPTS meant one attempt, not three. The first failing sweep moved the run to `incomplete`, which took it out of the leg's own scan for ever. The test covering the bound asserted `<= 3` and was satisfied by 1: a bound has two halves, and a test that only asserts the ceiling passes against a floor. 3. The first fix for (2) made the console lie. Spending every row's `revert_attempts` was a tidy way to take a `cleanup: false` run out of a counter-bounded scan, and the run page then rendered "3 attempts" beside resources nothing had ever tried. Found by opening the page. Both (1) and (2) are the same mistake: deriving "is there anything to do" from a summary column instead of from the rows. Neither was visible to a unit test, because a test that calls the sweep directly never asks what would have selected the run. The two properties that need the process to die were walked as the plan asks. With the module's perform() hanging, the placeholder existed while the dispatch was in flight and nothing was named; after taskkill and a restart the reclaim re-dispatched the same idempotency key, the retry re-used its own placeholder, and everything was given back. Then, with the module reporting one of two resources as no longer in force, the boot-time reconcile marked the other `orphaned` -- never `reverted`. This branch does NOT bump MODULE_API_VERSION, so the integration kit stays as Phase 7 left it: red until the Phase 16 cutover re-pins ci/core-ref.json. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -10,7 +10,16 @@
|
||||
// 1. **reclaim** — release leases whose holder died, never touching `attempts`
|
||||
// 2. **materialise** — sweep occurrences past their grace window into `missed`
|
||||
// 3. **advance** — claim each due run and move it through its phases
|
||||
// 4. **prune** — the `event_run_log` retention sweep, on its own long clock
|
||||
// 4. **cleanup** — give back what a TERMINAL run still holds (Phase 8)
|
||||
// 5. **prune** — the `event_run_log` retention sweep, on its own long clock
|
||||
//
|
||||
// **Cleanup is a leg rather than a limb of `advanceRun`**, and its position in
|
||||
// that list is load-bearing: it runs after advance, so a run that completes in
|
||||
// one tick is torn down in the same one, and it is one place rather than four, so
|
||||
// a process that dies mid-teardown resumes on the next tick instead of leaving a
|
||||
// world half-restored with nothing scheduled to finish it. §L's "cleanup steps
|
||||
// are generated from the ledger and run" on cancellation and abort as well as on
|
||||
// completion is one query here rather than a hook on each of the three.
|
||||
//
|
||||
// **What a phase advances on, as of Phase 5.** Every step terminal, and — if the
|
||||
// phase authored one — its GATE open as well. The gate is an ADDITIONAL
|
||||
@@ -69,6 +78,8 @@ const gates = require('../events/gates')
|
||||
const spec = require('../events/spec')
|
||||
const registries = require('../modules/registries')
|
||||
const { dispatchStep } = require('../events/dispatch')
|
||||
const ledger = require('../events/ledger')
|
||||
const cleanup = require('../events/cleanup')
|
||||
const authorize = require('../events/authorize')
|
||||
const log = require('./logger')('event-runner')
|
||||
|
||||
@@ -261,6 +272,27 @@ async function drainStep(run, step, now, carry = {}) {
|
||||
}
|
||||
}
|
||||
|
||||
// ── Rule 1: record BEFORE the dispatch, not after ──
|
||||
//
|
||||
// §D. A step that ledgers gets a placeholder keyed by its idempotency key,
|
||||
// written before anything reaches the module, so an answer that never comes
|
||||
// back still leaves cleanup something to act on. Recording afterwards would
|
||||
// make every object whose acknowledgement was lost invisible for ever, which is
|
||||
// the one failure the whole world-write half cannot tolerate.
|
||||
//
|
||||
// It is deliberately AFTER the permission check: a refused step never reaches
|
||||
// the module, so it has created nothing and must ledger nothing.
|
||||
let placeholderId = null
|
||||
try {
|
||||
placeholderId = await ledger.reserveStep(run, step, action)
|
||||
} catch (err) {
|
||||
// The ledger is what makes a world write recoverable, so a step that cannot
|
||||
// be recorded must not be dispatched. Transient — the next attempt tries the
|
||||
// insert again — because the alternative is an unrecorded world change.
|
||||
log.error('could not reserve the ledger row', { run: run.id, step: step.id, message: err.message })
|
||||
return applyFailure(run, step, `the resource ledger could not record this step: ${err.message}`)
|
||||
}
|
||||
|
||||
const result = await dispatchStep(step, { run })
|
||||
|
||||
if (result.actionVersionDrift) {
|
||||
@@ -273,6 +305,36 @@ async function drainStep(run, step, now, carry = {}) {
|
||||
})
|
||||
}
|
||||
|
||||
// ── …and promote it on the answer ──
|
||||
//
|
||||
// On both success shapes, because `await: 'human'` is a SUCCESS: the module did
|
||||
// its part and something outside the system has to happen next, and a cue's
|
||||
// confirm finishes the step without a second dispatch — so this is the only
|
||||
// moment its resources can be recorded. A failure records nothing and leaves the
|
||||
// placeholder standing, which is the whole point of writing one.
|
||||
if (result.outcome === 'done' || result.outcome === 'parked') {
|
||||
const recorded = await ledger.recordAnswer({
|
||||
run,
|
||||
step,
|
||||
action,
|
||||
placeholderId,
|
||||
resources: result.resources,
|
||||
})
|
||||
if (recorded.recorded > 0 || recorded.rejected.length > 0) {
|
||||
await logDb.write({
|
||||
runId: run.id,
|
||||
stepId: step.id,
|
||||
kind: 'resource.recorded',
|
||||
phase: step.phase,
|
||||
detail: {
|
||||
action: step.action_id,
|
||||
recorded: recorded.recorded,
|
||||
...(recorded.rejected.length ? { rejected: recorded.rejected } : {}),
|
||||
},
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
if (result.outcome === 'parked') {
|
||||
// The GM cue. The step stays `running` with a NULL lease: genuinely in
|
||||
// flight, nothing holding it, so the stale reclaim passes it by and a cue
|
||||
@@ -459,9 +521,11 @@ async function advanceRun(run, now) {
|
||||
}
|
||||
|
||||
if (run.status === 'ending') {
|
||||
// A run that reached the wind-down and then lost its process. Phase 8 puts
|
||||
// cleanup here; until then `ending` is a state a run passes through rather
|
||||
// than one it does work in, and completing it is the whole recovery.
|
||||
// A run that reached the wind-down and then lost its process. Completing it is
|
||||
// still the whole recovery even with a ledger in the picture: cleanup is a leg
|
||||
// of the tick over TERMINAL runs, so the row this leaves behind is exactly
|
||||
// what that leg is looking for, and doing the teardown here as well would be
|
||||
// the second call site the leg exists to avoid.
|
||||
await runsDb.transition(run.id, 'ending', 'completed')
|
||||
await logDb.write({ runId: run.id, kind: 'run.status', detail: { from: 'ending', to: 'completed' } })
|
||||
return 'completed'
|
||||
@@ -536,8 +600,9 @@ async function advanceRun(run, now) {
|
||||
const next = phases[phaseIndex + 1]
|
||||
if (!next) {
|
||||
// §E's `ending` exists for the reason `sending` does in the outbox — it is
|
||||
// what a claim sets — so the run passes through it even though Phase 2 has
|
||||
// no cleanup to do there. Phase 8 is what gives it work.
|
||||
// what a claim sets. The run still passes straight through it: the ledger's
|
||||
// teardown is the tick's cleanup leg, which runs after this one in the same
|
||||
// tick and takes the run as it now is.
|
||||
if (!(await runsDb.transition(run.id, 'running', 'ending'))) return 'taken'
|
||||
await logDb.write({ runId: run.id, kind: 'run.status', phase: phaseKey, detail: { from: 'running', to: 'ending' } })
|
||||
await runsDb.transition(run.id, 'ending', 'completed')
|
||||
@@ -790,6 +855,17 @@ async function tick(now = new Date()) {
|
||||
}
|
||||
if (due && due.length) log.info('event runs swept', { due: due.length, ...counts })
|
||||
|
||||
// **After advance, deliberately.** A run that reached `completed` two lines ago
|
||||
// is torn down in this tick rather than the next, so §L's "a run reaches
|
||||
// `completed` with `cleanup_status = 'incomplete'`" is what an operator sees
|
||||
// instead of a completed run that briefly claims it has cleanup pending.
|
||||
try {
|
||||
const swept = await cleanup.sweep()
|
||||
if (swept) log.info('event cleanup swept', { runs: swept })
|
||||
} catch (err) {
|
||||
log.error('cleanup sweep failed', { message: err.message })
|
||||
}
|
||||
|
||||
try {
|
||||
await prune(now)
|
||||
} catch (err) {
|
||||
|
||||
Reference in New Issue
Block a user