feat(events): the resource ledger, leases and cleanup (Phase 8)
Some checks failed
PR Checks / client-build (pull_request) Successful in 3m15s
PR Checks / server-tests (pull_request) Failing after 8m21s
PR Checks / bot-tests (pull_request) Successful in 11m12s

Event System Phase 8 (EVENTS_PLAN.md). Docs half: RunicGateway/docs#NNN.

One table, one core action, one route, one body field, and two members added to
MODULE_API 1.10.0 in place. The safety property the whole world-write half
depends on: core now remembers what a run changed in the world, and gives it
back on every terminal path.

Four decisions settled by the org lead on 2026-09-03, all as recommended:

- A lease is acquired by a new CORE action, `core.lease`. Section F puts the
  duration bound and the two-events-one-target conflict check on core's side of
  the seam, and a lease verb per module would be both re-implemented once per
  module, advisory everywhere.
- Record-before-confirm is a PLACEHOLDER keyed by the step's idempotency key. A
  spawn's ref does not exist until the module answers, so what core writes
  before the dispatch is `kind: '@step'`, `ref` = that key. If the answer never
  comes it stands, and cleanup calls revert() with the key and no resources --
  which is why section F's revert takes the key at all.
- Cleanup is one sweep over the ledger, not synthetic step rows. The
  step-shaped version costs a second retry counter beside `revert_attempts`.
- `reconcile` is declared here and TRIGGERED BY THE MODULE, through
  `ctx.events.reconcile()`. Core has no concept of the game being up, so it
  cannot decide when to ask; it asks once at its own boot.

MODULE_API stays 1.10.0. A protocol owes a bump once it has landed on `main`;
while it is on `edge` it is amended in place, so the whole module contract
reaches an author as one version they read once.

Verify

- `npm test` -- 2025 tests, 1935 pass, 89 skipped, 1 fail. That one is the
  pre-existing engagementManifest CRLF failure, in a file this branch does not
  touch (`edge` before: 1950/1876/73/1). +75 tests.
- The unique key was proved against a REAL MariaDB, because nothing else can
  prove it: whether multiple NULLs collide in a unique index, whether a STORED
  generated column is recomputed on UPDATE, and whether the SET NULL foreign key
  survives beside it are properties of the server. eventRunnerSql.test.js gained
  16 tests; 65 pass against the container. The real schema.sql was applied to a
  fresh database and to an existing one.
- Client: 362 pass, and it builds. routes:manifest and swagger -- one route
  added, none moved.

The live walk found three defects, and two of them are the phase's real finding

Driven by a throwaway `rig` module in website/modules/, deleted before commit.

1. A lease was never given back at all. `core.lease` reserves its own ledger
   row, so it never went through the ledger's dirty-marking, so a run holding
   only a lease kept `cleanup_status = 'not_required'` and the cleanup leg --
   which selected on `pending` -- never looked at it.
2. EVENT_REVERT_MAX_ATTEMPTS meant one attempt, not three. The first failing
   sweep moved the run to `incomplete`, which took it out of the leg's own scan
   for ever. The test covering the bound asserted `<= 3` and was satisfied by 1:
   a bound has two halves, and a test that only asserts the ceiling passes
   against a floor.
3. The first fix for (2) made the console lie. Spending every row's
   `revert_attempts` was a tidy way to take a `cleanup: false` run out of a
   counter-bounded scan, and the run page then rendered "3 attempts" beside
   resources nothing had ever tried. Found by opening the page.

Both (1) and (2) are the same mistake: deriving "is there anything to do" from a
summary column instead of from the rows. Neither was visible to a unit test,
because a test that calls the sweep directly never asks what would have selected
the run.

The two properties that need the process to die were walked as the plan asks.
With the module's perform() hanging, the placeholder existed while the dispatch
was in flight and nothing was named; after taskkill and a restart the reclaim
re-dispatched the same idempotency key, the retry re-used its own placeholder,
and everything was given back. Then, with the module reporting one of two
resources as no longer in force, the boot-time reconcile marked the other
`orphaned` -- never `reverted`.

This branch does NOT bump MODULE_API_VERSION, so the integration kit stays as
Phase 7 left it: red until the Phase 16 cutover re-pins ci/core-ref.json.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2026-09-03 21:19:27 -05:00
parent 57d183e921
commit fdc118166c
29 changed files with 3928 additions and 72 deletions

View File

@@ -13,6 +13,15 @@
// a human to go and do something. A deployment with no game module installed has
// a working event system made of exactly these.
//
// **Phase 8 added a fourth, and it is the odd one out on purpose.** `core.lease`
// names no game noun either — it borrows a value some module declared — but
// unlike the other three it genuinely changes the world, so it is `risk: 'change'`
// and therefore default-off, admin-only and cap-checked like any module verb.
// It is CORE's rather than each module's because §F puts the duration bound and
// the two-events-one-target conflict check on core's side of the seam: a lease
// verb per module would be that bound re-implemented once per module, advisory
// everywhere, and wrong in the first one that forgot it.
//
// **Phase 2 gave all three real bodies**, and between them they exercise every
// shape §F's envelope can take: `core.announce` does work and finishes,
// `core.wait` finishes while deferring what follows it, and `core.cue` succeeds
@@ -24,10 +33,42 @@
// which runs under `routeManifest.js` and `swagger.js` against a dead pool
// (MODULE_API.md §2.2). Nothing below runs at require time; the announce leg is
// looked up inside `perform()`, per call, which is also what makes a leg
// registered by a module that booted later reachable at all.
// registered by a module that booted later reachable at all. `core.lease` is the
// one action here that reaches a table, and it requires the model INSIDE
// `perform()` for the same reason — a top-level require would make this file
// build a pool during route-manifest generation.
const registries = require('../modules/registries')
/**
* Turn the `value` param's text into whatever the named lease says it holds.
*
* The range check is here too, and it is REQUIRED on the numeric types for the
* reason §F gives: unlike a cap, a bad lease value is in force the moment it is
* applied, so "0.5 to 5" is not advice.
*/
function coerceLeaseValue(lease, raw) {
const text = String(raw === undefined || raw === null ? '' : raw).trim()
if (lease.type === 'string') return { ok: true, value: text }
if (lease.type === 'bool') {
if (['true', '1', 'yes', 'on'].includes(text.toLowerCase())) return { ok: true, value: true }
if (['false', '0', 'no', 'off'].includes(text.toLowerCase())) return { ok: true, value: false }
return { ok: false, error: `"${raw}" is not a yes or no value for ${lease.label}` }
}
const n = Number(text)
if (text === '' || !Number.isFinite(n)) {
return { ok: false, error: `"${raw}" is not a number, and ${lease.label} holds one` }
}
if (lease.type === 'int' && !Number.isInteger(n)) {
return { ok: false, error: `${lease.label} holds a whole number, and "${raw}" is not one` }
}
if (n < lease.min || n > lease.max) {
return { ok: false, error: `${lease.label} accepts ${lease.min} to ${lease.max}, and "${raw}" is outside that` }
}
return { ok: true, value: n }
}
const ACTIONS = [
{
id: 'core.announce',
@@ -214,6 +255,164 @@ const ACTIONS = [
return { ok: true, await: 'human' }
},
},
{
id: 'core.lease',
label: 'Borrow a value',
description:
'Hold a module-declared value at a new setting for a bounded time, and put the old one back at teardown.',
// The world changes and it changes back, so `change` rather than
// `irreversible` — and `change`'s default `on_failure` is `pause`, which is
// the right stop for a run that failed halfway through altering the world.
risk: 'change',
// The one action core ships in this class. `override` is what tells the
// cleanup sweep to restore through the LEASE registry rather than through an
// action's `revert()`, which is why this action needs no `revert()` of its own
// and why the registry refuses one on it.
reversible: 'override',
version: 1,
params: [
{
name: 'lease',
type: 'string',
required: true,
example: 'uo.rate.skillgain',
source: 'core.options.leases',
description: 'Which declared value to borrow.',
},
{
// **A string, and the coercion is here rather than in the type system.**
// A param declares ONE type; a lease declares its own, and they are four
// different ones. Typing this `float` would make a boolean lease
// unauthorable and a string lease nonsense, so the field takes text and
// this action turns it into whatever the named lease said it holds — the
// one place that knows both halves.
name: 'value',
type: 'string',
required: true,
example: '3.0',
description: 'What to hold it at, in whatever type the lease declares.',
},
{
name: 'minutes',
type: 'int',
required: true,
example: 120,
description: 'How long to hold it. Core refuses more than the lease allows.',
},
],
// What a lease costs is the LEASE's business to bound, not a budget's:
// `maxDurationMs` and the numeric range are declared beside the callables and
// enforced below. A cap dimension here would be core inventing an accounting
// unit for something a module already bounds — and `registerEventBudgets`
// refuses a dimension nobody declared, which is exactly the rule that would
// then bite core's own action.
/**
* Read the baseline, reserve the target, apply the value.
*
* **This is rule 1 in its strongest form.** Unlike a spawn, a lease's target
* is knowable before the dispatch — it is the lease id the step names — so
* the ledger row is written with its real `kind` and `ref` BEFORE anything
* touches the world, and the two-events-one-target refusal comes from the
* unique index at that moment rather than from a check that read and then
* wrote. A second run asking for a lease another run holds comes back
* `refused`, in the same words a cap breach uses and for the same reason:
* nothing is broken, the deployment already has that value spoken for.
*
* The order is read then reserve then apply, and a failure at each stage
* undoes the one before it: a reservation whose `apply` refuses is released
* here rather than left for the sweep, because there is nothing out there to
* give back and a shard that is merely down must not lock a lease out for the
* length of a retry cycle.
*/
async perform({ runId, stepId, params, verify }) {
// eslint-disable-next-line global-require
const resourcesDb = require('../model/events/eventRunResources.db')
const lease = registries.eventLease(params.lease)
if (!lease) {
return { ok: false, retry: false, error: `no module registers the lease "${params.lease}"` }
}
const coerced = coerceLeaseValue(lease, params.value)
if (!coerced.ok) return { ok: false, retry: false, error: coerced.error }
const minutes = Number(params.minutes)
if (!Number.isFinite(minutes) || minutes <= 0) {
return { ok: false, retry: false, error: `"${params.minutes}" is not a number of minutes` }
}
const ms = Math.round(minutes * 60_000)
if (ms > lease.maxDurationMs) {
return {
ok: false,
retry: false,
error: `${lease.label} may be held for at most ${Math.floor(lease.maxDurationMs / 60_000)} minutes, not ${minutes}`,
}
}
// **The dry run stops here, and it has still checked everything worth
// checking**: the lease exists, the value is in range and the duration is
// allowed. What it deliberately does not do is reserve the target — a
// verify that took a lease would be a dry run that changed something, and
// it would then refuse the real run that followed it.
if (verify) return { ok: true }
const baseline = await lease.read()
if (!baseline || baseline.ok !== true) {
return { ok: false, error: `could not read the current value of ${lease.label}` }
}
const until = new Date(Date.now() + ms)
const reserved = await resourcesDb.reserve({
runId,
stepId,
owner: lease.owner || 'core',
kind: 'override',
ref: lease.id,
payload: { target: lease.id, baseline: baseline.value, applied: coerced.value, until: until.toISOString() },
leaseUntil: until,
})
if (!reserved.ok) {
const heldBy = reserved.holder ? ` (run ${reserved.holder.run_id})` : ''
return {
ok: false,
retry: false,
error: `${lease.label} is already leased by another run${heldBy}`,
}
}
// **`until` goes down the wire** (§F). The module passes it to its sidecar
// and the game side restores baseline when it passes, without being asked
// again — the fail-safe that makes an unattended, scheduled world change
// defensible, because the worst case is a world back at baseline early
// rather than one stuck changed indefinitely.
let applied
try {
applied = await lease.apply(coerced.value, until)
} catch (err) {
applied = { ok: false, error: err.message }
}
if (!applied || applied.ok !== true) {
await resourcesDb.markReverted(reserved.id)
return { ok: false, error: applied && applied.error ? String(applied.error) : `${lease.label} refused the new value` }
}
await resourcesDb.confirm(reserved.id)
// **The run now owes the world something, and something has to say so.**
// The generic path marks a run dirty when it records a module's reported
// resources; this action reserves its own row and never goes through it, so
// a run whose only resource was a lease would have kept `cleanup_status =
// 'not_required'` and never been swept. Found by the live walk, and the
// cleanup leg's own scan was widened to make the class impossible rather
// than only this instance.
// eslint-disable-next-line global-require
await require('../events/ledger').markRunDirty(runId)
return { ok: true }
},
},
]
// ── Core's own param option sources (§F, Phase 7) ──────────────────
@@ -237,6 +436,16 @@ const OPTION_SOURCES = [
return registries.announceLegs().map((l) => ({ value: l.leg, label: l.label || l.leg }))
},
},
{
id: 'core.options.leases',
label: 'Borrowable values',
description: 'Every value a module has declared this deployment may lease.',
async resolve() {
return registries
.allEventLeases()
.map((l) => ({ value: l.id, label: l.label, group: l.id.split('.')[0] }))
},
},
]
module.exports = { ACTIONS, OPTION_SOURCES }

View File

@@ -0,0 +1,427 @@
// ── Giving back what a run took ────────────────────────────────────────────
//
// EVENTS.md §C ("Cleanup is generated, never authored"), §L and its two ledger
// rules, and Phase 8 of EVENTS_PLAN.md. `events/ledger.js` is the write half;
// this is the undo half, plus the reconcile that answers "is any of it still
// there?" after something outside core restarted.
//
// **Cleanup is derived from the ledger, never authored.** An operator cannot be
// relied on to write the undo, and an aborted run never reaches the phase they
// wrote it in — so there is no cleanup phase in a spec and no `on_teardown` on an
// action. There is one function, it reads rows, and it runs on EVERY terminal
// path: completion, cancellation and abort alike.
//
// **It is not built out of `event_run_steps` rows** (org lead, 2026-09-03). The
// plan's phrase is "cleanup steps are generated from the ledger", and the
// tempting reading is a synthetic phase of real step rows so the console's
// per-step retry comes free. It is the wrong shape here for one concrete reason:
// `event_run_resources` already carries `revert_attempts` and `last_error`, so
// synthetic steps would put a second retry counter beside the first and the two
// would disagree the first time a step reverted three of its four resources.
// The manual retry the API surface promises is a route over the ledger —
// `POST /admin/events/runs/:runId/cleanup` — rather than a step control.
//
// **Where it runs from.** One place: the runner's cleanup leg, which finds
// terminal runs that still owe the world something and works their rows. Hooking
// each terminal path instead would be four call sites, three of which are inside
// a request, and none of which would survive the process dying mid-cleanup. The
// leg is ordered AFTER advance in the tick, so a run that completes in one tick
// is cleaned in the same one.
//
// **The scan's WHERE clause cost two live-walk findings, in opposite
// directions.** A run whose only resource was a LEASE never went through
// `ledger.markRunDirty` — `core.lease` reserves its own row — so its
// `cleanup_status` stayed `not_required` and the lease was never given back at
// all. And a run whose first sweep failed was moved to `incomplete` by that very
// sweep, so it was never picked up again: `MAX_REVERT_ATTEMPTS` meant ONE attempt
// rather than three. The first is why `not_required` is in the scan; the second
// is why `incomplete` is written HERE only once nothing retryable is left.
//
// **Rule 2 is what the bounds are for.** A revert that never succeeds must stay
// visible rather than cycle: `MAX_REVERT_ATTEMPTS` stops the automatic retry, the
// run reaches `completed` with `cleanup_status = 'incomplete'`, and the rows stay
// on the console with their last error. Only a human's cleanup clears the
// counter — Engagement Phase 14's rule, whose defect was a sweep that reset every
// stale row and made the ceiling unreachable for ever.
const resourcesDb = require('../model/events/eventRunResources.db')
const runsDb = require('../model/events/eventRuns.db')
const logDb = require('../model/events/eventRunLog.db')
const stepsDb = require('../model/events/eventRunSteps.db')
const registries = require('../modules/registries')
const { withDeadline } = require('./dispatch')
const log = require('../utils/logger')('events')
// How many times the automatic sweep will ask before leaving a resource for a
// human. Three, like a step's, and for the same reason: a fourth attempt against
// a shard that has answered the same way three times is not new information.
const MAX_REVERT_ATTEMPTS = Number(process.env.EVENT_REVERT_MAX_ATTEMPTS) || 3
// The bound on one revert call, when the action that made the resource is gone
// and there is no `budgetMs` to read. A restore is a round trip like any other.
const DEFAULT_REVERT_BUDGET_MS = 10_000
// How many runs one cleanup leg looks at, and how many resource groups it works
// per run. Bounds rather than targets, exactly like `RUN_BATCH`: the tick runs
// again, and an unbounded teardown is how one run's bad night stalls every other.
const CLEANUP_RUN_BATCH = Number(process.env.EVENT_CLEANUP_RUN_BATCH) || 10
const CLEANUP_GROUPS_PER_RUN = Number(process.env.EVENT_CLEANUP_GROUPS_PER_RUN) || 25
/**
* Classify one revert answer, with `dispatch.classify`'s posture: no shape a
* failure can take may read as success.
*
* The extra value here is `drifted`. It is NOT an error — the module did exactly
* what it was asked and found somebody else's value in place — so it is a third
* outcome rather than a failure with a flag, and the row it produces is the one
* §L wants surfaced beside the unreverted ones.
*/
function classifyRevert(raw, what) {
if (raw && raw.__timedOut) return { outcome: 'retry', error: raw.error }
if (raw === null || typeof raw !== 'object' || Array.isArray(raw)) {
return { outcome: 'retry', error: `${what} answered with no envelope` }
}
if (raw.ok === true) {
// §L, and the Rust wipe: "gone, and that is fine" is a successful revert. A
// module never has to distinguish "I deleted it" from "it was not there".
return { outcome: 'done', failed: Array.isArray(raw.failed) ? raw.failed.map(String) : [] }
}
if (raw.drifted === true) {
return {
outcome: 'drifted',
error: `the value is now ${JSON.stringify(raw.current)} rather than what this run applied, so it was left alone`,
}
}
return {
outcome: raw.retry === false ? 'terminal' : 'retry',
error: raw.error ? String(raw.error) : `${what} refused`,
}
}
/** Call a lease's `restore`, under a deadline, never throwing. */
async function restoreLease(row) {
const lease = registries.eventLease(row.ref)
if (!lease) {
// The module that owned it is uninstalled or failed to boot. Not a failure to
// retry away — nothing will change until an operator reinstalls it — and not
// an orphan either, because core has no idea whether the value is still
// applied. It stays unresolved with the reason on it, which is exactly what
// `cleanup_status = 'incomplete'` is for.
return { outcome: 'terminal', error: `no module registers the lease "${row.ref}"` }
}
const payload = row.payload || {}
let raw
try {
raw = await withDeadline(
() => lease.restore(payload.baseline, { expected: payload.applied, runId: row.run_id }),
DEFAULT_REVERT_BUDGET_MS,
row.ref,
)
} catch (err) {
return { outcome: 'retry', error: err.message }
}
return classifyRevert(raw, row.ref)
}
/** Call an action's `revert` over a group of its rows, under a deadline, never throwing. */
async function revertGroup(actionId, rows, idempotencyKey) {
const action = registries.eventAction(actionId)
if (!action || typeof action.revert !== 'function') {
return {
outcome: 'terminal',
error: action
? `${actionId} declares no revert()`
: `no module registers "${actionId}", so its resources cannot be given back`,
}
}
const payload = rows
.filter((r) => r.kind !== resourcesDb.STEP_KIND)
.map((r) => ({ kind: r.kind, ref: r.ref, payload: r.payload || null, memberKey: r.member_key || null }))
let raw
try {
raw = await withDeadline(
() => action.revert({ runId: rows[0].run_id, resources: payload, idempotencyKey }),
action.budgetMs || DEFAULT_REVERT_BUDGET_MS,
actionId,
)
} catch (err) {
// A module should not throw from `revert` any more than from `perform`, and
// one that does has produced a transient failure rather than a crashed sweep.
log.warn('event revert threw', { action: actionId, message: err.message })
return { outcome: 'retry', error: err.message }
}
return classifyRevert(raw, actionId)
}
/**
* Work one run's ledger once.
*
* Answers what it found and what it managed, and sets `cleanup_status` from the
* rows that are left rather than from what it did — the two differ whenever
* another writer touched the run, and the rows are the truth.
*
* `resetAttempts` is the human's flag. It is never set by the automatic leg.
*/
async function cleanupRun(run, { resetAttempts = false, actor = null } = {}) {
const summary = { attempted: 0, reverted: 0, drifted: 0, failed: 0, remaining: 0 }
if (resetAttempts) {
const cleared = await resourcesDb.resetAttempts(run.id)
await runsDb.setCleanupStatus(run.id, 'pending')
if (cleared) {
await logDb.write({
runId: run.id,
kind: 'cleanup.retry',
detail: { resources: cleared, by: actor },
})
}
}
const rows = await resourcesDb.unresolvedForRun(run.id, {
maxAttempts: resetAttempts ? null : MAX_REVERT_ATTEMPTS,
})
// **Grouped by the step that made them**, because that is what names the verb:
// the resource row records the module and the opaque names, the step records
// the action, and `revert()` takes a LIST so one round trip can give back
// twelve creatures. A lease is its own group of one — core restores it through
// the lease registry rather than through any action, which is the split §F
// draws and the reason `core.lease` needs no `revert()` of its own.
const leases = rows.filter((r) => r.kind === 'override')
const byStep = new Map()
for (const row of rows) {
if (row.kind === 'override') continue
const key = row.step_id === null ? `orphan:${row.id}` : `step:${row.step_id}`
if (!byStep.has(key)) byStep.set(key, [])
byStep.get(key).push(row)
}
const groups = [...leases.map((r) => ({ lease: r })), ...[...byStep.values()].map((rs) => ({ rows: rs }))]
for (const group of groups.slice(0, CLEANUP_GROUPS_PER_RUN)) {
if (group.lease) {
const row = group.lease
if (!(await resourcesDb.claimRevert(row.id))) continue
summary.attempted += 1
const verdict = await restoreLease(row)
await applyVerdict(run, [row], verdict, summary, row.ref)
continue
}
const rs = group.rows
// The step is what names the action and carries the idempotency key. A row
// whose step was deleted keeps the action id in its own payload, which is why
// the placeholder writes one.
const step = rs[0].step_id === null ? null : await stepsDb.getById(rs[0].step_id)
const actionId = step?.action_id || rs[0].payload?.action || null
if (!actionId) {
await noteUnrevertable(run, rs, 'nothing records which action created this', summary)
continue
}
const claimed = []
for (const row of rs) if (await resourcesDb.claimRevert(row.id)) claimed.push(row)
if (!claimed.length) continue
summary.attempted += claimed.length
const verdict = await revertGroup(actionId, claimed, step?.idempotency_key || rs[0].ref)
await applyVerdict(run, claimed, verdict, summary, actionId)
}
summary.remaining = await resourcesDb.unresolvedCount(run.id)
// **`incomplete` means "finished with, and not finished"**, so it is written
// only once there is nothing left this sweep will try. Writing it after the
// FIRST failure — which is what the first draft did — took the run straight out
// of the leg's own scan, and `MAX_REVERT_ATTEMPTS` quietly meant one attempt
// rather than three. Found by the live walk, watching `revert_attempts` sit at
// 1 through half a minute of ticks.
const retryable = await resourcesDb.unresolvedForRun(run.id, { maxAttempts: MAX_REVERT_ATTEMPTS })
const status = summary.remaining === 0 ? 'complete' : retryable.length ? 'pending' : 'incomplete'
await runsDb.setCleanupStatus(run.id, status)
if (summary.attempted > 0) {
await logDb.write({
runId: run.id,
kind: 'cleanup.swept',
detail: { ...summary, by: actor },
})
}
return summary
}
/** Write one verdict across the rows it covers, and count it. */
async function applyVerdict(run, rows, verdict, summary, what) {
for (const row of rows) {
if (verdict.outcome === 'done' && !(verdict.failed || []).includes(row.ref)) {
await resourcesDb.markReverted(row.id)
summary.reverted += 1
continue
}
if (verdict.outcome === 'drifted') {
await resourcesDb.failRevert(row.id, verdict.error, 'drifted')
summary.drifted += 1
continue
}
const error =
verdict.outcome === 'done'
? `${what} could not give "${row.ref}" back`
: verdict.error
await resourcesDb.failRevert(row.id, error, 'confirmed')
summary.failed += 1
}
await logDb.write({
runId: run.id,
kind: verdict.outcome === 'done' ? 'cleanup.reverted' : 'cleanup.failed',
detail: {
what,
outcome: verdict.outcome,
resources: rows.map((r) => `${r.kind}:${r.ref}`),
...(verdict.error ? { error: verdict.error } : {}),
},
})
}
/** A group core cannot even name a verb for. Counted as failed, and said once. */
async function noteUnrevertable(run, rows, reason, summary) {
for (const row of rows) {
if (!(await resourcesDb.claimRevert(row.id))) continue
await resourcesDb.failRevert(row.id, reason, 'confirmed')
summary.attempted += 1
summary.failed += 1
}
await logDb.write({
runId: run.id,
kind: 'cleanup.failed',
detail: { what: null, outcome: 'terminal', resources: rows.map((r) => `${r.kind}:${r.ref}`), error: reason },
})
}
/**
* The cleanup leg of the tick: every TERMINAL run with something left to give
* back.
*
* Terminal only. A run still in flight has a ledger that is still growing, and
* reverting a resource the next step is about to use would be core undoing an
* event while it is happening.
*/
async function sweep() {
// The ceiling goes INTO the query, so a run whose rows are all spent is not
// selected, worked over and found to have nothing to do on every tick for the
// rest of its life. It is also what excludes a run an admin cancelled without
// cleanup, whose counters were spent deliberately.
const candidates = await resourcesDb.runsNeedingCleanup(CLEANUP_RUN_BATCH, MAX_REVERT_ATTEMPTS)
let swept = 0
for (const candidate of candidates) {
if (!runsDb.TERMINAL.includes(candidate.status)) continue
try {
await cleanupRun(candidate)
swept += 1
} catch (err) {
log.error('event cleanup failed', { run: candidate.id, message: err.message })
}
}
return swept
}
/**
* Ask one module which of its ledgered resources the game still has (§L, and
* §N7's "the shard stays stateless about events").
*
* **Core cannot know when to ask**, and that is not an omission: §F says core has
* no concept of the game being up, because a module with six sidecars cannot
* answer that question in the singular. So the module triggers this, through
* `ctx.events.reconcile()`, when it sees its own reconnect — module-uo already
* watches `bootId` for exactly that. Core also asks once at boot, for its own
* restart.
*
* **A resource the module no longer has becomes `orphaned`, never `reverted`.**
* Reverting it would be core recording that it put something back when what
* actually happened is that the thing vanished while nobody was looking, and the
* two are different sentences to the operator reading the console afterwards.
*
* A module with no `reconcile()` on the action is not broken: core keeps
* believing its own ledger, which is precisely the behaviour before this phase.
*/
async function reconcileModule(owner) {
const rows = await resourcesDb.liveForModule(owner)
const summary = { asked: 0, inForce: 0, orphaned: 0, unanswered: 0 }
if (!rows.length) return summary
const byStep = new Map()
for (const row of rows) {
if (row.kind === resourcesDb.STEP_KIND) continue // nothing to ask about yet
const key = row.step_id === null ? `orphan:${row.id}` : `step:${row.step_id}`
if (!byStep.has(key)) byStep.set(key, [])
byStep.get(key).push(row)
}
for (const group of byStep.values()) {
const step = group[0].step_id === null ? null : await stepsDb.getById(group[0].step_id)
const actionId = step?.action_id || group[0].payload?.action || null
const action = actionId ? registries.eventAction(actionId) : null
if (!action || typeof action.reconcile !== 'function') {
summary.unanswered += group.length
continue
}
summary.asked += group.length
let raw
try {
raw = await withDeadline(
() =>
action.reconcile({
runId: group[0].run_id,
resources: group.map((r) => ({ kind: r.kind, ref: r.ref, payload: r.payload || null })),
}),
action.budgetMs || DEFAULT_REVERT_BUDGET_MS,
actionId,
)
} catch (err) {
log.warn('event reconcile threw', { action: actionId, message: err.message })
raw = null
}
// Same posture as everywhere else: nothing that is not an explicit answer
// counts as one. A module that could not answer leaves the ledger alone,
// because "I do not know" must never be read as "it is gone".
if (!raw || raw.__timedOut || raw.ok !== true || !Array.isArray(raw.inForce)) {
summary.unanswered += group.length
continue
}
const held = new Set(raw.inForce.map(String))
for (const row of group) {
if (held.has(row.ref)) {
summary.inForce += 1
continue
}
await resourcesDb.markOrphaned(row.id, 'the module reports this is no longer in force')
summary.orphaned += 1
await logDb.write({
runId: row.run_id,
kind: 'resource.orphaned',
detail: { module: owner, resource: `${row.kind}:${row.ref}`, action: actionId },
})
}
}
return summary
}
/** Ask every module that owns a live row. Core's own boot-time sweep. */
async function reconcileAll() {
const owners = await resourcesDb.modulesWithLiveRows()
const out = {}
for (const owner of owners) {
try {
out[owner] = await reconcileModule(owner)
} catch (err) {
log.error('event reconcile failed', { module: owner, message: err.message })
}
}
return out
}
module.exports = {
MAX_REVERT_ATTEMPTS,
classifyRevert,
cleanupRun,
sweep,
reconcileModule,
reconcileAll,
}

221
server/src/events/ledger.js Normal file
View File

@@ -0,0 +1,221 @@
// ── Recording what a run changed in the world ──────────────────────────────
//
// EVENTS.md §D and §L, and Phase 8 of EVENTS_PLAN.md. The write half of the
// resource ledger; `events/cleanup.js` is the read-and-undo half.
//
// **Rule 1 is the whole reason this file is not two lines inside `drainStep`.**
// A resource is recorded BEFORE it is confirmed. The obstacle is that a spawn's
// serial does not exist until the module answers, so there is nothing to write a
// row about yet — which is why what goes in before the dispatch is a PLACEHOLDER
// keyed by the step's idempotency key rather than by the object:
//
// pre-dispatch INSERT pending { kind: '@step', ref: <idempotency key> }
// answer INSERT confirmed { kind: 'creature', ref: '0x40001234' } × n
// resolve the placeholder
// ack lost the placeholder is still `pending`
// cleanup revert({ idempotencyKey, resources: [] })
//
// That last line is why §F's `revert({ runId, resources, idempotencyKey })` takes
// the key at all. A module that half-ran and never answered is reachable by its
// key and by nothing else, and Phase 11's plugin-side key ledger is what makes
// answering it exact. Until then the contract is still honest, because §L
// requires reverting something that does not exist to be a SUCCESS.
//
// **Recording is idempotent, and the database is what makes it so.** A retry
// re-dispatches the same idempotency key, and a module that answers with the same
// resources twice must not produce two rows. `uq_evres_target` refuses the second
// insert, and this file reads that refusal as "already recorded" rather than as an
// error — the same posture `materialisePhase`'s INSERT IGNORE takes.
//
// **A lease does not use the placeholder.** Its target is knowable before the
// dispatch — it is the lease id the step names — so `core.lease` reserves the
// real row first, which is both a stronger form of rule 1 and the only place the
// two-events-one-target refusal can happen before the world has been written to.
const resourcesDb = require('../model/events/eventRunResources.db')
const runsDb = require('../model/events/eventRuns.db')
const registries = require('../modules/registries')
const log = require('../utils/logger')('events')
// Which reversible classes get a pre-dispatch placeholder. `none` is gone once
// done and `self` undoes itself, so neither has anything core could come back
// for; `override` reserves its own target instead (see the header). That leaves
// `ledger` — the class that declares `revert()`, which is exactly the class whose
// refs core cannot know until the module speaks.
const PLACEHOLDER_CLASSES = ['ledger']
// A resource `kind` may be anything a module likes except core's own reserved
// one. Bounded to the column, and refused rather than truncated: a truncated ref
// is a cleanup call naming the wrong object.
const MAX_KIND = 64
const MAX_REF = 190
const MAX_MEMBER_KEY = 190
/** Does this action produce anything core will have to come back for? */
const ledgers = (action) => action && (action.reversible === 'ledger' || action.reversible === 'override')
/**
* Turn one entry of a module's `resources` array into a row, or say why not.
*
* Every failure here is the module's mistake rather than the world's, so none of
* them is a retry: a badly shaped resource will be just as badly shaped on the
* second attempt. They are logged and dropped, and the step still counts as done
* — because it IS done; something happened in the world, and refusing to record
* it would be the one outcome worse than recording it imperfectly.
*/
function normalise(entry, actionId) {
if (!entry || typeof entry !== 'object' || Array.isArray(entry)) {
return { ok: false, reason: `${actionId} reported a resource that is not an object` }
}
const kind = String(entry.kind || '')
const ref = String(entry.ref === undefined || entry.ref === null ? '' : entry.ref)
if (!kind || kind.length > MAX_KIND) {
return { ok: false, reason: `${actionId} reported a resource with a bad kind "${entry.kind}"` }
}
if (kind === resourcesDb.STEP_KIND) {
// Core's own. A module that could write one would be a module that could make
// its own step's placeholder look resolved.
return { ok: false, reason: `${actionId} reported a resource of the reserved kind "${kind}"` }
}
if (!ref || ref.length > MAX_REF) {
return { ok: false, reason: `${actionId} reported a resource with a bad ref "${entry.ref}"` }
}
const memberKey = entry.memberKey === undefined || entry.memberKey === null ? null : String(entry.memberKey)
if (memberKey !== null && memberKey.length > MAX_MEMBER_KEY) {
return { ok: false, reason: `${actionId} reported a resource with an over-long memberKey` }
}
let leaseUntil = null
if (entry.until !== undefined && entry.until !== null) {
const at = new Date(entry.until)
if (Number.isNaN(at.getTime())) {
return { ok: false, reason: `${actionId} reported a resource with a bad until "${entry.until}"` }
}
leaseUntil = at
}
// A borrowed value must name a lease core knows how to give back. Core restores
// an `override` through the lease registry — that is the split §F draws — so a
// ref naming nothing registered is a resource core would be recording with no
// way to undo it, which is the promise rule 2 exists to stop core making.
if (kind === 'override' && !registries.eventLease(ref)) {
return { ok: false, reason: `${actionId} reported a lease "${ref}" no module registers` }
}
return {
ok: true,
row: {
kind,
ref,
payload: entry.payload === undefined ? null : entry.payload,
leaseUntil,
memberKey,
},
}
}
/**
* Write the pre-dispatch placeholder for a step that is about to change the
* world. Answers the row id, or null when this action ledgers nothing.
*
* **A duplicate is not a failure.** A retry re-uses its step's idempotency key, so
* the second attempt's placeholder collides with the first's — and finding it
* already there is the correct answer, not an error. The existing row is reused.
*/
async function reserveStep(run, step, action) {
if (!ledgers(action) || !PLACEHOLDER_CLASSES.includes(action.reversible)) return null
const owner = action.owner || 'core'
const reserved = await resourcesDb.reserve({
runId: run.id,
stepId: step.id,
owner,
kind: resourcesDb.STEP_KIND,
ref: step.idempotency_key,
payload: { action: action.id, phase: step.phase, seq: step.seq },
})
if (reserved.ok) {
await markRunDirty(run.id)
return reserved.id
}
// The only way a '@step' row collides is with this step's own earlier attempt,
// because an idempotency key is minted once per step and never varies by
// attempt (§E). Reuse it.
const existing = await resourcesDb.findByTarget(owner, resourcesDb.STEP_KIND, step.idempotency_key)
return existing ? existing.id : null
}
/**
* Record what a module said it made, and close out the placeholder.
*
* Answers `{ recorded, rejected }` — how many rows went in, and the reasons any
* entry was dropped. Never throws: a step that changed the world has changed it,
* and a ledger that threw would turn a bookkeeping problem into a failed step and
* then into a retry of a world write that already happened.
*/
async function recordAnswer({ run, step, action, placeholderId, resources }) {
const out = { recorded: 0, rejected: [] }
if (!ledgers(action)) return out
const owner = action.owner || 'core'
const list = Array.isArray(resources) ? resources : []
for (const entry of list) {
const parsed = normalise(entry, action.id)
if (!parsed.ok) {
out.rejected.push(parsed.reason)
log.warn('event resource rejected', { run: run.id, step: step.id, reason: parsed.reason })
continue
}
try {
const reserved = await resourcesDb.reserve({
runId: run.id,
stepId: step.id,
owner,
kind: parsed.row.kind,
ref: parsed.row.ref,
payload: parsed.row.payload,
leaseUntil: parsed.row.leaseUntil,
memberKey: parsed.row.memberKey,
})
if (!reserved.ok) {
// Already ledgered — by this step's own earlier attempt, or (a module bug
// rather than a race) by another run that still holds the same target.
// Either way there is a live row for it and a second would be the double
// cleanup the unique key exists to prevent.
if (reserved.holder && reserved.holder.run_id !== run.id) {
out.rejected.push(`${parsed.row.kind} "${parsed.row.ref}" is already held by run ${reserved.holder.run_id}`)
}
continue
}
await resourcesDb.confirm(reserved.id)
out.recorded += 1
} catch (err) {
// Bookkeeping must not become the step's control flow.
out.rejected.push(err.message)
log.error('event resource insert failed', { run: run.id, step: step.id, message: err.message })
}
}
if (out.recorded > 0) await markRunDirty(run.id)
// The placeholder's job is over the moment the real rows exist. It is resolved
// even when the module reported nothing at all — an action that ledgers and
// then answers `ok` with an empty list is saying "I made nothing", and holding
// its placeholder open would make cleanup call `revert()` for a step that has
// nothing to give back on every terminal path for ever.
if (placeholderId) await resourcesDb.resolvePlaceholder(placeholderId)
return out
}
/**
* There is now something to clean up. Idempotent and guarded, so it can never
* walk a run back from `complete` or `incomplete` to `pending` — only a human's
* cleanup does that, and it does it deliberately.
*/
async function markRunDirty(runId) {
await runsDb.setCleanupStatus(runId, 'pending', ['not_required'])
}
module.exports = { ledgers, normalise, reserveStep, recordAnswer, markRunDirty, PLACEHOLDER_CLASSES }

View File

@@ -15,10 +15,13 @@
// diagnosis panel: a screen that explains why a phase has not started, beside a
// control that does something about it.
//
// **One of §I's controls is still not here.** `cleanup` needs Phase 8's resource
// ledger; there is nothing to revert, so cancel takes `{ reason }` and gains
// `cleanup` when there is something for it to do. Absent rather than inert,
// which is the posture Phase 1 set and every phase since has kept.
// **`cleanup` is the eighth, and Phase 8 is what gave it a ledger to work over.**
// It re-runs the teardown across every resource a run has not given back, and it
// is `admin` where the other seven are `admin` + `moderator`: it is not incident
// response, it is asking core to write to the world again. Its partner is
// cancel's new `cleanup: false`, which is §L's "cancelling WITHOUT cleanup is a
// separate, logged, admin-only action" — deliberately the flag that has to be
// asked for, because the safe default is to give back what the run took.
//
// **Every control is guarded on the status it may act from, and the guard is a
// WHERE clause rather than a read-then-write.** A run console rendered thirty
@@ -37,6 +40,7 @@ const runsDb = require('./eventRuns.db')
const stepsDb = require('./eventRunSteps.db')
const logDb = require('./eventRunLog.db')
const gatesDb = require('./eventPhaseGates.db')
const resourcesDb = require('./eventRunResources.db')
const gates = require('../../events/gates')
const MAX_REASON = 500
@@ -152,11 +156,23 @@ async function resume(runId, options = {}, userId = null) {
* and a second writer on that row would race the process that owns it. It
* finishes into a cancelled run, which is honest.
*/
async function cancel(runId, { reason } = {}, userId = null) {
async function cancel(runId, { reason, cleanup = true } = {}, userId = null, { isAdmin = true } = {}) {
const run = await loadRun(runId)
if (!run) return { ok: false, status: 404, errors: ['no such run'] }
if (runsDb.TERMINAL.includes(run.status)) return conflict(`this run is already ${run.status}`)
// §L: cancelling WITHOUT cleanup is a separate, logged, ADMIN-only action. The
// route itself is `admin` + `moderator`, so the narrower gate cannot live in
// middleware — which of the two you have to be depends on what is in the body,
// exactly as the authoring role floor does (§K).
if (cleanup === false && !isAdmin) {
return {
ok: false,
status: 403,
errors: ['leaving a run\'s world changes in place is an administrator\'s decision'],
}
}
const note = clean(reason)
const from = ['scheduled', 'starting', 'running', 'paused', 'ending']
if (!(await runsDb.transition(run.id, from, 'cancelled', { error: note || 'cancelled by staff' }))) {
@@ -168,9 +184,83 @@ async function cancel(runId, { reason } = {}, userId = null) {
runId: run.id,
kind: 'run.status',
phase: run.current_phase,
detail: { from: run.status, to: 'cancelled', control: 'cancel', by: userId, reason: note, cancelledSteps: closed },
detail: {
from: run.status,
to: 'cancelled',
control: 'cancel',
by: userId,
reason: note,
cancelledSteps: closed,
cleanup: cleanup !== false,
},
})
return { ok: true, run: await runsDb.getById(run.id), cancelledSteps: closed }
// **The teardown is not done here, and the request does not wait for it.**
// Cleanup is one leg of the runner's tick over terminal runs (§L), which is
// what makes it survive a process that dies halfway through it — and a cancel
// pressed at two in the morning must answer at once rather than after a dozen
// round trips to a shard that may be the reason it is being cancelled. The run
// is terminal the moment this returns, so the very next tick picks its ledger
// up.
//
// `cleanup: false` is the operator saying leave it. The resources stay
// unresolved and the run carries `incomplete`, which is the truthful value: the
// world changes are still up, they are listed on the console, and the log line
// above records who decided that.
let cleanupStatus = run.cleanup_status
if (cleanup === false && (await resourcesDb.unresolvedCount(run.id)) > 0) {
// `incomplete` is what takes the run out of the cleanup leg's scan, and it is
// the truthful value: the world changes are still up, they are listed on the
// console, and the log line above records who decided that.
//
// **The first draft spent every row's `revert_attempts` instead**, to stop the
// sweep by the same mechanism a failed retry does. It worked and it made the
// console lie: the run page rendered "3 attempts" beside resources nothing had
// ever tried, which reads as "core tried three times and could not". Found by
// opening the page. A counter that means two things is a counter a screen
// cannot render.
await runsDb.setCleanupStatus(run.id, 'incomplete')
cleanupStatus = 'incomplete'
}
return {
ok: true,
run: await runsDb.getById(run.id),
cancelledSteps: closed,
cleanup: cleanup !== false,
cleanupStatus,
}
}
/**
* Re-run cleanup over everything a run has not given back.
*
* The manual retry §L promises, and the only thing that clears
* `revert_attempts`. That licence is the same one a human's step retry has, and
* it is deliberately not extended to the automatic sweep: Engagement Phase 14's
* defect was exactly a sweep that reset every stale row, which made the attempt
* ceiling unreachable and left the row cycling for ever.
*
* Legal on a TERMINAL run only. A run still in flight has a ledger that is still
* growing, and reverting a resource the next step is about to use would be core
* undoing an event while it is happening.
*/
async function cleanupRun(runId, userId = null) {
const run = await loadRun(runId)
if (!run) return { ok: false, status: 404, errors: ['no such run'] }
if (!runsDb.TERMINAL.includes(run.status)) {
return conflict(`this run is still ${run.status}; cancel it before cleaning up after it`)
}
if (run.cleanup_status === 'not_required') {
return conflict('this run recorded no resources, so there is nothing to give back')
}
// eslint-disable-next-line global-require
const summary = await require('../../events/cleanup').cleanupRun(run, {
resetAttempts: true,
actor: userId,
})
return { ok: true, run: await runsDb.getById(run.id), summary }
}
/**
@@ -359,4 +449,4 @@ async function retryStep(runId, stepId, options = {}, userId = null) {
}
}
module.exports = { pause, resume, cancel, advancePhase, confirmStep, skipStep, retryStep }
module.exports = { pause, resume, cancel, cleanupRun, advancePhase, confirmStep, skipStep, retryStep }

View File

@@ -47,6 +47,16 @@ const KINDS = [
'run.budget', // the caps this run was seeded with, and which switch set each
'step.refused', // a step was not permitted: disabled, or over a cap
'version.verified', // a dry run passed against a version, unlocking scheduled starts
// Phase 8's six, and every one of them is an answer to "what did this event
// leave behind". `resource.recorded` is written at the ANSWER rather than at
// the placeholder, because a placeholder is a promise and the operator's
// question is about the world.
'resource.recorded', // a step reported what it created or borrowed, and it is ledgered
'resource.orphaned', // a module reports a ledgered resource is no longer in force
'cleanup.reverted', // a group of resources came back
'cleanup.failed', // a group did not, with the reason and how it was left
'cleanup.swept', // one pass over a run's ledger, and what it found
'cleanup.retry', // a human cleared the attempt counter and asked again
]
const hydrate = (row) => row && { ...row, detail: parseJson(row.detail, null) }

View File

@@ -0,0 +1,353 @@
// ── event_run_resources — SQL only ─────────────────────────────────────────
//
// EVENTS.md §D and §L ("The ledger's two rules"), and Phase 8 of EVENTS_PLAN.md.
// Everything one run created or leased, and what became of it.
//
// **Rule 1 lives in `reserve()`.** A resource is recorded BEFORE it is
// confirmed, so the placeholder this writes is the row that exists while the
// dispatch is in flight — and the row that SURVIVES when the acknowledgement is
// lost. Recording on the answer instead would make every object whose ack went
// missing invisible to cleanup for ever.
//
// **Rule 2 lives in the status column and in `failRevert()`.** A revert that
// never succeeds leaves its row unreverted, with the error on it, and the run
// completes with `cleanup_status = 'incomplete'` rather than being held open.
// Loud and sticky.
//
// **The unique key is enforced by the database, not by a read.** `reserve()`
// answers `{ ok: false, code: 'held' }` on a duplicate key rather than checking
// first and then inserting — two runs entering the same tick would both pass the
// check. It is the argument `event_run_budget.spend()` makes about the cap and
// `runsDb.transition` makes about a status, in the third place it applies.
const { query } = require('../../utils/db')
const { parseJson } = require('./eventJson')
// The one `kind` core owns. A module's kinds are opaque and stored verbatim; this
// one is core's own, and `registries` refuses a module resource that claims it.
const STEP_KIND = '@step'
// The statuses that mean "core still believes this resource is this run's". They
// are exactly the ones the `live_marker` generated column keeps non-NULL, so the
// unique target key holds while a row is in one of them and releases when it
// leaves. Duplicated here as a JavaScript list because the sweeps read by it too,
// and a second copy that can drift is better than a query that cannot express it.
const HELD = ['pending', 'confirmed', 'reverting']
// Every status that still wants a human or a retry: `HELD` plus the two that mean
// "we let go, and not cleanly". This is what "unreverted" means everywhere in
// this feature — the console's list, `cleanup_status`, and the manual retry.
const UNRESOLVED = [...HELD, 'orphaned', 'drifted']
const COLUMNS = `id, run_id, step_id, owner_module, kind, ref, payload, lease_until,
status, revert_attempts, last_error, member_key, created_at, updated_at`
// `payload` is opaque to core and stored verbatim, but it comes back as a string
// from the driver and every caller wants the object — the cleanup sweep reads a
// lease's baseline out of it, and the console renders it. Hydrated here for the
// same reason a step's params are: one place rather than at each read.
const hydrate = (row) => row && { ...row, payload: parseJson(row.payload, null) }
/**
* Record a resource that does not exist yet.
*
* Answers `{ ok: true, id }`, or `{ ok: false, code: 'held', holder }` when the
* target is already someone's — which is the lease conflict, surfaced as a
* refusal rather than a failure because nothing is wrong with the system: another
* run has the thing.
*
* **`ER_DUP_ENTRY` is the check.** The holder is looked up only to name it in the
* refusal, and only after the insert has already lost the race.
*/
async function reserve({ runId, stepId = null, owner, kind, ref, payload = null, leaseUntil = null, memberKey = null }) {
try {
const result = await query(
`INSERT INTO event_run_resources
(run_id, step_id, owner_module, kind, ref, payload, lease_until, member_key, status)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, 'pending')`,
[runId, stepId, owner, kind, ref, payload === null ? null : JSON.stringify(payload), leaseUntil, memberKey],
)
return { ok: true, id: Number(result.insertId) }
} catch (err) {
if (err && (err.code === 'ER_DUP_ENTRY' || err.errno === 1062)) {
const [holder] = await query(
`SELECT run_id, status FROM event_run_resources
WHERE owner_module = ? AND kind = ? AND ref = ? AND status IN (?, ?, ?)
LIMIT 1`,
[owner, kind, ref, ...HELD],
)
return { ok: false, code: 'held', holder: holder || null }
}
throw err
}
}
/**
* Promote a reserved row to `confirmed`, optionally attaching what the module
* finally said about it.
*
* Guarded on `pending` so a late answer cannot un-revert a row cleanup has
* already dealt with — the same reason every other write in this feature is a
* compare-and-set rather than a read-then-write.
*/
async function confirm(id, { payload, leaseUntil, memberKey } = {}) {
const sets = ["status = 'confirmed'"]
const params = []
if (payload !== undefined) {
sets.push('payload = ?')
params.push(payload === null ? null : JSON.stringify(payload))
}
if (leaseUntil !== undefined) {
sets.push('lease_until = ?')
params.push(leaseUntil)
}
if (memberKey !== undefined) {
sets.push('member_key = ?')
params.push(memberKey)
}
const result = await query(
`UPDATE event_run_resources SET ${sets.join(', ')} WHERE id = ? AND status = 'pending'`,
[...params, id],
)
return (result.affectedRows || 0) > 0
}
/**
* Resolve a step placeholder once the module has named what it actually made.
*
* The placeholder's whole job is over at this point: the real rows exist, so the
* `@step` row must stop being one of the things cleanup will try to revert.
* `reverted` is the honest terminal state for it — there is nothing left to undo
* that the rows it stood in for do not now cover — and it releases the
* idempotency key for a later run, which matters because keys are per step and a
* re-materialised step reuses its own.
*/
async function resolvePlaceholder(id) {
const result = await query(
`UPDATE event_run_resources
SET status = 'reverted', last_error = NULL
WHERE id = ? AND kind = ? AND status IN ('pending', 'confirmed')`,
[id, STEP_KIND],
)
return (result.affectedRows || 0) > 0
}
/**
* One row by its target, live or not — how a caller that lost the insert race
* finds the row it meant to write. Newest first, so a target that has been held
* and released several times answers with the current holder.
*/
async function findByTarget(owner, kind, ref) {
const [row] = await query(
`SELECT ${COLUMNS} FROM event_run_resources
WHERE owner_module = ? AND kind = ? AND ref = ?
ORDER BY id DESC LIMIT 1`,
[owner, kind, ref],
)
return hydrate(row) || null
}
/** One run's whole ledger, oldest first — the console's read. */
async function forRun(runId) {
const rows = await query(
`SELECT ${COLUMNS} FROM event_run_resources WHERE run_id = ? ORDER BY id`,
[runId],
)
return rows.map(hydrate)
}
/** The rows of one run that still want something: the cleanup sweep's input. */
async function unresolvedForRun(runId, { maxAttempts = null } = {}) {
const params = [runId, ...UNRESOLVED]
const attemptClause = maxAttempts === null ? '' : ' AND revert_attempts < ?'
if (maxAttempts !== null) params.push(maxAttempts)
const rows = await query(
`SELECT ${COLUMNS} FROM event_run_resources
WHERE run_id = ? AND status IN (?, ?, ?, ?, ?)${attemptClause}
ORDER BY id`,
params,
)
return rows.map(hydrate)
}
/** How many of one run's rows are still unresolved — what `cleanup_status` is derived from. */
async function unresolvedCount(runId) {
const [row] = await query(
`SELECT COUNT(*) AS n FROM event_run_resources
WHERE run_id = ? AND status IN (?, ?, ?, ?, ?)`,
[runId, ...UNRESOLVED],
)
return Number(row?.n || 0)
}
/** Unresolved counts for several runs at once, keyed by run id — the run LIST's read. */
async function unresolvedCounts(runIds) {
const ids = [...new Set(runIds || [])].filter(Boolean)
if (!ids.length) return new Map()
const rows = await query(
`SELECT run_id, COUNT(*) AS n FROM event_run_resources
WHERE run_id IN (${ids.map(() => '?').join(',')}) AND status IN (?, ?, ?, ?, ?)
GROUP BY run_id`,
[...ids, ...UNRESOLVED],
)
return new Map(rows.map((r) => [r.run_id, Number(r.n)]))
}
/**
* Claim one row for a revert: `pending | confirmed | orphaned | drifted → reverting`.
*
* The compare-and-set that keeps the cleanup leg and the manual cleanup route off
* each other's rows. `reverting` is deliberately not claimable — a row another
* pass is mid-revert on is left alone, exactly as a step with a live claim is.
*/
async function claimRevert(id) {
const result = await query(
`UPDATE event_run_resources
SET status = 'reverting'
WHERE id = ? AND status IN ('pending', 'confirmed', 'orphaned', 'drifted')`,
[id],
)
return (result.affectedRows || 0) > 0
}
/** The revert worked. `reverted` is terminal and releases the target. */
async function markReverted(id) {
await query(
`UPDATE event_run_resources SET status = 'reverted', last_error = NULL WHERE id = ?`,
[id],
)
}
/**
* The revert did not work, and the row goes back to being unresolved.
*
* `revert_attempts` is incremented here and NOWHERE else, and it is never reset by
* a sweep — Engagement Phase 14's rule, whose defect was a reclaim that returned
* every stale row to its start state and made the attempt ceiling unreachable, so
* the row cycled for ever and was never eligible for any retention sweep. The one
* thing that may reset it is a human pressing cleanup, which is the same licence
* a human's step retry has.
*
* `restoreTo` is where the row lands: `drifted` when the module says somebody else
* moved the value, `orphaned` when it says the thing is gone, and `confirmed`
* otherwise — still ours, still out there, try again.
*/
async function failRevert(id, error, restoreTo = 'confirmed') {
await query(
`UPDATE event_run_resources
SET status = ?, revert_attempts = revert_attempts + 1, last_error = ?
WHERE id = ?`,
[restoreTo, String(error || 'the revert did not answer').slice(0, 500), id],
)
}
/**
* A human is trying again: clear the attempt counter on one run's unresolved rows.
*
* Only ever called from the cleanup route with an actor behind it. The automatic
* leg must never do this (see `failRevert`).
*/
async function resetAttempts(runId) {
const result = await query(
`UPDATE event_run_resources
SET revert_attempts = 0
WHERE run_id = ? AND status IN (?, ?, ?, ?, ?)`,
[runId, ...UNRESOLVED],
)
return result.affectedRows || 0
}
/** Every live row one module owns, for the reconcile sweep. */
async function liveForModule(owner, { limit = 500 } = {}) {
const rows = await query(
`SELECT ${COLUMNS} FROM event_run_resources
WHERE owner_module = ? AND status IN ('pending', 'confirmed')
ORDER BY id LIMIT ?`,
[owner, Number(limit)],
)
return rows.map(hydrate)
}
/** Every module that currently owns a live row — who the reconcile sweep asks. */
async function modulesWithLiveRows() {
const rows = await query(
`SELECT DISTINCT owner_module FROM event_run_resources
WHERE status IN ('pending', 'confirmed')`,
)
return rows.map((r) => r.owner_module)
}
/**
* The game no longer has it. Never reached by a revert — a revert that finds
* nothing there is a SUCCESS (§L, and it is what a Rust wipe needs) — only by
* reconcile, which is a different question: nobody asked for this to go.
*/
async function markOrphaned(id, detail = null) {
await query(
`UPDATE event_run_resources
SET status = 'orphaned', last_error = ?
WHERE id = ? AND status IN ('pending', 'confirmed', 'reverting')`,
[detail === null ? null : String(detail).slice(0, 500), id],
)
}
/**
* Terminal runs that still owe the world something — the cleanup leg's scan.
*
* **Both halves of the WHERE were live-walk findings, and they are opposite
* mistakes.**
*
* `cleanup_status = 'pending'` alone missed a run whose only resource was a
* LEASE: `core.lease` reserves its own row and never goes through the ledger's
* `markRunDirty`, so the flag stayed `not_required` and the lease was never given
* back at all. Hence `not_required` is in the list — a terminal run with an
* unresolved row has something to do whatever any summary column says, and
* treating that combination as work is the fail-safe direction.
*
* And the run status filter alone made `MAX_REVERT_ATTEMPTS` mean ONE attempt,
* because the first failing sweep set `incomplete` and nothing looked at the run
* again. That is fixed in `cleanupRun`, which now only writes `incomplete` once
* there is nothing left it will try — so `incomplete` genuinely means "finished
* with, and not finished", which is exactly what excludes both a run whose
* retries are spent and a run an admin cancelled without cleanup.
*
* The attempt bound is in the join for a different reason: without it a run whose
* rows are all spent would be selected, worked over and found to have nothing to
* do on every tick for the rest of its life.
*/
async function runsNeedingCleanup(limit = 25, maxAttempts = 3) {
return query(
`SELECT DISTINCT r.id, r.status, r.cleanup_status, r.version_id, r.definition_id, r.scope
FROM event_runs r
JOIN event_run_resources res ON res.run_id = r.id
WHERE r.status IN ('completed', 'cancelled', 'failed', 'missed')
AND r.cleanup_status IN ('pending', 'not_required')
AND res.status IN (?, ?, ?, ?, ?)
AND res.revert_attempts < ?
ORDER BY r.id
LIMIT ?`,
[...UNRESOLVED, Number(maxAttempts), Number(limit)],
)
}
module.exports = {
STEP_KIND,
HELD,
UNRESOLVED,
reserve,
confirm,
resolvePlaceholder,
findByTarget,
forRun,
unresolvedForRun,
unresolvedCount,
unresolvedCounts,
claimRevert,
markReverted,
failRevert,
resetAttempts,
liveForModule,
modulesWithLiveRows,
markOrphaned,
runsNeedingCleanup,
}

View File

@@ -386,6 +386,32 @@ async function setHealth(id, health) {
return Number(result?.affectedRows || 0) === 1
}
/**
* Set `cleanup_status`, optionally guarded on where it is now (Phase 8).
*
* Four values and three writers, which is why the guard is a parameter rather
* than baked in. The ledger stamps `pending` the first time a run records
* anything, and it must do so only over `not_required` — a run already marked
* `complete` must not be walked back to `pending` by a late resource, and a
* `incomplete` one must not be silently tidied. The cleanup sweep sets `complete`
* or `incomplete` from what it found, unguarded, because the sweep IS the
* authority on that. A human's cleanup route re-opens `pending` deliberately, and
* says so in the log with the actor.
*
* **`pending` on a run that is still running is not a bug and reads correctly**:
* there is something to clean up and it has not happened yet. The alternative -
* a fifth value meaning "there will be something later" - is a state nothing
* would ever branch on.
*/
async function setCleanupStatus(id, to, from = null) {
const guard = from === null ? '' : ` AND cleanup_status IN (${from.map(() => '?').join(',')})`
const result = await query(
`UPDATE event_runs SET cleanup_status = ? WHERE id = ?${guard}`,
from === null ? [to, id] : [to, id, ...from],
)
return Number(result?.affectedRows || 0) === 1
}
/**
* Runs whose start instant passed more than their own grace window ago (§E, §L).
*
@@ -488,6 +514,7 @@ module.exports = {
statusOf,
transition,
setHealth,
setCleanupStatus,
concurrencyHolder,
reclaimStale,
terminalBefore,

View File

@@ -27,6 +27,7 @@ const definitionsDb = require('./eventDefinitions.db')
const versionsDb = require('./eventVersions.db')
const settingsDb = require('./eventActionSettings.db')
const budgetDb = require('./eventRunBudget.db')
const resourcesDb = require('./eventRunResources.db')
const authorize = require('../../events/authorize')
const MAX_SCOPE = 190
@@ -204,11 +205,12 @@ async function create(
async function detail(runId) {
const run = await db.getById(runId)
if (!run) return null
const [steps, counts, gateRows, budget] = await Promise.all([
const [steps, counts, gateRows, budget, resources] = await Promise.all([
stepsDb.listForRun(runId),
stepsDb.statusCounts(runId),
gatesDb.listForRun(runId),
budgetDb.forRun(runId),
resourcesDb.forRun(runId),
])
const now = new Date()
return {
@@ -226,6 +228,37 @@ async function detail(runId) {
cap: b.cap,
from: b.effective_from,
})),
// What this run changed in the world, and what became of it (Phase 8). The
// WHOLE ledger, reverted rows included, because "what did last night's
// invasion actually spawn, and did all of it come back" is the question this
// panel exists for and a list of only the failures cannot answer the second
// half of it.
//
// **The `@step` placeholders are filtered out.** They are core's own
// bookkeeping — a row that says "a dispatch is in flight and may have made
// something" — and the console's list is of things in the world. One left in
// would read as a resource nobody can name, which is exactly the confusion it
// exists to prevent internally.
resources: resources
.filter((r) => r.kind !== resourcesDb.STEP_KIND)
.map((r) => ({
id: r.id,
stepId: r.step_id,
module: r.owner_module,
kind: r.kind,
ref: r.ref,
payload: r.payload,
leaseUntil: r.lease_until,
status: r.status,
revertAttempts: r.revert_attempts,
lastError: r.last_error,
memberKey: r.member_key,
createdAt: r.created_at,
})),
// How many rows are still unresolved, counted over the WHOLE ledger rather
// than over the list above — a placeholder left standing by a lost
// acknowledgement is exactly the case `cleanup_status` must not call clean.
unresolvedResources: resources.filter((r) => resourcesDb.UNRESOLVED.includes(r.status)).length,
}
}

View File

@@ -228,6 +228,39 @@ function buildCtx(id, moduleRoot) {
emit: (triggerId, envelope) => {
engagementEmit.emit(id, triggerId, envelope)
},
// EVENTS.md §L, and the resource ledger (Phase 8). "On reconnect the runner
// asks each ledgered resource's module to reconcile" — and this is how the
// runner learns there has BEEN a reconnect.
//
// **Core cannot decide when to call this, and that is the contract rather
// than a gap.** §F: core has no concept of the game being up, because a
// module with six sidecars cannot answer that question in the singular. So
// the module says so, when it sees its own — module-uo already watches
// `bootId` to tell a shard restart from a sidecar reconnect, which is
// exactly the moment a ledger of live spawns has become a claim about a
// world that no longer exists.
//
// `id` is bound here and never taken from the arguments, like `emit` and
// `teams.activity.push` before it: a module reconciles its OWN ledger, and
// without the binding this would be a way to have core mark another
// module's resources orphaned.
//
// Fire-and-forget and returns undefined, for the third time and the same
// reason: this is called from inside a connection handler, and there is
// nothing a module could correctly do with a failure of core's bookkeeping.
reconcile: () => {
// eslint-disable-next-line global-require
require('../events/cleanup')
.reconcileModule(id)
.then(
(summary) => {
if (summary && summary.orphaned) {
log.warn('event resources orphaned on reconcile', { module: id, ...summary })
}
},
(err) => { log.error('ctx.events.reconcile failed', { module: id, message: err.message }) },
)
},
},
// The in-app sink (§5.1) — a module writing the inbox directly, without a
// rule. Live from Phase 7; it threw until the `user_notifications` table

View File

@@ -157,11 +157,11 @@ const eventBudgets = new Map()
// lease id → { owner, id, label, type, min, max, maxDurationMs, description,
// read, apply, restore } (§F "Leases: one more declaration", Phase 7).
//
// **Phase 7 registers a lease and nothing acquires one.** Core owns the duration
// and the conflict check, the module owns reading the current value and writing a
// new one — and both halves of that live in the resource ledger, which is Phase
// 8's. What is here is the declaration, its validation and its catalog entry, so
// that the module contract is one version rather than two.
// **Core owns the duration and the conflict check; the module owns reading the
// current value and writing a new one.** Phase 7 registered a lease and nothing
// acquired one; Phase 8 gave it a verb — `core.lease`, a CORE action, so the
// bound and the two-events-one-target refusal are enforced in one place rather
// than re-implemented by every module that ships a lease.
const eventLeases = new Map()
// source id → { owner, id, label, description, resolve } (§F "Param option
@@ -435,14 +435,14 @@ async function resolveAudience(id, params = {}) {
/**
* Every declaration WITHOUT its callables — what the admin catalog serves.
*
* `perform`, `revert` and `cost` are stripped for the same reason `resolve` is
* `perform`, `revert`, `reconcile` and `cost` are stripped for the same reason `resolve` is
* stripped from an audience and `handler` from a slash command: this is the
* object that leaves the process, and the browser's whole relationship with an
* action is naming one by id. §F's "a module registers actions server-side and
* adds no routes for them" is only true if the functions never ride out.
*/
const allEventActions = () =>
[...eventActions.values()].map(({ perform, revert, cost, ...rest }) => rest)
[...eventActions.values()].map(({ perform, revert, reconcile, cost, ...rest }) => rest)
/** One declaration, callables included. The runner's lookup (Phase 2). */
const eventAction = (id) => eventActions.get(id) || null
@@ -485,7 +485,7 @@ const isEventBudget = (id) => eventBudgets.has(id)
const allEventLeases = () =>
[...eventLeases.values()].map(({ read, apply: applyValue, restore, ...rest }) => rest)
/** One lease, callables included. Phase 8's lookup; nothing calls it yet. */
/** One lease, callables included. `core.lease` and the cleanup sweep read it. */
const eventLease = (id) => eventLeases.get(id) || null
/** Every option source WITHOUT its resolver — the authoring form's list. */
@@ -985,7 +985,7 @@ function checkActionParam(actionId, entry, seen) {
}
/**
* `registerEventActions([{ id, label, risk, reversible, version, budgetMs, cost, params, perform, revert }])`.
* `registerEventActions([{ id, label, risk, reversible, version, budgetMs, cost, params, perform, revert, reconcile }])`.
*
* A typed verb core may ask a registrant to carry out. Everything decidable from
* the argument alone is decided here, at the call; the collision — is this id
@@ -1042,6 +1042,23 @@ function checkEventActionShape(entry) {
`registerEventActions: ${a.id} declares revert() but is reversible: '${a.reversible}'`,
)
}
// §L's reconnect row, and it is OPTIONAL where `revert` is required (Phase 8).
// `revert` is how a run gives a resource back; `reconcile` is how a module says
// which of them the game still has after something outside core restarted. A
// module that cannot answer that question is not broken -- core simply keeps
// believing its own ledger, which is the pre-Phase-8 behaviour -- whereas a
// module that created something and cannot undo it has made a promise core has
// no way to keep. Only meaningful for an action that ledgers anything.
if (a.reconcile !== undefined) {
if (typeof a.reconcile !== 'function') {
throw new Error(`registerEventActions: ${a.id} reconcile must be a function`)
}
if (a.reversible === 'none' || a.reversible === 'self') {
throw new Error(
`registerEventActions: ${a.id} declares reconcile() but is reversible: '${a.reversible}' and ledgers nothing`,
)
}
}
if (a.cost !== undefined && typeof a.cost !== 'function') {
throw new Error(`registerEventActions: ${a.id} cost must be a function of its params`)
}
@@ -1076,6 +1093,7 @@ function checkEventActionShape(entry) {
cost: a.cost || null,
perform: a.perform,
revert: a.revert || null,
reconcile: a.reconcile || null,
}
}
@@ -1130,10 +1148,12 @@ function checkEventBudgetShape(entry) {
* thing a module must not be allowed to skip. A lease whose restore writes blindly
* is a lease that silently reverts an operator's manual fix.
*
* **Nothing acquires a lease in Phase 7.** This registers, validates and serves
* one; the ledger that holds it, the deadline that goes down the wire and the
* drift answer are Phase 8's. Declaring it now is what keeps the module contract
* one version rather than two.
* **A lease is acquired by `core.lease` and by nothing else** (Phase 8). The step
* names a lease id, a value and a duration; core reads the baseline, reserves the
* target in `event_run_resources` — which is where the two-events-one-target
* refusal comes from — applies the value with the deadline, and restores it at
* teardown through the same `restore()` the drift check lives in. A module ships
* the three callables and never has to own any of that.
*/
function checkEventLeaseShape(entry) {
const l = entry || {}

View File

@@ -9,13 +9,12 @@
// this screen does.
//
// **Phase 3 added the live run controls** at the bottom of this file: pause,
// resume, cancel, and a step's confirm, skip and retry. What is still absent is
// `advance`, `cleanup` and the action switchboard — `advance` has no honest
// meaning until Phase 5 gives a phase an advance condition, `cleanup` has no
// ledger to work over until Phase 8, and the switchboard is Phase 6's. Each of
// them is absent rather than stubbed, for the reason the whole set was in Phase
// 1: a control that returns 200 and does nothing is worse than one that is not
// there.
// resume, cancel, and a step's confirm, skip and retry. `advance` joined them in
// Phase 5, the action switchboard in Phase 6, and **`cleanup` in Phase 8** —
// each when the phase that gave it something to act on landed, and each absent
// rather than stubbed until then, for the reason the whole set was in Phase 1: a
// control that returns 200 and does nothing is worse than one that is not there.
// Nothing in the § API surface table is absent any more.
const registries = require('../../../modules/registries')
const spec = require('../../../events/spec')
@@ -315,6 +314,12 @@ exports.getRun = async (req, res) => {
// rather than "what is allowed now" — which is the question that survives
// an admin moving a switch tomorrow.
budget: found.budget,
// The resource ledger (Phase 8): everything this run created or borrowed, and
// what became of each. The WHOLE ledger, reverted rows included — "how much
// did last night's invasion spawn, and did all of it come back" is one
// question with two halves, and a list of only the failures answers neither.
resources: found.resources,
unresolvedResources: found.unresolvedResources,
})
}
@@ -678,14 +683,48 @@ exports.advanceRunPhase = async (req, res) => {
exports.cancelRun = async (req, res) => {
const runId = asId(req.params.runId)
if (!runId) return res.status(400).json({ error: 'bad run id' })
const result = await controls.cancel(runId, { reason: req.body?.reason }, req.user.id)
// **`cleanup` defaults to true and has to be asked out of.** §L makes cancelling
// WITHOUT cleanup the separate, admin-only, logged action, so an absent flag
// must mean "give back what this run took" — the safe direction, and the one a
// moderator's cancel at two in the morning takes without having to know the
// flag exists.
const withCleanup = req.body?.cleanup !== false
const result = await controls.cancel(
runId,
{ reason: req.body?.reason, cleanup: withCleanup },
req.user.id,
{ isAdmin: req.user.role === 'admin' },
)
if (!result.ok) return res.status(result.status || 409).json({ errors: result.errors })
await activity.log({
req,
action: 'event.run.cancelled',
detail: { runId, reason: req.body?.reason || null, cancelledSteps: result.cancelledSteps },
detail: {
runId,
reason: req.body?.reason || null,
cancelledSteps: result.cancelledSteps,
cleanup: result.cleanup,
},
})
return res.json({ run: shapeRun(result.run), cancelledSteps: result.cancelledSteps })
return res.json({
run: shapeRun(result.run),
cancelledSteps: result.cancelledSteps,
cleanup: result.cleanup,
})
}
/** POST /api/v1/admin/events/runs/:runId/cleanup */
exports.cleanupRun = async (req, res) => {
const runId = asId(req.params.runId)
if (!runId) return res.status(400).json({ error: 'bad run id' })
const result = await controls.cleanupRun(runId, req.user.id)
if (!result.ok) return res.status(result.status || 409).json({ errors: result.errors })
await activity.log({ req, action: 'event.run.cleaned', detail: { runId, ...result.summary } })
// **A 200 whatever the sweep found.** The request succeeded; some resources may
// still be out there, and answering 4xx would make "the shard refused to delete
// three of these" indistinguishable from "you sent a bad run id" — the same
// argument the dry run's findings make.
return res.json({ run: shapeRun(result.run), summary: result.summary })
}
/** POST /api/v1/admin/events/runs/:runId/steps/:stepId/confirm */

View File

@@ -17,8 +17,10 @@
// 5; **`verify` and the action switchboard arrived in Phase 6** — `verify` at
// `admin, editor` because a dry run dispatches nothing, and both halves of
// `/actions` at `admin`, because §K puts the switchboard in the same row as the
// world-changing actions it governs. `cleanup` is still absent rather than
// stubbed: there is no resource ledger until Phase 8.
// world-changing actions it governs. **`cleanup` completed the set in Phase 8**,
// and it is `admin` rather than admin+moderator for the same §K reason: it asks
// core to write to the world again, which is not incident response. There is no
// route in the § API surface table left absent.
//
// **Literal paths are declared before `/:id`**, so `/catalog`, `/series`,
// `/calendar` and `/runs` are never read as an event id.
@@ -207,9 +209,9 @@ eventsRouter.get(
'/runs/:runId',
// #swagger.tags = ['Admin · Events']
// #swagger.summary = 'One run: its status, health, cleanup state and every step with its params and idempotency key'
// #swagger.description = 'The run console. `counts` summarises the step list by status. Steps carry the idempotency key core minted at materialisation — stable across every attempt, which is what lets the game side recognise a repeat. `gates` is the diagnosis panel (Phase 5): one entry per phase that authored an advance condition, already rendered in the condition builder's own words — `gte` as "is at least", `present` as "is present" — with the tally, how long it has waited, and the last related firing whether or not it matched. A phase is waiting on its gate only once every one of its steps is terminal; `stalled` means an `on` gate has waited past EVENT_PHASE_STALL_MS, which is visibility and never a timeout — nothing advances a phase but its condition or a human.'
// #swagger.description = 'The run console. `counts` summarises the step list by status. Steps carry the idempotency key core minted at materialisation — stable across every attempt, which is what lets the game side recognise a repeat. `gates` is the diagnosis panel (Phase 5): one entry per phase that authored an advance condition, already rendered in the condition builder's own words — `gte` as "is at least", `present` as "is present" — with the tally, how long it has waited, and the last related firing whether or not it matched. A phase is waiting on its gate only once every one of its steps is terminal; `stalled` means an `on` gate has waited past EVENT_PHASE_STALL_MS, which is visibility and never a timeout — nothing advances a phase but its condition or a human. `budget` is the cap meter (Phase 6), and `resources` is the cleanup ledger (Phase 8): every object this run created and every value it borrowed, with what became of each — `confirmed` is still out there, `reverted` came back, `drifted` means somebody moved it and core left it alone, and `orphaned` means the module reports it is gone. `unresolvedResources` counts the ones still wanting something, including a placeholder left standing by a lost acknowledgement, which is why it can exceed the length of the list.'
// #swagger.security = [{ "cookieAuth": [] }, { "bearerAuth": [] }]
/* #swagger.responses[200] = { description: 'The run, its steps, the status counts and the phase gates', content: { "application/json": { schema: { type: "object", properties: { run: { type: "object", additionalProperties: true }, steps: { type: "array", items: { type: "object", additionalProperties: true } }, counts: { type: "object", additionalProperties: true }, gates: { type: "array", items: { type: "object", additionalProperties: true } } } } } } } */
/* #swagger.responses[200] = { description: 'The run, its steps, the status counts, the phase gates, the cap meter and the resource ledger', content: { "application/json": { schema: { type: "object", properties: { run: { type: "object", additionalProperties: true }, steps: { type: "array", items: { type: "object", additionalProperties: true } }, counts: { type: "object", additionalProperties: true }, gates: { type: "array", items: { type: "object", additionalProperties: true } }, budget: { type: "array", items: { type: "object", additionalProperties: true } }, resources: { type: "array", items: { type: "object", additionalProperties: true } }, unresolvedResources: { type: "integer" } } } } } } */
/* #swagger.responses[404] = { description: 'No such run', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
controller.getRun,
)
@@ -273,16 +275,29 @@ eventsRouter.post(
'/runs/:runId/cancel',
// #swagger.tags = ['Admin · Events']
// #swagger.summary = 'Cancel a run'
// #swagger.description = 'Legal from every non-terminal status, `scheduled` included. Pending steps and any parked cue are cancelled with it; a step with a live lease is left alone, because nothing can recall a command already sent and a second writer on that row would race the process dispatching it. `cleanup` is not a parameter yet — the resource ledger it would work over arrives in Phase 8, and a flag that changes nothing is worse than one that is not there.'
// #swagger.description = 'Legal from every non-terminal status, `scheduled` included. Pending steps and any parked cue are cancelled with it; a step with a live lease is left alone, because nothing can recall a command already sent and a second writer on that row would race the process dispatching it. `cleanup` arrived in Phase 8 and DEFAULTS TO TRUE: what the run created or borrowed is given back by the runner cleanup leg on its next tick, which is why this answers at once rather than after a round trip per resource. Sending `cleanup: false` deliberately leaves the world changes in place — that is admin-only even though the route is admin+moderator, because which of the two you have to be depends on what is in the body — and the run then carries `cleanup_status: incomplete` with every unreverted row listed on its console.'
// #swagger.security = [{ "cookieAuth": [] }, { "bearerAuth": [] }]
/* #swagger.requestBody = { required: false, content: { "application/json": { schema: { type: "object", properties: { reason: { type: "string", description: "Why. Recorded on the run and in its log, with the actor." } } } } } } */
/* #swagger.responses[200] = { description: 'The cancelled run and how many steps were closed out with it', content: { "application/json": { schema: { type: "object", properties: { run: { type: "object", additionalProperties: true }, cancelledSteps: { type: "integer" } } } } } } */
/* #swagger.requestBody = { required: false, content: { "application/json": { schema: { type: "object", properties: { reason: { type: "string", description: "Why. Recorded on the run and in its log, with the actor." }, cleanup: { type: "boolean", description: "Default true. False leaves the world changes from this run in place, and is admin-only." } } } } } } */
/* #swagger.responses[200] = { description: 'The cancelled run, how many steps were closed out with it, and whether cleanup was asked for', content: { "application/json": { schema: { type: "object", properties: { run: { type: "object", additionalProperties: true }, cancelledSteps: { type: "integer" }, cleanup: { type: "boolean" } } } } } } */
/* #swagger.responses[409] = { description: 'The run has already reached a terminal status', content: { "application/json": { schema: { type: "object", properties: { errors: { type: "array", items: { type: "string" } } } } } } } */
/* #swagger.responses[403] = { description: 'Not an admin or moderator', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
/* #swagger.responses[403] = { description: 'Not an admin or moderator, or a moderator asking to skip cleanup', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
liveControl,
controller.cancelRun,
)
eventsRouter.post(
'/runs/:runId/cleanup',
// #swagger.tags = ['Admin · Events']
// #swagger.summary = 'Re-run cleanup over everything this run has not given back'
// #swagger.description = 'The manual retry EVENTS.md §L promises, and the only thing that clears a resource attempt counter — the automatic sweep never does, because a sweep that reset every stale row is what made an attempt ceiling unreachable in the engagement workstream. Legal on a TERMINAL run only: a run still in flight has a ledger that is still growing, and reverting a resource the next step is about to use would be core undoing an event while it is happening. `admin` rather than admin+moderator, unlike the seven live controls beside it, because this is not incident response — it asks core to write to the world again, which §K puts in the same row as the world-changing actions themselves. Answers 200 whatever it found: some resources may still be out there, and a 4xx would make that indistinguishable from a bad run id.'
// #swagger.security = [{ "cookieAuth": [] }, { "bearerAuth": [] }]
/* #swagger.responses[200] = { description: 'The run and what the sweep managed', content: { "application/json": { schema: { type: "object", properties: { run: { type: "object", additionalProperties: true }, summary: { type: "object", properties: { attempted: { type: "integer" }, reverted: { type: "integer" }, drifted: { type: "integer" }, failed: { type: "integer" }, remaining: { type: "integer" } } } } } } } } */
/* #swagger.responses[409] = { description: 'The run is still in flight, or recorded no resources at all', content: { "application/json": { schema: { type: "object", properties: { errors: { type: "array", items: { type: "string" } } } } } } } */
/* #swagger.responses[403] = { description: 'Not an admin', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
adminOnly,
controller.cleanupRun,
)
eventsRouter.post(
'/runs/:runId/advance',
// #swagger.tags = ['Admin · Events']

View File

@@ -16,6 +16,7 @@ const teamForumUploadSweep = require('./utils/teamForumUploadSweep')
const teamDigestWorker = require('./utils/teamDigestWorker')
const engagementWorker = require('./utils/engagementWorker')
const eventRunner = require('./utils/eventRunner')
const eventCleanup = require('./events/cleanup')
const { ensureSchema, close } = require('./utils/db')
const { seedDefaults, createInitialAdminFromEnv } = require('../db/seed')
const settings = require('./model/settings/settings.model')
@@ -170,11 +171,25 @@ async function start() {
// enables a rule: core seeds none and `enabled` defaults to 0.
engagementWorker.start()
// Advance scheduled events (EVENTS.md §E). Materialise, advance, drain, and the
// run-log retention sweep. No-op until an admin publishes a definition and
// starts a run: core ships no event definitions.
// Advance scheduled events (EVENTS.md §E). Materialise, advance, drain, clean
// up and the run-log retention sweep. No-op until an admin publishes a
// definition and starts a run: core ships no event definitions.
eventRunner.start()
// **Core's own restart is the one reconnect core can see** (EVENTS.md §L,
// Phase 8). Every other one belongs to a module, which reports it through
// `ctx.events.reconcile()`; this is the case where the thing that restarted was
// this process, and the ledger it wakes up holding may describe a world that
// moved on while it was down. Awaited by nobody and never fatal: a module that
// cannot answer leaves its rows alone, which is the pre-Phase-8 behaviour.
eventCleanup
.reconcileAll()
.then((summaries) => {
const orphaned = Object.values(summaries).reduce((n, x) => n + (x.orphaned || 0), 0)
if (orphaned) log.warn('event resources orphaned at boot', { orphaned, summaries })
})
.catch((err) => log.error('boot reconcile failed', { message: err.message }))
setupShutdown(server, internalServer)
}

View File

@@ -10,7 +10,16 @@
// 1. **reclaim** — release leases whose holder died, never touching `attempts`
// 2. **materialise** — sweep occurrences past their grace window into `missed`
// 3. **advance** — claim each due run and move it through its phases
// 4. **prune** — the `event_run_log` retention sweep, on its own long clock
// 4. **cleanup** — give back what a TERMINAL run still holds (Phase 8)
// 5. **prune** — the `event_run_log` retention sweep, on its own long clock
//
// **Cleanup is a leg rather than a limb of `advanceRun`**, and its position in
// that list is load-bearing: it runs after advance, so a run that completes in
// one tick is torn down in the same one, and it is one place rather than four, so
// a process that dies mid-teardown resumes on the next tick instead of leaving a
// world half-restored with nothing scheduled to finish it. §L's "cleanup steps
// are generated from the ledger and run" on cancellation and abort as well as on
// completion is one query here rather than a hook on each of the three.
//
// **What a phase advances on, as of Phase 5.** Every step terminal, and — if the
// phase authored one — its GATE open as well. The gate is an ADDITIONAL
@@ -69,6 +78,8 @@ const gates = require('../events/gates')
const spec = require('../events/spec')
const registries = require('../modules/registries')
const { dispatchStep } = require('../events/dispatch')
const ledger = require('../events/ledger')
const cleanup = require('../events/cleanup')
const authorize = require('../events/authorize')
const log = require('./logger')('event-runner')
@@ -261,6 +272,27 @@ async function drainStep(run, step, now, carry = {}) {
}
}
// ── Rule 1: record BEFORE the dispatch, not after ──
//
// §D. A step that ledgers gets a placeholder keyed by its idempotency key,
// written before anything reaches the module, so an answer that never comes
// back still leaves cleanup something to act on. Recording afterwards would
// make every object whose acknowledgement was lost invisible for ever, which is
// the one failure the whole world-write half cannot tolerate.
//
// It is deliberately AFTER the permission check: a refused step never reaches
// the module, so it has created nothing and must ledger nothing.
let placeholderId = null
try {
placeholderId = await ledger.reserveStep(run, step, action)
} catch (err) {
// The ledger is what makes a world write recoverable, so a step that cannot
// be recorded must not be dispatched. Transient — the next attempt tries the
// insert again — because the alternative is an unrecorded world change.
log.error('could not reserve the ledger row', { run: run.id, step: step.id, message: err.message })
return applyFailure(run, step, `the resource ledger could not record this step: ${err.message}`)
}
const result = await dispatchStep(step, { run })
if (result.actionVersionDrift) {
@@ -273,6 +305,36 @@ async function drainStep(run, step, now, carry = {}) {
})
}
// ── …and promote it on the answer ──
//
// On both success shapes, because `await: 'human'` is a SUCCESS: the module did
// its part and something outside the system has to happen next, and a cue's
// confirm finishes the step without a second dispatch — so this is the only
// moment its resources can be recorded. A failure records nothing and leaves the
// placeholder standing, which is the whole point of writing one.
if (result.outcome === 'done' || result.outcome === 'parked') {
const recorded = await ledger.recordAnswer({
run,
step,
action,
placeholderId,
resources: result.resources,
})
if (recorded.recorded > 0 || recorded.rejected.length > 0) {
await logDb.write({
runId: run.id,
stepId: step.id,
kind: 'resource.recorded',
phase: step.phase,
detail: {
action: step.action_id,
recorded: recorded.recorded,
...(recorded.rejected.length ? { rejected: recorded.rejected } : {}),
},
})
}
}
if (result.outcome === 'parked') {
// The GM cue. The step stays `running` with a NULL lease: genuinely in
// flight, nothing holding it, so the stale reclaim passes it by and a cue
@@ -459,9 +521,11 @@ async function advanceRun(run, now) {
}
if (run.status === 'ending') {
// A run that reached the wind-down and then lost its process. Phase 8 puts
// cleanup here; until then `ending` is a state a run passes through rather
// than one it does work in, and completing it is the whole recovery.
// A run that reached the wind-down and then lost its process. Completing it is
// still the whole recovery even with a ledger in the picture: cleanup is a leg
// of the tick over TERMINAL runs, so the row this leaves behind is exactly
// what that leg is looking for, and doing the teardown here as well would be
// the second call site the leg exists to avoid.
await runsDb.transition(run.id, 'ending', 'completed')
await logDb.write({ runId: run.id, kind: 'run.status', detail: { from: 'ending', to: 'completed' } })
return 'completed'
@@ -536,8 +600,9 @@ async function advanceRun(run, now) {
const next = phases[phaseIndex + 1]
if (!next) {
// §E's `ending` exists for the reason `sending` does in the outbox — it is
// what a claim sets — so the run passes through it even though Phase 2 has
// no cleanup to do there. Phase 8 is what gives it work.
// what a claim sets. The run still passes straight through it: the ledger's
// teardown is the tick's cleanup leg, which runs after this one in the same
// tick and takes the run as it now is.
if (!(await runsDb.transition(run.id, 'running', 'ending'))) return 'taken'
await logDb.write({ runId: run.id, kind: 'run.status', phase: phaseKey, detail: { from: 'running', to: 'ending' } })
await runsDb.transition(run.id, 'ending', 'completed')
@@ -790,6 +855,17 @@ async function tick(now = new Date()) {
}
if (due && due.length) log.info('event runs swept', { due: due.length, ...counts })
// **After advance, deliberately.** A run that reached `completed` two lines ago
// is torn down in this tick rather than the next, so §L's "a run reaches
// `completed` with `cleanup_status = 'incomplete'`" is what an operator sees
// instead of a completed run that briefly claims it has cleanup pending.
try {
const swept = await cleanup.sweep()
if (swept) log.info('event cleanup swept', { runs: swept })
} catch (err) {
log.error('cleanup sweep failed', { message: err.message })
}
try {
await prune(now)
} catch (err) {