feat(events): the resource ledger, leases and cleanup (Phase 8)
Some checks failed
PR Checks / client-build (pull_request) Successful in 3m15s
PR Checks / server-tests (pull_request) Failing after 8m21s
PR Checks / bot-tests (pull_request) Successful in 11m12s

Event System Phase 8 (EVENTS_PLAN.md). Docs half: RunicGateway/docs#NNN.

One table, one core action, one route, one body field, and two members added to
MODULE_API 1.10.0 in place. The safety property the whole world-write half
depends on: core now remembers what a run changed in the world, and gives it
back on every terminal path.

Four decisions settled by the org lead on 2026-09-03, all as recommended:

- A lease is acquired by a new CORE action, `core.lease`. Section F puts the
  duration bound and the two-events-one-target conflict check on core's side of
  the seam, and a lease verb per module would be both re-implemented once per
  module, advisory everywhere.
- Record-before-confirm is a PLACEHOLDER keyed by the step's idempotency key. A
  spawn's ref does not exist until the module answers, so what core writes
  before the dispatch is `kind: '@step'`, `ref` = that key. If the answer never
  comes it stands, and cleanup calls revert() with the key and no resources --
  which is why section F's revert takes the key at all.
- Cleanup is one sweep over the ledger, not synthetic step rows. The
  step-shaped version costs a second retry counter beside `revert_attempts`.
- `reconcile` is declared here and TRIGGERED BY THE MODULE, through
  `ctx.events.reconcile()`. Core has no concept of the game being up, so it
  cannot decide when to ask; it asks once at its own boot.

MODULE_API stays 1.10.0. A protocol owes a bump once it has landed on `main`;
while it is on `edge` it is amended in place, so the whole module contract
reaches an author as one version they read once.

Verify

- `npm test` -- 2025 tests, 1935 pass, 89 skipped, 1 fail. That one is the
  pre-existing engagementManifest CRLF failure, in a file this branch does not
  touch (`edge` before: 1950/1876/73/1). +75 tests.
- The unique key was proved against a REAL MariaDB, because nothing else can
  prove it: whether multiple NULLs collide in a unique index, whether a STORED
  generated column is recomputed on UPDATE, and whether the SET NULL foreign key
  survives beside it are properties of the server. eventRunnerSql.test.js gained
  16 tests; 65 pass against the container. The real schema.sql was applied to a
  fresh database and to an existing one.
- Client: 362 pass, and it builds. routes:manifest and swagger -- one route
  added, none moved.

The live walk found three defects, and two of them are the phase's real finding

Driven by a throwaway `rig` module in website/modules/, deleted before commit.

1. A lease was never given back at all. `core.lease` reserves its own ledger
   row, so it never went through the ledger's dirty-marking, so a run holding
   only a lease kept `cleanup_status = 'not_required'` and the cleanup leg --
   which selected on `pending` -- never looked at it.
2. EVENT_REVERT_MAX_ATTEMPTS meant one attempt, not three. The first failing
   sweep moved the run to `incomplete`, which took it out of the leg's own scan
   for ever. The test covering the bound asserted `<= 3` and was satisfied by 1:
   a bound has two halves, and a test that only asserts the ceiling passes
   against a floor.
3. The first fix for (2) made the console lie. Spending every row's
   `revert_attempts` was a tidy way to take a `cleanup: false` run out of a
   counter-bounded scan, and the run page then rendered "3 attempts" beside
   resources nothing had ever tried. Found by opening the page.

Both (1) and (2) are the same mistake: deriving "is there anything to do" from a
summary column instead of from the rows. Neither was visible to a unit test,
because a test that calls the sweep directly never asks what would have selected
the run.

The two properties that need the process to die were walked as the plan asks.
With the module's perform() hanging, the placeholder existed while the dispatch
was in flight and nothing was named; after taskkill and a restart the reclaim
re-dispatched the same idempotency key, the retry re-used its own placeholder,
and everything was given back. Then, with the module reporting one of two
resources as no longer in force, the boot-time reconcile marked the other
`orphaned` -- never `reverted`.

This branch does NOT bump MODULE_API_VERSION, so the integration kit stays as
Phase 7 left it: red until the Phase 16 cutover re-pins ci/core-ref.json.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2026-09-03 21:19:27 -05:00
parent 57d183e921
commit fdc118166c
29 changed files with 3928 additions and 72 deletions

View File

@@ -4137,7 +4137,7 @@
],
"responses": {
"200": {
"description": "The run, its steps, the status counts and the phase gates",
"description": "The run, its steps, the status counts, the phase gates, the cap meter and the resource ledger",
"content": {
"application/json": {
"schema": {
@@ -4164,6 +4164,23 @@
"type": "object",
"additionalProperties": true
}
},
"budget": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": true
}
},
"resources": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": true
}
},
"unresolvedResources": {
"type": "integer"
}
}
}
@@ -4295,7 +4312,7 @@
"Admin · Events"
],
"summary": "Cancel a run",
"description": "Legal from every non-terminal status, `scheduled` included. Pending steps and any parked cue are cancelled with it; a step with a live lease is left alone, because nothing can recall a command already sent and a second writer on that row would race the process dispatching it. `cleanup` is not a parameter yet — the resource ledger it would work over arrives in Phase 8, and a flag that changes nothing is worse than one that is not there.",
"description": "Legal from every non-terminal status, `scheduled` included. Pending steps and any parked cue are cancelled with it; a step with a live lease is left alone, because nothing can recall a command already sent and a second writer on that row would race the process dispatching it. `cleanup` arrived in Phase 8 and DEFAULTS TO TRUE: what the run created or borrowed is given back by the runner cleanup leg on its next tick, which is why this answers at once rather than after a round trip per resource. Sending `cleanup: false` deliberately leaves the world changes in place — that is admin-only even though the route is admin+moderator, because which of the two you have to be depends on what is in the body — and the run then carries `cleanup_status: incomplete` with every unreverted row listed on its console.",
"parameters": [
{
"name": "runId",
@@ -4308,7 +4325,7 @@
],
"responses": {
"200": {
"description": "The cancelled run and how many steps were closed out with it",
"description": "The cancelled run, how many steps were closed out with it, and whether cleanup was asked for",
"content": {
"application/json": {
"schema": {
@@ -4320,6 +4337,9 @@
},
"cancelledSteps": {
"type": "integer"
},
"cleanup": {
"type": "boolean"
}
}
}
@@ -4330,7 +4350,7 @@
"description": "Bad Request"
},
"403": {
"description": "Not an admin or moderator",
"description": "Not an admin or moderator, or a moderator asking to skip cleanup",
"content": {
"application/json": {
"schema": {
@@ -4376,6 +4396,10 @@
"reason": {
"type": "string",
"description": "Why. Recorded on the run and in its log, with the actor."
},
"cleanup": {
"type": "boolean",
"description": "Default true. False leaves the world changes from this run in place, and is admin-only."
}
}
}
@@ -4384,6 +4408,102 @@
}
}
},
"/api/v1/admin/events/runs/{runId}/cleanup": {
"post": {
"tags": [
"Admin · Events"
],
"summary": "Re-run cleanup over everything this run has not given back",
"description": "The manual retry EVENTS.md §L promises, and the only thing that clears a resource attempt counter — the automatic sweep never does, because a sweep that reset every stale row is what made an attempt ceiling unreachable in the engagement workstream. Legal on a TERMINAL run only: a run still in flight has a ledger that is still growing, and reverting a resource the next step is about to use would be core undoing an event while it is happening. `admin` rather than admin+moderator, unlike the seven live controls beside it, because this is not incident response — it asks core to write to the world again, which §K puts in the same row as the world-changing actions themselves. Answers 200 whatever it found: some resources may still be out there, and a 4xx would make that indistinguishable from a bad run id.",
"parameters": [
{
"name": "runId",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "The run and what the sweep managed",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"run": {
"type": "object",
"additionalProperties": true
},
"summary": {
"type": "object",
"properties": {
"attempted": {
"type": "integer"
},
"reverted": {
"type": "integer"
},
"drifted": {
"type": "integer"
},
"failed": {
"type": "integer"
},
"remaining": {
"type": "integer"
}
}
}
}
}
}
}
},
"400": {
"description": "Bad Request"
},
"403": {
"description": "Not an admin",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Error"
}
}
}
},
"409": {
"description": "The run is still in flight, or recorded no resources at all",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"errors": {
"type": "array",
"items": {
"type": "string"
}
}
}
}
}
}
}
},
"security": [
{
"cookieAuth": []
},
{
"bearerAuth": []
}
]
}
},
"/api/v1/admin/events/runs/{runId}/log": {
"get": {
"tags": [