feat(events): the resource ledger, leases and cleanup (Phase 8) #190

Merged

whitlocktech merged 1 commits from feature/events-phase-8 into edge 2026-09-04 05:12:21 +00:00

Conversation 1 Commits 1 Files Changed 29 +3928 -72

wtclaude commented 2026-09-04 02:20:53 +00:00

Member

Copy Link

Event System Phase 8 (EVENTS_PLAN.md). Docs half: RunicGateway/docs#216.

This is the phase that makes an unattended, scheduled world change defensible. Core now records what a run did to the world before it does it, gives it back on every terminal path, and says so loudly when it cannot. Everything before this could announce, wait and cue a human; nothing before this could be trusted to spawn something.

One table (event_run_resources), one core action (core.lease), one option source (core.options.leases), one route added and none moved, one body field (cancel's { cleanup }), and two members added to MODULE_API 1.10.0 in place.

The version was amended, not bumped

reconcile() and ctx.events.reconcile() joined 1.10.0 rather than becoming 1.11.0 (org lead, 2026-09-03). A protocol owes a bump once it has landed on main; while it is on edge it is amended in place — the rule the Teams workstream arrived at, applied to a module API for the first time. 1.10.0 has not shipped, so the whole module contract reaches an author as one version they read once, which was the argument for putting the lease declaration in it a phase early.

This branch therefore does not touch MODULE_API_VERSION. The integration kit stays exactly as Phase 7 left it: red on purpose until the Phase 16 cutover re-pins ci/core-ref.json. Please still do not "fix" it there.

Four decisions, settled 2026-09-03, all as recommended

1. A lease is acquired by a new CORE action, core.lease. §F puts the duration bound and the two-events-one-target conflict check on core's side of the seam, and a lease verb per module would be both of those re-implemented once per module — advisory everywhere, and wrong in the first one that forgot. It is risk: 'change', so it is the first core action that is default-off, admin-only and cap-checked like any module verb.
2. Record-before-confirm is a PLACEHOLDER keyed by the step's idempotency key. A spawn's ref does not exist until the module answers, so there is nothing to write a row about — what core writes beforehand is kind: '@step', ref = that key. On the answer the reported resources are inserted confirmed and the placeholder is resolved; if the answer never comes it stands, and cleanup calls revert({ idempotencyKey, resources: [] }). That is why §F's revert takes the key at all. A lease skips it and reserves its real target instead — the same rule in a stronger form, and the only moment the conflict refusal can happen before the world has been written to.
3. Cleanup is one sweep over the ledger, not synthetic event_run_steps rows. The step-shaped reading buys the console's per-step retry for free and costs a second retry counter beside revert_attempts — two counters that disagree the first time a step reverts three of its four resources.
4. reconcile is declared here and TRIGGERED BY THE MODULE, through ctx.events.reconcile(). Core has no concept of the game being up (§F), so it cannot decide when to ask; module-uo already watches bootId. Core asks once at its own boot, which is the one reconnect it can see.

Four things the build settled

• The unique key is held by three statuses and released by three, which corrects §D's "among non-reverted rows" — written before the six statuses had their meanings. Taken literally it makes drifted and orphaned hold a target for ever, so one bad night would disable a lease permanently with no control able to clear it. Recorded as a dated amendment.
• MariaDB has no partial index, so the key is a STORED generated column that is NULL once the row is no longer ours — reading status alone. TEAMS.md §2.5 had to be corrected on this exact shape: MariaDB refuses ON DELETE SET NULL on a foreign key whose column is a base column of a stored generated column (error 1901), and step_id must stay SET NULL because a record of what was changed in the world has to outlive the row that scheduled it.
• Cleanup runs from one place, a fifth leg of the tick, ordered after advance so a run that completes in one tick is torn down in the same one. Hooking each terminal path would be four call sites, three inside a request, none surviving a process that died mid-teardown. It is also why cancel answers at once.
• What that leg SELECTS is the phase's real finding — see the walk below.

Verify

• npm test — 2025 tests, 1935 pass, 89 skipped, 1 fail. That one is the pre-existing engagementManifest.test.js CRLF failure, in a file this branch does not touch (edge before: 1950/1876/73/1). +75 tests.
• The unique key was proved against a real MariaDB, because nothing else can prove it. Whether multiple NULLs collide in a unique index, whether a STORED generated column is recomputed and re-indexed on UPDATE, and whether the SET NULL foreign key survives beside it are properties of the server. eventRunnerSql.test.js gained 16 tests; 65 pass against the container. The real schema.sql was applied to a fresh database and to an existing one — CREATE TABLE IF NOT EXISTS next to a generated column is where a migration silently does nothing.
• A latent bug in that file, found on the way. Its model-backed tests required a shipping model, whose pool utils/db builds at require time from DB_NAME — and utils/db's own dotenv.config() reads server/.env. So on a developer's machine those tests were reaching that developer's real schema while the fixtures they asserted against were written into the throwaway database next door. They passed only because both tables happened to exist in both.
• Client: 362 pass, and it builds. npm run routes:manifest and npm run swagger — one route added, none moved.
• The stale-stub trap, for the fourth time in this feature. Phase 4's expansion leg, Phase 5's gate read, Phase 6's settings read, now the ledger write. Unstubbed it is not a wrong answer, it is a ten-second ECONNREFUSED.

The live walk found three defects, and two of them are the phase's real finding

Driven by a throwaway rig module in website/modules/ — a real directory scanned by the real loader, deleted before commit — whose behaviour is driven by a JSON file on disk, so a revert could be made to fail and then succeed on the same boot. 34 assertions, all green at the end. rig declared coreApi: "^1.10.0" and module-uo's "^1.9.0" still resolved, on the same boot.

Defect 1 — a lease was never given back at all. core.lease reserves its own ledger row, so it never went through the ledger's dirty-marking, so a run holding only a lease kept cleanup_status = 'not_required' and the cleanup leg — which selected on pending — never looked at it. Every unit test passed: they call the sweep directly, and a test that calls the sweep never asks what would have SELECTED the run.

Defect 2 — EVENT_REVERT_MAX_ATTEMPTS meant one attempt, not three. The first failing sweep moved the run to incomplete, which took it out of the leg's own scan for ever. Visible only as revert_attempts sitting at 1 through half a minute of live ticks — and the test covering the bound asserted <= 3, which 1 satisfies. A bound has two halves, and a test that only asserts the ceiling passes against a floor. Both halves are asserted now.

Both are the same mistake: deriving "is there anything to do" from a summary column instead of from the rows that are the fact. The scan now includes not_required, and incomplete is written only once there is nothing left to try — so it means "finished with, and not finished", which is also what makes it the honest marker for a run an admin cancelled without cleanup.

Defect 3 — the first fix for defect 2 made the console lie. Spending every row's revert_attempts was a tidy way to take a cleanup: false run out of a counter-bounded scan, and the run page then rendered "3 attempts" beside resources nothing had ever tried — which reads as "core tried three times and could not". Found by opening the page, exactly like Phase 7's React defect. A counter that means two things is a counter a screen cannot render.

What the walk proved beyond the defects

Three wisps recorded and given back with the run reaching complete; a failing revert leaving the run completed + incomplete with the reason on each row; the cleanup route rescuing it, and a moderator refused it; the route refusing a run still in flight and one that recorded nothing; a lease taken with its baseline and applied value in the payload, and a second run refused it by name; a GM's mid-event edit producing drifted with the world left alone, and a later run still able to lease that target; and cancel { cleanup: false } refused to a moderator, allowed to an admin, and logged.

The two that needed the process to die

The plan's own verify line. With the module's perform() hanging: the placeholder existed while the dispatch was in flight and nothing was named — unresolvedResources: 1 over an empty list, which is rule 1 visible in the wild. After taskkill and a restart the reclaim re-dispatched the same idempotency key, the retry re-used its own placeholder rather than writing a second, and everything was given back. Then, with the module reporting one of two resources as no longer in force, core's boot-time reconcileAll() marked the other orphaned — never reverted — and logged it.

The console, in all three states

A clean run: neutral border, "Everything this run created or borrowed has been given back", green rows, no button. An unresolved run: amber border, "3 of these are still unresolved", "still out there", and Try cleanup again — clicked, answered 200, and flipped the panel green in place. A drifted lease: "someone else moved it", its deadline, its genuine attempt count, and the sentence naming the value that is there now.

---

• AI-assisted: authored with Claude Code (Claude Opus).

Event System **Phase 8** (`EVENTS_PLAN.md`). Docs half: **RunicGateway/docs#216**. > **This is the phase that makes an unattended, scheduled world change defensible.** Core now records what a run did to the world *before* it does it, gives it back on every terminal path, and says so loudly when it cannot. Everything before this could announce, wait and cue a human; nothing before this could be trusted to spawn something. One table (`event_run_resources`), one core action (`core.lease`), one option source (`core.options.leases`), **one route added and none moved**, one body field (`cancel`'s `{ cleanup }`), and two members added to **MODULE_API 1.10.0 in place**. ## The version was amended, not bumped `reconcile()` and `ctx.events.reconcile()` joined 1.10.0 rather than becoming 1.11.0 (org lead, 2026-09-03). **A protocol owes a bump once it has landed on `main`; while it is on `edge` it is amended in place** — the rule the Teams workstream arrived at, applied to a module API for the first time. 1.10.0 has not shipped, so the whole module contract reaches an author as one version they read once, which was the argument for putting the lease declaration in it a phase early. **This branch therefore does not touch `MODULE_API_VERSION`.** The integration kit stays exactly as Phase 7 left it: red on purpose until the Phase 16 cutover re-pins `ci/core-ref.json`. Please still do not "fix" it there. ## Four decisions, settled 2026-09-03, all as recommended 1. **A lease is acquired by a new CORE action, `core.lease`.** §F puts the duration bound and the two-events-one-target conflict check on core's side of the seam, and a lease verb per module would be both of those re-implemented once per module — advisory everywhere, and wrong in the first one that forgot. It is `risk: 'change'`, so it is the first core action that is default-off, admin-only and cap-checked like any module verb. 2. **Record-before-confirm is a PLACEHOLDER keyed by the step's idempotency key.** A spawn's `ref` does not exist until the module answers, so there is nothing to write a row *about* — what core writes beforehand is `kind: '@step'`, `ref` = that key. On the answer the reported resources are inserted `confirmed` and the placeholder is resolved; if the answer never comes it stands, and cleanup calls `revert({ idempotencyKey, resources: [] })`. **That is why §F's `revert` takes the key at all.** A lease skips it and reserves its real target instead — the same rule in a stronger form, and the only moment the conflict refusal can happen *before* the world has been written to. 3. **Cleanup is one sweep over the ledger, not synthetic `event_run_steps` rows.** The step-shaped reading buys the console's per-step retry for free and costs a second retry counter beside `revert_attempts` — two counters that disagree the first time a step reverts three of its four resources. 4. **`reconcile` is declared here and TRIGGERED BY THE MODULE**, through `ctx.events.reconcile()`. Core has *no concept of the game being up* (§F), so it cannot decide when to ask; module-uo already watches `bootId`. Core asks once at its own boot, which is the one reconnect it can see. ## Four things the build settled - **The unique key is held by three statuses and released by three**, which corrects §D's *"among non-reverted rows"* — written before the six statuses had their meanings. Taken literally it makes `drifted` and `orphaned` hold a target for ever, so one bad night would disable a lease permanently with no control able to clear it. Recorded as a dated amendment. - **MariaDB has no partial index**, so the key is a STORED generated column that is NULL once the row is no longer ours — reading `status` **alone**. `TEAMS.md` §2.5 had to be corrected on this exact shape: MariaDB refuses `ON DELETE SET NULL` on a foreign key whose column is a base column of a stored generated column (error 1901), and `step_id` must stay SET NULL because a record of what was changed in the world has to outlive the row that scheduled it. - **Cleanup runs from one place**, a fifth leg of the tick, ordered after advance so a run that completes in one tick is torn down in the same one. Hooking each terminal path would be four call sites, three inside a request, none surviving a process that died mid-teardown. It is also why `cancel` answers at once. - **What that leg SELECTS is the phase's real finding** — see the walk below. ## Verify - `npm test` — **2025 tests, 1935 pass, 89 skipped, 1 fail.** That one is the pre-existing `engagementManifest.test.js` CRLF failure, **in a file this branch does not touch** (`edge` before: 1950/1876/73/1). **+75 tests.** - **The unique key was proved against a real MariaDB, because nothing else can prove it.** Whether multiple NULLs collide in a unique index, whether a STORED generated column is recomputed and re-indexed on UPDATE, and whether the SET NULL foreign key survives beside it are properties of the server. `eventRunnerSql.test.js` gained 16 tests; **65 pass** against the container. The real `schema.sql` was applied to a fresh database *and* to an existing one — `CREATE TABLE IF NOT EXISTS` next to a generated column is where a migration silently does nothing. - **A latent bug in that file, found on the way.** Its model-backed tests required a shipping model, whose pool `utils/db` builds at require time from `DB_NAME` — and `utils/db`'s own `dotenv.config()` reads `server/.env`. So on a developer's machine those tests were reaching that developer's real schema while the fixtures they asserted against were written into the throwaway database next door. They passed only because both tables happened to exist in both. - Client: **362 pass**, and it builds. `npm run routes:manifest` and `npm run swagger` — **one route added, none moved.** - **The stale-stub trap, for the fourth time in this feature.** Phase 4's expansion leg, Phase 5's gate read, Phase 6's settings read, now the ledger write. Unstubbed it is not a wrong answer, it is a ten-second `ECONNREFUSED`. ## The live walk found three defects, and two of them are the phase's real finding Driven by a throwaway `rig` module in `website/modules/` — a real directory scanned by the real loader, **deleted before commit** — whose behaviour is driven by a JSON file on disk, so a revert could be made to fail and then succeed on the same boot. 34 assertions, all green at the end. `rig` declared `coreApi: "^1.10.0"` and module-uo's `"^1.9.0"` still resolved, on the same boot. **Defect 1 — a lease was never given back at all.** `core.lease` reserves its own ledger row, so it never went through the ledger's dirty-marking, so a run holding only a lease kept `cleanup_status = 'not_required'` and the cleanup leg — which selected on `pending` — never looked at it. Every unit test passed: they call the sweep directly, and **a test that calls the sweep never asks what would have SELECTED the run.** **Defect 2 — `EVENT_REVERT_MAX_ATTEMPTS` meant one attempt, not three.** The first failing sweep moved the run to `incomplete`, which took it out of the leg's own scan for ever. Visible only as `revert_attempts` sitting at 1 through half a minute of live ticks — and the test covering the bound asserted `<= 3`, which 1 satisfies. **A bound has two halves, and a test that only asserts the ceiling passes against a floor.** Both halves are asserted now. Both are the same mistake: deriving *"is there anything to do"* from a summary column instead of from the rows that are the fact. The scan now includes `not_required`, and `incomplete` is written only once there is nothing left to try — so it means *"finished with, and not finished"*, which is also what makes it the honest marker for a run an admin cancelled without cleanup. **Defect 3 — the first fix for defect 2 made the console lie.** Spending every row's `revert_attempts` was a tidy way to take a `cleanup: false` run out of a counter-bounded scan, and the run page then rendered *"3 attempts"* beside resources nothing had ever tried — which reads as *"core tried three times and could not"*. Found by opening the page, exactly like Phase 7's React defect. **A counter that means two things is a counter a screen cannot render.** ### What the walk proved beyond the defects Three wisps recorded and given back with the run reaching `complete`; a failing revert leaving the run `completed` + `incomplete` with the reason on each row; the cleanup route rescuing it, and a moderator refused it; the route refusing a run still in flight and one that recorded nothing; a lease taken with its baseline and applied value in the payload, and a **second run refused it by name**; a GM's mid-event edit producing `drifted` with the world left alone, and a later run still able to lease that target; and `cancel { cleanup: false }` refused to a moderator, allowed to an admin, and logged. ### The two that needed the process to die The plan's own verify line. With the module's `perform()` hanging: the placeholder existed while the dispatch was in flight and **nothing was named** — `unresolvedResources: 1` over an empty list, which is rule 1 visible in the wild. After `taskkill` and a restart the reclaim re-dispatched the **same** idempotency key, the retry re-used its own placeholder rather than writing a second, and everything was given back. Then, with the module reporting one of two resources as no longer in force, core's boot-time `reconcileAll()` marked the other **`orphaned`** — never `reverted` — and logged it. ### The console, in all three states A clean run: neutral border, *"Everything this run created or borrowed has been given back"*, green rows, no button. An unresolved run: amber border, *"3 of these are still unresolved"*, *"still out there"*, and **Try cleanup again** — clicked, answered `200`, and flipped the panel green in place. A drifted lease: *"someone else moved it"*, its deadline, its genuine attempt count, and the sentence naming the value that is there now. --- - [x] AI-assisted: authored with Claude Code (Claude Opus).

wtclaude added 1 commit 2026-09-04 02:20:55 +00:00

feat(events): the resource ledger, leases and cleanup (Phase 8) ... fdc118166c

Event System Phase 8 (EVENTS_PLAN.md). Docs half: RunicGateway/docs#NNN.

One table, one core action, one route, one body field, and two members added to
MODULE_API 1.10.0 in place. The safety property the whole world-write half
depends on: core now remembers what a run changed in the world, and gives it
back on every terminal path.

Four decisions settled by the org lead on 2026-09-03, all as recommended:

- A lease is acquired by a new CORE action, `core.lease`. Section F puts the
  duration bound and the two-events-one-target conflict check on core's side of
  the seam, and a lease verb per module would be both re-implemented once per
  module, advisory everywhere.
- Record-before-confirm is a PLACEHOLDER keyed by the step's idempotency key. A
  spawn's ref does not exist until the module answers, so what core writes
  before the dispatch is `kind: '@step'`, `ref` = that key. If the answer never
  comes it stands, and cleanup calls revert() with the key and no resources --
  which is why section F's revert takes the key at all.
- Cleanup is one sweep over the ledger, not synthetic step rows. The
  step-shaped version costs a second retry counter beside `revert_attempts`.
- `reconcile` is declared here and TRIGGERED BY THE MODULE, through
  `ctx.events.reconcile()`. Core has no concept of the game being up, so it
  cannot decide when to ask; it asks once at its own boot.

MODULE_API stays 1.10.0. A protocol owes a bump once it has landed on `main`;
while it is on `edge` it is amended in place, so the whole module contract
reaches an author as one version they read once.

Verify

- `npm test` -- 2025 tests, 1935 pass, 89 skipped, 1 fail. That one is the
  pre-existing engagementManifest CRLF failure, in a file this branch does not
  touch (`edge` before: 1950/1876/73/1). +75 tests.
- The unique key was proved against a REAL MariaDB, because nothing else can
  prove it: whether multiple NULLs collide in a unique index, whether a STORED
  generated column is recomputed on UPDATE, and whether the SET NULL foreign key
  survives beside it are properties of the server. eventRunnerSql.test.js gained
  16 tests; 65 pass against the container. The real schema.sql was applied to a
  fresh database and to an existing one.
- Client: 362 pass, and it builds. routes:manifest and swagger -- one route
  added, none moved.

The live walk found three defects, and two of them are the phase's real finding

Driven by a throwaway `rig` module in website/modules/, deleted before commit.

1. A lease was never given back at all. `core.lease` reserves its own ledger
   row, so it never went through the ledger's dirty-marking, so a run holding
   only a lease kept `cleanup_status = 'not_required'` and the cleanup leg --
   which selected on `pending` -- never looked at it.
2. EVENT_REVERT_MAX_ATTEMPTS meant one attempt, not three. The first failing
   sweep moved the run to `incomplete`, which took it out of the leg's own scan
   for ever. The test covering the bound asserted `<= 3` and was satisfied by 1:
   a bound has two halves, and a test that only asserts the ceiling passes
   against a floor.
3. The first fix for (2) made the console lie. Spending every row's
   `revert_attempts` was a tidy way to take a `cleanup: false` run out of a
   counter-bounded scan, and the run page then rendered "3 attempts" beside
   resources nothing had ever tried. Found by opening the page.

Both (1) and (2) are the same mistake: deriving "is there anything to do" from a
summary column instead of from the rows. Neither was visible to a unit test,
because a test that calls the sweep directly never asks what would have selected
the run.

The two properties that need the process to die were walked as the plan asks.
With the module's perform() hanging, the placeholder existed while the dispatch
was in flight and nothing was named; after taskkill and a restart the reclaim
re-dispatched the same idempotency key, the retry re-used its own placeholder,
and everything was given back. Then, with the module reporting one of two
resources as no longer in force, the boot-time reconcile marked the other
`orphaned` -- never `reverted`.

This branch does NOT bump MODULE_API_VERSION, so the integration kit stays as
Phase 7 left it: red until the Phase 16 cutover re-pins ci/core-ref.json.

Co-Authored-By: Claude <noreply@anthropic.com>

wtclaude referenced this pull request from RunicGateway/docs 2026-09-04 02:21:32 +00:00

docs(events): Phase 8 as built — the resource ledger, leases and cleanup #216

whitlocktech merged commit 4a91d74085 into edge 2026-09-04 05:12:21 +00:00

whitlocktech deleted branch feature/events-phase-8 2026-09-04 05:12:22 +00:00

whitlocktech referenced this issue from a commit 2026-09-04 05:12:23 +00:00

Merge pull request 'feat(events): the resource ledger, leases and cleanup (Phase 8)' (#190) from feature/events-phase-8 into edge

wtclaude referenced this pull request 2026-09-04 07:44:22 +00:00

fix(engagement): the trigger manifest was stale, and its check was crying wolf #191

wtclaude commented 2026-09-04 07:44:51 +00:00

Author

Member

Copy Link

Correction to this PR's Verify section, for the record.

It says the one failing test was "the pre-existing engagementManifest.test.js CRLF failure, in a file this branch does not touch". That diagnosis was wrong, and #189's body says the same thing, so it has now been repeated twice.

It was not CRLF, and it was not benign: engagement-triggers.json was genuinely stale. It embeds moduleApiVersion, and #189 bumped MODULE_API_VERSION to 1.10.0 without regenerating it, so the committed file has said 1.9.0 since that merge.

The CRLF part was real but secondary — the check compares bytes, and under core.autocrlf=true git checks the LF blob out as CRLF, so on Windows it fails for a spurious reason as well. That noise is precisely why the genuine staleness went unnoticed: the failure had already been written off as environmental, by me, twice.

Both are fixed in #191, which also proves the check is still live by breaking it deliberately. Nothing in Phase 8 itself is affected — the ledger, the lease and the cleanup are unchanged — but the "1 fail" number quoted above should be read as one real failure this branch did not cause and should not have waved past, rather than as environmental noise.

**Correction to this PR's Verify section, for the record.** It says the one failing test was *"the pre-existing `engagementManifest.test.js` CRLF failure, in a file this branch does not touch"*. **That diagnosis was wrong**, and #189's body says the same thing, so it has now been repeated twice. It was not CRLF, and it was not benign: **`engagement-triggers.json` was genuinely stale.** It embeds `moduleApiVersion`, and #189 bumped `MODULE_API_VERSION` to 1.10.0 without regenerating it, so the committed file has said `1.9.0` since that merge. The CRLF part was real but secondary — the check compares bytes, and under `core.autocrlf=true` git checks the LF blob out as CRLF, so on Windows it fails for a spurious reason *as well*. That noise is precisely why the genuine staleness went unnoticed: the failure had already been written off as environmental, by me, twice. Both are fixed in **#191**, which also proves the check is still live by breaking it deliberately. Nothing in Phase 8 itself is affected — the ledger, the lease and the cleanup are unchanged — but the *"1 fail"* number quoted above should be read as **one real failure this branch did not cause and should not have waved past**, rather than as environmental noise.

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

bug

Something is not working

duplicate

This issue or pull request already exists

enhancement

New feature

help wanted

Need some help

invalid

Something is wrong

question

More information is needed

severity:critical

Security audit: critical severity

severity:high

Security audit: high severity

severity:low

Security audit: low severity

severity:medium

Security audit: medium severity

wontfix

This won't be fixed

bug

Something is not working

duplicate

This issue or pull request already exists

enhancement

New feature

help wanted

Need some help

invalid

Something is wrong

question

More information is needed

wontfix

This won't be fixed

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

ksfixitman whitlocktech wtclaude

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: RunicGateway/website#190

No description provided.