diff --git a/website/EVENTS.md b/website/EVENTS.md index ae6f8bf..ce26c24 100644 --- a/website/EVENTS.md +++ b/website/EVENTS.md @@ -414,7 +414,7 @@ tables carry no module prefix. (The count said "nine" over a list of ten from th | `event_run_steps` | `run_id`, `phase`, `seq`, `action_id`, `params` JSON, `action_version`, `status`, `due_at`, `attempts`, `on_failure`, `idempotency_key`, `claimed_by`, `claim_expires_at`, `last_error`. `INDEX (status, due_at)` | The work queue, claimed with the outbox's compare-and-set. | | `event_action_settings` | `action_id` (the primary key), `enabled`, `caps` JSON (`{dimension: perRunCap}`), `updated_by`, `updated_at`. | **The deployment's switchboard, and the whole of the permission model beyond the role.** One row per action an admin has an opinion about; **a missing row is not "disabled", it is the default for the action's risk class** — see [§K](#k--security-model). Not a grant table — nobody is named, because the role check already answered who. Rows outlive their actions, so uninstalling a module and re-installing it restores the caps the operator chose. | | `event_run_budget` | `run_id`, `dimension`, `consumed`, `cap` **nullable**, `effective_from`. `UNIQUE (run_id, dimension)` | Consumption is incremented with a conditional update — `… SET consumed = consumed + ? WHERE run_id=? AND dimension=? AND (cap IS NULL OR consumed + ? <= cap)` — so the cap holds under concurrent steps without a transaction. **A NULL cap is uncapped and still a row**, so the console's meter counts what nothing bounds, and a *missing* row keeps its one meaning: a step spending a dimension its own run's version never priced, which is refused. `effective_from` names the action whose cap won, so a number on the meter traces back to a switch. | -| `event_run_resources` | `run_id`, `step_id`, `owner_module`, `kind` and `ref` (both module-opaque), `payload` JSON, `lease_until` nullable, `status` `ENUM('pending','confirmed','reverting','reverted','orphaned','drifted')`, `revert_attempts`, `last_error`, optional `member_key`. `UNIQUE (owner_module, kind, ref)` among non-reverted rows | **The cleanup ledger, and it holds both kinds of thing an event owns** — objects it created (`kind: 'creature'`, `ref` = a serial) and values it leased (`kind: 'override'`, `payload` = baseline + applied). `drifted` is the compare-and-set refusal; the unique index is what stops two events leasing one target. | +| `event_run_resources` | `run_id`, `step_id` **`SET NULL`**, `owner_module`, `kind` and `ref` (both module-opaque), `payload` JSON, `lease_until` nullable, `status` `ENUM('pending','confirmed','reverting','reverted','orphaned','drifted')`, `revert_attempts`, `last_error`, optional `member_key`. `UNIQUE (owner_module, kind, ref)` among the rows core still believes are ITS — see the amendment below | **The cleanup ledger, and it holds both kinds of thing an event owns** — objects it created (`kind: 'creature'`, `ref` = a serial) and values it leased (`kind: 'override'`, `payload` = baseline + applied). `drifted` is the compare-and-set refusal; the unique index is what stops two events leasing one target. `@step` is a reserved `kind` core owns (rule 1, below); a module reporting one is refused. | | `event_run_participants` | `run_id`, `user_id` nullable `SET NULL`, `member_key` module-opaque, `score`, `rank`, `joined_at`, `meta` JSON. `UNIQUE (run_id, member_key)` | Results and profile history read it. `SET NULL` not `CASCADE`, matching `engagement_sends`: a record of what happened must survive an account deletion. | | `event_run_phase_gates` | `run_id`, `phase`, `kind` `ENUM('after','on')`, `after_seconds`, `trigger_id`, `conditions` JSON, `needed`, `tally`, `entered_at`, `due_at`, `last_event` JSON, `satisfied_at`, `satisfied_by`, `forced_by`. `UNIQUE (run_id, phase)`, `INDEX (trigger_id, satisfied_at)` | **What a phase is waiting for, and how far it has got** (Phase 5). The one fact in this feature that is not derivable from a row somebody already wrote: `{ on: …, count: 3 }` counts things that happen *between* two ticks, and the runner is not running when they happen. The unique key is what makes opening a gate an `INSERT IGNORE`; the index is the emit path's only query and the one index here on a hot path. | | `event_run_log` | `run_id`, `step_id` nullable, `kind` (closed set), `phase`, `detail` JSON, `at`. | `activity_log.detail` is `TEXT` and unqueryable. "Why didn't phase 3 start?" must be a query. | @@ -723,9 +723,11 @@ module-uo's `coreApi: "^1.9.0"` continues to resolve. > allowlist for granting an item and for taking one back — and two actions declaring it separately > would be two allowlists that can disagree. > -> **A lease is declared here and acquired by nothing.** Core owns a lease's duration and its conflict -> check; both live in the resource ledger, which is Phase 8. Declaring it now is what keeps the module -> contract one version a module author reads once rather than two. +> **A lease is declared here and acquired by `core.lease`.** Core owns a lease's duration and its +> conflict check; both live in the resource ledger, which Phase 8 built. Declaring it in 1.10.0 rather +> than waiting is what keeps the module contract one version a module author reads once rather than +> two — and `reconcile` was folded into the same version for the same reason, in place, because a +> protocol owes a bump only once it has landed on `main`. ```js api.registerEventBudgets([ @@ -782,6 +784,16 @@ api.registerEventActions([{ A lease action declares what it can hold and how long it may hold it. Core owns the duration and the conflict check; the module owns reading the current value and writing a new one. +**The verb is core's** (org lead, 2026-09-03, Phase 8). A module declares the lease and never writes +one: an author puts `core.lease` in a step, naming the lease, a value and a number of minutes, and +core reads the baseline, reserves the target, applies the value with its deadline, and restores it at +teardown through the module's own `restore()`. A lease verb per module would be the `maxDurationMs` +bound and the two-events-one-target check re-implemented once per module — advisory everywhere, and +wrong in the first one that forgot. It also means `core.lease` needs no `revert()` of its own: an +`override` row is restored through the lease registry, which is exactly the split this section draws. +`core.lease` is `risk: 'change'`, so it is default-off, admin-only and cap-checked like any module +verb — the first core action for which that is true. + ```js api.registerEventLeases([{ id: 'uo.rate.skillgain', @@ -898,6 +910,16 @@ silently did nothing. - **`scope` is opaque and may be null.** Core passes it through and never interprets it. Core has **no concept of "the game being up"** — only `{ ok: false, retry: true }` — because a module with six sidecars cannot answer that question in the singular. +- **A module says when to reconcile, because core cannot.** §L's reconnect row asks each ledgered + resource's module what is still in force, and the trigger for that question is `ctx.events.reconcile()` + — the module's own call, made when it sees its own reconnect (module-uo already watches `bootId` to + tell a shard restart from a sidecar reconnect). Core has **no concept of the game being up**, which + is the same sentence three bullets down, so the alternative would be core guessing. Core asks once + at its own boot, which is the one reconnect it can see. `reconcile()` is OPTIONAL where `revert` is + required: a module that cannot answer leaves core believing its own ledger, which is the behaviour + before this phase, while a module that created something and cannot undo it has made a promise core + has no way to keep. Anything that is not an explicit `{ ok: true, inForce: [...] }` changes nothing: + **"I do not know" is never read as "it is gone"**. - **An action whose module is uninstalled goes dormant, never an error.** A step naming it fails `terminal` with the module named and the run degrades — never a silent skip. The authoring side draws the same line one step earlier, in the shape `engagement_rules` established for a dormant @@ -1123,9 +1145,12 @@ dimensions' labels. `check:modules` already fails core's build on a UO identifie > advance. It satisfies the gate and stops; the next tick performs the phase boundary, exactly as it > does after `resume`, so there is one implementation of what a phase boundary is rather than two. > -> **Cancel takes `{ reason }` and not `{ cleanup }`**, because the resource ledger a cleanup would -> work over arrives in Phase 8; a flag that changes nothing is the "control that answers 200 and does -> nothing" this plan has refused twice already. +> **All eight exist as of Phase 8, and cancel gained its `{ cleanup }`** now that there is a ledger +> for it to work over — a flag that changed nothing would have been the "control that answers 200 and +> does nothing" this plan has refused twice already, which is why it waited. It defaults to TRUE: +> §L makes cancelling *without* cleanup the separate, admin-only, logged action, so the safe direction +> is what a moderator's cancel does without having to know the flag exists. `cleanup` itself is the +> eighth control, and the one gated NARROWER than the rest — see [§K](#k--security-model). > **Retry is one control, not two.** A step may be retried only while its run is `paused`, and a > paused run is paused *at* that step — so re-queueing without resuming would leave the run in @@ -1235,7 +1260,7 @@ decided in [§N2](#n--decisions): starting a run is `admin` only, while stopping | Authoring and editing a draft | `admin`, `editor` | | Publishing a version, **starting a run** | `admin` only | | **Live control of a run in flight** — cancel, abort | `admin`, `moderator` | -| Any step whose action **changes the world** (`change`, `irreversible`), and the action switchboard | `admin` only | +| Any step whose action **changes the world** (`change`, `irreversible`), the action switchboard, and **re-running cleanup** | `admin` only | > **Why start and stop are gated differently.** Starting commits the deployment to everything the > definition contains, unattended, up to every cap it declares — it wants the narrowest gate there is. @@ -1332,10 +1357,10 @@ controller stamps it from the session. | **A step names an action this deployment has disabled** | The same `refused`, in the same words as a cap breach, with the reason naming the action by its label. It is not an authoring mistake — the plan was legal when it was written — so the switchboard is the fix, and a disabled action reached by a run in flight is exactly the case the dry run catches beforehand. Phase 6. | | **A scheduled occurrence of a version nobody dry-ran** | Not materialised at all, and logged once per definition per tick rather than once per occurrence — there is no run row to hang a diagnostic on, which is the point. The definition itself carries the fact, so the editor says so where its author is already looking. §K, Phase 6. | | **An action fails** | Per-step `on_failure`, defaulted from the risk class: `retry(n) → skip` for `notify`, `retry(n) → pause` for `change`, `retry(n) → abort_run` for `irreversible`. `pause` stops the run advancing and waits for a human — the right default when the world is half-changed. `n` is `EVENT_STEP_MAX_ATTEMPTS`, 3 by default. **All three dispositions write the STEP `failed`**: `on_failure` says what happens to the run, and a step attempted three times that never worked is `failed` under every one of them. `skipped` is reserved for a step a human skipped from the run console — a status meaning both "nobody ran this" and "this failed and we moved on" would make the console's summary line unreadable. | -| **A run is cancelled** | Pending steps `cancelled`; a running one is left to finish or time out (nothing can recall a sent command); cleanup steps are generated from the ledger and run. Cancelling *without* cleanup is a separate, logged, admin-only action. **A PARKED step is cancelled with the pending ones** (Phase 3): a cue is not a command already sent, it is an instruction nobody is holding, and leaving it `running` would have the console claim a cancelled event is still waiting for someone. The live lease is what tells the two apart, and it is in the `WHERE` clause. | +| **A run is cancelled** | Pending steps `cancelled`; a running one is left to finish or time out (nothing can recall a sent command); the ledger is then swept by the runner's cleanup leg on its next tick — the request does not wait for it, because a cancel pressed at 2am must answer at once rather than after a round trip per resource to a shard that may be why it is being pressed. Cancelling *without* cleanup is a separate, logged, admin-only action (`{ cleanup: false }`), and it leaves the run `cleanup_status: 'incomplete'` with every unreverted row on the console — the truthful value, because the world changes really are still up. **A PARKED step is cancelled with the pending ones** (Phase 3): a cue is not a command already sent, it is an instruction nobody is holding, and leaving it `running` would have the console claim a cancelled event is still waiting for someone. The live lease is what tells the two apart, and it is in the `WHERE` clause. | | **A human skips a step** | `skipped`, which is what that status is reserved for. Legal for a `pending` step and for a parked cue; refused for a step with a live lease, and unnecessary for a `failed` one — `nextOpenStep` already passes over it, so resuming the run carries the phase past it. | | **A human retries a step** | Only while the run is `paused`, and only on the furthest step its phase has reached. `attempts` returns to zero and the run resumes in the same action; both facts are in the log line with the actor. | -| **Cleanup itself fails** | The run reaches `completed` with `cleanup_status = 'incomplete'`, the unreverted resources listed and a manual retry offered. It does **not** stay `running` — an event whose world changes are still up is a real state, and pretending the event is in progress hides it. | +| **Cleanup itself fails** | The run reaches `completed` with `cleanup_status = 'incomplete'`, the unreverted resources listed and a manual retry offered. It does **not** stay `running` — an event whose world changes are still up is a real state, and pretending the event is in progress hides it. The automatic sweep stops after `EVENT_REVERT_MAX_ATTEMPTS` and leaves the rows for a person; `POST …/cleanup` is the only thing that clears the counter, with the actor logged. | ### The ledger's two rules @@ -1349,6 +1374,97 @@ twice. One that never succeeds stays visible until a human acts, because the alt `completed` row over a shard full of orphaned monsters — is the failure that would end this feature's credibility on its first bad night. +### What Phase 8 settled about the ledger + +**Rule 1 needed a mechanism, because a spawn's `ref` does not exist until the module answers.** What +goes in before the dispatch is a PLACEHOLDER keyed by the step's idempotency key — the reserved +`kind: '@step'`, `ref` = that key — and the reported resources are inserted `confirmed` on the answer, +resolving it. A dispatch that never answers leaves the placeholder standing, and cleanup then calls +`revert({ idempotencyKey, resources: [] })`. **That is why [§F](#f--the-module-contract)'s `revert` +takes the key at all**, and it is the case rule 1 exists for: record afterwards instead, and a +timed-out spawn is twelve creatures no cleanup will ever see. The placeholder is written AFTER the +permission check and BEFORE the module is reached — a refused step created nothing and must ledger +nothing. + +**A LEASE does not use the placeholder, and gets rule 1 in a stronger form.** Its target is knowable +before the dispatch — it is the lease id the step names — so `core.lease` writes the real row first. +That is also the only moment the two-events-one-target refusal can happen *before* the world has been +written to; a second run asking for a held lease comes back `refused`, in the same words a cap breach +uses and for the same reason. + +**Recording is idempotent because the database makes it so.** A retry re-sends the same idempotency +key and a module may honestly re-report the same resources; `uq_evres_target` refuses the second +insert and core reads that as "already recorded" rather than as an error — `materialisePhase`'s +INSERT IGNORE posture, one table along. + +**The unique key is held by three statuses and released by three** (amended 2026-09-03). The row +above said *"among non-reverted rows"*, which was written before the six statuses had their meanings; +taken literally it makes `drifted` and `orphaned` hold a target for ever, so one bad night would +disable a lease permanently with no control able to clear it. `pending`, `confirmed` and `reverting` +mean core still believes the resource is this run's, and hold the target. `reverted` gave it back, +`drifted` means somebody else has hold of the value and this run has deliberately let go, `orphaned` +means it vanished — none of the three is a claim, and all three stay LOUD by the other mechanism: +`cleanup_status = 'incomplete'` and a row on the run console, which is what rule 2 actually asks for. +MariaDB has no partial index, so the encoding is a STORED generated column that is NULL for the last +three; it reads `status` ALONE, because MariaDB refuses `ON DELETE SET NULL` on a foreign key whose +column is a base column of a stored generated column (error 1901) and `step_id` must stay SET NULL — +the same correction `TEAMS.md` §2.5 had to be given. + +**Cleanup is one sweep, not a set of synthetic steps.** The tempting reading of *"cleanup steps are +generated from the ledger"* is a synthetic phase of real `event_run_steps` rows, so the console's +per-step retry comes free. It is the wrong shape here for a concrete reason: `event_run_resources` +already carries `revert_attempts` and `last_error`, so synthetic steps would put a second retry +counter beside the first and the two would disagree the first time a step reverted three of its four +resources. The manual retry is a route over the ledger — `POST /admin/events/runs/:runId/cleanup` — +which is what the § API surface already promised. + +**It runs from ONE place: a fifth leg of the runner's tick.** Hooking each terminal path instead +would be four call sites, three of them inside a request, and none of them would survive the process +dying mid-teardown. The leg is ordered AFTER advance, so a run that completes in one tick is torn +down in the same one. `cancel` therefore answers at once and does not wait for the teardown — which +is also the right behaviour for a control pressed at two in the morning against a shard that may be +the reason it is being pressed. + +**What that leg SELECTS cost two live-walk defects, in opposite directions, and the pair is the +finding worth keeping.** Keying the scan on `cleanup_status = 'pending'` is the obvious reading of +the column, and it was wrong twice on the same boot: + +- A run whose only resource was a **lease** never went through the ledger's dirty-marking at all — + `core.lease` reserves its own row rather than reporting one — so the column stayed `not_required`, + the leg never looked at the run, and **the lease was never given back**. The scan now includes + `not_required`: a terminal run with an unresolved row has work to do whatever any summary column + says, and treating that combination as work is the fail-safe direction. +- A run whose first sweep FAILED was moved to `incomplete` **by that very sweep**, so it was never + picked up again and `EVENT_REVERT_MAX_ATTEMPTS` quietly meant one attempt rather than three. So + `incomplete` is now written only once there is nothing left to try — it means *"finished with, and + not finished"* — which is also what makes it the honest marker for a run an admin cancelled + **without** cleanup. + +Both are the same mistake: deriving *"is there anything to do"* from a summary column instead of from +the rows that are the fact. Neither was visible to a unit test, because a test that calls the sweep +directly never asks what would have selected the run. + +**And the first fix for the second one made the console lie.** Spending every row's +`revert_attempts` was a tidy way to take a `cleanup: false` run out of a scan bounded by the counter +— and the run page then rendered *"3 attempts"* beside resources nothing had ever tried, which reads +as *"core tried three times and could not"*. Found by opening the page. **A counter that means two +things is a counter a screen cannot render**, so the status column carries that decision instead. + +**Only a human clears `revert_attempts`.** The automatic sweep bounds itself at +`EVENT_REVERT_MAX_ATTEMPTS` and then leaves the rows for a person; the cleanup route resets the +counter and asks again, with the actor on the log line. This is Engagement Phase 14's rule stated a +third time — a sweep that returned every stale row to its start state made the attempt ceiling +unreachable, so the row cycled for ever and was never eligible for any retention sweep — and the +licence a human has here is the same one their step retry has. + +**Drift is a third outcome, not a failure with a flag.** The module did exactly what it was asked and +found somebody else's value in place, so `{ ok: false, drifted: true, current }` produces a `drifted` +row rather than a retry: asking again would find the same thing, and writing anyway would silently +revert an operator's manual fix. And **`orphaned` is only ever reached through reconcile**, never +through a revert — a revert that finds nothing there is a SUCCESS (§L, and what a Rust wipe needs), +whereas a resource the module reports missing is a thing that vanished while nobody was looking. +Those are two different sentences to the operator reading the console the morning after. + --- ## Versioning, and editing a live event @@ -1388,10 +1504,10 @@ no URL moved. | `GET /admin/events/runs` | staff | run history across definitions | | `GET /admin/events/runs/:runId` | staff | status, phase, steps, caps, resources, cleanup | | `GET /admin/events/runs/:runId/log` | staff | the diagnostic log | -| `POST /admin/events/runs/:runId/pause\|resume\|cancel` | admin, moderator | live control of a run in flight (Phase 3); `cancel` takes `{ reason }`, and gains `cleanup` with the ledger in Phase 8 | +| `POST /admin/events/runs/:runId/pause\|resume\|cancel` | admin, moderator | live control of a run in flight (Phase 3); `cancel` takes `{ reason }` and, since Phase 8, `{ cleanup }` — **defaulting to true**, so §L's *"cancelling WITHOUT cleanup is a separate, logged, admin-only action"* is the flag that has to be asked for. `cleanup: false` is refused to a moderator in the MODEL rather than in middleware, because which of the two roles you have to be depends on what is in the body. Either way the request answers at once: the teardown is the runner's cleanup leg, not this call | | `POST /admin/events/runs/:runId/advance` | admin, moderator | force the current phase past its advance condition (Phase 5). Legal only while the phase is genuinely waiting on its gate; `409` naming what it is waiting on otherwise. Satisfies the gate and stops — the next tick performs the boundary | | `POST /admin/events/runs/:runId/steps/:stepId/skip\|retry\|confirm` | admin, moderator | Phase 3. `confirm` resolves a GM cue step; `retry` re-queues the step a paused run is stopped at and resumes it | -| `POST /admin/events/runs/:runId/cleanup` | admin | re-run cleanup over unreverted resources | +| `POST /admin/events/runs/:runId/cleanup` | admin | re-run cleanup over unreverted resources (Phase 8). **`admin` rather than admin+moderator, unlike the seven live controls beside it**: this is not incident response, it asks core to write to the world again, which §K puts in the same row as the world-changing actions themselves. Legal on a TERMINAL run only — a run still in flight has a ledger that is still growing. It is the only thing that clears `revert_attempts`, and it answers **200 whatever it found**, for the dry run's reason: some resources may still be out there, and a 4xx would make that indistinguishable from a bad run id | | `GET /admin/events/catalog` | staff | registered actions, param schemas, risk classes, budget dimensions — and, since Phase 5, the **trigger** catalog and condition operators the advance form renders. Phase 7 added `budgets`, `leases` and `optionSources`: the other three registrations of the module contract, served beside the actions because the step editor needs all four to draw ONE step, and four requests to draw one form would be four chances to render half of it. Served here rather than borrowed from `/admin/engagement/triggers` because that route is `adminOnly` while a definition is authored by `admin` **and** `editor` | | `GET /admin/events/catalog/options/:sourceId` | staff | **the values behind a param's `source`** (Phase 7) — resolved by the module that registered it, on its own request rather than inside the catalog, because a source can be slow or down and must not take the catalog with it. A refusal is a **`200` with `ok: false` and a `reason`**, deliberately: the field degrades to free text with a warning rather than the form breaking | | `GET /admin/events/series` | staff | the arcs a definition may belong to | @@ -1440,11 +1556,18 @@ runner ticks every fifteen seconds, so a console rendered thirty seconds ago des since moved, and a control that checked in JavaScript and then wrote would race the tick it exists to interrupt. A refusal is a `409` naming the status the run is actually in. -**Phase 7 filled the option-source route, and the list of absent routes is now one item long — -`POST /admin/events/runs/:runId/cleanup`, which arrives with the ledger in Phase 8.** The route -answers a refusal with a `200`, which is the one thing about it worth stating twice: §F requires a -source that cannot answer to degrade its field rather than block the form, and a status code is how a -client tells the two apart. +**Phase 7 filled the option-source route, and Phase 8 filled the last one. Nothing in this table is +absent any more.** The option-source route answers a refusal with a `200`, which is the one thing +about it worth stating twice: §F requires a source that cannot answer to degrade its field rather +than block the form, and a status code is how a client tells the two apart. + +**Phase 8 added one route and one body field, and neither moved anything.** `cleanup` is the manual +retry; `cancel`'s `{ cleanup }` is the choice about what a cancelled run leaves behind. A run's +detail gained a fifth and sixth derived field: **`resources`**, the whole ledger with the `@step` +placeholders filtered out — they are core's own bookkeeping and a list of them would read as +resources nobody can name — and **`unresolvedResources`**, counted over the whole ledger *including* +the placeholders, which is why it can exceed the length of the list. That is deliberate: a run whose +answer was lost has something unresolved and nothing to show, and the console says exactly that. **Phase 6 added the three before it.** `GET/PUT /admin/events/actions` is the switchboard, `admin` in **both** directions — §K puts it in the same @@ -1464,8 +1587,8 @@ a draft would be a claim about a spec that changes under the author's hands. **F with a 200**: the request succeeded, the plan has problems, and answering 4xx would make "this event asks for 45 and you allow 30" indistinguishable from "you sent a bad event id". -Still absent, and still for a reason rather than for scope: `cleanup` and cancel's `cleanup` flag, -because there is no resource ledger until Phase 8. `advance` left this list in Phase 5. +Nothing is absent from this list any more: `advance` left it in Phase 5, `cleanup` and cancel's +`cleanup` flag in Phase 8. > **The whole-plan cost check is the one thing no other path makes.** Every per-step check the dry run > performs is also made at save or at dispatch; the TOTAL is not. Three steps each spawning 15 under a @@ -1523,7 +1646,7 @@ Phase 3 — "The Boss" has not started. | What a phase is waiting for, its tally and its last related firing | `event_run_phase_gates`, served already-rendered as a run's `gates` | | A phase opening a gate, and a gate opening — on a firing, a deadline or a human | `event_run_log`, kinds `phase.gate` and `phase.advanced` | | Module acknowledgement, or its absence with the budget exceeded | `event_run_steps.last_error` | -| Resources created, confirmed, leased, reverted, orphaned, drifted | `event_run_resources` | +| Resources created, confirmed, leased, reverted, orphaned, drifted | `event_run_resources`, plus six `event_run_log` kinds: `resource.recorded`, `resource.orphaned`, `cleanup.reverted`, `cleanup.failed`, `cleanup.swept`, `cleanup.retry`. `resource.recorded` is written at the ANSWER rather than at the placeholder, because a placeholder is a promise and the operator's question is about the world | One caution carried over from the engagement retention work: the run log is high-cardinality and grows per event, so it needs a retention sweep from the start — `engagementRetentionPrune` is the pattern, and the rule it learned is @@ -1555,7 +1678,7 @@ and needed no answer to [N1](#n--decisions). P11 and P12 were the gated pair; ** | **P5** ✓ | Conditions, phase advancement, and the "why didn't phase 3 start?" panel | `website` `docs` | | **P6** ✓ | Enablement, per-run caps, and the single `mayInvoke` decision point | `website` `docs` | | **P7** ✓ | The module contract — MODULE_API 1.10.0, proved with a throwaway module | `website` `docs` | -| **P8** | The resource ledger, leases and generated cleanup | `website` `docs` | +| **P8** ✓ | The resource ledger, leases and generated cleanup | `website` `docs` | | **P9** | UO wave 1 — the actions that need no protocol change | `module-uo` `docs` | | **P10** | Integrations — the `event.` triggers, participants, results, announce legs | `website` `docs` | | **P11** | *N1 answered.* Protocol: idempotency key, lease deadline, participation ledger | `servuo-plugins` `link` `module-uo` `installer` `docs` | diff --git a/website/EVENTS_PLAN.md b/website/EVENTS_PLAN.md index 0ee4536..016e0b8 100644 --- a/website/EVENTS_PLAN.md +++ b/website/EVENTS_PLAN.md @@ -866,23 +866,146 @@ it was carved out of has not been validated, and P9 should be the *second* consu ### Phase 8 — The resource ledger, leases and cleanup (`website` + `docs`) -`event_run_resources` with `lease_until`, the `drifted` status, and -`UNIQUE (owner_module, kind, ref)` among non-reverted rows — which is what makes two events unable to -lease one target, and produces a `refused` step at authoring time rather than a corrupted baseline at -runtime. +> **Complete.** `edge` in `website` and `docs`. One table (`event_run_resources`), one core action +> (`core.lease`) and one option source (`core.options.leases`), one route +> (`POST /admin/events/runs/:runId/cleanup`), one body field (`cancel`'s `{ cleanup }`), and two +> members added to MODULE_API **1.10.0 in place** — `reconcile()` on an action and +> `ctx.events.reconcile()` on the module context. +> +> **The version was amended rather than bumped** (org lead, 2026-09-03). A protocol owes a bump once +> it has landed on `main`; while it is on `edge` it is amended in place — the rule the Teams +> workstream arrived at, applied to a module API for the first time. 1.10.0 has not shipped, so the +> whole module contract reaches an author as one version they read once, which was the argument for +> putting the lease declaration in it a phase early. +> +> **The four decisions the org lead settled (2026-09-03), all as recommended:** +> +> - **A lease is acquired by a new CORE action, `core.lease`.** §F puts the duration bound and the +> two-events-one-target conflict check on core's side of the seam, and a lease verb per module +> would be both of those re-implemented once per module — advisory everywhere, and wrong in the +> first one that forgot. It is `risk: 'change'`, so it is the first core action that is default-off, +> admin-only and cap-checked like any module verb. +> - **Record-before-confirm is a PLACEHOLDER keyed by the step's idempotency key.** A spawn's `ref` +> does not exist until the module answers, so there is nothing to write a row about — the row core +> writes beforehand is `kind: '@step'`, `ref` = that key. On the answer the reported resources are +> inserted `confirmed` and the placeholder is resolved; if the answer never comes it stands, and +> cleanup calls `revert({ idempotencyKey, resources: [] })`. **That is why §F's `revert` takes the +> key at all.** A lease skips it and reserves its real target instead, which is the same rule in a +> stronger form. +> - **Cleanup is one sweep over the ledger, not synthetic `event_run_steps` rows.** The step-shaped +> version buys the console's per-step retry for free and costs a second retry counter beside +> `revert_attempts` — two counters that disagree the first time a step reverts three of its four +> resources. The manual retry is the route this plan already promised. +> - **`reconcile` is declared here and TRIGGERED BY THE MODULE**, through `ctx.events.reconcile()`. +> Core has no concept of the game being up (§F), so it cannot decide when to ask; module-uo already +> watches `bootId` to tell a shard restart from a sidecar reconnect, and that is the moment. Core +> asks once at its own boot, which is the one reconnect it can see. +> +> **Four things the build settled:** +> +> - **The unique key is held by three statuses and released by three**, which corrects §D's *"among +> non-reverted rows"* — written before the six statuses had their meanings. Taken literally it makes +> `drifted` and `orphaned` hold a target for ever, so one bad night would disable a lease +> permanently with no control able to clear it. `drifted` means somebody else has hold of the value +> and this run has let go; `orphaned` means it vanished. Neither is a claim, and both stay loud +> through `cleanup_status` and the console instead, which is what rule 2 actually asks for. +> Recorded as a dated amendment in `EVENTS.md`. +> - **MariaDB has no partial index, so the key is a STORED generated column that is NULL once the row +> is no longer ours** — and it reads `status` ALONE. `TEAMS.md` §2.5 had to be corrected on this +> exact shape: MariaDB refuses `ON DELETE SET NULL` on a foreign key whose column is a base column +> of a stored generated column (error 1901), and `step_id` must stay SET NULL because a record of +> what was changed in the world has to outlive the row that scheduled it. +> - **Cleanup runs from ONE place: a fifth leg of the runner's tick**, ordered after advance so a run +> that completes in one tick is torn down in the same one. Hooking each terminal path would be four +> call sites, three of them inside a request, and none would survive a process that died +> mid-teardown. It is also why `cancel` answers at once — the right behaviour for a control pressed +> at 2am against a shard that may be the reason. +> - **What that leg SELECTS is the phase's real finding, and it took the live walk twice.** See +> below: the obvious reading of `cleanup_status` stranded a lease outright, and then made +> `EVENT_REVERT_MAX_ATTEMPTS` mean one attempt. +> +> **Verified:** `npm test` — **2025 tests, 1935 pass, 89 skipped, 1 fail**, that one still the +> pre-existing `engagementManifest.test.js` CRLF failure in a file this branch does not touch +> (`edge` before: 1950/1876/73/1). **+75 tests**: 14 in the new `eventLedger.test.js`, 20 in the new +> `eventCleanup.test.js`, 16 in `eventRunnerSql.test.js` (which skip without a database — see +> below), 8 in `eventRunner.test.js`, 9 in `eventRunControls.test.js`, 5 in +> `eventModuleContract.test.js`, 2 in `eventActionRegistry.test.js` and 1 in `eventsRoles.test.js`. +> Client: **362 pass**; the client builds. `npm run routes:manifest` and `npm run swagger` — **one +> route added, none moved.** +> +> **The unique key was proved against a real MariaDB, because nothing else can prove it.** Whether +> multiple NULLs collide in a unique index, whether a STORED generated column is recomputed and +> re-indexed on UPDATE, and whether the SET NULL foreign key survives beside it are properties of the +> server and of nothing else. `eventRunnerSql.test.js` gained 14 tests covering all three, plus the +> ledger model's own statements: `DB_HOST=127.0.0.1 DB_PORT=3307 DB_USER=root DB_PASSWORD=… node +> --test test/eventRunnerSql.test.js` — **63 pass**. The real `schema.sql` was also applied to a fresh +> database *and* to an existing one, because `CREATE TABLE IF NOT EXISTS` next to a generated column +> is where a migration silently does nothing. +> +> **And that file had a latent bug this phase found.** Its model-backed tests required +> `eventRunBudget.db`, whose pool `utils/db` builds at require time from `DB_NAME` — and `utils/db`'s +> own `dotenv.config()` reads `server/.env`. So on a developer's machine those tests were reaching +> that developer's real schema while the fixtures they asserted against were being written into the +> throwaway database next door. They passed only because both tables happened to exist in both. +> `process.env.DB_NAME = DB` before the require fixes it, and the whole run is disposable again. +> +> **The stale-stub trap, for the fourth time in this feature.** Phase 4's expansion leg, Phase 5's +> gate read, Phase 6's settings read and now the ledger write: a new leg under a model needs a stub in +> every file that stubs that layer, and unstubbed it is not a wrong answer — it is a ten-second +> `ECONNREFUSED` against the dead port. One missing stub cost `eventsAdmin.test.js`'s run-detail test +> ten seconds and said nothing about the route it was testing. +> +> ### The live walk, and the three defects only it could find +> +> Driven by a throwaway `rig` module in `website/modules/` — a real module directory scanned by the +> real loader, **deleted before commit** — registering one ledgering action and one lease, with its +> behaviour driven by a JSON file on disk so a revert could be made to fail and then succeed on the +> same boot. 34 assertions, all green at the end. `rig` declared `coreApi: "^1.10.0"` and module-uo's +> `"^1.9.0"` still resolved, on the same boot: the additive claim proved rather than asserted. +> +> **Defect 1 — a lease was never given back at all.** `core.lease` reserves its own ledger row, so it +> never went through the ledger's dirty-marking, so a run holding only a lease kept +> `cleanup_status = 'not_required'` and the cleanup leg — which selected on `pending` — never looked +> at it. Every unit test passed: they call the sweep directly, and a test that calls the sweep never +> asks what would have SELECTED the run. +> +> **Defect 2 — `EVENT_REVERT_MAX_ATTEMPTS` meant one attempt, not three.** The first failing sweep +> moved the run to `incomplete`, which took it out of the leg's own scan for ever. Visible only as +> `revert_attempts` sitting at 1 through half a minute of live ticks; the test that covered the bound +> asserted `<= 3` and was satisfied by 1. **A bound has two halves — it retried, and it stopped — and +> a test that only asserts the ceiling passes against a floor.** Both halves are asserted now. +> +> **Defect 3 — the first fix for defect 2 made the console lie.** Spending every row's +> `revert_attempts` was a tidy way to take a `cleanup: false` run out of a counter-bounded scan, and +> the run page then rendered *"3 attempts"* beside resources nothing had ever tried — which reads as +> *"core tried three times and could not"*. Found by opening the page, exactly like Phase 7's React +> defect. **A counter that means two things is a counter a screen cannot render.** +> +> **What the walk proved, beyond the defects:** three wisps recorded and given back with the run +> reaching `complete`; a failing revert leaving the run `completed` + `incomplete` with the reason on +> each row; the cleanup route rescuing it and a moderator being refused it; the route refusing a run +> still in flight and one that recorded nothing; a lease taken, its baseline and applied value in the +> payload, and a **second run refused it by name**; a GM's mid-event edit producing `drifted` with the +> world left alone, and a later run still able to lease that target; and `cancel { cleanup: false }` +> refused to a moderator, allowed to an admin, and logged. +> +> **The two that needed the process to die**, which is the plan's own verify line. With the module's +> `perform()` hanging: the placeholder existed while the dispatch was in flight, **nothing was named** +> (`unresolvedResources: 1`, an empty list — rule 1 visible in the wild), and after `taskkill` and a +> restart the reclaim re-dispatched the SAME idempotency key, the retry re-used its own placeholder +> rather than writing a second, and everything was given back. Then, with the module reporting one of +> two resources as no longer in force, core's boot-time `reconcileAll()` marked the other `orphaned` +> — never `reverted` — and logged it. +> +> **The console, in all three states.** A clean run: neutral border, *"Everything this run created or +> borrowed has been given back"*, green rows, no button. An unresolved run: amber border, *"3 of these +> are still unresolved"*, *"still out there"*, and **Try cleanup again** — which was clicked, answered +> `200`, and flipped the panel green in place. A drifted lease: *"someone else moved it"*, its +> deadline, its genuine attempt count, and the sentence naming the value that is there now. +> +> **Ships:** the safety property the whole world-write half depends on. Also useful on its own — the +> platform gains a durable record of what it changed. -**Record before confirm.** The step writes a `pending` row, dispatches, and promotes it on the answer. -Recording afterwards makes every object whose acknowledgement was lost invisible to cleanup forever — -so reverting a resource that does not exist must be a *success*. - -Cleanup steps are **generated from the ledger** at teardown, on every terminal path — completion, -cancellation and abort alike. `cleanup_status` is its own column: a run reaches `completed` with -`cleanup_status = 'incomplete'` and stays on the admin screen, rather than being held `running`. - -Reconcile-on-reconnect: the runner asks each ledgered resource's module what is still in force. - -**Ships:** the safety property the whole world-write half depends on. Also useful on its own — the -platform gains a durable record of what it changed. **Verify:** `npm test`; a rig run that kills the process mid-run and confirms cleanup completes on restart; a run whose revert fails and stays visible. diff --git a/website/MODULE_API.md b/website/MODULE_API.md index 662d015..f14a8c6 100644 --- a/website/MODULE_API.md +++ b/website/MODULE_API.md @@ -35,12 +35,19 @@ module chunk evaluates, which is earlier than any network round trip could answe **1.10.0 — the event contract opens to modules: `api.registerEventActions(...)`, `api.registerEventBudgets(...)`, `api.registerEventLeases(...)` and -`api.registerEventOptionSources(...)`** (`website/EVENTS.md` §F, `EVENTS_PLAN.md` Phase 7). Four -additions and no removal, so minor; a module written against 1.9.0 registers no actions and its +`api.registerEventOptionSources(...)`** (`website/EVENTS.md` §F, `EVENTS_PLAN.md` Phases 7 and 8). +Four additions and no removal, so minor; a module written against 1.9.0 registers no actions and its deployment simply has fewer verbs an event can use — which is §F's own posture stated as a version rule, because core with none of this installed is still an event engine that can announce, wait, cue a human and publish results. +> **Phase 8 added `reconcile()` and `ctx.events.reconcile()` to this same version rather than to a +> new one** (org lead, 2026-09-03). A protocol owes a bump once it has landed on `main`; while it is +> on `edge` it is amended in place — the rule the Teams workstream arrived at, applied to a module +> API for the first time. 1.10.0 has not shipped, so the whole module contract reaches an author as +> one version they read once, which was the argument for putting the lease declaration here in the +> first place. + **Only one of the four is new machinery.** The ACTION registry has staged core's `core.announce`, `core.wait` and `core.cue` on every boot since Events Phase 1; what it never had was a way in — `loader.js` built its own `api` facade and had no method that delegated to it. So the seam a module @@ -70,15 +77,31 @@ api.registerEventActions([{ if (verify) return { ok: true } // dry run: validate, change NOTHING return { ok: true, resources: [{ kind: 'creature', ref: '0x40001234' }] } }, - async revert({ runId, resources, idempotencyKey }) { return { ok: true } }, // iff reversible: 'ledger' + // Required iff reversible: 'ledger'. Called by core's cleanup sweep at teardown, + // over the rows this action's `resources` produced — a LIST, so twelve creatures + // are one round trip. `{ ok: true }` reverts the group; `failed: ['0x...']` names + // the ones that did not come back. + async revert({ runId, resources, idempotencyKey }) { return { ok: true } }, + // OPTIONAL, and only on an action that ledgers. "Which of these does the game + // still have?" — asked after something outside core restarted. + async reconcile({ runId, resources }) { return { ok: true, inForce: ['0x40001234'] } }, }]) +// The module says WHEN, because core cannot: core has no concept of the game +// being up. module-uo already watches `bootId` to tell a shard restart from a +// sidecar reconnect, and that is the moment a ledger of live spawns has become a +// claim about a world that no longer exists. +ctx.events.reconcile() + api.registerEventOptionSources([{ id: 'uo.options.creatures', label: 'Creatures', async resolve() { return [{ value: 'Orc', label: 'Orc', group: 'Humanoid' }] }, }]) -api.registerEventLeases([{ // DECLARED here; acquired by nothing yet +// A value a run may borrow. The module ships the three callables; the VERB an +// author puts in a step is core's `core.lease`, so the duration bound and the +// two-events-one-target conflict check live in one place. +api.registerEventLeases([{ id: 'uo.rate.skillgain', label: 'Skill gain rate', type: 'float', min: 0.5, max: 5, maxDurationMs: 86400000, async read() { return { ok: true, value: 1.0 } }, @@ -109,9 +132,31 @@ field: `.`. An action names a VERB, a budget a RESOURCE, a lease a VALUE and an option source a CATALOG, so `uo.creatures` may legitimately appear in more than one — reading that as a collision would forbid the most natural set of names a module will ever write. -- **A lease is DECLARED at 1.10.0 and acquired by nothing.** Core owns a lease's duration and its - conflict check, and both live in the resource ledger, which is `EVENTS_PLAN.md` Phase 8. It is in - this version rather than the next so the module contract is one version an author reads once. +- **A lease is declared by a module and acquired by CORE.** The verb is `core.lease`, and the + module never writes one: core reads the baseline, reserves the target in the resource ledger — + which is where "two events cannot hold one target" comes from, as a unique index rather than as a + check — applies the value with the deadline, and restores it at teardown through the module's own + `restore()`. A lease verb per module would be that bound re-implemented once per module, advisory + everywhere, and wrong in the first one that forgot it. +- **`until` goes down the wire, and the game side must honour it without being asked again.** A + module that treats it as advisory has produced a lease that outlives an outage, which is the one + thing a lease exists to prevent. Core's copy of the deadline is for the console; the game's copy is + the fail-safe. +- **`revert` must be idempotent, and reverting something that does not exist is a SUCCESS.** Core + records a resource BEFORE it is confirmed (`EVENTS.md` §D rule 1), so a dispatch whose answer was + lost leaves a row for something that may never have existed — and cleanup will ask about it. A + module never has to tell "I deleted it" from "it was not there". This is also what a Rust-style + monthly wipe needs, and the second reason a lease's restore must be idempotent too. +- **`revert` is also called with NO resources and only an idempotency key.** That is the lost-answer + case: core knows a dispatch went out under that key and never learned what it made. A module that + can undo by key answers honestly; one that cannot answers `{ ok: false }` and the row stays visible + to an operator, which is the correct outcome rather than a silent one. +- **`reconcile` is optional where `revert` is required**, and the asymmetry is the design. A module + that cannot say what the game still has is not broken — core keeps believing its own ledger, which + is the behaviour before this version — whereas one that created something and cannot undo it has + made a promise core has no way to keep. Anything that is not an explicit `{ ok: true, inForce: [...] }` + leaves the ledger alone: **"I do not know" is never read as "it is gone"**, and a resource a module + reports missing becomes `orphaned` rather than `reverted`, because nobody asked for it to go. **1.9.0 — a module may ship its own message bodies and rules: `api.registerEngagementSeeds(...)`** (`website/ENGAGEMENT.md` Phase 11b, decision 7). One addition and no removal, so minor; a module @@ -576,7 +621,7 @@ api.registerSlashCommands([{ name, description, options, access, handler }]) // api.registerEventTriggers([{ id, label, kind, subjectKey, audience, ceiling, version, variables }]) // 1.7.0 api.registerAudiences([{ id, label, params, ceiling, resolve }]) // 1.7.0 api.registerEngagementSeeds({ templates, ruleGroups }) // 1.9.0 -api.registerEventActions([{ id, label, risk, reversible, cost, params, perform, revert }]) // 1.10.0 +api.registerEventActions([{ id, label, risk, reversible, cost, params, perform, revert, reconcile }]) // 1.10.0 api.registerEventBudgets([{ id, label, unit }]) // 1.10.0 api.registerEventLeases([{ id, label, type, min, max, maxDurationMs, read, apply, restore }]) // 1.10.0 api.registerEventOptionSources([{ id, label, resolve }]) // 1.10.0 @@ -954,7 +999,7 @@ are in §1.1 under **1.9.0**; four things are contract rather than implementatio **`registerEventActions([...])` / `registerEventBudgets([...])` / `registerEventLeases([...])` / `registerEventOptionSources([...])`** (1.10.0) are the event contract (`EVENTS.md` §F). The full -shapes and the six rules that come with them are in §1.1 under **1.10.0**; four things are contract +shapes and the ten rules that come with them are in §1.1 under **1.10.0**; six things are contract rather than implementation and belong here: - **An action is core CALLING THE MODULE**, like `registerTeamProvider` and `registerAnnounceLeg`'s @@ -975,6 +1020,19 @@ rather than implementation and belong here: the authoring form and it never raises. The alternative is a screen a module's outage can take away, for a field whose value the operator very often already knows — which is a worse failure than the typo the dropdown exists to prevent. +- **Core records what an action made BEFORE the action is dispatched, not after** (`EVENTS.md` §D + rule 1). A module's `resources` are the refs core did not know until the answer arrived; what core + wrote beforehand is a placeholder keyed by the step's idempotency key, so a dispatch whose answer + never came back is still something cleanup can act on. The consequence for a module author is the + whole reason `revert` takes `idempotencyKey` as well as `resources`: it will sometimes be called + with the key and an EMPTY list, meaning *"a command went out under this key and core never learned + what it did"*. Answering that honestly is what makes an unattended world write recoverable; a + module that cannot answer it says so, and the row stays visible to an operator. +- **Core owns cleanup, and it is derived rather than authored.** There is no `on_teardown` on an + action and no cleanup phase in a spec: an operator cannot be relied on to write the undo, and an + aborted run never reaches the phase they wrote it in. Cleanup is one sweep over the ledger and it + runs on every terminal path — completion, cancellation and abort alike — so a module's only job is + to answer `revert` correctly however many times it is asked. **An action whose module is uninstalled goes dormant, never an error.** A step already in a saved spec keeps it and a new step may not add one — the shape `engagement_rules` established for a dormant