feat(bridge): protocol 6 — an idempotency key, and champ.boss.killed (Phase 11a) #21

wtclaude · 2026-09-04T23:00:34Z

wtclaude commented

2026-09-04 23:00:34 +00:00

EVENTS_PLAN.md Phase 11a, the plugin half. The first protocol bump that adds a promise rather than data, and the shard is where the whole promise lives — the sidecar's part is one constant and one status mapping.

A command carrying an idempotencyKey is executed at most once. A repeat is answered with the original reply.

Nothing on the wire could say that before, and EVENTS.md §A row 7 records what it cost: "a command whose ack is lost is indistinguishable from one that never applied." So every world-writing verb had to be declared un-retryable — Phase 9 shipped uo.broadcast answering retry: false to everything, including a 503 from a shard that was merely restarting. Sound reasoning, real trade, and not one that survives an event that spawns creatures or holds a lease. §G says so in as many words: required before any world write ships.

overlay.toml is protocol = 6 in this PR, not a later one.

The four rules, each a decision rather than an implementation detail

Reserve on receipt, not on completion. A handler that finishes inside its own inbound call can never see a repeat — the Core thread takes one line at a time — but a handler that defers completes long after OnInboundLine returned, and that is precisely the window a lost ack opens. A repeat of a key still in flight gets bridge.busy: nothing runs, come back. Deliberately not spelled bridge.busy.error — nothing is wrong, the work is happening.
A key that has begun is never released. Not even when the handler throws. Releasing it would let a retry re-run a command that may have applied half of itself, which is the exact failure this file exists to prevent. A throw stores a bridge.error instead, so the retry gets a definite answer and the step fails once rather than looping.
A replay is stamped with the REPEAT's correlation id. The sidecar's reqId is a fresh per-process counter, so a retry waits on an id the first attempt never used; replaying the original verbatim would hang the call until the reply timeout — the very failure being answered. reqId, code and id are all handled, and a repeat correlating on a different field than the original is refused rather than answered under an id nobody is waiting on.
The bound is loud. One hour, 4096 keys, drop-oldest. The cap is the guarantee's one hole — an evicted key's repeat would apply twice — so evicting a key still inside its TTL prints a warning naming the count. If the promise is ever breached an operator reads it here rather than finding a doubled spawn in the world.

The gate is in BridgeBoot's inbound dispatch rather than in each handler, so it covers every kind including the ones a later protocol adds. A command with no key behaves exactly as before, which is what leaves the admin screens unchanged.

How the reply is captured. Handlers emit through BridgeLink.Emit, so there is nothing to return. While a keyed handler runs, Emit offers each line to the store, which keeps only the line the sidecar would correlate with this command. That filter is the point: an admin.audit frame emitted alongside the reply is a fact about the world and must not be replayed; the reply is an answer to a caller and must be. Capture happens before Emit's _running check, too — a reply the link was too dead to deliver is exactly the one a retry comes back for. Replay rewrites the correlation field in the stored text rather than round-tripping through JavaScriptSerializer, so a replayed reply is byte-for-byte the original apart from the field that had to change.

`champ.boss.killed`

Rides along because a bump costs a release, a bundle and an operator update on every shard — a field left out costs a second bump, not a follow-up commit.

§G had a boss defeat as 🔗 inferable from champ.update losing bossUp next to a mob.killed, and called that "slightly fragile". It is worse than that: bossUp also drops when a GM resets a spawn, when a boss despawns, and after a reconnect clears the sweep's diff cache. And the inference is silent about who fought.

Fired from EventSink.CreatureDeath and detected by type (BaseChampion, which BaseSeaChampion derives from), so a boss that popped and died inside one sweep interval is still reported — the sweep's map supplies only the altar context. The damage table is read here because it exists here and nowhere else: ServUO discards a creature's damage entries with the creature. Totals are summed per damager rather than trusted to be one entry each, and entries are reported whether or not ServUO considers them expired — expiry governs looting rights, and someone who fought two thirds of the fight and then died took part in it.

The live walk, and the defect it found

Real ServUO (43k mobiles, 209k items), the release Rust sidecar, and a new tools/scaffolding/BridgeProtocol6Probe.cs.

Three POST /admin/broadcast, two under one key. Attempt 2 answered {"kind":"admin.ok","replayed":true,"reqId":"r-2","t":1788550182074} — the first attempt's t, so it is stored rather than re-run, under r-2 rather than the r-1 the first attempt used. The shard's own audit trail holds two rows, not three. The id-correlated plane was walked too: two POST /towncrier under one key, the second replayed and correlated on id.

A champion killed in the middle of Britain emitted a frame with no region. Three things were true at once, and the naive check confirms the bug:

An active ChampionSpawn registers a ChampionSpawnRegion over its spawn area — constructed with a null name and the town region as its Parent. So the most specific region containing a champion boss is, by construction, the one region on the map guaranteed to have no name.
Mobile.Region never returns null: with no cached region it falls back to the map's unnamed default. The failure looks identical to "outdoors, nowhere in particular".
Region registration is deferred, so a lookup immediately after the altar is placed still answers "Britain" and one at the kill twenty seconds later does not. The probe printed region "Britain" on every run, including the ones whose frame carried nothing. A read at spawn time would have confirmed the bug into the design.

Fixed by walking outward to the nearest named ancestor — general rather than a champion special case, since house regions, dungeon sub-regions and guarded-zone overlays are all anonymous children of somewhere a player would name. The map's own default terminates the walk, so open countryside answers null and the field is omitted: a consumer reading region: "" cannot tell "nowhere in particular" from "the shard would not say".

Final frame: boss: Semidar, serial: 0x400150E8 / type: Abyss / level: 0 (the altar, attributed), region: "Britain", damagers [("tester", 100240), ("Darrow", 120)] — descending, and tester's registered 240 folded into the 100000 killing blow as one entry rather than two.

One thing the rig could not prove

bridge.busy. With today's synchronous handlers a repeat cannot arrive while the original runs, so the state is unreachable on a live shard until a handler defers. It is implemented, BridgeIdempotency.Hold/Complete are the door a deferring handler uses, and the sidecar's 425 mapping is unit-tested on all three responders. Phase 11b's leases are the first thing that can produce it, and proving it belongs in that walk rather than being claimed here.

Verification

Compiles clean against the real ServUO 57.4 reference assemblies (msbuild Scripts.csproj, 0 errors) — built offline with the shard stopped, because a running ServUO holds Scripts.dll and an in-place build fails MSB3027 rather than reporting a compile error.
The live walk above, with the boot log's 0 Error(s) read each cycle to confirm the new code is live rather than trusting a clean boot.
[bridge status reports idem(keys= seen= replayed= busy= evicted= uncorrelated=).

Rig note, recorded in the scaffolding README: deploy.ps1 overwrites Bridge.cfg, so every deploy strips Protocol6ProbeOnStart and AdminWriteEnabled. The probe then silently does nothing on the next boot. The README already warned about this and it still cost a cycle.

AI-assisted: Claude Code (Opus 5).

Sidecar: RunicGateway/link#36 · Module: RunicGateway/Module-uo#29 · Docs: RunicGateway/docs#219

🤖 Generated with Claude Code

`EVENTS_PLAN.md` Phase 11a, the plugin half. **The first protocol bump that adds a promise rather than data**, and the shard is where the whole promise lives — the sidecar's part is one constant and one status mapping. > A command carrying an `idempotencyKey` is executed **at most once**. A repeat is answered with the original reply. Nothing on the wire could say that before, and `EVENTS.md` §A row 7 records what it cost: *"a command whose ack is lost is indistinguishable from one that never applied."* So every world-writing verb had to be declared un-retryable — Phase 9 shipped `uo.broadcast` answering `retry: false` to *everything*, including a 503 from a shard that was merely restarting. Sound reasoning, real trade, and not one that survives an event that spawns creatures or holds a lease. §G says so in as many words: **required before any world write ships.** `overlay.toml` is `protocol = 6` **in this PR**, not a later one. ## The four rules, each a decision rather than an implementation detail - **Reserve on receipt, not on completion.** A handler that finishes inside its own inbound call can never see a repeat — the Core thread takes one line at a time — but a handler that *defers* completes long after `OnInboundLine` returned, and that is precisely the window a lost ack opens. A repeat of a key still in flight gets **`bridge.busy`**: nothing runs, come back. Deliberately not spelled `bridge.busy.error` — nothing is wrong, the work is happening. - **A key that has begun is never released.** Not even when the handler throws. Releasing it would let a retry re-run a command that may have applied half of itself, which is the exact failure this file exists to prevent. A throw stores a `bridge.error` instead, so the retry gets a definite answer and the step fails once rather than looping. - **A replay is stamped with the REPEAT's correlation id.** The sidecar's `reqId` is a fresh per-process counter, so a retry waits on an id the first attempt never used; replaying the original verbatim would hang the call until the reply timeout — the very failure being answered. `reqId`, `code` and `id` are all handled, and a repeat correlating on a *different* field than the original is refused rather than answered under an id nobody is waiting on. - **The bound is loud.** One hour, 4096 keys, drop-oldest. The cap is the guarantee's **one hole** — an evicted key's repeat *would* apply twice — so evicting a key still inside its TTL prints a warning naming the count. If the promise is ever breached an operator reads it here rather than finding a doubled spawn in the world. The gate is in `BridgeBoot`'s inbound dispatch rather than in each handler, so it covers **every** kind including the ones a later protocol adds. A command with no key behaves exactly as before, which is what leaves the admin screens unchanged. **How the reply is captured.** Handlers emit through `BridgeLink.Emit`, so there is nothing to return. While a keyed handler runs, `Emit` offers each line to the store, which keeps **only the line the sidecar would correlate with this command**. That filter is the point: an `admin.audit` frame emitted alongside the reply is a fact about the world and must not be replayed; the reply is an answer to a caller and must be. Capture happens *before* `Emit`'s `_running` check, too — a reply the link was too dead to deliver is exactly the one a retry comes back for. Replay rewrites the correlation field in the stored text rather than round-tripping through `JavaScriptSerializer`, so a replayed reply is byte-for-byte the original apart from the field that had to change. ## `champ.boss.killed` Rides along because a bump costs a release, a bundle and an operator update on every shard — a field left out costs a second bump, not a follow-up commit. §G had a boss defeat as 🔗 *inferable* from `champ.update` losing `bossUp` next to a `mob.killed`, and called that *"slightly fragile"*. It is worse than that: `bossUp` also drops when a GM resets a spawn, when a boss despawns, and after a reconnect clears the sweep's diff cache. And the inference is silent about who fought. Fired from `EventSink.CreatureDeath` and detected **by type** (`BaseChampion`, which `BaseSeaChampion` derives from), so a boss that popped and died inside one sweep interval is still reported — the sweep's map supplies only the altar *context*. The damage table is read here because it exists here and nowhere else: ServUO discards a creature's damage entries with the creature. Totals are **summed per damager** rather than trusted to be one entry each, and entries are reported whether or not ServUO considers them expired — expiry governs looting rights, and someone who fought two thirds of the fight and then died took part in it. ## The live walk, and the defect it found Real ServUO (43k mobiles, 209k items), the release Rust sidecar, and a new `tools/scaffolding/BridgeProtocol6Probe.cs`. **Three `POST /admin/broadcast`, two under one key.** Attempt 2 answered `{"kind":"admin.ok","replayed":true,"reqId":"r-2","t":1788550182074}` — the **first** attempt's `t`, so it is stored rather than re-run, under `r-2` rather than the `r-1` the first attempt used. The shard's own audit trail holds **two** rows, not three. The `id`-correlated plane was walked too: two `POST /towncrier` under one key, the second `replayed` and correlated on `id`. **A champion killed in the middle of Britain emitted a frame with no `region`.** Three things were true at once, and the naive check *confirms* the bug: 1. An active `ChampionSpawn` registers a `ChampionSpawnRegion` over its spawn area — constructed with a **null name** and the town region as its `Parent`. So the most specific region containing a champion boss is, by construction, the one region on the map guaranteed to have no name. 2. `Mobile.Region` never returns null: with no cached region it falls back to the map's **unnamed default**. The failure looks identical to "outdoors, nowhere in particular". 3. Region registration is **deferred**, so a lookup immediately after the altar is placed still answers `"Britain"` and one at the kill twenty seconds later does not. **The probe printed `region "Britain"` on every run, including the ones whose frame carried nothing.** A read at spawn time would have confirmed the bug into the design. Fixed by walking outward to the nearest **named** ancestor — general rather than a champion special case, since house regions, dungeon sub-regions and guarded-zone overlays are all anonymous children of somewhere a player would name. The map's own default terminates the walk, so open countryside answers null and the field is **omitted**: a consumer reading `region: ""` cannot tell "nowhere in particular" from "the shard would not say". Final frame: `boss: Semidar`, `serial: 0x400150E8` / `type: Abyss` / `level: 0` (the altar, attributed), `region: "Britain"`, damagers `[("tester", 100240), ("Darrow", 120)]` — descending, and `tester`'s registered 240 folded into the 100000 killing blow as one entry rather than two. ## One thing the rig could not prove **`bridge.busy`.** With today's synchronous handlers a repeat cannot arrive while the original runs, so the state is unreachable on a live shard until a handler defers. It is implemented, `BridgeIdempotency.Hold`/`Complete` are the door a deferring handler uses, and the sidecar's 425 mapping is unit-tested on all three responders. **Phase 11b's leases are the first thing that can produce it**, and proving it belongs in that walk rather than being claimed here. ## Verification - Compiles clean against the real ServUO 57.4 reference assemblies (`msbuild Scripts.csproj`, 0 errors) — **built offline with the shard stopped**, because a running ServUO holds `Scripts.dll` and an in-place build fails `MSB3027` rather than reporting a compile error. - The live walk above, with the boot log's `0 Error(s)` read each cycle to confirm the new code is live rather than trusting a clean boot. - `[bridge status` reports `idem(keys= seen= replayed= busy= evicted= uncorrelated=)`. **Rig note, recorded in the scaffolding README:** `deploy.ps1` overwrites `Bridge.cfg`, so every deploy strips `Protocol6ProbeOnStart` *and* `AdminWriteEnabled`. The probe then silently does nothing on the next boot. The README already warned about this and it still cost a cycle. - [x] AI-assisted: Claude Code (Opus 5). Sidecar: RunicGateway/link#36 · Module: RunicGateway/Module-uo#29 · Docs: RunicGateway/docs#219 🤖 Generated with [Claude Code](https://claude.com/claude-code)

wtclaude added 1 commit 2026-09-04 23:00:38 +00:00

feat(bridge): protocol 6 — an idempotency key, and champ.boss.killed (Phase 11a) 7aa7bc8032

A command carrying an `idempotencyKey` is now executed at most once: a repeat is
answered with the original reply rather than re-run. That is the precondition
every world verb in Phase 12 is waiting on, and it is what let `uo.broadcast`
stop being un-retryable.

The gate sits in BridgeBoot's inbound dispatch, not in each handler, so it covers
every kind including ones a later protocol adds. A command with no key behaves
exactly as it did before, which leaves the admin screens unchanged.

Four rules, each a decision rather than an implementation detail: reserve on
receipt (so a handler that defers is covered, answering `bridge.busy` to a repeat
in flight); a key that has begun is never released, not even when the handler
throws; a replay is stamped with the REPEAT's correlation id, because the
sidecar's reqId is fresh per call and replaying the original would hang the retry;
and the bound is loud, because an evicted key is the guarantee's one hole.

`champ.boss.killed` rides along because a bump costs a release, a bundle and an
operator update on every shard. It fires from EventSink.CreatureDeath, detected
by type so a boss that popped and died inside one sweep is still reported, and it
carries the damage table that exists at the death and nowhere else.

overlay.toml protocol = 6, in this commit rather than a later one.

Co-Authored-By: Claude <noreply@anthropic.com>

wtclaude referenced this pull request from RunicGateway/link

2026-09-04 23:00:59 +00:00

feat(sidecar): protocol 6 — carry the key, answer bridge.busy (Phase 11a) #36

wtclaude referenced this pull request from RunicGateway/Module-uo

2026-09-04 23:01:34 +00:00

feat(events): send the idempotency key, declare champ.boss.killed (Phase 11a) #29

wtclaude referenced this pull request from RunicGateway/docs

2026-09-04 23:02:08 +00:00

docs(link): protocol 6 as built — a guarantee, and the kind that pays for it #219

whitlocktech merged commit d2a12c46e2 into edge

2026-09-04 23:06:20 +00:00

whitlocktech deleted branch feature/protocol-v6-idempotency