docs(events): the cutover, the outage it survived, and the defect it exposed (Phase 16b cutover, 6 of 6) #231

wtclaude · 2026-09-10T01:22:08Z

wtclaude commented

2026-09-10 01:22:08 +00:00

The Event System cutover, step 6 of 6 — the last step, and it goes last on purpose: it carries the 16b record, which could not be written until the re-verify had run.

EVENTS_PLAN.md gains that record. 102/1, content only — no CRLF rewrite (checked against --numstat).

The cutover as executed

Six steps, and core lands before the module — which departs from the phase's own sentence and matches what the engagement cutover actually did, because Module-uo's ci/core-ref.json has to name a website main sha carrying MODULE_API 1.10.0 and that sha does not exist until core has landed. The protocol pair is one step, not two: bundle.yml's Gate 1 refuses a sidecar/overlay pair that disagrees on the protocol, so whichever lands first leaves a compose that cannot run.

Four decisions, all as recommended: that order; the app merges with no v* tag; the re-verify walks the whole delivery path; and edge stays standing in every repo. Releases: sidecar v2.2.0, overlay v1.2.0, bundle 2026.09.10 (protocol 7). No contract moves in this leg — MODULE_API_VERSION and EVENTS.md are untouched.

Three things in the record worth more than the chronology

1. A job's log IS readable on this Gitea — through the web route, not the API. /{owner}/{repo}/actions/runs/<n>/jobs/<j>/logs with a token, served as text/plain; step statuses come from the UI's own POST endpoint with a _csrf cookie. Every earlier phase diagnosed CI by reproducing jobs locally on the belief that logs were unreachable. Reading one turned four red jobs into four known causes in about ten minutes — and one of those causes was a log that did not exist, which is itself evidence: a job that dies in Set up job never uploads one.

2. Three of the four reds were infrastructure. A ten-minute Cloudflare outage sat in the middle of the window and killed both release runs: link's built every binary and died pushing the tag (error: 530), servuo-plugins' died inside Set up job after 11m52s. Neither pushed its tag before dying, so the orphaned-tag failure mode did not occur and a plain workflow_dispatch recovered both. The first to land left the pair mismatched and compose run 102 failed exactly as the PRs predicted; the second dispatched it again and 103 composed. link's rust-gates reds were curl: (6) Could not resolve host: sh.rustup.rs inside the runner.

3. The seventh defect of this phase, and the worst-hidden. server-tests had been red on every events PR since Phase 10 — eight PRs, every one # fail 1, always the same single test, merged over every time.

announce.js asked Intl.DateTimeFormat('en-GB', { …, hour12: true }), and that is not the same request as a 12-hour clock: for a locale whose default cycle is h23, Node 20 resolves it to h11 (hours 0–11) so midnight renders 0:00 am, while Node 22+ resolves it to h12. Same ICU (78.2) on both sides — V8's ECMA-402 behaviour, not locale data, so no amount of matching the runner's locale would have found it. The image ships node:20-alpine; a dev machine is newer. So it rendered correctly for everyone who reviewed it and wrongly for every real recipient, whose midnight event mail read "0:00 am" beside a schedule editor reading "12:00 AM".

The rule, now written down: hour12 is a request about a locale's preference, hourCycle is a request about the clock. Ask for the clock. Fixed in website#200, the form recurrence.js had already adopted for the mirror-image case.

The re-verify

The leg 16a could not do, since a locally built bundle cannot go through core's https-only, host-allowlisted installer:

installer      released v0.1.1, checksum matched against the release's own SHA256SUMS
bundle         2026.09.10, protocol 7 — both component checksums verified by the installer
overlay sync   a FIRST install into a stock 57.4 tree:  add=30 change=1 unchanged=0
script build   0 Warning(s) 0 Error(s)   ← no release had ever been asked to prove this
shard boot     [Bridge] enabled=True … adminWrite=True … events=True → connected to :7788
sidecar        server.hello, 208,568 items / 42,871 mobiles, x-uolink-version: 7
event plane    lease.list.ok (config AND targeted property leases), item.catalog.ok,
               GET /world/<run> an empty list rather than a 404
core           released main on a throwaway DB, capabilities: ["events"]
one event      published → run → completed, health ok, results published
the page       /site/events lists a run that finished two minutes earlier

The last row is 16a's calendar fix holding on main — before it, a run that had already started or finished was absent and the page rendered entries: []. A fresh Bridge.cfg still ships EventsEnabled=false and AdminWriteEnabled=false, confirmed from the released config rather than a working tree's.

One thing checked and deliberately not reported as a defect: a cancelled run appears on the public calendar. It is meant to — the entry carries its own status and the page renders a past cancelled run as "Did not happen".

The one leg still owed is the module's own install through core's installer, named as such in the record: it cannot run until Module-uo#34 cuts the release it would fetch.

The record also notes that 16c is now unblocked — checkFacts reads main, and main carries the engine, the module and the app, with the bundle triple the site quotes now fixed.

Pairs with link#40, servuo-plugins#26, website#199, website#200, Android-app#46, Module-uo#34 and Integration-kit#11.

AI-assisted: written with Claude Code (Claude Opus 5).

🤖 Generated with Claude Code

https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4

The Event System cutover, step 6 of 6 — the last step, and it goes last on purpose: it carries the 16b record, which could not be written until the re-verify had run. `EVENTS_PLAN.md` gains that record. `102/1`, content only — no CRLF rewrite (checked against `--numstat`). ## The cutover as executed Six steps, and **core lands before the module** — which departs from the phase's own sentence and matches what the engagement cutover actually did, because `Module-uo`'s `ci/core-ref.json` has to name a website `main` sha carrying MODULE_API 1.10.0 and that sha does not exist until core has landed. The protocol pair is **one** step, not two: `bundle.yml`'s Gate 1 refuses a sidecar/overlay pair that disagrees on the protocol, so whichever lands first leaves a compose that cannot run. Four decisions, all as recommended: that order; the app merges with **no `v*` tag**; the re-verify walks the whole delivery path; and **`edge` stays standing** in every repo. Releases: sidecar **v2.2.0**, overlay **v1.2.0**, bundle **2026.09.10 (protocol 7)**. No contract moves in this leg — `MODULE_API_VERSION` and `EVENTS.md` are untouched. ## Three things in the record worth more than the chronology **1. A job's log IS readable on this Gitea — through the web route, not the API.** `/{owner}/{repo}/actions/runs/<n>/jobs/<j>/logs` with a token, served as `text/plain`; step statuses come from the UI's own POST endpoint with a `_csrf` cookie. Every earlier phase diagnosed CI by reproducing jobs locally on the belief that logs were unreachable. Reading one turned four red jobs into four known causes in about ten minutes — and one of those causes was a log that **did not exist**, which is itself evidence: a job that dies in `Set up job` never uploads one. **2. Three of the four reds were infrastructure.** A ten-minute Cloudflare outage sat in the middle of the window and killed both release runs: `link`'s built every binary and died pushing the tag (`error: 530`), `servuo-plugins`' died inside `Set up job` after 11m52s. **Neither pushed its tag before dying**, so the orphaned-tag failure mode did not occur and a plain `workflow_dispatch` recovered both. The first to land left the pair mismatched and compose run 102 failed exactly as the PRs predicted; the second dispatched it again and 103 composed. `link`'s `rust-gates` reds were `curl: (6) Could not resolve host: sh.rustup.rs` inside the runner. **3. The seventh defect of this phase, and the worst-hidden.** `server-tests` had been red on every events PR since Phase 10 — eight PRs, every one `# fail 1`, always the same single test, merged over every time. `announce.js` asked `Intl.DateTimeFormat('en-GB', { …, hour12: true })`, and **that is not the same request as a 12-hour clock**: for a locale whose default cycle is h23, Node 20 resolves it to **`h11`** (hours 0–11) so midnight renders `0:00 am`, while Node 22+ resolves it to `h12`. **Same ICU (78.2) on both sides** — V8's ECMA-402 behaviour, not locale data, so no amount of matching the runner's locale would have found it. The image ships `node:20-alpine`; a dev machine is newer. So it rendered correctly for everyone who reviewed it and wrongly for every real recipient, whose midnight event mail read "0:00 am" beside a schedule editor reading "12:00 AM". **The rule, now written down: `hour12` is a request about a locale's preference, `hourCycle` is a request about the clock. Ask for the clock.** Fixed in `website#200`, the form `recurrence.js` had already adopted for the mirror-image case. ## The re-verify The leg 16a could not do, since a locally built bundle cannot go through core's https-only, host-allowlisted installer: ``` installer released v0.1.1, checksum matched against the release's own SHA256SUMS bundle 2026.09.10, protocol 7 — both component checksums verified by the installer overlay sync a FIRST install into a stock 57.4 tree: add=30 change=1 unchanged=0 script build 0 Warning(s) 0 Error(s) ← no release had ever been asked to prove this shard boot [Bridge] enabled=True … adminWrite=True … events=True → connected to :7788 sidecar server.hello, 208,568 items / 42,871 mobiles, x-uolink-version: 7 event plane lease.list.ok (config AND targeted property leases), item.catalog.ok, GET /world/<run> an empty list rather than a 404 core released main on a throwaway DB, capabilities: ["events"] one event published → run → completed, health ok, results published the page /site/events lists a run that finished two minutes earlier ``` The last row is **16a's calendar fix holding on `main`** — before it, a run that had already started or finished was absent and the page rendered `entries: []`. A fresh `Bridge.cfg` still ships `EventsEnabled=false` and `AdminWriteEnabled=false`, confirmed from the released config rather than a working tree's. One thing checked and deliberately **not** reported as a defect: a cancelled run appears on the public calendar. It is meant to — the entry carries its own `status` and the page renders a past cancelled run as **"Did not happen"**. **The one leg still owed** is the module's own install through core's installer, named as such in the record: it cannot run until `Module-uo#34` cuts the release it would fetch. The record also notes that 16c is now unblocked — `checkFacts` reads `main`, and `main` carries the engine, the module and the app, with the bundle triple the site quotes now fixed. Pairs with `link#40`, `servuo-plugins#26`, `website#199`, `website#200`, `Android-app#46`, `Module-uo#34` and `Integration-kit#11`. - [x] AI-assisted: written with Claude Code (Claude Opus 5). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4

wtclaude added 1 commit 2026-09-10 01:22:19 +00:00

docs(events): the cutover, the outage it survived, and the defect it exposed (Phase 16b) 1a7d2da4dd

`EVENTS_PLAN.md` gains the 16b record: the six steps and why core lands before
the module, the four decisions taken, the releases cut, and the re-verify against
artefacts an operator would actually download.

Three things in it are worth more than the chronology.

A job's log IS readable on this Gitea, through the web route rather than the
API. Every earlier phase diagnosed CI by reproducing jobs locally on the belief
that logs were unreachable; reading one turned four red jobs into four known
causes in about ten minutes. Three were the ten-minute Cloudflare outage in the
middle of the window and a runner that could not resolve sh.rustup.rs -- and
because neither release pushed its tag before dying, the orphaned-tag failure
mode did not occur and a plain workflow_dispatch recovered both.

The seventh defect of this phase: `server-tests` had been red on every events PR
since Phase 10, always the same single test, and the workstream merged over it
eight times. `announce.js` asked for `hour12: true`, which is not the same
request as a 12-hour clock -- for a locale whose default cycle is h23, Node 20
resolves it to h11 and midnight renders "0:00 am", while Node 22+ resolves it to
h12. Same ICU on both sides, so it is V8's ECMA-402 behaviour and not locale
data; the image ships node:20-alpine and a dev machine is newer, so it rendered
correctly for everyone who reviewed it and wrongly for every real recipient. The
rule is now written down: `hour12` is a request about a locale's preference,
`hourCycle` is a request about the clock -- ask for the clock.

And the re-verify itself: the released installer resolved bundle 2026.09.10,
verified both checksums, did a first install into a stock 57.4 tree, the overlay
compiled 0/0 -- which no release had ever been asked to prove -- the shard came
up with the events plane on and dialed the sidecar, the whole protocol-7 event
plane answered, and an event published on released `main` ran to `completed`
with its results published and its finished run visible on /site/events two
minutes later. That last line is 16a's calendar fix holding on `main`.

The module's own install through core's https installer is named as the one leg
still owed: it cannot run until Module-uo#34 cuts the release it fetches.

Diff is 102/1, content only -- no CRLF rewrite (checked against --numstat).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4

wtclaude added 1 commit 2026-09-10 02:06:54 +00:00

docs(events): the module leg of the re-verify, and the two defects it found bb75910a8f

The leg the first commit named as still owed is walked. Module-uo#34 cut v1.2.1,
so the module could arrive the way an operator's does: core fetched the release
MANIFEST over https from the allowlisted host, verified its sha256, and mounted
it; the four values the installer printed then produced `status: connected`,
`pluginConnected: true`, `protocol: 7`. Released core, released module, released
sidecar, released overlay.

That rig confirmed both Phase 16a fixes in the shipped artefacts rather than in a
working tree -- the atlas imports off a stock tree, and a world verb's teardown
leaves the shard answering `owned: [], pruned: 0`, which is the check the no-op
teardown hid behind.

It also found two more defects, both in the released bundle (Module-uo#35). The
aggregator discarded the UniqueId, so `uo.options.spawners` was empty and no
Phase 12b property lease was authorable at all -- while `PARSER_VERSION = 4`'s
own note said a point keeps that field and named Phase 12b as the reason. And a
landmark option value named 23 places at once: 558 landmarks under 320 distinct
`facet/name`, resolved by `.find()`, so an author who picked "Entrance - Destard"
got Blighted Grove with a successful run and no warning.

Both are recorded as one class, because that is the useful part: an option source
that answers empty, or answers with a value that does not identify one thing,
disables a feature silently. Nothing errors; the form simply cannot express the
thing, and a test that checks the parser, or the query, or the column in
isolation passes throughout.

Diff is 45/6, content only -- no CRLF rewrite (checked against --numstat).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4

whitlocktech merged commit e75ba0e6eb into edge

2026-09-10 02:14:24 +00:00

whitlocktech deleted branch docs/events-p16b-cutover

2026-09-10 02:14:26 +00:00

whitlocktech referenced this issue from a commit

2026-09-10 02:14:26 +00:00

Merge pull request 'docs(events): the cutover, the outage it survived, and the defect it exposed (Phase 16b cutover, 6 of 6)' (#231) from docs/events-p16b-cutover into edge

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: RunicGateway/docs#231