diff --git a/README.md b/README.md index 4110166..0edb670 100644 --- a/README.md +++ b/README.md @@ -65,7 +65,7 @@ particular game; a module is what makes it a site *for* one. | [PROTOCOL_2.md](link/PROTOCOL_2.md) | Protocol 2.0 / 2.1 design | | [v3.md](link/v3.md) | Protocol 3.0 design — shard content/standings streams + the visibility framework | | [v4.md](link/v4.md) | Protocol 4.0 — guild membership on the wire (`guild.roster`, `guild.leave`) | -| [v5.md](link/v5.md) | Protocol 5 — three enrichments in one bump: `house.decay`'s decay schedule, `vendor.listing`'s fee state, and `account.login.result`. **The current protocol**; built on `edge`, not yet released | +| [v5.md](link/v5.md) | Protocol 5 — three enrichments in one bump: `house.decay`'s decay schedule, `vendor.listing`'s fee state, and `account.login.result`. **The current protocol**, shipped 2026-09-01 as bundle 2026.09.01 (sidecar v2.1.0 + overlay v1.1.0) | | [ADMIN_CONTROLS.md](link/ADMIN_CONTROLS.md) | Staff write-plane (kick/ban/broadcast, page queue) | | [SHARD_PREREQS.md](link/SHARD_PREREQS.md) | Shard-side prerequisites for the bridge | | [PLAN.md](link/PLAN.md) | uo-link build plan | diff --git a/link/INTEGRATION.md b/link/INTEGRATION.md index 9657a35..9f4ab85 100644 --- a/link/INTEGRATION.md +++ b/link/INTEGRATION.md @@ -55,7 +55,7 @@ That is the same set of values Admin → Shard asks for — base URL and WS URL The wire protocol is versioned so a mismatch is caught immediately instead of failing weirdly. -The current version is **5**. It is built but **not yet released** — the last shipped pairing is protocol 4, in sidecar **v2.0.0** and overlay **v1.0.0**. +The current version is **5**, shipped on 2026-09-01 in sidecar **v2.1.0** and overlay **v1.1.0** — resolve them as bundle **2026.09.01**, never as "latest of each". The pairing before it was protocol 4, sidecar **v2.0.0** + overlay **v1.0.0**. - Every response carries an **`X-UOLink-Version: 5`** header. - `GET /health` and the WebSocket `ws.hello` frame include `"protocol": 5`. diff --git a/link/v5.md b/link/v5.md index e048fab..17ccd79 100644 --- a/link/v5.md +++ b/link/v5.md @@ -1,6 +1,6 @@ # Protocol 5 — Three enrichments, one bump -**Status:** Built, on `edge`. Not yet cut over to `main`, so not yet released. +**Status:** Released. Cut over to `main` on 2026-09-01 and shipped in sidecar **v2.1.0** + overlay **v1.1.0**, paired as bundle **2026.09.01**. **Date:** 2026-08-31 **Codebase:** ServUO 57.4, ``, net48 / x64, Expansion **EJ**. **Companion to** [`PLAN.md`](PLAN.md) (1.0 read/event plane), [`PROTOCOL_2.md`](PROTOCOL_2.md) (2.0 provisioning + world-state streams), [`v3.md`](v3.md) (3.0 shard content + the visibility framework), [`v4.md`](v4.md) (4.0 guild membership), [`INTEGRATION.md`](INTEGRATION.md) (website API). diff --git a/website/ENGAGEMENT.md b/website/ENGAGEMENT.md index d378d9e..320afec 100644 --- a/website/ENGAGEMENT.md +++ b/website/ENGAGEMENT.md @@ -1,30 +1,35 @@ # The Engagement System — findings and plan -**Status:** design of record. **Phases 1, 1a, 1b, 2, 3, 4a, 4b, 5a, 5b, 6, 7, 8 and 9 are built** -(Phase 1: website#165 + docs#178, with website#164 as its prerequisite; Phase 1a: website#166 + -docs#179; Phase 1b: website#167 + docs#180; Phase 2: website#168 + docs#181; Phase 3: website#169 + -docs#182; Phase 4a: website#170 + docs#183; Phase 4b: website#171 + docs#184; Phase 5a: website#172 + -docs#185; Phase 5b: website#173 + docs#186 + runicgateway.com#22; Phase 6: website#174 + docs#187 + -runicgateway.com#23; Phase 7: website#175 + docs#188 + runicgateway.com#24; Phase 8: the Android -half, Android-app#42 + docs#190; Phase 9: website#176 + docs#191); everything from Phase 10 on is -still design. **PHASE 10 IS BUILT (2026-08-31)** — the protocol bump; see its as-built below and -[`../link/v5.md`](../link/v5.md). **Phases 10 and 11 were both widened on 2026-08-31, by the org lead, before any code:** -the protocol bump carries three wire enrichments rather than one, and Phase 11 ships **every ✅ row of -§8.6** rather than a single rule. **Phase 11 was then scoped on the same day** — six decisions, three -carve-outs, a seventh value in the ceiling lattice and an 11a/11b split; see its own decision block. -The scope decisions -below are settled; **all nine questions in §7.1 are answered** - Q1, Q3, Q5 and Q7 on -2026-08-28, Q6 on 2026-08-29 at the start of Phase 2 (which also settled §7.2's namespace question), -**Q2 and Q4 on 2026-08-29 at the start of Phase 4**, and **Q8 on 2026-08-31 at the start of Phase -8**. Q1's answer added a whole phase (**Phase 1b**, unique email addresses); Q4's answer and the -phase's size split **Phase 4 into 4a and 4b**; **Q9** (core's own `news.post` emitter) was answered -**2026-08-31 at the start of Phase 11**, along with five further decisions that changed what that -phase ships — see its own decision block. Per CLAUDE.md § Conventions, no -implementation starts without the org lead's approval of the phase it belongs to. +**Status:** design of record. **The workstream is BUILT, CUT OVER and ACCEPTED (2026-09-01).** +Phases 0 through 13 are done and on `main` in every repository, and **Phase 13's acceptance walk +passed** — a clean install from `main`, released artefacts only, one email and one in-app item to the +linked owner and nothing to anyone else. It found two defects in shipped core; both are recorded in +Phase 13's as-built and neither is fixed there. **Phase 14 (retention) is scoped and not started.** Each phase carries its own as-built +section below, and the PRs are named there rather than here. -**Branching:** every phase lands on **`edge`** in its repo; `main` is touched once, by the cutover -(Phase 13). §6.0a records the blocking precondition — six `edge` branches are stale and two repos have -none. +The cutover cut `link` **v2.1.0**, the plugin overlay **v1.1.0**, `module-uo` **v1.1.0** and the +paired bundle **2026.09.01** (protocol **5**); `MODULE_API_VERSION` is **1.9.0**. Two of Phase 13's +seven steps were missed in the window and landed afterwards — see its as-built, which is also where +the four findings of the cutover itself are recorded. + +**Phases 10 and 11 were both widened on 2026-08-31, by the org lead, before any code:** the protocol +bump carries three wire enrichments rather than one, and Phase 11 ships **every ✅ row of §8.6** +rather than a single rule. **Phase 11 was then scoped on the same day** — six decisions, three +carve-outs, a seventh value in the ceiling lattice and an 11a/11b split; see its own decision block. +The scope decisions below are settled; **all nine questions in §7.1 are answered** — Q1, Q3, Q5 and +Q7 on 2026-08-28, Q6 on 2026-08-29 at the start of Phase 2 (which also settled §7.2's namespace +question), **Q2 and Q4 on 2026-08-29 at the start of Phase 4**, and **Q8 on 2026-08-31 at the start +of Phase 8**. Q1's answer added a whole phase (**Phase 1b**, unique email addresses); Q4's answer and +the phase's size split **Phase 4 into 4a and 4b**; **Q9** (core's own `news.post` emitter) was +answered **2026-08-31 at the start of Phase 11**, along with five further decisions that changed what +that phase ships — see its own decision block. Per CLAUDE.md § Conventions, no implementation starts +without the org lead's approval of the phase it belongs to, **and that still holds for Phase 14.** + +**Branching:** every phase landed on **`edge`** in its repo, and `main` was touched once, by the +cutover (Phase 13). §6.0a records the blocking precondition it opened with — six `edge` branches +stale, three repos with none — and Phase 13's as-built records how it closed: **every `edge` was +deleted on merge rather than fast-forwarded**, which is the convention now. Phase 14 comes after the +cutover and so goes to `main` through an ordinary feature branch. **Scope decisions, settled by the org lead (2026-08-28):** @@ -1388,7 +1393,8 @@ change is not complete until `docs/` reflects it" — is the floor; this table i | **11a** module-uo triggers | `modules/uo/API.md` — **the full trigger catalogue, its audiences and its ceilings**, not one entry · `modules/uo/README.md` · `website/ENGAGEMENT.md` §8.6 kept true as rows ship · `website/MODULE_API.md` §1.1 (**1.8.0**) and the ceiling vocabulary wherever it is enumerated · `BACKEND_DESIGN.md` — the `news.post` publish path now runs through the engine | `module-uo/README.md` | | **11b** Seeded rules + templates | `website/ENGAGEMENT.md` this phase as built · a release note naming **the seeded-disabled `news.post` rule as an upgrade step** (decision 5) — without it a deployment loses news push silently | **`runicgateway.com`**: `capabilities.mjs` and the notifications page — "one rule" and "the whole catalogue" are different marketing claims | | **12** Public site | `website/ENGAGEMENT.md` this phase as built, incl. the 12a/12b split and Phase 13's fill-in step | **`runicgateway.com`**, in full — see the phase. **Two PRs**: 12a (#26) mergeable now, 12b (#27) a draft held for the cutover window | -| **13** Cutover | `README.md` index rows · every doc's status line | `.profile/README.md` if this is a headline capability | +| **13** Cutover ✅ | `README.md` index rows · every doc's status line · **this phase as built**, incl. the two steps the window missed | `.profile/README.md` — the org lead called this a headline capability, so it landed | +| **14** Retention | `website/ENGAGEMENT.md` this phase as built · `BACKEND_DESIGN.md` table inventory · the settings keys it adds | **`runicgateway.com`**: `/privacy` + **`PLAY_DATA_SAFETY.md`**, both generated from `src/data/collection.mjs`, whose `deploy-engagement` row says "nothing here expires on its own" and stops being true | **One thing this table is protecting against.** `runicgateway.com` appears in eight rows, and it is the only repo here whose checks are *fetching* these values rather than being told them — see Phase 12. @@ -1843,7 +1849,7 @@ see something a preference actually governs. --- -### Phase 4 — The engine: rules, cooldowns, outbox +### Phase 4 — The engine: rules, cooldowns, outbox ✅ **Split into 4a and 4b** at the start of the phase, on the same argument that split Phase 5: the half that first makes this system capable of sending is worth reviewing without a React screen in the same @@ -2914,7 +2920,7 @@ just using, on a train — and fails for the case where the app was killed first --- -### Phase 9 — Deliverability: suppression and bounces +### Phase 9 — Deliverability: suppression and bounces ✅ `engagement_suppressions` and bounce/complaint capture per transport (SMTP has none — this is where the API-based providers earn their place). @@ -3028,7 +3034,7 @@ a bounce handler today. --- -### Phase 10 — Protocol bump: three wire enrichments *(parallel from day one)* +### Phase 10 — Protocol bump: three wire enrichments ✅ *(parallel from day one)* **Scope widened 2026-08-31, by the org lead, before any code.** The phase was written as the `house.decay` enrichment alone. It now carries **three** enrichments, for one reason that is a property @@ -3147,7 +3153,7 @@ once. --- -### Phase 11 — module-uo's triggers: the full catalogue, and core's `news.post` +### Phase 11 — module-uo's triggers: the full catalogue, and core's `news.post` ✅ **Scope widened 2026-08-31, by the org lead, before any code.** This phase used to say `module-uo` registers `uo.house.idoc_warning` "(and siblings)", and Part 8's preamble used to say "Phase 11 ships @@ -3590,7 +3596,7 @@ takes a fixed verb set, so `[p5probe` cannot be typed at a headless shard at all --- -### Phase 12 — `runicgateway.com`: the public site and the docs journey +### Phase 12 — `runicgateway.com`: the public site and the docs journey ✅ **This phase is not optional polish, and it is not "update the marketing copy". Two of its checks will fail the build on their own**, and they read from `main`, which fixes exactly when this has to @@ -3707,7 +3713,7 @@ has to sit outside the code span or the braces reach the published page. --- -### Phase 13 — The cutover: `edge` → `main`, in order +### Phase 13 — The cutover: `edge` → `main`, in order ✅ One PR per repo, all in one window. **The order is not cosmetic** — a `main` holding a v5 sidecar and a v4 overlay is a shard that cannot pair, and the installer refuses it by design. @@ -3742,6 +3748,158 @@ else, and `runicgateway.com` builds green with no fact disagreeing with its auth is then fast-forwarded to `main` again so the next workstream starts from a clean one — the step §6.0a found had been skipped after every previous cutover. +#### As built (2026-09-01) + +**The window ran in the order above and missed two steps.** Steps 1 (docs#200), 2a +(servuo-plugins#20), 2b (link#35), 3 (website#180), 4 (Module-uo#26) and 6 (Android-app#43) merged +together; **5 and 7 did not**, and were done afterwards as **Integration-kit#9** and +**runicgateway.com#28**, with **Module-uo#27** clearing a leftover from step 4. What the window cut: +`link` **v2.1.0**, the plugin overlay **v1.1.0**, `module-uo` **v1.1.0**, and the paired bundle +**2026.09.01** (protocol 5). `website` releases nothing by design. + +**Every `edge` was deleted rather than fast-forwarded**, in seven repos. The acceptance line above +asks for the fast-forward so the next workstream starts from a clean branch; deleting reaches the +same place more bluntly, since §6.0a's Phase -1 cuts `edge` fresh anyway — and a branch that does not +exist cannot be the stale one somebody branches from, which was the actual failure. Only +`Integration-kit` ended up in the state §6.0a warns about, one commit behind `main` once its own step +landed late; fast-forwarded on the day. **The convention is now delete-on-merge**, and Phase -1 cuts +`edge` rather than trusting one it finds. + +**1. Gitea's `raw` API route is CDN-cached for six hours, and a stale read fails BOTH ways.** +`checkFacts.mjs` failed `runicgateway.com` for a `moduleApi` that was correct: it had been served +`website`'s `version.js` from a fortnight earlier (`Cache-Control: public, max-age=21600`, +`cf-cache-status: HIT`, `Age: 15713`), and no edit in that repository could have made it pass. The +false red is the cheap half. **The same run reported the already-republished bundle triple as still +current** — a stale read is just as able to say "nothing has moved", and that repo's whole bargain is +that it goes red when the platform moves. All three cross-repository checks now read the `contents` +endpoint, which answers `private, must-revalidate` and is not cached; a request `Cache-Control: +no-cache` header does **not** bust the CDN, and a cache-busting query param was rejected as +papering over the mechanism rather than choosing the right one. + +**2. The Integration-kit equality check never goes red on its own, and step 5 is the only thing that +makes anyone look.** It clones the ref **the kit itself pins**, so a core that moves past that pin +changes nothing there. The kit sat three minor versions behind the platform for the whole +workstream — teaching a 1.6.0 contract with no triggers, audiences or seeds in it — and was green +throughout. Between cutovers the kit is not wrong, it is **dated**, and `ci/core-ref.json` is where +the date is written down. Read step 5's obligation as *someone re-reads the chapters*, never as +*CI will tell us*. + +**3. `registerEngagementSeeds` does not validate the body it seeds, and the kit's own example was +malformed.** The call checks that `blocks` is a non-empty array and stops; the body is validated by +the block registry, which runs in the template editor and in the renderer and nowhere else. The +draft template shipped a heading `level: 2` — the registry takes `'h1' | 'h2' | 'h3'` — and no block +`id`s at all, so it would have registered cleanly, seeded cleanly, and failed the first time an +operator opened it. It was found by running the template's `register()` through **core's real +`stage()` validators** at the pinned ref, which the kit's CI does not do and cannot: it runs the +template against `test/_fakes.js`, and a fake accepts what core refuses. Two smaller corrections came +out of the same run — core **does** validate `subjectKey` against the declared variables and refuses +the module, and `ctx.events.emit` **throws outside production** rather than only dropping and +logging. Both had been written into the chapter the other way round. + +The general lesson is §5.3's, arriving from the other side: a registration surface is only as +teachable as the thing that checks it. Where the registry validates, the book can describe the +error; where it does not — and `blocks` is the one place it does not — the book has to say so, and +the template needs a test of its own. It has one now, verified by breaking it. + +**4. `runicgateway.com`'s `pr-checks.yml` triggers on PRs into `main` only**, so not one of the seven +site PRs in this workstream was gated, and the cutover was their first CI run in eight phases. Same +shape as `android-app`'s trigger before Q8 fixed it (§6.0a), and left alone rather than fixed: that +repo's checks read the *source repos' `main`*, so running them on an `edge` PR would have been red +for the whole window by design. Phase 12's split exists for the same reason. What it costs is real +though — 12b's fill-in was verified locally and merged on that evidence alone. + +#### The acceptance walk — run 2026-09-01, PASSED + +Every clause of the acceptance line above, on a rig built the way an operator would build one rather +than the way a developer would. + +**A clean install from `main` alone.** The **released** installer v0.1.1, downloaded and checked +against its `SHA256SUMS` — the trust anchor the docs name, since releases are unsigned — resolved +bundle **2026.09.01** and reported the pair it was about to deploy: overlay **v1.1.0** protocol 5, +sidecar **v2.1.0** protocol 5. `--verify` first (`add=23 change=1`, nothing written), then the real +run. `dotnet build` on the synced tree: **0 errors** — the released overlay compiles against stock +ServUO 57.4. The shard booted, printed `[Bridge] enabled=True`, and dialled the sidecar; the site +reported `status=connected pluginConnected=true`. The website was a **fresh `main` clone** and +`module-uo` **v1.1.0 installed from its release manifest** through the `MODULES` declaration — sha +verified and unpacked by core, never copied in by hand. SMTP was configured against a mail catcher +and its test send arrived. + +**One email and one in-app item, to the linked owner, and nothing to anyone else.** Forcing +`0x400143DB` from IDOC to **Greatly** produced `event dispatched {rules:1, enqueued:2, deduped:0, +cooled:0, capped:0, cancelled:0}`, and after the rule's grace window both rows sent on the first +attempt. **Every row in every delivery table names the linked user and no other** — send log 2, +outbox 2, inbox 1, and nothing for the admin or for the second, unlinked player. The mail was the +in-universe body, correctly rendered: *"A notice concerning “Yewcross”"*, the stage in words, the +detail line, an absolute link to `/uo/houses` and a working unsubscribe link — no empty href, which +is the Phase 11b defect class staying fixed. + +**Two defects it found, both in shipped core, neither visible to any test:** + +1. **The one-shot rule-group guard is not atomic.** `moduleSeeds.js` `seedRuleGroup()` reads its + settings stamp, inserts the whole group, and writes the stamp **after the loop**. Two processes + booting within the same moment both read "not seeded" and both insert: the rig ended up with + **52 UO rules where the module ships 26**, under one stamp. The function's own comment states the + cost — *"a duplicate rule is two mails per event"* — which is exactly what its guard fails to + prevent. It was reached here by two app processes starting milliseconds apart, which is what a + `--scale app=2` or a rolling restart does on purpose. The fix is to claim before inserting: an + atomic conditional write of the stamp, or an insert idempotent per (owner, group, trigger). + Same shape as Phase 4a's `foundRows` finding — a guard that reads before it claims. +2. **A module-seeded rule that ships a `digest` body cannot be saved from the Rules screen.** + `checkSeedRule` permits `digest` explicitly, in as many words — *"a template slot rather than a + channel … allowed here and absent from `channels`"* — and sixteen in-universe UO rules ship one. + `engagementRules.model.js` then rejects any `templateKeys` entry that is not one of the rule's + channels, so an operator who opens **House — decay warning** and presses Save gets a 400 naming a + key they never typed. The only way to save is to delete the digest body, which silently removes + digest support from that rule. The two validators have to agree about what `digest` is. + +Neither is fixed here: this phase's PR is the documentation of a cutover, and a core change belongs +in its own. Both are recorded for the org lead to place. + +**Also still open:** nothing else. `.profile/README.md` (step 8) landed — the org lead called the +engagement system a headline capability — and Phase 12's second finding became **Phase 14**. + +--- + +### Phase 14 — Retention: the engagement schema has no sweep + +**Status: scoped, not started.** Phase 12 found this and recorded it without fixing it (finding 2 of +its as-built), on the grounds that a sweep is a `website` change and outside a documentation phase. +Nothing after it picked the finding up, so it is stated here as the phase it always was. + +**Four tables grow without bound**, and they are not one problem with one horizon: + +| Table | What accumulates | The constraint on a horizon | +| --- | --- | --- | +| `engagement_cooldowns` | one row per (rule, user, subject, channel) per fire, read once per fire | **`engagementCooldowns.db.prune(olderThan)` already exists and has no caller.** The horizon must exceed the longest `cooldown_seconds` on any enabled rule, or a pruned row makes the next fire a *first* fire — the model says so and leaves it to the caller | +| `engagement_outbox` | every enqueued delivery, including terminal `sent` / `failed` / `cancelled` / `suppressed` rows | only terminal rows are eligible; a `scheduled` row may be days out by design (`delay_seconds`) | +| `engagement_sends` (the send log) | one row per delivery attempt | **two live readers.** The per-rule hourly cap reads `idx_engs_rule_window (rule_id, created_at)`, so a horizon under an hour breaks Q3's ceiling; Admin → Engagement → Send Log is the operator's only answer to "was this person told", so a short one blinds the screen that exists to be looked at | +| `engagement_suppressions` | one row per suppressed address, forever | **the odd one out, and probably correct as it is.** A suppression is a standing decision; ageing out a `bounce` row means the next send re-mails an address that already hard-bounced, which is exactly how a sender loses a domain's reputation. If anything expires here it is `unverified`, and that is a decision to take rather than a default to assume | + +**The shape is already in the tree, twice.** `utils/teamActivityPrune.js` and +`utils/userNotificationsPrune.js` are the same worker — `setInterval` + `unref` + `stop()`, a horizon +read from a `*_retain_days` setting, started and stopped in `server.js`. Phase 7 wrote the second one +for `user_notifications` after finding that table had no policy either, with one policy difference +worth carrying: it deletes **read items only**, because age alone would destroy the evidence for "I +was never told". The same question has to be answered per table here rather than assumed. + +**Documentation this owes.** `runicgateway.com` again, and for Phase 12's reason: `/privacy` and the +generated **Play Data Safety** answers currently publish, for the `deploy-engagement` row, *"kept +until the operator removes them; nothing here expires on its own"*. That is the true answer today and +becomes false the moment this lands, and it is generated from one inventory (`src/data/collection.mjs`) +rather than written twice. Also `BACKEND_DESIGN.md`'s table inventory, and an operator-facing note for +whichever settings keys this adds. + +**For the org lead, before any code:** the four horizons (or three, plus "suppressions do not +expire"), and whether the send log's retention is allowed to be shorter than the operator screen's +usefulness — the alternative is that the screen learns to say "older entries have been swept" rather +than showing a silently truncated history. + +**Acceptance:** every one of the four tables has a stated policy — a sweep with a horizon, or a +recorded decision that it does not expire and why; the cooldown horizon is checked against the +longest enabled rule's cooldown rather than picked; `/privacy` and the Play answers are regenerated +from the inventory; and a rig run shows the sweep deleting terminal rows while leaving a `scheduled` +outbox row and an in-window send-log row alone. + --- ### Sequencing @@ -3756,9 +3914,11 @@ Stage C └─ 7 ── 8 (8 = app-store caden Stage D └─ 9 Stage E 10 ──────────────────────────────────── 11a ── 11b (10 parallel from day one; 11 needs 6 + 10) Stage F 12 ── 13 (12 written before 13, merged in its window) +Stage G └─ 14 (retention; after the cutover, so straight to `main`) ── all of the above onto `edge` ── 13 is the only thing that touches `main` + ...and 14, which comes after it ``` **Phase -1 is blocking and takes minutes.** Every `edge` is 0 ahead / 3–16 behind `main` (§6.0a), so @@ -3996,7 +4156,8 @@ Nothing here is "documentation to do at the end" — a phase is not done until i web channel that does not exist until Phase 7) · new admin pages for rules, templates and per-channel preferences · **`PLAY_DATA_SAFETY.md` + `/privacy`**, both generated from one inventory that an engagement mailer materially changes. -- `.profile/README.md` — only if this lands as a headline capability. +- `.profile/README.md` — only if this lands as a headline capability. **It did:** the org lead + called it one, and the org landing page names the engagement system in the Phase 13 window. ---