diff --git a/link/INTEGRATION.md b/link/INTEGRATION.md index 147f2f1..05cc847 100644 --- a/link/INTEGRATION.md +++ b/link/INTEGRATION.md @@ -283,11 +283,19 @@ Guilds expose only one in-game event (a member joining), so the roster is polled | kind | fields | notes | |------|--------|-------| -| `guild.update` | `id`, `name`, `abbr`, `members`, `online`, `alliance` (or null), `leader` (actor object or null) | A guild's roster/leader/alliance changed, or its first sight this connection. A **leave** shows up here as `members` dropping. | +| `guild.update` | `id`, `name`, `abbr`, `members`, `online`, `alliance` (or null), `leader` (actor object or null) | A guild's leader/alliance/name changed, its member count moved, or its first sight this connection. | | `guild.remove` | `id` | The guild disbanded (leader gone) or was removed. Drop the row. | | `guild.join` | `id`, `name`, `abbr`, `who` (actor object) | Real-time: a player joined a guild (`EventSink.JoinGuild`). | +| `guild.roster` **(4)** | `id`, `name`, `abbr`, `total`, `seq`, `more`, `members` (array of actor objects) | The full member list. Emitted whenever the member set changes. **`seq` 0 supersedes whatever roster you hold for that guild; `more: false` ends it.** | +| `guild.leave` **(4)** | `id`, `name`, `who` (serial string) | Real-time: a member left. Advisory — see below. | -The `leader`/`who` **actor object** is `{serial, name, acct?, webId?, player}` — `acct`/`webId` present when the mobile has an account / a linked website user. +The `leader`/`who` **actor object** is `{serial, name, acct?, webId?, player}` — `acct`/`webId` present when the mobile has an account / a linked website user. Note `guild.leave`'s `who` is a bare **serial string**, not an actor object: the mobile has already left, so there is nothing to attribute. + +**On Protocol 4.** Before it, a guild's membership was a *count* and a leave surfaced only as that count dropping. `guild.roster` carries the members themselves, and `guild.leave` names who went. + +`guild.leave` is **advisory**: any change to the member set re-emits the whole roster, so a consumer holding a membership table stays correct even if it ignores every leave event. Handle it when you want a "so-and-so left" feed to update without waiting for the sweep. + +**Rosters can arrive in several frames.** Members per frame are capped so a large guild cannot produce an unbounded line (~69 bytes per member; the default cap is 500). Every realistic guild arrives as one frame with `seq: 0, more: false` and needs no special handling — but if you consume the raw stream, accumulate from `seq` 0 and apply on `more: false`, discarding a partial roster if a frame arrives out of order or the shard reconnects. `GET /guilds` hands you rosters already reassembled. A guild with no members emits one frame with an empty array, so an emptied roster is distinguishable from an absent one. ```json {"kind":"guild.update","id":1042,"name":"The Silver Hand","abbr":"TSH","members":14, @@ -296,8 +304,16 @@ The `leader`/`who` **actor object** is `{serial, name, acct?, webId?, player}` "t":1752489280000} {"kind":"guild.join","id":1042,"name":"The Silver Hand","abbr":"TSH", "who":{"serial":"0x77","name":"Bran","acct":"bran","player":true},"t":1752489281000} +{"kind":"guild.roster","id":1042,"name":"The Silver Hand","abbr":"TSH", + "total":14,"seq":0,"more":false, + "members":[{"serial":"0x1A2B","name":"Darrow","acct":"whitlocktech","webId":"9931","player":true}, + {"serial":"0x77","name":"Bran","acct":"bran","player":true}], + "t":1752489282000} +{"kind":"guild.leave","id":1042,"name":"The Silver Hand","who":"0x77","t":1752489283000} ``` +`acct` is genuinely optional on a member — a character can have no account at all — so do not assume it is present. + Render the current board from `GET /guilds` (§6) on connect, then keep it live with these events. #### Town governors (Protocol 2.0) @@ -853,7 +869,17 @@ GET /guilds "t":1752489280000}, ... ] } ``` -Every guild's latest roster snapshot at once — the live board. Served from the sidecar's projection (no shard round-trip), kept current by the `guild.*` stream (§4). Render on load, then subscribe. Each entry is exactly a `guild.update` payload; ordered by name. Survives a sidecar restart. +Every guild's latest snapshot at once — the live board. Served from the sidecar's projection (no shard round-trip), kept current by the `guild.*` stream (§4). Render on load, then subscribe. Ordered by name. Survives a sidecar restart. + +Each entry is a `guild.update` payload **plus, from Protocol 4, a `roster` key** holding the member list — already reassembled, so the frame-splitting described in §4 never reaches this endpoint: + +``` +→ { "guilds": [ {"kind":"guild.update","id":1042, ..., "roster":[ + {"serial":"0x1A2B","name":"Darrow","acct":"whitlocktech","webId":"9931","player":true}, + {"serial":"0x77","name":"Bran","acct":"bran","player":true} ]}, ... ] } +``` + +A guild that has had a `guild.update` but no roster yet has **no `roster` key at all** — deliberately distinct from `"roster": []`, which means the guild is genuinely empty. Do not conflate "not known" with "known to be empty". ### Governor board (Protocol 2.0) diff --git a/link/PROTOCOL_2.md b/link/PROTOCOL_2.md index cd9944c..2b4f20b 100644 --- a/link/PROTOCOL_2.md +++ b/link/PROTOCOL_2.md @@ -236,6 +236,14 @@ These are **outbound** streams (shard → website), the natural extension of `PL ### 10.1 Guilds +> **Built in two halves.** 2.0 shipped the board (`guild.update` / `guild.remove`) plus the live +> `guild.join`, and folded membership into the board signature as a serial *sum* — so the site got a +> member **count** and no roster, and the `guild.leave` sketched below was never emitted. **Protocol 4 +> builds the rest of what this section describes**: the member-serial set, its diff, `guild.roster` +> and `guild.leave`. See [`v4.md`](v4.md). One correction from doing it — a *sum* is not a safe stand-in +> for a set, because one member joining and another leaving between two sweeps offset each other and +> the guild reads as unchanged. v4 holds the real set. + **Hook reality (verified):** - `EventSink.JoinGuild` is real — raised at `Scripts/Misc/Guild.cs:1597` when a mobile joins a guild. Usable as a live `guild.join`. diff --git a/link/v4.md b/link/v4.md new file mode 100644 index 0000000..726f73b --- /dev/null +++ b/link/v4.md @@ -0,0 +1,232 @@ +# Protocol 4 — Guild membership on the wire + +**Status:** Feature-complete on `edge`. All work lands on an `edge` branch in each repo; `edge` → `main` is the v4 cutover. +**Date:** 2026-08-17 +**Codebase:** ServUO 57.4, ``, net48 / x64, Expansion **EJ**. +**Companion to** [`PLAN.md`](PLAN.md) (1.0 read/event plane), [`PROTOCOL_2.md`](PROTOCOL_2.md) (2.0 provisioning + world-state streams), [`v3.md`](v3.md) (3.0 shard content + the visibility framework), [`INTEGRATION.md`](INTEGRATION.md) (website API). + +--- + +## 1. Why 4 + +Protocol 2 gave the website a guild board: name, abbreviation, leader, alliance, and a member +**count**. It could say a guild had 155 members. It could not say who they were, and there was no +event for anyone leaving one. + +That gap was known when 2.0 shipped. [`PROTOCOL_2.md` §10.1](PROTOCOL_2.md) sketched exactly this +design — hold a member-serial set, diff it each sweep, emit `guild.join`/`guild.leave` — and then +2.0 shipped only the half that needed no new state, folding membership into the board signature as a +serial *sum*. Protocol 4 builds the rest of what §10.1 described. + +The immediate consumer is a richer public Guilds page, which is worth the bump on its own merit. The +reason it is being built **now** is that [`../website/TEAMS.md`](../website/TEAMS.md) needs it: +platform Teams are sourced from game guilds, and nothing in Team core can be built against a count. +That is a dependency, not a coupling — nothing in this protocol version knows what a Team is. + +**Two kinds, both additive:** + +| Kind | Shape | Purpose | +|---|---|---| +| `guild.roster` | full member list, possibly split across frames | the membership itself; supersedes whatever was held | +| `guild.leave` | one departing member | the real-time counterpart to the existing `guild.join` | + +Nothing existing changed shape. `GET /guilds` grows a `roster` key; a v3 consumer that ignores it +keeps working. + +--- + +## 2. The shard side + +`BridgeSocial.cs` holds each guild's member serial **set** instead of folding it into the signature +as a sum. Two consequences, both the point of the change: + +- **A set comparison cannot collide.** The old sum could: one member joining and another leaving + between two sweeps offset each other, and the guild read as unchanged. +- **A set can be differenced.** Departures are the prior set minus the current one — which is what + makes a per-member `guild.leave` possible without a core tap, since ServUO raises no event for + leaving, disbanding, or a leader change. + +A changed set re-emits `guild.roster`, the whole member list. That is what lets `guild.leave` stay +**advisory**: a consumer building a "so-and-so left" feed wants the individual events, but a consumer +holding a membership table only needs the roster, so nothing downstream has to replay deltas to +remain correct. A dropped `guild.leave` costs latency, never accuracy. + +On a guild's **first** sweep there is no prior set, so nothing is reported as leaving. An unknown +roster becoming known is not 155 people leaving at once. + +### 2.1 Frames are capped, and split when they must be + +A roster is the only fat frame this bridge emits. Measured against a real 155-member guild: **10,812 +bytes, about 69 bytes per member.** The sidecar's `read_line` has no length bound, so an uncapped +roster is an unbounded line. + +Members per frame are therefore capped (`Bridge.GuildRosterMembersPerLine`, default 500 ≈ 35 KB), and +a guild over the cap is split into frames carrying `seq`, `more` and `total`: + +```jsonc +{"t":1786988708513,"kind":"guild.roster","id":1,"name":"The Silver Hand","abbr":"TSH", + "total":155,"seq":0,"more":true,"members":[ /* 50 actor objects */ ]} +{"t":1786988708514,"kind":"guild.roster","id":1,"name":"The Silver Hand","abbr":"TSH", + "total":155,"seq":1,"more":true,"members":[ /* 50 */ ]} +{"t":1786988708514,"kind":"guild.roster","id":1,"name":"The Silver Hand","abbr":"TSH", + "total":155,"seq":2,"more":true,"members":[ /* 50 */ ]} +{"t":1786988708514,"kind":"guild.roster","id":1,"name":"The Silver Hand","abbr":"TSH", + "total":155,"seq":3,"more":false,"members":[ /* 5 */ ]} +``` + +Reading the flags: **`seq` 0 begins a roster and supersedes whatever was held for that guild**; +`more: false` ends it. Every realistic guild is inside the cap and emits exactly one frame with +`seq: 0, more: false` — the same shape as if chunking did not exist. A guild with no members still +emits one frame with an empty array, or a consumer could never learn that a roster it holds has +emptied. + +`guild.leave` is a small frame naming the departed serial: + +```jsonc +{"t":1786988770099,"kind":"guild.leave","id":1,"name":"The Silver Hand","who":"0x1F5"} +``` + +### 2.2 The reconnect baseline is spread + +`BridgeLink.OnConnected` clears the diff caches, so after a reconnect **every** guild reads as +changed at once. The outbound queue is not the risk — the cap is 10,000 *lines* and a few hundred +guilds is a few hundred lines — but building hundreds of fat JSON frames in a single Core-thread tick +is exactly the stall this bridge exists to avoid. + +So at most `Bridge.GuildRosterGuildsPerTick` guilds (default 25) emit a roster per sweep. A guild over +budget keeps its previously held member set, so it still reads as changed on the next pass; its +`guild.update` has already gone, so the board's counts are current either way. While a baseline is +draining the sweep re-arms itself after 2 s rather than waiting a full `GuildSweepSeconds`, so +catch-up takes seconds instead of one sweep interval per batch. + +### 2.3 What a roster member carries + +Each entry is the standard actor object — `serial`, `name`, `player`, plus `acct` when the mobile has +an account and `webId` when that account is linked: + +```jsonc +{"serial":"0x1F5","name":"Seed000A","acct":"seed_000","webId":"42","player":true} +``` + +`acct` is **genuinely optional**: a `PlayerMobile` can have no `Account` at all, and the local test +world contains such mobiles. Consumers must not assume it is present. + +These identity fields are emitted unconditionally, by design — the sidecar is a forwarder, and +deciding who may see them is the website's job. See §4. + +--- + +## 3. The sidecar side + +`guild.roster` writes a **`members` column on the `guilds` board**, not a field inside the existing +`json` column. That column holds the verbatim `guild.update` line, and a roster written into it would +clobber the snapshot — name, abbreviation, leader, online count — that `guild.update` owns. Two +writers across two columns of one row keeps both as plain upserts: neither reads the other's value +first, so there is no read-modify-write and no ordering requirement between the two kinds. Either can +arrive first. + +`GET /guilds` folds the roster back in as `roster` at read time. A guild that has had a +`guild.update` but no `guild.roster` yet simply has **no `roster` key** — the honest representation +of "not known", and deliberately distinct from a guild whose roster is genuinely empty. + +`guild.leave` gets **no board projection at all**. The sidecar persists and broadcasts it like any +event and leaves the board alone; the roster self-corrects on the next `guild.roster`, which the +shard re-emits whenever the member set changes. Keeping the delta out of the projection is what keeps +the sidecar a forwarder rather than a thing that maintains state. + +### 3.1 Split rosters are reassembled in memory + +A roster split across frames is reassembled **before** it is stored, and written once on the frame +that closes it. Appending to the column per frame was rejected twice over: it would make the write a +read-modify-write — the exact thing the two-column split exists to avoid — and it would publish a +torn roster, since a reader hitting `GET /guilds` between frames would see a partial member list +presented as the whole truth. + +This is transport-level reassembly, the same category of work as turning bytes into a line, and it +holds nothing once a roster is complete. The ordinary single-frame case never enters the buffer at +all. Around it: a fresh `seq: 0` supersedes an abandoned partial, an out-of-order frame discards the +partial rather than storing one with an undetectable hole, a continuation with no start is ignored, a +`server.hello` drops every partial (a reconnected shard restarts each roster at 0), and accumulation +is bounded so a shard that never sends a closing frame cannot grow the buffer without limit. + +### 3.2 This is the first bump that needed a store migration + +`store.rs`'s `SCHEMA` is `CREATE TABLE IF NOT EXISTS`, which can add a table but **cannot add a +column to a table that already exists**. Every schema change up to and including Protocol 3.0 added +whole tables, so this never mattered and `ALTER TABLE` appeared nowhere in the repo's history. +`guilds.members` is the first column added to an existing table: without a mechanism it would simply +never reach an installed sidecar, and every roster write would fail. + +The counter is SQLite's own **`PRAGMA user_version`** — an integer in the database header, so it +costs no table and cannot drift from the file it describes. Each step runs in a transaction together +with the bump recording it, so a step lands completely or not at all and an interrupted run resumes +in the right place. + +- **A failure aborts startup**, which was already the behaviour. A half-migrated store answers the + website with confusing partial data, which is worse than being plainly absent — and the shard dials + *out*, so a sidecar that refuses to start never stalls the game. +- **A database written by a newer sidecar warns and continues.** Every step is additive, so a newer + schema has only columns an older reader ignores; refusing to start would turn rolling the binary + back — a recovery path — into a dead end. + +`sqlx::migrate!` was considered and rejected: it checksums each migration file, so editing an +already-released migration hard-fails startup, which is a poor trade for a schema of eleven +JSON-blob tables. + +--- + +## 4. Visibility + +Both kinds map to the existing **`guilds`** feature in the website's visibility framework +([`v3.md` §3](v3.md)). That mapping is required, not cosmetic: rule 2 fails an *unmapped* kind closed +to admin-only, which would have quietly kept rosters off the public Guilds page forever. + +Mapping them is safe because of how the framework strips fields. A roster is the first frame carrying +locked fields inside an **array** of actors rather than a single nested actor — but the projection +walker already recurses into arrays and matches `acct`/`webId` **by suffix, on meaning rather than +spelling**. So a member's account name is stripped below `admin` by exactly the rule that already +strips `guild.leader.acct`. + +The website stores `acct`/`web_id` per member, because that is what lets a linked member be matched to +a site user at all. It never projects them below `admin`. The difference this rule makes is a public +page listing character names versus one publishing 150 account names, so it carries a test of its own. + +--- + +## 5. Cross-repo obligations + +| Repo | Change | +|---|---| +| `servuo-plugins` | member-set diff, `guild.roster` + `guild.leave`, `BridgeJson.Actors`, two `Bridge.cfg` keys, **`overlay.toml` protocol → 4** | +| `link` | `PROTOCOL_VERSION` → 4, `members` column + `user_version` migration, roster reassembly, `GET /guilds` projection | +| `module-uo` | `shard_guild_members`, `guild.roster`/`guild.leave` ingest, the kind→feature map entries | +| `installer` | `backup.rs`'s stated reason for skipping the sidecar DB (§3.2 falsifies it) — docs only | +| `docs` | this file, `PROTOCOL_2.md` §10.1, `INTEGRATION.md` | + +**`overlay.toml` must move in the same PR as the emitters.** CI folds it into the release manifest and +the installer refuses to pair an overlay and a sidecar whose protocol numbers disagree, so a bump +landing separately would silently fail to compose into a bundle. + +--- + +## 6. Verification + +`servuo-plugins` has no CI build — the plugin compiles only inside ServUO — so "it compiled" is not +evidence and neither is a clean boot. What was actually run: + +1. **A throwaway one-guild spike first** (TEAMS.md Phase 0), against the real Rust sidecar rather than + a stub, to retire the unknowns before committing to a four-repo bump. It proved the two-column + board shape, measured the line, and surfaced §3.2 — the missing migration mechanism — which no + amount of reading would have. +2. **A live end-to-end run**: 155 members seeded from real `PlayerMobile`s on the local ServUO tree, + the cap forced down to 50 so the split path actually fired, producing 50/50/50/5 across four + frames; two members then removed on a timer, producing exactly two `guild.leave` frames with the + correct serials and a re-emitted roster at `total: 153`. +3. **A restart test with the shard stopped first**, so its reconnect could not re-emit and fake + persistence. The board's row came back byte-identical. + +Step 2 is what caught the reassembly bug in §3.1: every unit test passed through it, because they all +exercised a single-frame roster. The case does not arise until a guild exceeds the cap. + +Still outstanding for the cutover: the five-rung shard visibility walk against a live shard, confirming +`acct`/`webId` never reach a caller below their rung. diff --git a/website/TEAMS.md b/website/TEAMS.md index 479a937..dd4d6e4 100644 --- a/website/TEAMS.md +++ b/website/TEAMS.md @@ -1946,7 +1946,28 @@ Emit `guild.roster` for a **single** guild against the local ServUO tree store, survives a sidecar restart, and comes back out of `GET /guilds`. Then build Phase 1. Days, not weeks, and it retires the only unknown in the plan. -### Phase 1 — the roster on the wire (`servuo-plugins` + `link` + `module-uo` + `docs`) +### Phase 1 — the roster on the wire (`servuo-plugins` + `link` + `module-uo` + `installer` + `docs`) + +> **Amended 2026-08-17, after Phase 0 and while building this.** Two corrections to what follows. +> +> **The sidecar had no schema-migration mechanism, and this phase is the first change that needs +> one.** `store.rs`'s `SCHEMA` is `CREATE TABLE IF NOT EXISTS`, which can add a table but cannot add a +> column to one that already exists — and every schema change up to Protocol 3.0 happened to add +> whole tables, so `ALTER TABLE` appears nowhere in `link`'s history and the gap was invisible until +> `guilds.members`. Settled by the org lead: **`PRAGMA user_version` stepped migrations**, each step +> transactional with the bump recording it; a failure aborts startup (already the behaviour, and safe +> because the shard dials *out*), while a database from a *newer* sidecar warns and continues so a +> binary rollback stays a recovery path. Not `sqlx::migrate!`, whose per-file checksums hard-fail +> startup if a released migration is ever edited. +> +> **`installer` joins the phase**, which is why the heading names five repos rather than four. +> `backup.rs` justifies not copying the sidecar database on two claims: that every table is +> `IF NOT EXISTS` (which the migration above falsifies) and that the sweeps repopulate everything +> (already false — `events` is never pruned and the website backfills from `GET /history` on every +> reconnect). The behaviour is unchanged and correct; only its stated reason needed fixing, and a +> wrong reason left in place is what lets someone extend it to a case it never covered. +> +> The spec for all of it is [`../link/v4.md`](../link/v4.md). The prerequisite for everything. Nothing in Team core can be built against counts. @@ -1954,7 +1975,8 @@ The prerequisite for everything. Nothing in Team core can be built against count and `guild.leave`; `overlay.toml` protocol → 4. Sidecar: `members` on the `guilds` board, `PROTOCOL_VERSION` → 4, `GET /guilds` projection. `module-uo`: ingest both kinds, a `shard_guild_members` table, the kind→feature map entry and field projection for the new fields. -`docs/link/PROTOCOL_2.md` + `v3.md` + `INTEGRATION.md`. +A new `docs/link/v4.md` as the spec of record, plus `PROTOCOL_2.md` §10.1 (which sketched this design +in 2.0 and had it half-built) and `INTEGRATION.md`. **Ships:** a richer public Guilds page (real rosters) on its own merit, with no Team code anywhere. **Verify:** the five-rung shard visibility walk against a live ServUO + sidecar, confirming `acct`/`webId`