Merge pull request 'docs(link): protocol 4 — guild membership on the wire' (#153) from feat/teams-phase1-guild-roster into edge

Reviewed-on: #153

link/INTEGRATION.md

@@ -283,11 +283,19 @@ Guilds expose only one in-game event (a member joining), so the roster is polled
 
 | kind | fields | notes |
 |------|--------|-------|
-| `guild.update` | `id`, `name`, `abbr`, `members`, `online`, `alliance` (or null), `leader` (actor object or null) | A guild's roster/leader/alliance changed, or its first sight this connection. A **leave** shows up here as `members` dropping. |
+| `guild.update` | `id`, `name`, `abbr`, `members`, `online`, `alliance` (or null), `leader` (actor object or null) | A guild's leader/alliance/name changed, its member count moved, or its first sight this connection. |
 | `guild.remove` | `id` | The guild disbanded (leader gone) or was removed. Drop the row. |
 | `guild.join` | `id`, `name`, `abbr`, `who` (actor object) | Real-time: a player joined a guild (`EventSink.JoinGuild`). |
+| `guild.roster` **(4)** | `id`, `name`, `abbr`, `total`, `seq`, `more`, `members` (array of actor objects) | The full member list. Emitted whenever the member set changes. **`seq` 0 supersedes whatever roster you hold for that guild; `more: false` ends it.** |
+| `guild.leave` **(4)** | `id`, `name`, `who` (serial string) | Real-time: a member left. Advisory — see below. |
 
-The `leader`/`who` **actor object** is `{serial, name, acct?, webId?, player}` — `acct`/`webId` present when the mobile has an account / a linked website user.
+The `leader`/`who` **actor object** is `{serial, name, acct?, webId?, player}` — `acct`/`webId` present when the mobile has an account / a linked website user. Note `guild.leave`'s `who` is a bare **serial string**, not an actor object: the mobile has already left, so there is nothing to attribute.
+
+**On Protocol 4.** Before it, a guild's membership was a *count* and a leave surfaced only as that count dropping. `guild.roster` carries the members themselves, and `guild.leave` names who went.
+
+`guild.leave` is **advisory**: any change to the member set re-emits the whole roster, so a consumer holding a membership table stays correct even if it ignores every leave event. Handle it when you want a "so-and-so left" feed to update without waiting for the sweep.
+
+**Rosters can arrive in several frames.** Members per frame are capped so a large guild cannot produce an unbounded line (~69 bytes per member; the default cap is 500). Every realistic guild arrives as one frame with `seq: 0, more: false` and needs no special handling — but if you consume the raw stream, accumulate from `seq` 0 and apply on `more: false`, discarding a partial roster if a frame arrives out of order or the shard reconnects. `GET /guilds` hands you rosters already reassembled. A guild with no members emits one frame with an empty array, so an emptied roster is distinguishable from an absent one.
 
 ```json
 {"kind":"guild.update","id":1042,"name":"The Silver Hand","abbr":"TSH","members":14,
@@ -296,8 +304,16 @@ The `leader`/`who` **actor object** is `{serial, name, acct?, webId?, player}`
  "t":1752489280000}
 {"kind":"guild.join","id":1042,"name":"The Silver Hand","abbr":"TSH",
  "who":{"serial":"0x77","name":"Bran","acct":"bran","player":true},"t":1752489281000}
+{"kind":"guild.roster","id":1042,"name":"The Silver Hand","abbr":"TSH",
+ "total":14,"seq":0,"more":false,
+ "members":[{"serial":"0x1A2B","name":"Darrow","acct":"whitlocktech","webId":"9931","player":true},
+            {"serial":"0x77","name":"Bran","acct":"bran","player":true}],
+ "t":1752489282000}
+{"kind":"guild.leave","id":1042,"name":"The Silver Hand","who":"0x77","t":1752489283000}
 ```
 
+`acct` is genuinely optional on a member — a character can have no account at all — so do not assume it is present.
+
 Render the current board from `GET /guilds` (§6) on connect, then keep it live with these events.
 
 #### Town governors (Protocol 2.0)
@@ -853,7 +869,17 @@ GET /guilds
       "t":1752489280000}, ... ] }
 ```
 
-Every guild's latest roster snapshot at once — the live board. Served from the sidecar's projection (no shard round-trip), kept current by the `guild.*` stream (§4). Render on load, then subscribe. Each entry is exactly a `guild.update` payload; ordered by name. Survives a sidecar restart.
+Every guild's latest snapshot at once — the live board. Served from the sidecar's projection (no shard round-trip), kept current by the `guild.*` stream (§4). Render on load, then subscribe. Ordered by name. Survives a sidecar restart.
+
+Each entry is a `guild.update` payload **plus, from Protocol 4, a `roster` key** holding the member list — already reassembled, so the frame-splitting described in §4 never reaches this endpoint:
+
+```
+→ { "guilds": [ {"kind":"guild.update","id":1042, ..., "roster":[
+      {"serial":"0x1A2B","name":"Darrow","acct":"whitlocktech","webId":"9931","player":true},
+      {"serial":"0x77","name":"Bran","acct":"bran","player":true} ]}, ... ] }
+```
+
+A guild that has had a `guild.update` but no roster yet has **no `roster` key at all** — deliberately distinct from `"roster": []`, which means the guild is genuinely empty. Do not conflate "not known" with "known to be empty".
 
 ### Governor board (Protocol 2.0)
 

link/PROTOCOL_2.md

@@ -236,6 +236,14 @@ These are **outbound** streams (shard → website), the natural extension of `PL
 
 ### 10.1 Guilds
 
+> **Built in two halves.** 2.0 shipped the board (`guild.update` / `guild.remove`) plus the live
+> `guild.join`, and folded membership into the board signature as a serial *sum* — so the site got a
+> member **count** and no roster, and the `guild.leave` sketched below was never emitted. **Protocol 4
+> builds the rest of what this section describes**: the member-serial set, its diff, `guild.roster`
+> and `guild.leave`. See [`v4.md`](v4.md). One correction from doing it — a *sum* is not a safe stand-in
+> for a set, because one member joining and another leaving between two sweeps offset each other and
+> the guild reads as unchanged. v4 holds the real set.
+
 **Hook reality (verified):**
 
 - `EventSink.JoinGuild` is real — raised at `Scripts/Misc/Guild.cs:1597` when a mobile joins a guild. Usable as a live `guild.join`.

link/v4.md

@@ -0,0 +1,232 @@
+# Protocol 4 — Guild membership on the wire
+
+**Status:** Feature-complete on `edge`. All work lands on an `edge` branch in each repo; `edge` → `main` is the v4 cutover.
+**Date:** 2026-08-17
+**Codebase:** ServUO 57.4, `<servuo>`, net48 / x64, Expansion **EJ**.
+**Companion to** [`PLAN.md`](PLAN.md) (1.0 read/event plane), [`PROTOCOL_2.md`](PROTOCOL_2.md) (2.0 provisioning + world-state streams), [`v3.md`](v3.md) (3.0 shard content + the visibility framework), [`INTEGRATION.md`](INTEGRATION.md) (website API).
+
+---
+
+## 1. Why 4
+
+Protocol 2 gave the website a guild board: name, abbreviation, leader, alliance, and a member
+**count**. It could say a guild had 155 members. It could not say who they were, and there was no
+event for anyone leaving one.
+
+That gap was known when 2.0 shipped. [`PROTOCOL_2.md` §10.1](PROTOCOL_2.md) sketched exactly this
+design — hold a member-serial set, diff it each sweep, emit `guild.join`/`guild.leave` — and then
+2.0 shipped only the half that needed no new state, folding membership into the board signature as a
+serial *sum*. Protocol 4 builds the rest of what §10.1 described.
+
+The immediate consumer is a richer public Guilds page, which is worth the bump on its own merit. The
+reason it is being built **now** is that [`../website/TEAMS.md`](../website/TEAMS.md) needs it:
+platform Teams are sourced from game guilds, and nothing in Team core can be built against a count.
+That is a dependency, not a coupling — nothing in this protocol version knows what a Team is.
+
+**Two kinds, both additive:**
+
+| Kind | Shape | Purpose |
+|---|---|---|
+| `guild.roster` | full member list, possibly split across frames | the membership itself; supersedes whatever was held |
+| `guild.leave` | one departing member | the real-time counterpart to the existing `guild.join` |
+
+Nothing existing changed shape. `GET /guilds` grows a `roster` key; a v3 consumer that ignores it
+keeps working.
+
+---
+
+## 2. The shard side
+
+`BridgeSocial.cs` holds each guild's member serial **set** instead of folding it into the signature
+as a sum. Two consequences, both the point of the change:
+
+- **A set comparison cannot collide.** The old sum could: one member joining and another leaving
+  between two sweeps offset each other, and the guild read as unchanged.
+- **A set can be differenced.** Departures are the prior set minus the current one — which is what
+  makes a per-member `guild.leave` possible without a core tap, since ServUO raises no event for
+  leaving, disbanding, or a leader change.
+
+A changed set re-emits `guild.roster`, the whole member list. That is what lets `guild.leave` stay
+**advisory**: a consumer building a "so-and-so left" feed wants the individual events, but a consumer
+holding a membership table only needs the roster, so nothing downstream has to replay deltas to
+remain correct. A dropped `guild.leave` costs latency, never accuracy.
+
+On a guild's **first** sweep there is no prior set, so nothing is reported as leaving. An unknown
+roster becoming known is not 155 people leaving at once.
+
+### 2.1 Frames are capped, and split when they must be
+
+A roster is the only fat frame this bridge emits. Measured against a real 155-member guild: **10,812
+bytes, about 69 bytes per member.** The sidecar's `read_line` has no length bound, so an uncapped
+roster is an unbounded line.
+
+Members per frame are therefore capped (`Bridge.GuildRosterMembersPerLine`, default 500 ≈ 35 KB), and
+a guild over the cap is split into frames carrying `seq`, `more` and `total`:
+
+```jsonc
+{"t":1786988708513,"kind":"guild.roster","id":1,"name":"The Silver Hand","abbr":"TSH",
+ "total":155,"seq":0,"more":true,"members":[ /* 50 actor objects */ ]}
+{"t":1786988708514,"kind":"guild.roster","id":1,"name":"The Silver Hand","abbr":"TSH",
+ "total":155,"seq":1,"more":true,"members":[ /* 50 */ ]}
+{"t":1786988708514,"kind":"guild.roster","id":1,"name":"The Silver Hand","abbr":"TSH",
+ "total":155,"seq":2,"more":true,"members":[ /* 50 */ ]}
+{"t":1786988708514,"kind":"guild.roster","id":1,"name":"The Silver Hand","abbr":"TSH",
+ "total":155,"seq":3,"more":false,"members":[ /* 5 */ ]}
+```
+
+Reading the flags: **`seq` 0 begins a roster and supersedes whatever was held for that guild**;
+`more: false` ends it. Every realistic guild is inside the cap and emits exactly one frame with
+`seq: 0, more: false` — the same shape as if chunking did not exist. A guild with no members still
+emits one frame with an empty array, or a consumer could never learn that a roster it holds has
+emptied.
+
+`guild.leave` is a small frame naming the departed serial:
+
+```jsonc
+{"t":1786988770099,"kind":"guild.leave","id":1,"name":"The Silver Hand","who":"0x1F5"}
+```
+
+### 2.2 The reconnect baseline is spread
+
+`BridgeLink.OnConnected` clears the diff caches, so after a reconnect **every** guild reads as
+changed at once. The outbound queue is not the risk — the cap is 10,000 *lines* and a few hundred
+guilds is a few hundred lines — but building hundreds of fat JSON frames in a single Core-thread tick
+is exactly the stall this bridge exists to avoid.
+
+So at most `Bridge.GuildRosterGuildsPerTick` guilds (default 25) emit a roster per sweep. A guild over
+budget keeps its previously held member set, so it still reads as changed on the next pass; its
+`guild.update` has already gone, so the board's counts are current either way. While a baseline is
+draining the sweep re-arms itself after 2 s rather than waiting a full `GuildSweepSeconds`, so
+catch-up takes seconds instead of one sweep interval per batch.
+
+### 2.3 What a roster member carries
+
+Each entry is the standard actor object — `serial`, `name`, `player`, plus `acct` when the mobile has
+an account and `webId` when that account is linked:
+
+```jsonc
+{"serial":"0x1F5","name":"Seed000A","acct":"seed_000","webId":"42","player":true}
+```
+
+`acct` is **genuinely optional**: a `PlayerMobile` can have no `Account` at all, and the local test
+world contains such mobiles. Consumers must not assume it is present.
+
+These identity fields are emitted unconditionally, by design — the sidecar is a forwarder, and
+deciding who may see them is the website's job. See §4.
+
+---
+
+## 3. The sidecar side
+
+`guild.roster` writes a **`members` column on the `guilds` board**, not a field inside the existing
+`json` column. That column holds the verbatim `guild.update` line, and a roster written into it would
+clobber the snapshot — name, abbreviation, leader, online count — that `guild.update` owns. Two
+writers across two columns of one row keeps both as plain upserts: neither reads the other's value
+first, so there is no read-modify-write and no ordering requirement between the two kinds. Either can
+arrive first.
+
+`GET /guilds` folds the roster back in as `roster` at read time. A guild that has had a
+`guild.update` but no `guild.roster` yet simply has **no `roster` key** — the honest representation
+of "not known", and deliberately distinct from a guild whose roster is genuinely empty.
+
+`guild.leave` gets **no board projection at all**. The sidecar persists and broadcasts it like any
+event and leaves the board alone; the roster self-corrects on the next `guild.roster`, which the
+shard re-emits whenever the member set changes. Keeping the delta out of the projection is what keeps
+the sidecar a forwarder rather than a thing that maintains state.
+
+### 3.1 Split rosters are reassembled in memory
+
+A roster split across frames is reassembled **before** it is stored, and written once on the frame
+that closes it. Appending to the column per frame was rejected twice over: it would make the write a
+read-modify-write — the exact thing the two-column split exists to avoid — and it would publish a
+torn roster, since a reader hitting `GET /guilds` between frames would see a partial member list
+presented as the whole truth.
+
+This is transport-level reassembly, the same category of work as turning bytes into a line, and it
+holds nothing once a roster is complete. The ordinary single-frame case never enters the buffer at
+all. Around it: a fresh `seq: 0` supersedes an abandoned partial, an out-of-order frame discards the
+partial rather than storing one with an undetectable hole, a continuation with no start is ignored, a
+`server.hello` drops every partial (a reconnected shard restarts each roster at 0), and accumulation
+is bounded so a shard that never sends a closing frame cannot grow the buffer without limit.
+
+### 3.2 This is the first bump that needed a store migration
+
+`store.rs`'s `SCHEMA` is `CREATE TABLE IF NOT EXISTS`, which can add a table but **cannot add a
+column to a table that already exists**. Every schema change up to and including Protocol 3.0 added
+whole tables, so this never mattered and `ALTER TABLE` appeared nowhere in the repo's history.
+`guilds.members` is the first column added to an existing table: without a mechanism it would simply
+never reach an installed sidecar, and every roster write would fail.
+
+The counter is SQLite's own **`PRAGMA user_version`** — an integer in the database header, so it
+costs no table and cannot drift from the file it describes. Each step runs in a transaction together
+with the bump recording it, so a step lands completely or not at all and an interrupted run resumes
+in the right place.
+
+- **A failure aborts startup**, which was already the behaviour. A half-migrated store answers the
+  website with confusing partial data, which is worse than being plainly absent — and the shard dials
+  *out*, so a sidecar that refuses to start never stalls the game.
+- **A database written by a newer sidecar warns and continues.** Every step is additive, so a newer
+  schema has only columns an older reader ignores; refusing to start would turn rolling the binary
+  back — a recovery path — into a dead end.
+
+`sqlx::migrate!` was considered and rejected: it checksums each migration file, so editing an
+already-released migration hard-fails startup, which is a poor trade for a schema of eleven
+JSON-blob tables.
+
+---
+
+## 4. Visibility
+
+Both kinds map to the existing **`guilds`** feature in the website's visibility framework
+([`v3.md` §3](v3.md)). That mapping is required, not cosmetic: rule 2 fails an *unmapped* kind closed
+to admin-only, which would have quietly kept rosters off the public Guilds page forever.
+
+Mapping them is safe because of how the framework strips fields. A roster is the first frame carrying
+locked fields inside an **array** of actors rather than a single nested actor — but the projection
+walker already recurses into arrays and matches `acct`/`webId` **by suffix, on meaning rather than
+spelling**. So a member's account name is stripped below `admin` by exactly the rule that already
+strips `guild.leader.acct`.
+
+The website stores `acct`/`web_id` per member, because that is what lets a linked member be matched to
+a site user at all. It never projects them below `admin`. The difference this rule makes is a public
+page listing character names versus one publishing 150 account names, so it carries a test of its own.
+
+---
+
+## 5. Cross-repo obligations
+
+| Repo | Change |
+|---|---|
+| `servuo-plugins` | member-set diff, `guild.roster` + `guild.leave`, `BridgeJson.Actors`, two `Bridge.cfg` keys, **`overlay.toml` protocol → 4** |
+| `link` | `PROTOCOL_VERSION` → 4, `members` column + `user_version` migration, roster reassembly, `GET /guilds` projection |
+| `module-uo` | `shard_guild_members`, `guild.roster`/`guild.leave` ingest, the kind→feature map entries |
+| `installer` | `backup.rs`'s stated reason for skipping the sidecar DB (§3.2 falsifies it) — docs only |
+| `docs` | this file, `PROTOCOL_2.md` §10.1, `INTEGRATION.md` |
+
+**`overlay.toml` must move in the same PR as the emitters.** CI folds it into the release manifest and
+the installer refuses to pair an overlay and a sidecar whose protocol numbers disagree, so a bump
+landing separately would silently fail to compose into a bundle.
+
+---
+
+## 6. Verification
+
+`servuo-plugins` has no CI build — the plugin compiles only inside ServUO — so "it compiled" is not
+evidence and neither is a clean boot. What was actually run:
+
+1. **A throwaway one-guild spike first** (TEAMS.md Phase 0), against the real Rust sidecar rather than
+   a stub, to retire the unknowns before committing to a four-repo bump. It proved the two-column
+   board shape, measured the line, and surfaced §3.2 — the missing migration mechanism — which no
+   amount of reading would have.
+2. **A live end-to-end run**: 155 members seeded from real `PlayerMobile`s on the local ServUO tree,
+   the cap forced down to 50 so the split path actually fired, producing 50/50/50/5 across four
+   frames; two members then removed on a timer, producing exactly two `guild.leave` frames with the
+   correct serials and a re-emitted roster at `total: 153`.
+3. **A restart test with the shard stopped first**, so its reconnect could not re-emit and fake
+   persistence. The board's row came back byte-identical.
+
+Step 2 is what caught the reassembly bug in §3.1: every unit test passed through it, because they all
+exercised a single-frame roster. The case does not arise until a guild exceeds the cap.
+
+Still outstanding for the cutover: the five-rung shard visibility walk against a live shard, confirming
+`acct`/`webId` never reach a caller below their rung.

website/TEAMS.md

@@ -1946,7 +1946,28 @@ Emit `guild.roster` for a **single** guild against the local ServUO tree
 store, survives a sidecar restart, and comes back out of `GET /guilds`. Then build Phase 1. Days, not
 weeks, and it retires the only unknown in the plan.
 
-### Phase 1 — the roster on the wire (`servuo-plugins` + `link` + `module-uo` + `docs`)
+### Phase 1 — the roster on the wire (`servuo-plugins` + `link` + `module-uo` + `installer` + `docs`)
+
+> **Amended 2026-08-17, after Phase 0 and while building this.** Two corrections to what follows.
+>
+> **The sidecar had no schema-migration mechanism, and this phase is the first change that needs
+> one.** `store.rs`'s `SCHEMA` is `CREATE TABLE IF NOT EXISTS`, which can add a table but cannot add a
+> column to one that already exists — and every schema change up to Protocol 3.0 happened to add
+> whole tables, so `ALTER TABLE` appears nowhere in `link`'s history and the gap was invisible until
+> `guilds.members`. Settled by the org lead: **`PRAGMA user_version` stepped migrations**, each step
+> transactional with the bump recording it; a failure aborts startup (already the behaviour, and safe
+> because the shard dials *out*), while a database from a *newer* sidecar warns and continues so a
+> binary rollback stays a recovery path. Not `sqlx::migrate!`, whose per-file checksums hard-fail
+> startup if a released migration is ever edited.
+>
+> **`installer` joins the phase**, which is why the heading names five repos rather than four.
+> `backup.rs` justifies not copying the sidecar database on two claims: that every table is
+> `IF NOT EXISTS` (which the migration above falsifies) and that the sweeps repopulate everything
+> (already false — `events` is never pruned and the website backfills from `GET /history` on every
+> reconnect). The behaviour is unchanged and correct; only its stated reason needed fixing, and a
+> wrong reason left in place is what lets someone extend it to a case it never covered.
+>
+> The spec for all of it is [`../link/v4.md`](../link/v4.md).
 
 The prerequisite for everything. Nothing in Team core can be built against counts.
 
@@ -1954,7 +1975,8 @@ The prerequisite for everything. Nothing in Team core can be built against count
 and `guild.leave`; `overlay.toml` protocol → 4. Sidecar: `members` on the `guilds` board,
 `PROTOCOL_VERSION` → 4, `GET /guilds` projection. `module-uo`: ingest both kinds, a
 `shard_guild_members` table, the kind→feature map entry and field projection for the new fields.
-`docs/link/PROTOCOL_2.md` + `v3.md` + `INTEGRATION.md`.
+A new `docs/link/v4.md` as the spec of record, plus `PROTOCOL_2.md` §10.1 (which sketched this design
+in 2.0 and had it half-built) and `INTEGRATION.md`.
 
 **Ships:** a richer public Guilds page (real rosters) on its own merit, with no Team code anywhere.
 **Verify:** the five-rung shard visibility walk against a live ServUO + sidecar, confirming `acct`/`webId`