# Protocol 3.0 — Shard content, standings & the visibility framework **Status:** In progress. All work lands on an `edge` branch in each repo; `edge` → `main` is the v3 cutover. **Date:** 2026-07-28 **Codebase:** ServUO 57.4, ``, net48 / x64, Expansion **EJ**. **Companion to** [`PLAN.md`](PLAN.md) (1.0 read/event plane), [`PROTOCOL_2.md`](PROTOCOL_2.md) (2.0 provisioning + world-state streams), [`ADMIN_CONTROLS.md`](ADMIN_CONTROLS.md) (staff write plane), [`INTEGRATION.md`](INTEGRATION.md) (website API). ### Progress Each part is marked off here as it lands on `edge`. §9 carries the same state per sequencing row. | Order | Part | State | Landed on `edge` | |---|---|---|---| | 1 | **A** — visibility framework + actor-leak fix (§3) | ✅ **Done** | website [#109](https://gitea.whitlocktech.com/RunicGateway/website/pulls/109) + [#110](https://gitea.whitlocktech.com/RunicGateway/website/pulls/110), docs [#64](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/64) + [#65](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/65) | | 2 | **B/1** — `world.ruleset` (§5) | ✅ **Done** | servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66) | | 3 | **C** — spawn atlas (§6) | 🟡 **Data pipeline done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables); API/client PR next | | 4 | **B/2** — `points.board` (§7) | ⬜ Not started | — | | 5 | **B/3** — `vendor.listing` (§8) | ⬜ Not started | — | | 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | ⬜ Not started | — | --- ## 1. Why 3.0 A survey of the live ServUO tree against everything the bridge already surfaces end-to-end found that **the bridge covers live *activity* well and covers shard *content and standings* almost not at all.** Covered by 1.0 + 2.0: presence/online, region transitions, char vitals + profile + roster, house registry + IDOC decay, champion spawns, guild board, city governors + term history, help-page queue, total gold supply, player-vendor sales log, deaths/murders/kills, skill gains, fame/karma, quest completes, staff/cheat audit, account linking + creation, town crier + news. Not covered by anything: every leaderboard, every ruleset fact, every "where do I find X", and the entire player economy outside a player's own vendors. 3.0 has **three scope areas**: - **A — The visibility framework (§3).** Admin-configurable, per-feature and per-field audience control over every shard-derived surface on the website. Ships first; the rest depends on it. - **B — Three new wire streams (§5, §7, §8).** `world.ruleset`, `points.board`, `vendor.listing`/`vendor.listing.remove`. - **C — One website-only feature (§6).** The spawn atlas, built from static ServUO data files with no wire involvement at all. --- ## 2. Survey: the full gap list Recorded so the items *not* scoped for 3.0 aren't re-derived later. | # | Gap | Source on the shard | Value | Cost | Status | |---|---|---|---|---|---| | 1 | **Points/loyalty leaderboards** — 25 point currencies | `Scripts/Services/PointsSystems/PointsSystem.cs` → `static List Systems`, each `List{Player,Points}` | Very high | Low | **3.0 §7** | | 2 | **Shard ruleset page** | `Config/*.cfg` via `Server.Config.Get` | High | Very low | **3.0 §5** | | 3 | **Shard-wide marketplace** | `PlayerVendor.PlayerVendors` + `VendorSearch.cs` | Very high | High | **3.0 §8** | | 4 | **Spawn atlas / bestiary** | `Spawns/*.xml` (6,455 spawners), `RevampedSpawns/*.xml` (333), `Data/Regions.xml`, `Data/Locations/*.xml`, `Config/ChampionSpawns.xml`, `Data/teleporters.csv`, `Data/HarvestLocs/*` | High | Medium | **3.0 §6** | | 5 | VvV standings + battle status | `Services/ViceVsVirtue/{ViceVsVirtueSystem,GuildStats,VvVBattle}.cs` | High | Medium | `PROTOCOL_2.md` §10.4 deferred this pending "which PvP system does this shard run?" — **now answered: `VvV.cfg Enabled=True`, `Factions.cfg` off.** Unblocked, not scoped here | | 6 | Skill leaderboards + shard census | `Services/Reports/Reports.cs` → `GetSkillDistribution()`, `CompileGeneralStats()`, `StaffHistory` | High | Low | Spec'd `PROTOCOL_2.md` §14 (Part B phase 6), unbuilt. Shares §7's UI — fold in after | | 7 | Custom mounts/pets codex — ~35 across 4 tiers | `Scripts/Custom/{Companions,Legendary,Named,New Legacy}` | Medium-high | Very low | Pure wiki/CMS content, zero bridge work. The shard's most distinctive content, with zero site presence | | 8 | Community Collections progress | `Services/CommunityCollections/CollectionsSystem.cs` | Medium | Low | Natural public "community goal" widget | | 9 | Seasonal/holiday event calendar | `Services/Seasonal Events/SeasonalEventSystem.cs`, Krampus, Forsaken Foes | Medium | Low | "What's live now / what's next" | | 10 | Crafting / taming / harvesting feeds | `EventSink.CraftSuccess` / `TameCreature` / `ResourceHarvestSuccess` | Medium | Low | Spec'd `PROTOCOL_2.md` §11 #3/#4/#5, unbuilt | | 11 | Virtue progression | `EventSink.VirtueLevelChange`, `Services/Ethics/` | Medium | Low | Spec'd §11 #6, unbuilt | | 12 | Bulk Order Deeds + reward tables | `Services/BulkOrders/`, `Data/Bulk Orders/*` | Medium | Low | Feed spec'd §11 #7; the static reward tables are a free wiki page | | 13 | Guild wars | war state on `Guild` | Low-medium | Low | Spec'd §11 #8, unbuilt | | 14 | Astronomy discovery log | `Services/Astronomy/AstronomySystem.cs` (104 KB save) | Low | Low | Niche completion leaderboard | | 15 | In-game chat relay | `Services/Chat/`, `Logs/Chat/{General,Help,Trade,LFG}` | Low | Medium | Privacy-sensitive; staff-only at most | | 16 | Shard health telemetry | Crash logs, `LayerConflict.log`, `throttle.log`, `world.save.after` counts, AutoSave/AutoRestart schedule | Low-medium | Low | `world.save.after` is already ingested but never charted — world-size-over-time is nearly free | | 17 | Ultima Store / Sovereigns balance | `Store.cfg Enabled=True, CurrencyName=Sovereigns`; `UltimaStore.GetCurrency` | ? | Medium | Only worth it if sovereigns are actually sold | **Excluded permanently** — see [`ADMIN_CONTROLS.md`](ADMIN_CONTROLS.md) Tier H/N: firewall/IP-block, kill/resurrect, jail, item/gold grants, set-access-level, arbitrary `[set`/`[add`. **Noticed during the survey, out of scope:** `Scripts/Custom/PerryOwnerFix.cs` hardcodes an `EventSink.Login` hook granting `AccessLevel.Owner` to account `"ShardOnwerPerry"`. Worth reviewing independently of this work. --- ## 3. Part A — The visibility framework ✅ Done *Landed on `edge`: website [#109](https://gitea.whitlocktech.com/RunicGateway/website/pulls/109) (the framework) and [#110](https://gitea.whitlocktech.com/RunicGateway/website/pulls/110) (the REST-projection gap §3.6.1 records), docs [#64](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/64) + [#65](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/65). Smoke-tested across all five rungs per §11.* ### 3.1 The leak this replaces (verified 2026-07-28) `BridgeJson.Actor()` (`BridgeJson.cs:85-117`) writes `serial`, `name`, **`acct`**, **`webId`**, `player`. `shardState.model.js:346 shapeGuild()` returns `r.payload` verbatim, and `GET /api/v1/public/shard/guilds` (anonymous, `shard.controller.js:131`) serves it. **A guild leader's game account name and website user id are readable on an anonymous public endpoint today.** The same path exists for `shapeGovernor` → `/public/shard/governors`. `Actor` also feeds `guild.join`, `city.update` and `region.enter`, all three in `PUBLIC_KINDS` on the anonymous SSE stream. The framework below is the vehicle for the fix, and the reason it ships before anything else. ### 3.2 Where visibility lives **On the website, never in the sidecar.** The sidecar's job for 3.0 is unchanged in character: accept frames, persist them to its SQLite store, forward them verbatim over WS, and serve store-backed reads that survive a shard outage. It defines no access parameters, no audiences, no field projection, and advertises no capabilities. ### 3.3 The audience ladder `anonymous → logged_in → player → staff → admin`, each rung implying the ones below it. `viewerLevel(req)` resolves: no session ⇒ `anonymous`; authenticated ⇒ `logged_in`; authenticated with a linked shard account ⇒ `player`; moderator/admin role ⇒ `staff`/`admin`. **Staff always satisfy the `player` rung** even without a linked game account, consistent with the existing rule that `/player/*` is role-agnostic self-service. ### 3.4 Two limits an admin cannot override 1. **`acct` and `webId` are admin-only, always.** They are not in-game-visible and are not exposed as configurable fields. 2. **A kind absent from the kind→feature map is never broadcast below `admin`.** Fail closed. This preserves the property that today's static `PUBLIC_KINDS` allowlist is a security boundary rather than a convenience filter. ### 3.5 Configuration ```sql CREATE TABLE IF NOT EXISTS shard_feature_visibility ( feature VARCHAR(48) NOT NULL PRIMARY KEY, enabled TINYINT(1) NOT NULL DEFAULT 1, audience VARCHAR(20) NOT NULL DEFAULT 'anonymous', field_rules JSON NULL, -- {"": ""} for sensitive fields only updated_by INT NULL, updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4; ``` **Not** seeded on boot (this changed during implementation): an **absent row means "use the compiled default"**, so the table starts empty and only ever holds rows an admin has actually touched. The defaults live in one place — `FEATURES` in `shardVisibility.js` — instead of being duplicated into a seeder that could drift from it, and a DB blip degrades to those same defaults rather than to "everything is public". **All ten shard features are covered — the four new ones and the six that already ship — and every default reproduces today's behavior, so the retrofit is a no-op until an admin changes something.** | Feature | Default audience | Sensitive fields (default rung) | |---|---|---| | `status`, `activity`, `champs`, `guilds`, `governors` | `anonymous` | guilds/governors: `leaderAcct` / `leaderWebId` → **admin (locked)** | | `houses` | `anonymous` | `owner` → `staff`, `price` → `staff` (matches today's IDOC-only public view) | | `presence` | `anonymous` | `location` → `staff` (matches today's staff-only, location-gated `/online`) | | `ruleset` (new) | `anonymous` | `connect` → `anonymous` | | `atlas` (new) | `anonymous` | — | | `leaderboards` (new) | `anonymous` | `characterName` → `anonymous` | | `market` (new) | `anonymous` | `ownerName` → `anonymous`, `location` → `anonymous` | ### 3.6 Enforcement — three points, one config New `website/server/src/utils/shardVisibility.js`: - `LADDER = ['anonymous','logged_in','player','staff','admin']`, `rank()`, `meets(viewer, required)` - `viewerLevel(req)` (§3.3) - `KIND_FEATURE` — every event kind → its feature; unmapped ⇒ admin-only (§3.4) - `getConfig()` — DB-backed, cached ~5 s like `uoLinkClient`'s config cache, busted on admin `PUT` - `requireFeature(name)` — **404 when disabled** (don't leak existence), **403 when enabled but the viewer is below the audience** - `projectFeature(name, payload, viewerLevel)` — strips fields whose rung the viewer doesn't meet; `acct`/`webId` always stripped below `admin` Applied at: 1. **Routes** — `requireFeature(…)` on every `/public/shard/*`, `/public/atlas/*` and the shard-derived player routes; `projectFeature` in the controllers, replacing the ad-hoc `shapeGuild`-returns-payload-verbatim path. 2. **SSE** — `shardBroadcast.js` moves from *"one public channel with a static `PUBLIC_KINDS` allowlist plus one admin channel"* to **per-connection filtering**: each subscriber carries its `viewerLevel`; each frame is mapped kind→feature, gated on `enabled && meets(...)`, then passed through `projectFeature` before write. `PUBLIC_KINDS` becomes the seed data for `KIND_FEATURE` rather than a hardcoded gate. **This is the largest single change in Part A and where the security boundary now lives.** 3. **Nav** — `GET /api/v1/public/shard/features` returns only the features the calling viewer can see, so the SPA hides nav entries rather than rendering links that 403. ### 3.6.1 What the first implementation missed (found by the §11 smoke test, fixed) Part A shipped enforcement on the SSE path and on `/guilds` + `/governors`, but the **remaining public REST reads never called into it** — so the same event was projected live and served verbatim from history. Recorded because each miss is a shape the next phase can repeat: - **`/public/shard/feed` returned the stored payload as-is.** `actor.acct` / `actor.webId` were readable *anonymously* for every logged kind (`player.death`, `mob.killed`, `skill.gain`, `guild.join`, …) — broader than the §3.1 leak, which was limited to board holders. - **`/public/shard/idoc` returned `ownerAcct`.** Rule 1 keyed on the exact strings `acct`/`webId`, but `shapeHouse` flattens the actor into `ownerAcct` / `ownerName` / `ownerSerial`. The lock is now on the field's **meaning** — a key that is or ends in `acct`/`webId`, case-insensitively — so flattened spellings are covered and unwritten shapes fail closed. - **The `houses` field rules were dead config.** Neither `getIdoc` nor `getHouses` projected, so the panel offered toggles that did nothing. **Every feature's declared fields must name the keys the read model actually emits**, not just the wire frame's. - **`/feed` filtered on `PUBLIC_KINDS`**, a module-load constant derived from the compiled defaults, so live audience changes never reached it. `visibleKinds(level, config)` resolves the readable set from live config; it deliberately ignores the `stream` flag, which governs SSE fan-out only (market history stays readable with its firehose off). - **`shardEvents.db.list` treated an empty `kinds` array as "no filter"** and fell through to an unfiltered `SELECT`. A fully-gated config would have dumped the whole event log, staff audit included. An empty allowlist now serves nothing. - **`projectValue` recursed into every object**, so a `Date` column came back as `{}`. It walks arrays and plain objects only. The unit tests used JSON fixtures and could not have caught this — the live read did, which is the argument for §11's smoke test over tests alone. **The rule this leaves behind:** *a read path that returns shard data and does not call `projectFeature` is a bug.* Every new surface in Parts B and C — `/ruleset`, `/points`, `/market`, `/atlas` — must project, and must gate its kind set on live config rather than on `PUBLIC_KINDS`. ### 3.7 Admin surface `GET` / `PUT /api/v1/admin/shard/visibility` (admin-only). Validate feature names against the known set and rungs against the ladder; reject any attempt to set a locked field below `admin` — including its flattened spellings (`ownerAcct`, `leaderWebId`), see §3.6.1. Writes an `admin.audit`-style row so visibility changes are traceable. New client panel `routes/admin/ShardVisibility.jsx` at `/admin/shard-visibility`, linked from `ShardAdmin.jsx`. --- ## 4. The version bump and the rollout `PROTOCOL_VERSION` **2 → 3** in `link/sidecar/src/main.rs:27`. v3 is defined as *"adds `world.ruleset`, `points.board`, `vendor.listing` / `vendor.listing.remove`"*. A bump is an operator-visible hard cutover — `web.rs::gate` returns 409 on every protected route on mismatch, `uoLinkSocket.js::handleHello` closes the WS, and the website's declared version is the admin-set `uo_link_config.protocol` column — so it happens **exactly once**, at the end: - Cut an **`edge`** branch from `main` in each of `website/`, `link/`, `servuo-plugins/`, `docs/`. - Every phase PRs into `edge`, never `main`. Feature branches are cut from `edge`. - Part A lands first, alone. - When all parts are built and tested, one `edge` → `main` PR per repo, merged together. **That merge is the v3 cutover.** - A schema migration sets `uo_link_config.protocol` (the existing row **and** the column default) from 2 to 3, so the cutover doesn't require a manual admin edit. `UOLINK_PROTOCOL` still overrides. - No feature-negotiation array anywhere — v3 implies all three kinds. --- ## 5. Part B/1 — `world.ruleset` ✅ Done *Landed on `edge`: servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66). Implementation notes worth keeping:* - ***`shadowguard` is derived, not configured.*** `Shadowguard.cfg` carries only `ReadyDuration` and `RandomizeInstances` — there is no `Enabled` key — so the systems block reports `Core.TOL` (the expansion gate) instead. Same shape for `factions`: `Factions.cfg` has no `Enabled` either, and `Services/Factions/Core/Faction.cs` sets `Settings.Enabled = !ViceVsVirtueSystem.Enabled`, so the frame reads that static rather than inventing a key. **Where a system's on/off state is derived, read the system's own static; only read `Config.Get` where the .cfg key IS the truth.** - **`caps.skill` / `caps.totalSkill` are in tenths** (1000 = 100.0), the way ServUO stores them. Documented in `INTEGRATION.md` and converted in the client, because the raw number is actively misleading rather than merely unhelpful. - **`Config.Get` re-parses when the cached type differs.** `InternalGet` caches the parsed value on the entry and re-parses if `entry.Object is T` fails, so reading `PlayerCaps.SkillCap` as an `int` where ServUO reads it as a `double` is correct (both parse) — it just re-parses. Harmless, but worth knowing before assuming a shared cache. - **The plugin CAN be compile-verified**, contrary to "no standalone build": point Roslyn (`dotnet sdk/*/Roslyn/bincore/csc.dll`, `/langversion:7.3`, net48 reference assemblies) at the whole ServUO `Scripts` tree with `overlay/Scripts/Custom/Bridge/*.cs` substituted for the deployed copy, excluding `Scripts/obj` and `Scripts/bin`. 6,205 files, ~40 s, and it catches every signature error a boot would. Worth doing before every plugin PR. `PROTOCOL_2.md` §10.4 sketches a `world.systems` capability frame that was never implemented (`grep` returns nothing across all four repos). **`world.ruleset` subsumes it**, carrying a `systems` sub-object with the `cityLoyalty` / `vvv` / `factions` booleans §10.4 asked for. §10.4 is marked superseded; no orphan kind is left behind. ### 5.1 Plugin NEW `servuo-plugins/overlay/Scripts/Custom/Bridge/BridgeRuleset.cs`, modelled on `BridgeBoot.EmitHello` — **not** a sweep. Subscribes `BridgeLink.Connected_Core += Emit` so a sidecar that comes up second still learns the ruleset. Built from an **explicit allowlist** of `Server.Config.Get` calls. **Never enumerate `Config.Entries`** (`Server/Config.cs:162`) — it would sweep in secrets. An FNV-1a `rev` over the body makes an unchanged reconnect a site-side no-op (`String.GetHashCode()` is not stable across runs and must not be used). `BridgeConfig.cs` + `overlay/Config/Bridge.cfg`: `RulesetEnabled=true`, `PublicConnectAddress=""`, `RulesetIncludeSchedule=true`. `BridgeBoot.cs`: `reload` → re-emit, `status` → rev/bytes. Not wired to `sweepnow`; it isn't a sweep. ### 5.2 Payload Every block optional, omitted when its system is off: `shard`, `expansion`, `connect` (only from `PublicConnectAddress`), `systems{cityLoyalty,vvv,factions,siege,chat,store,dailyRares,honesty,shadowguard,treasureMaps,vetRewards,testCenter}`, `caps{skill:1000,totalSkill:7000,stat:225,str/dex/int:125,strMax/dexMax/intMax:150}`, `housing{accountHouseLimit:1}`, `accounts{perIp:3,charSlots:7,autoCreate}`, `vetRewards{enabled,rewardIntervalDays:30}`, `loot{feluccaLuckBonus:1000,feluccaBudgetBonus:100,feluccaMaxProps:11}`, `vendors{restockDelayMinutes,maxSell,economyStockAmount}`, `champions{powerScrolls:6,statScrolls:16,scrollChance,transcendenceChance,rankThresholds}`, `treasureMaps`, `vvv{enabled,startSilver:2000,enhancedRules}`, `store{enabled,currencyName}`, `schedule{autoSaveFrequencyMinutes,autoRestart*}`. **Excluded by name — in a code comment and here:** `Server.cfg` (Address/Listen/Port; only `PublicConnectAddress` is published), `Staff.cfg`, `Email.cfg`, `DataPath.cfg`, `Bridge.cfg`, `Compiler.cfg`, `Reports.cfg`, `Client.cfg`. ### 5.3 Sidecar and website Sidecar — `store.rs`: singleton `ruleset(id CHECK(id=1), rev, json, updated_t)` + upsert/get; `main.rs`: new arm in the board-projection match; `web.rs`: `GET /ruleset` served from the store, so it answers during a shard outage (`PROTOCOL_2.md` §12.2). Website — `uoLinkClient.getRuleset()`; `uoLinkSocket.backfill()` (object-shaped, so follow the `getPresence()` block's explicit form, not the array-only `snapshot()` helper); `shardIngest.js` → `shardState.setRuleset`, **not** in `LOGGED_KINDS` (it re-arrives every reconnect and `server.hello` already marks those); `KIND_FEATURE['world.ruleset'] = 'ruleset'`; `shard_ruleset` singleton table (`rev`, `expansion`, `payload JSON`, `t`); `GET /public/shard/ruleset` behind `requireFeature('ruleset')`, returning `null` ⇒ "not published yet". Client — NEW `routes/public/Rules.jsx` at `/site/rules`, alongside `/site/champs|guilds|governors|houses`; live via `useShardFeed({ filter: new Set(['world.ruleset']) })`. ### 5.4 Risk Perf is nil (~3 KB per connect). The only real risk is publishing a secret, mitigated by the explicit allowlist, the no-`Config.Entries` rule, the named exclusion list, and a manual eyeball of the emitted frame during verification. --- ## 6. Part C — Spawn atlas / bestiary (website-only) **No plugin, no sidecar, no `Bridge.cfg` knob, no new kinds.** Not part of the v3 wire change. > **Status:** data pipeline landed on `edge` — website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) > (parsers, build/import CLI, tables, artifact). API + client pages are the second website PR. > Part C ships as **two** website PRs, not one: the parsing half is where the correctness risk > lives, and burying it under routes and React would have meant reviewing it in a 10k-line diff. > Full operator documentation: [`docs/website/SPAWN_ATLAS.md`](../website/SPAWN_ATLAS.md). > > **§6 below is the original design and is partly superseded.** §6.1 records two decisions that were > rejected in review and replaced (the committed artifact, and the fixed facet list); §6.2 records > the corrections the real ServUO data forced. Read both before trusting §6. **Decision (revised at implementation time): the shard's ServUO tree is the single source of truth, re-derived on every server boot.** The original plan here was a committed generated artifact plus an idempotent import. That was rejected in review for two reasons, recorded in §6.1: a snapshot in the repo goes stale as a shard's maps change, and the design leaned on a fixed facet list that no shard is obliged to keep. Still not a browser-served blob; still parsed server-side only. New in `website/server/`: - `src/utils/spawnAtlasParse.js` — **pure functions, no fs**, so they are unit-testable in CI without a ServUO tree: `parseObjects2()`, `parsePoints()`, `parseRegions()`, `parseLocations()`, `resolveRegion()`. - ~~`scripts/buildSpawnAtlas.js` and a committed `db/data/spawnAtlas.*.json` artifact~~ — dropped, see §6.1 R1. Replaced by `src/utils/spawnAtlasSource.js` (the only thing that reads a ServUO tree, shared by the boot path and the CLI) and a `scripts/importSpawnAtlas.js` that is a thin CLI over the model. `package.json` gains `atlas:import` only. - `src/model/shardAtlas/{shardAtlas.db.js,shardAtlas.model.js}` following the `shardState` split. - `src/router/v1/public/atlas.{router,controller}.js`; `test/spawnAtlas.parse.test.js`. Two parsing notes that matter: - `` is `Type:MX=n:SB=…` segments joined by `:OBJ=` — verified against `trammel.xml`, where a single point carries six types. Split on `:OBJ=`; the token before the first `:` is the type. - **The high-value transform:** point-in-rect each spawn against the facet's `Regions.xml` rects (highest `priority` wins), falling back to the nearest `Data/Locations` landmark, else `"Wilderness"`. This is what turns *"lizardman at 5411,1234"* into ***"Despise, Felucca"*** and is the entire reason the page is worth building. `Regions.xml` is genuinely nested and needs a ~120-line recursive tokenizer **or** one devDependency (`fast-xml-parser`) — the server has zero XML deps today, so that is an explicit call to make at implementation time. The flat `` files need only regex/streaming; **do not** put 10.5 MB through a DOM parser. Tables: `shard_spawn_creatures` (slug PK, name, total, facets JSON), `shard_spawn_points` (slug, facet, x, y, region, landmark, max_count, tod_*), `shard_regions`, `shard_landmarks`, `shard_champion_spawns`, `shard_atlas_meta`. Plain `INDEX` on name, **not `FULLTEXT`** — ~1,500 creature rows makes a `LIKE` scan free, and FULLTEXT brings min-token-length trouble for names like "orc". No FKs, consistent with every existing `shard_*` table. Routes at `/api/v1/public/atlas`, **not** under `/shard` — the atlas is static shard *content*, not live shard *state*; it must not look sidecar-dependent, and unlike `/shard/*` it *should* be `siteMode`-gated like `/posts` and `/wiki`. `GET /creatures?q=&facet=`, `/creatures/:slug`, `/regions`, `/landmarks`, `/champions`, `/meta`, all behind `requireFeature('atlas')`. Admin: `GET /admin/shard/atlas/status` (artifact-vs-DB drift) and `POST /admin/shard/atlas/import`. **Build stays CLI-only.** Client: `routes/public/Atlas.jsx` (`/site/atlas`) and `AtlasCreature.jsx` (`/site/atlas/:slug`). **Payload risk** — *superseded by §6.1 R1; nothing is committed.* The field selection it describes still applies at parse time: every `` field the site cannot use (`UniqueId`, all trigger/refractory/proximity/sequential fields, sound ids) is dropped, keeping Name/Map/X/Y/W/H/Range/MaxCount/MinDelay/MaxDelay/TOD*/types. Parsed data never reaches the browser; the browser sees only paginated API responses. **Operator re-run story** — *revised by §6.1 R1.* Spawns changed → restart, or `npm run atlas:import` / `POST /admin/shard/atlas/import` to apply without one. `shard_atlas_meta` holds a sha256 per source file, so the server can tell on boot whether anything changed, and `GET /admin/shard/atlas/status` reports drift. If the change would remove a facet it is staged for approval rather than applied (§6.1 R3). Full detail in `docs/website/SPAWN_ATLAS.md`. ### 6.1 What implementation changed Two design decisions in §6 were rejected in review and replaced; the rest are corrections the real ServUO data forced. Kept as a diff rather than edited in place, because each is a trap the next person would otherwise re-enter. **R1. The committed artifact is gone — the tree is re-parsed on every boot.** §6 proposed building a generated artifact, committing it, and importing it. Two problems. A shard's maps change over its life, so a snapshot in the repo silently drifts from the world players actually see; and the build/ import split existed only to work around the website container not having a tree, which is a deployment question (mount it) rather than a reason to freeze data. The server now hashes the source files on boot and re-derives the atlas when they differ. `scripts/buildSpawnAtlas.js`, the 1.41 MB artifact, and the whole encode/decode seam it needed are deleted. **R2. Nothing may name a facet.** The first implementation carried a lookup table of the six stock UO facets to reconcile the spelling drift between sources. A shard may add facets, replace them outright, or rename them when its maps are updated, and a built-in list mishandles all three silently. Reconciliation is now by *matching* against the facet set discovered from the shard's own spawn and region data — exact key, then prefix in either direction — with an unmatched name keeping its own rather than being forced into a wrong bucket. **R3. Two contracts on the boot path.** It never blocks startup: no path, an unreadable mount, a malformed file or a database error is caught and logged, and the site comes up serving whatever atlas it had. And a refresh that would REMOVE a facet is never applied automatically — facet loss is indistinguishable at boot from a half-copied or mid-update tree, so it is staged in `shard_atlas_pending` for an admin to approve or reject. Only the decision is stored (source hashes + the facet diff, a few KB); approving re-parses, so what lands matches the tree at approval time. A rejection is remembered against those hashes so it does not re-prompt every restart. ### 6.2 What the build against real data changed Six corrections to the design above, from running it against stock ServUO 57.4. Kept as a diff rather than edited in place, because each one is a trap the next person would otherwise re-enter. **1. Six facets, not thirteen.** The design said `spawnAtlas..json ×13`, assuming one facet per spawn file. There are 13 files but only **6** facets — `Eodon.xml`, `GravewaterLake.xml`, `TreasuresOfKotl.xml` and the other named-area files carry TerMur/Trammel points. The facet comes from each record's own ``, never the file name, and the artifact shards 6 ways. **2. The XML dependency call: hand-rolled, zero deps.** §6 left `fast-xml-parser` vs a ~120-line tokenizer open. Resolved as the tokenizer — a deliberate *subset* parser covering only what these files use. The server keeps zero XML dependencies at any tier. **3. Facet names disagree between sources — a silent failure.** `Data/Locations/*.xml` spells them `Ter Mur` and `Tokuno Islands`; `` and `` say `TerMur` and `Tokuno`. Unreconciled, the landmark bucket is keyed differently from the points looking it up, so the fallback never fires and **every unregioned spawn in Ter Mur and Tokuno reads "Wilderness"** — a plausible-looking atlas that is quietly wrong for two facets. All facet names now pass through `normalizeFacet()`. **4. Spawn type tokens carry XmlSpawner directives.** `` types are not always bare class names: `Fairy,{RND,4,8}`, `alchemist/z/-50`, `Agralem/Name/Agralem`, `greatape,true`. Taken literally they invent creatures that do not exist *and* split real ones in two, since `Fairy` and `Fairy,{RND,4,8}` slug apart. 71 of 845 entries were affected; stripping at the first `/` or `,` leaves **800** real creatures. (The design's "~1,500 creature rows" estimate was high; 800 only reinforces the plain-`INDEX`-not-`FULLTEXT` call.) **5. The artifact would have been 1.41 MB, not "well under 1 MB" — and is now moot.** Dropping the unused `` fields as the design directed still left 4.40 MB; three further encodings brought it to 1.41 MB, and getting under 1 MB would have meant dropping the spawner `name`. The size budget in §6 was simply optimistic for 6,455 points. Superseded by §6.1 R1: there is no artifact, so there is no payload to budget and no encode/decode seam to keep in sync. **6. `DELETE`, not `TRUNCATE`.** The design said "TRUNCATE + batched INSERT in one transaction", which does not hold: `TRUNCATE` is DDL in MariaDB and implicitly commits, so a mid-import failure would leave the atlas half-loaded. `DELETE` is transactional, and at ~7k rows the cost is irrelevant. Point ids are also assigned explicitly rather than by `AUTO_INCREMENT`, because the join rows need them and `conn.batch()` reports no usable `insertId`. **Measured result:** 6,455 points, 800 creatures, 23,927 point/type rows, 387 regions, 558 landmarks, 25 champion altars. The placement transform resolves **83.2%** of points (3,689 by region, 1,690 by landmark, 1,086 Wilderness). **One thing the design got exactly right:** the point-in-rect transform really is the reason to build this. "Where does a lizardman spawn?" answers *Shrines, Isamu-Jima, Yew* across three facets. --- ## 7. Part B/2 — `points.board` Two deliverables: a diff sweep for the boards, and a `points` block folded into `char.profile` — the `PROTOCOL_2.md` §10.3 `titles` precedent (read-model enrichment, no new request kind). ### 7.1 Plugin NEW `BridgePoints.cs`, copying the `BridgeHousing.cs` diff-sweep shape (`Initialize` → `ServerStarted`, `Connected_Core += OnConnected` clearing `_last` + `Rearm()`, `SweepOnce()`, `Status()`, skip when `!BridgeLink.Connected`, try/catch throughout). Which systems: default to `PointsSystem.Systems` filtered to `ShowOnLoyaltyGump == true` — reuse the shard's own "this is player-facing" signal rather than inventing one. `Bridge.cfg PointsSystems=` overrides. Null-guard `PointsSystem.Systems`; it is a mutable static populated by 25 separate subsystem constructors. **The perf trap.** `PlayerTable` is a plain `List`, and `QueensLoyalty` has `AutoAdd`, so it can hold an entry for every `PlayerMobile` that ever existed. A naive `.OrderByDescending().Take(N)` across 25 systems is 25 full sorts — at 20,000 historical characters, ~7.5 M comparisons, tens of ms on the Core thread. `BRIDGE_PLUGIN_PLAN.md` §1 found that nothing except bulk profile generation comes close to a frame budget; this would be the second thing that does. **Mitigation — single-pass bounded selection** into a fixed N-element sorted array (N=10): O(n·N) with tiny constants and one allocation. Skip `Player == null || Deleted` and `Points <= 0`. ~500 k cheap iterations at a 300 s interval. Diff signature per system: `concat(serial + ":" + (long)points)` over the top N, plus the entry count. **No `points.remove`** — the system set is fixed, the same argument `city.update` already uses. ### 7.2 Payload — one frame per system 25 × ~600 B rather than one 12 KB frame, matching `champ.update` / `guild.update`: ```jsonc {"t":…,"kind":"points.board","system":"QueensLoyalty", "nameString":"Queen's Loyalty","nameNumber":1114938, "maxPoints":30000,"showOnGump":true,"players":842, "top":[{"rank":1,"serial":"0x1A2B","name":"Darrow","points":29500}, …]} ``` `nameString` **and** `nameNumber` are both emitted (a `TextDefinition` may be a cliloc), resolved website-side — the contract `titles.reward` already documents at `BridgeProfile.cs:107-110`. **Entries are written inline as `{serial, name}` — never via `BridgeJson.Actor`.** Deliberate even though the website can now reveal fields by rung: `acct`/`webId` are not needed here, because the website resolves serial→user from its own `shard_account_links` mirror for staff views. Keep the wire minimal. ### 7.3 `char.profile` enrichment `BridgeProfile.cs` gains `WritePoints(sb, m)` alongside `WriteTitles`: `"points":[{system,nameString,points,maxPoints}]`, omitting systems with no entry or 0 points. **Deliberately no `rank`** — computing it means scanning each system's `PlayerTable` once per profile (25 × n), which would dominate the measured 0.069 ms/profile budget. The website derives rank from the board when the character appears in the top N. Gate behind `PointsProfileRank=false` if it is ever wanted. ### 7.4 Config, sidecar, website `Bridge.cfg`: `PointsSweepSeconds=300`, `PointsLeaderboardEnabled=true`, `PointsTopN=10`, `PointsSystems=` (blank ⇒ auto), `PointsProfileEnabled=true`, `PointsProfileRank=false`. `BridgeBoot.cs`: `Rearm()` in `reload`, `SweepOnce()` in `sweepnow`, `Status()` in both. Sidecar — `points_boards(system PK, name, json, updated_t)`; `main.rs` arm keyed on `system`; `GET /points` and `GET /points/:system`. Website — `shard_points_boards(system PK, name, name_cliloc, max_points, players, show_on_gump, payload JSON, t)`. **The top-N list stays in `payload`** — a fixed-size list read whole, exactly like `shard_governors.candidates`. Do not normalize into a `shard_points_entries` table until a per-character reverse lookup is actually needed. `shardIngest.js` → `upsertPointsBoard`, **not** in `LOGGED_KINDS` (board state, like `guild.update`). `KIND_FEATURE['points.board'] = 'leaderboards'`, with `characterName` as its per-field rule. `GET /public/shard/points` and `/points/:system` behind `requireFeature('leaderboards')`; validate `system` ≤ 48 chars. **No new player route** — per-character points ride inside `char.profile`, already served by `GET /player/shard/char/:serial` with its `shardLinks.ownsAccount` check. Client — NEW `routes/public/Leaderboards.jsx` at `/site/leaderboards`; a "Loyalty & Points" section added to `components/CharacterSheet.jsx`, one edit serving both `PlayerCharacter.jsx` and `AdminCharacter.jsx`. --- ## 8. Part B/3 — `vendor.listing` ### 8.1 It cannot be an RPC, and this is load-bearing `rpc.rs::try_route` correlates on the **first** frame carrying a matching `reqId` and resolves a single `oneshot`. A chunked reply sharing one `reqId` would deliver chunk 1 to the HTTP caller and **leak chunks 2..N onto the broadcast feed**. `REPLY_TIMEOUT` is 10 s (the client waits 12 s), so a whole-world snapshot could not fit regardless. ⇒ **a per-vendor diff sweep on the broadcast stream**, like `champ.update` / `house.update`. The existing per-account `vendor.snapshot` RPC is untouched; the player portal keeps using it. Kinds: `vendor.listing` (one frame per vendor, authoritative for that vendor) and `vendor.listing.remove`. Payload: `serial, shopName, owner:{serial,name}, map, x, y, region, house, count, truncated, items:[{serial,itemId,hue,amount,price,name,cliloc,child}]`. ### 8.2 Two perf traps Measured baseline (`BRIDGE_PLUGIN_PLAN.md` §1): 30 vendors / 1,200 listings = 0.343 ms via `pack.Items` + `GetVendorItem`; extrapolated to 500 vendors / 40,000 listings ≈ 12 ms per full pass. Except: 1. **`VendorSearch.GetItemName(Item)` is a packet builder, not a field read.** It constructs an `ObjectPropertyList`, calls `GetProperties`, serialises, then byte-parses the packet (`VendorSearch.cs:681-789`) — per item. Across 40,000 items in one tick that is a multi-hundred-millisecond stall. **Mandatory: never call it in the sweep.** Emit `itemId`, `hue`, `amount`, `price`, `item.Name` (the plain field, null for most) and `item.LabelNumber`, resolving display names website-side — exactly what `char.profile.equipment` already does (`BridgeProfile.cs:173`). 2. **`VendorSearch.GetItems(PlayerVendor)` is private** (`:791`). The reusable public API is `GetItems(Container, List)` (`:807`), which recurses into sub-containers, so real item counts run above the top-level `pack.Items` the 0.343 ms measurement used. Budget accordingly. ### 8.3 Mitigations - **Amortized round-robin sweep** — `MarketSweepSeconds=60`, at most `MarketSweepBatch=25` vendors per tick, with a persistent cursor over `PlayerVendor.PlayerVendors`. Full coverage in `ceil(vendors/25) × 60 s`, with **per-tick cost bounded independent of world size**. This is the one genuinely new pattern versus the existing sweeps and should be flagged in review. - **Per-vendor signature diff** (`count | Σ(serial ^ price) | x | y | shopName`), as `BridgeHousing` does — most vendors are static, so steady-state emission is near zero. - **`MarketMaxListings=250`**, then `"truncated":true`. `BridgeJson.Parse` caps *inbound* at 1 MB; outbound is uncapped and `shard.rs::read_line` will allocate whatever arrives. - On `Connected_Core`, clear `_last` **and reset the cursor**; the re-emit is self-throttled by the round-robin window. ### 8.4 Player opt-out and privacy **Honour `pv.VendorSearch`** — ServUO's own per-vendor opt-out, which `DoSearch` filters on (`:62`). Skip opted-out vendors entirely; the seen-set removal then drops them from the board, so **a player who hid their vendor in game is hidden on the website too.** Also skip `Map == null || Map.Internal` and `Backpack == null`, matching `DoSearch`. A vendor's shop name, owner character name and location are **already globally visible in-game** — the stock Vendor Search gump surfaces exactly this set to any player — which is why they default to `anonymous`. They remain per-field configurable (`ownerName`, `location`) so an admin can tighten them. Account name and website user id never go on the wire. ### 8.5 Sidecar and website Sidecar — one table `vendors(serial PK, shop_name, owner_name, map, x, y, region, count, json, updated_t)` storing the whole-vendor blob. **No `vendor_items` table** — the sidecar's job here is outage resilience (`PROTOCOL_2.md` §12.2), not search; search lives in MariaDB. Endpoint is **`GET /market`**, not `/vendors` — axum would route the latter fine, but the collision with the per-account RPC is a readability trap. Website — `shard_vendors` + `shard_vendor_items` (indexes on `vendor_serial`, `price`, `item_id`, `display_name`; delete-then-insert per vendor in one transaction; no FKs). `shardIngest.js` handles both kinds; **not** in `LOGGED_KINDS`. `KIND_FEATURE['vendor.listing'] = 'market'`, but the market feature's **SSE mapping is disabled by default**: a live firehose of full vendor inventories would be the site's single biggest bandwidth consumer, and no page needs it live. The page is a paginated DB query with a staleness stamp; an admin can turn the stream on. `uoLinkSocket` paginates `/market` on reconnect, bounded by `MARKET_SNAPSHOT_MAX = 5000` vendors so a pathological world cannot hang startup. `GET /public/shard/market?q=&minPrice=&maxPrice=&itemId=&map=®ion=&sort=&limit=&offset=` (limit 1..100, default 50; `q` ≤ 60 chars; `sort ∈ {price_asc, price_desc, recent}`) and `/market/vendors/:serial`, behind `requireFeature('market')`. **Rate-limit it** — this is the first genuinely expensive public endpoint; `express-rate-limit` is already a dependency. ### 8.6 The open dependency — cliloc names `CharacterSheet.jsx:14-15` already documents the gap ("without a cliloc table on the site we can only show literals") and renders equipment as `id {itemId}`. Search-by-name needs that table. - **Recommended:** `scripts/buildClilocs.js` reads the UO client's `Cliloc.enu` → committed `db/data/clilocs.json`; ingest denormalizes into `shard_vendor_items.display_name`. Same build-artifact pattern as §6, and it **also fixes the character sheet**. - **Fallback:** ship with item-art + price + region filters, and name search only over renamed items. This decision is the reason §8 is sequenced last. ### 8.7 Client `routes/public/Market.jsx` at `/site/market`, with a *"prices last refreshed N minutes ago"* banner driven by `staleAt` (the oldest `shard_vendors.updated_at`). The round-robin sweep means data is inherently up to one full cycle old, and the UI must say so. --- ## 9. Sequencing | Order | Part | Repos touched | Wire change | State | |---|---|---|---|---| | 1 | **A** — visibility framework + actor-leak fix | website, docs | none | ✅ Done | | 2 | **B/1** — `world.ruleset` (§5) | all four | new kind | ✅ Done | | 3 | **C** — spawn atlas (§6) | website, docs | none | 🟡 Pipeline done, API/client next | | 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | ⬜ | | 5 | **B/3** — `vendor.listing` (§8) | all four | new kinds | ⬜ | | 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | ⬜ | --- ## 10. Documentation obligations - This file (`link/v3.md`) is the canonical 3.0 design. - `PROTOCOL_2.md` §10.4 gains a note that `world.systems` is superseded by `world.ruleset`, and that the deferred VvV question is answered (`VvV.cfg Enabled=True`, Factions off). - `INTEGRATION.md` — catalog entries and §6 consumer sections for each new kind, plus the v2→v3 upgrade note for operators. - `PLAN.md` — phasing. - `website/BACKEND_DESIGN.md` — every new table and route, and **the visibility framework as a security contract**: the audience ladder, the two locked rules, and the fail-closed kind map belong in the security section. - NEW `website/SHARD_VISIBILITY.md` — admin-facing: what each feature exposes, what each rung means, what cannot be loosened. - NEW `website/SPAWN_ATLAS.md`, NEW `website/MARKETPLACE.md`. - `PROJECT_TREE.md` in each touched repo. - `npm run swagger` **and** `npm run routes:manifest` on every route-touching PR — both are committed artifacts, and `test/routeManifest.test.js` fails on drift. **Follow-up, not scoped for 3.0:** the Android app consumes the same public/player shard API and will need `/public/shard/features` to hide its own nav. Track separately against `android-app/`. --- ## 11. Verification **Plugin** — `servuo-plugins\deploy.ps1 -ServerPath -Verify`, inspect the ADD/CHANGE list, then re-run without `-Verify` (ServUO must be stopped). Boot with `tools/stub_sidecar.ps1` listening and **confirm the compile banner in the console, not merely the absence of errors** — `BRIDGE_PLUGIN_PLAN.md` §1 warns that a failing build is silently ignored and the previous `Scripts.dll` reloads. Then `[bridge status`, `[bridge sweepnow`, `[bridge reload`. - §5: eyeball the emitted `world.ruleset` frame for anything sourced from `Server.cfg`, `Staff.cfg`, `Email.cfg`, `DataPath.cfg` or `Bridge.cfg`. - §8: with a seeded world, time one sweep tick and confirm the batch cap holds it under ~1 ms. **Sidecar** — `cargo build && cargo clippy`; `curl -H "Authorization: Bearer " localhost:8080/ruleset` (and `/points`, `/market`); confirm `X-UOLink-Version: 3` and that a client declaring 2 receives a 409. **Website server** — `DB_HOST=127.0.0.1 DB_PORT=59999 node --test`. New tests, each modelled on an existing sibling: `test/shardVisibility.test.js`, `test/shardBroadcast.visibility.test.js`, `test/shardIngest.{ruleset,points,market}.test.js` (after `shardIngest.protocol2.test.js` — stubbed deps, asserting routing and `logged` flags), `test/spawnAtlas.parse.test.js` (pure functions, inline fixtures). Then `npm run routes:manifest` and `npm run swagger`, committing both. **Full stack** — against a local MariaDB: apply `db/schema.sql` (idempotent), start the server, confirm `uoLinkSocket` backfill logs the new snapshot lines and that `uo_link_config.protocol` migrated to 3, then load `/site/rules`, `/site/atlas`, `/site/leaderboards`, `/site/market`. **Visibility smoke test** — for each of the five rungs, walk every shard page and confirm gating and field projection match the configured matrix. Same shape as the 200-routes × 5-access-levels sweep already run for the domain split. --- ## 12. Critical files | File | Why | |---|---| | `website/server/src/utils/shardBroadcast.js` | The security boundary; reworked from a static allowlist to per-connection audience filtering. **The highest-risk file in 3.0.** | | `website/server/src/utils/shardVisibility.js` (new) | Ladder, kind→feature map, projection | | `website/server/src/utils/shardIngest.js` | The dispatcher every new kind routes through | | `website/server/src/model/shardState/shardState.model.js` | The `shape*` projections, including the `shapeGuild` leak §3.1 fixes | | `servuo-plugins/overlay/Scripts/Custom/Bridge/BridgeHousing.cs` | Cleanest copy of the diff-sweep pattern; template for `BridgePoints.cs` and `BridgeMarket.cs` | | `link/sidecar/src/main.rs` | `PROTOCOL_VERSION` 2→3 and the board-projection match | | `website/server/db/schema.sql` | All new `shard_*` tables plus the `uo_link_config.protocol` migration |