diff --git a/README.md b/README.md index 9ceffd5..679624b 100644 --- a/README.md +++ b/README.md @@ -19,6 +19,11 @@ ci/ cross-cutting CI/quality notes | [BACKEND_DESIGN.md](website/BACKEND_DESIGN.md) | API contract, DB schema, security model | | [HERO_EDITOR.md](website/HERO_EDITOR.md) | Hero canvas editor feature spec | | [WIKI_UPGRADE.md](website/WIKI_UPGRADE.md) | Wiki subsystem upgrade notes | +| [SHARD_VISIBILITY.md](website/SHARD_VISIBILITY.md) | Who sees which shard data — the admin-configurable audience framework | +| [SPAWN_ATLAS.md](website/SPAWN_ATLAS.md) | The bestiary / spawn atlas: what the shard contains, parsed from its own ServUO tree | +| [CLILOCS.md](website/CLILOCS.md) | UO's id → name table: converting one from your client so items have names | +| [UOFIDDLER.md](website/UOFIDDLER.md) | **Operator runbook** — step-by-step extraction from your own UO client (cliloc table, creature art) | +| [MARKETPLACE.md](website/MARKETPLACE.md) | The player-vendor index: how it is gathered, what it costs, how to tune it | | [website-README.md](website/website-README.md) | Snapshot of the website repo's README (setup/run reference) | | [PROJECT_TREE.md](website/PROJECT_TREE.md) | Auto-generated snapshot of the repo's tracked file layout | @@ -27,6 +32,7 @@ ci/ cross-cutting CI/quality notes |---|---| | [INTEGRATION.md](link/INTEGRATION.md) | How the website integrates with the uo-link sidecar | | [PROTOCOL_2.md](link/PROTOCOL_2.md) | Protocol 2.0 / 2.1 design | +| [v3.md](link/v3.md) | Protocol 3.0 design — shard content/standings streams + the visibility framework | | [ADMIN_CONTROLS.md](link/ADMIN_CONTROLS.md) | Staff write-plane (kick/ban/broadcast, page queue) | | [SHARD_PREREQS.md](link/SHARD_PREREQS.md) | Shard-side prerequisites for the bridge | | [PLAN.md](link/PLAN.md) | uo-link build plan | diff --git a/android/PLAN.md b/android/PLAN.md index 8891a0d..af75c06 100644 --- a/android/PLAN.md +++ b/android/PLAN.md @@ -1,6 +1,6 @@ # Android App — Plan -Status: **M0–M7 landed; M7 (push notifications) both parts done — Part 1 backend (website#78) and Part 2 app (Android-app#15) plus a small `push.ntfyUrl` settings addition (website#79). Remaining: set the shard's `NTFY_*` deploy config so push lights up, and cut the v1 tag. M9 (native SSO login) is now underway backend-first — the Mobile SSO Authorization Bridge is being built in `website/` + `docs/` ahead of the app-side client (§4.2, §9 M9); custom-scheme callback only for now, App Links deferred (see [`APP_LINKS.md`](./APP_LINKS.md)).** This document is the +Status: **M0–M7 landed; M7 (push notifications) both parts done — Part 1 backend (website#78) and Part 2 app (Android-app#15) plus a small `push.ntfyUrl` settings addition (website#79). Remaining: set the shard's `NTFY_*` deploy config so push lights up, and cut the v1 tag. M9 (native SSO login) is now underway backend-first — the Mobile SSO Authorization Bridge is being built in `website/` + `docs/` ahead of the app-side client (§4.2, §9 M9); custom-scheme callback only for now, App Links deferred (see [`APP_LINKS.md`](./APP_LINKS.md)). **M11 (Protocol 3.0 shard parity)** is scoped and next: the app sees none of the four shard features v3 added (`ruleset`, `leaderboards`, `market`, `atlas`) and does not consult `GET /public/shard/features`, so it gates shard nav on session role alone while an admin can switch any of those surfaces off or raise its audience — the v3 `edge` → `main` cutover is held until it lands (§9 M11).** This document is the design contract for the `RunicGateway/Android-app` repo. It was written before implementation so the API changes it depends on could be landed in `website/` and `docs/` first. The authoritative API reference is the committed OpenAPI spec at `website/server/swagger/swagger-output.json` (regenerated @@ -631,6 +631,8 @@ not rank). | News & content | everyone | `/public/posts/:category`, `/public/pages/:slug` | | Wiki | everyone | `/public/wiki`, `/public/wiki/categories`, `/public/wiki/tags`, `/public/wiki/:slug` | | Shard (live) | everyone | `/public/shard/*` + `/public/shard/stream` (SSE) | +| **Rules / Leaderboards / Market** | everyone, *if the shard publishes them* | `/public/shard/{ruleset,points,market}` (M11) | +| **Atlas** (bestiary) | everyone, *if the shard publishes it* | `/public/atlas/*` (M11) | | Contact | everyone | `/public/contact` | | **My Account** | signed-in | `/player/account/*` (or `/admin/account/*` for staff — see §6.4) | | **My Characters / Vendors / Houses** | `player` (linked) | `/player/shard/*` | @@ -639,6 +641,12 @@ not rank). Guidelines: - The menu is **declarative + data-driven**, not a pile of `if role ==` checks — one list of entries with a `minAccess`/`requiredCapability` field, filtered by the session. +- **Session role is not the only gate on shard surfaces (M11).** Every shard-derived feature is + *admin-configurable* — it can be switched off or raised to a higher audience rung — so a shard entry + is filtered by the session role **and** by `GET /public/shard/features`, which reports the features + the caller may actually reach. While that answer is unknown (in flight, or the lookup failed) the app + shows everything: the server gates regardless, and a nav that flickers in on every load is worse than + a link that briefly `403`s. - Never hide the fact that more exists behind auth in a way that misleads; anonymous users see public groups and a "Sign in" affordance. - The server is the source of truth: a hidden/greyed item is a UX convenience; every gated call still @@ -663,9 +671,15 @@ Guidelines: ### 6.2 Public shard (live) - Status/online/feed/economy/champs/guilds/governors(+history)/presence/houses/idoc — the `/public/shard/*` GETs. -- **Live updates** — subscribe to `GET /public/shard/stream` (SSE, safe kinds only) and patch the - in-memory boards in place (champ/guild/city/house/presence update+remove frames). Reconnect with - backoff; fall back to poll if SSE drops. +- **Live updates** — subscribe to `GET /public/shard/stream` (SSE) and patch the in-memory boards in + place (champ/guild/city/house/presence update+remove frames). Reconnect with backoff; fall back to + poll if SSE drops. What arrives on the stream is **resolved from the caller's audience rung at + subscribe time**, not from a fixed allowlist (Protocol 3.0 §3.6) — the stream request carries the + bearer like every other call, so a signed-in app session sees exactly what the same account sees on + the web. +- **Visibility + the Protocol 3.0 surfaces (M11)** — `GET /public/shard/features` drives which of these + the menu offers; `GET /public/shard/{ruleset,points,points/:system,market,market/meta,market/vendors/:serial}` + and `GET /public/atlas/*` are the new reads. Full contract and traps in §9 M11. ### 6.3 Player self-service & game data (bearer) - **Account** — `GET /player/account`; `PATCH /player/account/username`; @@ -675,6 +689,11 @@ Guidelines: - **My game data** — `GET /player/shard/roster/:account`, `/char/:serial`, `/vendors/:account`, `/sales`, `/houses`. All ownership-checked server-side; a `503` means shard/sidecar down → show an "offline, retry" state (see §7). +- **The character sheet carries two things the app does not yet read (M11):** the `points` block + (per-character loyalty/points standings, Protocol 3.0 §7.3) and the server-resolved cliloc names on + `equipment[].clilocName` / `titles.rewardResolved` (§8.6). Both are served **ungated** on this route — + a character's own standings are self-service data and do not depend on the public `leaderboards` + feature being visible, which is the behavior the app must mirror rather than re-gate. - **Presentation is text-only for v1.** Character sheets and vendor listings render as data/text — no item icons or paperdoll art. A richer "pretty paperdoll" view is a **future** enhancement (pending the art/asset work on the platform side) and is explicitly out of the first release. @@ -883,10 +902,98 @@ push, and Play (M6–M8) follow the designed app. shard-write actions degrade gracefully when the sidecar is offline. Excluded: hero/CMS block editor, Discord-bot config, uo-link config, OAuth-provider setup. +12. **M11 — Protocol 3.0 shard parity** (post-v1; scoped 2026-07-30). The website's Protocol 3.0 work + added four shard features and, with them, an **admin-configurable visibility framework** the app + knows nothing about. `link/v3.md` §10 deferred the app side as a follow-up; it is now scoped + deliberately, and **the v3 `edge` → `main` cutover is held until both parts land** so web and app + surface the same shard on the same day (decided 2026-07-30). + + Neither part is coupled to the cutover *merge order*, which is what makes holding it a schedule + decision rather than a technical one: against a pre-v3 website every new route and + `/public/shard/features` simply `404`s, and each consumer below falls back to exactly today's + behavior. The app declares no protocol version and never talks to the sidecar. + + - **Part 1 — the visibility rules + the read-model adds.** The security-shaped half, reviewed on + its own: + - `GET /public/shard/features` → `{ level, features[] }`: the features **this caller** may reach. + A new singleton cache mirrors the web client's (`lib/useShardFeatures.js`): per-viewer but + stable for a session, invalidated on sign-in/out and on a server switch. + - `MenuEntry` gains `feature: String?` beside its existing `access`, so the one declarative menu + (§5) filters on the session role **and** the shard's live feature config. While the lookup is + in flight or has failed, **show everything** — the same deliberate fail-open the web client + takes, because the server gates regardless and a nav that flickers in on every load is worse + than a link that briefly `403`s. The gate is server-side; hiding is presentation. + - **`404` and `403` mean different things here** and neither is a generic error: + `requireFeature` `404`s a *disabled* feature (deliberately not disclosing that it exists) and + `403`s a viewer *below its audience*. Both render "not available on this shard", alongside the + existing `503` = shard offline (§7). + - **The `level` from `/features` is authoritative — do not re-derive the rung from the role.** + The server's ladder is `anonymous → logged_in → player → staff → admin`, where `player` means + *a linked game account* and staff always satisfy `player` (the same superset rule `Menu.kt` + already encodes as `isPlayer || isStaff`). + - **`char.profile.points`** → the "Loyalty & Points" block the web character sheet gained: + `CharProfileDto.points[{system, nameString, points, maxPoints, rank?}]`. Three traps, all of + them things a real shard does and a fake one does not (`v3.md` §7.5): `maxPoints == 0` means + **uncapped** and is the *common* case, so nothing may divide by it; `nameString` is usually + `null` because most systems name themselves with a cliloc, making the humanise-the-`system`-key + path the **primary** one rather than a fallback; and `rank` is absent unless the shard runs + `PointsProfileRank=true` — absent and "unranked" are different answers. + - **Cliloc-resolved names** (`v3.md` §8.6, already live on the website): `EquipmentDto` gains + `name` + `clilocName` and `TitlesDto` gains `rewardResolved`, so equipment stops rendering as a + layer or a bare id. Precedence is `name → clilocName → layer`: a player-given name outranks the + resolved type name, and the server applies the same order. A shard with no cliloc table + configured sends neither field and the sheet renders exactly as it does today. + - `ActorDto` keeps its `acct` / `webId` fields (nullable, so nothing breaks) but its KDoc stops + describing them as available: they are **locked to the admin rung**, always, and stripped from + every response below it. + - **Part 2 — the four new screens**, each hidden by its feature name in the menu: + - **Rules** — `GET /public/shard/ruleset` (`ruleset`). A `null` body means "the shard has not + published its ruleset yet", which is a different state from the feature being disabled. Every + block is optional and omitted when its system is off. **`caps.skill` / `caps.totalSkill` are in + tenths** (1000 = 100.0) and must be converted — the raw number is actively misleading, not + merely unhelpful. Live via the `world.ruleset` frame, which is on the public stream by default. + - **Leaderboards** — `GET /public/shard/points`, `/points/:system` (`leaderboards`). The same + `maxPoints`/`nameString` traps as the profile block. Live via `points.board`. + - **Market** — `GET /public/shard/market` (`q`, `minPrice`, `maxPrice`, `itemId`, `map`, `region`, + `sort`, `limit`, `offset`), `/market/meta` for the filter options + staleness, and + `/market/vendors/:serial` (`market`). Four things this screen must get right: it is the site's + first **rate-limited** public endpoint, so handle `429` the way the contact form does; the + *"prices last refreshed N minutes ago"* banner is **required, not decoration** — the shard + sweeps vendors round-robin, so a listing can legitimately be a full cycle stale and a page + implying live prices sends people to an item that sold twenty minutes ago; a `truncated` shop + must say so; and `location` is a **nested object** that an admin may gate away entirely, which + the vendor screen renders as "hidden by the shard" (a real answer) rather than as blank + coordinates — same for `ownerName` / `ownerSerial`. **The `market` SSE fan-out is off by + default** (a live firehose of vendor inventories would be the site's biggest bandwidth + consumer), so the screen is a plain paginated read and must never depend on live frames. + - **Atlas** — `GET /public/atlas/{creatures,creatures/:slug,regions,landmarks,champions,meta}` + (`atlas`). Note the path: `/public/atlas`, **not** `/public/shard` — the atlas is static shard + *content*, not live shard *state*, and unlike `/shard/*` it **is** `siteMode`-gated like + `/posts` and `/wiki`, so a site in maintenance mode withholds it independently of the sidecar. + Three traps. Two are units/naming, from `v3.md` §6.3: respawn delays are **seconds** + throughout, and `points` is a *count* on the search route while `spawners` is the *list* on + the detail route. The third is a **shape**: `places` is a list of + `{facet, label, spawners, maxAlive}` **objects**, not of place-name strings — it is the + aggregate the screen exists to show ("Shrines, Isamu-Jima, Yew"), it arrives only on the + detail route, and typing it `List` makes that whole route fail to decode while the + request itself returns `200`. + - **Verification** — the five-rung walk (`anonymous`, `logged_in`, `player`, `staff`, `admin`) + against a local website on the cutover branch, per + [`../link/v3.md`](../link/v3.md) §11 and the shard-visibility smoke harness; plus one pass with + **every feature disabled** in Admin → Shard Visibility, confirming the app *hides* each surface + instead of erroring on it. Unit tests cover the menu filter (role × feature set), the + `404`/`403`/`503` mapping, and DTO decode for each new shape. **Decode tests must feed real + captured JSON**, not DTOs built in Kotlin: the fakes under `data/api/fake/` construct objects + directly, so they can never catch a wire/type mismatch — which is how the `places` shape above + shipped past a green suite. + - **Excluded**, in the same class as M10's exclusions: the admin *configuration* panels — Shard + Visibility, Spawn Atlas and Cliloc import — alongside the hero/CMS block editor, Discord-bot + config, uo-link config and OAuth-provider setup. + ### Deferred (not a milestone) -- **`/api/mobile` facade migration + app-version floor** — briefly planned as M11 (2026-07-22), now - **deferred with no app work scheduled**. The website's router refactor is being done in place with +- **`/api/mobile` facade migration + app-version floor** — briefly planned as its own milestone + (2026-07-22), now **deferred with no app work scheduled**. The website's router refactor is being done in place with every URL byte-identical and `/api/v1` is not being retired, so the app's ~70 hardcoded `api/v1/…` endpoints, its SSE path, and its SSO URLs keep working untouched. If the mobile contract ever needs to diverge from web, the migration comes back — starting from a one-line alias mount on the server, diff --git a/link/INTEGRATION.md b/link/INTEGRATION.md index ef9105f..a9fa80d 100644 --- a/link/INTEGRATION.md +++ b/link/INTEGRATION.md @@ -31,18 +31,32 @@ Missing or wrong token → **401** `{"error":"missing or invalid auth token"}`. The wire protocol is versioned so a mismatch is caught immediately instead of failing weirdly. -- Every response carries an **`X-UOLink-Version: 2`** header. -- `GET /health` and the WebSocket `ws.hello` frame include `"protocol": 2`. -- **Optionally**, send `X-UOLink-Version: 2` on your requests. If it disagrees with the sidecar, the request is rejected **409 Conflict**: +- Every response carries an **`X-UOLink-Version: 3`** header. +- `GET /health` and the WebSocket `ws.hello` frame include `"protocol": 3`. +- **Optionally**, send `X-UOLink-Version: 3` on your requests. If it disagrees with the sidecar, the request is rejected **409 Conflict**: ```json - { "error": "protocol version mismatch", "sidecar_protocol": 2, "client_protocol": "1" } + { "error": "protocol version mismatch", "sidecar_protocol": 3, "client_protocol": "2" } ``` Pin the version you built against and compare it to the header (or `/health.protocol`) at startup. **v2 (Protocol 2.0)** added the account-provisioning surface (§6.x: `POST /accounts/create`, `DELETE /link/{account}`) and the `account.*` events. Outbound event kinds are **additive** — a v1 client that ignores unknown kinds keeps working against the live feed — but the new *endpoints* require a v2 sidecar. If you send `X-UOLink-Version: 1`, calls to the new endpoints are refused with the 409 above. +**v3 (Protocol 3.0)** adds `world.ruleset`, `points.board` and `vendor.listing` / +`vendor.listing.remove`, with the `GET /ruleset`, `/points` and `/market` reads that serve them from +the sidecar's store. Same shape as the v2 bump: the event kinds are additive, so a v2 client that +ignores unknown kinds keeps working against the live feed, but the three new endpoints require a v3 +sidecar. There is deliberately **no feature-negotiation array** — v3 implies all three kinds, so the +version number alone tells you what is available. + +**Upgrading a v2 integration.** The bump is an operator-visible hard break in one direction only: a +client still declaring `2` gets a 409 on every protected route and, on the WebSocket, a closed +connection on the `ws.hello` mismatch. So update the pinned version at the same time you deploy the +v3 sidecar. Nothing that existed in v2 changed shape, so that is the whole migration — the website +does it with a one-shot boot migration of its `uo_link_config.protocol` row ([`v3.md`](v3.md) §4.1); +a third-party client changes the constant it sends. + --- ## 3. Health @@ -54,7 +68,7 @@ GET /health (no auth) ```json { "status": "ok", // "ok" when plugin connected AND db reachable, else "degraded" - "protocol": 1, + "protocol": 3, "plugin_connected": true, // is the shard link up right now? "database": "ok", // "ok" | "error" "uptime": "3d 12h", @@ -77,7 +91,7 @@ A push-only stream of game events as they happen. You do **not** send commands o **On connect**, the first frame is: ```json -{ "kind": "ws.hello", "protocol": 1 } +{ "kind": "ws.hello", "protocol": 3 } ``` **Then** a continuous stream of event frames, each with at least `t` (epoch ms) and `kind`. Route on `kind`. @@ -315,6 +329,198 @@ The house registry — one row per house, complementing the `house.decay` *trans Render from `GET /houses` (§6) on connect, then keep live with these events. +#### Shard ruleset (Protocol 3.0) + +How the shard is actually configured, published by the shard itself. **Not a sweep** — it changes only +when an operator edits `Config/*.cfg`, so it is emitted once per shard↔sidecar connect (and on +`[bridge reload`), exactly like `server.hello`. + +| kind | fields | notes | +|------|--------|-------| +| `world.ruleset` | `rev`, `shard`, `expansion`, `connect?`, `systems`, `caps`, `housing`, `accounts`, `vetRewards`, `loot`, `vendors`, `champions?`, `treasureMaps`, `vvv?`, `store`, `schedule?` | The whole ruleset, always complete — **never a delta**, so the latest frame replaces the previous one outright. Every block except `shard`/`expansion` is optional and is **omitted when its system is off**, so absence means "not applicable here", not "unknown". | + +`rev` is the shard's FNV-1a of the body: identical `rev` means the ruleset is unchanged and this frame +is just a reconnect re-send, so a consumer can skip the write. It is deliberately **not** +`String.GetHashCode()`, which is seeded per process and would change on every shard restart. + +```json +{"kind":"world.ruleset","rev":"1a2b3c4d","shard":"UOMysticmoon","expansion":"EJ", + "systems":{"cityLoyalty":true,"vvv":true,"factions":false,"siege":false,"chat":true, + "store":true,"dailyRares":true,"honesty":true,"shadowguard":true, + "treasureMaps":true,"vetRewards":true,"testCenter":false}, + "caps":{"skill":1000,"totalSkill":7000,"stat":225,"str":125,"dex":125,"int":125, + "strMax":150,"dexMax":150,"intMax":150}, + "housing":{"accountHouseLimit":1}, + "accounts":{"perIp":3,"charSlots":7,"autoCreate":true}, + "vetRewards":{"enabled":true,"rewardIntervalDays":30}, + "loot":{"feluccaLuckBonus":1000,"feluccaBudgetBonus":100,"feluccaMaxProps":11}, + "vendors":{"restockDelayMinutes":60,"maxSell":500,"economyStockAmount":500}, + "champions":{"powerScrolls":6,"statScrolls":16,"scrollChance":0.1, + "transcendenceChance":50.0,"rankThresholds":[5,10,13]}, + "treasureMaps":{"enabled":true,"lootChance":0.01,"resetDays":30}, + "vvv":{"enabled":true,"startSilver":2000,"enhancedRules":false}, + "store":{"enabled":true,"currencyName":"Sovereigns"}, + "schedule":{"autoSaveEnabled":true,"autoSaveFrequencyMinutes":15,"autoRestartEnabled":false}, + "t":1752489280000} +``` + +**Two things consumers get wrong.** + +1. **`caps.skill` and `caps.totalSkill` are in tenths**, the way ServUO stores them: `1000` is `100.0` + skill and `7000` is `700.0` total. Rendering the raw number is actively misleading. The other caps + (`stat`, `str`, …) are plain integers. +2. **`connect` is present only if the operator set `Bridge.PublicConnectAddress`.** The shard's real + listen address (`Server.cfg`) is never published; nor are `Staff.cfg`, `Email.cfg`, `DataPath.cfg`, + `Bridge.cfg`, `Compiler.cfg`, `Reports.cfg` or `Client.cfg`. The frame is built from an explicit + allowlist in `BridgeRuleset.cs` — `Config.Entries` is never enumerated, because that would sweep in + every key on the server. + +Absent entirely if the shard runs `Bridge.RulesetEnabled=false` or an older plugin. Render from +`GET /ruleset` (§6) on connect, then keep live with this event. + +This **supersedes the `world.systems` frame** sketched in [`PROTOCOL_2.md`](PROTOCOL_2.md) §10.4 and +never implemented; the `systems` block above is what that asked for. + +#### Points / loyalty leaderboards (Protocol 3.0) + +ServUO carries ~25 separate point currencies — Queen's Loyalty, Void Pool, Casino, Clean Up Britannia, +the nine city loyalties, Blackthorn, the Doom / Khaldun / Kotl treasure systems — every one a standing +players accumulate over months, and none of them visible outside an in-game gump before 3.0. + +A diff sweep (default 300 s), **one frame per system** rather than one large frame for all of them, +matching `champ.update` / `guild.update`. A system is emitted only when its top N or its participant +count actually changes. + +| kind | fields | notes | +|------|--------|-------| +| `points.board` | `system`, `nameString`, `nameNumber`, `maxPoints`, `showOnGump`, `players`, `top[]` | One system's complete board — **never a delta**. The latest frame for a `system` replaces the previous one outright. `top[]` entries are `{rank, serial, name, points}`. | + +`system` is the shard's own `PointsType` enum name (`QueensLoyalty`, `CleanUpBritannia`, …) and is the +board's stable key. There is deliberately **no `points.remove`**: the set of systems is fixed at startup +by `PointsSystem.Configure`, so a system cannot disappear at runtime — the same argument `city.update` +makes for cities. + +```json +{"kind":"points.board","system":"QueensLoyalty", + "nameString":"Queen's Loyalty","nameNumber":1114938, + "maxPoints":15000,"showOnGump":true,"players":842, + "top":[{"rank":1,"serial":"0x1A2B","name":"Darrow","points":29500}, + {"rank":2,"serial":"0x1A2C","name":"Mireille","points":21000}], + "t":1752489280000} +``` + +**Four things consumers get wrong.** + +1. **`maxPoints` of `0` means UNCAPPED, not "zero points allowed".** ServUO's idiom for an uncapped + system is `double.MaxValue` (`DespiseCrystals`, `ShameCrystals` and `VoidPool` all use it), which + the plugin normalises to `0` rather than emitting a nonsense integer. On a real shard **most + systems are uncapped**, so a UI that renders `points / maxPoints` must special-case this or it will + divide by zero on the common path. +2. **`nameString` is usually `null`.** The shard's `Name` is a `TextDefinition`, which may carry a + literal *or* a cliloc id, and in practice most systems use the cliloc — so `nameNumber` is set and + `nameString` is `null`. Resolve clilocs consumer-side; failing that, humanising the `system` key + ("CleanUpBritannia" → "Clean Up Britannia") reads better than showing a bare number. This is the + same contract `titles.reward` already documents. +3. **`players` counts players who actually hold points**, not the size of the system's table. Ten of + the ~25 systems have `AutoAdd = true` and therefore keep a zero-point row for every character that + has ever logged in, so the raw table size would report the shard's entire character census as that + system's participants. +4. **Entries carry `serial` and `name` only — never `acct` or `webId`.** A board is the widest-audience + surface the bridge has, so the account name of every ranked player deliberately does not cross the + wire; resolve serial → site user from your own link mirror if you need it. + +Absent entirely if the shard runs `Bridge.PointsLeaderboardEnabled=false` or an older plugin. Render +from `GET /points` (§6) on connect, then keep live with this event. + +##### `char.profile` gains a `points` block + +Read-model enrichment on the existing kind — there is **no** request kind for one character's points, +the same precedent `titles` set in [`PROTOCOL_2.md`](PROTOCOL_2.md) §10.3: + +```json +"points":[{"system":"QueensLoyalty","nameString":"Queen's Loyalty","nameNumber":1114938, + "points":29500,"maxPoints":15000}] +``` + +Systems where the character has no entry, or an entry at zero, are **omitted** — otherwise every sheet +would carry ~25 zeroes. `maxPoints` follows the same `0 == uncapped` rule as the board. + +`rank` is **absent by default** and appears only when the shard runs `Bridge.PointsProfileRank=true`: +a points lookup stops at the character's own row, but a rank must count every row that beats them, in +every system, on every profile build. Derive rank from `points.board` instead for anyone in the top N. + +#### Player-vendor marketplace (Protocol 3.0) + +The shard-wide shop index: every player vendor's shop name, owner, location and priced inventory — +the same set the in-game **Vendor Search** gump reads, published so a site can offer the same search +from outside the game. + +An **amortized round-robin diff sweep**, not a snapshot RPC, and the distinction is load-bearing: +`rpc.rs::try_route` correlates a reply on the FIRST frame carrying a matching `reqId`, so a chunked +reply sharing one `reqId` would deliver chunk 1 to the HTTP caller and leak chunks 2..N onto the +broadcast feed. A whole-world snapshot could not fit in one frame inside the 10 s reply timeout +either. The per-account `vendor.snapshot` RPC (§5) is unaffected and still serves the player portal. + +Each tick inventories at most `Bridge.MarketSweepBatch` vendors (default 25) starting from a +persistent cursor, so **per-tick cost is bounded independently of world size**; full coverage takes +`ceil(vendors / batch) × MarketSweepSeconds`. A vendor is emitted only when its contents, prices, +shop name or location actually change. + +| kind | fields | notes | +|------|--------|-------| +| `vendor.listing` | `serial`, `shopName`, `ownerSerial`, `ownerName`, `location{}`, `count`, `total`, `truncated`, `items[]` | One vendor's complete shop — **never a delta**. The latest frame for a `serial` replaces the previous one outright. | +| `vendor.listing.remove` | `serial` | The shop is gone from the index: dismissed, expired, or its owner switched off the in-game Vendor Search flag. | + +```json +{"kind":"vendor.listing","serial":"0x40001234", + "shopName":"Darrow's Bargains","ownerSerial":"0x1A2B","ownerName":"Darrow", + "location":{"map":"Trammel","x":1421,"y":1699,"z":0, + "region":"Britain","house":"Darrow's Villa"}, + "count":2,"total":2,"truncated":false, + "items":[{"serial":"0x40012ABC","itemId":3922,"hue":0,"amount":1, + "price":25000,"name":null,"cliloc":1023721}, + {"serial":"0x40012ABD","itemId":7026,"hue":1157,"amount":3, + "price":500,"name":"a shard sigil","cliloc":1041243}], + "t":1752489280000} +``` + +**Six things consumers get wrong.** + +1. **`name` is `null` for nearly every item; `cliloc` is the real label.** Items carry a + `LabelNumber`, not a name. The plugin deliberately never calls `VendorSearch.GetItemName`, which + builds an `ObjectPropertyList`, serialises it and byte-parses the packet **per item** — a + multi-hundred-millisecond stall across a full pass. (It would not work anyway: every current + client ships its cliloc files compressed and ServUO's bundled `Ultima.StringList` cannot read + them, so the in-game gump has the same gap.) Resolve clilocs consumer-side; a non-null `name` is a + player-set literal and is strictly more specific, so **prefer it over the cliloc**. +2. **`location` is one nested object, and it may be absent entirely.** It is nested so that a + consumer gating vendor whereabouts gates one field rather than five that can drift apart — the + website's `market.location` rule removes the whole object. Treat a missing `location` as "not + published", not as an error. +3. **`truncated` means the shop holds more than the frame carries.** `count` is what was published, + `total` is what the shop actually holds, capped by `Bridge.MarketMaxListings` (default 250). A + commodity reseller with thousands of stacked resources is real and an uncapped frame for one is + measured in megabytes. Say "showing 250 of 3,104" rather than presenting a partial shop as + complete. +4. **`child: true` means the price buys the ENCLOSING CONTAINER.** ServUO prices a container as a + unit and everything inside inherits that price with no `VendorItem` of its own; `DoSearch` + surfaces the same flag. A UI that prints the container's price against each item inside it is + lying about the shard. +5. **Opted-out vendors are absent, and that is a privacy control.** `pv.VendorSearch` is the player's + own in-game toggle and the sweep honours it — hide your vendor in game and it is hidden here too. + The same goes for `Map.Internal` and a null backpack, matching `DoSearch`. Process + `vendor.listing.remove` promptly: it is how a player *revoking* that consent reaches you. +6. **Prices are inherently stale, by design.** The round-robin sweep means a shop can be a full cycle + behind. Any UI over this must say how old the data may be — the website derives it from the oldest + vendor row. + +Entries carry `ownerSerial`/`ownerName` and **never `acct` or `webId`**, the same rule `points.board` +follows. Absent entirely if the shard runs `Bridge.MarketEnabled=false` or an older plugin. Render +from `GET /market` (§6) on connect, then keep live with these events — though note that a live +firehose of whole vendor inventories is the largest stream the bridge produces, and a consumer that +only needs a browsable index (as the website does) is better served by the REST read plus the +periodic re-sweep. + --- ## 5. REST — read queries @@ -354,7 +560,7 @@ Full character sheet: stats, all trained skills, worn equipment with flattened i Field notes: - `skills[].base` is trained value, `value` includes item/temp bonuses, `cap` is the cap. **Do not assume `base <= cap`** — GM characters can exceed it. - `equipment[].mods` is a flattened map of every non-zero AOS attribute on the item (weapon or armor). Empty `{}` for plain items. -- Item names are usually **clilocs**, not strings: use `name` when present, otherwise resolve `cliloc` against a UO cliloc table on the site. +- Item names are usually **clilocs**, not strings: use `name` when present, otherwise resolve `cliloc` against a UO cliloc table on the site. **Do not expect the shard to resolve them for you** — on any modern client ServUO's own `Ultima.StringList` cannot read the client's compressed cliloc files, so `VendorSearch.GetItemName` returns `item.Name` and the in-game Vendor Search gump has the same gap. Building that table is a consumer-side job; the website's is described in [`website/CLILOCS.md`](../website/CLILOCS.md). - `titles` (Protocol 2.0): `selected` is the index into `reward` currently displayed (`-1` if none). `fameKarma`/`skill` are computed display titles, omitted when the character has none. `reward` entries may be a **cliloc number as a string** or a literal string — resolve numeric ones against your cliloc table, same as item names. - Errors: unknown account → **404** `{"kind":"bridge.error","reason":"unknown account"}`; bad slot → **404**/**400** similarly. @@ -658,6 +864,72 @@ GET /houses Every house's latest snapshot — owner→houses map. Served from the sidecar's projection, kept current by the `house.*` stream (§4). Ordered by name. Survives a sidecar restart. +### Shard ruleset (Protocol 3.0) + +``` +GET /ruleset +→ { "ruleset": {"kind":"world.ruleset","rev":"1a2b3c4d","shard":"UOMysticmoon", + "expansion":"EJ","systems":{...},"caps":{...},"accounts":{...}, ... } } +``` + +The shard's published ruleset (§4 for the full frame and its two gotchas). Served from the sidecar's +store, so it **answers while the shard is down** — a rules page that goes blank during a restart is +worse than one that is briefly stale. Keep it current with the `world.ruleset` stream. + +`{"ruleset": null}` means the shard has never published one — an older plugin, or +`Bridge.RulesetEnabled=false`. That is a real answer distinct from a published ruleset, and worth +rendering differently ("not published yet") rather than as an empty ruleset. + +### Points / loyalty leaderboards (Protocol 3.0) + +``` +GET /points +→ { "boards": [ {"kind":"points.board","system":"QueensLoyalty","nameString":"Queen's Loyalty", + "nameNumber":1114938,"maxPoints":15000,"showOnGump":true,"players":842, + "top":[{"rank":1,"serial":"0x1A2B","name":"Darrow","points":29500}, ...],"t":...}, ... ] } + +GET /points/{system} # e.g. /points/QueensLoyalty +→ {"kind":"points.board","system":"QueensLoyalty", ... } +``` + +Every system's latest board, or one by its `PointsType` name (§4 for the frame and its four gotchas). +Served from the sidecar's projection, kept current by the `points.board` stream, ordered by display +name. Survives a sidecar restart — which matters more here than for live state, since these are +standings built over months and blanking them during a restart reads as data loss. + +`GET /points/{system}` returns **404** for a system the shard has never published (an unknown name, or +one excluded by `Bridge.PointsSystems`). That is distinct from a published board nobody has scored in +yet, which is **200** with an empty `top[]` — and the two are worth rendering differently. + +### Player-vendor marketplace (Protocol 3.0) + +``` +GET /market?limit=200&offset=0 +→ { "vendors": [ {"kind":"vendor.listing","serial":"0x40001234", + "shopName":"Darrow's Bargains","ownerSerial":"0x1A2B","ownerName":"Darrow", + "location":{"map":"Trammel","x":1421,"y":1699,"z":0, + "region":"Britain","house":"Darrow's Villa"}, + "count":2,"total":2,"truncated":false,"items":[ ... ],"t":...}, ... ], + "total": 137, "limit": 200, "offset": 0 } +``` + +Every vendor's latest shop, exactly as `vendor.listing` published it (§4 for the frame and its six +gotchas). Served from the sidecar's projection, so it answers while the shard is down. + +**This is the only PAGED read the sidecar serves**, because it is the only board that can be a whole +world's inventory. `limit` is clamped to 1..1000 (default 200); `total` is returned so a caller knows +when to stop rather than paging until it sees a short page, which would race a concurrent sweep. +Ordering is by **serial**, not by shop name — a serial is stable while a shop name is renameable, so +a rename mid-walk cannot make a vendor skip or repeat a page. + +The route is `/market` and deliberately **not** `/vendors`: `/vendors/{account}` next door is the +per-account RPC (§5), and two routes a prefix apart meaning "this player's shops" and "every shop on +the shard" is a trap nobody wins. + +Frames are served **verbatim**, owner names and coordinates included. That is not an oversight: the +sidecar defines no audiences. Deciding who may see what is the consuming site's job — see +[`v3.md`](v3.md) §3 for how the website does it. + --- ## 7. Status codes @@ -683,7 +955,7 @@ Every house's latest snapshot — owner→houses map. Served from the sidecar's A typical character page: ```js -const H = { "Authorization": `Bearer ${TOKEN}`, "X-UOLink-Version": "2" }; +const H = { "Authorization": `Bearer ${TOKEN}`, "X-UOLink-Version": "3" }; // 1. render the roster const roster = await fetch(`${BASE}/roster/${account}`, { headers: H }).then(r => r.json()); diff --git a/link/PLAN.md b/link/PLAN.md index fff21e0..5bf9b88 100644 --- a/link/PLAN.md +++ b/link/PLAN.md @@ -284,6 +284,15 @@ Counts in `hello` are a live snapshot taken on the Core thread, not a cached val `Item.Name` is frequently `null`; the display name is `LabelNumber`, a cliloc id. **There is no `Data/Cliloc.enu` in this repo** — `BRIDGE_FINDINGS.md` §IV.4 is wrong about this. Cliloc data lives in the client install, which `DataPath` resolves to `D:\Games\Electronic Arts\Ultima Online Classic\`. Ship **both** `name` (when non-null) and `cliloc`, and resolve the number **on the website** against a cliloc map. That avoids a server-side dependency on the client directory. +**Update (3.0).** That recommendation held, and the reason it had to hold turned out to be stronger +than "avoids a dependency": **ServUO cannot resolve clilocs either.** Every current client ships its +`Cliloc.*` files compressed, and the bundled `Ultima.StringList` reads only the older plain layout — +so `VendorSearch.StringList` is null and `VendorSearch.GetItemName` returns `item.Name` on any modern +shard. The in-game Vendor Search gump has the same gap, which is why `vendor.listing` never calls it. +Pushing name resolution to the plugin was never an option. See [`v3.md`](v3.md) §8.6 and +`docs/website/CLILOCS.md` for how the site gets a table instead (the operator converts one from their +own client, once). + --- ## 8. Corrections to `BRIDGE_FINDINGS.md` @@ -315,6 +324,35 @@ Counts in `hello` are a live snapshot taken on the Core thread, not a cached val 7. **Core edit: `PlayerVendorSale`** (§6). Then the cheat-detection feed. 8. **Cheat signals.** `FastWalk`, `OnPropertyChanged` audit, vendor-sale anomaly detection in the sidecar. +**Beyond 1.0.** Phases above are the 1.0 read/event plane. Protocol 2.0's phasing (provisioning + +world-state boards) is [`PROTOCOL_2.md`](PROTOCOL_2.md) §13; Protocol 3.0's (visibility framework, +shard content and standings) is [`v3.md`](v3.md) §9, which also tracks what has landed. Shipped from +3.0 so far: **Part A** — the visibility framework — **`world.ruleset`** ([`v3.md`](v3.md) §5), +`BridgeRuleset.cs`, the first bridge stream that is neither an event subscription nor a sweep (it is +emitted once per connect, like `server.hello`, because shard config changes only when an operator +edits a file) — the **spawn atlas** ([`v3.md`](v3.md) §6), which is website-only and touches no wire +at all — and **`points.board`** ([`v3.md`](v3.md) §7), `BridgePoints.cs`, the loyalty/points +leaderboards. `BridgePoints` is the widest read the bridge performs: ten of ServUO's ~25 point systems +keep a row for every character ever created, so it selects the top N in a single bounded pass rather +than sorting, and runs on a deliberately slow 300 s interval. + +Also shipped: **`vendor.listing`** ([`v3.md`](v3.md) §8), `BridgeMarket.cs`, the shard-wide +player-vendor index. It introduces the one sweep pattern the bridge did not previously have — an +**amortized round-robin**. Every other sweep walks its whole collection per tick, which is fine for +tens of houses or a fixed set of point systems and is not fine for a world of shops whose inventories +recurse into containers. `BridgeMarket` inventories at most `MarketSweepBatch` vendors per tick from +a persistent cursor, so the per-tick cost is bounded by the batch rather than by world size, and full +coverage takes `ceil(vendors / batch) x MarketSweepSeconds`. Measured at **15.4 ms** for a cold tick +of 25 vendors x 40 listings and **0.3 ms** in steady state (the per-vendor diff), on a shard of 209k +items / 43k mobiles. It is also the first stream to honour a per-player privacy toggle: ServUO's own +`PlayerVendor.VendorSearch` flag, so a shop hidden in game is hidden on the site. + +That completes 3.0's feature work, so the last step is the version itself: `PROTOCOL_VERSION` **2 → +3** and the coordinated `edge` → `main` merge across all four repos ([`v3.md`](v3.md) §4 and §4.1). +The bump is deliberately the *only* thing that happens at that moment — v3 adds kinds and endpoints +but changes nothing that already existed in v2 — so the operator-visible break is limited to +re-pinning the version, which the website does for itself in a one-shot boot migration. + ### Config keys (`Config/Bridge.cfg`) ```ini @@ -328,6 +366,13 @@ EconomySweepSeconds=300 Read in `Configure()` via `Config.Get("Bridge.", default)`. Key scope is the filename: `Bridge.cfg` + `StatSweepSeconds` → `Bridge.StatSweepSeconds`. +The set above is the 1.0 sample, not the current one — every later phase added keys (sweep intervals +for each board, the town-crier/news caps, the admin write plane, account provisioning, and 3.0's +`RulesetEnabled` / `PublicConnectAddress` / `RulesetIncludeSchedule`, and the `Points*` and `Market*` +blocks). +**`servuo-plugins/overlay/Config/Bridge.cfg` +is the authoritative, commented list**; `BridgeConfig.cs` holds the defaults. + --- ## 11. Phase 1 acceptance diff --git a/link/PROTOCOL_2.md b/link/PROTOCOL_2.md index bc0dec7..cd9944c 100644 --- a/link/PROTOCOL_2.md +++ b/link/PROTOCOL_2.md @@ -285,6 +285,18 @@ City titles and faction/VvV merchant titles (`CityLoyaltySystem.ApplyCityTitle`, ### 10.4 Factions / Vice vs Virtue +> **Status update (Protocol 3.0, 2026-07-28).** +> +> - **The deferred question is answered.** This shard runs **Vice vs Virtue** (`VvV.cfg Enabled=True`); +> old Factions is off, and in stock ServUO that is not a coincidence — +> `Services/Factions/Core/Faction.cs` sets `Settings.Enabled = !ViceVsVirtueSystem.Enabled`, so the +> two are mutually exclusive by construction. The `vvv.standings` / `vvv.battle` streams below are +> therefore unblocked, but are **not** scoped for 3.0 (see [`v3.md`](v3.md) §2 row 5). +> - **`world.systems` is superseded by `world.ruleset`** ([`v3.md`](v3.md) §5), which shipped in 3.0. +> It was never implemented under this name. `world.ruleset` carries the same +> `systems{cityLoyalty, vvv, factions, …}` sub-object this section asked for, plus the rest of the +> shard's published ruleset, so no orphan kind is left behind. Do not implement `world.systems`. + **Which system is live is a shard decision — verify before building.** Two exist: - **Old Factions** (`Scripts/Services/Factions`): `Faction.Commander` (leader, `Faction.cs:160`), `Faction.Election`, `Faction.Members` (`List`), and faction-controlled **Towns** (`Town.cs` — each town has an owning faction, a sheriff, and finance). Config-gated and, on most modern shards, **off**. @@ -300,7 +312,10 @@ City titles and faction/VvV merchant titles (`CityLoyaltySystem.ApplyCityTitle`, {"kind":"vvv.standings","order":142000,"chaos":138500,"leaderSide":"Order"} ``` -> Start by detecting which system is enabled at boot and streaming only that one; emit a one-time `world.systems` frame (what's on: cityLoyalty, vvv, factions) so the website renders the right panels instead of guessing. +> Start by detecting which system is enabled at boot and streaming only that one. ~~emit a one-time +> `world.systems` frame (what's on: cityLoyalty, vvv, factions) so the website renders the right panels +> instead of guessing.~~ — **superseded: `world.ruleset` already carries that `systems` block** (see the +> status note at the top of this section). ## 11. Further integration points — a menu to pick from diff --git a/link/v3.md b/link/v3.md new file mode 100644 index 0000000..cd79d2a --- /dev/null +++ b/link/v3.md @@ -0,0 +1,1037 @@ +# Protocol 3.0 — Shard content, standings & the visibility framework + +**Status:** Feature-complete on `edge`; the cutover (order 6) is in review. All work lands on an `edge` branch in each repo; `edge` → `main` is the v3 cutover. +**Date:** 2026-07-28 +**Codebase:** ServUO 57.4, ``, net48 / x64, Expansion **EJ**. +**Companion to** [`PLAN.md`](PLAN.md) (1.0 read/event plane), [`PROTOCOL_2.md`](PROTOCOL_2.md) (2.0 provisioning + world-state streams), [`ADMIN_CONTROLS.md`](ADMIN_CONTROLS.md) (staff write plane), [`INTEGRATION.md`](INTEGRATION.md) (website API). + +### Progress + +Each part is marked off here as it lands on `edge`. §9 carries the same state per sequencing row. + +| Order | Part | State | Landed on `edge` | +|---|---|---|---| +| 1 | **A** — visibility framework + actor-leak fix (§3) | ✅ **Done** | website [#109](https://gitea.whitlocktech.com/RunicGateway/website/pulls/109) + [#110](https://gitea.whitlocktech.com/RunicGateway/website/pulls/110), docs [#64](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/64) + [#65](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/65) | +| 2 | **B/1** — `world.ruleset` (§5) | ✅ **Done** | servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66) | +| 3 | **C** — spawn atlas (§6) | ✅ **Done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables) + [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113) (API + pages + admin panel), docs [#67](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/67) + [#68](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/68) | +| 4 | **B/2** — `points.board` (§7) | ✅ **Done** | servuo-plugins [#4](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/4), link [#18](https://gitea.whitlocktech.com/RunicGateway/link/pulls/18), website [#114](https://gitea.whitlocktech.com/RunicGateway/website/pulls/114), docs [#69](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/69) | +| 5a | **B/3 dependency** — cliloc table (§8.6) | ✅ **Done** | website [#115](https://gitea.whitlocktech.com/RunicGateway/website/pulls/115), docs [#70](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/70) | +| 5b | **B/3** — `vendor.listing` (§8) | ✅ **Done** | servuo-plugins [#5](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/5), link [#19](https://gitea.whitlocktech.com/RunicGateway/link/pulls/19), website [#116](https://gitea.whitlocktech.com/RunicGateway/website/pulls/116), docs [#71](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/71) | +| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | 🟨 In review | the bump: link [#20](https://gitea.whitlocktech.com/RunicGateway/link/pulls/20), website [#117](https://gitea.whitlocktech.com/RunicGateway/website/pulls/117), docs [#72](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/72) — then `edge` → `main`: servuo-plugins [#6](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/6), link [#21](https://gitea.whitlocktech.com/RunicGateway/link/pulls/21), website [#118](https://gitea.whitlocktech.com/RunicGateway/website/pulls/118), docs [#73](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/73) | + +Order 5 split in two once §8.6's cliloc dependency turned out to be a client-format problem rather +than a parser (see §8.6). 5a is website-only and lands first so the marketplace ships with real item +names; 5b is the four-repo wire change. + +**The `edge` → `main` half of order 6 is held for Android parity** (decided 2026-07-30, see §10): the +app sees none of the four new features and gates shard nav on session role alone, so merging the +cutover first would ship a shard whose app client silently disagrees with the web client about what is +public. The **bump** PRs into `edge` are unaffected and merge normally. + +--- + +## 1. Why 3.0 + +A survey of the live ServUO tree against everything the bridge already surfaces end-to-end found that +**the bridge covers live *activity* well and covers shard *content and standings* almost not at all.** + +Covered by 1.0 + 2.0: presence/online, region transitions, char vitals + profile + roster, house +registry + IDOC decay, champion spawns, guild board, city governors + term history, help-page queue, +total gold supply, player-vendor sales log, deaths/murders/kills, skill gains, fame/karma, quest +completes, staff/cheat audit, account linking + creation, town crier + news. + +Not covered by anything: every leaderboard, every ruleset fact, every "where do I find X", and the +entire player economy outside a player's own vendors. + +3.0 has **three scope areas**: + +- **A — The visibility framework (§3).** Admin-configurable, per-feature and per-field audience + control over every shard-derived surface on the website. Ships first; the rest depends on it. +- **B — Three new wire streams (§5, §7, §8).** `world.ruleset`, `points.board`, + `vendor.listing`/`vendor.listing.remove`. +- **C — One website-only feature (§6).** The spawn atlas, built from static ServUO data files with no + wire involvement at all. + +--- + +## 2. Survey: the full gap list + +Recorded so the items *not* scoped for 3.0 aren't re-derived later. + +| # | Gap | Source on the shard | Value | Cost | Status | +|---|---|---|---|---|---| +| 1 | **Points/loyalty leaderboards** — 25 point currencies | `Scripts/Services/PointsSystems/PointsSystem.cs` → `static List Systems`, each `List{Player,Points}` | Very high | Low | **3.0 §7** | +| 2 | **Shard ruleset page** | `Config/*.cfg` via `Server.Config.Get` | High | Very low | **3.0 §5** | +| 3 | **Shard-wide marketplace** | `PlayerVendor.PlayerVendors` + `VendorSearch.cs` | Very high | High | **3.0 §8** | +| 4 | **Spawn atlas / bestiary** | `Spawns/*.xml` (6,455 spawners), `RevampedSpawns/*.xml` (333), `Data/Regions.xml`, `Data/Locations/*.xml`, `Config/ChampionSpawns.xml`, `Data/teleporters.csv`, `Data/HarvestLocs/*` | High | Medium | **3.0 §6** | +| 5 | VvV standings + battle status | `Services/ViceVsVirtue/{ViceVsVirtueSystem,GuildStats,VvVBattle}.cs` | High | Medium | `PROTOCOL_2.md` §10.4 deferred this pending "which PvP system does this shard run?" — **now answered: `VvV.cfg Enabled=True`, `Factions.cfg` off.** Unblocked, not scoped here | +| 6 | Skill leaderboards + shard census | `Services/Reports/Reports.cs` → `GetSkillDistribution()`, `CompileGeneralStats()`, `StaffHistory` | High | Low | Spec'd `PROTOCOL_2.md` §14 (Part B phase 6), unbuilt. Shares §7's UI — fold in after | +| 7 | Custom mounts/pets codex — ~35 across 4 tiers | `Scripts/Custom/{Companions,Legendary,Named,New Legacy}` | Medium-high | Very low | Pure wiki/CMS content, zero bridge work. The shard's most distinctive content, with zero site presence | +| 8 | Community Collections progress | `Services/CommunityCollections/CollectionsSystem.cs` | Medium | Low | Natural public "community goal" widget | +| 9 | Seasonal/holiday event calendar | `Services/Seasonal Events/SeasonalEventSystem.cs`, Krampus, Forsaken Foes | Medium | Low | "What's live now / what's next" | +| 10 | Crafting / taming / harvesting feeds | `EventSink.CraftSuccess` / `TameCreature` / `ResourceHarvestSuccess` | Medium | Low | Spec'd `PROTOCOL_2.md` §11 #3/#4/#5, unbuilt | +| 11 | Virtue progression | `EventSink.VirtueLevelChange`, `Services/Ethics/` | Medium | Low | Spec'd §11 #6, unbuilt | +| 12 | Bulk Order Deeds + reward tables | `Services/BulkOrders/`, `Data/Bulk Orders/*` | Medium | Low | Feed spec'd §11 #7; the static reward tables are a free wiki page | +| 13 | Guild wars | war state on `Guild` | Low-medium | Low | Spec'd §11 #8, unbuilt | +| 14 | Astronomy discovery log | `Services/Astronomy/AstronomySystem.cs` (104 KB save) | Low | Low | Niche completion leaderboard | +| 15 | In-game chat relay | `Services/Chat/`, `Logs/Chat/{General,Help,Trade,LFG}` | Low | Medium | Privacy-sensitive; staff-only at most | +| 16 | Shard health telemetry | Crash logs, `LayerConflict.log`, `throttle.log`, `world.save.after` counts, AutoSave/AutoRestart schedule | Low-medium | Low | `world.save.after` is already ingested but never charted — world-size-over-time is nearly free | +| 17 | Ultima Store / Sovereigns balance | `Store.cfg Enabled=True, CurrencyName=Sovereigns`; `UltimaStore.GetCurrency` | ? | Medium | Only worth it if sovereigns are actually sold | + +**Excluded permanently** — see [`ADMIN_CONTROLS.md`](ADMIN_CONTROLS.md) Tier H/N: firewall/IP-block, +kill/resurrect, jail, item/gold grants, set-access-level, arbitrary `[set`/`[add`. + +**Noticed during the survey, out of scope:** `Scripts/Custom/PerryOwnerFix.cs` hardcodes an +`EventSink.Login` hook granting `AccessLevel.Owner` to account `"ShardOnwerPerry"`. Worth reviewing +independently of this work. + +--- + +## 3. Part A — The visibility framework ✅ Done + +*Landed on `edge`: website [#109](https://gitea.whitlocktech.com/RunicGateway/website/pulls/109) (the framework) +and [#110](https://gitea.whitlocktech.com/RunicGateway/website/pulls/110) (the REST-projection gap §3.6.1 +records), docs [#64](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/64) + [#65](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/65). +Smoke-tested across all five rungs per §11.* + +### 3.1 The leak this replaces (verified 2026-07-28) + +`BridgeJson.Actor()` (`BridgeJson.cs:85-117`) writes `serial`, `name`, **`acct`**, **`webId`**, +`player`. `shardState.model.js:346 shapeGuild()` returns `r.payload` verbatim, and +`GET /api/v1/public/shard/guilds` (anonymous, `shard.controller.js:131`) serves it. **A guild +leader's game account name and website user id are readable on an anonymous public endpoint today.** +The same path exists for `shapeGovernor` → `/public/shard/governors`. `Actor` also feeds +`guild.join`, `city.update` and `region.enter`, all three in `PUBLIC_KINDS` on the anonymous SSE +stream. + +The framework below is the vehicle for the fix, and the reason it ships before anything else. + +### 3.2 Where visibility lives + +**On the website, never in the sidecar.** The sidecar's job for 3.0 is unchanged in character: accept +frames, persist them to its SQLite store, forward them verbatim over WS, and serve store-backed reads +that survive a shard outage. It defines no access parameters, no audiences, no field projection, and +advertises no capabilities. + +### 3.3 The audience ladder + +`anonymous → logged_in → player → staff → admin`, each rung implying the ones below it. + +`viewerLevel(req)` resolves: no session ⇒ `anonymous`; authenticated ⇒ `logged_in`; authenticated +with a linked shard account ⇒ `player`; moderator/admin role ⇒ `staff`/`admin`. **Staff always +satisfy the `player` rung** even without a linked game account, consistent with the existing rule +that `/player/*` is role-agnostic self-service. + +### 3.4 Two limits an admin cannot override + +1. **`acct` and `webId` are admin-only, always.** They are not in-game-visible and are not exposed as + configurable fields. +2. **A kind absent from the kind→feature map is never broadcast below `admin`.** Fail closed. This + preserves the property that today's static `PUBLIC_KINDS` allowlist is a security boundary rather + than a convenience filter. + +### 3.5 Configuration + +```sql +CREATE TABLE IF NOT EXISTS shard_feature_visibility ( + feature VARCHAR(48) NOT NULL PRIMARY KEY, + enabled TINYINT(1) NOT NULL DEFAULT 1, + audience VARCHAR(20) NOT NULL DEFAULT 'anonymous', + field_rules JSON NULL, -- {"": ""} for sensitive fields only + updated_by INT NULL, + updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4; +``` + +**Not** seeded on boot (this changed during implementation): an **absent row means "use the compiled +default"**, so the table starts empty and only ever holds rows an admin has actually touched. The +defaults live in one place — `FEATURES` in `shardVisibility.js` — instead of being duplicated into a +seeder that could drift from it, and a DB blip degrades to those same defaults rather than to +"everything is public". **All ten shard features are covered — the four new ones and the six that +already ship — and every default reproduces today's behavior, so the retrofit is a no-op until an +admin changes something.** + +| Feature | Default audience | Sensitive fields (default rung) | +|---|---|---| +| `status`, `activity`, `champs`, `guilds`, `governors` | `anonymous` | guilds/governors: `leaderAcct` / `leaderWebId` → **admin (locked)** | +| `houses` | `anonymous` | `owner` → `staff`, `price` → `staff` (matches today's IDOC-only public view) | +| `presence` | `anonymous` | `location` → `staff` (matches today's staff-only, location-gated `/online`) | +| `ruleset` (new) | `anonymous` | `connect` → `anonymous` | +| `atlas` (new) | `anonymous` | — | +| `leaderboards` (new) | `anonymous` | `characterName` → `anonymous` | +| `market` (new) | `anonymous` | `ownerName` → `anonymous`, `location` → `anonymous` | + +### 3.6 Enforcement — three points, one config + +New `website/server/src/utils/shardVisibility.js`: + +- `LADDER = ['anonymous','logged_in','player','staff','admin']`, `rank()`, `meets(viewer, required)` +- `viewerLevel(req)` (§3.3) +- `KIND_FEATURE` — every event kind → its feature; unmapped ⇒ admin-only (§3.4) +- `getConfig()` — DB-backed, cached ~5 s like `uoLinkClient`'s config cache, busted on admin `PUT` +- `requireFeature(name)` — **404 when disabled** (don't leak existence), **403 when enabled but the + viewer is below the audience** +- `projectFeature(name, payload, viewerLevel)` — strips fields whose rung the viewer doesn't meet; + `acct`/`webId` always stripped below `admin` + +Applied at: + +1. **Routes** — `requireFeature(…)` on every `/public/shard/*`, `/public/atlas/*` and the + shard-derived player routes; `projectFeature` in the controllers, replacing the ad-hoc + `shapeGuild`-returns-payload-verbatim path. +2. **SSE** — `shardBroadcast.js` moves from *"one public channel with a static `PUBLIC_KINDS` + allowlist plus one admin channel"* to **per-connection filtering**: each subscriber carries its + `viewerLevel`; each frame is mapped kind→feature, gated on `enabled && meets(...)`, then passed + through `projectFeature` before write. `PUBLIC_KINDS` becomes the seed data for `KIND_FEATURE` + rather than a hardcoded gate. **This is the largest single change in Part A and where the security + boundary now lives.** +3. **Nav** — `GET /api/v1/public/shard/features` returns only the features the calling viewer can + see, so the SPA hides nav entries rather than rendering links that 403. + +### 3.6.1 What the first implementation missed (found by the §11 smoke test, fixed) + +Part A shipped enforcement on the SSE path and on `/guilds` + `/governors`, but the **remaining public +REST reads never called into it** — so the same event was projected live and served verbatim from +history. Recorded because each miss is a shape the next phase can repeat: + +- **`/public/shard/feed` returned the stored payload as-is.** `actor.acct` / `actor.webId` were + readable *anonymously* for every logged kind (`player.death`, `mob.killed`, `skill.gain`, + `guild.join`, …) — broader than the §3.1 leak, which was limited to board holders. +- **`/public/shard/idoc` returned `ownerAcct`.** Rule 1 keyed on the exact strings `acct`/`webId`, + but `shapeHouse` flattens the actor into `ownerAcct` / `ownerName` / `ownerSerial`. The lock is now + on the field's **meaning** — a key that is or ends in `acct`/`webId`, case-insensitively — so + flattened spellings are covered and unwritten shapes fail closed. +- **The `houses` field rules were dead config.** Neither `getIdoc` nor `getHouses` projected, so the + panel offered toggles that did nothing. **Every feature's declared fields must name the keys the + read model actually emits**, not just the wire frame's. +- **`/feed` filtered on `PUBLIC_KINDS`**, a module-load constant derived from the compiled defaults, + so live audience changes never reached it. `visibleKinds(level, config)` resolves the readable set + from live config; it deliberately ignores the `stream` flag, which governs SSE fan-out only (market + history stays readable with its firehose off). +- **`shardEvents.db.list` treated an empty `kinds` array as "no filter"** and fell through to an + unfiltered `SELECT`. A fully-gated config would have dumped the whole event log, staff audit + included. An empty allowlist now serves nothing. +- **`projectValue` recursed into every object**, so a `Date` column came back as `{}`. It walks + arrays and plain objects only. The unit tests used JSON fixtures and could not have caught this — + the live read did, which is the argument for §11's smoke test over tests alone. + +**The rule this leaves behind:** *a read path that returns shard data and does not call +`projectFeature` is a bug.* Every new surface in Parts B and C — `/ruleset`, `/points`, `/market`, +`/atlas` — must project, and must gate its kind set on live config rather than on `PUBLIC_KINDS`. + +### 3.7 Admin surface + +`GET` / `PUT /api/v1/admin/shard/visibility` (admin-only). Validate feature names against the known +set and rungs against the ladder; reject any attempt to set a locked field below `admin` — including +its flattened spellings (`ownerAcct`, `leaderWebId`), see §3.6.1. Writes an +`admin.audit`-style row so visibility changes are traceable. New client panel +`routes/admin/ShardVisibility.jsx` at `/admin/shard-visibility`, linked from `ShardAdmin.jsx`. + +--- + +## 4. The version bump and the rollout + +`PROTOCOL_VERSION` **2 → 3** in `link/sidecar/src/main.rs:27`. v3 is defined as *"adds +`world.ruleset`, `points.board`, `vendor.listing` / `vendor.listing.remove`"*. + +A bump is an operator-visible hard cutover — `web.rs::gate` returns 409 on every protected route on +mismatch, `uoLinkSocket.js::handleHello` closes the WS, and the website's declared version is the +admin-set `uo_link_config.protocol` column — so it happens **exactly once**, at the end: + +- Cut an **`edge`** branch from `main` in each of `website/`, `link/`, `servuo-plugins/`, `docs/`. +- Every phase PRs into `edge`, never `main`. Feature branches are cut from `edge`. +- Part A lands first, alone. +- When all parts are built and tested, one `edge` → `main` PR per repo, merged together. **That merge + is the v3 cutover.** +- A schema migration sets `uo_link_config.protocol` (the existing row **and** the column default) + from 2 to 3, so the cutover doesn't require a manual admin edit. `UOLINK_PROTOCOL` still overrides. +- No feature-negotiation array anywhere — v3 implies all three kinds. + +### 4.1 What the bump actually touches + +The version lives in five places, and all five move together: + +| Where | Change | +|---|---| +| `link/sidecar/src/main.rs` | `PROTOCOL_VERSION` 2 → 3 (with the v3 note beside the v2 one), plus the sidecar README's worked example | +| `website/server/db/schema.sql` | `uo_link_config.protocol` column default 1 → 3, plus the boot migration below | +| `website/server/src/model/uoLinkConfig/uoLinkConfig.model.js` | `DEFAULT_PROTOCOL` — what a site with nothing saved yet declares | +| `website/server/src/utils/uoLinkClient.js` + `uoLinkSocket.js` | the `config.protocol || …` fallbacks, so an unset value can never quietly send `1` and 409 with a confusing message | +| `website/client/.../ShardAdmin.jsx`, `website/.env.example` | the admin form's initial value and the documented env default | + +**The migration has to be one-shot, and that is the only subtle part.** `schema.sql` is re-run on +*every* boot (`utils/db.js::ensureSchema`), and every other statement in its migration block is an +idempotent `ADD COLUMN IF NOT EXISTS` / `MODIFY`. A bare `UPDATE uo_link_config SET protocol = 3` +would not be idempotent in the sense that matters: `protocol` is **admin-editable**, so an operator +who deliberately pins an older sidecar in Admin → Shard would silently be un-pinned on the next +restart. It is therefore gated on a marker row in `settings`: + +```sql +ALTER TABLE uo_link_config MODIFY COLUMN protocol INT NOT NULL DEFAULT 3; +UPDATE uo_link_config SET protocol = 3 + WHERE id = 1 AND protocol < 3 + AND NOT EXISTS (SELECT 1 FROM settings WHERE `key` = 'uo_link_protocol_3_migrated'); +INSERT IGNORE INTO settings (`key`, value) VALUES ('uo_link_protocol_3_migrated', '1'); +``` + +The marker is written *after* the `UPDATE`, so the first boot on the new build migrates and every +later boot is a no-op. A fresh install has no `uo_link_config` row to update and simply gets the +marker plus the new column default. `protocol < 3` rather than `= 2` so an install that never left +the old default of `1` is carried across too — it could not have been talking to a v2 sidecar +anyway. + +--- + +## 5. Part B/1 — `world.ruleset` ✅ Done + +*Landed on `edge`: servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66). Implementation notes worth keeping:* + +- ***`shadowguard` is derived, not configured.*** `Shadowguard.cfg` carries only `ReadyDuration` and + `RandomizeInstances` — there is no `Enabled` key — so the systems block reports `Core.TOL` + (the expansion gate) instead. Same shape for `factions`: `Factions.cfg` has no `Enabled` either, and + `Services/Factions/Core/Faction.cs` sets `Settings.Enabled = !ViceVsVirtueSystem.Enabled`, so the + frame reads that static rather than inventing a key. **Where a system's on/off state is derived, read + the system's own static; only read `Config.Get` where the .cfg key IS the truth.** +- **`caps.skill` / `caps.totalSkill` are in tenths** (1000 = 100.0), the way ServUO stores them. + Documented in `INTEGRATION.md` and converted in the client, because the raw number is actively + misleading rather than merely unhelpful. +- **`Config.Get` re-parses when the cached type differs.** `InternalGet` caches the parsed value on + the entry and re-parses if `entry.Object is T` fails, so reading `PlayerCaps.SkillCap` as an `int` + where ServUO reads it as a `double` is correct (both parse) — it just re-parses. Harmless, but worth + knowing before assuming a shared cache. +- **The plugin CAN be compile-verified**, contrary to "no standalone build": point Roslyn + (`dotnet sdk/*/Roslyn/bincore/csc.dll`, `/langversion:7.3`, net48 reference assemblies) at the whole + ServUO `Scripts` tree with `overlay/Scripts/Custom/Bridge/*.cs` substituted for the deployed copy, + excluding `Scripts/obj` and `Scripts/bin`. 6,205 files, ~40 s, and it catches every signature error + a boot would. Worth doing before every plugin PR. + +`PROTOCOL_2.md` §10.4 sketches a `world.systems` capability frame that was never implemented +(`grep` returns nothing across all four repos). **`world.ruleset` subsumes it**, carrying a `systems` +sub-object with the `cityLoyalty` / `vvv` / `factions` booleans §10.4 asked for. §10.4 is marked +superseded; no orphan kind is left behind. + +### 5.1 Plugin + +NEW `servuo-plugins/overlay/Scripts/Custom/Bridge/BridgeRuleset.cs`, modelled on +`BridgeBoot.EmitHello` — **not** a sweep. Subscribes `BridgeLink.Connected_Core += Emit` so a sidecar +that comes up second still learns the ruleset. + +Built from an **explicit allowlist** of `Server.Config.Get` calls. **Never enumerate +`Config.Entries`** (`Server/Config.cs:162`) — it would sweep in secrets. An FNV-1a `rev` over the body +makes an unchanged reconnect a site-side no-op (`String.GetHashCode()` is not stable across runs and +must not be used). + +`BridgeConfig.cs` + `overlay/Config/Bridge.cfg`: `RulesetEnabled=true`, `PublicConnectAddress=""`, +`RulesetIncludeSchedule=true`. `BridgeBoot.cs`: `reload` → re-emit, `status` → rev/bytes. Not wired +to `sweepnow`; it isn't a sweep. + +### 5.2 Payload + +Every block optional, omitted when its system is off: + +`shard`, `expansion`, `connect` (only from `PublicConnectAddress`), +`systems{cityLoyalty,vvv,factions,siege,chat,store,dailyRares,honesty,shadowguard,treasureMaps,vetRewards,testCenter}`, +`caps{skill:1000,totalSkill:7000,stat:225,str/dex/int:125,strMax/dexMax/intMax:150}`, +`housing{accountHouseLimit:1}`, `accounts{perIp:3,charSlots:7,autoCreate}`, +`vetRewards{enabled,rewardIntervalDays:30}`, +`loot{feluccaLuckBonus:1000,feluccaBudgetBonus:100,feluccaMaxProps:11}`, +`vendors{restockDelayMinutes,maxSell,economyStockAmount}`, +`champions{powerScrolls:6,statScrolls:16,scrollChance,transcendenceChance,rankThresholds}`, +`treasureMaps`, `vvv{enabled,startSilver:2000,enhancedRules}`, `store{enabled,currencyName}`, +`schedule{autoSaveFrequencyMinutes,autoRestart*}`. + +**Excluded by name — in a code comment and here:** `Server.cfg` (Address/Listen/Port; only +`PublicConnectAddress` is published), `Staff.cfg`, `Email.cfg`, `DataPath.cfg`, `Bridge.cfg`, +`Compiler.cfg`, `Reports.cfg`, `Client.cfg`. + +### 5.3 Sidecar and website + +Sidecar — `store.rs`: singleton `ruleset(id CHECK(id=1), rev, json, updated_t)` + upsert/get; +`main.rs`: new arm in the board-projection match; `web.rs`: `GET /ruleset` served from the store, so +it answers during a shard outage (`PROTOCOL_2.md` §12.2). + +Website — `uoLinkClient.getRuleset()`; `uoLinkSocket.backfill()` (object-shaped, so it cannot use the +array-only `snapshot()` helper — but it **must still go through `shardIngest.ingest()`**, as +`ingestEach` does, rather than calling `shardState.setRuleset` directly: the two arrival orders have +to produce the same stored frame, and a direct call quietly made backfill a second writer that +skipped the normalization below); `shardIngest.js` → +`shardState.setRuleset`, **not** in `LOGGED_KINDS` (it re-arrives every reconnect and `server.hello` +already marks those); `KIND_FEATURE['world.ruleset'] = 'ruleset'`; `shard_ruleset` singleton table +(`rev`, `expansion`, `payload JSON`, `t`); `GET /public/shard/ruleset` behind +`requireFeature('ruleset')`, returning `null` ⇒ "not published yet". + +Client — NEW `routes/public/Rules.jsx` at `/site/rules`, alongside +`/site/champs|guilds|governors|houses`; live via `useShardFeed({ filter: new Set(['world.ruleset']) })`. + +**The `shard` field falls back to the instance's own name.** ServUO ships `Server.cfg` with +`Name=My Shard`, so an operator who never edited it publishes that verbatim — which is the shard +saying *unnamed*, not naming anything, and the rules page then reads "My Shard" under a header +carrying the real one. `shardIngest` substitutes `settings.getInstanceName()` (the admin-editable +site title, else `BRAND_NAME` — the same resolution `getPublic().brand.name` uses, so one install +never shows two names) when `shard` is absent, blank, or exactly the stock default, matched +case-insensitively and trim-tolerantly but only as a **whole** value: a shard genuinely called +*"My Shard Reborn"* has named itself and keeps it. Applied at **ingest**, not on read, because the +ruleset is also broadcast live — the same object goes to the SSE fan-out, so a read-time +substitution would be undone by the next reconnect's frame. + +### 5.4 Risk + +Perf is nil (~3 KB per connect). The only real risk is publishing a secret, mitigated by the explicit +allowlist, the no-`Config.Entries` rule, the named exclusion list, and a manual eyeball of the emitted +frame during verification. + +--- + +## 6. Part C — Spawn atlas / bestiary (website-only) + +**No plugin, no sidecar, no `Bridge.cfg` knob, no new kinds.** Not part of the v3 wire change. + +> **Status:** complete on `edge` — website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) +> (parsers, import CLI, tables) and [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113) +> (the six public routes, the five admin ones, `/site/atlas` + `/site/atlas/:slug`, and the +> Admin → Spawn Atlas panel). +> Part C ships as **two** website PRs, not one: the parsing half is where the correctness risk +> lives, and burying it under routes and React would have meant reviewing it in a 10k-line diff. +> Full operator documentation: [`docs/website/SPAWN_ATLAS.md`](../website/SPAWN_ATLAS.md). +> +> **§6 below is the original design and is partly superseded.** §6.1 records two decisions that were +> rejected in review and replaced (the committed artifact, and the fixed facet list); §6.2 records +> the corrections the real ServUO data forced. Read both before trusting §6. + +**Decision (revised at implementation time): the shard's ServUO tree is the single source of truth, +re-derived on every server boot.** The original plan here was a committed generated artifact plus an +idempotent import. That was rejected in review for two reasons, recorded in §6.1: a snapshot in the +repo goes stale as a shard's maps change, and the design leaned on a fixed facet list that no shard +is obliged to keep. Still not a browser-served blob; still parsed server-side only. + +New in `website/server/`: + +- `src/utils/spawnAtlasParse.js` — **pure functions, no fs**, so they are unit-testable in CI without + a ServUO tree: `parseObjects2()`, `parsePoints()`, `parseRegions()`, `parseLocations()`, + `resolveRegion()`. +- ~~`scripts/buildSpawnAtlas.js` and a committed `db/data/spawnAtlas.*.json` artifact~~ — dropped, + see §6.1 R1. Replaced by `src/utils/spawnAtlasSource.js` (the only thing that reads a ServUO tree, + shared by the boot path and the CLI) and a `scripts/importSpawnAtlas.js` that is a thin CLI over + the model. `package.json` gains `atlas:import` only. +- `src/model/shardAtlas/{shardAtlas.db.js,shardAtlas.model.js}` following the `shardState` split. +- `src/router/v1/public/atlas.{router,controller}.js`; `test/spawnAtlas.parse.test.js`. + +Two parsing notes that matter: + +- `` is `Type:MX=n:SB=…` segments joined by `:OBJ=` — verified against `trammel.xml`, where + a single point carries six types. Split on `:OBJ=`; the token before the first `:` is the type. +- **The high-value transform:** point-in-rect each spawn against the facet's `Regions.xml` rects + (highest `priority` wins), falling back to the nearest `Data/Locations` landmark, else + `"Wilderness"`. This is what turns *"lizardman at 5411,1234"* into ***"Despise, Felucca"*** and is + the entire reason the page is worth building. `Regions.xml` is genuinely nested and needs a ~120-line + recursive tokenizer **or** one devDependency (`fast-xml-parser`) — the server has zero XML deps + today, so that is an explicit call to make at implementation time. The flat `` files need + only regex/streaming; **do not** put 10.5 MB through a DOM parser. + +Tables: `shard_spawn_creatures` (slug PK, name, total, facets JSON), `shard_spawn_points` (slug, +facet, x, y, region, landmark, max_count, tod_*), `shard_regions`, `shard_landmarks`, +`shard_champion_spawns`, `shard_atlas_meta`. Plain `INDEX` on name, **not `FULLTEXT`** — ~1,500 +creature rows makes a `LIKE` scan free, and FULLTEXT brings min-token-length trouble for names like +"orc". No FKs, consistent with every existing `shard_*` table. + +Routes at `/api/v1/public/atlas`, **not** under `/shard` — the atlas is static shard *content*, not +live shard *state*; it must not look sidecar-dependent, and unlike `/shard/*` it *should* be +`siteMode`-gated like `/posts` and `/wiki`. `GET /creatures?q=&facet=`, `/creatures/:slug`, +`/regions`, `/landmarks`, `/champions`, `/meta`, all behind `requireFeature('atlas')`. Admin: +`GET /admin/shard/atlas/status` (artifact-vs-DB drift) and `POST /admin/shard/atlas/import`. **Build +stays CLI-only.** + +Client: `routes/public/Atlas.jsx` (`/site/atlas`) and `AtlasCreature.jsx` (`/site/atlas/:slug`). + +**Payload risk** — *superseded by §6.1 R1; nothing is committed.* The field selection it describes +still applies at parse time: every `` field the site cannot use (`UniqueId`, all +trigger/refractory/proximity/sequential fields, sound ids) is dropped, keeping +Name/Map/X/Y/W/H/Range/MaxCount/MinDelay/MaxDelay/TOD*/types. Parsed data never reaches the browser; +the browser sees only paginated API responses. + +**Operator re-run story** — *revised by §6.1 R1.* Spawns changed → restart, or +`npm run atlas:import` / `POST /admin/shard/atlas/import` to apply without one. `shard_atlas_meta` +holds a sha256 per source file, so the server can tell on boot whether anything changed, and +`GET /admin/shard/atlas/status` reports drift. If the change would remove a facet it is staged for +approval rather than applied (§6.1 R3). Full detail in `docs/website/SPAWN_ATLAS.md`. + +### 6.1 What implementation changed + +Two design decisions in §6 were rejected in review and replaced; the rest are corrections the real +ServUO data forced. Kept as a diff rather than edited in place, because each is a trap the next +person would otherwise re-enter. + +**R1. The committed artifact is gone — the tree is re-parsed on every boot.** §6 proposed building a +generated artifact, committing it, and importing it. Two problems. A shard's maps change over its +life, so a snapshot in the repo silently drifts from the world players actually see; and the build/ +import split existed only to work around the website container not having a tree, which is a +deployment question (mount it) rather than a reason to freeze data. The server now hashes the source +files on boot and re-derives the atlas when they differ. `scripts/buildSpawnAtlas.js`, the 1.41 MB +artifact, and the whole encode/decode seam it needed are deleted. + +**R2. Nothing may name a facet.** The first implementation carried a lookup table of the six stock +UO facets to reconcile the spelling drift between sources. A shard may add facets, replace them +outright, or rename them when its maps are updated, and a built-in list mishandles all three +silently. Reconciliation is now by *matching* against the facet set discovered from the shard's own +spawn and region data — exact key, then prefix in either direction — with an unmatched name keeping +its own rather than being forced into a wrong bucket. + +**R3. Two contracts on the boot path.** It never blocks startup: no path, an unreadable mount, a +malformed file or a database error is caught and logged, and the site comes up serving whatever +atlas it had. And a refresh that would REMOVE a facet is never applied automatically — facet loss +is indistinguishable at boot from a half-copied or mid-update tree, so it is staged in +`shard_atlas_pending` for an admin to approve or reject. Only the decision is stored (source hashes ++ the facet diff, a few KB); approving re-parses, so what lands matches the tree at approval time. +A rejection is remembered against those hashes so it does not re-prompt every restart. + +### 6.2 What the build against real data changed + +Six corrections to the design above, from running it against stock ServUO 57.4. Kept as a diff +rather than edited in place, because each one is a trap the next person would otherwise re-enter. + +**1. Six facets, not thirteen.** The design said `spawnAtlas..json ×13`, assuming one facet +per spawn file. There are 13 files but only **6** facets — `Eodon.xml`, `GravewaterLake.xml`, +`TreasuresOfKotl.xml` and the other named-area files carry TerMur/Trammel points. The facet comes +from each record's own ``, never the file name, and the artifact shards 6 ways. + +**2. The XML dependency call: hand-rolled, zero deps.** §6 left `fast-xml-parser` vs a ~120-line +tokenizer open. Resolved as the tokenizer — a deliberate *subset* parser covering only what these +files use. The server keeps zero XML dependencies at any tier. + +**3. Facet names disagree between sources — a silent failure.** `Data/Locations/*.xml` spells them +`Ter Mur` and `Tokuno Islands`; `` and `` say `TerMur` and `Tokuno`. Unreconciled, +the landmark bucket is keyed differently from the points looking it up, so the fallback never fires +and **every unregioned spawn in Ter Mur and Tokuno reads "Wilderness"** — a plausible-looking atlas +that is quietly wrong for two facets. All facet names now pass through `normalizeFacet()`. + +**4. Spawn type tokens carry XmlSpawner directives.** `` types are not always bare class +names: `Fairy,{RND,4,8}`, `alchemist/z/-50`, `Agralem/Name/Agralem`, `greatape,true`. Taken literally +they invent creatures that do not exist *and* split real ones in two, since `Fairy` and +`Fairy,{RND,4,8}` slug apart. 71 of 845 entries were affected; stripping at the first `/` or `,` +leaves **800** real creatures. (The design's "~1,500 creature rows" estimate was high; 800 only +reinforces the plain-`INDEX`-not-`FULLTEXT` call.) + +**5. The artifact would have been 1.41 MB, not "well under 1 MB" — and is now moot.** Dropping the +unused `` fields as the design directed still left 4.40 MB; three further encodings brought +it to 1.41 MB, and getting under 1 MB would have meant dropping the spawner `name`. The size budget +in §6 was simply optimistic for 6,455 points. Superseded by §6.1 R1: there is no artifact, so there +is no payload to budget and no encode/decode seam to keep in sync. + +**6. `DELETE`, not `TRUNCATE`.** The design said "TRUNCATE + batched INSERT in one transaction", +which does not hold: `TRUNCATE` is DDL in MariaDB and implicitly commits, so a mid-import failure +would leave the atlas half-loaded. `DELETE` is transactional, and at ~7k rows the cost is +irrelevant. Point ids are also assigned explicitly rather than by `AUTO_INCREMENT`, because the +join rows need them and `conn.batch()` reports no usable `insertId`. + +**Measured result:** 6,455 points, 800 creatures, 23,927 point/type rows, 387 regions, 558 +landmarks, 25 champion altars. The placement transform resolves **83.2%** of points (3,689 by +region, 1,690 by landmark, 1,086 Wilderness). + +**One thing the design got exactly right:** the point-in-rect transform really is the reason to +build this. "Where does a lizardman spawn?" answers *Shrines, Isamu-Jima, Yew* across three facets. + +### 6.3 What the API/client half added + +The second website PR built the six public routes, the five admin ones, `/site/atlas` + +`/site/atlas/:slug`, and the Admin → Spawn Atlas panel. Three things it changed or established: + +**1. Respawn delays were being read in the wrong unit — sometimes.** XmlSpawner writes +`MinDelay`/`MaxDelay` in minutes and switches to seconds only when a delay does not divide into +whole minutes, flagging that per record with `DelayInSec` +(`XmlSpawner2.cs:7462-7480`, read back at `:6345-6358`). So a `5` means five *minutes* on one +spawner and five *seconds* on the next, both plausible, and the pipeline stored the raw number. +170 of 6,455 stock spawners are second-flagged — few enough to look like noise on a page and be +believed. The parser now normalises to **seconds**, and the API and UI carry seconds throughout. +*This is the class of bug §6.2 is a list of: the atlas still builds, it is just quietly wrong.* + +**2. The hash gate needed a parser version, and this generalises.** Fixing the parse exposed that +"has the tree changed?" is the wrong question on its own — an install whose maps never change would +have kept serving the old readings forever, because the only thing compared was the tree. +`spawnAtlasSource.PARSER_VERSION` is stored in `shard_atlas_meta` beside the source hashes, and a +mismatch counts as drift. Any future parse correction lands on the next boot without an operator +having to know it happened. **Bump it whenever the parser derives different data from identical +files.** + +**3. `points` is a count; `spawners` is the list.** The first cut of the detail route spread the +creature row and then set `points` to the array of spawn points — the same key meaning a number on +the search route and an array on the detail route. Renamed before it shipped, and worth recording +because the two names are one letter apart in meaning and it reads as correct. + +**On projection.** The `atlas` feature declares no sensitive fields, so `projectFeature` is a no-op +on every one of these routes today. Every handler calls it anyway, per §3.6.1's rule — the point of +the rule is that the *first* field that needs gating is covered by construction rather than by a +retrofit nobody remembers to do. + +--- + +## 7. Part B/2 — `points.board` ✅ Done + +*Landed on `edge`: servuo-plugins [#4](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/4), +link [#18](https://gitea.whitlocktech.com/RunicGateway/link/pulls/18), website [#114](https://gitea.whitlocktech.com/RunicGateway/website/pulls/114), +docs [#69](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/69). Verified against the real ServUO tree +per §11 — see §7.5 for what that run changed.* + +Two deliverables: a diff sweep for the boards, and a `points` block folded into `char.profile` — +the `PROTOCOL_2.md` §10.3 `titles` precedent (read-model enrichment, no new request kind). + +### 7.1 Plugin + +NEW `BridgePoints.cs`, copying the `BridgeHousing.cs` diff-sweep shape (`Initialize` → +`ServerStarted`, `Connected_Core += OnConnected` clearing `_last` + `Rearm()`, `SweepOnce()`, +`Status()`, skip when `!BridgeLink.Connected`, try/catch throughout). + +Which systems: default to `PointsSystem.Systems` filtered to `ShowOnLoyaltyGump == true` — reuse the +shard's own "this is player-facing" signal rather than inventing one. `Bridge.cfg PointsSystems=` +overrides. Null-guard `PointsSystem.Systems`; it is a mutable static populated by 25 separate +subsystem constructors. + +**The perf trap.** `PlayerTable` is a plain `List`, and `QueensLoyalty` has `AutoAdd`, so +it can hold an entry for every `PlayerMobile` that ever existed. A naive +`.OrderByDescending().Take(N)` across 25 systems is 25 full sorts — at 20,000 historical characters, +~7.5 M comparisons, tens of ms on the Core thread. `BRIDGE_PLUGIN_PLAN.md` §1 found that nothing +except bulk profile generation comes close to a frame budget; this would be the second thing that +does. + +**Mitigation — single-pass bounded selection** into a fixed N-element sorted array (N=10): O(n·N) with +tiny constants and one allocation. Skip `Player == null || Deleted` and `Points <= 0`. ~500 k cheap +iterations at a 300 s interval. + +Diff signature per system: `concat(serial + ":" + (long)points)` over the top N, plus the entry count. +**No `points.remove`** — the system set is fixed, the same argument `city.update` already uses. + +### 7.2 Payload — one frame per system + +25 × ~600 B rather than one 12 KB frame, matching `champ.update` / `guild.update`: + +```jsonc +{"t":…,"kind":"points.board","system":"QueensLoyalty", + "nameString":"Queen's Loyalty","nameNumber":1114938, + "maxPoints":30000,"showOnGump":true,"players":842, + "top":[{"rank":1,"serial":"0x1A2B","name":"Darrow","points":29500}, …]} +``` + +`nameString` **and** `nameNumber` are both emitted (a `TextDefinition` may be a cliloc), resolved +website-side — the contract `titles.reward` already documents at `BridgeProfile.cs:107-110`. + +**Entries are written inline as `{serial, name}` — never via `BridgeJson.Actor`.** Deliberate even +though the website can now reveal fields by rung: `acct`/`webId` are not needed here, because the +website resolves serial→user from its own `shard_account_links` mirror for staff views. Keep the wire +minimal. + +### 7.3 `char.profile` enrichment + +`BridgeProfile.cs` gains `WritePoints(sb, m)` alongside `WriteTitles`: +`"points":[{system,nameString,points,maxPoints}]`, omitting systems with no entry or 0 points. + +**Deliberately no `rank`** — computing it means scanning each system's `PlayerTable` once per profile +(25 × n), which would dominate the measured 0.069 ms/profile budget. The website derives rank from +the board when the character appears in the top N. Gate behind `PointsProfileRank=false` if it is +ever wanted. + +### 7.4 Config, sidecar, website + +`Bridge.cfg`: `PointsSweepSeconds=300`, `PointsLeaderboardEnabled=true`, `PointsTopN=10`, +`PointsSystems=` (blank ⇒ auto), `PointsProfileEnabled=true`, `PointsProfileRank=false`. +`BridgeBoot.cs`: `Rearm()` in `reload`, `SweepOnce()` in `sweepnow`, `Status()` in both. + +Sidecar — `points_boards(system PK, name, json, updated_t)`; `main.rs` arm keyed on `system`; +`GET /points` and `GET /points/:system`. + +Website — `shard_points_boards(system PK, name, name_cliloc, max_points, players, show_on_gump, +payload JSON, t)`. **The top-N list stays in `payload`** — a fixed-size list read whole, exactly like +`shard_governors.candidates`. Do not normalize into a `shard_points_entries` table until a +per-character reverse lookup is actually needed. `shardIngest.js` → `upsertPointsBoard`, **not** in +`LOGGED_KINDS` (board state, like `guild.update`). `KIND_FEATURE['points.board'] = 'leaderboards'`, +with `characterName` as its per-field rule. `GET /public/shard/points` and `/points/:system` behind +`requireFeature('leaderboards')`; validate `system` ≤ 48 chars. + +**No new player route** — per-character points ride inside `char.profile`, already served by +`GET /player/shard/char/:serial` with its `shardLinks.ownsAccount` check. + +Client — NEW `routes/public/Leaderboards.jsx` at `/site/leaderboards`; a "Loyalty & Points" section +added to `components/CharacterSheet.jsx`, one edit serving both `PlayerCharacter.jsx` and +`AdminCharacter.jsx`. + +**An unscored board still renders a row.** Most systems on a young shard have `top: []`, and a page +of blank cards reads as broken rather than as new — so a board with no entries shows a single +placeholder bearing the **instance's own name** with an em dash where a score goes, above the +existing "nobody has earned points here yet" line. It is deliberately **not** shaped like an entry — +no rank, no medal, no bar, muted — because a placeholder that looked like a real standing would be a +fabricated one; the first real entry replaces it outright. Purely presentational: the API keeps +sending an empty `top`, so no consumer ever receives an invented row. Web and app render it the same +way (`Leaderboards.jsx`, `LeaderboardsScreen.kt`). + +### 7.5 What the run against a real shard changed + +The plan above was written from reading `PointsSystem.cs`. Booting the actual shard (ServUO 57.4, a +43,011-mobile world) and letting one sweep run corrected four things — all of them invisible to a +fake-shard test, because a fake shard emits whatever the spec says it should. + +1. **`maxPoints` overflowed to `long.MinValue`.** `MaxPoints` is a `double`, and ServUO's idiom for an + uncapped system is `double.MaxValue` — which `DespiseCrystals`, `ShameCrystals` and `VoidPool` all + use. `(long)double.MaxValue` in C# is an **unchecked** conversion: it does not throw, it yields + `long.MinValue`, and the first real sweep published + `"maxPoints": -9223372036854775808` for three of the five live boards. Fixed with `Cap()` / + `Score()` converters that normalise anything unrepresentable to `0`, which is now the wire's + documented **"uncapped"** value. Worth stating plainly because it inverts the obvious reading: + **on a real shard, `maxPoints: 0` is the common case, not an edge case**, so any UI dividing by it + must special-case it. +2. **`nameString` is usually `null`.** Most systems define their `Name` as a cliloc rather than a + literal: four of the five boards on the live shard came back `nameString: null` with only + `nameNumber` set. The humanise-the-`system`-key fallback is therefore the *primary* display path, + not a defensive nicety, and both the leaderboards page and the character sheet lead with it. +3. **`GetEntry`/`GetPoints` cannot be used in the read model.** `GetEntry(from, create: false)` still + calls `AddEntry` when the system has `AutoAdd` (`PointsSystem.cs:207`) — it **mutates the world**. + Ten of the ~25 systems have `AutoAdd = true`, so a profile built with the obvious accessor would + have appended up to ten rows to the points save file every time anyone viewed a character sheet. + `BridgeProfile.WritePoints` hand-rolls a read-only scan instead, and says so loudly. +4. **`players` had to be redefined.** §7.2 called for "the entry count", but those same ten `AutoAdd` + systems hold a zero-point row per character ever created — so the raw count reports the shard's + whole census as one system's participants. It is now the number of players actually holding points, + which is both the honest number and a strictly better diff signal (it moves when someone scores, + not when someone logs in for the first time). + +One deviation from the plan as written, for the same class of reason: §7.4 named the per-field +visibility rule `characterName`, but `projectValue` matches on the **literal JSON key**, and the wire +key is `name`. A rule under the descriptive name would have been silently inert — an admin tightening +character names would have got no enforcement and no error, exactly the failure §3.6.1 records for the +flattened `ownerAcct`. `FEATURES.leaderboards.fields` therefore keys on `name`, with a test that fails +if it is renamed back. + +--- + +## 8. Part B/3 — `vendor.listing` 🟨 In review + +### 8.1 It cannot be an RPC, and this is load-bearing + +`rpc.rs::try_route` correlates on the **first** frame carrying a matching `reqId` and resolves a +single `oneshot`. A chunked reply sharing one `reqId` would deliver chunk 1 to the HTTP caller and +**leak chunks 2..N onto the broadcast feed**. `REPLY_TIMEOUT` is 10 s (the client waits 12 s), so a +whole-world snapshot could not fit regardless. + +⇒ **a per-vendor diff sweep on the broadcast stream**, like `champ.update` / `house.update`. The +existing per-account `vendor.snapshot` RPC is untouched; the player portal keeps using it. + +Kinds: `vendor.listing` (one frame per vendor, authoritative for that vendor) and +`vendor.listing.remove`. Payload: `serial, shopName, owner:{serial,name}, map, x, y, region, house, +count, truncated, items:[{serial,itemId,hue,amount,price,name,cliloc,child}]`. + +### 8.2 Two perf traps + +Measured baseline (`BRIDGE_PLUGIN_PLAN.md` §1): 30 vendors / 1,200 listings = 0.343 ms via +`pack.Items` + `GetVendorItem`; extrapolated to 500 vendors / 40,000 listings ≈ 12 ms per full pass. +Except: + +1. **`VendorSearch.GetItemName(Item)` is a packet builder, not a field read.** It constructs an + `ObjectPropertyList`, calls `GetProperties`, serialises, then byte-parses the packet + (`VendorSearch.cs:681-789`) — per item. Across 40,000 items in one tick that is a + multi-hundred-millisecond stall. **Mandatory: never call it in the sweep.** Emit `itemId`, `hue`, + `amount`, `price`, `item.Name` (the plain field, null for most) and `item.LabelNumber`, resolving + display names website-side — exactly what `char.profile.equipment` already does + (`BridgeProfile.cs:173`). +2. **`VendorSearch.GetItems(PlayerVendor)` is private** (`:791`). The reusable public API is + `GetItems(Container, List)` (`:807`), which recurses into sub-containers, so real item counts + run above the top-level `pack.Items` the 0.343 ms measurement used. Budget accordingly. + +### 8.3 Mitigations + +- **Amortized round-robin sweep** — `MarketSweepSeconds=60`, at most `MarketSweepBatch=25` vendors per + tick, with a persistent cursor over `PlayerVendor.PlayerVendors`. Full coverage in + `ceil(vendors/25) × 60 s`, with **per-tick cost bounded independent of world size**. This is the one + genuinely new pattern versus the existing sweeps and should be flagged in review. +- **Per-vendor signature diff** (`count | Σ(serial ^ price) | x | y | shopName`), as `BridgeHousing` + does — most vendors are static, so steady-state emission is near zero. +- **`MarketMaxListings=250`**, then `"truncated":true`. `BridgeJson.Parse` caps *inbound* at 1 MB; + outbound is uncapped and `shard.rs::read_line` will allocate whatever arrives. +- On `Connected_Core`, clear `_last` **and reset the cursor**; the re-emit is self-throttled by the + round-robin window. + +### 8.4 Player opt-out and privacy + +**Honour `pv.VendorSearch`** — ServUO's own per-vendor opt-out, which `DoSearch` filters on (`:62`). +Skip opted-out vendors entirely; the seen-set removal then drops them from the board, so **a player +who hid their vendor in game is hidden on the website too.** Also skip `Map == null || Map.Internal` +and `Backpack == null`, matching `DoSearch`. + +A vendor's shop name, owner character name and location are **already globally visible in-game** — the +stock Vendor Search gump surfaces exactly this set to any player — which is why they default to +`anonymous`. They remain per-field configurable (`ownerName`, `location`) so an admin can tighten +them. Account name and website user id never go on the wire. + +### 8.5 Sidecar and website + +Sidecar — one table `vendors(serial PK, shop_name, owner_name, map, x, y, region, count, json, +updated_t)` storing the whole-vendor blob. **No `vendor_items` table** — the sidecar's job here is +outage resilience (`PROTOCOL_2.md` §12.2), not search; search lives in MariaDB. Endpoint is +**`GET /market`**, not `/vendors` — axum would route the latter fine, but the collision with the +per-account RPC is a readability trap. + +Website — `shard_vendors` + `shard_vendor_items` (indexes on `vendor_serial`, `price`, `item_id`, +`display_name`; delete-then-insert per vendor in one transaction; no FKs). `shardIngest.js` handles +both kinds; **not** in `LOGGED_KINDS`. + +`KIND_FEATURE['vendor.listing'] = 'market'`, but the market feature's **SSE mapping is disabled by +default**: a live firehose of full vendor inventories would be the site's single biggest bandwidth +consumer, and no page needs it live. The page is a paginated DB query with a staleness stamp; an +admin can turn the stream on. `uoLinkSocket` paginates `/market` on reconnect, bounded by +`MARKET_SNAPSHOT_MAX = 5000` vendors so a pathological world cannot hang startup. + +`GET /public/shard/market?q=&minPrice=&maxPrice=&itemId=&map=®ion=&sort=&limit=&offset=` +(limit 1..100, default 50; `q` ≤ 60 chars; `sort ∈ {price_asc, price_desc, recent}`) and +`/market/vendors/:serial`, behind `requireFeature('market')`. **Rate-limit it** — this is the first +genuinely expensive public endpoint; `express-rate-limit` is already a dependency. + +### 8.6 The open dependency — cliloc names ✅ Resolved (shipped ahead of §8) + +`CharacterSheet.jsx:14-15` documented the gap ("without a cliloc table on the site we can only +show literals") and rendered equipment as `id {itemId}`. Search-by-name needs that table. + +**Resolved as its own website-only change, landed BEFORE the market so `/site/market` ships with real +item names.** Full design and operator guide: [`docs/website/CLILOCS.md`](../website/CLILOCS.md). +Ingest denormalizes the resolved name into `shard_vendor_items.display_name` as planned. + +Two things in the original recommendation above turned out to be wrong, and both are worth recording +because the reasoning generalises. + +**1. The committed `db/data/clilocs.json` artifact was dropped.** It predates the two Part C +corrections (§6.1) and violates both: no committed snapshot of derived content, and nothing +EA-derived ever shipped. UO's strings are EA's, exactly as the creature sprites are. Replaced with +the §6 pattern instead — parse on every boot from an operator-configured path, hash-gated, output +gitignored, `PARSER_VERSION` counted as drift. + +**2. `scripts/buildClilocs.js reads the UO client's Cliloc.enu` is not possible, and the reason +matters.** **Every current client ships its cliloc files COMPRESSED** — all eight `Cliloc.*` files +open with a DWORD whose high byte is `0x8E`, the "Mythic" container. The plain layout (`02 00 00 00 +01 00`, then `{int32 number, byte flag, uint16 length, UTF-8}`) is what those files looked like +*before* that change. Parsing a compressed file as plain does not fail cleanly: it yields ~19k +"records" with negative ids, 1,722 distinct keys out of 19,508, one 62 KB "string", and a truncation +somewhere in the middle. + +Decompressing means porting an inverse-BWT coder with a 1 KB frequency header — a few hundred lines +whose failure mode is plausible-looking garbage rather than an error. Two facts closed off the +alternatives: + +- **ServUO cannot read it either.** Its bundled `Ultima.StringList` implements only the plain layout, + so on a modern client `VendorSearch.StringList` is null and `VendorSearch.GetItemName` returns + `item.Name`. **The in-game Vendor Search gump has the same gap** — which also means §8.2's warning + never to call `GetItemName` in the sweep costs us nothing we could otherwise have had. +- The shard therefore cannot supply names on our behalf, so this could not be pushed to the plugin. + +⇒ **the operator converts once, from their own client, and the site reads the result.** Accepted +shapes are the plain binary layout and a `numbertext` export; the site sniffs which. +`server/tools/cliloc-export/` drives UOFiddler's `Ultima.dll` (the decompressor that already exists) +and writes the plain form. A shard that never converts is fully supported — names render as ids, +exactly as before. + +**Shards edit items and add new ones**, and those carry ids no stock client table has — so this reads +a **set** of sources, not one file, hash-gated together and re-read on every boot exactly as §6 reads +the ServUO tree: a base (the converted client table) plus every overlay under `custom/`, later +winning. Adding one custom item therefore never means re-exporting a 5 MB client file. Measured on +the live shard for scale: its script tree references **16,434** cliloc ids and only **37** are absent +from stock — tens against a 67k base, which is why an overlay and not a second table. `custom/` is the +one convention here that is ours rather than the shard's, because **ServUO has no server-side notion +of a custom cliloc**: they live in the patched client a shard distributes, and nothing in the tree +declares them. + +That set also brings back a hazard a single file did not have, and §8.6 answers it the way §6 does. A +corrupt source fails the parse loudly, but a source that has **vanished** parses perfectly and imports +a table quietly missing everything it contributed — an unmounted volume is indistinguishable from a +deliberate deletion. So it is **staged, not applied** (`status: 'needsReview'`), reported by both the +import and `status()`, and accepted with `{approve:true}`. It is a flag rather than §6's +approve/reject pair because the atlas stores a pending decision *so that approving re-parses*; here +nothing is stored, so re-reading at approval time is automatic. + +Five traps found by building it, all recorded in `CLILOCS.md`: + +- **`StringList.SaveStringList` RE-COMPRESSES on save.** It looks exactly like the export path and is + not; its output is byte-identical to its compressed input, because its purpose is round-tripping a + file back into the client. +- **Trimming a text line before splitting silently drops half the table.** Roughly half of a real + cliloc table is empty strings (ids the client reserves), exported as `1005008`. Trimming eats + the trailing separator, leaving a bare number that then looks like a header row — 55,994 of 123,490 + entries vanished, and the import still looked successful. +- **`Number('')` is `0`, not `NaN`.** A line starting with a separator imports as a bogus cliloc 0 + unless the empty field is rejected explicitly. +- **Tidying punctuation unconditionally corrupts real names.** Stripping leftover brackets is right + after a placeholder is removed (`[~1_stuff~]` → nothing) and wrong otherwise: a shard's custom + `"Runic Gateway Sigil (v2)"` rendered as `"(v2"`. Same shape as the `%` rule. **Found only by + running a shard-style overlay through it** — every stock-table fixture passed. +- **Source labels must be forward-slashed and root-relative**, or the same directory fingerprints + differently on Windows and Linux and every boot looks like a change. The identical bug §6 records. + +Blank entries are dropped at import (123,490 parsed → **67,496** stored), which also makes the binary +and text paths converge on identical content. + +### 8.7 Client + +`routes/public/Market.jsx` at `/site/market`, with a *"prices last refreshed N minutes ago"* banner +driven by `staleAt` (the oldest `shard_vendors.updated_at`). The round-robin sweep means data is +inherently up to one full cycle old, and the UI must say so. + +Shipped with a second page, `routes/public/MarketVendor.jsx` at `/site/market/vendors/:serial` — +where a search result points. It is the only surface that can render the two states the result list +cannot: a `truncated` shop (*"showing 250 of 3,104 — this shop holds more than the shard +publishes"*) and a `location` an admin has gated away, which is a real answer rather than an empty +coordinate. + +### 8.8 What the build changed + +Four things the implementation settled differently from §8 as written, all of them found by building +against the live shard. + +**1. `location` is a nested object, not flat `map`/`x`/`y`/`region`.** §8.1's payload sketch had them +flat, and it would have made `market.location` — a rule Part A pre-wired — **inert**, exactly like +the `characterName` miss §7.5 records: `projectValue` matches literal JSON keys, so there is no +`location` key for the rule to match. Flat keys would have needed five rules that could drift apart. +Nesting makes one rule hide the facet, the coordinates, the region and the house together, on the +live frame and the stored read model alike, because both now spell it the same way. + +The other pre-wired rule, `market.ownerName`, checked out — it is a real key on the frame. Owner is +written as flat `ownerSerial`/`ownerName` rather than through `BridgeJson.Actor`, which would add +`acct` and `webId`; same argument `points.board` makes. `ownerSerial` was **added** to the +configurable fields alongside `ownerName`, because an admin who hides the owner's name and leaves a +serial every other board resolves back to that name has not hidden anything. + +**2. The per-vendor diff signature is the full listing set, not §8.3's `count | Σ(serial ^ price)`.** +That hash collides on the single most common change a shop makes: two items swapping prices, which +is what re-pricing looks like. The signature is built over the same buffer the frame is written +from, in the same order, so a match really does mean an identical frame. + +**3. There is no `payload` column on `shard_vendors`.** §8.5 implied the board pattern (whole frame +in JSON, columns hoisted for display). It does not apply here: the items ARE the searchable rows, so +they are normalized into `shard_vendor_items` and there is nothing left worth duplicating. The +sidecar keeps the whole blob, because outage resilience is its job and search is not. + +**4. Sweep cost is reported, and a slow tick warns.** The batch cap is a *claim* about per-tick cost, +and an operator tuning `MarketSweepBatch` was otherwise tuning blind. `[bridge status` now carries +`lastMs`/`maxMs`, and a tick over 50 ms prints a rate-limited warning naming the knob. + +Measured on the live shard (27 vendors × 40 listings, 209k items / 43k mobiles): + +| | | +|---|---| +| First tick — 25 vendors emitted cold | **15.4 ms** | +| Second tick — the remaining 2 | **3.4 ms** | +| Steady state — nothing changed | **0.3 ms** | +| Website `/market` search over 1,040 listings | 1,040 total, names resolved | +| Cliloc re-resolution pass over 1,040 rows | **50 ms** | + +The diff is what makes the steady state ~free; the batch cap is what bounds the cold case. Note the +arithmetic the warning exists for: at the default cap of 250 listings, a batch of 25 **full** shops +is 6,250 items ≈ 95 ms — over budget. Real shops hold tens, which is why 25 is the default, but a +shard of commodity resellers should lower the batch, and now it will be told to. + +Two smaller things worth not rediscovering: + +- **`BridgeJson.Escape` takes a NON-NULL string** — it dereferences `value.Length` immediately — and + `BridgeJson.Str` writes its own `,"key":` prefix, so neither serves a value inside a hand-built + object. Nearly everything this frame writes is legitimately null (an item's plain `Name` is null + for almost every item; a vendor in the street has no house), so that is the common path, not an + edge case. `BridgeMarket.Text()` is the two-line writer that was missing. +- **The ServUO console writes in the OS code page**, so an em dash in a `Console.WriteLine` renders + as `???` in the log an operator would paste into an issue. Bridge console output is ASCII. + +Search-side, one thing the site had to fix rather than inherit: `%` and `_` in a user's query are +**LIKE** metacharacters, not SQL ones, so parameterization does not neutralize them — a search for +`%` would otherwise match every listing on the shard. `shardMarket.db.js` escapes them. (The atlas's +`LIKE` searches predate this and have the same shape over a much smaller table; worth a follow-up, +not a blocker here.) + +--- + +## 9. Sequencing + +| Order | Part | Repos touched | Wire change | State | +|---|---|---|---|---| +| 1 | **A** — visibility framework + actor-leak fix | website, docs | none | ✅ Done | +| 2 | **B/1** — `world.ruleset` (§5) | all four | new kind | ✅ Done | +| 3 | **C** — spawn atlas (§6) | website, docs | none | ✅ Done | +| 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | ✅ Done | +| 5a | **B/3 dependency** — cliloc table (§8.6) | website, docs | none | ✅ Done | +| 5b | **B/3** — `vendor.listing` (§8) | all four | new kinds | ✅ Done | +| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | 🟨 In review — `edge` → `main` held for Android parity (§10) | + +--- + +## 10. Documentation obligations + +- This file (`link/v3.md`) is the canonical 3.0 design. +- `PROTOCOL_2.md` §10.4 gains a note that `world.systems` is superseded by `world.ruleset`, and that + the deferred VvV question is answered (`VvV.cfg Enabled=True`, Factions off). +- `INTEGRATION.md` — catalog entries and §6 consumer sections for each new kind, plus the v2→v3 + upgrade note for operators. +- `PLAN.md` — phasing. +- `website/BACKEND_DESIGN.md` — every new table and route, and **the visibility framework as a + security contract**: the audience ladder, the two locked rules, and the fail-closed kind map belong + in the security section. +- NEW `website/SHARD_VISIBILITY.md` — admin-facing: what each feature exposes, what each rung means, + what cannot be loosened. +- NEW `website/SPAWN_ATLAS.md`, NEW `website/CLILOCS.md`, NEW `website/MARKETPLACE.md`. +- `PROJECT_TREE.md` in each touched repo. +- `npm run swagger` **and** `npm run routes:manifest` on every route-touching PR — both are committed + artifacts, and `test/routeManifest.test.js` fails on drift. + +**Android parity — now scoped, and it gates the cutover (decided 2026-07-30).** This was written as a +"track separately" follow-up. It was re-examined before the cutover and the gap is wider than nav +hiding: the app consumes the same public/player shard API but has **no consumer for any of the four new +features** (`ruleset`, `leaderboards`, `market`, `atlas`), no `points` block on its character sheet, no +cliloc-resolved item names (§8.6), and — the part that matters for §3 — **it gates shard navigation on +session role alone**, so an admin who disables a feature or raises its audience leaves the app +rendering entries that `404`/`403` into a generic error where the web client hides them. + +Two things were verified as already correct and are recorded so they are not re-derived: the app's SSE +request rides the same authenticated OkHttp client as every other call, so an app session resolves to +the same audience rung as the same account on the web; and every shard DTO in the app is +nullable-with-defaults, so field projection strips fields without a deserialization failure. + +Scoped as **M11 in [`../android/PLAN.md`](../android/PLAN.md) §9**, two PRs (the visibility rules + +read-model adds, then the four screens). `edge` → `main` is held until both land, so web and app +surface the same shard on the same day. Neither PR is coupled to the merge order — on a pre-v3 website +every new route and `/public/shard/features` `404`s and the app falls back to today's behavior — so +holding the cutover is a schedule decision, not a technical dependency. + +--- + +## 11. Verification + +**Plugin** — `servuo-plugins\deploy.ps1 -ServerPath -Verify`, inspect the ADD/CHANGE list, +then re-run without `-Verify` (ServUO must be stopped). Boot with `tools/stub_sidecar.ps1` listening +and **confirm the compile banner in the console, not merely the absence of errors** — +`BRIDGE_PLUGIN_PLAN.md` §1 warns that a failing build is silently ignored and the previous +`Scripts.dll` reloads. Then `[bridge status`, `[bridge sweepnow`, `[bridge reload`. + +- §5: eyeball the emitted `world.ruleset` frame for anything sourced from `Server.cfg`, `Staff.cfg`, + `Email.cfg`, `DataPath.cfg` or `Bridge.cfg`. +- §8: with a seeded world, time one sweep tick and confirm the batch cap holds it under ~1 ms. + +**Sidecar** — `cargo build && cargo clippy`; `curl -H "Authorization: Bearer " +localhost:8080/ruleset` (and `/points`, `/market`); confirm `X-UOLink-Version: 3` and that a client +declaring 2 receives a 409. + +**Website server** — `DB_HOST=127.0.0.1 DB_PORT=59999 node --test`. New tests, each modelled on an +existing sibling: `test/shardVisibility.test.js`, `test/shardBroadcast.visibility.test.js`, +`test/shardIngest.{ruleset,points,market}.test.js` (after `shardIngest.protocol2.test.js` — stubbed +deps, asserting routing and `logged` flags), `test/spawnAtlas.parse.test.js` (pure functions, inline +fixtures). Then `npm run routes:manifest` and `npm run swagger`, committing both. + +**Full stack** — against a local MariaDB: apply `db/schema.sql` (idempotent), start the server, +confirm `uoLinkSocket` backfill logs the new snapshot lines and that `uo_link_config.protocol` +migrated to 3, then load `/site/rules`, `/site/atlas`, `/site/leaderboards`, `/site/market`. + +**Visibility smoke test** — for each of the five rungs, walk every shard page and confirm gating and +field projection match the configured matrix. Same shape as the 200-routes × 5-access-levels sweep +already run for the domain split. + +--- + +## 12. Critical files + +| File | Why | +|---|---| +| `website/server/src/utils/shardBroadcast.js` | The security boundary; reworked from a static allowlist to per-connection audience filtering. **The highest-risk file in 3.0.** | +| `website/server/src/utils/shardVisibility.js` (new) | Ladder, kind→feature map, projection | +| `website/server/src/utils/shardIngest.js` | The dispatcher every new kind routes through | +| `website/server/src/model/shardState/shardState.model.js` | The `shape*` projections, including the `shapeGuild` leak §3.1 fixes | +| `servuo-plugins/overlay/Scripts/Custom/Bridge/BridgeHousing.cs` | Cleanest copy of the diff-sweep pattern; template for `BridgePoints.cs` and `BridgeMarket.cs` | +| `link/sidecar/src/main.rs` | `PROTOCOL_VERSION` 2→3 and the board-projection match | +| `website/server/db/schema.sql` | All new `shard_*` tables plus the `uo_link_config.protocol` migration | diff --git a/website/BACKEND_DESIGN.md b/website/BACKEND_DESIGN.md index ccb4f0d..409569c 100644 --- a/website/BACKEND_DESIGN.md +++ b/website/BACKEND_DESIGN.md @@ -99,8 +99,12 @@ server/ pages.router.js (2) /public/pages — the draft-preview route precedes /:slug and is deliberately not site-mode gated - shard.router.js (12) /public/shard/* incl. the anonymous + shard.router.js (14) /public/shard/* incl. the anonymous SSE stream; never site-mode gated + atlas.router.js (6) /public/atlas/* — the spawn atlas. + NOT under /shard: nothing here + touches the sidecar, and unlike + /shard it IS site-mode gated site.router.js (4) /settings /status /version /contact — the group-root singletons; declares no router-level middleware @@ -358,6 +362,232 @@ analogue to a password — and there is no hash-lookup constraint (verification unused rows and `bcrypt.compare`s each, like password verification). `used_at` is the single-use marker. Cleared wholesale on TOTP disable / password change / password reset. +### shard_ruleset — the shard's published ruleset (Protocol 3.0) + +Singleton row (`id = 1`, CHECK-constrained) holding the latest `world.ruleset` frame: `rev`, +`expansion`, `payload` JSON (the whole frame), `t`, `updated_at`. The shard re-emits the complete +ruleset on every sidecar connect, so this is an **overwrite, not an append** — and the kind is +deliberately **not** in `LOGGED_KINDS`, since logging it would put a duplicate row in `shard_events` +on every reconnect while `server.hello` already marks each of those. + +The frame is stored whole rather than normalized into columns: it is a flat description of server +config that is read as one page, so splitting it up would mean a schema change every time the shard +grows a new block. `rev` (the shard's FNV-1a of the body) and `expansion` are hoisted only because +they are cheap to display — the same payload-plus-hoisted-columns shape `shard_champs` uses. + +**No row means the shard has never published one** (an older plugin, or `Bridge.RulesetEnabled=false`), +served as `null` rather than `{}`: "not published yet" and "published, everything off" are different +answers and the page renders them differently. + +### shard_points_boards — points / loyalty leaderboards (Protocol 3.0) + +One row per point system, keyed by the shard's own `PointsType` name (`QueensLoyalty`, +`CleanUpBritannia`, …). The shard carries ~25 of these, each a standing players build over months. +Columns: `system` (PK), `name`, `name_cliloc`, `max_points`, `players`, `show_on_gump`, `payload` JSON +(the whole `points.board` frame), `t`, `updated_at`. + +**The top-N list stays inside `payload`** rather than being normalized into a `shard_points_entries` +table. It is a fixed-size list (10 by default) that is only ever read whole — exactly like +`shard_governors.candidates` — so normalizing buys nothing until something needs a per-character +reverse lookup, and a character's own standings already ride inside `char.profile` instead. + +Board state, not events: `points.board` is **not** in `LOGGED_KINDS`, for the same reason +`guild.update` isn't. The shard emits a frame every time anyone's score moves a top ten, so logging +would grow `shard_events` without bound for something whose only interesting value is its latest +version. There is also **no delete path** — the shard's set of systems is fixed at startup, so there is +no `points.remove` to mirror. + +Two values carry non-obvious meanings, both set by the plugin and both documented in +[`link/INTEGRATION.md`](../link/INTEGRATION.md) §4: + +- **`max_points = 0` means uncapped**, and on a real shard that is the *common* case (ServUO's + uncapped idiom is `double.MaxValue`, which the plugin normalises to 0). Anything rendering + `points / max_points` must special-case it. +- **`name` is usually NULL**, with `name_cliloc` set instead — most systems name themselves with a + cliloc rather than a literal. Listing therefore orders by `COALESCE(name, system)`, so boards + awaiting cliloc resolution sort by their own key rather than clumping together under NULL. + +### shard_vendors / shard_vendor_items — the player-vendor marketplace (Protocol 3.0) + +The shard-wide shop index, fed by `vendor.listing` / `vendor.listing.remove`. One row per player +vendor and one per priced listing. Full operator detail in [`MARKETPLACE.md`](MARKETPLACE.md); the +design is `docs/link/v3.md` §8. + +| Table | Shape | +|---|---| +| `shard_vendors` | `serial` (PK), `shop_name`, `owner_serial`, `owner_name`, `map`/`x`/`y`/`z`, `region`, `house`, `item_count`, `item_total`, `truncated`, `t`, `updated_at`. Indexes on owner, map, region and `updated_at`. | +| `shard_vendor_items` | `id` (PK), `vendor_serial`, `serial`, `item_id`, `hue`, `amount`, `price`, `name`, `cliloc`, `display_name`, `child`. Indexes on `vendor_serial`, `price`, `item_id`, `display_name`, and `(display_name, price)`. | + +**Ingest is per-vendor and authoritative**: the frame is the whole shop, so ingest is +delete-then-insert of that vendor's listings inside one transaction. All-or-nothing matters +specifically because the two writes are "the shop" and "what is in it" — a failure between them +leaves a shop advertising an inventory it no longer has, which is visibly wrong and indistinguishable +from a genuinely empty shop. No foreign keys, consistent with every other `shard_*` table. + +**There is deliberately no `payload` column**, unlike `shard_points_boards` directly above. The +board's top-N is a fixed-size list read whole, so it lives in JSON; here the items *are* the +searchable rows, so they are normalized and nothing is left worth duplicating. The sidecar keeps the +whole blob — outage resilience is its job, search is ours. + +Market state, not events: neither kind is in `LOGGED_KINDS`, and this is the strongest case of the +three v3 kinds. One frame carries up to 250 listings and the sweep re-emits a shop on any price +change, so logging would turn `shard_events` into a price history nobody reads. + +Two columns carry non-obvious meanings: + +- **`item_count` vs `item_total`.** `item_count` is what the frame published; `item_total` is what + the shop actually holds. They differ when `truncated` — the shard caps listings per frame + (`Bridge.MarketMaxListings`, 250 by default), and a commodity reseller with thousands of stacks + genuinely exceeds it. Any UI must show both or it presents a partial shop as complete. +- **`display_name` is denormalized at ingest**, resolved from the item's literal `name` (preferred — + a player set it, so it is more specific) else its `cliloc` against `shard_clilocs`. Resolving at + query time would put the cliloc table on the hot path and make search-by-name impossible. Because + the shard's diff sweep will not re-send an unchanged shop just because the site learned what its + items are called, **a cliloc import triggers a bulk re-resolution** of this column (after a boot + import and after an admin import; ~50 ms per thousand rows, never throws). + +`updated_at` is written explicitly on every upsert rather than left to `ON UPDATE CURRENT_TIMESTAMP`, +which MariaDB does not fire when every column is written back unchanged. A shop re-published +identically is still *freshly confirmed*, and without this the staleness banner would age a perfectly +current shop forever. + +### shard_feature_visibility — per-feature audience config (Protocol 3.0) + +One row per shard feature: `feature` (PK), `enabled`, `audience` (a rung on the ladder in §6.5), +`stream` (whether the feature's kinds fan out over SSE at all), `field_rules` JSON (`{field: rung}` +for the sensitive fields only), `updated_by`, `updated_at`. + +**An absent row means "use the compiled default", and the compiled defaults reproduce pre-3.0 +behavior — so an empty table is a no-op and there is nothing to seed.** Stored rows are merged over +the defaults on read, which is also where the invariants are re-applied: a row naming an unknown +feature is ignored (a stale row must not resurrect a removed feature), an invalid rung falls back to +the default rather than failing open, and a rule touching a locked field (`acct` / `webId`) is +discarded. See §6.5. + +### shard_spawn_* / shard_regions / shard_landmarks / shard_champion_spawns / shard_atlas_meta — the spawn atlas (Protocol 3.0) + +Static shard **content**, not live shard state. Nothing here comes from the sidecar: the atlas is +derived from the shard's own ServUO tree, re-read on **every server boot** and hash-gated so an +unchanged tree costs one read pass and no write. Nothing is precomputed and committed — a shard's +maps change over its life, and a snapshot in the repo would silently drift from the world players +actually see. These tables stay populated whether the shard is up or not. Full operator detail in +[`SPAWN_ATLAS.md`](SPAWN_ATLAS.md); the design is `docs/link/v3.md` §6. + +**No facet name appears anywhere in the code.** A shard may add facets, replace them, or rename them +when its maps are updated; the facet set is discovered from the tree, and the loose spellings in +`Data/Locations` are matched against it rather than looked up in a table. + +| Table | Key columns | +|---|---| +| `shard_spawn_creatures` | `slug` PK, `name`, `total`, `points`, `facets` JSON, `art` NULL | +| `shard_spawn_points` | `id` PK, `facet`, `name`, `x`, `y`, `width`, `height`, `spawn_range`, `max_count`, `min_delay`, `max_delay`, `tod_start/end/mode`, `region`, `landmark`, `label` | +| `shard_spawn_point_types` | `(point_id, slug)` PK, `max_count` | +| `shard_regions` | `facet`, `name`, `type`, `priority`, `parent`, `rects` JSON | +| `shard_landmarks` | `facet`, `name`, `grp`, `x`, `y`, `z` | +| `shard_champion_spawns` | `slug` PK, `name`, `grp`, `type`, `random_type`, `facet`, `x`, `y`, `z`, `radius`, `label` | +| `shard_atlas_meta` | Singleton (`id = 1`), `payload` JSON (counts, a sha256 per source file, `parserVersion`), `imported_at` | +| `shard_atlas_pending` | Singleton (`id = 1`), `status` (`pending`/`rejected`), `payload` JSON, `detected_at` | + +The first seven are **import-owned**: a refresh empties and reloads every one inside a single +transaction, so a failed reload leaves the previous atlas intact rather than a half-loaded world. +Nothing else writes to them, and nothing holds a foreign key to them — no FKs at all, consistent with +every other `shard_*` table. + +**`shard_atlas_pending` is the security-relevant one.** A refresh that would REMOVE a facet is never +applied automatically: facet loss is indistinguishable at boot from a half-copied or mid-update tree, +so it is staged here for an admin to approve or reject, and **startup is never blocked by it**. Only +the decision is stored — source hashes plus the facet diff, a few KB — and approving re-parses the +tree, so a multi-megabyte blob never lands in the database and what gets applied matches the tree at +approval time. A rejection is remembered against those exact hashes so a declined refresh does not +re-prompt on every restart. Everything else (new facets, renamed regions, changed spawns) applies +immediately, since none of it can destroy data an operator would miss. + +The boot refresh is **best-effort by contract**: no configured path, an unreadable mount, a malformed +file or a database error is caught and logged, and the site comes up serving whatever atlas it had. +The tree path comes from the `spawn_atlas_servuo_path` setting, falling back to `SERVUO_PATH`. + +**A refresh re-derives when the tree changed OR the parser did.** `spawnAtlasSource.PARSER_VERSION` +is stored in `shard_atlas_meta` beside the source hashes and bumped whenever the parser produces +different data from identical files. Hashing the tree alone would strand an install whose maps never +change on whatever an older build derived — a corrected parse would ship and never reach the data. + +Four column choices worth stating, because each one is a trap: + +- **`spawn_range`, not `range`**, and **`grp`, not `group`** — both are reserved words. +- **`DELETE`, not `TRUNCATE`.** `TRUNCATE` is DDL in MariaDB and implicitly commits, which would + defeat the all-or-nothing reload. At ~7k rows the difference does not matter. +- **Point ids are assigned explicitly**, not left to `AUTO_INCREMENT`: the `shard_spawn_point_types` + rows need to know them, and `conn.batch()` reports no usable `insertId` for a multi-row insert. +- **Plain `INDEX` on `name`, deliberately not `FULLTEXT`.** ~800 creature rows makes a `LIKE` scan + free, and FULLTEXT's minimum token length would break searches for names like "orc". + +`shard_champion_spawns` is the *configured* altar roster ("there is an Unholy Terror altar in +Deceit"). The live `champ.update` feed in `shard_champs` is the separate answer to "it is on level 3 +right now". Both exist; they are not the same data. + +**`shard_spawn_creatures.art` is always NULL on a fresh import.** The project ships no creature +artwork: sprites live in the operator's own client `.mul`/`.uop` files and are theirs, not ours to +redistribute. An operator supplies art via a gitignored map plus images under the (already +gitignored) `server/uploads/atlas/`. Text-only is the normal, supported state. + +### shard_clilocs / shard_cliloc_meta — UO's localization table (Protocol 3.0) + +Items on the wire carry a `LabelNumber`, not a name. The bridge has always sent it — +`char.profile.equipment.cliloc`, reward titles as a cliloc number in string form, and one per +marketplace listing — but with no table to resolve it against, the character sheet could only render +`id 1023721` where the game renders "quarter staff". + +| Table | Shape | +|---|---| +| `shard_clilocs` | `number` INT PK, `flag`, `text` TEXT | +| `shard_cliloc_meta` | Singleton (`id = 1`), `payload` JSON (source file, sha256, count, `parserVersion`), `imported_at` | + +Import-owned and all-or-nothing in one transaction, same contract as the atlas — including **`DELETE`, +not `TRUNCATE`**, for the same reason. + +**Sourced from files the operator supplies**, at a path from the `cliloc_client_path` setting falling +back to `UO_CLIENT_PATH`. Nothing client-derived is committed: UO's strings are EA's, exactly as the +creature sprites are. A shard with nothing configured is fully supported — names render as ids. Full +design and operator guide: [`CLILOCS.md`](CLILOCS.md). + +**It reads a SET of sources, not one file**, because shards edit items and add new ones and those +carry cliloc ids no stock client table has. A base (the converted client table) plus every overlay +under `custom/` are re-read on every boot and hash-gated **together**, exactly as the atlas re-reads +`Regions.xml` + `Locations/*.xml` + `Spawns/*.xml` + `ChampionSpawns.xml`. Later sources win, so an +overlay both adds ids and overrides stock ones, and adding one custom item never means re-exporting a +5 MB client file. Scale, measured on the live shard: its script tree references 16,434 cliloc ids and +only 37 are absent from stock — tens of entries against a 67k base, which is why this is an overlay +and not a second table. + +The conversion step is not avoidable: **every current client ships its cliloc files compressed** +(first DWORD's high byte `0x8E`), and ServUO's own bundled `Ultima.StringList` cannot read that +either — so the shard cannot supply names on our behalf. The plain layout and a delimited text export +are both accepted, sniffed by header rather than extension. + +Three decisions worth stating: + +- **`text` is TEXT, not VARCHAR.** Long property descriptions reach 12 KB. The index that matters for + marketplace search is the denormalized `shard_vendor_items.display_name`, not this table. +- **Blank entries are dropped at import** — 123,490 parsed → **67,496** stored. Roughly half a cliloc + table is empty strings for ids the client reserves and never uses; a row that resolves to no name is + indistinguishable from no row at all, and dropping them makes the binary and text imports converge + on identical content. +- **Two refusals, one of them the atlas's.** A corrupt source fails the parse on a truncated record, + so it is caught outright and leaves the previous table serving. But a source that has **vanished** + parses perfectly and imports a table quietly missing everything it contributed — an unmounted volume + and a deliberate deletion are indistinguishable from here, which is precisely the ambiguity the + atlas stages a facet removal for. So it is escalated: `status: 'needsReview'`, nothing applied, + `missingSources` reported by both the import and `status()`, and an admin accepts it with + `{approve:true}`. That is a flag rather than the atlas's approve/reject pair because the atlas + stores a pending decision so that approving **re-parses** the tree; here nothing is stored, so + re-reading at approval time is automatic. + +**Resolution is server-side and there is no public route.** The table is never served *as* a table: +67k rows would dwarf any page using them, and the Android client consumes the same already-resolved +JSON. `resolveMany()` returns only ids that resolved to something displayable — placeholders like +`~1_val~` are stripped, since the bridge sends the id and never the property packet that carries the +arguments — and it never throws, because a cliloc lookup is decoration on a character sheet. + --- ## 4. API contract @@ -372,7 +602,7 @@ are authoritative, and they answer different questions: | Artifact | Source of truth for | Generated by | |---|---|---| -| `server/routes.manifest.json` — mirrored as [api-route-inventory.json](./api-route-inventory.json) | **What URLs exist.** 200 public routes + 2 on the internal listener, sorted, method + path only. | `npm run routes:manifest`, by walking the live Express stack | +| `server/routes.manifest.json` — mirrored as [api-route-inventory.json](./api-route-inventory.json) | **What URLs exist.** 215 public routes + 2 on the internal listener, sorted, method + path only. | `npm run routes:manifest`, by walking the live Express stack | | `server/swagger/swagger-output.json` — served at `/api/docs` | **What each route means.** Parameters, bodies, response codes, security. | `npm run swagger`, from `#swagger.*` annotations | The split is deliberate: Swagger is annotation-derived, so an unannotated route is invisible in it and @@ -556,6 +786,19 @@ from the per-route **siteMode** middleware (§5), never from an auth gate. | GET | `/wiki` | list of pages (slug + title) | | GET | `/wiki/:slug` | single page | | POST | `/contact` | (rate-limited) send mail via SMTP; if unconfigured, respond `{fallback:"mailto", email}` | +| GET | `/shard/ruleset` | the shard's own published ruleset (Protocol 3.0 `world.ruleset`): expansion, which optional systems are on, skill/stat caps, account and house limits, champion scroll rules, the save/restart schedule. Served from `shard_ruleset`, so it renders while the shard is down; live via `world.ruleset` on `/shard/stream`. Behind `requireFeature('ruleset')`. **`null`** means the shard has never published one — a real answer, distinct from a published ruleset. `caps.skill` / `caps.totalSkill` are in **tenths** (1000 = 100.0). | +| GET | `/shard/points` | every points/loyalty leaderboard the shard publishes (Protocol 3.0 `points.board`) — Queen's Loyalty, Void Pool, the nine city loyalties, Clean Up Britannia, … Served from `shard_points_boards`, so it renders while the shard is down; live via `points.board` on `/shard/stream`. Behind `requireFeature('leaderboards')`, ordered by display name. **`maxPoints: 0` means uncapped** (the common case), and `nameString` is usually `null` with `nameNumber` holding a cliloc — resolve client-side or humanise the `system` key. | +| GET | `/shard/points/:system` | one board by the shard's `PointsType` name (e.g. `QueensLoyalty`); `:system` must match `/^[A-Za-z][A-Za-z0-9_]{0,47}$/` or **400** before any query runs. **404** = the shard has never published that system, which is distinct from a published board nobody has scored in yet (**200** with an empty `top`). | +| GET | `/shard/market?q=&minPrice=&maxPrice=&itemId=&map=®ion=&sort=&limit=&offset=` | search the player-vendor marketplace (Protocol 3.0 `vendor.listing`). Returns **listings**, not vendors — "who sells X and for how much" is the question, and a vendor-shaped result would make every caller flatten the shops back out. Served from `shard_vendors` + `shard_vendor_items`, so it renders while the shard is down. Behind `requireFeature('market')` **and rate-limited** — the first genuinely expensive public read on the site (a `LIKE` scan plus a `COUNT` over what is typically the largest `shard_*` table, reachable with no session). `sort ∈ {price_asc, price_desc, recent}`. `q` matches the resolved display name **or** the item's literal name, with `%`/`_` escaped: they are `LIKE` metacharacters, not SQL ones, so parameterization alone would let `?q=%` match every listing on the shard. Every response repeats `staleAt` (the oldest vendor row) because the shard sweeps round-robin — a banner that ages with the results it labels, not one fetched once. | +| GET | `/shard/market/meta` | index size, staleness (`staleAt`/`freshAt`) and which facets and regions actually hold vendors, so a client builds its filters without running a search it will discard. | +| GET | `/shard/market/vendors/:serial` | one shop and its listings; `:serial` must match `/^0x[0-9A-Fa-f]{1,16}$/` or **400** before any query runs. **404** = a serial the index has never seen, which also covers a vendor since dismissed or hidden — to an anonymous caller those are the same answer, and distinguishing them would leak that a hidden vendor exists. `truncated` (with `total` exceeding `count`) means the shop holds more than the shard publishes per frame. | +| GET | `/shard/features` | the shard features **this caller** may reach plus the audience rung they resolved to (§6.5), so a client hides nav it can't follow. Reports only what the caller can see — the list itself never discloses a gated feature. Consumed by the SPA header and (pending) the Android nav. | +| GET | `/atlas/creatures?q=&facet=&limit=&offset=` | the bestiary, most numerous first, with an unpaginated `total`. Static content parsed from the shard's ServUO tree — **not** sidecar-backed, which is why the atlas sits outside `/shard`, and unlike `/shard/*` it **is** site-mode gated. Behind `requireFeature('atlas')`. `?facet=` is matched exactly and never validated against a list (no facet name exists in the code); the filter is an `EXISTS` over the points rather than a JSON path or `JSON_SEARCH` built from caller input, whose `%`/`_` wildcards would make `?facet=%` match everything. | +| GET | `/atlas/creatures/:slug` | one creature: `places` (the point-in-rect aggregate — "lizardman → Shrines, Isamu-Jima, Yew"), `spawners` (the bounded raw list, with `spawnersTruncated`), `alsoHere`. **`points` is a COUNT and `spawners` is the LIST** — named apart so one key never means a number on one route and an array on another. `minDelay`/`maxDelay` are in **seconds**, normalised at parse time from the source's per-record minutes-or-seconds. 404 = no such creature in this atlas. | +| GET | `/atlas/regions?facet=&q=` | named regions and the rectangles that placed each spawner | +| GET | `/atlas/landmarks?facet=&q=` | points of interest, labelled by `group` ("Covetous", not "Level 1") | +| GET | `/atlas/champions?facet=` | the **configured** altar roster. Not `/shard/champs`, which is the live board. | +| GET | `/atlas/meta` | facets, counts and when the atlas was parsed. Game-world facts only — the ServUO path, source hashes and any pending refresh are operator detail and live on the admin route. | Public content GETs pass through the **siteMode** gate (§5). @@ -570,6 +813,10 @@ the whole gate. The ops/config capabilities — `uo-link`, `email`, `discord-bot linking carries no extra gate and the in-game staff operations carry `modAccess`. There is no residual file: every admin route is declared in a capability router. +`GET`/`PUT /admin/shard/visibility` are the third tier on that mixed prefix: **`adminOnly`**, because +they decide what *anonymous* visitors can see (§6.5). They sit above `modAccess` deliberately — a +moderator can ban a player but cannot decide what the public internet reads. + `GET /dashboard` and `PUT /site-mode` are the one place where a **single screen spans two tiers**: the dashboard is staff-wide, but the site-mode toggle on it is `adminOnly`. The client must therefore gate that control on its own (`Dashboard.jsx` renders it only for `role === 'admin'`) rather than relying on @@ -595,6 +842,13 @@ file a route sits in — that is the property the route manifest freezes. | GET | `/users/:id/trusted-devices` | list a user's active trusted devices (never tokens) | | DELETE | `/users/:id/trusted-devices` · `…/:deviceId` | revoke all / one of a user's trusted devices (logs `admin.trusted_device.revoke[_all]`) | | POST | `/users/:id/mfa/reset` | recover a locked-out user: disable TOTP + revoke all trusted devices + clear recovery codes (logs `admin.user.totp.reset`) | +| GET | `/shard/atlas` | spawn-atlas status (`adminOnly`): the ServUO path, whether the tree is readable, whether it has drifted from what is loaded, counts, facets, and any refresh staged for review. The public `/atlas/meta` reports the game world only; the filesystem detail is here. | +| POST | `/shard/atlas/import` | re-import without restarting; `{force}` ignores the hash gate. **An unreadable tree answers 200 with `status:"unavailable"`, not 500** — `refresh()` reports outcomes rather than throwing (the boot path must never be blocked by a bad tree) and that contract is preserved at the API. | +| POST | `/shard/atlas/approve` · `/shard/atlas/reject` | answer a refresh staged because it would REMOVE a facet. Approving **re-parses** the tree, so what lands matches it at approval time; rejecting is remembered against those source hashes so it does not re-prompt every restart. 404 when nothing is staged. | +| PUT | `/shard/atlas/path` | point the atlas at a different tree (persisted as `spawn_atlas_servuo_path`, which wins over `SERVUO_PATH`). Blank clears it. Deliberately **does not import** — moving the mount and reloading the world are separate decisions — and returns fresh status so the panel can offer the import next. | +| GET | `/shard/clilocs` | cliloc-table status (`adminOnly`): every source found now (base first, then `custom/` overlays in merge order), what each contributed at the last import, readability, drift across the set, the entry count, and `missingSources`. `configured:false` is a supported state — item names then render as ids. No public counterpart: the table is never served *as* a table. | +| POST | `/shard/clilocs/import` | reload after a client patch or an overlay edit; `{force}` ignores the hash gate, `{approve}` accepts a **vanished** source (refused by default — see the table notes above). **A missing path — or the likely mistake of pointing at the client's own COMPRESSED `Cliloc.enu` — answers 200 with `status:"unavailable"` and a `code`, not 500.** `COMPRESSED` is called out by name: a 500 would say only "something broke", and the operator needs to be told which file to convert. | +| PUT | `/shard/clilocs/path` | point the site at a different cliloc base file or directory (persisted as `cliloc_client_path`, which wins over `UO_CLIENT_PATH`). Overlays are read from `custom/` beside it either way. Blank clears it. Deliberately **does not import**, same reasoning as the atlas path. | Every admin write logs to `activity_log`. @@ -628,7 +882,11 @@ who"; `activity_log` provides the history feed. - **Cookie**: `httpOnly`, `sameSite=Lax`, `path=/`, and **`secure` decided per-request** (`COOKIE_SECURE=auto` → `secure: req.secure`). - **Trusted-device MFA.** A second, separate httpOnly cookie (`rg_trust`, default 30d) — opaque, sha256-hashed server-side in `trusted_devices` — lets a browser/app **skip the TOTP step** (never the password) on future logins. It is a server-side, per-row-revocable record (never a JWT claim), so the stateless session JWT is unchanged and trust stays revocable. It only ever gates the **second factor**; it deliberately outlives logout, and is cleared on untrust / password change / password reset / TOTP disable. **Recovery codes** (bcrypt, single-use) are the 2FA-lockout fallback. All admin trusted-device/MFA actions and the self actions (`auth.login.trusted_device`, `account.trusted_device.*`, `account.recovery_code*`, `admin.trusted_device.*`, `admin.user.totp.reset`) are audit-logged. See `docs/website/TRUSTED_DEVICES_MFA.md`. This is the key to dual access: the cookie is `Secure` when reached through Pangolin (HTTPS, `X-Forwarded-Proto: https`) but **not** `Secure` when reached directly over the LAN IP on plain HTTP — so login works in both. `COOKIE_SECURE=true|false` can force it. Requires `trust proxy` (below). `localhost:5173` (Vite) and `localhost:3000` are same-site, so the cookie flows in dev too. - **bcrypt** hashing (cost 10+); plaintext passwords never stored, logged, or returned. -- **Rate limiting** (`express-rate-limit`) on `/auth/login` and `/public/contact`. +- **Rate limiting** (`express-rate-limit`) on `/auth/login`, `/public/contact`, and — the only limited + *read* — `/public/shard/market` and `/public/shard/market/vendors/:serial` (60/min/IP). Every other + public read is an indexed lookup of bounded size; the marketplace search is a `LIKE` scan plus a + `COUNT` over the largest `shard_*` table, anonymous by default, so it is the one public GET that is + worth money to serve. - **Validation** (`express-validator`) on all writes; centralized error handler. - **helmet** with a Content-Security-Policy tuned for the built React SPA. The policies now live in **`server/src/config/csp.js`** (`app.js` only wires them up): @@ -672,6 +930,77 @@ who"; `activity_log` provides the history feed. - **`app.set('trust proxy', 1)`** so secure cookies, `req.ip`, and rate-limiting work behind Pangolin. - **CORS**: same-origin in prod (SPA served by Express). Dev only: allow `CLIENT_ORIGIN` (Vite, `http://localhost:5173`) with `credentials:true`. +### 6.5 Shard visibility — the audience boundary (Protocol 3.0) + +Every shard-derived surface is gated by an **admin-configurable, per-feature and per-field** audience +setting. This **replaces** the static `PUBLIC_KINDS` allowlist that used to be the whole boundary. +Policy lives in `utils/shardVisibility.js`; rows live in `shard_feature_visibility`; the admin surface +is `GET`/`PUT /admin/shard/visibility` (`adminOnly`). Admin-facing guide: +[`SHARD_VISIBILITY.md`](SHARD_VISIBILITY.md). Design: [`../link/v3.md`](../link/v3.md) §3. + +**The ladder.** `anonymous < logged_in < player < staff < admin`, each rung implying the ones below. +`viewerLevel(req)` resolves it: no session ⇒ `anonymous`; authenticated ⇒ `logged_in`; authenticated +with a linked game account ⇒ `player`; moderator ⇒ `staff`; admin ⇒ `admin`. **Staff satisfy the +`player` rung without a linked account** (consistent with `/player/*` being role-agnostic). +**`editor` gets no shard privilege** — it is a content role, and mapping it to `staff` would silently +widen what editors see. + +**Two invariants that are code, not configuration.** Both are enforced server-side and both reject +rather than silently ignore: + +1. **`acct` and `webId` are admin-only, always.** They are not exposed as configurable fields, and a + stored row attempting to loosen them is discarded on read as well as rejected on write. A character + name is visible in game; the account behind it and the website user it links to are not. + The lock is on the field's **meaning, not one spelling**: `isLockedField(key)` matches a key that + *is* or *ends in* `acct`/`webId`, case-insensitively, so the flattened forms the read models emit + (`shapeHouse` → `ownerAcct`, `shapeGuild` → `leaderWebId`) are covered too. An exact-key check was + the original implementation and it let `GET /public/shard/idoc` serve `ownerAcct` anonymously. +2. **A kind absent from `KIND_FEATURE` is never broadcast below `admin`.** Fail closed. This is what + keeps the kind map a security boundary rather than a convenience filter, and it means a shard that + starts emitting an unknown event degrades to staff-only, never to public. + +**Fail-closed everywhere else too.** An unreadable visibility config withholds every public frame; a +DB failure falls back to the compiled defaults (pre-3.0 behavior), not to open; an unresolvable viewer +subscribes as `anonymous`. The ladder comparison uses **asymmetric** fallbacks by design — an unknown +*viewer* level floors to the bottom rung and an unknown *requirement* ceils to admin, so an +unrecognised value loses on both sides. (A single shared fallback cannot do that: whichever direction +it picks, it fails open on one side.) + +**Three enforcement points, one config:** + +| Where | Mechanism | +|---|---| +| Routes | `requireFeature(name)` — **404** when the feature is disabled (don't leak that it exists), **403** when the caller is below its audience. `projectFeature` then strips out-of-rung fields from the body. | +| SSE (`utils/shardBroadcast.js`) | Per-connection filtering. A subscriber's rung is resolved **once at subscribe time and frozen** for that connection, so a long-lived stream can't gain privilege; each frame is then mapped kind→feature, gated, and field-projected per viewer. Two subscribers can legitimately receive different versions of one event, or one of them nothing. | +| Nav | `GET /public/shard/features` returns only what the caller may reach, so the SPA never renders a link that would 403. Presentation only. | + +Config reads are cached ~5s, so admin changes take effect within seconds **including on already-open +streams**. `PUBLIC_KINDS` still exists and is still exported (`notificationStreams.js`) but is now +**derived** from the kind map rather than hand-maintained, so the two cannot drift. + +**`PUBLIC_KINDS` is a module-load constant and must not be used to answer "may this caller read this +kind?"** — it is computed from the compiled *defaults*, so it cannot see an admin's changes. Use +`visibleKinds(level, config)`, which resolves against the live config. `/feed` uses it; it originally +used `PUBLIC_KINDS` and consequently kept serving `guild.join` to anonymous callers after an admin had +moved `guilds` to `staff`. `visibleKinds` deliberately ignores the `stream` flag: that governs SSE +fan-out only, so a feature whose live firehose ships off (market) stays readable from stored history. + +**Every read path that returns shard data must call `projectFeature`.** The stored-history endpoints +are not exempt — `/feed` returns the same events the stream does, and returning them unprojected +reopens on the REST side exactly what the stream closes. Relatedly, `shardEvents.db.list` treats an +**empty** `kinds` array as "serve nothing", never "no filter"; the fall-through it used to take would +have turned a fully-gated config into a dump of the entire event log. + +`projectFeature` walks **arrays and plain objects only**. A `Date`, `Buffer` or other class instance +is passed through as a value — rebuilding one key-by-key yields `{}`, which is the difference between +the pure-JSON wire frames and the DB-backed read models whose rows carry real `Date` columns. + +**Defaults reproduce pre-3.0 behavior exactly**, so installing the framework is a no-op until an admin +changes something — with deliberate exceptions, which are the leaks it was written to close. +`/public/shard/guilds`, `/public/shard/governors` and `/public/shard/feed` previously returned the raw +stored payload, whose actors carry `acct` and `webId`; `/public/shard/idoc` returned the flattened +`ownerAcct`. All are now stripped for every caller below admin. + --- ## 7. Email diff --git a/website/CLILOCS.md b/website/CLILOCS.md new file mode 100644 index 0000000..f2e1413 --- /dev/null +++ b/website/CLILOCS.md @@ -0,0 +1,309 @@ +# Cliloc table (item and title names) + +**Status:** Complete on `edge` — website [#115](https://gitea.whitlocktech.com/RunicGateway/website/pulls/115), docs [#70](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/70). +**Design:** [`docs/link/v3.md` §8.6](../link/v3.md) — Protocol 3.0, the dependency Part B/3 was sequenced behind. + +A "cliloc" is UO's localization table: an integer id mapped to a display string. +**Items on the wire carry a `LabelNumber`, not a name.** The bridge has always +sent that number — `char.profile.equipment` has a `cliloc` field, reward titles +arrive as a cliloc number in string form, and every marketplace listing carries +one — but the site had no table to look it up in, so a character sheet could only +render `id 1023721` where the game renders **"quarter staff"**. + +The number was never the missing piece. The table was. + +## Why the operator has to convert the file + +This is the awkward part, and it is not avoidable: + +**Every current UO client ships its cliloc files compressed.** All eight +`Cliloc.*` files in a modern client (`chs`, `cht`, `deu`, `enu`, `esp`, `fra`, +`jpn`, `kor`) begin with a DWORD whose high byte is `0x8E` — the "Mythic" +compressed container. The plain layout this site parses is what those files +looked like *before* that change. + +Decompressing it means an inverse-BWT coder with a frequency header — a few +hundred lines of bit-level work whose failure mode is plausible-looking garbage +rather than an error. The site has no business carrying that at runtime. + +Two facts make the alternatives worse, not better: + +- **ServUO cannot read it either.** Its bundled `Ultima.StringList` implements + only the plain layout, so on a modern client `VendorSearch.StringList` is null + and `VendorSearch.GetItemName` returns `item.Name` — usually nothing. The + shard cannot supply names on our behalf; the in-game Vendor Search gump has the + same gap. +- **Nothing client-derived may be committed.** UO's strings are EA's. The repo + ships no string table for the same reason it ships no artwork and no map + snapshot — see [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md). + +So the conversion happens **once, on the operator's machine, against their own +client**, and the site reads the result from a path it is given. A shard that +never does this is in a fully supported state: names render as ids, exactly as +they did before the table existed. + +## Converting + +> **Step-by-step operator instructions — where to get UOFiddler, where your +> client files are, and how to verify the import — are in +> [`UOFIDDLER.md`](UOFIDDLER.md).** This section covers the formats and the +> reasoning behind them. + +Either format below is accepted; the site sniffs which one it was handed. + +| Format | Fidelity | Notes | +|---|---|---| +| **Plain binary** (recommended) | Exact | 6-byte header, then `{int32 number, byte flag, uint16 length, UTF-8}` records | +| Delimited text | Loses leading/trailing whitespace | `numbertext` per line; a header row, blank lines and `#` comments are ignored | + +The whitespace caveat is real but cosmetic: ~1,300 of the 123,490 entries in a +stock `Cliloc.enu` are label prefixes like `"max = "` whose trailing space is +meaningful when the client concatenates a value onto them. Nothing on this site +concatenates, and every consumer passes through `displayText()`, which trims. + +### Using the bundled tool + +`server/tools/cliloc-export/` is a small .NET console app that drives +[UOFiddler](https://github.com/polserver/UOFiddler)'s `Ultima.dll` — the +decompressor that already exists and is already maintained — and writes the plain +format. It loads that DLL **reflectively** so it compiles against any SDK, and it +writes the records by hand because UOFiddler's own `SaveStringList` *re-compresses* +on save (its purpose is round-tripping a file back into the client, so its output +is byte-identical to its input — a trap worth knowing about). + +```bash +cd website/server/tools/cliloc-export +dotnet build -c Release + +# binary (recommended) +dotnet run -- "/Ultima.dll" "/Cliloc.enu" /srv/uo-data/clilocs.plain + +# or tab-delimited +dotnet run -- "/Ultima.dll" "/Cliloc.enu" /srv/uo-data/clilocs.tsv --tsv +``` + +A UOFiddler GUI export works too, but **not unmodified**: its Cliloc tab writes +`Number;Text;Flag` — three columns, the flag *last* — and the parser reads +`numbertext`, so the trailing field is absorbed into the name and +every item renders as `quarter staff;0`. Stripping it is one `sed`, given in +[`UOFIDDLER.md`](UOFIDDLER.md) §Route B. + +The parser already tolerates `number,flag,text`, with the flag in the *middle*. +It is not extended to cover the trailing form because a final `;0` is +indistinguishable from a name that genuinely ends that way — a heuristic there +would corrupt real names to save the operator one command. + +## Shard-added and shard-edited items + +**Shards edit items and add new ones**, and those carry cliloc ids no stock +client table has. The table is therefore built from a **set** of sources, all +re-read on every boot and hash-gated together — the same shape as the spawn +atlas, which reads `Regions.xml` + `Locations/*.xml` + `Spawns/*.xml` + +`ChampionSpawns.xml` and merges them: + +``` +/ + clilocs.plain ← base: the converted client table + custom/ + 01-uomysticmoon.tsv ← overlays: shard additions and overrides + 02-events.tsv +``` + +Overlays use the same delimited-text format, are read in **sorted order**, and +**later sources win** — so an overlay both *adds* ids the client never had and +*overrides* stock ones the shard has re-purposed. Any `.tsv`, `.csv`, `.txt`, +`.enu` or `.plain` file in `custom/` is picked up; anything else (a `README.md`, +say) is ignored. + +Adding, editing or removing any overlay counts as drift, so a new custom item +needs only a file edit and a restart — or the admin panel's Import button. +**Adding one item never means re-exporting a 5 MB client file.** + +The import result reports what each source contributed, which is how you confirm +an overlay took effect — `overrode: 0` on a file meant to re-label stock items +says it did not: + +```json +"sources": [ + { "label": "clilocs.plain", "kind": "base", "entries": 123490, "added": 123490, "overrode": 0 }, + { "label": "custom/uomysticmoon.tsv", "kind": "custom", "entries": 2, "added": 1, "overrode": 1 } +] +``` + +**Why a convention rather than discovery.** Everywhere else this pipeline follows +the shard's own files, but **ServUO has no server-side notion of a custom +cliloc** — they live in the patched client a shard distributes to its players, +and nothing in the tree declares them. There is nothing to discover, so `custom/` +is the one thing here that is our convention rather than the shard's. (An +operator who *does* patch their client cliloc needs no overlay at all: convert +the patched file and their edits are simply in the base.) + +Measured on the live shard for scale: its script tree references **16,434** cliloc +ids and only **37** are absent from the stock client table — tens of entries +against a 67k base, which is what makes an overlay the right shape rather than a +second full table. + +## Configuring the path + +Two ways to point at the sources, the setting winning over the environment: + +| Source | Notes | +|---|---| +| `cliloc_client_path` setting | Admin-editable (Admin → Shard); takes effect on the next refresh without a redeploy | +| `UO_CLIENT_PATH` env var | The deploy-time default, since the path usually describes a mount the deployment sets up | + +The value may be **the base file itself or a directory to search**, because both +are natural answers to "where is it". Overlays are read from a `custom/` +directory beside the base **either way** — pointing at a file does not forfeit +them. + +A directory is searched case-insensitively (the client writes `Cliloc.enu` on +Windows; the site usually runs on Linux) for, in order: `clilocs.tsv`, +`clilocs.csv`, `clilocs.plain`, `cliloc.plain`, `cliloc.plain.enu`, +`cliloc.enu.plain`, `clilocs.txt`, `cliloc.enu`. + +That ordering puts explicitly-converted names first on purpose. Pointing the +setting straight at an unconverted client directory finds `cliloc.enu`, which is +compressed — and the site says so by name rather than failing obscurely: + +``` +status: unavailable +code: COMPRESSED +reason: This is a compressed (Mythic-format) cliloc file, which the site cannot + read. Convert it to the plain format first — see docs/website/CLILOCS.md. +``` + +## Refresh contract + +Identical in shape to the spawn atlas, and for the same reasons: + +- **It never blocks startup.** No path, an unreadable file, a wrong-format file, + a database error — all caught and logged. The site comes up either way. +- **Hash-gated.** The boot path hashes the file and skips the parse entirely when + it matches what is loaded, which is every restart that did not follow a client + patch. Measured on a stock table: **14 ms** for the no-op, **663 ms** for a full + parse and replace. +- **A `PARSER_VERSION` bump also counts as drift**, so a corrected parse reaches + an install whose client never patches. + +### Two ways a refresh is refused + +**A corrupt file** — the realistic failure for any single source — makes the +parser fail on a truncated record rather than yield a plausible-but-short table, +so it is caught outright. Verified: a file truncated to half its length reports + +``` +code: TRUNCATED +reason: Truncated record header at byte 2486759 (74909 entries read) +``` + +and the rows already loaded are untouched. A malformed overlay names the file it +came from (`custom/broken.tsv: No cliloc entries found…`), because "which of my +six overlay files is broken" is otherwise a guessing game. + +**A source that has VANISHED** is the hazard a single file did not have. It +parses perfectly and imports a table quietly missing everything that file +contributed — and an unmounted volume looks exactly like a deliberate deletion +from here. This is the same ambiguity the atlas stages a facet removal for, so it +is escalated rather than applied: + +``` +status: needsReview +reason: 1 previously-loaded cliloc source(s) are missing; + the existing table is unchanged +missingSources: ["custom/uomysticmoon.tsv"] +``` + +`status()` reports `missingSources` too, so the panel can show it before anyone +clicks Import. An admin accepts it by re-running the import with +`{ "approve": true }`. + +**Why that is a flag and not the atlas's approve/reject pair.** The atlas stores +a pending decision in its own table so that approving *re-parses the tree*, which +is what keeps a multi-megabyte blob out of the database and makes the applied +result match the tree at approval time. Here nothing is stored, so re-reading at +approval time is automatic — the decision is a single boolean on the import an +admin was already going to run. + +## What gets stored + +| | | +|---|---| +| Parsed from a stock `Cliloc.enu` | **123,490** entries | +| Of those, empty strings | **55,994** (ids the client reserves and never uses) | +| Stored in `shard_clilocs` | **67,496** | + +Blank entries are dropped at import. A row resolving to no name is +indistinguishable from no row at all to every caller, and dropping them makes the +binary and text imports converge on **identical** content — the binary format +carries the blanks explicitly and a text export may or may not, depending on the +tool. Verified: both formats import to the same 67,496 rows with the same keys. + +`text` is `TEXT`, not `VARCHAR`: the long property descriptions reach 12 KB, and +silently truncating them would be worse than storing them. The index that matters +for marketplace search is on the denormalized `shard_vendor_items.display_name`, +not here. + +## How names are applied + +**Resolution happens server-side.** The table is never served *as* a table and +there is no public route for it. Two reasons: 67k rows would dwarf any page that +used them, and the Android client consumes the same JSON and would otherwise need +its own copy. + +`resolveMany()` takes a batch of ids and returns a `Map` holding only those that +resolved to something displayable, so "no such id" and "id with no usable name" +collapse into one branch at the call site. It never throws — a cliloc lookup is +decoration on someone's character sheet, and a database blip must not fail the +sheet. A capped in-process cache fronts it; measured cold **4.2 ms**, warm +**0.015 ms**. + +### `displayText()` + +Cliloc strings interpolate arguments the client pulls from an item's property +list — `~1_val~`, `~2_NAME~`. **We never have those**: the bridge sends the id, +not the packet. So a name carrying them is reduced to what is actually knowable. + +| Raw | Displayed | +|---|---| +| `quarter staff` | `quarter staff` | +| `cold damage ~1_val~%` | `cold damage` | +| `[~1_stuff~]` | *(nothing — the whole string was the argument)* | +| `50%` | `50%` | +| `Runic Gateway Sigil (v2)` | `Runic Gateway Sigil (v2)` | + +**Punctuation is only tidied when a placeholder was actually removed.** The +trailing `%` in row two is the unit belonging to the number we never had, and the +brackets in row three only ever wrapped the argument — but a string with no +placeholder has no such debris, and trimming it anyway corrupts real names. Rows +four and five are the ones that caught it: a shard's custom +`"Runic Gateway Sigil (v2)"` rendered as `"(v2"` while the bracket trim was +unconditional. + +### Consumers + +- **Character sheet equipment.** `enrichCharProfile` attaches `clilocName` to each + item. A player-given `name` always wins — "Bob's lucky axe" must not be + relabelled "hatchet" — and the client re-states that precedence. +- **Reward titles.** `titles.rewardResolved` is a parallel array with the numeric + entries turned into words (`null` where nothing resolved). The sheet used to + *skip* numeric reward titles entirely, having no way to render them. +- **Marketplace listings** (Protocol 3.0 §8) denormalize the resolved name into + `shard_vendor_items.display_name` so search can index it. + +## Admin surface + +All admin-only, alongside the atlas under Admin → Shard: + +| Route | Purpose | +|---|---| +| `GET /api/v1/admin/shard/clilocs` | Sources found, what each contributed at the last import, readability, drift, entry count, `missingSources` | +| `POST /api/v1/admin/shard/clilocs/import` | Reload after a client patch or an overlay edit; `{ "force": true }` reimports an unchanged set, `{ "approve": true }` accepts a vanished source | +| `PUT /api/v1/admin/shard/clilocs/path` | Set the path; blank disables resolution | + +A refresh **result is not an exception**: a missing file, or the likely mistake of +pointing at the client's own compressed `Cliloc.enu`, answers `200` with +`status: "unavailable"` and a reason. A `500` would say only "something broke"; +the operator needs to be told which file to convert. Setting the path +deliberately does **not** import as a side effect — the response carries the +refreshed status so the panel can offer that as the next step. diff --git a/website/MARKETPLACE.md b/website/MARKETPLACE.md new file mode 100644 index 0000000..5212b9f --- /dev/null +++ b/website/MARKETPLACE.md @@ -0,0 +1,167 @@ +# Marketplace — the player-vendor index + +**Status:** On `edge` — servuo-plugins [#5](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/5), link [#19](https://gitea.whitlocktech.com/RunicGateway/link/pulls/19), website [#116](https://gitea.whitlocktech.com/RunicGateway/website/pulls/116). +**Design:** [`docs/link/v3.md` §8](../link/v3.md) — Protocol 3.0 Part B/3. +**Depends on:** [`CLILOCS.md`](CLILOCS.md) — without a cliloc table, listings render as item ids. + +The marketplace is a searchable index of every player vendor on the shard: what +each shop is selling, for how much, and where it is standing. It is the same set +the in-game **Vendor Search** gump reads, offered from outside the game — so a +player can find the vanquishing kryss they want before logging in, and someone +who does not play at all can see that the economy exists. + +Page: `/site/market`, plus `/site/market/vendors/:serial` for one shop. + +## Three things the pages must say out loud + +Everything below follows from how the data is gathered, and each has a visible +consequence the UI is required to surface. + +**1. The prices are not live.** The shard sweeps vendors **round-robin** — at +most `Bridge.MarketSweepBatch` shops per tick — so a given shop can be a full +cycle behind. The page carries a *"prices last refreshed N minutes ago"* banner +driven by the **oldest** vendor row, not the newest: the one stale shop is the +one that wastes somebody's trip. + +**2. A shop can be truncated.** `Bridge.MarketMaxListings` (250 by default) caps +how many listings one frame carries. A commodity reseller with thousands of +stacked resources is a real thing, and an uncapped frame for one is measured in +megabytes. Over the cap the shop reports `truncated`, and the vendor page says +*"showing 250 of 3,104 — this shop holds more than the shard publishes"* rather +than presenting a partial shop as complete. + +**3. An item may have no name.** Items on the wire carry a cliloc id, not a name. +On a shard whose operator has not converted a cliloc table +([`CLILOCS.md`](CLILOCS.md)) the honest render is the item id — never an invented +label, which would be indistinguishable from a real one. + +## Privacy: the player's own toggle wins + +Only vendors whose owner left the in-game **Vendor Search** flag ON are ever sent +to the site. A player who hides their shop in game is hidden here too, and no +admin setting overrides that. When they hide one that was already indexed, the +shard emits `vendor.listing.remove` and the row is deleted — so revoking consent +takes effect, it does not merely stop refreshing. + +Shop name, owner character name and location default to **Everyone**, because the +stock Vendor Search gump already shows exactly that set to any player in game. +They remain admin-configurable; see [`SHARD_VISIBILITY.md`](SHARD_VISIBILITY.md). +Account names and website user ids never cross the wire at all. + +## How it is put together + +``` +ServUO uo-link sidecar website +────── ─────────────── ─────── +BridgeMarket.cs vendors table shard_vendors + round-robin sweep ──────► (whole frame blob) ──────► shard_vendor_items + per-vendor diff GET /market (paged) + display_name + vendor.listing resolved at ingest + vendor.listing.remove +``` + +**The shard side** walks at most `MarketSweepBatch` vendors per tick from a +persistent cursor, diffs each against what it last published, and emits a whole +frame for any shop that moved. Per-tick cost is therefore bounded by the batch, +not by how many vendors the world holds — full coverage takes +`ceil(vendors / batch) × MarketSweepSeconds`. + +**The sidecar** stores each frame whole and serves `GET /market`, its only paged +read. It normalizes nothing and defines no audiences: it is a dumb forwarder, and +search is the website's job. + +**The website** splits each frame into a vendor row and its listings, replacing +that vendor's whole listing set inside one transaction (the frame is +authoritative for that vendor, never a delta). Item names are resolved against +the cliloc table **on the way in** and stored denormalized, which is what makes +search-by-name possible and keeps the cliloc table off the hot path. + +## Operating it + +Everything is in `Config/Bridge.cfg` on the shard. There is nothing to configure +on the website. + +| Setting | Default | What it does | +|---|---|---| +| `MarketEnabled` | `true` | Master switch. Off publishes nothing; the page shows an empty index. | +| `MarketSweepSeconds` | `60` | Tick interval. | +| `MarketSweepBatch` | `25` | Vendors inventoried per tick. Clamped 1..500. | +| `MarketMaxListings` | `250` | Per-shop listing cap, after which `truncated`. Clamped 1..5000. | + +**Faster coverage vs. per-tick cost.** Lowering `MarketSweepSeconds` or raising +`MarketSweepBatch` both refresh the index sooner and both cost more per tick. +The expensive part is the item walk, which recurses into every container a vendor +is selling — so a shard of big shops should raise the interval rather than the +batch. + +`[bridge status` reports the sweep, including `lastMs` and `maxMs`: + +``` +market(enabled=True sweeps=42 scanned=108 emitted=27 removed=0 skipped=0 + truncated=0 tracked=27 vendors=27 cursor=2 batch=25 lastMs=0.31 maxMs=15.40) +``` + +A tick over **50 ms** prints a rate-limited warning naming the knob: + +``` +[Bridge] market sweep took 82.4 ms (budget 50 ms) - lower Bridge.MarketSweepBatch (now 25) if this persists +``` + +Measured on a shard with 27 vendors × 40 listings (209k items, 43k mobiles): +**15.4 ms** for the first cold tick of 25 vendors, **0.3 ms** in steady state — +the diff is what makes an unchanged world nearly free. Note the arithmetic: 25 +*full* shops at the 250-listing cap is 6,250 items ≈ 95 ms, over budget. Real +shops hold tens, which is why 25 is the default and why the warning exists. + +`[bridge sweepnow` runs one tick immediately; `[bridge reload` re-reads the +settings above without a restart. + +## Names arriving late + +Item names come from the cliloc table, and the market sweep will **not** re-send +an unchanged shop just because the site learned what its items are called. So a +cliloc import triggers a bulk re-resolution of every stored listing — otherwise +an operator who configures clilocs after the first sweep would see item ids until +every shop happened to change on its own. It runs after a boot import and after +an admin import, takes ~50 ms per thousand listings, and never throws: a failure +leaves names exactly as they were. + +## API + +All under `/api/v1/public/shard`, gated by the `market` feature and +**rate-limited** — these are the first genuinely expensive public reads on the +site (a `LIKE` scan plus a `COUNT` over what is typically the largest `shard_*` +table, reachable with no session). + +| Route | What | +|---|---| +| `GET /market` | Search. Returns **listings**, not vendors — "who sells X and for how much" is the question. `?q=&minPrice=&maxPrice=&itemId=&map=®ion=&sort=&limit=&offset=`, `sort ∈ {price_asc, price_desc, recent}`. | +| `GET /market/meta` | Index size, staleness, and which facets and regions actually hold vendors — so a client builds its filters without running a search it will discard. | +| `GET /market/vendors/:serial` | One shop and its listings. **404** for a serial the index has never seen, which also covers a vendor since dismissed or hidden — to an anonymous caller those are the same answer. | + +`q` matches the resolved display name **or** the item's own literal name, because +an item with a player-set name (most of what is worth searching for on a +player-run shard) may carry a generic cliloc. `%` and `_` in a query are escaped: +they are `LIKE` metacharacters, not SQL ones, so parameterization alone would let +a search for `%` match every listing on the shard. + +Full schemas are in the OpenAPI spec (`ShardMarketPage`, `ShardMarketVendor`, +`ShardMarketMeta`, `ShardMarketListing`, `ShardMarketLocation`). + +## Tables + +`shard_vendors` (one row per shop) and `shard_vendor_items` (one row per priced +listing). Both are ingest-owned; nothing else writes to them. No foreign keys, +in keeping with every other `shard_*` table — the ingest transaction is what +keeps them consistent, and an FK would turn a malformed frame into a failed write +rather than a dropped row. + +There is deliberately **no `payload` column** on `shard_vendors`, unlike the +points board next door. The board's top-N is a fixed-size list read whole, so it +lives in JSON; here the items *are* the searchable rows, so they are normalized +and there is nothing left worth duplicating. The sidecar keeps the whole blob, +because outage resilience is its job. + +`shard_vendor_items.display_name` is denormalized and indexed (alone, and +composite with `price` for "cheapest matching X"). See "Names arriving late" +above for how it is kept current. diff --git a/website/SHARD_VISIBILITY.md b/website/SHARD_VISIBILITY.md new file mode 100644 index 0000000..31254f9 --- /dev/null +++ b/website/SHARD_VISIBILITY.md @@ -0,0 +1,158 @@ +# Shard visibility — who sees which shard data + +**Status:** Built (Protocol 3.0 Part A). Admin → Shard Visibility. +**Audience:** shard owners and admins. +**Companion to** [`../link/v3.md`](../link/v3.md) §3 (the design) and +[`BACKEND_DESIGN.md`](BACKEND_DESIGN.md) §6 (the security contract). + +The website surfaces a lot of live shard data. What your players, your staff and the anonymous +internet may each see is **yours to decide**, per feature, from Admin → Shard Visibility. + +Nothing changes until you change it: every setting ships at the value that reproduces how the site +behaved before this panel existed. + +--- + +## 1. The audience ladder + +Five rungs. Each one includes everyone below it. + +| Rung | In the UI | Who that is | +|---|---|---| +| `anonymous` | **Everyone** | Anyone at all, signed in or not. | +| `logged_in` | **Signed in** | Any registered account, whether or not they've linked a game account. | +| `player` | **Linked players** | Accounts with a linked in-game account. **Staff always qualify**, linked or not. | +| `staff` | **Staff** | Admins and moderators. | +| `admin` | **Admins only** | Admins. | + +Two notes that surprise people: + +- **`editor` is a content role, not a shard role.** Editors write news and wiki pages; they get no + shard privilege from that. An editor is treated by link status like any other member. This matches + the rest of the site, where shard staff powers are admin-or-moderator. +- **Staff satisfy `player` without linking.** Otherwise an admin would be locked out of surfaces + they'd gated to players, which is how the `/player/*` routes already behave. + +## 2. What you can set per feature + +**Enabled.** Off means gone. The feature's pages return “not found”, not “forbidden” — a disabled +feature doesn't advertise that it exists. + +**Who can see it.** The minimum rung, from the ladder above. + +**Live updates.** Whether this feature pushes changes to open pages in real time. Turning it off +doesn't break the page; it just refreshes on load instead of updating in place. + +**Sensitive fields.** Some features expose a field that deserves its own rung — you can publish the +board while holding back one column. See the table in §3. + +## 3. The features, and their defaults + +| Feature | What it exposes | Default | Sensitive fields | +|---|---|---|---| +| **Shard status** | Connection state, online count, gold-supply series | Everyone | — | +| **Activity feed** | Deaths, kills, skill gains, quests, logins | Everyone | — | +| **Champion spawns** | The live champion / mini-champ / sea-boss board | Everyone | — | +| **Guilds** | Guild rosters, alliances, leaders | Everyone | — | +| **Town governors** | City Loyalty governors, elections, term history | Everyone | — | +| **Houses / IDOC** | Houses in danger | Everyone | House owner → Staff · House price → Staff | +| **Players online** | Population aggregate, staff-online widget | Everyone | In-game location → Staff | +| **Shard rules** | Skill/stat caps, house limits, vet rewards, the ruleset | Everyone | Connect address → Everyone | +| **Spawn atlas** | Bestiary and spawn locations (static content) | Everyone | — | +| **Leaderboards** | Point and loyalty standings | Everyone | Character names → Everyone | +| **Marketplace** | The shard-wide player-vendor index | Everyone, **live updates off** | Vendor owner name → Everyone · Vendor owner character id → Everyone · In-game location → Everyone | + +**Why the marketplace ships with live updates off.** A live feed of every vendor's full inventory +would be the single largest thing the site sends. No page needs it — the marketplace is a search over +stored data with a “prices last refreshed N minutes ago” stamp. Turn it on only if you want it. + +**Why the marketplace's fields default to Everyone.** A vendor's shop name, its owner's character +name and where it is standing are *already* visible to every player in game: the stock Vendor Search +gump surfaces exactly that set to anyone who opens it. Publishing them on the site is not a new +disclosure. They stay configurable because a shard may still prefer to keep its economy behind a +login — and because "already public in game" is a judgement about your shard, not ours. + +**Location is one setting covering four things.** Hiding it removes the facet, the coordinates, the +region *and* the house name together. That is deliberate: those are four ways of saying the same +thing, and a setting that hid the coordinates while publishing the house name would not have hidden +anything. + +**Hiding the owner name also hides the owner character id.** They are separate settings so you can +be explicit, but leaving the id published while hiding the name achieves nothing — the leaderboards +and guild boards resolve that same id back to a character name. Set both. + +**What hiding a vendor cannot do.** Only vendors whose owner left the in-game *Vendor Search* flag ON +are ever sent to the site, so a player who hides their shop in game is hidden here too — and no +setting on this page can override that. It works the other way as well: these settings control who +sees the index, not whether players can find each other's shops in game. + +**Why house owner/price default to Staff.** The public Houses page has always been a "where are the +falling houses" board — location only. Owner and price are the staff view. That split is preserved. + +## 4. What you cannot change + +Two rules are enforced in code and are not settings. Attempting to set them returns an error rather +than silently ignoring you. + +**1. Game account names and website user ids are admin-only, always.** +`acct` and `webId` never appear below the admin rung on any surface. A character *name* is visible in +game to anyone standing next to them; the **account** behind it is not, and neither is the website +user it's linked to. Publishing those would disclose something the shard itself doesn't, and would +tie a player's in-game identity to their forum identity without their consent. + +This rule matches the *meaning* of a field, not one spelling of it. Some responses nest the player +who owns a record (`leader.acct`); others flatten it into the row (`ownerAcct`, `leaderWebId`, +`governorAcct`). Every one of those is locked, and the admin API refuses to configure any of them — +so a new response shape can't quietly reopen the hole by naming the field differently. + +**2. Unknown event kinds are never broadcast below admin.** +The live stream maps each event kind to a feature. A kind with no mapping — a new event from a shard +plugin the site doesn't know yet, say — goes to admins only. It fails closed. This is what keeps the +stream safe by default when the shard starts sending something new: the worst case is that staff see +it and players don't, never the reverse. + +## 5. How it's enforced + +Three places, one config: + +- **Page and API requests** are checked before the handler runs, and the response is then stripped of + any field above the caller's rung. +- **The live stream** resolves a viewer's rung once, when they connect, and freezes it for that + connection — a long-open page can't gain privilege because something changed underneath it. Each + event is then gated and stripped per viewer, so two people watching the same page can legitimately + receive different versions of the same event, or one of them nothing. +- **Navigation** hides links a viewer can't follow, so they don't hit a wall. This is presentation + only — the gate is server-side either way. + +**Stored history answers the same way the live stream does.** The activity feed reads from the event +log rather than the live stream, but it resolves the *same* question against the *same* config: which +kinds you may read, and which fields survive. So moving a feature up a rung hides it from the history +as well as the stream — there is no back door where yesterday's copy of an event is more revealing +than today's. + +One deliberate asymmetry: turning **live updates** off for a feature stops the push, not the reading. +The marketplace ships this way — its history and its pages are public, only the firehose is off. + +Changes take effect within about five seconds, **including on streams that are already open**. You +don't need to restart anything. + +If the database is briefly unreachable, the site falls back to the built-in defaults — the pre-v3 +behavior — rather than to "everything is public". + +## 6. Worked examples + +**"I want a private shard — nothing public until people register."** +Set every feature to **Signed in**. Anonymous visitors still get the site itself; the shard data +disappears from the nav. + +**"Publish the market, but don't tie vendors to players."** +Marketplace → Everyone, with **Vendor owner name** → Staff. Prices, items and locations stay public; +who owns each vendor doesn't. + +**"Leaderboards for members only."** +Leaderboards → **Linked players**. Anyone who's linked a game account sees the standings; drive-by +visitors don't. + +**"Let players see house owners."** +Houses → Everyone, **House owner** → Linked players. Note this is a real disclosure: house ownership +is visible in game, but the website makes it searchable in a way the game doesn't. diff --git a/website/SPAWN_ATLAS.md b/website/SPAWN_ATLAS.md new file mode 100644 index 0000000..343ad62 --- /dev/null +++ b/website/SPAWN_ATLAS.md @@ -0,0 +1,355 @@ +# Spawn atlas + +**Status:** Complete on `edge` — data pipeline in website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112), API + pages in website [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113). +**Design:** [`docs/link/v3.md` §6](../link/v3.md) — Protocol 3.0 Part C. + +The spawn atlas is a browsable catalogue of what the shard *contains*: which +creatures spawn, where, how many, and which champion altars are configured. It +answers "where do I find a lizardman?" with **"Shrines, Yew, Isamu-Jima"** rather +than with a list of raw coordinates. + +## Two things that shape the whole design + +**The shard's ServUO tree is the single source of truth.** Nothing is +precomputed and committed to the repository. A shard's maps change over its +lifetime — facets get added, replaced, or renamed — and a snapshot in the repo +would silently drift from the world players actually see. The atlas is therefore +re-derived from the tree **on every server boot**. + +**Facets are not a fixed list.** Nothing in the codebase names Felucca, Trammel, +or any other stock facet. The facet set is whatever the shard's own files +declare, discovered at parse time. A shard running entirely custom maps gets +exactly the same treatment as a stock one, with no code change. + +## What it is not + +The atlas is **static shard content, not live shard state.** + +- It does **not** come from the sidecar. Nothing here touches the bridge, and + there is no event kind, no wire change and no `PROTOCOL_VERSION` bump for it. + Part C is website-only. +- It stays fully populated while the shard is down. +- Its champion table (`shard_champion_spawns`) is the *configured roster* — + "there is an Unholy Terror altar in Deceit". The live `champ.update` feed in + `shard_champs` is the separate, sidecar-fed answer to "it is on level 3 right + now". Both exist; do not conflate them. + +Routes live at `/api/v1/public/atlas`, deliberately **not** under `/shard`, +because `/shard/*` means sidecar-dependent. + +## Configuring the tree + +The website needs to be able to *read* the ServUO tree — same host, a bind mount, +or a shared volume. Two ways to point at it, the setting winning over the +environment: + +| Source | Notes | +|---|---| +| `spawn_atlas_servuo_path` setting | Admin-editable; changes take effect on the next refresh without a redeploy | +| `SERVUO_PATH` env var | The deploy-time default, since the path usually describes a mount the deployment sets up | + +With neither set the atlas is simply skipped — the site runs normally without +one. + +## The boot path + +On every start the server hashes the source files and compares them against what +is loaded. Unchanged (the normal case on a restart) costs one read pass, ~120 ms, +and no database write. A real change costs a ~400 ms parse and a reload. + +Two contracts govern it: + +**1. It never blocks startup.** No configured path, an unreadable mount, a +malformed file, a database error — every one is caught and logged, and the site +comes up serving whatever atlas it already had. + +**2. A facet disappearing is never applied automatically.** Losing a facet looks +exactly like a half-copied or mid-update tree, and boot cannot tell that apart +from a real map change. That refresh is *staged* for a human instead. Everything +else — new facets, renamed regions, changed spawns — applies immediately, since +none of it can destroy something an operator would miss. + +``` +boot + └─ path configured? no ──▶ skip + └─ tree readable? no ──▶ warn, carry on + └─ hashes changed? no ──▶ done (nothing parsed) + └─ parse + └─ a facet would be removed? + no ──▶ import + yes ──▶ stage for admin review; atlas unchanged +``` + +### Approving or rejecting a staged refresh + +Only the *decision* is stored, never the parsed world — a few KB of source hashes +plus the facet diff. Approving **re-parses** the tree, so what lands matches the +tree at approval time rather than at boot, and a multi-megabyte blob never sits +in the database. + +A rejection is remembered against those exact source hashes, so a declined +refresh does not re-prompt on every restart. Change the tree and the hashes +differ, which asks again. + +From **Admin → Spawn Atlas**, or from the CLI: + +```bash +cd website/server +npm run atlas:import -- --status # what is loaded, and what is pending +npm run atlas:import -- --approve # apply the staged refresh +npm run atlas:import -- --reject # keep the current atlas, dismiss it +``` + +## The CLI + +The server refreshes itself on boot, so this is for applying a map change +*without* a restart, and for the approve/reject flow above. + +```bash +npm run atlas:import # import if the tree differs +npm run atlas:import -- --servuo # override the path for this run +npm run atlas:import -- --force # reimport even if unchanged +``` + +`--servuo` is a per-run override and deliberately does **not** persist — changing +where the atlas permanently reads from is an admin action, not a side effect of a +one-off import. + +## Sources + +| File | Count (stock ServUO 57.4) | Used for | +|---|---|---| +| `Spawns/*.xml` | 13 files, ~10.5 MB | Every spawner: location, size, delays, time-of-day, creature types | +| `Data/Regions.xml` | 129 KB, nested | Named regions and their rectangles | +| `Data/Locations/*.xml` | 6 files | Landmarks (dungeon levels, town markers) | +| `Config/ChampionSpawns.xml` | 4.8 KB | Configured champion altars | + +**A stock tree has 13 spawn files but only 6 facets.** `Eodon.xml`, +`GravewaterLake.xml`, `TreasuresOfKotl.xml` and the other named-area files hold +TerMur/Trammel points. The facet always comes from each record's own ``, +never from the file name. + +## How a coordinate becomes a place name + +This is the transform the atlas exists for, in `resolveRegion()`: + +1. The highest-`priority` named region whose rectangle contains the point. Ties + break toward the **smallest** rect, so a specific room wins over the + dungeon-wide rect enclosing it. +2. Otherwise the nearest landmark within the landmark radius (200 tiles by + default), labelled by its **group** ("Covetous"), not its individual marker + ("Level 1"). +3. Otherwise `"Wilderness"`. + +The radius cap in step 2 is what keeps step 3 reachable. Without it the nearest +landmark is always *some* landmark however far away, and open countryside gets +labelled with a dungeon on the far side of the map. + +Against stock ServUO this resolves **83.2%** of points (5,369 of 6,455): 3,681 by +region, 1,688 by landmark, 1,086 Wilderness. + +## Three quirks in the source data + +Each of these is silent if unhandled — the atlas still builds, it is just wrong. + +**Facet names disagree between sources.** `Data/Locations/*.xml` spells them +`Ter Mur` and `Tokuno Islands`, while `` and `` say `TerMur` and +`Tokuno`. Unreconciled, the landmark bucket is keyed differently from the points +looking it up, so the fallback never fires and every unregioned spawn on those +facets reads "Wilderness". + +This is reconciled **by matching, not by a lookup table** — there is no list of +facet names anywhere. `facetKey()` collapses spelling differences (lowercase, +alphanumerics only), and `resolveFacetName()` matches a loose spelling against +the canonical set discovered from the shard's own spawn and region data, by exact +key then by prefix in either direction. A name matching nothing keeps its own +name: forcing a wrong match would file a real custom facet's landmarks under the +wrong facet, which is worse than leaving it alone. + +**Spawn type tokens carry XmlSpawner directives.** The `` type is not +always a bare class name: + +``` +Fairy,{RND,4,8} alchemist/z/-50 Agralem/Name/Agralem +GargishRouser,1 greatape,true GargishRefugee/hue/34532 +``` + +Taken literally these invent creatures that do not exist *and* split real ones in +two, because `Fairy` and `Fairy,{RND,4,8}` slug apart into separate entries. 71 of +845 were affected. Everything from the first `/` or `,` is stripped, leaving 800 +real creatures. + +**Case is inconsistent across files.** The same creature is `Lizardman` in one +file and `lizardman` in another. Slugging collapses them correctly, but the +display name is chosen deterministically — most common spelling wins, ties break +to the more capitalised form, then alphabetically — because otherwise it would +depend on file read order and change on an unrelated restart. + +## Tables + +All are **import-owned**: a refresh empties and reloads them in one transaction, +so a failed reload leaves the previous atlas intact rather than a half-loaded +world. Nothing else writes to them and nothing holds a foreign key to them — no +FKs at all, consistent with every other `shard_*` table. Full column listings in +[`BACKEND_DESIGN.md`](BACKEND_DESIGN.md). + +| Table | Rows (stock) | Notes | +|---|---|---| +| `shard_spawn_creatures` | 800 | `slug` PK; `total` = sum of each type's own max; nullable `art` | +| `shard_spawn_points` | 6,455 | `spawn_range`, since `range` is reserved in MariaDB | +| `shard_spawn_point_types` | 23,927 | The many-to-many; one spawner commonly carries six types | +| `shard_regions` | 387 | Flattened out of the nesting; `rects` JSON | +| `shard_landmarks` | 558 | `grp`, since `group` is reserved in SQL | +| `shard_champion_spawns` | 25 | Configured altars, not the live feed | +| `shard_atlas_meta` | 1 | Singleton; source hashes, for the change check | +| `shard_atlas_pending` | 0–1 | Singleton; a staged refresh awaiting admin review | + +`shard_spawn_creatures.name` carries a plain `INDEX`, deliberately **not +`FULLTEXT`**: ~800 rows makes a `LIKE` scan free, and FULLTEXT's minimum token +length would break searches for names like "orc". + +The reload uses `DELETE`, not `TRUNCATE` — `TRUNCATE` is DDL in MariaDB and would +implicitly commit, defeating the all-or-nothing guarantee. Point ids are assigned +explicitly rather than left to `AUTO_INCREMENT`, because the join rows need them +and `conn.batch()` reports no usable `insertId`. + +## Artwork — operator-supplied, never shipped + +**This project ships no creature art and no extraction tooling, and never will.** +UO sprites live in the operator's own client `.mul`/`.uop` files. They are the +operator's, not ours to redistribute. + +The atlas is fully functional as text. `shard_spawn_creatures.art` is nullable +and is NULL on every fresh import; pages render without images, which is the +normal and supported state, not a degraded one. + +An operator who wants art — step-by-step, with the UOFiddler side spelled out, in +[`UOFIDDLER.md`](UOFIDDLER.md) §Part 2: + +1. Extracts it from **their own** client files (UOFiddler, ClassicUO tooling, or + any art extractor). +2. Drops the images under `server/uploads/atlas/`. +3. Copies `server/db/data/spawnAtlas.art.example.json` to `spawnAtlas.art.json` + and maps creature slugs to file names. +4. Restarts, or runs `npm run atlas:import -- --force`. + +Both `spawnAtlas.art.json` and `server/uploads/` are gitignored, so neither the +map nor the images can be committed by accident. + +## Code layout + +| File | Role | +|---|---| +| `src/utils/spawnAtlasParse.js` | **Pure and fs-free** parsers, so CI covers them with no ServUO tree. Zero dependencies. | +| `src/utils/spawnAtlasSource.js` | The only thing that reads a ServUO tree; shared by the boot path and the CLI | +| `src/model/shardAtlas/shardAtlas.db.js` | The one-transaction replace | +| `src/model/shardAtlas/shardAtlas.model.js` | The refresh decision, staging, approve/reject | +| `scripts/importSpawnAtlas.js` | Thin CLI over the model | + +Parsing notes: + +- `Regions.xml`, `Locations/*.xml` and `ChampionSpawns.xml` genuinely nest, and + get a small hand-rolled **subset** tokenizer — elements, attributes, + self-closing tags, comments, the XML declaration, CDATA, and the five + predefined entities plus numeric refs. It is not a general-purpose XML parser + and must not be reused as one. +- The ~10.5 MB of `Spawns/*.xml` never touches that tokenizer. Those records are + flat, so they get a streaming regex sweep instead; a DOM would allocate a node + per element across ~40 fields on every record to keep 14 of them. **Do not put + the Points files through a DOM parser.** +- `` is `Type:MX=n:SB=…` segments joined by `:OBJ=`. Split on `:OBJ=` + *first* — a naive `split(':')` shreds it. A single Trammel point carries six + types. +- **Respawn delays are stored in two different units, per record.** XmlSpawner + writes `MinDelay`/`MaxDelay` in minutes, and switches to seconds only when a + spawner's delay does not divide into whole minutes — flagging that with + `DelayInSec` on the same record. A `5` therefore means five *minutes* on one + spawner and five *seconds* on the next, and both are plausible respawn times, + so a reader assuming either unit is silently wrong about the other. Stock + ServUO 57.4 has ~170 second-flagged spawners out of 6,455. The parser + normalises everything to **seconds**; the API and UI carry seconds throughout. + +### The parser version + +`spawnAtlasSource.js` exports `PARSER_VERSION`, stored in `shard_atlas_meta` +alongside the source hashes and bumped whenever the parser derives **different +data from identical files** — a fixed misreading, a new field, a changed unit. + +A refresh re-derives when the tree changed **or** the parser did. Hashing the +tree alone would be a trap: an install whose maps never change would keep serving +whatever an older build derived, indefinitely, and a deploy that corrects the +parse would never reach the data. A version mismatch counts as drift, so the +correction lands on the next boot without an operator having to know it happened. + +## The API + +Everything is served from MariaDB. Nothing on this path touches the sidecar, so +the pages stay complete while the shard is down — which is why the routes sit at +`/api/v1/public/atlas` and **not** under `/public/shard`, where a prefix means +"sidecar-dependent". Unlike `/shard/*`, they *are* `siteMode`-gated, like +`/posts` and `/wiki`: a bestiary is site content and follows site content's rules. + +Every route carries `requireFeature('atlas')` — **404** when an admin has +disabled the feature (its pages must not reveal that it exists) and **403** when +the caller sits below its configured audience. The default is `anonymous`, so the +gates are inert until an admin changes something. Responses are field-projected +like every other shard read; `atlas` declares no sensitive fields today, and the +projection call is there so the first one that does is covered by construction +rather than by a retrofit ([`v3.md` §3.6.1](../link/v3.md)). + +| Route | Answers | +|---|---| +| `GET /atlas/creatures?q=&facet=&limit=&offset=` | The bestiary, most numerous first, paginated with an unpaginated `total` | +| `GET /atlas/creatures/:slug?facet=&points=` | One creature: `places`, `spawners`, `alsoHere` | +| `GET /atlas/regions?facet=&q=` | Named regions and their rectangles | +| `GET /atlas/landmarks?facet=&q=` | Points of interest, labelled by `group` | +| `GET /atlas/champions?facet=` | The configured altar roster | +| `GET /atlas/meta` | Facets, counts and when the atlas was parsed | + +Two shapes worth knowing: + +- **`places` is the aggregate the atlas exists for.** "Lizardman → Shrines, + Isamu-Jima, Yew", grouped in SQL rather than by summing 6,455 point rows in + Node. `spawners` is the raw list underneath it, bounded, with + `spawnersTruncated` saying when it was cut. +- **`points` is a COUNT, `spawners` is the LIST.** The two are named apart + deliberately: the same key meaning a number on the search route and an array on + the detail route is the kind of thing a client only discovers in production. + +`GET /atlas/meta` reports the **game world only**. The ServUO path, the per-file +hashes and any pending refresh describe the operator's filesystem, and live on +the admin route instead. + +A facet is never validated against a list — nothing in the codebase names one. +`?facet=` is length-bounded and matched exactly, so an unknown name returns an +empty result rather than an error. The filter is an `EXISTS` over the points and +deliberately not a JSON path or `JSON_SEARCH` built from caller input: that +function treats `%` and `_` as wildcards, which would make `?facet=%` match +everything. + +## The admin panel + +**Admin → Spawn Atlas** (`/admin/shard-atlas`, admin-only — it reads a path on +the server's filesystem and replaces every atlas table, which is closer to a +deploy action than to moderation). + +| Route | Does | +|---|---| +| `GET /admin/shard/atlas` | Status: path, readable, drift, counts, facets, pending | +| `POST /admin/shard/atlas/import` | Import now; `{ force: true }` ignores the hash gate | +| `POST /admin/shard/atlas/approve` | Apply a staged refresh, facet loss and all | +| `POST /admin/shard/atlas/reject` | Keep the current atlas; remember the decision | +| `PUT /admin/shard/atlas/path` | Point the atlas at a different tree | + +Three behaviours that are deliberate: + +- **An unreadable tree is a 200, not a 500.** `refresh()` reports outcomes rather + than throwing, because the boot path must never be stopped by a bad tree, and + that contract is preserved at the API. The panel says *"The tree could not be + read: …"*; a 500 would say only that something broke. +- **Setting the path does not import.** Moving the mount and reloading the world + are separate decisions, and an operator fixing a typo should not have a + multi-thousand-row replace happen under them. The response carries fresh status + so the panel can offer the import as the next step. +- **Every action is written to the admin activity log** (`shard.atlas.import` / + `.approve` / `.reject` / `.path`). diff --git a/website/UOFIDDLER.md b/website/UOFIDDLER.md new file mode 100644 index 0000000..96d8e69 --- /dev/null +++ b/website/UOFIDDLER.md @@ -0,0 +1,282 @@ +# Extracting from your own UO client (UOFiddler) + +**Audience:** the shard operator, once, at setup time. +**Related:** [`CLILOCS.md`](CLILOCS.md) (why the cliloc conversion is unavoidable), +[`SPAWN_ATLAS.md`](SPAWN_ATLAS.md) (where creature art fits). + +Two features read data that **only exists inside a UO client**, and a UO client's +files are EA's, not ours to redistribute. So neither this repo nor any image we +publish can ship them — the operator extracts from **their own** client, once, +and points the site at the result. + +| Feature | What it needs | Required? | Without it | +|---|---|---|---| +| **Item / title names** ([`CLILOCS.md`](CLILOCS.md)) | `Cliloc.enu`, converted | No | Names render as raw ids — `id 1023721` instead of *quarter staff* | +| **Creature art** ([`SPAWN_ATLAS.md`](SPAWN_ATLAS.md)) | Sprites from `.mul`/`.uop` | No | Atlas pages render as text, which is the normal state | + +**Both are optional and neither is load-bearing.** A shard that never does any of +this is fully supported. Do part one and skip part two if art is not worth your +time — they share only the tool. + +Everything you extract stays **outside the repository**: the converted cliloc +file lives at a path you choose, and `spawnAtlas.art.json` plus `server/uploads/` +are gitignored, so none of it can be committed by accident. + +--- + +## Part 0 — Get UOFiddler + +[UOFiddler](https://github.com/polserver/UOFiddler) is the community client-file +editor. We use it because its `Ultima.dll` already contains the cliloc +decompressor, maintained by people who do this for a living. + +1. Download the latest release zip from + — one asset, named + `UOFiddler-.zip` (4.22.2 is ~2 MB). +2. Extract it. The zip contains a single top-level folder, and the two files that + matter are at **its root**: + + ``` + UOFiddler-4.22.2/ + Ultima.dll ← the decompressor (Part 1 needs this path) + UoFiddler.exe ← the GUI (Part 2 needs this) + plugins/ + … + ``` + +3. **Runtime:** UOFiddler 4.22.2 is built for **.NET 10**. Running `UoFiddler.exe` + needs the .NET 10 **Desktop** Runtime (Windows only); loading `Ultima.dll` from + the converter in Part 1 needs the .NET 10 runtime. Install from + . + +### Finding your client files + +The cliloc file is in your **UO client installation directory**, not in your +ServUO tree — the shard server has no copy of it. Look for `Cliloc.enu` (English; +the other seven are `chs`, `cht`, `deu`, `esp`, `fra`, `jpn`, `kor`) beside +`art.mul` / `artLegacyMUL.uop`. The EA Classic Client's default location is: + +``` +C:\Program Files (x86)\Electronic Arts\Ultima Online Classic\ +``` + +**If your shard distributes its own patched client to players, use that copy.** +Any cliloc edits you shipped to players are then already in the base table and +you need no overlay for them (see [`CLILOCS.md`](CLILOCS.md) §Shard-added and +shard-edited items). + +--- + +## Part 1 — Convert the cliloc table + +**Goal:** turn the client's compressed `Cliloc.enu` into a file the site can +read, and point the site at it. + +The site cannot read `Cliloc.enu` directly. Every modern client compresses it +(the "Mythic" container), and so does ServUO's own bundled `Ultima.StringList` — +which is why the shard cannot supply names on our behalf either. The full +reasoning is in [`CLILOCS.md`](CLILOCS.md) §Why the operator has to convert the +file; this section is just the procedure. + +Two routes. **The bundled tool is the recommended one** — the GUI export needs a +fixup step, described below. + +### Route A — the bundled converter (recommended) + +Needs a .NET SDK (any version 8 or newer — the project targets `net8.0` and rolls +forward, so whatever you have works) **plus** the .NET 10 runtime from Part 0, +which is what actually loads `Ultima.dll`. + +```bash +cd website/server/tools/cliloc-export +dotnet build -c Release + +# plain binary — recommended, exact +dotnet run -c Release -- \ + "/path/to/UOFiddler-4.22.2/Ultima.dll" \ + "/path/to/UO client/Cliloc.enu" \ + /srv/uo-data/clilocs.plain + +# or tab-delimited text, if you want to eyeball or hand-edit it +dotnet run -c Release -- \ + "/path/to/UOFiddler-4.22.2/Ultima.dll" \ + "/path/to/UO client/Cliloc.enu" \ + /srv/uo-data/clilocs.tsv --tsv +``` + +Expected output for a stock English client: + +``` +wrote 123490 entries to /srv/uo-data/clilocs.plain (maxTextBytes=12150, skippedOversize=0) +``` + +**Sanity-check that number.** A stock `Cliloc.enu` is ~123,000 entries. A few +hundred means it read something else and you should not ship the result. The +tool exits non-zero and says `no entries were written — is that a cliloc file?` +when it gets nothing at all. + +The conversion runs on whatever machine has the client (usually Windows), and the +site reads the output wherever it runs — so **copy the output file to the server** +if those are different machines. It is a single self-contained file (~5 MB); the +`--tsv` form is larger but diff-able. + +
+Errors you may hit + +| Message | Cause | +|---|---| +| `Ultima.StringList not found — is that really UOFiddler's Ultima.dll?` | First argument points at some other `Ultima.dll` (ServUO ships one too — it is **not** the same assembly and cannot do this) | +| `You must install .NET to run this application` | Missing the .NET 10 runtime from Part 0 step 3 | +| `Unexpected Ultima.StringList API` | UOFiddler older than 4.21 | +| `usage: clilocexport …` | Fewer than three arguments | + +
+ +### Route B — the UOFiddler GUI + +Use this if you would rather not install a .NET SDK. **It needs one extra step**, +so do not skip the fixup. + +1. Launch `UoFiddler.exe` and point it at your client directory when it asks + (or **Options → Path Settings**). +2. Open the **Cliloc** tab and use its **export to CSV** action. +3. It writes `CliLoc.csv` to UOFiddler's configured output path, in **three** + columns with a header row: + + ``` + Number;Text;Flag + 1023721;quarter staff;0 + ``` + +4. **Strip the trailing flag column.** The site's text parser reads + `numbertext`, so that third field is otherwise absorbed into the name + and every item on the site renders as `quarter staff;0`. + + ```bash + sed -E 's/;[0-9]+$//' CliLoc.csv > clilocs.csv + ``` + + ```powershell + Get-Content CliLoc.csv | + ForEach-Object { $_ -replace ';\d+$','' } | + Set-Content -Encoding utf8 clilocs.csv + ``` + + The header row needs no removal — a line whose first field is not an integer + is skipped. Blank entries (`1005008;`) survive the fixup correctly and are + dropped at import, as intended. + +5. Copy `clilocs.csv` to the server. + +**Why the fixup is not just done for us:** the parser already handles +`number,flag,text` — the flag in the *middle*, which is what several exports +emit. UOFiddler puts it at the *end*, where it is indistinguishable from a name +that genuinely ends in `;0`. One `sed` on the operator's side beats a parser +heuristic that would corrupt real names. + +### Point the site at it + +Two ways, the setting winning over the environment: + +| Where | How | +|---|---| +| **Admin → Shard → cliloc path** | Takes effect on the next refresh, no redeploy | +| `UO_CLIENT_PATH` env var | The deploy-time default | + +The value may be **the file itself or a directory to search** — both are natural +answers to "where is it", and overlays are picked up either way. + +Setting the path deliberately does **not** import as a side effect. Click +**Import** (or `POST /api/v1/admin/shard/clilocs/import`) to load it. + +### Verify + +`GET /api/v1/admin/shard/clilocs`, or the Admin → Shard panel, reports what each +source contributed: + +```json +"sources": [ + { "label": "clilocs.plain", "kind": "base", "entries": 123490, "added": 123490, "overrode": 0 } +] +``` + +Roughly **67,500 rows stored** from a stock table is correct — about half a +cliloc table is empty strings for ids the client reserves and never uses. + +Then load any character sheet with equipment: items should show names rather than +`id 1023721`. + +
+What a refusal means + +A bad file answers `200` with a `status` and a named reason, not a `500` — you +need to be told *which file* to fix. + +| `code` | Meaning | +|---|---| +| `COMPRESSED` | You pointed at the raw client `Cliloc.enu`. Convert it — this whole page. | +| `TRUNCATED` | Half-copied file. Re-copy; the loaded table is untouched. | +| `EMPTY` | A text source with no parseable rows — the file is named in the reason. | +| `status: needsReview` + `missingSources` | A previously-loaded source has vanished (unmounted volume? deliberate deletion?). Nothing changes until you re-import with `{ "approve": true }`. | + +
+ +### Custom items — do *not* re-export for these + +Shard-added items carry ids no client table has. Drop a small delimited file in a +`custom/` directory beside the base file and re-import: + +``` +/srv/uo-data/ + clilocs.plain ← base, from this guide + custom/ + 01-uomysticmoon.tsv ← your additions and overrides +``` + +Files are read in sorted order and **later sources win**, so an overlay both adds +new ids and overrides stock ones you have re-purposed. **Adding one item never +means re-exporting a 5 MB client file.** Details in [`CLILOCS.md`](CLILOCS.md). + +--- + +## Part 2 — Creature art for the spawn atlas (optional) + +**Goal:** put sprites on atlas pages. Purely cosmetic — the atlas is fully +functional as text, and `art` is NULL on every fresh import. + +**This project ships no art and no art-extraction tooling, and never will.** + +1. In `UoFiddler.exe` (paths configured as in Route B step 1), open the + **Animations** tab for creature sprites — or **Items** for object art — find + the creature, and export as PNG. Right-click an entry for its export options, + or use the tab's *Export All* action for a batch. (4.22.2 added an export + option to the Animation tab's thumbnail list, which is the convenient one + here.) +2. Put the images under `server/uploads/atlas/`. +3. Copy `server/db/data/spawnAtlas.art.example.json` to `spawnAtlas.art.json` in + the same directory and map creature slugs to file names: + + ```json + { + "lizardman": "lizardman.png", + "orc": "orc.png" + } + ``` + + **Keys are the slugs the atlas API reports**, derived from the type names in + your own shard's `Spawns/*.xml` — read them off the atlas rather than guessing. + A creature with no entry renders without art, which is the default. + +4. Restart, or `npm run atlas:import -- --force`. + +The art map is re-read on every atlas refresh, so adding one image is an edit plus +a refresh. Both `spawnAtlas.art.json` and `server/uploads/` are gitignored. + +--- + +## Licensing, briefly + +UO's strings and sprites are EA's. Extracting from **your own** client for +**your own** shard is the arrangement here; redistributing the extracted files is +not something this project does or can advise on. That is the whole reason this +page exists instead of a download link. diff --git a/website/api-route-inventory.json b/website/api-route-inventory.json index 63a0207..f42fcb4 100644 --- a/website/api-route-inventory.json +++ b/website/api-route-inventory.json @@ -257,6 +257,26 @@ "method": "GET", "path": "/api/v1/admin/shard/accounts" }, + { + "method": "GET", + "path": "/api/v1/admin/shard/atlas" + }, + { + "method": "POST", + "path": "/api/v1/admin/shard/atlas/approve" + }, + { + "method": "POST", + "path": "/api/v1/admin/shard/atlas/import" + }, + { + "method": "PUT", + "path": "/api/v1/admin/shard/atlas/path" + }, + { + "method": "POST", + "path": "/api/v1/admin/shard/atlas/reject" + }, { "method": "GET", "path": "/api/v1/admin/shard/audit" @@ -313,6 +333,14 @@ "method": "GET", "path": "/api/v1/admin/shard/vendors/:account" }, + { + "method": "GET", + "path": "/api/v1/admin/shard/visibility" + }, + { + "method": "PUT", + "path": "/api/v1/admin/shard/visibility" + }, { "method": "PUT", "path": "/api/v1/admin/site-mode" @@ -705,6 +733,30 @@ "method": "GET", "path": "/api/v1/player/shard/vendors/:account" }, + { + "method": "GET", + "path": "/api/v1/public/atlas/champions" + }, + { + "method": "GET", + "path": "/api/v1/public/atlas/creatures" + }, + { + "method": "GET", + "path": "/api/v1/public/atlas/creatures/:slug" + }, + { + "method": "GET", + "path": "/api/v1/public/atlas/landmarks" + }, + { + "method": "GET", + "path": "/api/v1/public/atlas/meta" + }, + { + "method": "GET", + "path": "/api/v1/public/atlas/regions" + }, { "method": "POST", "path": "/api/v1/public/contact" @@ -737,6 +789,10 @@ "method": "GET", "path": "/api/v1/public/shard/economy" }, + { + "method": "GET", + "path": "/api/v1/public/shard/features" + }, { "method": "GET", "path": "/api/v1/public/shard/feed" @@ -769,6 +825,10 @@ "method": "GET", "path": "/api/v1/public/shard/presence" }, + { + "method": "GET", + "path": "/api/v1/public/shard/ruleset" + }, { "method": "GET", "path": "/api/v1/public/shard/status"