RunicGateway/docs
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: RunicGateway:10ae129b943a5f71caf93368b6833e27d8362526
Branches Tags
RunicGateway:main RunicGateway:docs/installer-plan RunicGateway:docs/android-v3-parity RunicGateway:docs/spawn-atlas RunicGateway:docs/ci-sonarqube RunicGateway:docs/android-m6-landed RunicGateway:chore/open-source-governance RunicGateway:feature/branding
...
compare: RunicGateway:docs/android-v3-parity
Branches Tags
RunicGateway:main RunicGateway:docs/installer-plan RunicGateway:docs/android-v3-parity RunicGateway:docs/spawn-atlas RunicGateway:docs/ci-sonarqube RunicGateway:docs/android-m6-landed RunicGateway:chore/open-source-governance RunicGateway:feature/branding

17 Commits
10ae129b94 ... docs/andro

Author SHA1 Message Date
wtclaude
f6a22734d3 docs(android): link the M11 app PRs and name the remaining gate 
Both parts are built: Android-app#30 (the visibility rules + read-model adds)
and #31 (the four screens, stacked on it). The on-device five-rung walk against
a website on the cutover branch is what edge->main is now actually waiting on.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-30 02:51:53 -05:00
wtclaude
5a091157d6 docs(android): scope M11 — Protocol 3.0 shard parity for the app 
The v3 work added four shard features and an admin-configurable visibility
framework the Android client knows nothing about. v3.md §10 deferred the app
side as a follow-up; re-examining it before the cutover found the gap is wider
than nav hiding:

  - no consumer for any of ruleset / leaderboards / market / atlas,
  - no `points` block on the character sheet (§7.3),
  - no cliloc-resolved item names (§8.6), and
  - shard nav gated on session role alone, so an admin who disables a feature
    or raises its audience leaves the app rendering entries that 404/403 into a
    generic error where the web client hides them.

Scoped as PLAN.md §9 M11 in two PRs (the visibility rules + read-model adds,
then the four screens), with the traps a real shard exposes recorded inline:
uncapped `maxPoints: 0`, cliloc-named boards with a null `nameString`, skill
caps in tenths, the required market staleness banner, the market stream being
off by default, atlas delays in seconds, and `points`-count vs `spawners`-list.

edge → main is held until both land so web and app surface the same shard on
the same day. Neither PR is coupled to the merge order — on a pre-v3 website
every new route and /public/shard/features 404s and the app falls back to
today's behavior — so holding the cutover is a schedule decision, not a
technical dependency.

Also records two things verified as already correct, so they are not
re-derived: the app's SSE request rides the authenticated client (same audience
rung as the same account on web), and every shard DTO is nullable-with-defaults
(field projection cannot cause a decode failure).

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-30 00:46:16 -05:00
Colby Whitlock
3a1bbdd165 Merge pull request 'docs(link): the Protocol 3.0 cutover (v3.md order 6)' (#72) from docs/protocol-3-cutover into edge 
Reviewed-on: #72
 2026-07-30 03:03:10 +00:00
wtclaude
32def88c4e docs(link): fill in the cutover PR numbers 
The order-6 row was written before the seven PRs existed. Same follow-up as the
cliloc row got.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-29 18:09:50 -05:00
wtclaude
71207cef16 docs(link): the Protocol 3.0 cutover (v3.md order 6) 
INTEGRATION.md was written for the window that just closed -- it told integrators
the version had NOT been bumped yet and that a sidecar on `edge` reports 2 while
already carrying v3 kinds. That guidance is now wrong in the direction that
matters, so the version section states 3 (header, /health, ws.hello, the 409
example and the §8 worked example) and replaces the "until then" paragraph with
what a v2 integration actually has to do to upgrade: change the constant it
sends, and nothing else, because nothing that existed in v2 changed shape.

v3.md gains §4.1 for what the bump touches and, more importantly, WHY the
website's boot migration is gated on a marker row: schema.sql is re-run on every
boot and uo_link_config.protocol is admin-editable, so an ungated UPDATE would
silently un-pin an operator running an older sidecar. That is the one piece of
the cutover a reader could not infer from the code being one constant.

Progress tables: 5b done, 6 in review.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-29 18:04:00 -05:00
Colby Whitlock
cdea1aa7cd Merge pull request 'docs(link): the player-vendor marketplace (Protocol 3.0 §8)' (#71) from docs/vendor-listing into edge 
Reviewed-on: #71
 2026-07-29 20:02:31 +00:00
wtclaude
6ce60a82c3 docs(link): the player-vendor marketplace (Protocol 3.0 §8) 
Documents order 5b across the four repos, and records what building it changed
about §8 as designed.

- NEW website/MARKETPLACE.md — the operator guide: what the pages must say out
  loud and why, the privacy contract (the player's in-game Vendor Search toggle
  wins, and no admin setting overrides it), the Bridge.cfg knobs and how they
  trade against each other, and the measured sweep costs.
- INTEGRATION.md — catalog entry for vendor.listing / vendor.listing.remove with
  its six consumer gotchas, and the GET /market REST section (the sidecar's only
  paged read, and why it orders by serial rather than shop name).
- BACKEND_DESIGN.md — shard_vendors / shard_vendor_items, the routes, and the
  marketplace search as the only rate-limited public read.
- SHARD_VISIBILITY.md — why the market's fields default to Everyone (the in-game
  gump already shows exactly that set), why location is one setting covering
  four things, and why hiding the owner name without the owner id achieves
  nothing.
- PLAN.md — the amortized round-robin as the one sweep pattern the bridge did not
  previously have, and an update to §7's cliloc note: pushing name resolution to
  the plugin was never an option, because ServUO cannot read a modern client's
  compressed cliloc files either.
- v3.md §8.8 — the four things the build settled differently, chief among them
  that §8.1's FLAT location payload would have made Part A's pre-wired
  market.location rule inert, exactly like the characterName miss one part
  earlier.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-29 09:52:14 -05:00
Colby Whitlock
70d49b7792 Merge pull request 'docs(link): the cliloc table, and why §8.6's recommendation was not implementable' (#70) from docs/cliloc-table into edge 
Reviewed-on: #70
 2026-07-29 11:58:26 +00:00
wtclaude
ee0c146d7a docs(link): cliloc overlays for shard-added and shard-edited items 
Shards edit items and add new ones, carrying cliloc ids no stock client table
has. The cliloc table now reads a SET of sources rather than one file — a base
plus every operator-maintained overlay under `custom/`, hash-gated together and
re-read on every boot, exactly as §6 reads the ServUO tree. Matching docs for
website #115.

website/CLILOCS.md gains a "Shard-added and shard-edited items" section: the
directory layout, merge precedence, the per-source breakdown an operator uses
to confirm an overlay took effect, and why `custom/` is a convention we chose
rather than one discovered from the shard — ServUO has no server-side notion of
a custom cliloc, they live in the patched client a shard distributes and
nothing in the tree declares them. Scale recorded: the live shard's script tree
references 16,434 cliloc ids and only 37 are absent from stock, which is why
this is an overlay and not a second table.

"Why there is no staged-approval flow" is replaced by "Two ways a refresh is
refused", because the set brings back the hazard a single file did not have. A
corrupt source fails the parse loudly; a source that has VANISHED parses
perfectly and imports a table quietly missing everything it contributed. That
is the same ambiguity §6 stages a facet removal for, so it is staged here too
(`needsReview`, `{approve:true}` to accept) — as a flag rather than §6's
approve/reject pair, because the atlas stores a pending decision SO THAT
approving re-parses, and here nothing is stored.

Two more traps recorded in §8.6 (now five), both found by running a shard-style
overlay rather than another stock-table fixture:

- Tidying punctuation unconditionally corrupts real names — a custom
  "Runic Gateway Sigil (v2)" rendered as "(v2". Stripping leftover brackets is
  right after a placeholder is removed and wrong otherwise, the same condition
  the `%` rule already had.
- Source labels must be forward-slashed and root-relative or the same directory
  fingerprints differently on Windows and Linux and every boot looks like a
  change — the identical bug §6 records.

BACKEND_DESIGN.md: the source set, the two refusals, and the updated admin
route notes (`approve`, `missingSources`, overlays found beside a file path).

Documentation only. Every claim was written after the behaviour was observed:
the counts come from the live shard's script tree, and the refusal/approve flow
and per-source breakdown are copied from real runs in website #115.

PROJECT_TREE.md files are deliberately untouched — CI-generated by the
sync-project-tree workflow.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-29 06:46:43 -05:00
wtclaude
e3aabf9e3e docs(link): fill in the cliloc PR numbers, correct the cliloc file count 
The progress table and CLILOCS.md carried #TBD placeholders until the PRs
existed; they now point at website #115 and docs #70.

Also corrects "all four Cliloc.* files" to eight (chs, cht, deu, enu, esp,
fra, jpn, kor) in both v3.md §8.6 and CLILOCS.md — the compression marker was
checked against every one of them, and understating the sample weakens the
claim it supports.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-29 04:25:06 -05:00
wtclaude
be9f5019fa docs(link): the cliloc table, and why §8.6's recommendation was not implementable 
Protocol 3.0 §8.6 resolved as its own website-only change, landing ahead of
§8 so the marketplace ships with real item names. Matching documentation for
website #TBD.

NEW website/CLILOCS.md — operator-facing: why the conversion step exists, how
to convert, how to configure the path, the refresh contract, what gets stored
and how names are applied.

link/v3.md §8.6 rewritten. Two things in the original recommendation turned
out to be wrong, and both are recorded because the reasoning generalises:

1. The committed db/data/clilocs.json artifact predates the Part C
   corrections (§6.1) and violates both — no committed snapshot of derived
   content, and nothing EA-derived ever shipped. UO's strings are EA's,
   exactly as the creature sprites are.
2. "scripts/buildClilocs.js reads the UO client's Cliloc.enu" is not
   possible. EVERY current client ships its cliloc files compressed (first
   DWORD's high byte 0x8E, the Mythic container); the plain layout is what
   those files looked like before that change, and parsing one as the other
   does not fail cleanly — it yields ~19k records with negative ids, 1,722
   distinct keys out of 19,508, one 62 KB "string", and a truncation
   somewhere in the middle. ServUO's own Ultima.StringList cannot read it
   either, so VendorSearch.GetItemName is already inert on such a shard and
   the work could not be pushed to the plugin.

That second point also retires an open question in §8.2: the warning never to
call GetItemName in the market sweep costs us nothing we could otherwise have
had, because the in-game Vendor Search gump has the same gap.

Three traps found by building it are recorded: StringList.SaveStringList
RE-COMPRESSES on save (its output is byte-identical to its compressed input,
because its purpose is round-tripping a file back into the client); trimming a
text line before splitting silently drops the ~half of a table that is empty
strings; and Number('') is 0, not NaN.

Also updated:
- Progress and §9 sequencing tables: order 5 split into 5a (this, website
  only) and 5b (the four-repo wire change).
- website/BACKEND_DESIGN.md — shard_clilocs / shard_cliloc_meta, the three
  admin routes, and why there is no staged-approval flow and no public route.
- link/INTEGRATION.md — the char.profile field note now says explicitly not to
  expect the shard to resolve clilocs, and points at CLILOCS.md.
- §10 documentation obligations list CLILOCS.md.

Documentation only. Every claim was written after the corresponding behaviour
was observed running: the compressed-format finding and the parse failures
come from the real client files on this machine, and the counts (123,490
parsed → 67,496 stored) and timings from importing them into the live MariaDB.

PROJECT_TREE.md files are deliberately untouched — they are CI-generated by
the sync-project-tree workflow and say so in their header.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-29 04:22:05 -05:00
Colby Whitlock
f715323aa0 Merge pull request 'docs(link): points.board, the leaderboards API, and what a real shard changed' (#69) from feat/points-board into edge 
Reviewed-on: #69
Reviewed-by: Colby Whitlock <whitlocktech@gmail.com>
 2026-07-29 07:52:25 +00:00
wtclaude
8e857a9c8d docs(link): points.board, the leaderboards API, and what a real shard changed 
Protocol 3.0 §7 lands across servuo-plugins, link and website; this is the
matching documentation.

INTEGRATION.md
  - points.board in the event catalog: one frame per system, never a delta, no
    points.remove (the shard's system set is fixed at startup). Four gotchas
    called out, all of them things a consumer gets wrong by default:
      1. maxPoints: 0 means UNCAPPED, not "zero allowed" — and on a real shard
         it is the COMMON case, so anything rendering points/maxPoints must
         special-case it or divide by zero on the happy path.
      2. nameString is usually null, with a cliloc in nameNumber — so
         humanising the system key is the primary display path, not a fallback.
      3. players counts players actually holding points, not table size: ten of
         the ~25 systems keep a zero-point row per character ever created, so
         the raw count would report the shard's whole census.
      4. Entries carry serial + name only, never acct/webId.
  - The char.profile `points` block, and why `rank` is absent by default.
  - GET /points and /points/:system, including why 404 (never published) and
    200-with-empty-top (published, nobody scored) are different answers.

v3.md
  - B/2 marked done in both the progress table and §9.
  - NEW §7.5, "what the run against a real shard changed" — the same record §6.1
    and §6.2 keep. Four corrections the plan could not have anticipated from
    reading PointsSystem.cs, the sharpest being that (long)double.MaxValue is an
    unchecked conversion yielding long.MinValue, which published
    "maxPoints": -9223372036854775808 on the first live sweep. Also records that
    GetEntry/GetPoints mutate the world on AutoAdd systems and so cannot be used
    in a read model, and the one deliberate deviation from §7.4: the visibility
    field rule must key on the wire's `name`, not the descriptive
    `characterName`, or it is silently inert.

BACKEND_DESIGN.md — shard_points_boards (including why the top-N list stays in
the payload and why listing orders by COALESCE(name, system)), plus the two new
public routes.

PLAN.md — 3.0 phasing brought current: the spawn atlas and points.board added to
what has shipped, and the Points* keys noted in the config-key paragraph.

PROJECT_TREE.md files are deliberately untouched — they are CI-generated and say
so.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-07-28 21:05:04 -05:00
Colby Whitlock
64fb7edc3e Merge pull request 'docs(website): the spawn atlas API, the admin panel, and the delay-unit trap' (#68) from docs/spawn-atlas-api into edge 
Reviewed-on: #68
Reviewed-by: Colby Whitlock <whitlocktech@gmail.com>
 2026-07-29 00:53:54 +00:00
wtclaude
be7e1a69ce docs(website): the spawn atlas API, the admin panel, and the delay-unit trap 
Docs half of website #113 (Protocol 3.0 Part C, second website PR).

Carries the atlas rewrite that missed #67: that PR merged before the "derive
from the tree on every boot" commit was pushed, so `edge` currently describes
the build/import-artifact design that was rejected in review, not what shipped
in website #112. It lands here.

New in SPAWN_ATLAS.md: the six public routes and five admin ones, and three
behaviours that read as bugs unless they are written down — an unreadable tree
answers 200 with status "unavailable" rather than 500 (refresh reports outcomes
so boot is never blocked by a bad tree, and the contract is preserved at the
API), setting the ServUO path deliberately does not import, and `points` is a
count while `spawners` is the list.

Also the delay-unit trap: XmlSpawner stores MinDelay/MaxDelay in minutes OR
seconds per record, decided by that record's own DelayInSec flag, so a `5` is
five minutes on one spawner and five seconds on the next. Both are plausible
respawn times, which is what makes it silent. 170 of 6,455 stock spawners are
second-flagged. And PARSER_VERSION, which exists because hashing the tree alone
would strand an install whose maps never change on whatever an older parser
derived.

BACKEND_DESIGN.md gains the routes, the router-map entry, and the parser-version
rule. v3.md marks Part C done and records in 6.3 what the API half found.

api-route-inventory.json refreshed from the live manifest — it had drifted to
200 routes before this PR (real count was 204) and is now 215.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U7CBg11prhLimL9iHSX1bP
 2026-07-28 19:51:50 -05:00
Claude
1b7da860b5 Merge remote-tracking branch 'origin/edge' into docs/spawn-atlas 2026-07-28 16:47:54 -05:00
wtclaude
ff1c2064a5 docs(website): the atlas reads the shard's tree on every boot, not a snapshot 
Follows the redesign in website #112. Two decisions from the original §6 were
rejected in review and replaced; the docs now describe what was actually built.

**The committed artifact is gone.** A shard's maps change over its life, so a
snapshot in the repo silently drifts from the world players actually see. The
ServUO tree is the single source of truth and the atlas is re-derived on every
server boot, hash-gated so an unchanged tree costs one read pass and no write.

**Nothing may name a facet.** The first implementation carried a lookup table of
the six stock UO facets. A shard may add facets, replace them outright, or rename
them when its maps are updated, and a built-in list mishandles all three
silently. Reconciliation is now by matching against the facet set discovered from
the shard's own data, with an unmatched name keeping its own rather than being
forced into a wrong bucket.

## Changes

- **`website/SPAWN_ATLAS.md`** rewritten: the two ideas that shape the design,
  how to configure the tree path, the boot flow as a decision tree, the
  approve/reject flow, and the code layout. The artwork policy is unchanged and
  still explicit — no art ever ships, operators extract their own from their own
  client files.
- **`link/v3.md` §6.1 (new)** records the two rejected decisions plus the two
  boot-path contracts. The old "what real data changed" list becomes §6.2. §6's
  now-superseded passages — the artifact bullet, the payload budget, the operator
  re-run story — are marked rather than deleted, so the reasoning stays legible.
- **`website/BACKEND_DESIGN.md`** documents `shard_atlas_pending` and the two
  contracts that make it safe: a facet removal is staged for a human, and the
  boot refresh can never block startup.

The two contracts are the part worth reviewing. Losing a facet is
indistinguishable at boot from a half-copied or mid-update tree, so it is staged
rather than applied; and no failure mode of the atlas — missing path, unreadable
mount, malformed file, database error — is allowed to stop the site coming up.

---

- [x] AI-assisted: written with **Claude Code** (Claude Opus 5), reviewed before opening.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U7CBg11prhLimL9iHSX1bP
 2026-07-28 16:44:39 -05:00
11 changed files with 1674 additions and 184 deletions
Expand all files Collapse all files

3
README.md
View File

@@ -20,6 +20,9 @@ ci/ cross-cutting CI/quality notes
| [HERO_EDITOR.md](website/HERO_EDITOR.md) | Hero canvas editor feature spec |
| [WIKI_UPGRADE.md](website/WIKI_UPGRADE.md) | Wiki subsystem upgrade notes |
| [SHARD_VISIBILITY.md](website/SHARD_VISIBILITY.md) | Who sees which shard data — the admin-configurable audience framework |
| [SPAWN_ATLAS.md](website/SPAWN_ATLAS.md) | The bestiary / spawn atlas: what the shard contains, parsed from its own ServUO tree |
| [CLILOCS.md](website/CLILOCS.md) | UO's id → name table: converting one from your client so items have names |
| [MARKETPLACE.md](website/MARKETPLACE.md) | The player-vendor index: how it is gathered, what it costs, how to tune it |
| [website-README.md](website/website-README.md) | Snapshot of the website repo's README (setup/run reference) |
| [PROJECT_TREE.md](website/PROJECT_TREE.md) | Auto-generated snapshot of the repo's tracked file layout |

115
android/PLAN.md
View File

@@ -1,6 +1,6 @@
# Android App — Plan
Status: **M0–M7 landed; M7 (push notifications) both parts done — Part 1 backend (website#78) and Part 2 app (Android-app#15) plus a small `push.ntfyUrl` settings addition (website#79). Remaining: set the shard's `NTFY_*` deploy config so push lights up, and cut the v1 tag. M9 (native SSO login) is now underway backend-first — the Mobile SSO Authorization Bridge is being built in `website/` + `docs/` ahead of the app-side client (§4.2, §9 M9); custom-scheme callback only for now, App Links deferred (see [`APP_LINKS.md`](./APP_LINKS.md)).** This document is the
Status: **M0–M7 landed; M7 (push notifications) both parts done — Part 1 backend (website#78) and Part 2 app (Android-app#15) plus a small `push.ntfyUrl` settings addition (website#79). Remaining: set the shard's `NTFY_*` deploy config so push lights up, and cut the v1 tag. M9 (native SSO login) is now underway backend-first — the Mobile SSO Authorization Bridge is being built in `website/` + `docs/` ahead of the app-side client (§4.2, §9 M9); custom-scheme callback only for now, App Links deferred (see [`APP_LINKS.md`](./APP_LINKS.md)). **M11 (Protocol 3.0 shard parity)** is scoped and next: the app sees none of the four shard features v3 added (`ruleset`, `leaderboards`, `market`, `atlas`) and does not consult `GET /public/shard/features`, so it gates shard nav on session role alone while an admin can switch any of those surfaces off or raise its audience — the v3 `edge` → `main` cutover is held until it lands (§9 M11).** This document is the
design contract for the `RunicGateway/Android-app` repo. It was written before implementation so the
API changes it depends on could be landed in `website/` and `docs/` first. The authoritative API
reference is the committed OpenAPI spec at `website/server/swagger/swagger-output.json` (regenerated
@@ -631,6 +631,8 @@ not rank).
| News & content | everyone | `/public/posts/:category`, `/public/pages/:slug` |
| Wiki | everyone | `/public/wiki`, `/public/wiki/categories`, `/public/wiki/tags`, `/public/wiki/:slug` |
| Shard (live) | everyone | `/public/shard/*` + `/public/shard/stream` (SSE) |
| **Rules / Leaderboards / Market** | everyone, *if the shard publishes them* | `/public/shard/{ruleset,points,market}` (M11) |
| **Atlas** (bestiary) | everyone, *if the shard publishes it* | `/public/atlas/*` (M11) |
| Contact | everyone | `/public/contact` |
| **My Account** | signed-in | `/player/account/*` (or `/admin/account/*` for staff — see §6.4) |
| **My Characters / Vendors / Houses** | `player` (linked) | `/player/shard/*` |
@@ -639,6 +641,12 @@ not rank).
Guidelines:
- The menu is **declarative + data-driven**, not a pile of `if role ==` checks — one list of entries
with a `minAccess`/`requiredCapability` field, filtered by the session.
- **Session role is not the only gate on shard surfaces (M11).** Every shard-derived feature is
*admin-configurable* — it can be switched off or raised to a higher audience rung — so a shard entry
is filtered by the session role **and** by `GET /public/shard/features`, which reports the features
the caller may actually reach. While that answer is unknown (in flight, or the lookup failed) the app
shows everything: the server gates regardless, and a nav that flickers in on every load is worse than
a link that briefly `403`s.
- Never hide the fact that more exists behind auth in a way that misleads; anonymous users see public
groups and a "Sign in" affordance.
- The server is the source of truth: a hidden/greyed item is a UX convenience; every gated call still
@@ -663,9 +671,15 @@ Guidelines:
### 6.2 Public shard (live)
- Status/online/feed/economy/champs/guilds/governors(+history)/presence/houses/idoc — the
`/public/shard/*` GETs.
- **Live updates** — subscribe to `GET /public/shard/stream` (SSE, safe kinds only) and patch the
in-memory boards in place (champ/guild/city/house/presence update+remove frames). Reconnect with
backoff; fall back to poll if SSE drops.
- **Live updates** — subscribe to `GET /public/shard/stream` (SSE) and patch the in-memory boards in
place (champ/guild/city/house/presence update+remove frames). Reconnect with backoff; fall back to
poll if SSE drops. What arrives on the stream is **resolved from the caller's audience rung at
subscribe time**, not from a fixed allowlist (Protocol 3.0 §3.6) — the stream request carries the
bearer like every other call, so a signed-in app session sees exactly what the same account sees on
the web.
- **Visibility + the Protocol 3.0 surfaces (M11)** — `GET /public/shard/features` drives which of these
the menu offers; `GET /public/shard/{ruleset,points,points/:system,market,market/meta,market/vendors/:serial}`
and `GET /public/atlas/*` are the new reads. Full contract and traps in §9 M11.
### 6.3 Player self-service & game data (bearer)
- **Account** — `GET /player/account`; `PATCH /player/account/username`;
@@ -675,6 +689,11 @@ Guidelines:
- **My game data** — `GET /player/shard/roster/:account`, `/char/:serial`, `/vendors/:account`,
`/sales`, `/houses`. All ownership-checked server-side; a `503` means shard/sidecar down → show an
"offline, retry" state (see §7).
- **The character sheet carries two things the app does not yet read (M11):** the `points` block
(per-character loyalty/points standings, Protocol 3.0 §7.3) and the server-resolved cliloc names on
`equipment[].clilocName` / `titles.rewardResolved` (§8.6). Both are served **ungated** on this route —
a character's own standings are self-service data and do not depend on the public `leaderboards`
feature being visible, which is the behavior the app must mirror rather than re-gate.
- **Presentation is text-only for v1.** Character sheets and vendor listings render as data/text — no
item icons or paperdoll art. A richer "pretty paperdoll" view is a **future** enhancement (pending the
art/asset work on the platform side) and is explicitly out of the first release.
@@ -883,10 +902,94 @@ push, and Play (M6–M8) follow the designed app.
shard-write actions degrade gracefully when the sidecar is offline. Excluded: hero/CMS block
editor, Discord-bot config, uo-link config, OAuth-provider setup.
12. **M11 — Protocol 3.0 shard parity** (post-v1; scoped 2026-07-30). The website's Protocol 3.0 work
added four shard features and, with them, an **admin-configurable visibility framework** the app
knows nothing about. `link/v3.md` §10 deferred the app side as a follow-up; it is now scoped
deliberately, and **the v3 `edge` → `main` cutover is held until both parts land** so web and app
surface the same shard on the same day (decided 2026-07-30).
Neither part is coupled to the cutover *merge order*, which is what makes holding it a schedule
decision rather than a technical one: against a pre-v3 website every new route and
`/public/shard/features` simply `404`s, and each consumer below falls back to exactly today's
behavior. The app declares no protocol version and never talks to the sidecar.
**Both parts are built and in review:** Part 1 `RunicGateway/Android-app#30`, Part 2 (stacked on
it) `#31`. 336 unit tests pass and lint is clean on both; the on-device five-rung walk below is
the remaining gate, and it is what the cutover is actually waiting on.
- **Part 1 — the visibility rules + the read-model adds.** The security-shaped half, reviewed on
its own:
- `GET /public/shard/features` → `{ level, features[] }`: the features **this caller** may reach.
A new singleton cache mirrors the web client's (`lib/useShardFeatures.js`): per-viewer but
stable for a session, invalidated on sign-in/out and on a server switch.
- `MenuEntry` gains `feature: String?` beside its existing `access`, so the one declarative menu
(§5) filters on the session role **and** the shard's live feature config. While the lookup is
in flight or has failed, **show everything** — the same deliberate fail-open the web client
takes, because the server gates regardless and a nav that flickers in on every load is worse
than a link that briefly `403`s. The gate is server-side; hiding is presentation.
- **`404` and `403` mean different things here** and neither is a generic error:
`requireFeature` `404`s a *disabled* feature (deliberately not disclosing that it exists) and
`403`s a viewer *below its audience*. Both render "not available on this shard", alongside the
existing `503` = shard offline (§7).
- **The `level` from `/features` is authoritative — do not re-derive the rung from the role.**
The server's ladder is `anonymous → logged_in → player → staff → admin`, where `player` means
*a linked game account* and staff always satisfy `player` (the same superset rule `Menu.kt`
already encodes as `isPlayer || isStaff`).
- **`char.profile.points`** → the "Loyalty & Points" block the web character sheet gained:
`CharProfileDto.points[{system, nameString, points, maxPoints, rank?}]`. Three traps, all of
them things a real shard does and a fake one does not (`v3.md` §7.5): `maxPoints == 0` means
**uncapped** and is the *common* case, so nothing may divide by it; `nameString` is usually
`null` because most systems name themselves with a cliloc, making the humanise-the-`system`-key
path the **primary** one rather than a fallback; and `rank` is absent unless the shard runs
`PointsProfileRank=true` — absent and "unranked" are different answers.
- **Cliloc-resolved names** (`v3.md` §8.6, already live on the website): `EquipmentDto` gains
`name` + `clilocName` and `TitlesDto` gains `rewardResolved`, so equipment stops rendering as a
layer or a bare id. Precedence is `name → clilocName → layer`: a player-given name outranks the
resolved type name, and the server applies the same order. A shard with no cliloc table
configured sends neither field and the sheet renders exactly as it does today.
- `ActorDto` keeps its `acct` / `webId` fields (nullable, so nothing breaks) but its KDoc stops
describing them as available: they are **locked to the admin rung**, always, and stripped from
every response below it.
- **Part 2 — the four new screens**, each hidden by its feature name in the menu:
- **Rules** — `GET /public/shard/ruleset` (`ruleset`). A `null` body means "the shard has not
published its ruleset yet", which is a different state from the feature being disabled. Every
block is optional and omitted when its system is off. **`caps.skill` / `caps.totalSkill` are in
tenths** (1000 = 100.0) and must be converted — the raw number is actively misleading, not
merely unhelpful. Live via the `world.ruleset` frame, which is on the public stream by default.
- **Leaderboards** — `GET /public/shard/points`, `/points/:system` (`leaderboards`). The same
`maxPoints`/`nameString` traps as the profile block. Live via `points.board`.
- **Market** — `GET /public/shard/market` (`q`, `minPrice`, `maxPrice`, `itemId`, `map`, `region`,
`sort`, `limit`, `offset`), `/market/meta` for the filter options + staleness, and
`/market/vendors/:serial` (`market`). Four things this screen must get right: it is the site's
first **rate-limited** public endpoint, so handle `429` the way the contact form does; the
*"prices last refreshed N minutes ago"* banner is **required, not decoration** — the shard
sweeps vendors round-robin, so a listing can legitimately be a full cycle stale and a page
implying live prices sends people to an item that sold twenty minutes ago; a `truncated` shop
must say so; and `location` is a **nested object** that an admin may gate away entirely, which
the vendor screen renders as "hidden by the shard" (a real answer) rather than as blank
coordinates — same for `ownerName` / `ownerSerial`. **The `market` SSE fan-out is off by
default** (a live firehose of vendor inventories would be the site's biggest bandwidth
consumer), so the screen is a plain paginated read and must never depend on live frames.
- **Atlas** — `GET /public/atlas/{creatures,creatures/:slug,regions,landmarks,champions,meta}`
(`atlas`). Note the path: `/public/atlas`, **not** `/public/shard` — the atlas is static shard
*content*, not live shard *state*, and unlike `/shard/*` it **is** `siteMode`-gated like
`/posts` and `/wiki`, so a site in maintenance mode withholds it independently of the sidecar.
Two units/naming traps from `v3.md` §6.3: respawn delays are **seconds** throughout, and
`points` is a *count* on the search route while `spawners` is the *list* on the detail route.
- **Verification** — the five-rung walk (`anonymous`, `logged_in`, `player`, `staff`, `admin`)
against a local website on the cutover branch, per
[`../link/v3.md`](../link/v3.md) §11 and the shard-visibility smoke harness; plus one pass with
**every feature disabled** in Admin → Shard Visibility, confirming the app *hides* each surface
instead of erroring on it. Unit tests cover the menu filter (role × feature set), the
`404`/`403`/`503` mapping, and DTO decode for each new shape.
- **Excluded**, in the same class as M10's exclusions: the admin *configuration* panels — Shard
Visibility, Spawn Atlas and Cliloc import — alongside the hero/CMS block editor, Discord-bot
config, uo-link config and OAuth-provider setup.
### Deferred (not a milestone)
- **`/api/mobile` facade migration + app-version floor** — briefly planned as M11 (2026-07-22), now
**deferred with no app work scheduled**. The website's router refactor is being done in place with
- **`/api/mobile` facade migration + app-version floor** — briefly planned as its own milestone
(2026-07-22), now **deferred with no app work scheduled**. The website's router refactor is being done in place with
every URL byte-identical and `/api/v1` is not being retired, so the app's ~70 hardcoded `api/v1/…`
endpoints, its SSE path, and its SSO URLs keep working untouched. If the mobile contract ever needs
to diverge from web, the migration comes back — starting from a one-line alias mount on the server,

230
link/INTEGRATION.md
View File

@@ -31,31 +31,31 @@ Missing or wrong token → **401** `{"error":"missing or invalid auth token"}`.
The wire protocol is versioned so a mismatch is caught immediately instead of failing weirdly.
- Every response carries an **`X-UOLink-Version: 2`** header.
- `GET /health` and the WebSocket `ws.hello` frame include `"protocol": 2`.
- **Optionally**, send `X-UOLink-Version: 2` on your requests. If it disagrees with the sidecar, the request is rejected **409 Conflict**:
- Every response carries an **`X-UOLink-Version: 3`** header.
- `GET /health` and the WebSocket `ws.hello` frame include `"protocol": 3`.
- **Optionally**, send `X-UOLink-Version: 3` on your requests. If it disagrees with the sidecar, the request is rejected **409 Conflict**:
```json
{ "error": "protocol version mismatch", "sidecar_protocol": 2, "client_protocol": "1" }
{ "error": "protocol version mismatch", "sidecar_protocol": 3, "client_protocol": "2" }
```
Pin the version you built against and compare it to the header (or `/health.protocol`) at startup.
**v2 (Protocol 2.0)** added the account-provisioning surface (§6.x: `POST /accounts/create`, `DELETE /link/{account}`) and the `account.*` events. Outbound event kinds are **additive** — a v1 client that ignores unknown kinds keeps working against the live feed — but the new *endpoints* require a v2 sidecar. If you send `X-UOLink-Version: 1`, calls to the new endpoints are refused with the 409 above.
**v3 (Protocol 3.0) is being built and the version has not been bumped yet.** It is defined as *adds
`world.ruleset`, `points.board`, `vendor.listing` / `vendor.listing.remove`*, and the bump to
`X-UOLink-Version: 3` happens **exactly once**, at the end, when [`v3.md`](v3.md) §4's `edge` → `main`
cutover lands — because a bump is an operator-visible hard break (409 on every protected route, and
the website's WS closes on the `ws.hello` mismatch), so doing it per phase would break the site
repeatedly.
**v3 (Protocol 3.0)** adds `world.ruleset`, `points.board` and `vendor.listing` /
`vendor.listing.remove`, with the `GET /ruleset`, `/points` and `/market` reads that serve them from
the sidecar's store. Same shape as the v2 bump: the event kinds are additive, so a v2 client that
ignores unknown kinds keeps working against the live feed, but the three new endpoints require a v3
sidecar. There is deliberately **no feature-negotiation array** — v3 implies all three kinds, so the
version number alone tells you what is available.
Until then, sidecars on `edge` still report `2` while already carrying some v3 kinds and endpoints.
That is safe in the direction that matters: event kinds are additive, and a client that ignores
unknown kinds and tolerates a `404` on a not-yet-present endpoint keeps working. What you must **not**
do is infer feature availability from the version number during this window — probe the endpoint, or
treat a missing `world.ruleset` as "this shard hasn't published one". There is deliberately **no
feature-negotiation array**: v3 implies all three kinds.
**Upgrading a v2 integration.** The bump is an operator-visible hard break in one direction only: a
client still declaring `2` gets a 409 on every protected route and, on the WebSocket, a closed
connection on the `ws.hello` mismatch. So update the pinned version at the same time you deploy the
v3 sidecar. Nothing that existed in v2 changed shape, so that is the whole migration — the website
does it with a one-shot boot migration of its `uo_link_config.protocol` row ([`v3.md`](v3.md) §4.1);
a third-party client changes the constant it sends.
---
@@ -68,7 +68,7 @@ GET /health (no auth)
```json
{
"status": "ok", // "ok" when plugin connected AND db reachable, else "degraded"
"protocol": 1,
"protocol": 3,
"plugin_connected": true, // is the shard link up right now?
"database": "ok", // "ok" | "error"
"uptime": "3d 12h",
@@ -91,7 +91,7 @@ A push-only stream of game events as they happen. You do **not** send commands o
**On connect**, the first frame is:
```json
{ "kind": "ws.hello", "protocol": 1 }
{ "kind": "ws.hello", "protocol": 3 }
```
**Then** a continuous stream of event frames, each with at least `t` (epoch ms) and `kind`. Route on `kind`.
@@ -381,6 +381,146 @@ Absent entirely if the shard runs `Bridge.RulesetEnabled=false` or an older plug
This **supersedes the `world.systems` frame** sketched in [`PROTOCOL_2.md`](PROTOCOL_2.md) §10.4 and
never implemented; the `systems` block above is what that asked for.
#### Points / loyalty leaderboards (Protocol 3.0)
ServUO carries ~25 separate point currencies — Queen's Loyalty, Void Pool, Casino, Clean Up Britannia,
the nine city loyalties, Blackthorn, the Doom / Khaldun / Kotl treasure systems — every one a standing
players accumulate over months, and none of them visible outside an in-game gump before 3.0.
A diff sweep (default 300 s), **one frame per system** rather than one large frame for all of them,
matching `champ.update` / `guild.update`. A system is emitted only when its top N or its participant
count actually changes.
| kind | fields | notes |
|------|--------|-------|
| `points.board` | `system`, `nameString`, `nameNumber`, `maxPoints`, `showOnGump`, `players`, `top[]` | One system's complete board — **never a delta**. The latest frame for a `system` replaces the previous one outright. `top[]` entries are `{rank, serial, name, points}`. |
`system` is the shard's own `PointsType` enum name (`QueensLoyalty`, `CleanUpBritannia`, …) and is the
board's stable key. There is deliberately **no `points.remove`**: the set of systems is fixed at startup
by `PointsSystem.Configure`, so a system cannot disappear at runtime — the same argument `city.update`
makes for cities.
```json
{"kind":"points.board","system":"QueensLoyalty",
"nameString":"Queen's Loyalty","nameNumber":1114938,
"maxPoints":15000,"showOnGump":true,"players":842,
"top":[{"rank":1,"serial":"0x1A2B","name":"Darrow","points":29500},
{"rank":2,"serial":"0x1A2C","name":"Mireille","points":21000}],
"t":1752489280000}
```
**Four things consumers get wrong.**
1. **`maxPoints` of `0` means UNCAPPED, not "zero points allowed".** ServUO's idiom for an uncapped
system is `double.MaxValue` (`DespiseCrystals`, `ShameCrystals` and `VoidPool` all use it), which
the plugin normalises to `0` rather than emitting a nonsense integer. On a real shard **most
systems are uncapped**, so a UI that renders `points / maxPoints` must special-case this or it will
divide by zero on the common path.
2. **`nameString` is usually `null`.** The shard's `Name` is a `TextDefinition`, which may carry a
literal *or* a cliloc id, and in practice most systems use the cliloc — so `nameNumber` is set and
`nameString` is `null`. Resolve clilocs consumer-side; failing that, humanising the `system` key
("CleanUpBritannia" → "Clean Up Britannia") reads better than showing a bare number. This is the
same contract `titles.reward` already documents.
3. **`players` counts players who actually hold points**, not the size of the system's table. Ten of
the ~25 systems have `AutoAdd = true` and therefore keep a zero-point row for every character that
has ever logged in, so the raw table size would report the shard's entire character census as that
system's participants.
4. **Entries carry `serial` and `name` only — never `acct` or `webId`.** A board is the widest-audience
surface the bridge has, so the account name of every ranked player deliberately does not cross the
wire; resolve serial → site user from your own link mirror if you need it.
Absent entirely if the shard runs `Bridge.PointsLeaderboardEnabled=false` or an older plugin. Render
from `GET /points` (§6) on connect, then keep live with this event.
##### `char.profile` gains a `points` block
Read-model enrichment on the existing kind — there is **no** request kind for one character's points,
the same precedent `titles` set in [`PROTOCOL_2.md`](PROTOCOL_2.md) §10.3:
```json
"points":[{"system":"QueensLoyalty","nameString":"Queen's Loyalty","nameNumber":1114938,
"points":29500,"maxPoints":15000}]
```
Systems where the character has no entry, or an entry at zero, are **omitted** — otherwise every sheet
would carry ~25 zeroes. `maxPoints` follows the same `0 == uncapped` rule as the board.
`rank` is **absent by default** and appears only when the shard runs `Bridge.PointsProfileRank=true`:
a points lookup stops at the character's own row, but a rank must count every row that beats them, in
every system, on every profile build. Derive rank from `points.board` instead for anyone in the top N.
#### Player-vendor marketplace (Protocol 3.0)
The shard-wide shop index: every player vendor's shop name, owner, location and priced inventory —
the same set the in-game **Vendor Search** gump reads, published so a site can offer the same search
from outside the game.
An **amortized round-robin diff sweep**, not a snapshot RPC, and the distinction is load-bearing:
`rpc.rs::try_route` correlates a reply on the FIRST frame carrying a matching `reqId`, so a chunked
reply sharing one `reqId` would deliver chunk 1 to the HTTP caller and leak chunks 2..N onto the
broadcast feed. A whole-world snapshot could not fit in one frame inside the 10 s reply timeout
either. The per-account `vendor.snapshot` RPC (§5) is unaffected and still serves the player portal.
Each tick inventories at most `Bridge.MarketSweepBatch` vendors (default 25) starting from a
persistent cursor, so **per-tick cost is bounded independently of world size**; full coverage takes
`ceil(vendors / batch) × MarketSweepSeconds`. A vendor is emitted only when its contents, prices,
shop name or location actually change.
| kind | fields | notes |
|------|--------|-------|
| `vendor.listing` | `serial`, `shopName`, `ownerSerial`, `ownerName`, `location{}`, `count`, `total`, `truncated`, `items[]` | One vendor's complete shop — **never a delta**. The latest frame for a `serial` replaces the previous one outright. |
| `vendor.listing.remove` | `serial` | The shop is gone from the index: dismissed, expired, or its owner switched off the in-game Vendor Search flag. |
```json
{"kind":"vendor.listing","serial":"0x40001234",
"shopName":"Darrow's Bargains","ownerSerial":"0x1A2B","ownerName":"Darrow",
"location":{"map":"Trammel","x":1421,"y":1699,"z":0,
"region":"Britain","house":"Darrow's Villa"},
"count":2,"total":2,"truncated":false,
"items":[{"serial":"0x40012ABC","itemId":3922,"hue":0,"amount":1,
"price":25000,"name":null,"cliloc":1023721},
{"serial":"0x40012ABD","itemId":7026,"hue":1157,"amount":3,
"price":500,"name":"a shard sigil","cliloc":1041243}],
"t":1752489280000}
```
**Six things consumers get wrong.**
1. **`name` is `null` for nearly every item; `cliloc` is the real label.** Items carry a
`LabelNumber`, not a name. The plugin deliberately never calls `VendorSearch.GetItemName`, which
builds an `ObjectPropertyList`, serialises it and byte-parses the packet **per item** — a
multi-hundred-millisecond stall across a full pass. (It would not work anyway: every current
client ships its cliloc files compressed and ServUO's bundled `Ultima.StringList` cannot read
them, so the in-game gump has the same gap.) Resolve clilocs consumer-side; a non-null `name` is a
player-set literal and is strictly more specific, so **prefer it over the cliloc**.
2. **`location` is one nested object, and it may be absent entirely.** It is nested so that a
consumer gating vendor whereabouts gates one field rather than five that can drift apart — the
website's `market.location` rule removes the whole object. Treat a missing `location` as "not
published", not as an error.
3. **`truncated` means the shop holds more than the frame carries.** `count` is what was published,
`total` is what the shop actually holds, capped by `Bridge.MarketMaxListings` (default 250). A
commodity reseller with thousands of stacked resources is real and an uncapped frame for one is
measured in megabytes. Say "showing 250 of 3,104" rather than presenting a partial shop as
complete.
4. **`child: true` means the price buys the ENCLOSING CONTAINER.** ServUO prices a container as a
unit and everything inside inherits that price with no `VendorItem` of its own; `DoSearch`
surfaces the same flag. A UI that prints the container's price against each item inside it is
lying about the shard.
5. **Opted-out vendors are absent, and that is a privacy control.** `pv.VendorSearch` is the player's
own in-game toggle and the sweep honours it — hide your vendor in game and it is hidden here too.
The same goes for `Map.Internal` and a null backpack, matching `DoSearch`. Process
`vendor.listing.remove` promptly: it is how a player *revoking* that consent reaches you.
6. **Prices are inherently stale, by design.** The round-robin sweep means a shop can be a full cycle
behind. Any UI over this must say how old the data may be — the website derives it from the oldest
vendor row.
Entries carry `ownerSerial`/`ownerName` and **never `acct` or `webId`**, the same rule `points.board`
follows. Absent entirely if the shard runs `Bridge.MarketEnabled=false` or an older plugin. Render
from `GET /market` (§6) on connect, then keep live with these events — though note that a live
firehose of whole vendor inventories is the largest stream the bridge produces, and a consumer that
only needs a browsable index (as the website does) is better served by the REST read plus the
periodic re-sweep.
---
## 5. REST — read queries
@@ -420,7 +560,7 @@ Full character sheet: stats, all trained skills, worn equipment with flattened i
Field notes:
- `skills[].base` is trained value, `value` includes item/temp bonuses, `cap` is the cap. **Do not assume `base <= cap`** — GM characters can exceed it.
- `equipment[].mods` is a flattened map of every non-zero AOS attribute on the item (weapon or armor). Empty `{}` for plain items.
- Item names are usually **clilocs**, not strings: use `name` when present, otherwise resolve `cliloc` against a UO cliloc table on the site.
- Item names are usually **clilocs**, not strings: use `name` when present, otherwise resolve `cliloc` against a UO cliloc table on the site. **Do not expect the shard to resolve them for you** — on any modern client ServUO's own `Ultima.StringList` cannot read the client's compressed cliloc files, so `VendorSearch.GetItemName` returns `item.Name` and the in-game Vendor Search gump has the same gap. Building that table is a consumer-side job; the website's is described in [`website/CLILOCS.md`](../website/CLILOCS.md).
- `titles` (Protocol 2.0): `selected` is the index into `reward` currently displayed (`-1` if none). `fameKarma`/`skill` are computed display titles, omitted when the character has none. `reward` entries may be a **cliloc number as a string** or a literal string — resolve numeric ones against your cliloc table, same as item names.
- Errors: unknown account → **404** `{"kind":"bridge.error","reason":"unknown account"}`; bad slot → **404**/**400** similarly.
@@ -740,6 +880,56 @@ worse than one that is briefly stale. Keep it current with the `world.ruleset` s
`Bridge.RulesetEnabled=false`. That is a real answer distinct from a published ruleset, and worth
rendering differently ("not published yet") rather than as an empty ruleset.
### Points / loyalty leaderboards (Protocol 3.0)
```
GET /points
→ { "boards": [ {"kind":"points.board","system":"QueensLoyalty","nameString":"Queen's Loyalty",
"nameNumber":1114938,"maxPoints":15000,"showOnGump":true,"players":842,
"top":[{"rank":1,"serial":"0x1A2B","name":"Darrow","points":29500}, ...],"t":...}, ... ] }
GET /points/{system} # e.g. /points/QueensLoyalty
→ {"kind":"points.board","system":"QueensLoyalty", ... }
```
Every system's latest board, or one by its `PointsType` name (§4 for the frame and its four gotchas).
Served from the sidecar's projection, kept current by the `points.board` stream, ordered by display
name. Survives a sidecar restart — which matters more here than for live state, since these are
standings built over months and blanking them during a restart reads as data loss.
`GET /points/{system}` returns **404** for a system the shard has never published (an unknown name, or
one excluded by `Bridge.PointsSystems`). That is distinct from a published board nobody has scored in
yet, which is **200** with an empty `top[]` — and the two are worth rendering differently.
### Player-vendor marketplace (Protocol 3.0)
```
GET /market?limit=200&offset=0
→ { "vendors": [ {"kind":"vendor.listing","serial":"0x40001234",
"shopName":"Darrow's Bargains","ownerSerial":"0x1A2B","ownerName":"Darrow",
"location":{"map":"Trammel","x":1421,"y":1699,"z":0,
"region":"Britain","house":"Darrow's Villa"},
"count":2,"total":2,"truncated":false,"items":[ ... ],"t":...}, ... ],
"total": 137, "limit": 200, "offset": 0 }
```
Every vendor's latest shop, exactly as `vendor.listing` published it (§4 for the frame and its six
gotchas). Served from the sidecar's projection, so it answers while the shard is down.
**This is the only PAGED read the sidecar serves**, because it is the only board that can be a whole
world's inventory. `limit` is clamped to 1..1000 (default 200); `total` is returned so a caller knows
when to stop rather than paging until it sees a short page, which would race a concurrent sweep.
Ordering is by **serial**, not by shop name — a serial is stable while a shop name is renameable, so
a rename mid-walk cannot make a vendor skip or repeat a page.
The route is `/market` and deliberately **not** `/vendors`: `/vendors/{account}` next door is the
per-account RPC (§5), and two routes a prefix apart meaning "this player's shops" and "every shop on
the shard" is a trap nobody wins.
Frames are served **verbatim**, owner names and coordinates included. That is not an oversight: the
sidecar defines no audiences. Deciding who may see what is the consuming site's job — see
[`v3.md`](v3.md) §3 for how the website does it.
---
## 7. Status codes
@@ -765,7 +955,7 @@ rendering differently ("not published yet") rather than as an empty ruleset.
A typical character page:
```js
const H = { "Authorization": `Bearer ${TOKEN}`, "X-UOLink-Version": "2" };
const H = { "Authorization": `Bearer ${TOKEN}`, "X-UOLink-Version": "3" };
// 1. render the roster
const roster = await fetch(`${BASE}/roster/${account}`, { headers: H }).then(r => r.json());

40
link/PLAN.md
View File

@@ -284,6 +284,15 @@ Counts in `hello` are a live snapshot taken on the Core thread, not a cached val
`Item.Name` is frequently `null`; the display name is `LabelNumber`, a cliloc id. **There is no `Data/Cliloc.enu` in this repo** — `BRIDGE_FINDINGS.md` §IV.4 is wrong about this. Cliloc data lives in the client install, which `DataPath` resolves to `D:\Games\Electronic Arts\Ultima Online Classic\`. Ship **both** `name` (when non-null) and `cliloc`, and resolve the number **on the website** against a cliloc map. That avoids a server-side dependency on the client directory.
**Update (3.0).** That recommendation held, and the reason it had to hold turned out to be stronger
than "avoids a dependency": **ServUO cannot resolve clilocs either.** Every current client ships its
`Cliloc.*` files compressed, and the bundled `Ultima.StringList` reads only the older plain layout —
so `VendorSearch.StringList` is null and `VendorSearch.GetItemName` returns `item.Name` on any modern
shard. The in-game Vendor Search gump has the same gap, which is why `vendor.listing` never calls it.
Pushing name resolution to the plugin was never an option. See [`v3.md`](v3.md) §8.6 and
`docs/website/CLILOCS.md` for how the site gets a table instead (the operator converts one from their
own client, once).
---
## 8. Corrections to `BRIDGE_FINDINGS.md`
@@ -318,10 +327,31 @@ Counts in `hello` are a live snapshot taken on the Core thread, not a cached val
**Beyond 1.0.** Phases above are the 1.0 read/event plane. Protocol 2.0's phasing (provisioning +
world-state boards) is [`PROTOCOL_2.md`](PROTOCOL_2.md) §13; Protocol 3.0's (visibility framework,
shard content and standings) is [`v3.md`](v3.md) §9, which also tracks what has landed. Shipped from
3.0 so far: **Part A** — the visibility framework — and **`world.ruleset`** ([`v3.md`](v3.md) §5),
`BridgeRuleset.cs`, the first bridge stream that is neither an event subscription nor a sweep: it is
3.0 so far: **Part A** — the visibility framework — **`world.ruleset`** ([`v3.md`](v3.md) §5),
`BridgeRuleset.cs`, the first bridge stream that is neither an event subscription nor a sweep (it is
emitted once per connect, like `server.hello`, because shard config changes only when an operator
edits a file.
edits a file) — the **spawn atlas** ([`v3.md`](v3.md) §6), which is website-only and touches no wire
at all — and **`points.board`** ([`v3.md`](v3.md) §7), `BridgePoints.cs`, the loyalty/points
leaderboards. `BridgePoints` is the widest read the bridge performs: ten of ServUO's ~25 point systems
keep a row for every character ever created, so it selects the top N in a single bounded pass rather
than sorting, and runs on a deliberately slow 300 s interval.
Also shipped: **`vendor.listing`** ([`v3.md`](v3.md) §8), `BridgeMarket.cs`, the shard-wide
player-vendor index. It introduces the one sweep pattern the bridge did not previously have — an
**amortized round-robin**. Every other sweep walks its whole collection per tick, which is fine for
tens of houses or a fixed set of point systems and is not fine for a world of shops whose inventories
recurse into containers. `BridgeMarket` inventories at most `MarketSweepBatch` vendors per tick from
a persistent cursor, so the per-tick cost is bounded by the batch rather than by world size, and full
coverage takes `ceil(vendors / batch) x MarketSweepSeconds`. Measured at **15.4 ms** for a cold tick
of 25 vendors x 40 listings and **0.3 ms** in steady state (the per-vendor diff), on a shard of 209k
items / 43k mobiles. It is also the first stream to honour a per-player privacy toggle: ServUO's own
`PlayerVendor.VendorSearch` flag, so a shop hidden in game is hidden on the site.
That completes 3.0's feature work, so the last step is the version itself: `PROTOCOL_VERSION` **2 →
3** and the coordinated `edge` → `main` merge across all four repos ([`v3.md`](v3.md) §4 and §4.1).
The bump is deliberately the *only* thing that happens at that moment — v3 adds kinds and endpoints
but changes nothing that already existed in v2 — so the operator-visible break is limited to
re-pinning the version, which the website does for itself in a one-shot boot migration.
### Config keys (`Config/Bridge.cfg`)
@@ -338,7 +368,9 @@ Read in `Configure()` via `Config.Get<T>("Bridge.<Key>", default)`. Key scope is
The set above is the 1.0 sample, not the current one — every later phase added keys (sweep intervals
for each board, the town-crier/news caps, the admin write plane, account provisioning, and 3.0's
`RulesetEnabled` / `PublicConnectAddress` / `RulesetIncludeSchedule`). **`servuo-plugins/overlay/Config/Bridge.cfg`
`RulesetEnabled` / `PublicConnectAddress` / `RulesetIncludeSchedule`, and the `Points*` and `Market*`
blocks).
**`servuo-plugins/overlay/Config/Bridge.cfg`
is the authoritative, commented list**; `BridgeConfig.cs` holds the defaults.
---

407
link/v3.md
View File

@@ -1,6 +1,6 @@
# Protocol 3.0 — Shard content, standings & the visibility framework
**Status:** In progress. All work lands on an `edge` branch in each repo; `edge` → `main` is the v3 cutover.
**Status:** Feature-complete on `edge`; the cutover (order 6) is in review. All work lands on an `edge` branch in each repo; `edge` → `main` is the v3 cutover.
**Date:** 2026-07-28
**Codebase:** ServUO 57.4, `<servuo>`, net48 / x64, Expansion **EJ**.
**Companion to** [`PLAN.md`](PLAN.md) (1.0 read/event plane), [`PROTOCOL_2.md`](PROTOCOL_2.md) (2.0 provisioning + world-state streams), [`ADMIN_CONTROLS.md`](ADMIN_CONTROLS.md) (staff write plane), [`INTEGRATION.md`](INTEGRATION.md) (website API).
@@ -13,10 +13,20 @@ Each part is marked off here as it lands on `edge`. §9 carries the same state p
|---|---|---|---|
| 1 | **A** — visibility framework + actor-leak fix (§3) | ✅ **Done** | website [#109](https://gitea.whitlocktech.com/RunicGateway/website/pulls/109) + [#110](https://gitea.whitlocktech.com/RunicGateway/website/pulls/110), docs [#64](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/64) + [#65](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/65) |
| 2 | **B/1** — `world.ruleset` (§5) | ✅ **Done** | servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66) |
| 3 | **C** — spawn atlas (§6) | 🟡 **Data pipeline done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables); API/client PR next |
| 4 | **B/2** — `points.board` (§7) | ⬜ Not started | — |
| 5 | **B/3** — `vendor.listing` (§8) | ⬜ Not started | — |
| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | ⬜ Not started | — |
| 3 | **C** — spawn atlas (§6) | ✅ **Done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables) + [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113) (API + pages + admin panel), docs [#67](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/67) + [#68](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/68) |
| 4 | **B/2** — `points.board` (§7) | ✅ **Done** | servuo-plugins [#4](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/4), link [#18](https://gitea.whitlocktech.com/RunicGateway/link/pulls/18), website [#114](https://gitea.whitlocktech.com/RunicGateway/website/pulls/114), docs [#69](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/69) |
| 5a | **B/3 dependency** — cliloc table (§8.6) | ✅ **Done** | website [#115](https://gitea.whitlocktech.com/RunicGateway/website/pulls/115), docs [#70](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/70) |
| 5b | **B/3** — `vendor.listing` (§8) | ✅ **Done** | servuo-plugins [#5](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/5), link [#19](https://gitea.whitlocktech.com/RunicGateway/link/pulls/19), website [#116](https://gitea.whitlocktech.com/RunicGateway/website/pulls/116), docs [#71](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/71) |
| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | 🟨 In review | the bump: link [#20](https://gitea.whitlocktech.com/RunicGateway/link/pulls/20), website [#117](https://gitea.whitlocktech.com/RunicGateway/website/pulls/117), docs [#72](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/72) — then `edge` → `main`: servuo-plugins [#6](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/6), link [#21](https://gitea.whitlocktech.com/RunicGateway/link/pulls/21), website [#118](https://gitea.whitlocktech.com/RunicGateway/website/pulls/118), docs [#73](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/73) |
Order 5 split in two once §8.6's cliloc dependency turned out to be a client-format problem rather
than a parser (see §8.6). 5a is website-only and lands first so the marketplace ships with real item
names; 5b is the four-repo wire change.
**The `edge` → `main` half of order 6 is held for Android parity** (decided 2026-07-30, see §10): the
app sees none of the four new features and gates shard nav on session role alone, so merging the
cutover first would ship a shard whose app client silently disagrees with the web client about what is
public. The **bump** PRs into `edge` are unaffected and merge normally.
---
@@ -237,6 +247,39 @@ admin-set `uo_link_config.protocol` column — so it happens **exactly once**, a
from 2 to 3, so the cutover doesn't require a manual admin edit. `UOLINK_PROTOCOL` still overrides.
- No feature-negotiation array anywhere — v3 implies all three kinds.
### 4.1 What the bump actually touches
The version lives in five places, and all five move together:
| Where | Change |
|---|---|
| `link/sidecar/src/main.rs` | `PROTOCOL_VERSION` 2 → 3 (with the v3 note beside the v2 one), plus the sidecar README's worked example |
| `website/server/db/schema.sql` | `uo_link_config.protocol` column default 1 → 3, plus the boot migration below |
| `website/server/src/model/uoLinkConfig/uoLinkConfig.model.js` | `DEFAULT_PROTOCOL` — what a site with nothing saved yet declares |
| `website/server/src/utils/uoLinkClient.js` + `uoLinkSocket.js` | the `config.protocol || …` fallbacks, so an unset value can never quietly send `1` and 409 with a confusing message |
| `website/client/.../ShardAdmin.jsx`, `website/.env.example` | the admin form's initial value and the documented env default |
**The migration has to be one-shot, and that is the only subtle part.** `schema.sql` is re-run on
*every* boot (`utils/db.js::ensureSchema`), and every other statement in its migration block is an
idempotent `ADD COLUMN IF NOT EXISTS` / `MODIFY`. A bare `UPDATE uo_link_config SET protocol = 3`
would not be idempotent in the sense that matters: `protocol` is **admin-editable**, so an operator
who deliberately pins an older sidecar in Admin → Shard would silently be un-pinned on the next
restart. It is therefore gated on a marker row in `settings`:
```sql
ALTER TABLE uo_link_config MODIFY COLUMN protocol INT NOT NULL DEFAULT 3;
UPDATE uo_link_config SET protocol = 3
WHERE id = 1 AND protocol < 3
AND NOT EXISTS (SELECT 1 FROM settings WHERE `key` = 'uo_link_protocol_3_migrated');
INSERT IGNORE INTO settings (`key`, value) VALUES ('uo_link_protocol_3_migrated', '1');
```
The marker is written *after* the `UPDATE`, so the first boot on the new build migrates and every
later boot is a no-op. A fresh install has no `uo_link_config` row to update and simply gets the
marker plus the new column default. `protocol < 3` rather than `= 2` so an install that never left
the old default of `1` is carried across too — it could not have been talking to a v2 sidecar
anyway.
---
## 5. Part B/1 — `world.ruleset` ✅ Done
@@ -329,28 +372,33 @@ frame during verification.
**No plugin, no sidecar, no `Bridge.cfg` knob, no new kinds.** Not part of the v3 wire change.
> **Status:** data pipeline landed on `edge` — website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112)
> (parsers, build/import CLI, tables, artifact). API + client pages are the second website PR.
> **Status:** complete on `edge` — website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112)
> (parsers, import CLI, tables) and [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113)
> (the six public routes, the five admin ones, `/site/atlas` + `/site/atlas/:slug`, and the
> Admin → Spawn Atlas panel).
> Part C ships as **two** website PRs, not one: the parsing half is where the correctness risk
> lives, and burying it under routes and React would have meant reviewing it in a 10k-line diff.
> Full operator documentation: [`docs/website/SPAWN_ATLAS.md`](../website/SPAWN_ATLAS.md).
>
> §6.1 below records where the shipped implementation differs from this design. The differences are
> all things the real ServUO data forced, not changes of mind.
> **§6 below is the original design and is partly superseded.** §6.1 records two decisions that were
> rejected in review and replaced (the committed artifact, and the fixed facet list); §6.2 records
> the corrections the real ServUO data forced. Read both before trusting §6.
**Decision: committed generated artifact + idempotent DB import**, split in two because the build
needs the ServUO tree (which the website container does not have) and the import does not. Not
runtime import (10.5 MB of XML per boot), not a browser-served blob.
**Decision (revised at implementation time): the shard's ServUO tree is the single source of truth,
re-derived on every server boot.** The original plan here was a committed generated artifact plus an
idempotent import. That was rejected in review for two reasons, recorded in §6.1: a snapshot in the
repo goes stale as a shard's maps change, and the design leaned on a fixed facet list that no shard
is obliged to keep. Still not a browser-served blob; still parsed server-side only.
New in `website/server/`:
- `src/utils/spawnAtlasParse.js` — **pure functions, no fs**, so they are unit-testable in CI without
a ServUO tree: `parseObjects2()`, `parsePoints()`, `parseRegions()`, `parseLocations()`,
`resolveRegion()`.
- `scripts/buildSpawnAtlas.js` (`--servuo <path> --out db/data/`) and `scripts/importSpawnAtlas.js`
(TRUNCATE + batched INSERT in one transaction); `package.json` scripts `atlas:build`, `atlas:import`.
- `db/data/spawnAtlas.<facet>.json` ×13 + `spawnAtlas.index.json` (creatures, champions, regions,
landmarks, meta with per-source-file hashes).
- ~~`scripts/buildSpawnAtlas.js` and a committed `db/data/spawnAtlas.*.json` artifact~~ — dropped,
see §6.1 R1. Replaced by `src/utils/spawnAtlasSource.js` (the only thing that reads a ServUO tree,
shared by the boot path and the CLI) and a `scripts/importSpawnAtlas.js` that is a thin CLI over
the model. `package.json` gains `atlas:import` only.
- `src/model/shardAtlas/{shardAtlas.db.js,shardAtlas.model.js}` following the `shardState` split.
- `src/router/v1/public/atlas.{router,controller}.js`; `test/spawnAtlas.parse.test.js`.
@@ -381,18 +429,48 @@ stays CLI-only.**
Client: `routes/public/Atlas.jsx` (`/site/atlas`) and `AtlasCreature.jsx` (`/site/atlas/:slug`).
**Payload risk** — a monolithic artifact would be 2–3 MB of committed JSON. Shard per facet and drop
every `<Points>` field the site cannot use (`UniqueId`, all trigger/refractory/proximity/sequential
fields, sound ids), keeping Name/Map/X/Y/W/H/Range/MaxCount/MinDelay/MaxDelay/TOD*/types — well under
1 MB. The artifact never reaches the browser; the browser sees only paginated API responses.
**Payload risk** — *superseded by §6.1 R1; nothing is committed.* The field selection it describes
still applies at parse time: every `<Points>` field the site cannot use (`UniqueId`, all
trigger/refractory/proximity/sequential fields, sound ids) is dropped, keeping
Name/Map/X/Y/W/H/Range/MaxCount/MinDelay/MaxDelay/TOD*/types. Parsed data never reaches the browser;
the browser sees only paginated API responses.
**Operator re-run story** — spawns changed → `npm run atlas:build -- --servuo <path>` on a machine
with the tree → commit the regenerated `db/data/spawnAtlas.*.json` → deploy → `npm run atlas:import`
(or `POST /admin/shard/atlas/import`). `shard_atlas_meta.source` holds per-file hashes, so
`GET /admin/shard/atlas/status` reports when the DB is behind the artifact. Full detail in
`docs/website/SPAWN_ATLAS.md`.
**Operator re-run story** — *revised by §6.1 R1.* Spawns changed → restart, or
`npm run atlas:import` / `POST /admin/shard/atlas/import` to apply without one. `shard_atlas_meta`
holds a sha256 per source file, so the server can tell on boot whether anything changed, and
`GET /admin/shard/atlas/status` reports drift. If the change would remove a facet it is staged for
approval rather than applied (§6.1 R3). Full detail in `docs/website/SPAWN_ATLAS.md`.
### 6.1 What the build against real data changed
### 6.1 What implementation changed
Two design decisions in §6 were rejected in review and replaced; the rest are corrections the real
ServUO data forced. Kept as a diff rather than edited in place, because each is a trap the next
person would otherwise re-enter.
**R1. The committed artifact is gone — the tree is re-parsed on every boot.** §6 proposed building a
generated artifact, committing it, and importing it. Two problems. A shard's maps change over its
life, so a snapshot in the repo silently drifts from the world players actually see; and the build/
import split existed only to work around the website container not having a tree, which is a
deployment question (mount it) rather than a reason to freeze data. The server now hashes the source
files on boot and re-derives the atlas when they differ. `scripts/buildSpawnAtlas.js`, the 1.41 MB
artifact, and the whole encode/decode seam it needed are deleted.
**R2. Nothing may name a facet.** The first implementation carried a lookup table of the six stock
UO facets to reconcile the spelling drift between sources. A shard may add facets, replace them
outright, or rename them when its maps are updated, and a built-in list mishandles all three
silently. Reconciliation is now by *matching* against the facet set discovered from the shard's own
spawn and region data — exact key, then prefix in either direction — with an unmatched name keeping
its own rather than being forced into a wrong bucket.
**R3. Two contracts on the boot path.** It never blocks startup: no path, an unreadable mount, a
malformed file or a database error is caught and logged, and the site comes up serving whatever
atlas it had. And a refresh that would REMOVE a facet is never applied automatically — facet loss
is indistinguishable at boot from a half-copied or mid-update tree, so it is staged in
`shard_atlas_pending` for an admin to approve or reject. Only the decision is stored (source hashes
+ the facet diff, a few KB); approving re-parses, so what lands matches the tree at approval time.
A rejection is remembered against those hashes so it does not re-prompt every restart.
### 6.2 What the build against real data changed
Six corrections to the design above, from running it against stock ServUO 57.4. Kept as a diff
rather than edited in place, because each one is a trap the next person would otherwise re-enter.
@@ -419,12 +497,11 @@ they invent creatures that do not exist *and* split real ones in two, since `Fai
leaves **800** real creatures. (The design's "~1,500 creature rows" estimate was high; 800 only
reinforces the plain-`INDEX`-not-`FULLTEXT` call.)
**5. The artifact is 1.41 MB, not "well under 1 MB".** Dropping the unused `<Points>` fields as the
design directed still left 4.40 MB. Three further encodings — `facet` dropped per record,
default-valued fields omitted rather than written as `0`, and `types` as `[name, max]` tuples
(~24,000 entries × 15 bytes of repeated key names) — brought it to 1.41 MB. Getting under 1 MB would
mean dropping the spawner `name`, which is the only human handle on a specific spawner and worth
keeping. `encodePoint()` and `readPoint()` are exact inverses and are round-tripped in tests.
**5. The artifact would have been 1.41 MB, not "well under 1 MB" — and is now moot.** Dropping the
unused `<Points>` fields as the design directed still left 4.40 MB; three further encodings brought
it to 1.41 MB, and getting under 1 MB would have meant dropping the spawner `name`. The size budget
in §6 was simply optimistic for 6,455 points. Superseded by §6.1 R1: there is no artifact, so there
is no payload to budget and no encode/decode seam to keep in sync.
**6. `DELETE`, not `TRUNCATE`.** The design said "TRUNCATE + batched INSERT in one transaction",
which does not hold: `TRUNCATE` is DDL in MariaDB and implicitly commits, so a mid-import failure
@@ -439,9 +516,46 @@ region, 1,690 by landmark, 1,086 Wilderness).
**One thing the design got exactly right:** the point-in-rect transform really is the reason to
build this. "Where does a lizardman spawn?" answers *Shrines, Isamu-Jima, Yew* across three facets.
### 6.3 What the API/client half added
The second website PR built the six public routes, the five admin ones, `/site/atlas` +
`/site/atlas/:slug`, and the Admin → Spawn Atlas panel. Three things it changed or established:
**1. Respawn delays were being read in the wrong unit — sometimes.** XmlSpawner writes
`MinDelay`/`MaxDelay` in minutes and switches to seconds only when a delay does not divide into
whole minutes, flagging that per record with `DelayInSec`
(`XmlSpawner2.cs:7462-7480`, read back at `:6345-6358`). So a `5` means five *minutes* on one
spawner and five *seconds* on the next, both plausible, and the pipeline stored the raw number.
170 of 6,455 stock spawners are second-flagged — few enough to look like noise on a page and be
believed. The parser now normalises to **seconds**, and the API and UI carry seconds throughout.
*This is the class of bug §6.2 is a list of: the atlas still builds, it is just quietly wrong.*
**2. The hash gate needed a parser version, and this generalises.** Fixing the parse exposed that
"has the tree changed?" is the wrong question on its own — an install whose maps never change would
have kept serving the old readings forever, because the only thing compared was the tree.
`spawnAtlasSource.PARSER_VERSION` is stored in `shard_atlas_meta` beside the source hashes, and a
mismatch counts as drift. Any future parse correction lands on the next boot without an operator
having to know it happened. **Bump it whenever the parser derives different data from identical
files.**
**3. `points` is a count; `spawners` is the list.** The first cut of the detail route spread the
creature row and then set `points` to the array of spawn points — the same key meaning a number on
the search route and an array on the detail route. Renamed before it shipped, and worth recording
because the two names are one letter apart in meaning and it reads as correct.
**On projection.** The `atlas` feature declares no sensitive fields, so `projectFeature` is a no-op
on every one of these routes today. Every handler calls it anyway, per §3.6.1's rule — the point of
the rule is that the *first* field that needs gating is covered by construction rather than by a
retrofit nobody remembers to do.
---
## 7. Part B/2 — `points.board`
## 7. Part B/2 — `points.board` ✅ Done
*Landed on `edge`: servuo-plugins [#4](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/4),
link [#18](https://gitea.whitlocktech.com/RunicGateway/link/pulls/18), website [#114](https://gitea.whitlocktech.com/RunicGateway/website/pulls/114),
docs [#69](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/69). Verified against the real ServUO tree
per §11 — see §7.5 for what that run changed.*
Two deliverables: a diff sweep for the boards, and a `points` block folded into `char.profile` —
the `PROTOCOL_2.md` §10.3 `titles` precedent (read-model enrichment, no new request kind).
@@ -524,9 +638,46 @@ Client — NEW `routes/public/Leaderboards.jsx` at `/site/leaderboards`; a "Loya
added to `components/CharacterSheet.jsx`, one edit serving both `PlayerCharacter.jsx` and
`AdminCharacter.jsx`.
### 7.5 What the run against a real shard changed
The plan above was written from reading `PointsSystem.cs`. Booting the actual shard (ServUO 57.4, a
43,011-mobile world) and letting one sweep run corrected four things — all of them invisible to a
fake-shard test, because a fake shard emits whatever the spec says it should.
1. **`maxPoints` overflowed to `long.MinValue`.** `MaxPoints` is a `double`, and ServUO's idiom for an
uncapped system is `double.MaxValue` — which `DespiseCrystals`, `ShameCrystals` and `VoidPool` all
use. `(long)double.MaxValue` in C# is an **unchecked** conversion: it does not throw, it yields
`long.MinValue`, and the first real sweep published
`"maxPoints": -9223372036854775808` for three of the five live boards. Fixed with `Cap()` /
`Score()` converters that normalise anything unrepresentable to `0`, which is now the wire's
documented **"uncapped"** value. Worth stating plainly because it inverts the obvious reading:
**on a real shard, `maxPoints: 0` is the common case, not an edge case**, so any UI dividing by it
must special-case it.
2. **`nameString` is usually `null`.** Most systems define their `Name` as a cliloc rather than a
literal: four of the five boards on the live shard came back `nameString: null` with only
`nameNumber` set. The humanise-the-`system`-key fallback is therefore the *primary* display path,
not a defensive nicety, and both the leaderboards page and the character sheet lead with it.
3. **`GetEntry`/`GetPoints` cannot be used in the read model.** `GetEntry(from, create: false)` still
calls `AddEntry` when the system has `AutoAdd` (`PointsSystem.cs:207`) — it **mutates the world**.
Ten of the ~25 systems have `AutoAdd = true`, so a profile built with the obvious accessor would
have appended up to ten rows to the points save file every time anyone viewed a character sheet.
`BridgeProfile.WritePoints` hand-rolls a read-only scan instead, and says so loudly.
4. **`players` had to be redefined.** §7.2 called for "the entry count", but those same ten `AutoAdd`
systems hold a zero-point row per character ever created — so the raw count reports the shard's
whole census as one system's participants. It is now the number of players actually holding points,
which is both the honest number and a strictly better diff signal (it moves when someone scores,
not when someone logs in for the first time).
One deviation from the plan as written, for the same class of reason: §7.4 named the per-field
visibility rule `characterName`, but `projectValue` matches on the **literal JSON key**, and the wire
key is `name`. A rule under the descriptive name would have been silently inert — an admin tightening
character names would have got no enforcement and no error, exactly the failure §3.6.1 records for the
flattened `ownerAcct`. `FEATURES.leaderboards.fields` therefore keys on `name`, with a test that fails
if it is renamed back.
---
## 8. Part B/3 — `vendor.listing`
## 8. Part B/3 — `vendor.listing` 🟨 In review
### 8.1 It cannot be an RPC, and this is load-bearing
@@ -607,17 +758,86 @@ admin can turn the stream on. `uoLinkSocket` paginates `/market` on reconnect, b
`/market/vendors/:serial`, behind `requireFeature('market')`. **Rate-limit it** — this is the first
genuinely expensive public endpoint; `express-rate-limit` is already a dependency.
### 8.6 The open dependency — cliloc names
### 8.6 The open dependency — cliloc names ✅ Resolved (shipped ahead of §8)
`CharacterSheet.jsx:14-15` already documents the gap ("without a cliloc table on the site we can only
show literals") and renders equipment as `id {itemId}`. Search-by-name needs that table.
`CharacterSheet.jsx:14-15` documented the gap ("without a cliloc table on the site we can only
show literals") and rendered equipment as `id {itemId}`. Search-by-name needs that table.
- **Recommended:** `scripts/buildClilocs.js` reads the UO client's `Cliloc.enu` → committed
`db/data/clilocs.json`; ingest denormalizes into `shard_vendor_items.display_name`. Same
build-artifact pattern as §6, and it **also fixes the character sheet**.
- **Fallback:** ship with item-art + price + region filters, and name search only over renamed items.
**Resolved as its own website-only change, landed BEFORE the market so `/site/market` ships with real
item names.** Full design and operator guide: [`docs/website/CLILOCS.md`](../website/CLILOCS.md).
Ingest denormalizes the resolved name into `shard_vendor_items.display_name` as planned.
This decision is the reason §8 is sequenced last.
Two things in the original recommendation above turned out to be wrong, and both are worth recording
because the reasoning generalises.
**1. The committed `db/data/clilocs.json` artifact was dropped.** It predates the two Part C
corrections (§6.1) and violates both: no committed snapshot of derived content, and nothing
EA-derived ever shipped. UO's strings are EA's, exactly as the creature sprites are. Replaced with
the §6 pattern instead — parse on every boot from an operator-configured path, hash-gated, output
gitignored, `PARSER_VERSION` counted as drift.
**2. `scripts/buildClilocs.js reads the UO client's Cliloc.enu` is not possible, and the reason
matters.** **Every current client ships its cliloc files COMPRESSED** — all eight `Cliloc.*` files
open with a DWORD whose high byte is `0x8E`, the "Mythic" container. The plain layout (`02 00 00 00
01 00`, then `{int32 number, byte flag, uint16 length, UTF-8}`) is what those files looked like
*before* that change. Parsing a compressed file as plain does not fail cleanly: it yields ~19k
"records" with negative ids, 1,722 distinct keys out of 19,508, one 62 KB "string", and a truncation
somewhere in the middle.
Decompressing means porting an inverse-BWT coder with a 1 KB frequency header — a few hundred lines
whose failure mode is plausible-looking garbage rather than an error. Two facts closed off the
alternatives:
- **ServUO cannot read it either.** Its bundled `Ultima.StringList` implements only the plain layout,
so on a modern client `VendorSearch.StringList` is null and `VendorSearch.GetItemName` returns
`item.Name`. **The in-game Vendor Search gump has the same gap** — which also means §8.2's warning
never to call `GetItemName` in the sweep costs us nothing we could otherwise have had.
- The shard therefore cannot supply names on our behalf, so this could not be pushed to the plugin.
⇒ **the operator converts once, from their own client, and the site reads the result.** Accepted
shapes are the plain binary layout and a `number<TAB|,|;>text` export; the site sniffs which.
`server/tools/cliloc-export/` drives UOFiddler's `Ultima.dll` (the decompressor that already exists)
and writes the plain form. A shard that never converts is fully supported — names render as ids,
exactly as before.
**Shards edit items and add new ones**, and those carry ids no stock client table has — so this reads
a **set** of sources, not one file, hash-gated together and re-read on every boot exactly as §6 reads
the ServUO tree: a base (the converted client table) plus every overlay under `custom/`, later
winning. Adding one custom item therefore never means re-exporting a 5 MB client file. Measured on
the live shard for scale: its script tree references **16,434** cliloc ids and only **37** are absent
from stock — tens against a 67k base, which is why an overlay and not a second table. `custom/` is the
one convention here that is ours rather than the shard's, because **ServUO has no server-side notion
of a custom cliloc**: they live in the patched client a shard distributes, and nothing in the tree
declares them.
That set also brings back a hazard a single file did not have, and §8.6 answers it the way §6 does. A
corrupt source fails the parse loudly, but a source that has **vanished** parses perfectly and imports
a table quietly missing everything it contributed — an unmounted volume is indistinguishable from a
deliberate deletion. So it is **staged, not applied** (`status: 'needsReview'`), reported by both the
import and `status()`, and accepted with `{approve:true}`. It is a flag rather than §6's
approve/reject pair because the atlas stores a pending decision *so that approving re-parses*; here
nothing is stored, so re-reading at approval time is automatic.
Five traps found by building it, all recorded in `CLILOCS.md`:
- **`StringList.SaveStringList` RE-COMPRESSES on save.** It looks exactly like the export path and is
not; its output is byte-identical to its compressed input, because its purpose is round-tripping a
file back into the client.
- **Trimming a text line before splitting silently drops half the table.** Roughly half of a real
cliloc table is empty strings (ids the client reserves), exported as `1005008<TAB>`. Trimming eats
the trailing separator, leaving a bare number that then looks like a header row — 55,994 of 123,490
entries vanished, and the import still looked successful.
- **`Number('')` is `0`, not `NaN`.** A line starting with a separator imports as a bogus cliloc 0
unless the empty field is rejected explicitly.
- **Tidying punctuation unconditionally corrupts real names.** Stripping leftover brackets is right
after a placeholder is removed (`[~1_stuff~]` → nothing) and wrong otherwise: a shard's custom
`"Runic Gateway Sigil (v2)"` rendered as `"(v2"`. Same shape as the `%` rule. **Found only by
running a shard-style overlay through it** — every stock-table fixture passed.
- **Source labels must be forward-slashed and root-relative**, or the same directory fingerprints
differently on Windows and Linux and every boot looks like a change. The identical bug §6 records.
Blank entries are dropped at import (123,490 parsed → **67,496** stored), which also makes the binary
and text paths converge on identical content.
### 8.7 Client
@@ -625,6 +845,75 @@ This decision is the reason §8 is sequenced last.
driven by `staleAt` (the oldest `shard_vendors.updated_at`). The round-robin sweep means data is
inherently up to one full cycle old, and the UI must say so.
Shipped with a second page, `routes/public/MarketVendor.jsx` at `/site/market/vendors/:serial` —
where a search result points. It is the only surface that can render the two states the result list
cannot: a `truncated` shop (*"showing 250 of 3,104 — this shop holds more than the shard
publishes"*) and a `location` an admin has gated away, which is a real answer rather than an empty
coordinate.
### 8.8 What the build changed
Four things the implementation settled differently from §8 as written, all of them found by building
against the live shard.
**1. `location` is a nested object, not flat `map`/`x`/`y`/`region`.** §8.1's payload sketch had them
flat, and it would have made `market.location` — a rule Part A pre-wired — **inert**, exactly like
the `characterName` miss §7.5 records: `projectValue` matches literal JSON keys, so there is no
`location` key for the rule to match. Flat keys would have needed five rules that could drift apart.
Nesting makes one rule hide the facet, the coordinates, the region and the house together, on the
live frame and the stored read model alike, because both now spell it the same way.
The other pre-wired rule, `market.ownerName`, checked out — it is a real key on the frame. Owner is
written as flat `ownerSerial`/`ownerName` rather than through `BridgeJson.Actor`, which would add
`acct` and `webId`; same argument `points.board` makes. `ownerSerial` was **added** to the
configurable fields alongside `ownerName`, because an admin who hides the owner's name and leaves a
serial every other board resolves back to that name has not hidden anything.
**2. The per-vendor diff signature is the full listing set, not §8.3's `count | Σ(serial ^ price)`.**
That hash collides on the single most common change a shop makes: two items swapping prices, which
is what re-pricing looks like. The signature is built over the same buffer the frame is written
from, in the same order, so a match really does mean an identical frame.
**3. There is no `payload` column on `shard_vendors`.** §8.5 implied the board pattern (whole frame
in JSON, columns hoisted for display). It does not apply here: the items ARE the searchable rows, so
they are normalized into `shard_vendor_items` and there is nothing left worth duplicating. The
sidecar keeps the whole blob, because outage resilience is its job and search is not.
**4. Sweep cost is reported, and a slow tick warns.** The batch cap is a *claim* about per-tick cost,
and an operator tuning `MarketSweepBatch` was otherwise tuning blind. `[bridge status` now carries
`lastMs`/`maxMs`, and a tick over 50 ms prints a rate-limited warning naming the knob.
Measured on the live shard (27 vendors × 40 listings, 209k items / 43k mobiles):
| | |
|---|---|
| First tick — 25 vendors emitted cold | **15.4 ms** |
| Second tick — the remaining 2 | **3.4 ms** |
| Steady state — nothing changed | **0.3 ms** |
| Website `/market` search over 1,040 listings | 1,040 total, names resolved |
| Cliloc re-resolution pass over 1,040 rows | **50 ms** |
The diff is what makes the steady state ~free; the batch cap is what bounds the cold case. Note the
arithmetic the warning exists for: at the default cap of 250 listings, a batch of 25 **full** shops
is 6,250 items ≈ 95 ms — over budget. Real shops hold tens, which is why 25 is the default, but a
shard of commodity resellers should lower the batch, and now it will be told to.
Two smaller things worth not rediscovering:
- **`BridgeJson.Escape` takes a NON-NULL string** — it dereferences `value.Length` immediately — and
`BridgeJson.Str` writes its own `,"key":` prefix, so neither serves a value inside a hand-built
object. Nearly everything this frame writes is legitimately null (an item's plain `Name` is null
for almost every item; a vendor in the street has no house), so that is the common path, not an
edge case. `BridgeMarket.Text()` is the two-line writer that was missing.
- **The ServUO console writes in the OS code page**, so an em dash in a `Console.WriteLine` renders
as `???` in the log an operator would paste into an issue. Bridge console output is ASCII.
Search-side, one thing the site had to fix rather than inherit: `%` and `_` in a user's query are
**LIKE** metacharacters, not SQL ones, so parameterization does not neutralize them — a search for
`%` would otherwise match every listing on the shard. `shardMarket.db.js` escapes them. (The atlas's
`LIKE` searches predate this and have the same shape over a much smaller table; worth a follow-up,
not a blocker here.)
---
## 9. Sequencing
@@ -633,10 +922,11 @@ inherently up to one full cycle old, and the UI must say so.
|---|---|---|---|---|
| 1 | **A** — visibility framework + actor-leak fix | website, docs | none | ✅ Done |
| 2 | **B/1** — `world.ruleset` (§5) | all four | new kind | ✅ Done |
| 3 | **C** — spawn atlas (§6) | website, docs | none | 🟡 Pipeline done, API/client next |
| 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | ⬜ |
| 5 | **B/3** — `vendor.listing` (§8) | all four | new kinds | ⬜ |
| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | ⬜ |
| 3 | **C** — spawn atlas (§6) | website, docs | none | ✅ Done |
| 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | ✅ Done |
| 5a | **B/3 dependency** — cliloc table (§8.6) | website, docs | none | ✅ Done |
| 5b | **B/3** — `vendor.listing` (§8) | all four | new kinds | ✅ Done |
| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | 🟨 In review — `edge` → `main` held for Android parity (§10) |
---
@@ -653,13 +943,32 @@ inherently up to one full cycle old, and the UI must say so.
in the security section.
- NEW `website/SHARD_VISIBILITY.md` — admin-facing: what each feature exposes, what each rung means,
what cannot be loosened.
- NEW `website/SPAWN_ATLAS.md`, NEW `website/MARKETPLACE.md`.
- NEW `website/SPAWN_ATLAS.md`, NEW `website/CLILOCS.md`, NEW `website/MARKETPLACE.md`.
- `PROJECT_TREE.md` in each touched repo.
- `npm run swagger` **and** `npm run routes:manifest` on every route-touching PR — both are committed
artifacts, and `test/routeManifest.test.js` fails on drift.
**Follow-up, not scoped for 3.0:** the Android app consumes the same public/player shard API and will
need `/public/shard/features` to hide its own nav. Track separately against `android-app/`.
**Android parity — now scoped, and it gates the cutover (decided 2026-07-30).** This was written as a
"track separately" follow-up. It was re-examined before the cutover and the gap is wider than nav
hiding: the app consumes the same public/player shard API but has **no consumer for any of the four new
features** (`ruleset`, `leaderboards`, `market`, `atlas`), no `points` block on its character sheet, no
cliloc-resolved item names (§8.6), and — the part that matters for §3 — **it gates shard navigation on
session role alone**, so an admin who disables a feature or raises its audience leaves the app
rendering entries that `404`/`403` into a generic error where the web client hides them.
Two things were verified as already correct and are recorded so they are not re-derived: the app's SSE
request rides the same authenticated OkHttp client as every other call, so an app session resolves to
the same audience rung as the same account on the web; and every shard DTO in the app is
nullable-with-defaults, so field projection strips fields without a deserialization failure.
Scoped as **M11 in [`../android/PLAN.md`](../android/PLAN.md) §9**, two PRs: the visibility rules +
read-model adds ([Android-app #30](https://gitea.whitlocktech.com/RunicGateway/Android-app/pulls/30))
and the four screens ([#31](https://gitea.whitlocktech.com/RunicGateway/Android-app/pulls/31), stacked
on it). Both are **built and in review**; the on-device five-rung walk (§11) against a website on the
cutover branch is the remaining gate. `edge` → `main` is held until both land, so web and app surface
the same shard on the same day. Neither PR is coupled to the merge order — on a pre-v3 website
every new route and `/public/shard/features` `404`s and the app falls back to today's behavior — so
holding the cutover is a schedule decision, not a technical dependency.
---

197
website/BACKEND_DESIGN.md
View File

@@ -101,6 +101,10 @@ server/
deliberately not site-mode gated
shard.router.js (14) /public/shard/* incl. the anonymous
SSE stream; never site-mode gated
atlas.router.js (6) /public/atlas/* — the spawn atlas.
NOT under /shard: nothing here
touches the sidecar, and unlike
/shard it IS site-mode gated
site.router.js (4) /settings /status /version /contact —
the group-root singletons; declares no
router-level middleware
@@ -375,6 +379,78 @@ they are cheap to display — the same payload-plus-hoisted-columns shape `shard
served as `null` rather than `{}`: "not published yet" and "published, everything off" are different
answers and the page renders them differently.
### shard_points_boards — points / loyalty leaderboards (Protocol 3.0)
One row per point system, keyed by the shard's own `PointsType` name (`QueensLoyalty`,
`CleanUpBritannia`, …). The shard carries ~25 of these, each a standing players build over months.
Columns: `system` (PK), `name`, `name_cliloc`, `max_points`, `players`, `show_on_gump`, `payload` JSON
(the whole `points.board` frame), `t`, `updated_at`.
**The top-N list stays inside `payload`** rather than being normalized into a `shard_points_entries`
table. It is a fixed-size list (10 by default) that is only ever read whole — exactly like
`shard_governors.candidates` — so normalizing buys nothing until something needs a per-character
reverse lookup, and a character's own standings already ride inside `char.profile` instead.
Board state, not events: `points.board` is **not** in `LOGGED_KINDS`, for the same reason
`guild.update` isn't. The shard emits a frame every time anyone's score moves a top ten, so logging
would grow `shard_events` without bound for something whose only interesting value is its latest
version. There is also **no delete path** — the shard's set of systems is fixed at startup, so there is
no `points.remove` to mirror.
Two values carry non-obvious meanings, both set by the plugin and both documented in
[`link/INTEGRATION.md`](../link/INTEGRATION.md) §4:
- **`max_points = 0` means uncapped**, and on a real shard that is the *common* case (ServUO's
uncapped idiom is `double.MaxValue`, which the plugin normalises to 0). Anything rendering
`points / max_points` must special-case it.
- **`name` is usually NULL**, with `name_cliloc` set instead — most systems name themselves with a
cliloc rather than a literal. Listing therefore orders by `COALESCE(name, system)`, so boards
awaiting cliloc resolution sort by their own key rather than clumping together under NULL.
### shard_vendors / shard_vendor_items — the player-vendor marketplace (Protocol 3.0)
The shard-wide shop index, fed by `vendor.listing` / `vendor.listing.remove`. One row per player
vendor and one per priced listing. Full operator detail in [`MARKETPLACE.md`](MARKETPLACE.md); the
design is `docs/link/v3.md` §8.
| Table | Shape |
|---|---|
| `shard_vendors` | `serial` (PK), `shop_name`, `owner_serial`, `owner_name`, `map`/`x`/`y`/`z`, `region`, `house`, `item_count`, `item_total`, `truncated`, `t`, `updated_at`. Indexes on owner, map, region and `updated_at`. |
| `shard_vendor_items` | `id` (PK), `vendor_serial`, `serial`, `item_id`, `hue`, `amount`, `price`, `name`, `cliloc`, `display_name`, `child`. Indexes on `vendor_serial`, `price`, `item_id`, `display_name`, and `(display_name, price)`. |
**Ingest is per-vendor and authoritative**: the frame is the whole shop, so ingest is
delete-then-insert of that vendor's listings inside one transaction. All-or-nothing matters
specifically because the two writes are "the shop" and "what is in it" — a failure between them
leaves a shop advertising an inventory it no longer has, which is visibly wrong and indistinguishable
from a genuinely empty shop. No foreign keys, consistent with every other `shard_*` table.
**There is deliberately no `payload` column**, unlike `shard_points_boards` directly above. The
board's top-N is a fixed-size list read whole, so it lives in JSON; here the items *are* the
searchable rows, so they are normalized and nothing is left worth duplicating. The sidecar keeps the
whole blob — outage resilience is its job, search is ours.
Market state, not events: neither kind is in `LOGGED_KINDS`, and this is the strongest case of the
three v3 kinds. One frame carries up to 250 listings and the sweep re-emits a shop on any price
change, so logging would turn `shard_events` into a price history nobody reads.
Two columns carry non-obvious meanings:
- **`item_count` vs `item_total`.** `item_count` is what the frame published; `item_total` is what
the shop actually holds. They differ when `truncated` — the shard caps listings per frame
(`Bridge.MarketMaxListings`, 250 by default), and a commodity reseller with thousands of stacks
genuinely exceeds it. Any UI must show both or it presents a partial shop as complete.
- **`display_name` is denormalized at ingest**, resolved from the item's literal `name` (preferred —
a player set it, so it is more specific) else its `cliloc` against `shard_clilocs`. Resolving at
query time would put the cliloc table on the hot path and make search-by-name impossible. Because
the shard's diff sweep will not re-send an unchanged shop just because the site learned what its
items are called, **a cliloc import triggers a bulk re-resolution** of this column (after a boot
import and after an admin import; ~50 ms per thousand rows, never throws).
`updated_at` is written explicitly on every upsert rather than left to `ON UPDATE CURRENT_TIMESTAMP`,
which MariaDB does not fire when every column is written back unchanged. A shop re-published
identically is still *freshly confirmed*, and without this the staleness banner would age a perfectly
current shop forever.
### shard_feature_visibility — per-feature audience config (Protocol 3.0)
One row per shard feature: `feature` (PK), `enabled`, `audience` (a rung on the ladder in §6.5),
@@ -391,9 +467,15 @@ discarded. See §6.5.
### shard_spawn_* / shard_regions / shard_landmarks / shard_champion_spawns / shard_atlas_meta — the spawn atlas (Protocol 3.0)
Static shard **content**, not live shard state. Nothing here comes from the sidecar: the atlas is
built from a ServUO tree by a CLI script, committed as JSON under `server/db/data/`, and loaded with
`npm run atlas:import`. These tables stay populated whether the shard is up or not. Full operator
detail in [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md); the design is `docs/link/v3.md` §6.
derived from the shard's own ServUO tree, re-read on **every server boot** and hash-gated so an
unchanged tree costs one read pass and no write. Nothing is precomputed and committed — a shard's
maps change over its life, and a snapshot in the repo would silently drift from the world players
actually see. These tables stay populated whether the shard is up or not. Full operator detail in
[`SPAWN_ATLAS.md`](SPAWN_ATLAS.md); the design is `docs/link/v3.md` §6.
**No facet name appears anywhere in the code.** A shard may add facets, replace them, or rename them
when its maps are updated; the facet set is discovered from the tree, and the loose spellings in
`Data/Locations` are matched against it rather than looked up in a table.
| Table | Key columns |
|---|---|
@@ -403,13 +485,32 @@ detail in [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md); the design is `docs/link/v3.md` §
| `shard_regions` | `facet`, `name`, `type`, `priority`, `parent`, `rects` JSON |
| `shard_landmarks` | `facet`, `name`, `grp`, `x`, `y`, `z` |
| `shard_champion_spawns` | `slug` PK, `name`, `grp`, `type`, `random_type`, `facet`, `x`, `y`, `z`, `radius`, `label` |
| `shard_atlas_meta` | Singleton (`id = 1`), `payload` JSON, `imported_at` |
| `shard_atlas_meta` | Singleton (`id = 1`), `payload` JSON (counts, a sha256 per source file, `parserVersion`), `imported_at` |
| `shard_atlas_pending` | Singleton (`id = 1`), `status` (`pending`/`rejected`), `payload` JSON, `detected_at` |
All of them are **import-owned**: `atlas:import` empties and reloads every one inside a single
transaction, so a failed import leaves the previous atlas intact rather than a half-loaded world.
The first seven are **import-owned**: a refresh empties and reloads every one inside a single
transaction, so a failed reload leaves the previous atlas intact rather than a half-loaded world.
Nothing else writes to them, and nothing holds a foreign key to them — no FKs at all, consistent with
every other `shard_*` table.
**`shard_atlas_pending` is the security-relevant one.** A refresh that would REMOVE a facet is never
applied automatically: facet loss is indistinguishable at boot from a half-copied or mid-update tree,
so it is staged here for an admin to approve or reject, and **startup is never blocked by it**. Only
the decision is stored — source hashes plus the facet diff, a few KB — and approving re-parses the
tree, so a multi-megabyte blob never lands in the database and what gets applied matches the tree at
approval time. A rejection is remembered against those exact hashes so a declined refresh does not
re-prompt on every restart. Everything else (new facets, renamed regions, changed spawns) applies
immediately, since none of it can destroy data an operator would miss.
The boot refresh is **best-effort by contract**: no configured path, an unreadable mount, a malformed
file or a database error is caught and logged, and the site comes up serving whatever atlas it had.
The tree path comes from the `spawn_atlas_servuo_path` setting, falling back to `SERVUO_PATH`.
**A refresh re-derives when the tree changed OR the parser did.** `spawnAtlasSource.PARSER_VERSION`
is stored in `shard_atlas_meta` beside the source hashes and bumped whenever the parser produces
different data from identical files. Hashing the tree alone would strand an install whose maps never
change on whatever an older build derived — a corrected parse would ship and never reach the data.
Four column choices worth stating, because each one is a trap:
- **`spawn_range`, not `range`**, and **`grp`, not `group`** — both are reserved words.
@@ -429,6 +530,64 @@ artwork: sprites live in the operator's own client `.mul`/`.uop` files and are t
redistribute. An operator supplies art via a gitignored map plus images under the (already
gitignored) `server/uploads/atlas/`. Text-only is the normal, supported state.
### shard_clilocs / shard_cliloc_meta — UO's localization table (Protocol 3.0)
Items on the wire carry a `LabelNumber`, not a name. The bridge has always sent it —
`char.profile.equipment.cliloc`, reward titles as a cliloc number in string form, and one per
marketplace listing — but with no table to resolve it against, the character sheet could only render
`id 1023721` where the game renders "quarter staff".
| Table | Shape |
|---|---|
| `shard_clilocs` | `number` INT PK, `flag`, `text` TEXT |
| `shard_cliloc_meta` | Singleton (`id = 1`), `payload` JSON (source file, sha256, count, `parserVersion`), `imported_at` |
Import-owned and all-or-nothing in one transaction, same contract as the atlas — including **`DELETE`,
not `TRUNCATE`**, for the same reason.
**Sourced from files the operator supplies**, at a path from the `cliloc_client_path` setting falling
back to `UO_CLIENT_PATH`. Nothing client-derived is committed: UO's strings are EA's, exactly as the
creature sprites are. A shard with nothing configured is fully supported — names render as ids. Full
design and operator guide: [`CLILOCS.md`](CLILOCS.md).
**It reads a SET of sources, not one file**, because shards edit items and add new ones and those
carry cliloc ids no stock client table has. A base (the converted client table) plus every overlay
under `custom/` are re-read on every boot and hash-gated **together**, exactly as the atlas re-reads
`Regions.xml` + `Locations/*.xml` + `Spawns/*.xml` + `ChampionSpawns.xml`. Later sources win, so an
overlay both adds ids and overrides stock ones, and adding one custom item never means re-exporting a
5 MB client file. Scale, measured on the live shard: its script tree references 16,434 cliloc ids and
only 37 are absent from stock — tens of entries against a 67k base, which is why this is an overlay
and not a second table.
The conversion step is not avoidable: **every current client ships its cliloc files compressed**
(first DWORD's high byte `0x8E`), and ServUO's own bundled `Ultima.StringList` cannot read that
either — so the shard cannot supply names on our behalf. The plain layout and a delimited text export
are both accepted, sniffed by header rather than extension.
Three decisions worth stating:
- **`text` is TEXT, not VARCHAR.** Long property descriptions reach 12 KB. The index that matters for
marketplace search is the denormalized `shard_vendor_items.display_name`, not this table.
- **Blank entries are dropped at import** — 123,490 parsed → **67,496** stored. Roughly half a cliloc
table is empty strings for ids the client reserves and never uses; a row that resolves to no name is
indistinguishable from no row at all, and dropping them makes the binary and text imports converge
on identical content.
- **Two refusals, one of them the atlas's.** A corrupt source fails the parse on a truncated record,
so it is caught outright and leaves the previous table serving. But a source that has **vanished**
parses perfectly and imports a table quietly missing everything it contributed — an unmounted volume
and a deliberate deletion are indistinguishable from here, which is precisely the ambiguity the
atlas stages a facet removal for. So it is escalated: `status: 'needsReview'`, nothing applied,
`missingSources` reported by both the import and `status()`, and an admin accepts it with
`{approve:true}`. That is a flag rather than the atlas's approve/reject pair because the atlas
stores a pending decision so that approving **re-parses** the tree; here nothing is stored, so
re-reading at approval time is automatic.
**Resolution is server-side and there is no public route.** The table is never served *as* a table:
67k rows would dwarf any page using them, and the Android client consumes the same already-resolved
JSON. `resolveMany()` returns only ids that resolved to something displayable — placeholders like
`~1_val~` are stripped, since the bridge sends the id and never the property packet that carries the
arguments — and it never throws, because a cliloc lookup is decoration on a character sheet.
---
## 4. API contract
@@ -443,7 +602,7 @@ are authoritative, and they answer different questions:
| Artifact | Source of truth for | Generated by |
|---|---|---|
| `server/routes.manifest.json` — mirrored as [api-route-inventory.json](./api-route-inventory.json) | **What URLs exist.** 200 public routes + 2 on the internal listener, sorted, method + path only. | `npm run routes:manifest`, by walking the live Express stack |
| `server/routes.manifest.json` — mirrored as [api-route-inventory.json](./api-route-inventory.json) | **What URLs exist.** 215 public routes + 2 on the internal listener, sorted, method + path only. | `npm run routes:manifest`, by walking the live Express stack |
| `server/swagger/swagger-output.json` — served at `/api/docs` | **What each route means.** Parameters, bodies, response codes, security. | `npm run swagger`, from `#swagger.*` annotations |
The split is deliberate: Swagger is annotation-derived, so an unannotated route is invisible in it and
@@ -628,7 +787,18 @@ from the per-route **siteMode** middleware (§5), never from an auth gate.
| GET | `/wiki/:slug` | single page |
| POST | `/contact` | (rate-limited) send mail via SMTP; if unconfigured, respond `{fallback:"mailto", email}` |
| GET | `/shard/ruleset` | the shard's own published ruleset (Protocol 3.0 `world.ruleset`): expansion, which optional systems are on, skill/stat caps, account and house limits, champion scroll rules, the save/restart schedule. Served from `shard_ruleset`, so it renders while the shard is down; live via `world.ruleset` on `/shard/stream`. Behind `requireFeature('ruleset')`. **`null`** means the shard has never published one — a real answer, distinct from a published ruleset. `caps.skill` / `caps.totalSkill` are in **tenths** (1000 = 100.0). |
| GET | `/shard/points` | every points/loyalty leaderboard the shard publishes (Protocol 3.0 `points.board`) — Queen's Loyalty, Void Pool, the nine city loyalties, Clean Up Britannia, … Served from `shard_points_boards`, so it renders while the shard is down; live via `points.board` on `/shard/stream`. Behind `requireFeature('leaderboards')`, ordered by display name. **`maxPoints: 0` means uncapped** (the common case), and `nameString` is usually `null` with `nameNumber` holding a cliloc — resolve client-side or humanise the `system` key. |
| GET | `/shard/points/:system` | one board by the shard's `PointsType` name (e.g. `QueensLoyalty`); `:system` must match `/^[A-Za-z][A-Za-z0-9_]{0,47}$/` or **400** before any query runs. **404** = the shard has never published that system, which is distinct from a published board nobody has scored in yet (**200** with an empty `top`). |
| GET | `/shard/market?q=&minPrice=&maxPrice=&itemId=&map=&region=&sort=&limit=&offset=` | search the player-vendor marketplace (Protocol 3.0 `vendor.listing`). Returns **listings**, not vendors — "who sells X and for how much" is the question, and a vendor-shaped result would make every caller flatten the shops back out. Served from `shard_vendors` + `shard_vendor_items`, so it renders while the shard is down. Behind `requireFeature('market')` **and rate-limited** — the first genuinely expensive public read on the site (a `LIKE` scan plus a `COUNT` over what is typically the largest `shard_*` table, reachable with no session). `sort ∈ {price_asc, price_desc, recent}`. `q` matches the resolved display name **or** the item's literal name, with `%`/`_` escaped: they are `LIKE` metacharacters, not SQL ones, so parameterization alone would let `?q=%` match every listing on the shard. Every response repeats `staleAt` (the oldest vendor row) because the shard sweeps round-robin — a banner that ages with the results it labels, not one fetched once. |
| GET | `/shard/market/meta` | index size, staleness (`staleAt`/`freshAt`) and which facets and regions actually hold vendors, so a client builds its filters without running a search it will discard. |
| GET | `/shard/market/vendors/:serial` | one shop and its listings; `:serial` must match `/^0x[0-9A-Fa-f]{1,16}$/` or **400** before any query runs. **404** = a serial the index has never seen, which also covers a vendor since dismissed or hidden — to an anonymous caller those are the same answer, and distinguishing them would leak that a hidden vendor exists. `truncated` (with `total` exceeding `count`) means the shop holds more than the shard publishes per frame. |
| GET | `/shard/features` | the shard features **this caller** may reach plus the audience rung they resolved to (§6.5), so a client hides nav it can't follow. Reports only what the caller can see — the list itself never discloses a gated feature. Consumed by the SPA header and (pending) the Android nav. |
| GET | `/atlas/creatures?q=&facet=&limit=&offset=` | the bestiary, most numerous first, with an unpaginated `total`. Static content parsed from the shard's ServUO tree — **not** sidecar-backed, which is why the atlas sits outside `/shard`, and unlike `/shard/*` it **is** site-mode gated. Behind `requireFeature('atlas')`. `?facet=` is matched exactly and never validated against a list (no facet name exists in the code); the filter is an `EXISTS` over the points rather than a JSON path or `JSON_SEARCH` built from caller input, whose `%`/`_` wildcards would make `?facet=%` match everything. |
| GET | `/atlas/creatures/:slug` | one creature: `places` (the point-in-rect aggregate — "lizardman → Shrines, Isamu-Jima, Yew"), `spawners` (the bounded raw list, with `spawnersTruncated`), `alsoHere`. **`points` is a COUNT and `spawners` is the LIST** — named apart so one key never means a number on one route and an array on another. `minDelay`/`maxDelay` are in **seconds**, normalised at parse time from the source's per-record minutes-or-seconds. 404 = no such creature in this atlas. |
| GET | `/atlas/regions?facet=&q=` | named regions and the rectangles that placed each spawner |
| GET | `/atlas/landmarks?facet=&q=` | points of interest, labelled by `group` ("Covetous", not "Level 1") |
| GET | `/atlas/champions?facet=` | the **configured** altar roster. Not `/shard/champs`, which is the live board. |
| GET | `/atlas/meta` | facets, counts and when the atlas was parsed. Game-world facts only — the ServUO path, source hashes and any pending refresh are operator detail and live on the admin route. |
Public content GETs pass through the **siteMode** gate (§5).
@@ -672,6 +842,13 @@ file a route sits in — that is the property the route manifest freezes.
| GET | `/users/:id/trusted-devices` | list a user's active trusted devices (never tokens) |
| DELETE | `/users/:id/trusted-devices` · `…/:deviceId` | revoke all / one of a user's trusted devices (logs `admin.trusted_device.revoke[_all]`) |
| POST | `/users/:id/mfa/reset` | recover a locked-out user: disable TOTP + revoke all trusted devices + clear recovery codes (logs `admin.user.totp.reset`) |
| GET | `/shard/atlas` | spawn-atlas status (`adminOnly`): the ServUO path, whether the tree is readable, whether it has drifted from what is loaded, counts, facets, and any refresh staged for review. The public `/atlas/meta` reports the game world only; the filesystem detail is here. |
| POST | `/shard/atlas/import` | re-import without restarting; `{force}` ignores the hash gate. **An unreadable tree answers 200 with `status:"unavailable"`, not 500** — `refresh()` reports outcomes rather than throwing (the boot path must never be blocked by a bad tree) and that contract is preserved at the API. |
| POST | `/shard/atlas/approve` · `/shard/atlas/reject` | answer a refresh staged because it would REMOVE a facet. Approving **re-parses** the tree, so what lands matches it at approval time; rejecting is remembered against those source hashes so it does not re-prompt every restart. 404 when nothing is staged. |
| PUT | `/shard/atlas/path` | point the atlas at a different tree (persisted as `spawn_atlas_servuo_path`, which wins over `SERVUO_PATH`). Blank clears it. Deliberately **does not import** — moving the mount and reloading the world are separate decisions — and returns fresh status so the panel can offer the import next. |
| GET | `/shard/clilocs` | cliloc-table status (`adminOnly`): every source found now (base first, then `custom/` overlays in merge order), what each contributed at the last import, readability, drift across the set, the entry count, and `missingSources`. `configured:false` is a supported state — item names then render as ids. No public counterpart: the table is never served *as* a table. |
| POST | `/shard/clilocs/import` | reload after a client patch or an overlay edit; `{force}` ignores the hash gate, `{approve}` accepts a **vanished** source (refused by default — see the table notes above). **A missing path — or the likely mistake of pointing at the client's own COMPRESSED `Cliloc.enu` — answers 200 with `status:"unavailable"` and a `code`, not 500.** `COMPRESSED` is called out by name: a 500 would say only "something broke", and the operator needs to be told which file to convert. |
| PUT | `/shard/clilocs/path` | point the site at a different cliloc base file or directory (persisted as `cliloc_client_path`, which wins over `UO_CLIENT_PATH`). Overlays are read from `custom/` beside it either way. Blank clears it. Deliberately **does not import**, same reasoning as the atlas path. |
Every admin write logs to `activity_log`.
@@ -705,7 +882,11 @@ who"; `activity_log` provides the history feed.
- **Cookie**: `httpOnly`, `sameSite=Lax`, `path=/`, and **`secure` decided per-request** (`COOKIE_SECURE=auto` → `secure: req.secure`).
- **Trusted-device MFA.** A second, separate httpOnly cookie (`rg_trust`, default 30d) — opaque, sha256-hashed server-side in `trusted_devices` — lets a browser/app **skip the TOTP step** (never the password) on future logins. It is a server-side, per-row-revocable record (never a JWT claim), so the stateless session JWT is unchanged and trust stays revocable. It only ever gates the **second factor**; it deliberately outlives logout, and is cleared on untrust / password change / password reset / TOTP disable. **Recovery codes** (bcrypt, single-use) are the 2FA-lockout fallback. All admin trusted-device/MFA actions and the self actions (`auth.login.trusted_device`, `account.trusted_device.*`, `account.recovery_code*`, `admin.trusted_device.*`, `admin.user.totp.reset`) are audit-logged. See `docs/website/TRUSTED_DEVICES_MFA.md`. This is the key to dual access: the cookie is `Secure` when reached through Pangolin (HTTPS, `X-Forwarded-Proto: https`) but **not** `Secure` when reached directly over the LAN IP on plain HTTP — so login works in both. `COOKIE_SECURE=true|false` can force it. Requires `trust proxy` (below). `localhost:5173` (Vite) and `localhost:3000` are same-site, so the cookie flows in dev too.
- **bcrypt** hashing (cost 10+); plaintext passwords never stored, logged, or returned.
- **Rate limiting** (`express-rate-limit`) on `/auth/login` and `/public/contact`.
- **Rate limiting** (`express-rate-limit`) on `/auth/login`, `/public/contact`, and — the only limited
*read* — `/public/shard/market` and `/public/shard/market/vendors/:serial` (60/min/IP). Every other
public read is an indexed lookup of bounded size; the marketplace search is a `LIKE` scan plus a
`COUNT` over the largest `shard_*` table, anonymous by default, so it is the one public GET that is
worth money to serve.
- **Validation** (`express-validator`) on all writes; centralized error handler.
- **helmet** with a Content-Security-Policy tuned for the built React SPA. The policies now live in
**`server/src/config/csp.js`** (`app.js` only wires them up):

296
website/CLILOCS.md Normal file
View File

@@ -0,0 +1,296 @@
# Cliloc table (item and title names)
**Status:** Complete on `edge` — website [#115](https://gitea.whitlocktech.com/RunicGateway/website/pulls/115), docs [#70](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/70).
**Design:** [`docs/link/v3.md` §8.6](../link/v3.md) — Protocol 3.0, the dependency Part B/3 was sequenced behind.
A "cliloc" is UO's localization table: an integer id mapped to a display string.
**Items on the wire carry a `LabelNumber`, not a name.** The bridge has always
sent that number — `char.profile.equipment` has a `cliloc` field, reward titles
arrive as a cliloc number in string form, and every marketplace listing carries
one — but the site had no table to look it up in, so a character sheet could only
render `id 1023721` where the game renders **"quarter staff"**.
The number was never the missing piece. The table was.
## Why the operator has to convert the file
This is the awkward part, and it is not avoidable:
**Every current UO client ships its cliloc files compressed.** All eight
`Cliloc.*` files in a modern client (`chs`, `cht`, `deu`, `enu`, `esp`, `fra`,
`jpn`, `kor`) begin with a DWORD whose high byte is `0x8E` — the "Mythic"
compressed container. The plain layout this site parses is what those files
looked like *before* that change.
Decompressing it means an inverse-BWT coder with a frequency header — a few
hundred lines of bit-level work whose failure mode is plausible-looking garbage
rather than an error. The site has no business carrying that at runtime.
Two facts make the alternatives worse, not better:
- **ServUO cannot read it either.** Its bundled `Ultima.StringList` implements
only the plain layout, so on a modern client `VendorSearch.StringList` is null
and `VendorSearch.GetItemName` returns `item.Name` — usually nothing. The
shard cannot supply names on our behalf; the in-game Vendor Search gump has the
same gap.
- **Nothing client-derived may be committed.** UO's strings are EA's. The repo
ships no string table for the same reason it ships no artwork and no map
snapshot — see [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md).
So the conversion happens **once, on the operator's machine, against their own
client**, and the site reads the result from a path it is given. A shard that
never does this is in a fully supported state: names render as ids, exactly as
they did before the table existed.
## Converting
Either format below is accepted; the site sniffs which one it was handed.
| Format | Fidelity | Notes |
|---|---|---|
| **Plain binary** (recommended) | Exact | 6-byte header, then `{int32 number, byte flag, uint16 length, UTF-8}` records |
| Delimited text | Loses leading/trailing whitespace | `number<TAB\|,\|;>text` per line; a header row, blank lines and `#` comments are ignored |
The whitespace caveat is real but cosmetic: ~1,300 of the 123,490 entries in a
stock `Cliloc.enu` are label prefixes like `"max = "` whose trailing space is
meaningful when the client concatenates a value onto them. Nothing on this site
concatenates, and every consumer passes through `displayText()`, which trims.
### Using the bundled tool
`server/tools/cliloc-export/` is a small .NET console app that drives
[UOFiddler](https://github.com/polserver/UOFiddler)'s `Ultima.dll` — the
decompressor that already exists and is already maintained — and writes the plain
format. It loads that DLL **reflectively** so it compiles against any SDK, and it
writes the records by hand because UOFiddler's own `SaveStringList` *re-compresses*
on save (its purpose is round-tripping a file back into the client, so its output
is byte-identical to its input — a trap worth knowing about).
```bash
cd website/server/tools/cliloc-export
dotnet build -c Release
# binary (recommended)
dotnet run -- "<UOFiddler>/Ultima.dll" "<UO client>/Cliloc.enu" /srv/uo-data/clilocs.plain
# or tab-delimited
dotnet run -- "<UOFiddler>/Ultima.dll" "<UO client>/Cliloc.enu" /srv/uo-data/clilocs.tsv --tsv
```
A UOFiddler GUI export works equally well — anything producing one of the two
shapes above is fine.
## Shard-added and shard-edited items
**Shards edit items and add new ones**, and those carry cliloc ids no stock
client table has. The table is therefore built from a **set** of sources, all
re-read on every boot and hash-gated together — the same shape as the spawn
atlas, which reads `Regions.xml` + `Locations/*.xml` + `Spawns/*.xml` +
`ChampionSpawns.xml` and merges them:
```
<cliloc path>/
clilocs.plain ← base: the converted client table
custom/
01-uomysticmoon.tsv ← overlays: shard additions and overrides
02-events.tsv
```
Overlays use the same delimited-text format, are read in **sorted order**, and
**later sources win** — so an overlay both *adds* ids the client never had and
*overrides* stock ones the shard has re-purposed. Any `.tsv`, `.csv`, `.txt`,
`.enu` or `.plain` file in `custom/` is picked up; anything else (a `README.md`,
say) is ignored.
Adding, editing or removing any overlay counts as drift, so a new custom item
needs only a file edit and a restart — or the admin panel's Import button.
**Adding one item never means re-exporting a 5 MB client file.**
The import result reports what each source contributed, which is how you confirm
an overlay took effect — `overrode: 0` on a file meant to re-label stock items
says it did not:
```json
"sources": [
{ "label": "clilocs.plain", "kind": "base", "entries": 123490, "added": 123490, "overrode": 0 },
{ "label": "custom/uomysticmoon.tsv", "kind": "custom", "entries": 2, "added": 1, "overrode": 1 }
]
```
**Why a convention rather than discovery.** Everywhere else this pipeline follows
the shard's own files, but **ServUO has no server-side notion of a custom
cliloc** — they live in the patched client a shard distributes to its players,
and nothing in the tree declares them. There is nothing to discover, so `custom/`
is the one thing here that is our convention rather than the shard's. (An
operator who *does* patch their client cliloc needs no overlay at all: convert
the patched file and their edits are simply in the base.)
Measured on the live shard for scale: its script tree references **16,434** cliloc
ids and only **37** are absent from the stock client table — tens of entries
against a 67k base, which is what makes an overlay the right shape rather than a
second full table.
## Configuring the path
Two ways to point at the sources, the setting winning over the environment:
| Source | Notes |
|---|---|
| `cliloc_client_path` setting | Admin-editable (Admin → Shard); takes effect on the next refresh without a redeploy |
| `UO_CLIENT_PATH` env var | The deploy-time default, since the path usually describes a mount the deployment sets up |
The value may be **the base file itself or a directory to search**, because both
are natural answers to "where is it". Overlays are read from a `custom/`
directory beside the base **either way** — pointing at a file does not forfeit
them.
A directory is searched case-insensitively (the client writes `Cliloc.enu` on
Windows; the site usually runs on Linux) for, in order: `clilocs.tsv`,
`clilocs.csv`, `clilocs.plain`, `cliloc.plain`, `cliloc.plain.enu`,
`cliloc.enu.plain`, `clilocs.txt`, `cliloc.enu`.
That ordering puts explicitly-converted names first on purpose. Pointing the
setting straight at an unconverted client directory finds `cliloc.enu`, which is
compressed — and the site says so by name rather than failing obscurely:
```
status: unavailable
code: COMPRESSED
reason: This is a compressed (Mythic-format) cliloc file, which the site cannot
read. Convert it to the plain format first — see docs/website/CLILOCS.md.
```
## Refresh contract
Identical in shape to the spawn atlas, and for the same reasons:
- **It never blocks startup.** No path, an unreadable file, a wrong-format file,
a database error — all caught and logged. The site comes up either way.
- **Hash-gated.** The boot path hashes the file and skips the parse entirely when
it matches what is loaded, which is every restart that did not follow a client
patch. Measured on a stock table: **14 ms** for the no-op, **663 ms** for a full
parse and replace.
- **A `PARSER_VERSION` bump also counts as drift**, so a corrected parse reaches
an install whose client never patches.
### Two ways a refresh is refused
**A corrupt file** — the realistic failure for any single source — makes the
parser fail on a truncated record rather than yield a plausible-but-short table,
so it is caught outright. Verified: a file truncated to half its length reports
```
code: TRUNCATED
reason: Truncated record header at byte 2486759 (74909 entries read)
```
and the rows already loaded are untouched. A malformed overlay names the file it
came from (`custom/broken.tsv: No cliloc entries found…`), because "which of my
six overlay files is broken" is otherwise a guessing game.
**A source that has VANISHED** is the hazard a single file did not have. It
parses perfectly and imports a table quietly missing everything that file
contributed — and an unmounted volume looks exactly like a deliberate deletion
from here. This is the same ambiguity the atlas stages a facet removal for, so it
is escalated rather than applied:
```
status: needsReview
reason: 1 previously-loaded cliloc source(s) are missing;
the existing table is unchanged
missingSources: ["custom/uomysticmoon.tsv"]
```
`status()` reports `missingSources` too, so the panel can show it before anyone
clicks Import. An admin accepts it by re-running the import with
`{ "approve": true }`.
**Why that is a flag and not the atlas's approve/reject pair.** The atlas stores
a pending decision in its own table so that approving *re-parses the tree*, which
is what keeps a multi-megabyte blob out of the database and makes the applied
result match the tree at approval time. Here nothing is stored, so re-reading at
approval time is automatic — the decision is a single boolean on the import an
admin was already going to run.
## What gets stored
| | |
|---|---|
| Parsed from a stock `Cliloc.enu` | **123,490** entries |
| Of those, empty strings | **55,994** (ids the client reserves and never uses) |
| Stored in `shard_clilocs` | **67,496** |
Blank entries are dropped at import. A row resolving to no name is
indistinguishable from no row at all to every caller, and dropping them makes the
binary and text imports converge on **identical** content — the binary format
carries the blanks explicitly and a text export may or may not, depending on the
tool. Verified: both formats import to the same 67,496 rows with the same keys.
`text` is `TEXT`, not `VARCHAR`: the long property descriptions reach 12 KB, and
silently truncating them would be worse than storing them. The index that matters
for marketplace search is on the denormalized `shard_vendor_items.display_name`,
not here.
## How names are applied
**Resolution happens server-side.** The table is never served *as* a table and
there is no public route for it. Two reasons: 67k rows would dwarf any page that
used them, and the Android client consumes the same JSON and would otherwise need
its own copy.
`resolveMany()` takes a batch of ids and returns a `Map` holding only those that
resolved to something displayable, so "no such id" and "id with no usable name"
collapse into one branch at the call site. It never throws — a cliloc lookup is
decoration on someone's character sheet, and a database blip must not fail the
sheet. A capped in-process cache fronts it; measured cold **4.2 ms**, warm
**0.015 ms**.
### `displayText()`
Cliloc strings interpolate arguments the client pulls from an item's property
list — `~1_val~`, `~2_NAME~`. **We never have those**: the bridge sends the id,
not the packet. So a name carrying them is reduced to what is actually knowable.
| Raw | Displayed |
|---|---|
| `quarter staff` | `quarter staff` |
| `cold damage ~1_val~%` | `cold damage` |
| `[~1_stuff~]` | *(nothing — the whole string was the argument)* |
| `50%` | `50%` |
| `Runic Gateway Sigil (v2)` | `Runic Gateway Sigil (v2)` |
**Punctuation is only tidied when a placeholder was actually removed.** The
trailing `%` in row two is the unit belonging to the number we never had, and the
brackets in row three only ever wrapped the argument — but a string with no
placeholder has no such debris, and trimming it anyway corrupts real names. Rows
four and five are the ones that caught it: a shard's custom
`"Runic Gateway Sigil (v2)"` rendered as `"(v2"` while the bracket trim was
unconditional.
### Consumers
- **Character sheet equipment.** `enrichCharProfile` attaches `clilocName` to each
item. A player-given `name` always wins — "Bob's lucky axe" must not be
relabelled "hatchet" — and the client re-states that precedence.
- **Reward titles.** `titles.rewardResolved` is a parallel array with the numeric
entries turned into words (`null` where nothing resolved). The sheet used to
*skip* numeric reward titles entirely, having no way to render them.
- **Marketplace listings** (Protocol 3.0 §8) denormalize the resolved name into
`shard_vendor_items.display_name` so search can index it.
## Admin surface
All admin-only, alongside the atlas under Admin → Shard:
| Route | Purpose |
|---|---|
| `GET /api/v1/admin/shard/clilocs` | Sources found, what each contributed at the last import, readability, drift, entry count, `missingSources` |
| `POST /api/v1/admin/shard/clilocs/import` | Reload after a client patch or an overlay edit; `{ "force": true }` reimports an unchanged set, `{ "approve": true }` accepts a vanished source |
| `PUT /api/v1/admin/shard/clilocs/path` | Set the path; blank disables resolution |
A refresh **result is not an exception**: a missing file, or the likely mistake of
pointing at the client's own compressed `Cliloc.enu`, answers `200` with
`status: "unavailable"` and a reason. A `500` would say only "something broke";
the operator needs to be told which file to convert. Setting the path
deliberately does **not** import as a side effect — the response carries the
refreshed status so the panel can offer that as the next step.

167
website/MARKETPLACE.md Normal file
View File

@@ -0,0 +1,167 @@
# Marketplace — the player-vendor index
**Status:** On `edge` — servuo-plugins [#5](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/5), link [#19](https://gitea.whitlocktech.com/RunicGateway/link/pulls/19), website [#116](https://gitea.whitlocktech.com/RunicGateway/website/pulls/116).
**Design:** [`docs/link/v3.md` §8](../link/v3.md) — Protocol 3.0 Part B/3.
**Depends on:** [`CLILOCS.md`](CLILOCS.md) — without a cliloc table, listings render as item ids.
The marketplace is a searchable index of every player vendor on the shard: what
each shop is selling, for how much, and where it is standing. It is the same set
the in-game **Vendor Search** gump reads, offered from outside the game — so a
player can find the vanquishing kryss they want before logging in, and someone
who does not play at all can see that the economy exists.
Page: `/site/market`, plus `/site/market/vendors/:serial` for one shop.
## Three things the pages must say out loud
Everything below follows from how the data is gathered, and each has a visible
consequence the UI is required to surface.
**1. The prices are not live.** The shard sweeps vendors **round-robin** — at
most `Bridge.MarketSweepBatch` shops per tick — so a given shop can be a full
cycle behind. The page carries a *"prices last refreshed N minutes ago"* banner
driven by the **oldest** vendor row, not the newest: the one stale shop is the
one that wastes somebody's trip.
**2. A shop can be truncated.** `Bridge.MarketMaxListings` (250 by default) caps
how many listings one frame carries. A commodity reseller with thousands of
stacked resources is a real thing, and an uncapped frame for one is measured in
megabytes. Over the cap the shop reports `truncated`, and the vendor page says
*"showing 250 of 3,104 — this shop holds more than the shard publishes"* rather
than presenting a partial shop as complete.
**3. An item may have no name.** Items on the wire carry a cliloc id, not a name.
On a shard whose operator has not converted a cliloc table
([`CLILOCS.md`](CLILOCS.md)) the honest render is the item id — never an invented
label, which would be indistinguishable from a real one.
## Privacy: the player's own toggle wins
Only vendors whose owner left the in-game **Vendor Search** flag ON are ever sent
to the site. A player who hides their shop in game is hidden here too, and no
admin setting overrides that. When they hide one that was already indexed, the
shard emits `vendor.listing.remove` and the row is deleted — so revoking consent
takes effect, it does not merely stop refreshing.
Shop name, owner character name and location default to **Everyone**, because the
stock Vendor Search gump already shows exactly that set to any player in game.
They remain admin-configurable; see [`SHARD_VISIBILITY.md`](SHARD_VISIBILITY.md).
Account names and website user ids never cross the wire at all.
## How it is put together
```
ServUO uo-link sidecar website
────── ─────────────── ───────
BridgeMarket.cs vendors table shard_vendors
round-robin sweep ──────► (whole frame blob) ──────► shard_vendor_items
per-vendor diff GET /market (paged) + display_name
vendor.listing resolved at ingest
vendor.listing.remove
```
**The shard side** walks at most `MarketSweepBatch` vendors per tick from a
persistent cursor, diffs each against what it last published, and emits a whole
frame for any shop that moved. Per-tick cost is therefore bounded by the batch,
not by how many vendors the world holds — full coverage takes
`ceil(vendors / batch) × MarketSweepSeconds`.
**The sidecar** stores each frame whole and serves `GET /market`, its only paged
read. It normalizes nothing and defines no audiences: it is a dumb forwarder, and
search is the website's job.
**The website** splits each frame into a vendor row and its listings, replacing
that vendor's whole listing set inside one transaction (the frame is
authoritative for that vendor, never a delta). Item names are resolved against
the cliloc table **on the way in** and stored denormalized, which is what makes
search-by-name possible and keeps the cliloc table off the hot path.
## Operating it
Everything is in `Config/Bridge.cfg` on the shard. There is nothing to configure
on the website.
| Setting | Default | What it does |
|---|---|---|
| `MarketEnabled` | `true` | Master switch. Off publishes nothing; the page shows an empty index. |
| `MarketSweepSeconds` | `60` | Tick interval. |
| `MarketSweepBatch` | `25` | Vendors inventoried per tick. Clamped 1..500. |
| `MarketMaxListings` | `250` | Per-shop listing cap, after which `truncated`. Clamped 1..5000. |
**Faster coverage vs. per-tick cost.** Lowering `MarketSweepSeconds` or raising
`MarketSweepBatch` both refresh the index sooner and both cost more per tick.
The expensive part is the item walk, which recurses into every container a vendor
is selling — so a shard of big shops should raise the interval rather than the
batch.
`[bridge status` reports the sweep, including `lastMs` and `maxMs`:
```
market(enabled=True sweeps=42 scanned=108 emitted=27 removed=0 skipped=0
truncated=0 tracked=27 vendors=27 cursor=2 batch=25 lastMs=0.31 maxMs=15.40)
```
A tick over **50 ms** prints a rate-limited warning naming the knob:
```
[Bridge] market sweep took 82.4 ms (budget 50 ms) - lower Bridge.MarketSweepBatch (now 25) if this persists
```
Measured on a shard with 27 vendors × 40 listings (209k items, 43k mobiles):
**15.4 ms** for the first cold tick of 25 vendors, **0.3 ms** in steady state —
the diff is what makes an unchanged world nearly free. Note the arithmetic: 25
*full* shops at the 250-listing cap is 6,250 items ≈ 95 ms, over budget. Real
shops hold tens, which is why 25 is the default and why the warning exists.
`[bridge sweepnow` runs one tick immediately; `[bridge reload` re-reads the
settings above without a restart.
## Names arriving late
Item names come from the cliloc table, and the market sweep will **not** re-send
an unchanged shop just because the site learned what its items are called. So a
cliloc import triggers a bulk re-resolution of every stored listing — otherwise
an operator who configures clilocs after the first sweep would see item ids until
every shop happened to change on its own. It runs after a boot import and after
an admin import, takes ~50 ms per thousand listings, and never throws: a failure
leaves names exactly as they were.
## API
All under `/api/v1/public/shard`, gated by the `market` feature and
**rate-limited** — these are the first genuinely expensive public reads on the
site (a `LIKE` scan plus a `COUNT` over what is typically the largest `shard_*`
table, reachable with no session).
| Route | What |
|---|---|
| `GET /market` | Search. Returns **listings**, not vendors — "who sells X and for how much" is the question. `?q=&minPrice=&maxPrice=&itemId=&map=&region=&sort=&limit=&offset=`, `sort ∈ {price_asc, price_desc, recent}`. |
| `GET /market/meta` | Index size, staleness, and which facets and regions actually hold vendors — so a client builds its filters without running a search it will discard. |
| `GET /market/vendors/:serial` | One shop and its listings. **404** for a serial the index has never seen, which also covers a vendor since dismissed or hidden — to an anonymous caller those are the same answer. |
`q` matches the resolved display name **or** the item's own literal name, because
an item with a player-set name (most of what is worth searching for on a
player-run shard) may carry a generic cliloc. `%` and `_` in a query are escaped:
they are `LIKE` metacharacters, not SQL ones, so parameterization alone would let
a search for `%` match every listing on the shard.
Full schemas are in the OpenAPI spec (`ShardMarketPage`, `ShardMarketVendor`,
`ShardMarketMeta`, `ShardMarketListing`, `ShardMarketLocation`).
## Tables
`shard_vendors` (one row per shop) and `shard_vendor_items` (one row per priced
listing). Both are ingest-owned; nothing else writes to them. No foreign keys,
in keeping with every other `shard_*` table — the ingest transaction is what
keeps them consistent, and an FK would turn a malformed frame into a failed write
rather than a dropped row.
There is deliberately **no `payload` column** on `shard_vendors`, unlike the
points board next door. The board's top-N is a fixed-size list read whole, so it
lives in JSON; here the items *are* the searchable rows, so they are normalized
and there is nothing left worth duplicating. The sidecar keeps the whole blob,
because outage resilience is its job.
`shard_vendor_items.display_name` is denormalized and indexed (alone, and
composite with `price` for "cheapest matching X"). See "Names arriving late"
above for how it is kept current.

22
website/SHARD_VISIBILITY.md
View File

@@ -60,12 +60,32 @@ board while holding back one column. See the table in §3.
| **Shard rules** | Skill/stat caps, house limits, vet rewards, the ruleset | Everyone | Connect address → Everyone |
| **Spawn atlas** | Bestiary and spawn locations (static content) | Everyone | — |
| **Leaderboards** | Point and loyalty standings | Everyone | Character names → Everyone |
| **Marketplace** | The shard-wide player-vendor index | Everyone, **live updates off** | Vendor owner name → Everyone · Location → Everyone |
| **Marketplace** | The shard-wide player-vendor index | Everyone, **live updates off** | Vendor owner name → Everyone · Vendor owner character id → Everyone · In-game location → Everyone |
**Why the marketplace ships with live updates off.** A live feed of every vendor's full inventory
would be the single largest thing the site sends. No page needs it — the marketplace is a search over
stored data with a “prices last refreshed N minutes ago” stamp. Turn it on only if you want it.
**Why the marketplace's fields default to Everyone.** A vendor's shop name, its owner's character
name and where it is standing are *already* visible to every player in game: the stock Vendor Search
gump surfaces exactly that set to anyone who opens it. Publishing them on the site is not a new
disclosure. They stay configurable because a shard may still prefer to keep its economy behind a
login — and because "already public in game" is a judgement about your shard, not ours.
**Location is one setting covering four things.** Hiding it removes the facet, the coordinates, the
region *and* the house name together. That is deliberate: those are four ways of saying the same
thing, and a setting that hid the coordinates while publishing the house name would not have hidden
anything.
**Hiding the owner name also hides the owner character id.** They are separate settings so you can
be explicit, but leaving the id published while hiding the name achieves nothing — the leaderboards
and guild boards resolve that same id back to a character name. Set both.
**What hiding a vendor cannot do.** Only vendors whose owner left the in-game *Vendor Search* flag ON
are ever sent to the site, so a player who hides their shop in game is hidden here too — and no
setting on this page can override that. It works the other way as well: these settings control who
sees the index, not whether players can find each other's shops in game.
**Why house owner/price default to Staff.** The public Houses page has always been a "where are the
falling houses" board — location only. Owner and price are the staff view. That split is preserved.

321
website/SPAWN_ATLAS.md
View File

@@ -1,6 +1,6 @@
# Spawn atlas
**Status:** Data pipeline landed on `edge` (website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112)); API and client pages follow in a second PR.
**Status:** Complete on `edge` — data pipeline in website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112), API + pages in website [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113).
**Design:** [`docs/link/v3.md` §6](../link/v3.md) — Protocol 3.0 Part C.
The spawn atlas is a browsable catalogue of what the shard *contains*: which
@@ -8,6 +8,19 @@ creatures spawn, where, how many, and which champion altars are configured. It
answers "where do I find a lizardman?" with **"Shrines, Yew, Isamu-Jima"** rather
than with a list of raw coordinates.
## Two things that shape the whole design
**The shard's ServUO tree is the single source of truth.** Nothing is
precomputed and committed to the repository. A shard's maps change over its
lifetime — facets get added, replaced, or renamed — and a snapshot in the repo
would silently drift from the world players actually see. The atlas is therefore
re-derived from the tree **on every server boot**.
**Facets are not a fixed list.** Nothing in the codebase names Felucca, Trammel,
or any other stock facet. The facet set is whatever the shard's own files
declare, discovered at parse time. A shard running entirely custom maps gets
exactly the same treatment as a stock one, with no code change.
## What it is not
The atlas is **static shard content, not live shard state.**
@@ -24,44 +37,83 @@ The atlas is **static shard content, not live shard state.**
Routes live at `/api/v1/public/atlas`, deliberately **not** under `/shard`,
because `/shard/*` means sidecar-dependent.
## The two commands
## Configuring the tree
Building needs a ServUO tree. Importing does not. That split is the whole design:
the website container ships the committed artifact but has no ServUO tree, so it
can import but never build.
The website needs to be able to *read* the ServUO tree — same host, a bind mount,
or a shared volume. Two ways to point at it, the setting winning over the
environment:
```bash
# On a machine that has the ServUO tree (writes server/db/data/spawnAtlas.*.json)
cd website/server
npm run atlas:build -- --servuo /path/to/ServUO
| Source | Notes |
|---|---|
| `spawn_atlas_servuo_path` setting | Admin-editable; changes take effect on the next refresh without a redeploy |
| `SERVUO_PATH` env var | The deploy-time default, since the path usually describes a mount the deployment sets up |
# Anywhere, including the deployed container
npm run atlas:import
With neither set the atlas is simply skipped — the site runs normally without
one.
## The boot path
On every start the server hashes the source files and compares them against what
is loaded. Unchanged (the normal case on a restart) costs one read pass, ~120 ms,
and no database write. A real change costs a ~400 ms parse and a reload.
Two contracts govern it:
**1. It never blocks startup.** No configured path, an unreadable mount, a
malformed file, a database error — every one is caught and logged, and the site
comes up serving whatever atlas it already had.
**2. A facet disappearing is never applied automatically.** Losing a facet looks
exactly like a half-copied or mid-update tree, and boot cannot tell that apart
from a real map change. That refresh is *staged* for a human instead. Everything
else — new facets, renamed regions, changed spawns — applies immediately, since
none of it can destroy something an operator would miss.
```
boot
└─ path configured? no ──▶ skip
└─ tree readable? no ──▶ warn, carry on
└─ hashes changed? no ──▶ done (nothing parsed)
└─ parse
└─ a facet would be removed?
no ──▶ import
yes ──▶ stage for admin review; atlas unchanged
```
`atlas:build` flags:
### Approving or rejecting a staged refresh
| Flag | Default | Meaning |
|---|---|---|
| `--servuo` | *(required)* | ServUO server root — the directory holding `Spawns/`, `Data/`, `Config/` |
| `--out` | `server/db/data` | Where to write the artifact |
| `--landmark-radius` | `200` | Max tile distance for the landmark fallback |
Only the *decision* is stored, never the parsed world — a few KB of source hashes
plus the facet diff. Approving **re-parses** the tree, so what lands matches the
tree at approval time rather than at boot, and a multi-megabyte blob never sits
in the database.
`atlas:import` takes `--dir` (default `server/db/data`).
A rejection is remembered against those exact source hashes, so a declined
refresh does not re-prompt on every restart. Change the tree and the hashes
differ, which asks again.
## Operator re-run story
From **Admin → Spawn Atlas**, or from the CLI:
Spawns changed → rebuild → commit → deploy → import.
```bash
cd website/server
npm run atlas:import -- --status # what is loaded, and what is pending
npm run atlas:import -- --approve # apply the staged refresh
npm run atlas:import -- --reject # keep the current atlas, dismiss it
```
1. `npm run atlas:build -- --servuo <path>` on a machine with the tree.
2. Commit the regenerated `server/db/data/spawnAtlas.*.json`.
3. Deploy.
4. `npm run atlas:import`, or `POST /api/v1/admin/shard/atlas/import`.
## The CLI
`shard_atlas_meta` stores a sha256 per source file, so
`GET /api/v1/admin/shard/atlas/status` reports when the database is behind the
committed artifact. **Build stays CLI-only** — there is no admin button that
reads a ServUO tree.
The server refreshes itself on boot, so this is for applying a map change
*without* a restart, and for the approve/reject flow above.
```bash
npm run atlas:import # import if the tree differs
npm run atlas:import -- --servuo <path> # override the path for this run
npm run atlas:import -- --force # reimport even if unchanged
```
`--servuo` is a per-run override and deliberately does **not** persist — changing
where the atlas permanently reads from is an admin action, not a side effect of a
one-off import.
## Sources
@@ -72,11 +124,10 @@ reads a ServUO tree.
| `Data/Locations/*.xml` | 6 files | Landmarks (dungeon levels, town markers) |
| `Config/ChampionSpawns.xml` | 4.8 KB | Configured champion altars |
**There are 13 spawn files but only 6 facets.** `Eodon.xml`,
**A stock tree has 13 spawn files but only 6 facets.** `Eodon.xml`,
`GravewaterLake.xml`, `TreasuresOfKotl.xml` and the other named-area files hold
TerMur/Trammel points. The facet always comes from each record's own `<Map>`,
never from the file name, and the artifact shards by facet — Felucca, Trammel,
Ilshenar, Malas, Tokuno, TerMur.
never from the file name.
## How a coordinate becomes a place name
@@ -85,16 +136,17 @@ This is the transform the atlas exists for, in `resolveRegion()`:
1. The highest-`priority` named region whose rectangle contains the point. Ties
break toward the **smallest** rect, so a specific room wins over the
dungeon-wide rect enclosing it.
2. Otherwise the nearest landmark within `--landmark-radius` tiles, labelled by
its **group** ("Covetous"), not its individual marker ("Level 1").
2. Otherwise the nearest landmark within the landmark radius (200 tiles by
default), labelled by its **group** ("Covetous"), not its individual marker
("Level 1").
3. Otherwise `"Wilderness"`.
The radius cap in step 2 is what keeps step 3 reachable. Without it the nearest
landmark is always *some* landmark however far away, and open countryside gets
labelled with a dungeon on the far side of the map.
Against stock ServUO this resolves **83.2%** of points (5,369 of 6,455): 3,689 by
region, 1,690 by landmark, 1,086 Wilderness.
Against stock ServUO this resolves **83.2%** of points (5,369 of 6,455): 3,681 by
region, 1,688 by landmark, 1,086 Wilderness.
## Three quirks in the source data
@@ -102,11 +154,17 @@ Each of these is silent if unhandled — the atlas still builds, it is just wron
**Facet names disagree between sources.** `Data/Locations/*.xml` spells them
`Ter Mur` and `Tokuno Islands`, while `<Map>` and `<Facet name>` say `TerMur` and
`Tokuno`. Unreconciled, the landmark bucket for those two facets is keyed
differently from the points looking it up, so the fallback never fires and every
unregioned spawn in Ter Mur and Tokuno reads "Wilderness". All facet names are
canonicalised through `normalizeFacet()`; unknown facets pass through unchanged
so a custom shard facet still gets an atlas.
`Tokuno`. Unreconciled, the landmark bucket is keyed differently from the points
looking it up, so the fallback never fires and every unregioned spawn on those
facets reads "Wilderness".
This is reconciled **by matching, not by a lookup table** — there is no list of
facet names anywhere. `facetKey()` collapses spelling differences (lowercase,
alphanumerics only), and `resolveFacetName()` matches a loose spelling against
the canonical set discovered from the shard's own spawn and region data, by exact
key then by prefix in either direction. A name matching nothing keeps its own
name: forcing a wrong match would file a real custom facet's landmarks under the
wrong facet, which is worse than leaving it alone.
**Spawn type tokens carry XmlSpawner directives.** The `<Objects2>` type is not
always a bare class name:
@@ -125,38 +183,35 @@ real creatures.
file and `lizardman` in another. Slugging collapses them correctly, but the
display name is chosen deterministically — most common spelling wins, ties break
to the more capitalised form, then alphabetically — because otherwise it would
depend on file read order and produce a spurious artifact diff on every unrelated
rebuild.
depend on file read order and change on an unrelated restart.
## Artifact format
## Tables
`server/db/data/`, all committed:
All are **import-owned**: a refresh empties and reloads them in one transaction,
so a failed reload leaves the previous atlas intact rather than a half-loaded
world. Nothing else writes to them and nothing holds a foreign key to them — no
FKs at all, consistent with every other `shard_*` table. Full column listings in
[`BACKEND_DESIGN.md`](BACKEND_DESIGN.md).
| File | Contents |
|---|---|
| `spawnAtlas.meta.json` | Indented. Build time, counts, sha256 per source file |
| `spawnAtlas.index.json` | Compact. Facets, creatures, regions, landmarks, champions |
| `spawnAtlas.<facet>.json` × 6 | Compact. That facet's spawn points |
| Table | Rows (stock) | Notes |
|---|---|---|
| `shard_spawn_creatures` | 800 | `slug` PK; `total` = sum of each type's own max; nullable `art` |
| `shard_spawn_points` | 6,455 | `spawn_range`, since `range` is reserved in MariaDB |
| `shard_spawn_point_types` | 23,927 | The many-to-many; one spawner commonly carries six types |
| `shard_regions` | 387 | Flattened out of the nesting; `rects` JSON |
| `shard_landmarks` | 558 | `grp`, since `group` is reserved in SQL |
| `shard_champion_spawns` | 25 | Configured altars, not the live feed |
| `shard_atlas_meta` | 1 | Singleton; source hashes, for the change check |
| `shard_atlas_pending` | 0–1 | Singleton; a staged refresh awaiting admin review |
**1.41 MB total**, down from 4.40 MB. The design budgeted "well under 1 MB",
which turned out optimistic for 6,455 points; three encodings closed most of the
gap:
`shard_spawn_creatures.name` carries a plain `INDEX`, deliberately **not
`FULLTEXT`**: ~800 rows makes a `LIKE` scan free, and FULLTEXT's minimum token
length would break searches for names like "orc".
- `facet` is dropped per record — the shard file names it once at the top.
- Fields at their default are omitted rather than written as `0`. Most spawners
are a single point with no time-of-day gating, so `width`, `height`, `range`
and the three `tod*` fields are zero on the large majority of records.
- `types` are `[name, max]` tuples. There are ~24,000 type entries and
`{"type":"Orc","max":1}` spends 15 bytes apiece restating two key names that
never vary.
`label` is not stored at all — it is exactly `region || landmark || "Wilderness"`
and the importer recomputes it.
`buildSpawnAtlas.encodePoint()` and `importSpawnAtlas.readPoint()` are exact
inverses, round-tripped in `test/spawnAtlas.build.test.js`. **Change one, change
both.** The artifact never reaches the browser; the browser sees only paginated
API responses.
The reload uses `DELETE`, not `TRUNCATE` — `TRUNCATE` is DDL in MariaDB and would
implicitly commit, defeating the all-or-nothing guarantee. Point ids are assigned
explicitly rather than left to `AUTO_INCREMENT`, because the join rows need them
and `conn.batch()` reports no usable `insertId`.
## Artwork — operator-supplied, never shipped
@@ -175,41 +230,22 @@ An operator who wants art:
2. Drops the images under `server/uploads/atlas/`.
3. Copies `server/db/data/spawnAtlas.art.example.json` to `spawnAtlas.art.json`
and maps creature slugs to file names.
4. Re-runs `npm run atlas:import`.
4. Restarts, or runs `npm run atlas:import -- --force`.
Both `spawnAtlas.art.json` and `server/uploads/` are gitignored, so neither the
map nor the images can be committed by accident.
## Tables
## Code layout
All six are **import-owned**: `atlas:import` empties and reloads them in one
transaction. Nothing else writes to them and nothing holds a foreign key to them
— no FKs at all, consistent with every other `shard_*` table. Full column
listings in [`BACKEND_DESIGN.md`](BACKEND_DESIGN.md).
| File | Role |
|---|---|
| `src/utils/spawnAtlasParse.js` | **Pure and fs-free** parsers, so CI covers them with no ServUO tree. Zero dependencies. |
| `src/utils/spawnAtlasSource.js` | The only thing that reads a ServUO tree; shared by the boot path and the CLI |
| `src/model/shardAtlas/shardAtlas.db.js` | The one-transaction replace |
| `src/model/shardAtlas/shardAtlas.model.js` | The refresh decision, staging, approve/reject |
| `scripts/importSpawnAtlas.js` | Thin CLI over the model |
| Table | Rows (stock) | Notes |
|---|---|---|
| `shard_spawn_creatures` | 800 | `slug` PK; `total` = sum of each type's own max; nullable `art` |
| `shard_spawn_points` | 6,455 | `spawn_range`, since `range` is reserved in MariaDB |
| `shard_spawn_point_types` | 23,927 | The many-to-many; one spawner commonly carries six types |
| `shard_regions` | 387 | Flattened out of the nesting; `rects` JSON |
| `shard_landmarks` | 558 | `grp`, since `group` is reserved in SQL |
| `shard_champion_spawns` | 25 | Configured altars, not the live feed |
| `shard_atlas_meta` | 1 | Singleton (`id = 1`); source hashes for the drift check |
`shard_spawn_creatures.name` carries a plain `INDEX`, deliberately **not
`FULLTEXT`**: ~800 rows makes a `LIKE` scan free, and FULLTEXT's minimum token
length would break searches for names like "orc".
The importer uses `DELETE`, not `TRUNCATE` — `TRUNCATE` is DDL in MariaDB and
would implicitly commit, defeating the all-or-nothing reload. Point ids are
assigned explicitly rather than left to `AUTO_INCREMENT`, because the join rows
need to know them and `conn.batch()` reports no usable `insertId`.
## Parsing notes
`server/src/utils/spawnAtlasParse.js` is **pure and fs-free**, so CI covers it
with no ServUO tree. It adds **zero dependencies**.
Parsing notes:
- `Regions.xml`, `Locations/*.xml` and `ChampionSpawns.xml` genuinely nest, and
get a small hand-rolled **subset** tokenizer — elements, attributes,
@@ -223,3 +259,96 @@ with no ServUO tree. It adds **zero dependencies**.
- `<Objects2>` is `Type:MX=n:SB=…` segments joined by `:OBJ=`. Split on `:OBJ=`
*first* — a naive `split(':')` shreds it. A single Trammel point carries six
types.
- **Respawn delays are stored in two different units, per record.** XmlSpawner
writes `MinDelay`/`MaxDelay` in minutes, and switches to seconds only when a
spawner's delay does not divide into whole minutes — flagging that with
`DelayInSec` on the same record. A `5` therefore means five *minutes* on one
spawner and five *seconds* on the next, and both are plausible respawn times,
so a reader assuming either unit is silently wrong about the other. Stock
ServUO 57.4 has ~170 second-flagged spawners out of 6,455. The parser
normalises everything to **seconds**; the API and UI carry seconds throughout.
### The parser version
`spawnAtlasSource.js` exports `PARSER_VERSION`, stored in `shard_atlas_meta`
alongside the source hashes and bumped whenever the parser derives **different
data from identical files** — a fixed misreading, a new field, a changed unit.
A refresh re-derives when the tree changed **or** the parser did. Hashing the
tree alone would be a trap: an install whose maps never change would keep serving
whatever an older build derived, indefinitely, and a deploy that corrects the
parse would never reach the data. A version mismatch counts as drift, so the
correction lands on the next boot without an operator having to know it happened.
## The API
Everything is served from MariaDB. Nothing on this path touches the sidecar, so
the pages stay complete while the shard is down — which is why the routes sit at
`/api/v1/public/atlas` and **not** under `/public/shard`, where a prefix means
"sidecar-dependent". Unlike `/shard/*`, they *are* `siteMode`-gated, like
`/posts` and `/wiki`: a bestiary is site content and follows site content's rules.
Every route carries `requireFeature('atlas')` — **404** when an admin has
disabled the feature (its pages must not reveal that it exists) and **403** when
the caller sits below its configured audience. The default is `anonymous`, so the
gates are inert until an admin changes something. Responses are field-projected
like every other shard read; `atlas` declares no sensitive fields today, and the
projection call is there so the first one that does is covered by construction
rather than by a retrofit ([`v3.md` §3.6.1](../link/v3.md)).
| Route | Answers |
|---|---|
| `GET /atlas/creatures?q=&facet=&limit=&offset=` | The bestiary, most numerous first, paginated with an unpaginated `total` |
| `GET /atlas/creatures/:slug?facet=&points=` | One creature: `places`, `spawners`, `alsoHere` |
| `GET /atlas/regions?facet=&q=` | Named regions and their rectangles |
| `GET /atlas/landmarks?facet=&q=` | Points of interest, labelled by `group` |
| `GET /atlas/champions?facet=` | The configured altar roster |
| `GET /atlas/meta` | Facets, counts and when the atlas was parsed |
Two shapes worth knowing:
- **`places` is the aggregate the atlas exists for.** "Lizardman → Shrines,
Isamu-Jima, Yew", grouped in SQL rather than by summing 6,455 point rows in
Node. `spawners` is the raw list underneath it, bounded, with
`spawnersTruncated` saying when it was cut.
- **`points` is a COUNT, `spawners` is the LIST.** The two are named apart
deliberately: the same key meaning a number on the search route and an array on
the detail route is the kind of thing a client only discovers in production.
`GET /atlas/meta` reports the **game world only**. The ServUO path, the per-file
hashes and any pending refresh describe the operator's filesystem, and live on
the admin route instead.
A facet is never validated against a list — nothing in the codebase names one.
`?facet=` is length-bounded and matched exactly, so an unknown name returns an
empty result rather than an error. The filter is an `EXISTS` over the points and
deliberately not a JSON path or `JSON_SEARCH` built from caller input: that
function treats `%` and `_` as wildcards, which would make `?facet=%` match
everything.
## The admin panel
**Admin → Spawn Atlas** (`/admin/shard-atlas`, admin-only — it reads a path on
the server's filesystem and replaces every atlas table, which is closer to a
deploy action than to moderation).
| Route | Does |
|---|---|
| `GET /admin/shard/atlas` | Status: path, readable, drift, counts, facets, pending |
| `POST /admin/shard/atlas/import` | Import now; `{ force: true }` ignores the hash gate |
| `POST /admin/shard/atlas/approve` | Apply a staged refresh, facet loss and all |
| `POST /admin/shard/atlas/reject` | Keep the current atlas; remember the decision |
| `PUT /admin/shard/atlas/path` | Point the atlas at a different tree |
Three behaviours that are deliberate:
- **An unreadable tree is a 200, not a 500.** `refresh()` reports outcomes rather
than throwing, because the boot path must never be stopped by a bad tree, and
that contract is preserved at the API. The panel says *"The tree could not be
read: …"*; a 500 would say only that something broke.
- **Setting the path does not import.** Moving the mount and reloading the world
are separate decisions, and an operator fixing a typo should not have a
multi-thousand-row replace happen under them. The response carries fresh status
so the panel can offer the import as the next step.
- **Every action is written to the admin activity log** (`shard.atlas.import` /
`.approve` / `.reject` / `.path`).

60
website/api-route-inventory.json
View File

@@ -257,6 +257,26 @@
"method": "GET",
"path": "/api/v1/admin/shard/accounts"
},
{
"method": "GET",
"path": "/api/v1/admin/shard/atlas"
},
{
"method": "POST",
"path": "/api/v1/admin/shard/atlas/approve"
},
{
"method": "POST",
"path": "/api/v1/admin/shard/atlas/import"
},
{
"method": "PUT",
"path": "/api/v1/admin/shard/atlas/path"
},
{
"method": "POST",
"path": "/api/v1/admin/shard/atlas/reject"
},
{
"method": "GET",
"path": "/api/v1/admin/shard/audit"
@@ -313,6 +333,14 @@
"method": "GET",
"path": "/api/v1/admin/shard/vendors/:account"
},
{
"method": "GET",
"path": "/api/v1/admin/shard/visibility"
},
{
"method": "PUT",
"path": "/api/v1/admin/shard/visibility"
},
{
"method": "PUT",
"path": "/api/v1/admin/site-mode"
@@ -705,6 +733,30 @@
"method": "GET",
"path": "/api/v1/player/shard/vendors/:account"
},
{
"method": "GET",
"path": "/api/v1/public/atlas/champions"
},
{
"method": "GET",
"path": "/api/v1/public/atlas/creatures"
},
{
"method": "GET",
"path": "/api/v1/public/atlas/creatures/:slug"
},
{
"method": "GET",
"path": "/api/v1/public/atlas/landmarks"
},
{
"method": "GET",
"path": "/api/v1/public/atlas/meta"
},
{
"method": "GET",
"path": "/api/v1/public/atlas/regions"
},
{
"method": "POST",
"path": "/api/v1/public/contact"
@@ -737,6 +789,10 @@
"method": "GET",
"path": "/api/v1/public/shard/economy"
},
{
"method": "GET",
"path": "/api/v1/public/shard/features"
},
{
"method": "GET",
"path": "/api/v1/public/shard/feed"
@@ -769,6 +825,10 @@
"method": "GET",
"path": "/api/v1/public/shard/presence"
},
{
"method": "GET",
"path": "/api/v1/public/shard/ruleset"
},
{
"method": "GET",
"path": "/api/v1/public/shard/status"