Shards edit items and add new ones, and those carry cliloc ids no stock client
table has. Reading exactly one converted file meant an operator had to
re-export 5 MB every time they added one item — friction enough that the table
would simply go stale, which is the failure the spawn atlas was redesigned to
avoid in the first place.
So this mirrors spawnAtlasSource.readSources(): a BASE (the converted client
table) plus every operator-maintained overlay under `custom/`, all re-read on
every boot and hash-gated as a SET. Later sources win, so an overlay both adds
ids the client never had and overrides stock ones the shard re-purposed.
Adding, editing or removing any overlay counts as drift.
`custom/` is the one convention here that is ours rather than the shard's, and
deliberately so: ServUO has no server-side notion of a custom cliloc — they
live in the patched client a shard distributes, and nothing in the tree
declares them. There is nothing to discover. (An operator who does patch their
client cliloc needs no overlay: convert the patched file and the edits are in
the base.) Scale, measured on the live shard: its script tree references 16,434
cliloc ids and only 37 are absent from stock — tens against a 67k base, which
is why this is an overlay and not a second table.
The set brings back a hazard a single file did not have, and it gets the
atlas's answer. A corrupt source fails the parse loudly, but a source that has
VANISHED parses perfectly and imports a table quietly missing everything it
contributed — an unmounted volume is indistinguishable from a deliberate
deletion. So it is staged, not applied (`needsReview`), reported by both the
import and status(), and accepted with `{approve:true}`. That is a flag rather
than the atlas's approve/reject pair because the atlas stores a pending
decision SO THAT approving re-parses; here nothing is stored, so re-reading at
approval time is automatic.
Also reports a per-source breakdown (entries/added/overrode) on import and in
status, which is how an operator confirms an overlay took effect — "overrode: 0"
on a file meant to re-label stock items says it did not.
Two bugs this surfaced, both found by running a shard-style overlay rather than
by another stock-table fixture:
- displayText tidied punctuation unconditionally, so a custom
"Runic Gateway Sigil (v2)" rendered as "(v2". Stripping leftover brackets is
right after a placeholder is removed and wrong otherwise — the same condition
the `%` rule already had.
- CANDIDATE_NAMES did not include `clilocs.plain`, which is the exact filename
CLILOCS.md and the export tool's README tell operators to write. Pointing at
the directory they were told to create failed with NO_FILE.
Verified end to end against the live MariaDB and a real server boot: base-only
import, overlay adding one id and overriding another (per-source breakdown
correct), unchanged set as a no-op, an edited overlay re-importing and
withdrawing its override, a vanished overlay refused with the table intact,
status reporting missingSources, approve applying it, and a file-path
configuration still finding overlays beside it. All three resolve correctly
through the running server: shard-added, overridden and stock. 646 server tests
pass (16 new in clilocSource.test.js, 3 new in clilocParse.test.js); swagger,
routes.manifest.json and routes.guards.json regenerated.
Co-Authored-By: Claude <noreply@anthropic.com>
Protocol 3.0 §8.6 (docs/link/v3.md), the dependency order 5 was sequenced
behind. Items on the wire carry a LabelNumber, not a name — the bridge has
always sent it (char.profile.equipment.cliloc, reward titles as a cliloc
number in string form, and one per marketplace listing) but the site had no
table to resolve it against, so a character sheet could only render
`id 1023721` where the game renders "quarter staff".
The number was never the missing piece. The table was.
Sourced from a file the operator converts once from their own client, at a
path from the `cliloc_client_path` setting falling back to UO_CLIENT_PATH.
Nothing client-derived is committed: UO's strings are EA's, exactly as the
creature sprites are. A shard with nothing configured is fully supported —
names render as ids, as they did before.
The conversion step is not avoidable, and that is the substantive finding
here: every current client ships its cliloc files COMPRESSED (first DWORD's
high byte 0x8E, the Mythic container), and ServUO's own bundled
Ultima.StringList cannot read that either — so VendorSearch.GetItemName is
already inert on such a shard and the plugin could not supply names instead.
v3.md's original "read the client's Cliloc.enu" recommendation was therefore
not implementable as written, and its committed db/data/clilocs.json artifact
also predates the Part C corrections (no committed derived snapshots, nothing
EA-derived shipped). Replaced with the spawn-atlas pattern: parse on boot from
an operator-configured path, hash-gated, output gitignored.
- utils/clilocParse.js — pure parsers, fs-free so the suite runs in CI.
Accepts the plain binary layout and delimited text, sniffed by header rather
than extension. Rejects a compressed file BY NAME: without that check the
plain parser reads it as ~19k records of negative ids and 60 KB "strings"
before dying mid-file, and the resulting error names the wrong problem.
displayText() drops the ~1_val~ arguments the bridge never sends.
- utils/clilocSource.js — the fs layer. hashSource reports `compressed` so the
admin panel can flag an unconverted file WITHOUT parsing 5 MB per poll;
otherwise pointing at a client directory reports a healthy file with pending
drift ("ready to import") and the operator only finds out on failure.
- model/shardClilocs — refresh/status/lookup. All-or-nothing replace (DELETE,
not TRUNCATE — TRUNCATE is DDL in MariaDB and implicitly commits). Batched
server-side resolution behind a capped cache; never throws, because a cliloc
lookup is decoration on a character sheet.
- Deliberately NO staged-approval flow, unlike the atlas: the atlas escalates
facet loss because a half-copied tree and a real map change are
indistinguishable from inside the process, whereas a partial cliloc copy
makes the parser fail on a truncated record. The ambiguity the atlas must
escalate is one this parser simply detects.
- No public route. The table is never served AS a table: 67k rows would dwarf
any page using them, and the Android client consumes the same resolved JSON.
Two parser bugs found by building it, both now covered by tests: trimming a
text line before splitting ate the trailing separator on empty-text entries
and silently dropped 55,994 of 123,490 while still reporting success; and
Number('') is 0, not NaN, so a line starting with a separator imported as a
bogus cliloc 0.
Verified against the real client table (123,490 entries) and the live MariaDB:
import 663 ms, hash-gated boot no-op 14 ms, cold resolve 4.2 ms / warm 0.015 ms.
Binary and TSV imports converge on the same 67,496 rows with identical keys
(blank entries — half the table — are dropped at import). A file truncated to
half its length is refused with TRUNCATED and leaves the previous table
serving. Boot logs verified for both the import and the compressed-file
warning; neither blocks startup. All three admin routes exercised over HTTP
with a real session. 629 server tests pass; client builds clean; swagger,
routes.manifest.json and routes.guards.json regenerated.
Not covered by an automated test: the character sheet renders resolved names
in presentational React with no DOM test harness in this repo, and was not
rendered against a live linked-player profile — that needs a logged-in player
with a linked game account and a shard answering a profile RPC.
Co-Authored-By: Claude <noreply@anthropic.com>
Protocol 3.0 §7 (docs/link/v3.md). The shard publishes ~25 points/loyalty
leaderboards — Queen's Loyalty, Void Pool, the nine city loyalties, Clean Up
Britannia — and the site renders them, plus each character's own standings on
their sheet.
Server
- shard_points_boards: one row per system, keyed by the shard's PointsType
name. The top-N list stays inside `payload` — a fixed-size list read whole,
exactly like shard_governors.candidates. Normalizing into an entries table
buys nothing until something needs a per-character reverse lookup, and a
character's own standings already ride inside char.profile.
- shardIngest routes points.board to upsertPointsBoard and deliberately does
NOT log it: this is board state like guild.update, and the shard emits a
frame every time anyone's score moves a top ten.
- uoLinkSocket backfills /points through snapshot() with ingestEach rather
than a replace*: there is no points.remove and the system set is fixed, so
upserting IS the reconciliation, and a system the operator later excludes
keeps its last-known board rather than vanishing.
- GET /public/shard/points and /points/:system behind
requireFeature('leaderboards'), both projected per §3.6.1. :system is
constrained to an identifier before any query runs; 404 for a system never
published, distinct from a published board nobody has scored in (200, empty
top).
The leaderboards field rule now keys on `name`, not `characterName`
Part A pre-wired FEATURES.leaderboards.fields = { characterName: ... }, but
projectValue matches on the LITERAL JSON key and the wire key is `name`. As
written the rule was inert: an admin tightening character names would have got
no enforcement and no error — precisely the failure §3.6.1 records for the
flattened `ownerAcct` spelling. Fixed, with a test that fails if it is renamed
back, and the admin panel's FIELD_LABEL carries the meaning instead.
Client
- routes/public/Leaderboards.jsx at /site/leaderboards. A points.board frame
describes ONE system, so live frames merge over the fetched set by system
key rather than replacing it wholesale the way the ruleset does. Filter
matches board name, system key, or any ranked player — the last is what
makes it useful ("where do I appear?").
- A "Loyalty & Points" section in CharacterSheet.jsx, one edit serving both
PlayerCharacter and AdminCharacter.
- Both treat maxPoints: 0 as UNCAPPED and both fall back to humanising the
system key when nameString is null. Neither is defensive padding: on a real
shard uncapped and cliloc-only names are the majority case.
Verified end to end against the local MariaDB, the Rust sidecar, and the real
ServUO shard: backfill from /points, live SSE delivery (a board absent from the
initial fetch appearing without a reload, and an existing one updating in
place), REST reflecting the overwrite, and the gate at every rung — 200 by
default with names, names stripped but points kept at fieldRules name=staff, 403
plus dropped from /features at audience=staff, 404 when disabled. Page rendered
clean, no console errors beyond the pre-existing React Router v7 warnings.
605 server tests pass; routes.manifest.json, routes.guards.json and the OpenAPI
spec regenerated.
Co-Authored-By: Claude <noreply@anthropic.com>
Protocol 3.0 order 3 (Part C), second of two website PRs. #112 built the data
pipeline; this makes it reachable — six public routes, five admin ones, two
public pages and an admin panel. Still website-only: no plugin, no sidecar, no
new event kinds, no wire change.
The API sits at /api/v1/public/atlas, not under /public/shard. Nothing here
touches the sidecar, so the pages stay complete while the shard is down, and a
/shard prefix would imply a dependency the atlas does not have. Unlike /shard/*
it IS site-mode gated, like /posts and /wiki: a bestiary is site content.
Every route carries requireFeature('atlas') and projects its response. The atlas
feature declares no sensitive fields, so the projection is a no-op today — the
call is there because v3.md 3.6.1's rule is that the FIRST field needing a gate
should be covered by construction rather than by a retrofit.
Two bugs the UI surfaced, both fixed here:
Respawn delays were stored in the wrong unit, sometimes. XmlSpawner writes
MinDelay/MaxDelay in minutes and switches to seconds only when a delay does not
divide into whole minutes, flagging it per record with DelayInSec. A `5` means
five minutes on one spawner and five seconds on the next, both plausible, and
the pipeline stored the raw number. 170 of 6,455 stock spawners are second
flagged. The parser normalises to seconds; the API and UI carry seconds.
That exposed the hash gate as a trap. "Has the tree changed?" is the wrong
question on its own: an install whose maps never change would have kept serving
the old readings forever, because the only thing compared was the tree.
PARSER_VERSION is now stored beside the source hashes and a mismatch counts as
drift, so any future parse correction lands on the next boot.
Also renamed the detail route's spawn-point array to `spawners` — it was
`points`, which is the COUNT on the search route, so one key meant a number in
one place and an array in the other.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U7CBg11prhLimL9iHSX1bP
Replaces the committed-artifact design from the first commit. Two problems with
it, both raised in review:
**Facets are not a fixed list.** The first pass carried a hardcoded table of the
six stock UO facets to reconcile the spelling drift between sources. That is
wrong: a shard may add facets, replace them outright, or rename them when its
maps are updated, and a built-in list quietly mishandles all three. Nothing in
the atlas names a facet any more. The facet set is discovered from the tree —
spawn records and region definitions are the authority — and the loose spellings
in Data/Locations are matched against it by key and prefix. Custom facets get
identical treatment; the tests use `Sosaria` and `Underdark` precisely so a
stock-facet assumption cannot creep back in.
**A snapshot goes stale.** Maps change over a server's life, so a build-once
artifact silently drifts from the world players actually see. The tree is now
the single source of truth and the atlas is re-derived on every boot.
## What that changed
- **The committed artifact is gone** — 1.41 MB of generated JSON removed, along
with `scripts/buildSpawnAtlas.js` and the whole encode/decode seam it needed
(`encodePoint`/`readPoint`, the tuple encoding, the omitted-defaults scheme and
their round-trip tests). Nothing to keep in sync, nothing to go stale.
- **NEW `src/utils/spawnAtlasSource.js`** — the only thing that touches a ServUO
tree; shared by the boot path and the CLI. Parsers stay pure and fs-free.
- **NEW `src/model/shardAtlas/`** — `.db.js` (the one-transaction replace) and
`.model.js` (the refresh decision).
- **`scripts/importSpawnAtlas.js`** is now a thin CLI over the model:
`--servuo`, `--force`, `--approve`, `--reject`, `--status`. `atlas:build` is
gone; `atlas:import` remains.
- Path comes from the `spawn_atlas_servuo_path` admin setting, falling back to
`SERVUO_PATH`. The setting wins, matching how the rest of the shard
integration is admin-managed rather than env-configured.
## Two contracts on the boot path
**It never blocks startup.** No path, an unreadable mount, a malformed file, a
database error — every one is caught and logged, and the site comes up serving
whatever atlas it already had. Verified by booting the real server with no path,
a broken path, and a good path.
**A facet disappearing is never applied automatically.** Losing a facet is the
signature of a half-copied or mid-update tree as much as of a real map change,
and boot cannot tell them apart. The refresh is staged in `shard_atlas_pending`
for an admin to approve or reject, and startup continues regardless. Additions
and every other change apply immediately, since none of them can destroy
something an operator would miss.
Only the decision is stored, not the parsed world: a few KB of source hashes and
the facet diff. Approving re-parses, so what gets applied matches the tree at
approval time rather than at boot. A rejection is remembered against those exact
hashes, so a declined refresh does not re-prompt on every restart — changing the
tree changes the hashes and asks again.
Hash-gated, so the common case (restart, maps unchanged) reads and hashes the
tree (~120 ms) and writes nothing. A real change costs a ~400 ms parse.
The admin approve/reject UI is part of the second PR, with the rest of the
routes and pages. Until then the CLI covers it.
## Verification
- **564 server tests pass**, 28 new in `spawnAtlas.source.test.js` covering the
custom-facet build, the spelling reconciliation, hash gating, and every branch
of the refresh decision — including that `refreshOnBoot` survives a database
that throws on every call.
- End-to-end against the local MariaDB and the real ServUO tree: 6,455 points,
800 creatures, 23,927 point/type rows, 387 regions, 558 landmarks, 25 altars,
83.2% of points resolved to a place name.
- The facet gate exercised against a real tree copy with `malas.xml` removed:
staged rather than applied, atlas untouched with all 293 Malas points intact,
reject then stays quiet on re-run, approve applies and drops the facet.
- Booted the real server under all three source conditions; none blocked.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U7CBg11prhLimL9iHSX1bP
Protocol 3.0 order 3 (Part C), first of two website PRs. This half is the data
pipeline only — parsers, the build/import CLI, and the tables. No routes and no
client, so nothing is user-visible yet; the API and pages follow in PR 2.
Part C is website-only: no plugin, no sidecar, no new event kinds, no wire
change.
## Parsing
`src/utils/spawnAtlasParse.js` is pure and fs-free so CI covers it with no
ServUO tree. Zero new dependencies — `Regions.xml` genuinely nests, so it gets a
small hand-rolled subset tokenizer rather than a new XML package. The 10.5 MB of
`Spawns/*.xml` never touches it: those records are flat and get a streaming
regex sweep instead.
The high-value transform is point-in-rect placement — highest region priority
wins, ties break to the smaller rect, then a nearest-landmark fallback within
200 tiles, else "Wilderness". That is what turns "lizardman at 5411,1234" into
"Despise, Felucca", and it resolves 83.2% of points (5,369 of 6,455).
Three things the real data forced, none of which were in the design:
- **Only 6 facets, not 13.** `Eodon.xml`, `GravewaterLake.xml` and the other
named-area files carry TerMur/Trammel points, so the facet comes from each
record's own `<Map>` and the artifact shards 6 ways.
- **Facet names disagree across sources.** `Data/Locations/*.xml` spells them
`Ter Mur` and `Tokuno Islands`; `<Map>` and `<Facet name>` say `TerMur` and
`Tokuno`. Unreconciled this is silent — the landmark fallback simply never
fires on those facets and every unregioned spawn there reads "Wilderness".
- **Spawn type tokens carry XmlSpawner directives**: `Fairy,{RND,4,8}`,
`alchemist/z/-50`, `Agralem/Name/Agralem`. Taken literally these invent
creatures that do not exist AND split real ones in two, since `Fairy` and
`Fairy,{RND,4,8}` slug apart. 71 of 845 entries were affected; stripping at
the first `/` or `,` leaves 800 clean ones.
## Artifact
`npm run atlas:build -- --servuo <path>` writes `db/data/spawnAtlas.*.json`:
6 facet shards + a compact index + a small indented `meta`. 1.41 MB committed,
down from 4.40 MB by dropping `facet` per record, omitting defaulted fields, and
tuple-encoding the ~24,000 type entries. `encodePoint()` and the importer's
`readPoint()` are exact inverses and are round-tripped in tests.
Display spelling is chosen deterministically (most common, ties to the
capitalised form) because the spawn files are inconsistent about case and the
name would otherwise depend on file read order — a spurious diff on every
unrelated rebuild.
## Import
`npm run atlas:import` needs no ServUO tree, which is the whole reason build and
import are separate: the container has the artifact but not the tree. It
reloads all six tables in one transaction (DELETE, not TRUNCATE, which is DDL
and would implicitly commit), so a failed import leaves the previous atlas
intact.
## No artwork, by design
The repo ships no creature art and no extraction tooling. Sprites live in the
operator's own client `.mul`/`.uop` files and are theirs, not ours to
redistribute. `shard_spawn_creatures.art` is nullable and NULL on every fresh
import; an operator who wants art extracts it themselves, drops it under
`server/uploads/atlas/` (already gitignored) and maps slugs in a gitignored
`spawnAtlas.art.json`. Text-only is the normal, fully supported state.
## Verification
- **544 server tests pass**, 57 new across `spawnAtlas.parse.test.js` (the
`:OBJ=` split, directive stripping, nested-region priority inheritance,
half-open rects, the facet reconciliation, tokenizer edge cases) and
`spawnAtlas.build.test.js` (aggregation, deterministic naming, and the
encode/decode round trip).
- Built and imported for real against the local MariaDB and the ServUO tree at
`C:\Users\colby\Desktop\ServUO`: 6,455 points, 800 creatures, 23,927
point/type rows, 387 regions, 558 landmarks, 25 champion altars.
- "Where does a lizardman spawn?" answers Shrines / Isamu-Jima / Yew across
Felucca, Trammel and Tokuno.
No routes changed, so the OpenAPI spec and route manifest are untouched.
---
- [x] AI-assisted: written with **Claude Code** (Claude Opus 5), reviewed before opening.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U7CBg11prhLimL9iHSX1bP
Protocol 3.0 §5 (docs/link/v3.md). The shard publishes its own ruleset —
expansion, which optional systems are on, skill/stat caps, account and house
limits, champion scroll rules, the save/restart schedule — and the site renders
it, so the rules page cannot drift from how the shard actually plays.
Server
- shard_ruleset: a singleton table (id = 1) holding the whole frame in
`payload`, with `rev` and `expansion` hoisted. Nothing is normalized out:
the frame is a flat description of config read as one page, and splitting it
into columns would mean a schema change every time the shard grows a block.
- shardIngest routes world.ruleset to setRuleset and deliberately does NOT
log it — the shard re-emits the whole ruleset on every sidecar connect, so
logging would append a duplicate row per reconnect, and server.hello already
marks each of those.
- uoLinkSocket backfills GET /ruleset explicitly rather than via snapshot(),
which asserts an array; this covers the order where the sidecar was already
up and holding the ruleset when we reconnected.
- GET /public/shard/ruleset behind requireFeature('ruleset') and projected,
per §3.6.1's rule that a shard read which doesn't project is a bug. `null`
means the shard has never published one — a real answer, distinct from a
published ruleset, and the page says so.
Client
- routes/public/Rules.jsx at /site/rules, live via world.ruleset (a frame is a
complete ruleset, not a delta, so the newest one wins outright). Caps are
rendered from tenths — 7000 is 700.0, and showing the raw number would
mislead. A systems key this build doesn't know still renders, humanised, so
a newer plugin can't go invisible against an older client.
- Nav entry gated on the `ruleset` feature, so it hides rather than 403s.
Verified end to end against the local MariaDB and a sidecar fed by a fake shard:
backfill snapshot, live SSE delivery of a changed ruleset, REST reflecting the
overwrite, an empty /feed (not logged), and the gate — 200 by default, 403 at
audience=staff (and dropped from /features so nav hides it), 404 when disabled.
Page rendered clean at all breakpoints checked, no console errors.
497 server tests pass; routes.manifest.json, routes.guards.json and the OpenAPI
spec regenerated.
Co-Authored-By: Claude <noreply@anthropic.com>
Protocol 3.0 Part A follow-up, found by the live five-rung smoke test.
Part A implemented the visibility framework correctly on the SSE path
and on /guilds + /governors, but the remaining public REST reads never
called into it. The result was that one event was projected live and
served verbatim from history:
* GET /public/shard/feed returned the stored payload as-is, so
actor.acct and actor.webId were readable ANONYMOUSLY for every
logged kind - player.death, player.murdered, mob.killed,
quest.complete, skill.gain, fame/karma.change, mob.login/logout,
guild.join. Broader than the guild-leader leak Part A set out to
close, since it covers every player rather than board holders.
* GET /public/shard/idoc returned ownerAcct - the house owner's game
account - to anonymous callers.
* The `houses` field rules (owner/price -> staff) were dead config:
neither getIdoc nor getHouses projected, so an admin could set them
in the panel and nothing happened.
* /feed filtered on PUBLIC_KINDS, a module-load constant derived from
the compiled DEFAULTS, so live audience changes did not reach it.
With `guilds` moved to staff, /guilds 403'd while /feed happily
served guild.join to anonymous.
Four fixes, all at the root rather than per-route:
1. Rule 1 now matches a field's MEANING, not one spelling. The wire
nests actors (leader.acct) but the read models flatten them
(shapeHouse -> ownerAcct, shapeGuild -> leaderWebId), and an
exact-key check missed every flattened one. isLockedField() locks a
key that is or ends in acct/webId, case-insensitively, so it fails
closed for shapes not yet written. The admin PUT rejects those
spellings too - `ownerAcct` is no longer configurable.
2. visibleKinds(level, config) resolves readable kinds from the LIVE
config; getFeed uses it and projects each row against its own kind's
feature. Deliberately independent of the `stream` flag, which governs
SSE fan-out only - so market history stays readable with its firehose
off. This makes the set a superset of PUBLIC_KINDS by exactly the two
vendor kinds.
3. getIdoc/getHouses/getChamps/getPresence project, so every shard
surface honours the same config.
4. shardEvents.db.list treats an EMPTY kinds array as "serve nothing".
It previously fell through to the unfiltered query, so a fully-gated
config would have dumped the whole event log, staff audit included.
Also fixes a bug introduced while wiring this up: projectValue recursed
into any object, so a Date column came back as {}. It now walks arrays
and plain objects only. The unit tests used JSON fixtures and could not
have caught it - the live /idoc read did.
Verified live against MariaDB + a stub sidecar, all five rungs: 13
routes x 5 rungs, defaults reproducing pre-v3 access exactly, zero
acct/webId below admin on any read, unmapped kinds (staff.command,
cheat.detect, login.attempt) reaching only admin on SSE, and audience /
enabled / stream changes taking effect live on an already-open stream.
Tests: 487 server (+9). Swagger regenerated; route manifest unchanged.
Co-Authored-By: Claude <noreply@anthropic.com>
Protocol 3.0 Part A. Replaces the static PUBLIC_KINDS allowlist - which
was the entire public/admin boundary - with per-feature, per-field
audience control an admin owns from Admin -> Shard Visibility.
Closes a live leak. BridgeJson.Actor() writes acct and webId;
shapeGuild() returned the stored payload verbatim; GET
/api/v1/public/shard/guilds is anonymous. Guild leaders' game account
names and website user ids were readable by anyone, and the same path
existed for governors. Both are now projected.
The ladder is anonymous < logged_in < player < staff < admin, each rung
implying the ones below. Staff satisfy `player` without a linked account
(as /player/* already does); `editor` is a content role and gets no
shard privilege, since mapping it to staff would silently widen what
editors see.
Two invariants are code, not configuration, and both reject rather than
silently ignore:
1. acct/webId are admin-only always - not configurable, discarded on
read as well as rejected on write.
2. A kind absent from KIND_FEATURE never reaches anyone below admin.
Fail closed, so a shard emitting a new event degrades to staff-only
rather than to public.
Enforcement is three points over one config: requireFeature() on routes
(404 disabled, 403 out-of-rung) plus field projection; per-connection
filtering on SSE, where a subscriber's rung is resolved once at subscribe
time and frozen so a long-open stream cannot gain privilege; and
/public/shard/features so the SPA hides links it cannot follow.
PUBLIC_KINDS still exists and is still exported (/feed filtering,
notificationStreams) but is now derived from the kind map, so the two
can no longer drift. Defaults reproduce pre-3.0 behavior exactly - a
test pins the derived set against the old allowlist.
Also fixes an SSE resource leak found while testing: a client dropped
because its write threw was removed from the bucket but its keepalive
interval was never cleared, firing forever on a dead socket. Both paths
now go through one drop().
Tests: 478 server (33 new across shardVisibility + shardBroadcast),
43 client. Route manifest and OpenAPI spec regenerated.
Co-Authored-By: Claude <noreply@anthropic.com>
"Trust this device" did nothing for anyone who signs in with Google or Discord.
sso.controller went straight from needsTotp(user) to staging a pending-TOTP
challenge and never consulted resolveTrustedDevice, so an SSO user was asked for
a code on EVERY sign-in no matter how many times they had ticked the box — and
POST /auth/sso/totp accepted only `code`, so that step could not establish a
trust either. The password paths (web + native) were unaffected and already
worked; this closes the gap for SSO, on the website AND in the Android app.
Server:
- finishLogin and finishMobileLogin now run the same trusted-device check as
auth.controller.login, via one shared helper: honor a trust that belongs to
THIS user, stamp last_used_at, log auth.login.trusted_device. A store error
falls through to the challenge — fail closed to asking for the code.
- POST /auth/sso/totp gains optional trustDevice + deviceName, sets the rg_trust
cookie, and mirrors the password path's { trustLimitReached, devices } response
at the cap (the sign-in still completes). Recovery codes stay password-only.
Android coverage, without leaking a secret into a URL:
- The app opens SSO in a Custom Tab, which shares the system browser's cookie
jar, so the rg_trust cookie set on that TOTP form is presented back on the next
app sign-in. That alone makes native SSO skip the code. Passing the app's token
into the start URL was rejected — it would put a 256-bit secret in query
strings, Referer headers and access logs.
- To also cover the app's NATIVE password login, ticking the box sets
mobile_auth_sessions.trust_device (a boolean; never the token), and
/auth/mobile/sso/exchange mints a platform:'mobile' trust and returns
{ trustToken }. Minting there keeps the raw token on an authenticated
app→server call, out of the deep link and out of the bridge row. Best-effort:
at the cap the response just omits it rather than failing a good sign-in.
Client: the trust checkbox is no longer hidden on the SSO second step, on both
the admin and player login screens. On the mobile bridge the deep-link redirect
takes priority over the cap prompt — the sign-in succeeded and the link is
single-use, so stalling there would strand the app.
Tests: 8 new cases in server/test/ssoTrustedDevice.test.js (verified to fail
against the pre-fix controller). Full suites green — server 445, client 43 —
and routes.manifest.json is a zero-line diff: no URL moved, only +2 handlers on
/auth/sso/totp in routes.guards.json for the two new validators. Swagger
regenerated. Verified live against the running server and real MariaDB: the TOTP
step issues rg_trust and persists the row, a subsequent SSO callback carrying it
skips the code, and an invalid trust is still challenged.
Co-Authored-By: Claude <noreply@anthropic.com>
`uoLinkClient.call()` resolved the uo-link config OUTSIDE its try/catch.
resolveConfig() decrypts the stored auth token, and secretBox.decrypt throws
when the ciphertext can't be authenticated — SECRET_ENC_KEY rotated, or a DB
dump restored into an environment keyed differently. That throw escaped the
client entirely, breaking its documented "never throws / always returns
{ ok, data, status }" contract and turning a misconfiguration into a 500 on
every route that does a live sidecar round-trip:
GET /admin/uo-link/config
GET /{admin,player}/shard/char/:serial
GET /{admin,player}/shard/roster/:account
GET /{admin,player}/shard/vendors/:account
Found by a live smoke test of all 200 routes at every access level. Public
shard routes were unaffected because they read the DB via getSafe(), which
never decrypts.
Move resolveConfig() inside the try so the failure returns the standard
{ ok: false } shape, and log it at ERROR with a distinct message: a wrong key
previously looked identical to "the shard is offline", with no clue why.
Those routes now degrade to 503, and GET /admin/uo-link/config returns 200
again — it is the screen an admin needs to re-enter the token and recover, so
having it 500 locked them out of the fix.
Also gate the admin Dashboard's site-mode toggle. PUT /admin/site-mode is
adminOnly, but the button rendered for every staff role, and toggle() had a
try/finally with no catch — so an editor clicking it got an unhandled promise
rejection and zero UI feedback. Gate the control on role === 'admin' (the rule
AdminLayout already documents: never show a non-admin a control that would 403)
and surface a message if the call is refused anyway.
Adds server/test/uoLinkClient.test.js, which fails against the unfixed client.
Co-Authored-By: Claude <noreply@anthropic.com>
The last split PR of docs/website/API_V2_PLAN.md § Phase 2. public.routes.js,
player.routes.js and auth.routes.js are deleted; each group is now a directory
whose index.js owns the group gate and the mount table and declares no routes.
Every one of the 200 manifest routes is now in a capability router.
public/ posts (2) wiki (4) pages (2) shard (12) site (4, group root)
player/ account (8) shard (8) appeals (4), behind noindex + requireAuth
auth/ login (2) register (1) invite (2) password (3) session (2, root)
No URL moves. All four gates zero-diff: routes.manifest.json (200 public + 2
internal), routes.guards.json, swagger-output.json (198 operations), and
docs/website/api-route-inventory.json was already in sync. 434 tests green.
Notes on the non-mechanical parts:
- public/index.js and auth/index.js carry no group gate, deliberately, and say
so. The public surface is anonymous by contract (logged-out SPA, Discord bot,
Android ShardStreamClient on /public/shard/stream); /auth is where a caller
becomes authenticated. player/index.js gates on requireAuth only, never
requireRole('player') — staff are a superset of players.
- GET /auth/me has a mount-order dependency: use('/me', meRouter) matches the
bare /me, so the request runs meRouter's noindex + requireAuth and falls
through. session.router.js must stay mounted last. Verified by the
counterfactual — mounting it first still 401s but drops X-Robots-Tag, which
no manifest or guards file can see.
- loginGuards moved to auth/loginGuards.js (frozen) rather than being copied
into the three routers that spread it; sso.routes.js drops its duplicate.
- The :param shadowing check was re-run in dispatch order against the built
stack: 86 routes, 64 literal, none shadowed. /public/wiki/{categories,tags}
ahead of /:slug is the only ordering-sensitive pair.
Co-Authored-By: Claude <noreply@anthropic.com>
PR 4 of the in-place admin router split (docs/website/API_V2_PLAN.md § Phase 2),
and the last admin one: it moves the entire residual 33 and DELETES
admin.routes.js. Every one of the 110 admin routes is now declared in a
capability router. No URL, gate or handler changes.
shard.router.js (16) /admin/shard
uoLink.router.js ( 5) /admin/uo-link
email.router.js ( 6) /admin/email
discordBot.router.js ( 2) /admin/discord-bot
settings.router.js ( 2) /admin/settings
dashboard.router.js ( 2) GET /dashboard + PUT /site-mode, at the group root
admin.routes.js deleted, was 33
No gate moved to router level. Every adminOnly in the residual file was
per-route, and modAccess on /shard must stay per-route because half that router
must not have it — which keeps the per-route handler count intact, the one
number routes.guards.json can actually check.
/shard is the first prefix where two tiers share one router: 7 self-service
account-linking routes (no extra gate, served by the same player/shard
controller handlers, tagged `Admin · Account`) alongside 9 in-game staff ops on
modAccess. Prefix ownership beats tag grouping — splitting by tag would put two
routers under one prefix for no gain. The tag mismatch stays; retagging is a
real spec diff and belongs in a PR about tags.
dashboard.router.js is the one router mounted at the group root rather than a
prefix: GET /dashboard and PUT /site-mode share no path segment. That is safe
only because the file declares no router-level middleware — a bare use(gate) in
a root-mounted router would run for every request passing through toward
another mount. The file carries a comment saying so.
Acceptance — all four gates zero-diff:
routes.manifest.json unchanged (200 public + 2 internal)
routes.guards.json unchanged (no route lost or gained a gate)
swagger-output.json unchanged (198 operations)
api-route-inventory.json already in sync
plus 434 server tests green.
Verified separately, because no gate can catch it: introspecting the built
stack, all 59 literal admin paths still dispatch to their own layer — nothing
is captured first by a /:param sibling. The manifest sorts its entries, so
declaration order is invisible to it.
Also repoints the comments that referenced admin.routes.js by name
(botActivity/moderation controllers, the town-crier cap mirror in
announceJobs.logic.js) and generalizes the "the path is on the line after
router.get(" rationale in routeManifest.js, README.md and pr-checks.yml, which
was never about that one file.
Co-Authored-By: Claude <noreply@anthropic.com>
PR 3 of the in-place admin router split (docs/website/API_V2_PLAN.md § Phase 2).
Moves the content tier out of the residual admin.routes.js into one router file
per capability, each mounted at the prefix it already owned. No URL, gate or
handler changes.
posts.router.js ( 9) /admin/posts
uploads.router.js ( 1) /admin/uploads
wiki.router.js (14) /admin/wiki
pages.router.js ( 7) /admin/pages
admin.routes.js (33) residual, was 64
All four capabilities are editor tier, so no gate moved: the shared
`noindex, isLoggedIn, staffOnly` in admin/index.js is their whole gate.
The multer config moved to admin/imageUpload.js because the two routes that
share it (POST /posts/upload and POST /uploads) now live in different files;
duplicating a mimetype allowlist is how the two copies drift. It stays in
admin/ because UPLOAD_DIR is resolved relative to __dirname.
Acceptance — all four gates zero-diff:
routes.manifest.json unchanged (200 public + 2 internal)
routes.guards.json unchanged (no route lost or gained a gate)
swagger-output.json unchanged (198 operations)
api-route-inventory.json already in sync
plus 434 server tests green.
Verified separately, because no gate can catch it: the wiki router's literal
/categories and /tags paths still precede /:slug in declaration order. The
manifest sorts its entries, so a reordering there would be invisible.
Co-Authored-By: Claude <noreply@anthropic.com>
PR 2 of the domain split (docs/website/API_V2_PLAN.md § Phase 2). Carves 18 more
routes out of admin.routes.js into one router file per business capability,
in place, with every URL unchanged:
moderation.router.js (15) /admin/moderation modAccess at router level
botActivity.router.js (2) /admin/bot-activity adminOnly per route
activity.router.js (1) /admin/activity staff-wide, no extra gate
The residual admin.routes.js drops from 82 routes to 64.
Moderation was already gated by a prefix mount (adminRouter.use('/moderation',
modAccess)), so moderationRouter.use(modAccess) is the exact equivalent now that
the router is mounted at a prefix. Bot-activity's adminOnly was per-route and is
deliberately kept per-route: that is what holds the per-route handler count in
routes.guards.json, the only signal that would catch a dropped gate, since
requireRole(...) returns an anonymous arrow and never appears by name.
/activity gets its own file rather than waiting for dashboard.router.js in PR 4
— it is the staff audit log, a different capability from the dashboard's stats
overview and from the botScore middleware's in-memory ban state.
Acceptance:
- routes.manifest.json zero-diff (200 public + 2 internal)
- routes.guards.json zero-diff
- swagger-output.json zero-diff (198 operations)
- api-route-inventory.json already in sync
- 434 server tests green
- role gates verified identical to main by reading the requireRole role sets
off the live Express stack for every moved route plus untouched controls
Co-Authored-By: Claude <noreply@anthropic.com>
First of the five domain-split PRs in docs/website/API_V2_PLAN.md § Phase 2. Pure
mechanical re-wiring: routes move between files, no handler, gate, validator or
annotation changes, and not one URL moves.
New src/router/v1/admin/index.js owns the two things the group shares — the
`noindex, isLoggedIn, staffOnly` gate and the mount table — and declares no routes
itself. The gate sits ahead of every mount so a capability router extracted in a
later PR cannot silently ship without it. Four capability routers mount at the
prefix they already owned inside the monolith:
account.router.js 6 routes -> /admin/account (self-service, no adminOnly)
users.router.js 15 routes -> /admin/users (adminOnly, router-level)
invites.router.js 3 routes -> /admin/invites (adminOnly, per-route)
authProviders.router.js 4 routes -> /admin/auth (adminOnly, per-route)
admin.routes.js keeps the other 82 (6+15+3+4+82 = the 110 inventoried admin
routes) and is mounted last at the group root; none of the four prefixes appears
in it, so nothing depends on mount ordering. It disappears when PR 5 lands.
Handlers still live in admin.controller.js and usersShard.controller.js — this
re-wires routes, not logic. `adminOnly` moves with the routes that use it, and
`usersRouter.use(adminOnly)` is exactly equivalent to the old
`adminRouter.use('/users', adminOnly)` now that the router is mounted at /users.
All three generated gates are zero-diff:
routes.manifest.json unchanged (200 public + 2 internal)
routes.guards.json unchanged — no route lost or gained a gate
swagger-output.json unchanged, byte-for-byte
The spec staying byte-identical depends on the path normalization landed in the
preceding commit; without it the four collection routes would have documented as
/api/v1/admin/{users,invites,account}/ with a trailing slash.
Server tests green (434/434).
Co-Authored-By: Claude <noreply@anthropic.com>
Prepares the committed spec for the admin router domain split
(docs/website/API_V2_PLAN.md § Phase 2) by post-processing swagger-autogen's
output in swagger/swagger.js. No route, handler or annotation changes.
Trailing slashes are stripped from path keys. swagger-autogen builds a path by
string-concatenating the mount prefix with the route argument, so a capability
router mounted at /users whose collection route is router.get('/') documents as
/api/v1/admin/users/ — advertising a URL no client calls while dropping the one
the SPA, the Android app and the Discord bot all do. Express is indifferent
(non-strict routing treats the two as one route, and routes.manifest.json records
the canonical slash-less form), but the published spec is a contract. The split
creates one of these per capability router, so it is fixed once here rather than
by contorting the route declarations in every router file.
Path keys are also sorted. The generator emits them in router-traversal order, so
moving a route between files rewrites most of this ~5k-line committed artifact
even when the API is provably unchanged, burying the one line a reviewer needs to
see. OpenAPI attaches no meaning to path order, and scripts/routeManifest.js
already sorts for the same reason.
Verified inert: the regenerated spec is byte-for-byte the sorted form of the
previously committed one — same 198 operations, zero added or removed, and no
trailing-slash keys (there were none to strip yet; the guard is for the split).
A collision after normalization throws rather than silently dropping an
operation. Server tests green (434/434).
Co-Authored-By: Claude <noreply@anthropic.com>
Phase 1 of docs/website/API_V2_PLAN.md. The tightened policy ships on
Content-Security-Policy-Report-Only alongside the unchanged enforced one for a
release; a follow-up PR flips it after the soak comes back clean.
The plan expected a two-directive delta. It is one. `form-action 'self'` was
described as absent because it is not in the directives object in app.js — but
the middleware runs with `useDefaults: true` and helmet's defaults already
supply it, so the header served in production has carried it all along. Caught
by capturing the live header from the running app instead of reading the config.
It is now written out explicitly in config/csp.js regardless: a security
directive should not depend on a third-party library's default surviving its
next major version. The enforced header's contents do not change at all, and a
test pins it verbatim.
So the whole behavioural delta is `frame-ancestors 'self'` -> `'none'`. That is
still the directive most worth soaking: a frame-ancestors report is generated by
the browser of whoever framed the site, which is the only way to find out that
something legitimately embeds us before an enforcing policy breaks it.
The policies move to config/csp.js, with the report-only one derived by spread
from the enforced one so the two cannot drift and the object reads as a diff.
`report-to` needs somewhere to point, so this adds POST /api/csp-report --
same-origin on purpose, since reports describe attacks against this site and
should not go to a third-party collector. It is mounted outside /api/v1 next to
/api/health: the browser learns the path from the policy header, never from a
client build, so it is not versioned client contract.
It is necessarily unauthenticated -- browsers send reports with no session, and
gating it would silence exactly the anonymous visitors worth hearing about -- so
it is bounded on every axis:
* both wire formats, since report-uri (Firefox/Safari) sends hyphenated keys
in application/csp-report and report-to (Chrome) sends camelCase envelopes
in application/reports+json; handling one silently drops half the browsers,
* report-to also needs the Reporting-Endpoints response header or it is inert,
* 16 KB body cap, per-IP rate limit, fixed field allowlist, every logged field
truncated (script-sample is attacker-influenced and can carry a whole inline
script),
* always 204, even for malformed input: a 4xx would reach the global error
handler, which logs the offending body -- turning an open endpoint into a
log-flood primitive.
Nothing is persisted; reports go to the `csp` log tag.
routes.manifest.json moves 199 -> 200, which is the freeze from PR 0 working as
designed: the one new URL is visible as a reviewed +1 rather than slipping
through. Swagger regenerated to match.
Co-Authored-By: Claude <noreply@anthropic.com>
PR 0 of the router domain split (docs/website/API_V2_PLAN.md § Phase 2). The
split promises that admin.routes.js can be carved into one router file per
business capability without moving a single URL. That promise has to be proved
by a diff, not asserted in review — this lands the tool that proves it, with no
router file moved.
scripts/routeManifest.js walks the live Express stack (runtime introspection,
not source parsing: route paths in admin.routes.js sit on the line *after*
`adminRouter.get(`, which defeats greps) and writes a sorted { method, path }
list to routes.manifest.json. It reproduces the frozen baseline in
docs/website/api-route-inventory.json byte-for-byte — 199 public routes plus 2
on the internal listener — so the freeze is confirmed accurate, not just
claimed.
Scope is /api/** and /.well-known/** plus the internal app. The SPA catch-all,
/uploads and /brand are filesystem-conditional static mounts, so including them
would make the output depend on whether CI had built the client. Static mounts
are not API contract.
Also emits routes.guards.json — a review aid, not a contract: per route, the
handler count and the *named* middleware on its mount chain. Router-level
`use(noindex, isLoggedIn, staffOnly)` gates never appear in an individual
route's own stack, so an extracted capability router that forgot to re-apply
one would otherwise publish authenticated endpoints silently. Names are a hint
only (requireRole(...) returns an anonymous arrow), but a vanished requireAuth
is unambiguous — and the test suite asserts every /admin/** and /player/**
route still carries it.
The plan's optional unauthenticated-status snapshot was tried and dropped, as
it allowed: against the dead-port mariadb pool the tests use, the sweep sits on
the pool's acquire timeout and had not finished after two minutes. A flaky
two-minute gate is worse than none; the requireAuth assertion covers the same
regression deterministically.
CI runs `npm run routes:manifest -- --check` on every PR, so a URL change can
only merge by deliberately committing the new manifest.
Co-Authored-By: Claude <noreply@anthropic.com>
Add a sync-project-tree workflow that regenerates this repo's tracked-file
tree and opens (or force-updates) a PR against RunicGateway/docs whenever the
layout on main changes. Never writes to the docs repo's main directly. Reuses
the existing REGISTRY_USER / REGISTRY_TOKEN secrets. Tree rendering lives in
.gitea/scripts/gen_tree.py (deterministic, dirs-first ordering).
Co-Authored-By: Claude <noreply@anthropic.com>
windowValue mapped only the 24h/7d keys and used `?? row.d30` as the fallback:
const col = { '24h': row.d1, '7d': row.d7 }[key] ?? row.d30
so a null d1/d7 (which the function is documented to tolerate) returned the
30-day count instead of 0, inflating the 24h/7d moderation tiles. It happens to
be masked today because `SUM(created_at >= ?)` nulls d1/d7/d30 only in unison,
but the contract is wrong and the existing test used an all-null row that hid it.
Map all three window keys explicitly so each reads its own column and a null
coerces to 0 via `Number(col) || 0`. Add a regression test with a null narrow
column and a non-null d30.
Co-Authored-By: Claude <noreply@anthropic.com>
The `:discordId` param validator on the five admin moderation routes used
`/^d{1,32}$/`, which matches 1-32 literal `d` characters instead of digits.
A real numeric Discord snowflake failed validation, so every
`/moderation/user/:discordId*` endpoint returned a 400 for valid input.
The backslash was dropped in a prior code-smell cleanup (12d50fd) that
intended `[0-9]` -> `\d`. Restore `\d` so the regex matches digits again.
Co-Authored-By: Claude <noreply@anthropic.com>
The ntfy service was configured with no published host port, on the
assumption that the public reverse proxy shares the compose network and
can dial ntfy:80 directly. It does not — Pangolin runs outside the
compose network and reaches every service through a published host port
(exactly why `app` publishes 3000). With no published port there was
nothing for the notification subdomain to forward to, so push delivery
could never work in production.
Publish container :80 on a host port (NTFY_HOST_PORT, default 2586,
binds 0.0.0.0 like `app`) and correct the now-inaccurate comments in
docker-compose.yml and ntfy/server.yml. Document NTFY_HOST_PORT in
.env.example. No code change — deploy config only.
Co-Authored-By: Claude <noreply@anthropic.com>
Staff are a superset of players — every player ability plus their staff
tools on top — but the /player/* group ran requireRole('player'), so a
signed-in admin/editor/moderator got 403 on their own linked game
accounts (e.g. GET /player/shard/accounts). On the Android client this
hid "My characters" and greyed the personal notification streams for
staff accounts, even when they had linked characters.
Drop the role gate: the group is now requireAuth-only. Every handler is
already self-scoped to the caller by req.user.id (with the pre-existing
isAdmin bypass still letting a genuine admin read any character), so this
only ever widens access to the caller's OWN data. Staff also reach the
identical self-scoped handlers under /admin/shard/* (same controller).
- player.routes.js: requireRole('player') -> requireAuth; corrected the
five stale "Player role required" 403 descriptions and regenerated
swagger-output.json.
- New test/playerRouteAccess.test.js mounts the router and asserts
player/admin/editor/moderator all reach the handler, anon still 401s,
and a disabled account still 403s. Suite: 420 pass.
Co-Authored-By: Claude <noreply@anthropic.com>
Add opt-in "Trust this device" so a browser/app skips the TOTP step (never
the password) for 30 days, single-use bcrypt recovery codes as a 2FA-lockout
fallback, and admin trusted-device/MFA-reset management — backend, web UI,
OpenAPI spec, and tests.
- Schema: trusted_devices (sha256 token hash, looked up by unique index) and
recovery_codes (bcrypt, single-use). Both additive/idempotent.
- Session service: trust-token mint/hash/resolve + cap helpers; new rg_trust
httpOnly cookie (survives logout, revoked on untrust/password change/reset/
TOTP disable). JWTs stay stateless — trust is a server-side row, not a claim.
- Web + mobile login accept a trusted-device token / recovery code; login/totp
gains trustDevice + recoveryCode. Cap of 10/user with NO silent pruning — an
over-cap trust returns 409/trustLimitReached and the client prompts to revoke.
- Self-service /auth/me/trusted-devices* + recovery-codes*; admin
/admin/users/:id/trusted-devices* + /mfa/reset. All actions audit-logged.
- Client: "Trust this device" + recovery-code login options, one-time recovery
code display, Trusted Devices + Recovery Codes account panels, a TOTP-styled
revoke-to-continue cap modal, and admin per-user security controls.
- OpenAPI regenerated; 33 new server tests (all suites green).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add an Architecture section with a Mermaid diagram of the full data path
(SPA/mobile clients -> layered Express backend -> MariaDB, and the uo-link
sidecar bridge to the ServUO shard) plus a Contents entry. Same diagram is
mirrored in the docs repo (docs/website/ARCHITECTURE.md).
Co-Authored-By: Claude <noreply@anthropic.com>
