RunicGateway/docs
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(modules): close Phase 4 — de-UO the backend reference, record the acceptance run

Browse Source 
Phase 4 slice 4. BACKEND_DESIGN.md had never been de-UO'd: Phase 3 rewrote core's
code and README, but §5.2's identifier check reads code, not prose, so nothing
ever looked at the design document. It still described 27 shard_* tables, 20 UO
route rows and the shard visibility ladder as core's, a phase after core stopped
being able to serve any of them.

Moved, text unchanged:

  BACKEND_DESIGN §3   six shard_* schema sections (226 lines) → modules/uo/SCHEMA.md
  BACKEND_DESIGN §4   13 public + 7 admin UO route rows       → modules/uo/API.md
  BACKEND_DESIGN §6.5 the audience ladder (70 lines)          → modules/uo/API.md §4

Core keeps the seam and gains the eight /admin/modules routes it had never
documented. §6.5 becomes "Module-owned audience boundaries": core's security
boundary ends at authentication, roles and the session, and a module that serves
game data brings its own.

Also fixed on the way: users.router.js was still listed as 15 routes (it is 9 —
six went to the extension slot), and the push section still promised
config/shardStreams.js "moves out with it" four slices after it left.

The acceptance table now carries results. Criterion 2 was proved for real against
module-uo v0.3.0 on an empty database, which turned up the uninstall ordering
defect fixed in website#146.

AI disclosure: this contribution was AI-assisted (Claude Code).

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
claude
2026-08-12 09:16:29 -05:00
parent 49ab5a1266
commit 42867d7ef8
7 changed files with 622 additions and 378 deletions
Download Patch File Download Diff File Expand all files Collapse all files

440
website/BACKEND_DESIGN.md
View File

@@ -100,12 +100,6 @@ server/
pages.router.js (2) /public/pages — the draft-preview
route precedes /:slug and is
deliberately not site-mode gated
shard.router.js (14) /public/shard/* incl. the anonymous
SSE stream; never site-mode gated
atlas.router.js (6) /public/atlas/* — the spawn atlas.
NOT under /shard: nothing here
touches the sidecar, and unlike
/shard it IS site-mode gated
modules.router.js (1) /public/modules — the installed-module
list a client feature-detects against.
A real prefix and not a fifth singleton
@@ -115,17 +109,18 @@ server/
site.router.js (4) /settings /status /version /contact —
the group-root singletons; declares no
router-level middleware
public.controller.js + shard.controller.js
public.controller.js
(/public/shard and /public/atlas are module-uo's — see
../modules/uo/API.md)
player/ index.js owns the shared `noindex, requireAuth` gate
(authenticated, ANY role — staff are a superset
of players) and the mount table
account.router.js (8) /player/account — credentials, TOTP,
linked identities; handlers shared
with /admin/account and /auth/me
shard.router.js (8) /player/shard — linking + own roster,
vendors, chars, sales, houses
appeals.router.js (4) /player/appeals
shard.controller.js + appeals.controller.js
appeals.controller.js
(/player/shard is module-uo's)
settings/ index.js owns the shared `noindex, requireAuth` gate
(authenticated, ANY role) and the mount table.
A fifth group, for site-wide settings that
@@ -145,7 +140,10 @@ server/
`noindex, isLoggedIn, staffOnly` gate and
declares no routes itself
account.router.js (6) /admin/account — self-service, no adminOnly
users.router.js (15) /admin/users — adminOnly
users.router.js (9) /admin/users — adminOnly. The six
/users/:id/shard/* routes are a
MODULE's, reached through the
admin.users.detail extension slot
invites.router.js (3) /admin/invites — adminOnly
authProviders.router.js (4) /admin/auth — adminOnly
moderation.router.js (15) /admin/moderation — modAccess
@@ -162,13 +160,11 @@ server/
pages.router.js (7) /admin/pages — CMS page builder
imageUpload.js shared multer config for the two
upload routes above (not a router)
shard.router.js (16) /admin/shard — 7 self-service
account-linking routes (no extra
gate, handlers shared with
/player/shard) + 9 in-game staff
ops on modAccess, per route
uoLink.router.js (5) /admin/uo-link — sidecar config,
town crier, admin SSE — adminOnly
modules.router.js (8) /admin/modules — adminOnly, the
module delivery surface: install
from a manifest URL, enable,
disable, uninstall, purge, restart
and the source allowlist
email.router.js (6) /admin/email — Gmail OAuth2
delivery — adminOnly
discordBot.router.js (2) /admin/discord-bot — adminOnly
@@ -177,7 +173,7 @@ server/
and carries its own key allowlist
(theming/nav keys + the hero draft)
so it can never drop site_mode or
the uo-link config; POST
a module's own seeded row; POST
/brand-asset/:slot uploads a
logo/hero/favicon and writes the
brand_assets row in the same call
@@ -191,6 +187,7 @@ server/
admin.controller.js + the per-capability controllers
(already domain-split; the split PRs re-wire
routes, not logic)
(/admin/shard and /admin/uo-link are module-uo's)
model/
users/ users.model.js + users.db.js
posts/ posts.model.js + posts.db.js (news/five-on-friday/newsletter/screenshots)
@@ -326,8 +323,10 @@ base `NAV` arrays are client constants, and duplicating them server-side would
create a second source of truth for navigation that drifts the first time a route
is added. `client/src/lib/navOverrides.js` drops an unknown `to` at merge time
instead, which is also what makes deleting a route in code safe. The merge runs
*before* the role and shard-feature filters in `SiteHeader.jsx` /
`AdminLayout.jsx`, which are unchanged and remain the boundary — a stored
*before* the role and feature filters in `SiteHeader.jsx` / `AdminLayout.jsx` —
a `feature` on a nav row is resolved by the module that **registered** the row
(`client/src/modules/featureGate.js`), so no flag string carries a parsed prefix
and core learns nothing about a game — and those filters remain the boundary: a stored
`hidden: false` on a gated item shows nobody anything. `hidden: false` is
accepted (the editor sends it mid-edit) but never stored, so hiding stays
subtractive. `hidden` on `/admin/navigation` is dropped for `nav_admin`, because
@@ -471,231 +470,15 @@ analogue to a password — and there is no hash-lookup constraint (verification
unused rows and `bcrypt.compare`s each, like password verification). `used_at` is the single-use
marker. Cleared wholesale on TOTP disable / password change / password reset.
### shard_ruleset — the shard's published ruleset (Protocol 3.0)
### The 27 shard tables — module-owned (module system)
Singleton row (`id = 1`, CHECK-constrained) holding the latest `world.ruleset` frame: `rev`,
`expansion`, `payload` JSON (the whole frame), `t`, `updated_at`. The shard re-emits the complete
ruleset on every sidecar connect, so this is an **overwrite, not an append** — and the kind is
deliberately **not** in `LOGGED_KINDS`, since logging it would put a duplicate row in `shard_events`
on every reconnect while `server.hello` already marks each of those.
`shard_*` and `uo_link_config` are **not core's**. They are created and dropped by `module-uo`'s own
schema fragment, and a core running without that module has none of them. Their shapes and the
reasoning behind them live with the module:
[`../modules/uo/SCHEMA.md`](../modules/uo/SCHEMA.md).
The frame is stored whole rather than normalized into columns: it is a flat description of server
config that is read as one page, so splitting it up would mean a schema change every time the shard
grows a new block. `rev` (the shard's FNV-1a of the body) and `expansion` are hoisted only because
they are cheap to display — the same payload-plus-hoisted-columns shape `shard_champs` uses.
**No row means the shard has never published one** (an older plugin, or `Bridge.RulesetEnabled=false`),
served as `null` rather than `{}`: "not published yet" and "published, everything off" are different
answers and the page renders them differently.
### shard_points_boards — points / loyalty leaderboards (Protocol 3.0)
One row per point system, keyed by the shard's own `PointsType` name (`QueensLoyalty`,
`CleanUpBritannia`, …). The shard carries ~25 of these, each a standing players build over months.
Columns: `system` (PK), `name`, `name_cliloc`, `max_points`, `players`, `show_on_gump`, `payload` JSON
(the whole `points.board` frame), `t`, `updated_at`.
**The top-N list stays inside `payload`** rather than being normalized into a `shard_points_entries`
table. It is a fixed-size list (10 by default) that is only ever read whole — exactly like
`shard_governors.candidates` — so normalizing buys nothing until something needs a per-character
reverse lookup, and a character's own standings already ride inside `char.profile` instead.
Board state, not events: `points.board` is **not** in `LOGGED_KINDS`, for the same reason
`guild.update` isn't. The shard emits a frame every time anyone's score moves a top ten, so logging
would grow `shard_events` without bound for something whose only interesting value is its latest
version. There is also **no delete path** — the shard's set of systems is fixed at startup, so there is
no `points.remove` to mirror.
Two values carry non-obvious meanings, both set by the plugin and both documented in
[`link/INTEGRATION.md`](../link/INTEGRATION.md) §4:
- **`max_points = 0` means uncapped**, and on a real shard that is the *common* case (ServUO's
uncapped idiom is `double.MaxValue`, which the plugin normalises to 0). Anything rendering
`points / max_points` must special-case it.
- **`name` is usually NULL**, with `name_cliloc` set instead — most systems name themselves with a
cliloc rather than a literal. Listing therefore orders by `COALESCE(name, system)`, so boards
awaiting cliloc resolution sort by their own key rather than clumping together under NULL.
### shard_vendors / shard_vendor_items — the player-vendor marketplace (Protocol 3.0)
The shard-wide shop index, fed by `vendor.listing` / `vendor.listing.remove`. One row per player
vendor and one per priced listing. Full operator detail in [`MARKETPLACE.md`](MARKETPLACE.md); the
design is `docs/link/v3.md` §8.
| Table | Shape |
|---|---|
| `shard_vendors` | `serial` (PK), `shop_name`, `owner_serial`, `owner_name`, `map`/`x`/`y`/`z`, `region`, `house`, `item_count`, `item_total`, `truncated`, `t`, `updated_at`. Indexes on owner, map, region and `updated_at`. |
| `shard_vendor_items` | `id` (PK), `vendor_serial`, `serial`, `item_id`, `hue`, `amount`, `price`, `name`, `cliloc`, `display_name`, `child`. Indexes on `vendor_serial`, `price`, `item_id`, `display_name`, and `(display_name, price)`. |
**Ingest is per-vendor and authoritative**: the frame is the whole shop, so ingest is
delete-then-insert of that vendor's listings inside one transaction. All-or-nothing matters
specifically because the two writes are "the shop" and "what is in it" — a failure between them
leaves a shop advertising an inventory it no longer has, which is visibly wrong and indistinguishable
from a genuinely empty shop. No foreign keys, consistent with every other `shard_*` table.
**There is deliberately no `payload` column**, unlike `shard_points_boards` directly above. The
board's top-N is a fixed-size list read whole, so it lives in JSON; here the items *are* the
searchable rows, so they are normalized and nothing is left worth duplicating. The sidecar keeps the
whole blob — outage resilience is its job, search is ours.
Market state, not events: neither kind is in `LOGGED_KINDS`, and this is the strongest case of the
three v3 kinds. One frame carries up to 250 listings and the sweep re-emits a shop on any price
change, so logging would turn `shard_events` into a price history nobody reads.
Two columns carry non-obvious meanings:
- **`item_count` vs `item_total`.** `item_count` is what the frame published; `item_total` is what
the shop actually holds. They differ when `truncated` — the shard caps listings per frame
(`Bridge.MarketMaxListings`, 250 by default), and a commodity reseller with thousands of stacks
genuinely exceeds it. Any UI must show both or it presents a partial shop as complete.
- **`display_name` is denormalized at ingest**, resolved from the item's literal `name` (preferred —
a player set it, so it is more specific) else its `cliloc` against `shard_clilocs`. Resolving at
query time would put the cliloc table on the hot path and make search-by-name impossible. Because
the shard's diff sweep will not re-send an unchanged shop just because the site learned what its
items are called, **a cliloc import triggers a bulk re-resolution** of this column (after a boot
import and after an admin import; ~50 ms per thousand rows, never throws).
`updated_at` is written explicitly on every upsert rather than left to `ON UPDATE CURRENT_TIMESTAMP`,
which MariaDB does not fire when every column is written back unchanged. A shop re-published
identically is still *freshly confirmed*, and without this the staleness banner would age a perfectly
current shop forever.
### shard_feature_visibility — per-feature audience config (Protocol 3.0)
One row per shard feature: `feature` (PK), `enabled`, `audience` (a rung on the ladder in §6.5),
`stream` (whether the feature's kinds fan out over SSE at all), `field_rules` JSON (`{field: rung}`
for the sensitive fields only), `updated_by`, `updated_at`.
**An absent row means "use the compiled default", and the compiled defaults reproduce pre-3.0
behavior — so an empty table is a no-op and there is nothing to seed.** Stored rows are merged over
the defaults on read, which is also where the invariants are re-applied: a row naming an unknown
feature is ignored (a stale row must not resurrect a removed feature), an invalid rung falls back to
the default rather than failing open, and a rule touching a locked field (`acct` / `webId`) is
discarded. See §6.5.
### shard_spawn_* / shard_regions / shard_landmarks / shard_champion_spawns / shard_atlas_meta — the spawn atlas (Protocol 3.0)
Static shard **content**, not live shard state. Nothing here comes from the sidecar: the atlas is
derived from the shard's own ServUO tree, re-read on **every server boot** and hash-gated so an
unchanged tree costs one read pass and no write. Nothing is precomputed and committed — a shard's
maps change over its life, and a snapshot in the repo would silently drift from the world players
actually see. These tables stay populated whether the shard is up or not. Full operator detail in
[`SPAWN_ATLAS.md`](SPAWN_ATLAS.md); the design is `docs/link/v3.md` §6.
**No facet name appears anywhere in the code.** A shard may add facets, replace them, or rename them
when its maps are updated; the facet set is discovered from the tree, and the loose spellings in
`Data/Locations` are matched against it rather than looked up in a table.
| Table | Key columns |
|---|---|
| `shard_spawn_creatures` | `slug` PK, `name`, `total`, `points`, `facets` JSON, `art` NULL |
| `shard_spawn_points` | `id` PK, `facet`, `name`, `x`, `y`, `width`, `height`, `spawn_range`, `max_count`, `min_delay`, `max_delay`, `tod_start/end/mode`, `region`, `landmark`, `label` |
| `shard_spawn_point_types` | `(point_id, slug)` PK, `max_count` |
| `shard_regions` | `facet`, `name`, `type`, `priority`, `parent`, `rects` JSON |
| `shard_landmarks` | `facet`, `name`, `grp`, `x`, `y`, `z` |
| `shard_champion_spawns` | `slug` PK, `name`, `grp`, `type`, `random_type`, `facet`, `x`, `y`, `z`, `radius`, `label` |
| `shard_atlas_meta` | Singleton (`id = 1`), `payload` JSON (counts, a sha256 per source file, `parserVersion`), `imported_at` |
| `shard_atlas_pending` | Singleton (`id = 1`), `status` (`pending`/`rejected`), `payload` JSON, `detected_at` |
The first seven are **import-owned**: a refresh empties and reloads every one inside a single
transaction, so a failed reload leaves the previous atlas intact rather than a half-loaded world.
Nothing else writes to them, and nothing holds a foreign key to them — no FKs at all, consistent with
every other `shard_*` table.
**`shard_atlas_pending` is the security-relevant one.** A refresh that would REMOVE a facet is never
applied automatically: facet loss is indistinguishable at boot from a half-copied or mid-update tree,
so it is staged here for an admin to approve or reject, and **startup is never blocked by it**. Only
the decision is stored — source hashes plus the facet diff, a few KB — and approving re-parses the
tree, so a multi-megabyte blob never lands in the database and what gets applied matches the tree at
approval time. A rejection is remembered against those exact hashes so a declined refresh does not
re-prompt on every restart. Everything else (new facets, renamed regions, changed spawns) applies
immediately, since none of it can destroy data an operator would miss.
The boot refresh is **best-effort by contract**: no configured path, an unreadable mount, a malformed
file or a database error is caught and logged, and the site comes up serving whatever atlas it had.
The tree path comes from the `spawn_atlas_servuo_path` setting, falling back to `SERVUO_PATH`.
**A refresh re-derives when the tree changed OR the parser did.** `spawnAtlasSource.PARSER_VERSION`
is stored in `shard_atlas_meta` beside the source hashes and bumped whenever the parser produces
different data from identical files. Hashing the tree alone would strand an install whose maps never
change on whatever an older build derived — a corrected parse would ship and never reach the data.
Four column choices worth stating, because each one is a trap:
- **`spawn_range`, not `range`**, and **`grp`, not `group`** — both are reserved words.
- **`DELETE`, not `TRUNCATE`.** `TRUNCATE` is DDL in MariaDB and implicitly commits, which would
defeat the all-or-nothing reload. At ~7k rows the difference does not matter.
- **Point ids are assigned explicitly**, not left to `AUTO_INCREMENT`: the `shard_spawn_point_types`
rows need to know them, and `conn.batch()` reports no usable `insertId` for a multi-row insert.
- **Plain `INDEX` on `name`, deliberately not `FULLTEXT`.** ~800 creature rows makes a `LIKE` scan
free, and FULLTEXT's minimum token length would break searches for names like "orc".
`shard_champion_spawns` is the *configured* altar roster ("there is an Unholy Terror altar in
Deceit"). The live `champ.update` feed in `shard_champs` is the separate answer to "it is on level 3
right now". Both exist; they are not the same data.
**`shard_spawn_creatures.art` is always NULL on a fresh import.** The project ships no creature
artwork: sprites live in the operator's own client `.mul`/`.uop` files and are theirs, not ours to
redistribute. An operator supplies art via a gitignored map plus images under the (already
gitignored) `server/uploads/atlas/`. Text-only is the normal, supported state.
### shard_clilocs / shard_cliloc_meta — UO's localization table (Protocol 3.0)
Items on the wire carry a `LabelNumber`, not a name. The bridge has always sent it —
`char.profile.equipment.cliloc`, reward titles as a cliloc number in string form, and one per
marketplace listing — but with no table to resolve it against, the character sheet could only render
`id 1023721` where the game renders "quarter staff".
| Table | Shape |
|---|---|
| `shard_clilocs` | `number` INT PK, `flag`, `text` TEXT |
| `shard_cliloc_meta` | Singleton (`id = 1`), `payload` JSON (source file, sha256, count, `parserVersion`), `imported_at` |
Import-owned and all-or-nothing in one transaction, same contract as the atlas — including **`DELETE`,
not `TRUNCATE`**, for the same reason.
**Sourced from files the operator supplies**, at a path from the `cliloc_client_path` setting falling
back to `UO_CLIENT_PATH`. Nothing client-derived is committed: UO's strings are EA's, exactly as the
creature sprites are. A shard with nothing configured is fully supported — names render as ids. Full
design and operator guide: [`CLILOCS.md`](CLILOCS.md).
**It reads a SET of sources, not one file**, because shards edit items and add new ones and those
carry cliloc ids no stock client table has. A base (the converted client table) plus every overlay
under `custom/` are re-read on every boot and hash-gated **together**, exactly as the atlas re-reads
`Regions.xml` + `Locations/*.xml` + `Spawns/*.xml` + `ChampionSpawns.xml`. Later sources win, so an
overlay both adds ids and overrides stock ones, and adding one custom item never means re-exporting a
5 MB client file. Scale, measured on the live shard: its script tree references 16,434 cliloc ids and
only 37 are absent from stock — tens of entries against a 67k base, which is why this is an overlay
and not a second table.
The conversion step is not avoidable: **every current client ships its cliloc files compressed**
(first DWORD's high byte `0x8E`), and ServUO's own bundled `Ultima.StringList` cannot read that
either — so the shard cannot supply names on our behalf. The plain layout and a delimited text export
are both accepted, sniffed by header rather than extension.
Three decisions worth stating:
- **`text` is TEXT, not VARCHAR.** Long property descriptions reach 12 KB. The index that matters for
marketplace search is the denormalized `shard_vendor_items.display_name`, not this table.
- **Blank entries are dropped at import** — 123,490 parsed → **67,496** stored. Roughly half a cliloc
table is empty strings for ids the client reserves and never uses; a row that resolves to no name is
indistinguishable from no row at all, and dropping them makes the binary and text imports converge
on identical content.
- **Two refusals, one of them the atlas's.** A corrupt source fails the parse on a truncated record,
so it is caught outright and leaves the previous table serving. But a source that has **vanished**
parses perfectly and imports a table quietly missing everything it contributed — an unmounted volume
and a deliberate deletion are indistinguishable from here, which is precisely the ambiguity the
atlas stages a facet removal for. So it is escalated: `status: 'needsReview'`, nothing applied,
`missingSources` reported by both the import and `status()`, and an admin accepts it with
`{approve:true}`. That is a flag rather than the atlas's approve/reject pair because the atlas
stores a pending decision so that approving **re-parses** the tree; here nothing is stored, so
re-reading at approval time is automatic.
**Resolution is server-side and there is no public route.** The table is never served *as* a table:
67k rows would dwarf any page using them, and the Android client consumes the same already-resolved
JSON. `resolveMany()` returns only ids that resolved to something displayable — placeholders like
`~1_val~` are stripped, since the bridge sends the id and never the property packet that carries the
arguments — and it never throws, because a cliloc lookup is decoration on a character sheet.
The prefixes are grandfathered ([`MODULE_API.md`](MODULE_API.md) §6.5) — a new module prefixes its
tables with its own id.
### installed_modules — what is installed, and what happened to it (module system)
@@ -863,12 +646,12 @@ for web back-compat.
**The `/player/*` group is self-service, not player-only.** Staff are a **superset** of players — every
player ability plus their staff tools on top — so the whole group (`account.router.js`,
`shard.router.js`, `appeals.router.js`, mounted by `player/index.js`) sits behind the shared
`noindex, requireAuth` gate **only**, never `requireRole('player')`. Every handler is self-scoped to the caller by `req.user.id`, so an admin/editor/
`appeals.router.js`, mounted by `player/index.js`, plus whatever a module mounts here) sits behind
the shared `noindex, requireAuth` gate **only**, never `requireRole('player')`. Every handler is self-scoped to the caller by `req.user.id`, so an admin/editor/
moderator using it sees only their **own** linked accounts and characters (with the pre-existing
`isAdmin` bypass still letting a genuine admin read *any* character). Staff also reach the identical
self-scoped handlers under `/admin/shard/*` (same controller) for the web admin surface; the two are
interchangeable. This is why a staff account with linked game characters gets its "My characters" and
`isAdmin` bypass still letting a genuine admin read *any* character). `module-uo` inherits the rule
and relies on it: its `/player/shard/*` handlers are the identical self-scoped ones it also serves
under `/admin/shard/*`, so the two are interchangeable. This is why a staff account with linked game characters gets its "My characters" and
personal notification streams on the mobile client — the group no longer 403s a non-`player` role.
**Password reset.** Uses the same audited pattern as `user_invites`: an opaque 32-byte token
@@ -886,16 +669,17 @@ checked content over the authenticated API. Two producers fan out through the on
ingest dispatcher (`utils/shardIngest`, beside the SSE broadcast) for shard-derived streams, and the
create/publish-post path for `news.post`. The catalog is assembled at boot by
`modules/registries.js` from core's own streams (`config/coreStreams.js` — just `news.post`) plus
each installed module's; the shard streams and their event→stream mapping are
`config/shardStreams.js`, which belongs to module-uo and moves out with it
(MODULE_SYSTEM.md §1.8). Security invariants:
- **Same public/admin split as the SSE feed.** Public streams are drawn *only* from the SSE
`PUBLIC_KINDS` allowlist; a sensitive kind (audit/cheat/IP/login-attempt) can never produce a public
push.
- **Personal streams are owner-keyed.** `vendor.sale` / `house.idoc` / `account.login` are delivered
only to the *owning* user's devices, resolved via `shardLinks` (the same ownership check as
`/player/shard/*`) in `utils/shardPush.js`. `utils/pushDispatch.js` itself only publishes to a
stream id someone else resolved — it has no idea what a shard event is.
each installed module's. The seven shard streams and their event→stream mapping left with
`module-uo` in Phase 3 and are registered by it; their ids are grandfathered to that module
([`MODULE_API.md`](MODULE_API.md) §6.5) because they are stored in `notification_subs` and read by
the shipped Android app. Security invariants:
- **Whether a stream is safe to publish is the registering module's decision, and it stays inside
that module.** `module-uo` applies the same public/admin split as its SSE feed — public streams are
drawn only from its own allowlist, so a sensitive kind (audit/cheat/IP/login-attempt) can never
produce a public push — and resolves personal streams (`vendor.sale`, `house.idoc`,
`account.login`) to the *owning* user's devices through its own ownership check. Core never sees a
shard event. **`utils/pushDispatch.js` publishes to a stream id someone else resolved** and knows
nothing about what produced it, which is what lets a second game's module reuse the whole pipe.
- **SSRF guard.** A device `endpoint` is a client-supplied URL the server POSTs to, so registration and
every publish validate it is HTTPS, non-private/loopback, and (when configured) on the shard's ntfy
allow-set (`NTFY_BASE_URL` / `NTFY_ALLOWED_ORIGINS`).
@@ -973,7 +757,8 @@ expiry (~5 min); `/exchange` is rate-limited per-IP. The bridge tables self-prun
**No group gate, deliberately.** This surface is anonymous by design: the SPA renders it logged-out,
the Discord bot reads it with no credentials, and the Android `ShardStreamClient` consumes
`/public/shard/stream` without an `Authorization` header. Content visibility during maintenance comes
module-uo's `/public/shard/stream` without an `Authorization` header — a module mounting here
inherits the same "no gate" and owns whatever gate it adds. Content visibility during maintenance comes
from the per-route **siteMode** middleware (§5), never from an auth gate.
| Method | Path | Notes |
@@ -987,19 +772,7 @@ from the per-route **siteMode** middleware (§5), never from an auth gate.
| GET | `/wiki` | list of pages (slug + title) |
| GET | `/wiki/:slug` | single page |
| POST | `/contact` | (rate-limited) send mail via SMTP; if unconfigured, respond `{fallback:"mailto", email}` |
| GET | `/shard/ruleset` | the shard's own published ruleset (Protocol 3.0 `world.ruleset`): expansion, which optional systems are on, skill/stat caps, account and house limits, champion scroll rules, the save/restart schedule. Served from `shard_ruleset`, so it renders while the shard is down; live via `world.ruleset` on `/shard/stream`. Behind `requireFeature('ruleset')`. **`null`** means the shard has never published one — a real answer, distinct from a published ruleset. `caps.skill` / `caps.totalSkill` are in **tenths** (1000 = 100.0). |
| GET | `/shard/points` | every points/loyalty leaderboard the shard publishes (Protocol 3.0 `points.board`) — Queen's Loyalty, Void Pool, the nine city loyalties, Clean Up Britannia, … Served from `shard_points_boards`, so it renders while the shard is down; live via `points.board` on `/shard/stream`. Behind `requireFeature('leaderboards')`, ordered by display name. **`maxPoints: 0` means uncapped** (the common case), and `nameString` is usually `null` with `nameNumber` holding a cliloc — resolve client-side or humanise the `system` key. |
| GET | `/shard/points/:system` | one board by the shard's `PointsType` name (e.g. `QueensLoyalty`); `:system` must match `/^[A-Za-z][A-Za-z0-9_]{0,47}$/` or **400** before any query runs. **404** = the shard has never published that system, which is distinct from a published board nobody has scored in yet (**200** with an empty `top`). |
| GET | `/shard/market?q=&minPrice=&maxPrice=&itemId=&map=&region=&sort=&limit=&offset=` | search the player-vendor marketplace (Protocol 3.0 `vendor.listing`). Returns **listings**, not vendors — "who sells X and for how much" is the question, and a vendor-shaped result would make every caller flatten the shops back out. Served from `shard_vendors` + `shard_vendor_items`, so it renders while the shard is down. Behind `requireFeature('market')` **and rate-limited** — the first genuinely expensive public read on the site (a `LIKE` scan plus a `COUNT` over what is typically the largest `shard_*` table, reachable with no session). `sort ∈ {price_asc, price_desc, recent}`. `q` matches the resolved display name **or** the item's literal name, with `%`/`_` escaped: they are `LIKE` metacharacters, not SQL ones, so parameterization alone would let `?q=%` match every listing on the shard. Every response repeats `staleAt` (the oldest vendor row) because the shard sweeps round-robin — a banner that ages with the results it labels, not one fetched once. |
| GET | `/shard/market/meta` | index size, staleness (`staleAt`/`freshAt`) and which facets and regions actually hold vendors, so a client builds its filters without running a search it will discard. |
| GET | `/shard/market/vendors/:serial` | one shop and its listings; `:serial` must match `/^0x[0-9A-Fa-f]{1,16}$/` or **400** before any query runs. **404** = a serial the index has never seen, which also covers a vendor since dismissed or hidden — to an anonymous caller those are the same answer, and distinguishing them would leak that a hidden vendor exists. `truncated` (with `total` exceeding `count`) means the shop holds more than the shard publishes per frame. |
| GET | `/shard/features` | the shard features **this caller** may reach plus the audience rung they resolved to (§6.5), so a client hides nav it can't follow. Reports only what the caller can see — the list itself never discloses a gated feature. Consumed by the SPA header and (pending) the Android nav. |
| GET | `/atlas/creatures?q=&facet=&limit=&offset=` | the bestiary, most numerous first, with an unpaginated `total`. Static content parsed from the shard's ServUO tree — **not** sidecar-backed, which is why the atlas sits outside `/shard`, and unlike `/shard/*` it **is** site-mode gated. Behind `requireFeature('atlas')`. `?facet=` is matched exactly and never validated against a list (no facet name exists in the code); the filter is an `EXISTS` over the points rather than a JSON path or `JSON_SEARCH` built from caller input, whose `%`/`_` wildcards would make `?facet=%` match everything. |
| GET | `/atlas/creatures/:slug` | one creature: `places` (the point-in-rect aggregate — "lizardman → Shrines, Isamu-Jima, Yew"), `spawners` (the bounded raw list, with `spawnersTruncated`), `alsoHere`. **`points` is a COUNT and `spawners` is the LIST** — named apart so one key never means a number on one route and an array on another. `minDelay`/`maxDelay` are in **seconds**, normalised at parse time from the source's per-record minutes-or-seconds. 404 = no such creature in this atlas. |
| GET | `/atlas/regions?facet=&q=` | named regions and the rectangles that placed each spawner |
| GET | `/atlas/landmarks?facet=&q=` | points of interest, labelled by `group` ("Covetous", not "Level 1") |
| GET | `/atlas/champions?facet=` | the **configured** altar roster. Not `/shard/champs`, which is the live board. |
| GET | `/atlas/meta` | facets, counts and when the atlas was parsed. Game-world facts only — the ServUO path, source hashes and any pending refresh are operator detail and live on the admin route. |
| — | `/shard/*` · `/atlas/*` | **Served by `module-uo`, not by core** (25 routes). Documented in [`../modules/uo/API.md`](../modules/uo/API.md); absent entirely when the module is not installed, which is a 404 and not an error. |
Public content GETs pass through the **siteMode** gate (§5).
@@ -1021,14 +794,18 @@ These rows are configuration that happens to need a login.
`users`, `invites`, `auth/providers` and `bot-activity` add `adminOnly` on top, and `moderation` adds
`modAccess` (admin + moderator, so editors are excluded). The content capabilities — `posts`,
`uploads`, `wiki`, `pages` — add nothing: managing content is the editor tier's job, so `staffOnly` is
the whole gate. The ops/config capabilities — `uo-link`, `email`, `discord-bot`, `settings`, and
`PUT /site-mode` — are `adminOnly`; `shard` is the one mixed prefix, where self-service account
linking carries no extra gate and the in-game staff operations carry `modAccess`. There is no residual
file: every admin route is declared in a capability router.
the whole gate. The ops/config capabilities — `modules`, `email`, `discord-bot`, `settings`, and
`PUT /site-mode` — are `adminOnly`. There is no residual file: every admin route is declared in a
capability router.
`GET`/`PUT /admin/shard/visibility` are the third tier on that mixed prefix: **`adminOnly`**, because
they decide what *anonymous* visitors can see (§6.5). They sit above `modAccess` deliberately — a
moderator can ban a player but cannot decide what the public internet reads.
**A module mounts into this group as a peer**, at a prefix it claims and core has verified nothing
else owns; the shared gate above applies to it, and any gate beyond that is the module's own. So the
mixed-tier prefixes here are `module-uo`'s `/admin/shard` and `/admin/uo-link`, documented in
[`../modules/uo/API.md`](../modules/uo/API.md) — not core's, and absent from this table.
`/admin/modules` is `adminOnly` rather than `staffOnly` for the reason the endpoint exists: it
installs code that will run inside the server process at the next boot. An editor or a moderator has
no business doing that, and the group gate alone would let them.
`GET /dashboard` and `PUT /site-mode` are the one place where a **single screen spans two tiers**: the
dashboard is staff-wide, but the site-mode toggle on it is `adminOnly`. The client must therefore gate
@@ -1057,13 +834,15 @@ file a route sits in — that is the property the route manifest freezes.
| GET | `/users/:id/trusted-devices` | list a user's active trusted devices (never tokens) |
| DELETE | `/users/:id/trusted-devices` · `…/:deviceId` | revoke all / one of a user's trusted devices (logs `admin.trusted_device.revoke[_all]`) |
| POST | `/users/:id/mfa/reset` | recover a locked-out user: disable TOTP + revoke all trusted devices + clear recovery codes (logs `admin.user.totp.reset`) |
| GET | `/shard/atlas` | spawn-atlas status (`adminOnly`): the ServUO path, whether the tree is readable, whether it has drifted from what is loaded, counts, facets, and any refresh staged for review. The public `/atlas/meta` reports the game world only; the filesystem detail is here. |
| POST | `/shard/atlas/import` | re-import without restarting; `{force}` ignores the hash gate. **An unreadable tree answers 200 with `status:"unavailable"`, not 500** — `refresh()` reports outcomes rather than throwing (the boot path must never be blocked by a bad tree) and that contract is preserved at the API. |
| POST | `/shard/atlas/approve` · `/shard/atlas/reject` | answer a refresh staged because it would REMOVE a facet. Approving **re-parses** the tree, so what lands matches it at approval time; rejecting is remembered against those source hashes so it does not re-prompt every restart. 404 when nothing is staged. |
| PUT | `/shard/atlas/path` | point the atlas at a different tree (persisted as `spawn_atlas_servuo_path`, which wins over `SERVUO_PATH`). Blank clears it. Deliberately **does not import** — moving the mount and reloading the world are separate decisions — and returns fresh status so the panel can offer the import next. |
| GET | `/shard/clilocs` | cliloc-table status (`adminOnly`): every source found now (base first, then `custom/` overlays in merge order), what each contributed at the last import, readability, drift across the set, the entry count, and `missingSources`. `configured:false` is a supported state — item names then render as ids. No public counterpart: the table is never served *as* a table. |
| POST | `/shard/clilocs/import` | reload after a client patch or an overlay edit; `{force}` ignores the hash gate, `{approve}` accepts a **vanished** source (refused by default — see the table notes above). **A missing path — or the likely mistake of pointing at the client's own COMPRESSED `Cliloc.enu` — answers 200 with `status:"unavailable"` and a `code`, not 500.** `COMPRESSED` is called out by name: a 500 would say only "something broke", and the operator needs to be told which file to convert. |
| PUT | `/shard/clilocs/path` | point the site at a different cliloc base file or directory (persisted as `cliloc_client_path`, which wins over `UO_CLIENT_PATH`). Overlays are read from `custom/` beside it either way. Blank clears it. Deliberately **does not import**, same reasoning as the atlas path. |
| GET | `/modules` | installed modules reconciled across all four sources of truth — the `installed_modules` row, the live loader record, the modules volume, and the `MODULES` declaration — plus the install-source allowlist. They are allowed to disagree, and the screen renders the disagreement rather than picking one (module system, [`MODULE_SYSTEM.md`](MODULE_SYSTEM.md) §2.4) |
| POST | `/modules` | install or upgrade from a release **install-manifest URL**: allowlisted `https` host, declared `sha256`, whole-archive inspection, unpack into a scratch dir, move into place last. Takes effect at the next restart. `adminOnly`, rate-limited, audit-logged — this endpoint installs code that will run in the server process |
| PUT | `/modules/sources` | replace the host allowlist. Seeded from `MODULE_SOURCE_HOSTS` on a fresh install and DB-owned from then on, so changing the variable never overwrites an operator's choice. An **empty list forbids every install**, never permits all |
| POST | `/modules/restart` | graceful shutdown so module changes take effect; the supervisor brings the process back (`restart: unless-stopped` on the shipped compose). Emits `SIGTERM` **as an event** rather than signalling the pid — `process.kill` is unconditional termination on Windows |
| POST | `/modules/:id/enable` | move the row to `enabled`. Deliberately does **not** touch the loader: there is no `onBoot` re-dispatch, so the screen asks for a restart |
| POST | `/modules/:id/disable` | the one module action that takes effect immediately — dispatches that module's `onShutdown`, then its routes, nav and client chunk answer 404. A real kill switch, not a visibility flag |
| POST | `/modules/:id/purge` | run a **disabled** module's `purge.sql`, dropping its tables and data. `409` while it is still running; `400` if it ships no `purge.sql` |
| DELETE | `/modules/:id[?purge=true]` | uninstall: stop, then (with `purge=true`) drop its data, then delete its directory. Non-destructive by default — the row stays `disabled` and the data is left for a reinstall to pick up. The purge option lives here because it cannot live after: `purge.sql` is a file inside the directory being deleted |
| — | `/shard/*` · `/uo-link/*` | **Served by `module-uo`, not by core** (33 routes). Documented in [`../modules/uo/API.md`](../modules/uo/API.md) |
Every admin write logs to `activity_log`.
@@ -1133,11 +912,11 @@ who"; `activity_log` provides the history feed.
- **Cookie**: `httpOnly`, `sameSite=Lax`, `path=/`, and **`secure` decided per-request** (`COOKIE_SECURE=auto` → `secure: req.secure`).
- **Trusted-device MFA.** A second, separate httpOnly cookie (`rg_trust`, default 30d) — opaque, sha256-hashed server-side in `trusted_devices` — lets a browser/app **skip the TOTP step** (never the password) on future logins. It is a server-side, per-row-revocable record (never a JWT claim), so the stateless session JWT is unchanged and trust stays revocable. It only ever gates the **second factor**; it deliberately outlives logout, and is cleared on untrust / password change / password reset / TOTP disable. **Recovery codes** (bcrypt, single-use) are the 2FA-lockout fallback. All admin trusted-device/MFA actions and the self actions (`auth.login.trusted_device`, `account.trusted_device.*`, `account.recovery_code*`, `admin.trusted_device.*`, `admin.user.totp.reset`) are audit-logged. See `docs/website/TRUSTED_DEVICES_MFA.md`. This is the key to dual access: the cookie is `Secure` when reached through Pangolin (HTTPS, `X-Forwarded-Proto: https`) but **not** `Secure` when reached directly over the LAN IP on plain HTTP — so login works in both. `COOKIE_SECURE=true|false` can force it. Requires `trust proxy` (below). `localhost:5173` (Vite) and `localhost:3000` are same-site, so the cookie flows in dev too.
- **bcrypt** hashing (cost 10+); plaintext passwords never stored, logged, or returned.
- **Rate limiting** (`express-rate-limit`) on `/auth/login`, `/public/contact`, and — the only limited
*read* — `/public/shard/market` and `/public/shard/market/vendors/:serial` (60/min/IP). Every other
public read is an indexed lookup of bounded size; the marketplace search is a `LIKE` scan plus a
`COUNT` over the largest `shard_*` table, anonymous by default, so it is the one public GET that is
worth money to serve.
- **Rate limiting** (`express-rate-limit`) on `/auth/login`, `/public/contact`, and the account-change
routes. Every core public read is an indexed lookup of bounded size, so none is limited. A module
gets the same factory through `ctx.middleware.rateLimit` and is expected to use it on any read that
is expensive to serve — `module-uo` limits its marketplace search (60/min/IP), the one public GET
in the system that costs real money to answer.
- **Validation** (`express-validator`) on all writes; centralized error handler.
- **helmet** with a Content-Security-Policy tuned for the built React SPA. The policies now live in
**`server/src/config/csp.js`** (`app.js` only wires them up):
@@ -1181,76 +960,17 @@ who"; `activity_log` provides the history feed.
- **`app.set('trust proxy', 1)`** so secure cookies, `req.ip`, and rate-limiting work behind Pangolin.
- **CORS**: same-origin in prod (SPA served by Express). Dev only: allow `CLIENT_ORIGIN` (Vite, `http://localhost:5173`) with `credentials:true`.
### 6.5 Shard visibility — the audience boundary (Protocol 3.0)
### 6.5 Module-owned audience boundaries
Every shard-derived surface is gated by an **admin-configurable, per-feature and per-field** audience
setting. This **replaces** the static `PUBLIC_KINDS` allowlist that used to be the whole boundary.
Policy lives in `utils/shardVisibility.js`; rows live in `shard_feature_visibility`; the admin surface
is `GET`/`PUT /admin/shard/visibility` (`adminOnly`). Admin-facing guide:
[`SHARD_VISIBILITY.md`](SHARD_VISIBILITY.md). Design: [`../link/v3.md`](../link/v3.md) §3.
Core's security boundaries end at authentication, roles and the session. A module that serves
game data brings its own audience rules, and core does not police them beyond the gates it hands
over (`requireAuth`, `requireRole`, the tier group gates).
**The ladder.** `anonymous < logged_in < player < staff < admin`, each rung implying the ones below.
`viewerLevel(req)` resolves it: no session ⇒ `anonymous`; authenticated ⇒ `logged_in`; authenticated
with a linked game account ⇒ `player`; moderator ⇒ `staff`; admin ⇒ `admin`. **Staff satisfy the
`player` rung without a linked account** (consistent with `/player/*` being role-agnostic).
**`editor` gets no shard privilege** — it is a content role, and mapping it to `staff` would silently
widen what editors see.
**Two invariants that are code, not configuration.** Both are enforced server-side and both reject
rather than silently ignore:
1. **`acct` and `webId` are admin-only, always.** They are not exposed as configurable fields, and a
stored row attempting to loosen them is discarded on read as well as rejected on write. A character
name is visible in game; the account behind it and the website user it links to are not.
The lock is on the field's **meaning, not one spelling**: `isLockedField(key)` matches a key that
*is* or *ends in* `acct`/`webId`, case-insensitively, so the flattened forms the read models emit
(`shapeHouse` → `ownerAcct`, `shapeGuild` → `leaderWebId`) are covered too. An exact-key check was
the original implementation and it let `GET /public/shard/idoc` serve `ownerAcct` anonymously.
2. **A kind absent from `KIND_FEATURE` is never broadcast below `admin`.** Fail closed. This is what
keeps the kind map a security boundary rather than a convenience filter, and it means a shard that
starts emitting an unknown event degrades to staff-only, never to public.
**Fail-closed everywhere else too.** An unreadable visibility config withholds every public frame; a
DB failure falls back to the compiled defaults (pre-3.0 behavior), not to open; an unresolvable viewer
subscribes as `anonymous`. The ladder comparison uses **asymmetric** fallbacks by design — an unknown
*viewer* level floors to the bottom rung and an unknown *requirement* ceils to admin, so an
unrecognised value loses on both sides. (A single shared fallback cannot do that: whichever direction
it picks, it fails open on one side.)
**Three enforcement points, one config:**
| Where | Mechanism |
|---|---|
| Routes | `requireFeature(name)` — **404** when the feature is disabled (don't leak that it exists), **403** when the caller is below its audience. `projectFeature` then strips out-of-rung fields from the body. |
| SSE (`utils/shardBroadcast.js`) | Per-connection filtering. A subscriber's rung is resolved **once at subscribe time and frozen** for that connection, so a long-lived stream can't gain privilege; each frame is then mapped kind→feature, gated, and field-projected per viewer. Two subscribers can legitimately receive different versions of one event, or one of them nothing. |
| Nav | `GET /public/shard/features` returns only what the caller may reach, so the SPA never renders a link that would 403. Presentation only. |
Config reads are cached ~5s, so admin changes take effect within seconds **including on already-open
streams**. `PUBLIC_KINDS` still exists and is still exported (`utils/shardBroadcast.js`) but is now
**derived** from the kind map rather than hand-maintained, so the two cannot drift.
**`PUBLIC_KINDS` is a module-load constant and must not be used to answer "may this caller read this
kind?"** — it is computed from the compiled *defaults*, so it cannot see an admin's changes. Use
`visibleKinds(level, config)`, which resolves against the live config. `/feed` uses it; it originally
used `PUBLIC_KINDS` and consequently kept serving `guild.join` to anonymous callers after an admin had
moved `guilds` to `staff`. `visibleKinds` deliberately ignores the `stream` flag: that governs SSE
fan-out only, so a feature whose live firehose ships off (market) stays readable from stored history.
**Every read path that returns shard data must call `projectFeature`.** The stored-history endpoints
are not exempt — `/feed` returns the same events the stream does, and returning them unprojected
reopens on the REST side exactly what the stream closes. Relatedly, `shardEvents.db.list` treats an
**empty** `kinds` array as "serve nothing", never "no filter"; the fall-through it used to take would
have turned a fully-gated config into a dump of the entire event log.
`projectFeature` walks **arrays and plain objects only**. A `Date`, `Buffer` or other class instance
is passed through as a value — rebuilding one key-by-key yields `{}`, which is the difference between
the pure-JSON wire frames and the DB-backed read models whose rows carry real `Date` columns.
**Defaults reproduce pre-3.0 behavior exactly**, so installing the framework is a no-op until an admin
changes something — with deliberate exceptions, which are the leaks it was written to close.
`/public/shard/guilds`, `/public/shard/governors` and `/public/shard/feed` previously returned the raw
stored payload, whose actors carry `acct` and `webId`; `/public/shard/idoc` returned the flattened
`ownerAcct`. All are now stripped for every caller below admin.
`module-uo`'s is the worked example, and it is a real boundary rather than a convenience filter: an
admin-configurable, per-feature and per-field audience ladder with fail-closed defaults, applied at
routes, at SSE subscribe time and at the nav. It used to be documented here as core's; it moved to
[`../modules/uo/API.md`](../modules/uo/API.md) §4 when Phase 4 closed, with the admin-facing guide
still at [`SHARD_VISIBILITY.md`](SHARD_VISIBILITY.md).
---