diff --git a/modules/uo/API.md b/modules/uo/API.md new file mode 100644 index 0000000..af69c1a --- /dev/null +++ b/modules/uo/API.md @@ -0,0 +1,155 @@ +# module-uo — its HTTP surface + +The **72 URLs** `module-uo` serves, and the audience boundary that gates them. Frozen in the module's +own [`routes.manifest.json`](https://gitea.whitlocktech.com/RunicGateway/Module-uo/src/branch/main/routes.manifest.json) +and documented operation-by-operation in its +[`swagger-fragment.json`](https://gitea.whitlocktech.com/RunicGateway/Module-uo/src/branch/main/swagger-fragment.json), +which core merges into `/api/docs.json` while the module is running — so the live Swagger UI is +always the most complete answer. + +This page moved out of [`BACKEND_DESIGN.md`](../../website/BACKEND_DESIGN.md) §4 and §6.5 when +Phase 4 closed ([`MODULE_SYSTEM.md`](../../website/MODULE_SYSTEM.md) §2.7.2). Core's API contract +describes core's routes; these are the module's, and core cannot answer them with the module absent. +The tables and the reasoning are unchanged. + +**Every URL is byte-identical to the one core served before the extraction** — that is §1.2 of the +module plan, and it is what lets the shipped Android app keep calling +`POST /api/v1/admin/shard/kick` and the Discord bot keep reading `/api/v1/public/shard/*` without +knowing a module answers now. + +## 1. The mounts + +| Mount | Routes | Tier and gate | +|---|---|---| +| `/api/v1/public/shard` | 19 | Anonymous. **Never site-mode gated** — the shard surface stays readable during maintenance, per feature audience. | +| `/api/v1/public/atlas` | 6 | Anonymous, and unlike `/shard` it **is** site-mode gated: nothing here touches the sidecar, it is parsed shard content. | +| `/api/v1/admin/shard` | 26 | Behind core's `isLoggedIn + noindex + staffOnly` group gate, then **mixed per route** — see below. | +| `/api/v1/admin/uo-link` | 7 | `adminOnly`. The sidecar connection config, its live status, the admin SSE stream and the town crier. | +| `/api/v1/player/shard` | 8 | `requireAuth`, **any role** — staff are a superset of players — and every handler is self-scoped to `req.user.id`. | +| `/api/v1/admin/users/:id/shard/*` | 6 | `adminOnly`. The module's routes hanging off a **core** resource, through core's `admin.users.detail` extension slot: core owns the user, the module owns what it knows about their game accounts. | + +**`/admin/shard` is the one mixed prefix**, and it is mixed because it carries three different jobs: + +- the **self-service account-linking** routes carry no gate beyond `staffOnly` — they are the same + handlers `/player/shard` serves, reached from the admin surface; +- the **in-game staff operations** (kick, ban, unban, broadcast, pages) carry `modAccess` + (admin + moderator, so editors are excluded); +- **`GET`/`PUT /admin/shard/visibility` are `adminOnly`**, a third tier above `modAccess`, because + they decide what *anonymous* visitors can see (§4). A moderator can ban a player but cannot decide + what the public internet reads. + +Every admin write logs to core's one `activity_log`, through `ctx.activity.log` +([`MODULE_API.md`](../../website/MODULE_API.md) §2.7) — an admin action a module performs is not +allowed its own audit trail. + +## 2. Public routes worth their own note + +The full list is in the manifest and the merged spec. These are the ones that carried a design note +in core's contract before the extraction; the pre-3.0 ingest routes (`/shard/status`, `/feed`, +`/online`, `/economy`, `/houses`, `/idoc`, `/champs`, `/guilds`, `/governors` and the `/stream` SSE +feed) are described where their wire frames are, in +[`link/PLAN.md`](../../link/PLAN.md) §5 and +[`link/INTEGRATION.md`](../../link/INTEGRATION.md). + +| Method | Path | Notes | +|---|---|---| +| GET | `/shard/ruleset` | the shard's own published ruleset (Protocol 3.0 `world.ruleset`): expansion, which optional systems are on, skill/stat caps, account and house limits, champion scroll rules, the save/restart schedule. Served from `shard_ruleset`, so it renders while the shard is down; live via `world.ruleset` on `/shard/stream`. Behind `requireFeature('ruleset')`. **`null`** means the shard has never published one — a real answer, distinct from a published ruleset. `caps.skill` / `caps.totalSkill` are in **tenths** (1000 = 100.0). | +| GET | `/shard/points` | every points/loyalty leaderboard the shard publishes (Protocol 3.0 `points.board`) — Queen's Loyalty, Void Pool, the nine city loyalties, Clean Up Britannia, … Served from `shard_points_boards`, so it renders while the shard is down; live via `points.board` on `/shard/stream`. Behind `requireFeature('leaderboards')`, ordered by display name. **`maxPoints: 0` means uncapped** (the common case), and `nameString` is usually `null` with `nameNumber` holding a cliloc — resolve client-side or humanise the `system` key. | +| GET | `/shard/points/:system` | one board by the shard's `PointsType` name (e.g. `QueensLoyalty`); `:system` must match `/^[A-Za-z][A-Za-z0-9_]{0,47}$/` or **400** before any query runs. **404** = the shard has never published that system, which is distinct from a published board nobody has scored in yet (**200** with an empty `top`). | +| GET | `/shard/market?q=&minPrice=&maxPrice=&itemId=&map=®ion=&sort=&limit=&offset=` | search the player-vendor marketplace (Protocol 3.0 `vendor.listing`). Returns **listings**, not vendors — "who sells X and for how much" is the question, and a vendor-shaped result would make every caller flatten the shops back out. Served from `shard_vendors` + `shard_vendor_items`, so it renders while the shard is down. Behind `requireFeature('market')` **and rate-limited** — the first genuinely expensive public read on the site (a `LIKE` scan plus a `COUNT` over what is typically the largest `shard_*` table, reachable with no session). `sort ∈ {price_asc, price_desc, recent}`. `q` matches the resolved display name **or** the item's literal name, with `%`/`_` escaped: they are `LIKE` metacharacters, not SQL ones, so parameterization alone would let `?q=%` match every listing on the shard. Every response repeats `staleAt` (the oldest vendor row) because the shard sweeps round-robin — a banner that ages with the results it labels, not one fetched once. | +| GET | `/shard/market/meta` | index size, staleness (`staleAt`/`freshAt`) and which facets and regions actually hold vendors, so a client builds its filters without running a search it will discard. | +| GET | `/shard/market/vendors/:serial` | one shop and its listings; `:serial` must match `/^0x[0-9A-Fa-f]{1,16}$/` or **400** before any query runs. **404** = a serial the index has never seen, which also covers a vendor since dismissed or hidden — to an anonymous caller those are the same answer, and distinguishing them would leak that a hidden vendor exists. `truncated` (with `total` exceeding `count`) means the shop holds more than the shard publishes per frame. | +| GET | `/shard/features` | the shard features **this caller** may reach plus the audience rung they resolved to (§4 below), so a client hides nav it can't follow. Reports only what the caller can see — the list itself never discloses a gated feature. Consumed by the SPA header and (pending) the Android nav. | +| GET | `/atlas/creatures?q=&facet=&limit=&offset=` | the bestiary, most numerous first, with an unpaginated `total`. Static content parsed from the shard's ServUO tree — **not** sidecar-backed, which is why the atlas sits outside `/shard`, and unlike `/shard/*` it **is** site-mode gated. Behind `requireFeature('atlas')`. `?facet=` is matched exactly and never validated against a list (no facet name exists in the code); the filter is an `EXISTS` over the points rather than a JSON path or `JSON_SEARCH` built from caller input, whose `%`/`_` wildcards would make `?facet=%` match everything. | +| GET | `/atlas/creatures/:slug` | one creature: `places` (the point-in-rect aggregate — "lizardman → Shrines, Isamu-Jima, Yew"), `spawners` (the bounded raw list, with `spawnersTruncated`), `alsoHere`. **`points` is a COUNT and `spawners` is the LIST** — named apart so one key never means a number on one route and an array on another. `minDelay`/`maxDelay` are in **seconds**, normalised at parse time from the source's per-record minutes-or-seconds. 404 = no such creature in this atlas. | +| GET | `/atlas/regions?facet=&q=` | named regions and the rectangles that placed each spawner | +| GET | `/atlas/landmarks?facet=&q=` | points of interest, labelled by `group` ("Covetous", not "Level 1") | +| GET | `/atlas/champions?facet=` | the **configured** altar roster. Not `/shard/champs`, which is the live board. | +| GET | `/atlas/meta` | facets, counts and when the atlas was parsed. Game-world facts only — the ServUO path, source hashes and any pending refresh are operator detail and live on the admin route. | + +## 3. Admin routes worth their own note + +The two content imports — the spawn atlas and the cliloc table — whose behaviour is a decision +rather than a passthrough. The shard-ops routes (`kick`, `ban`, `unban`, `broadcast`, `pages`), the +account-linking routes and the sidecar config under `/admin/uo-link` are in the merged spec. + +| Method | Path | Purpose | +|---|---|---| +| GET | `/shard/atlas` | spawn-atlas status (`adminOnly`): the ServUO path, whether the tree is readable, whether it has drifted from what is loaded, counts, facets, and any refresh staged for review. The public `/atlas/meta` reports the game world only; the filesystem detail is here. | +| POST | `/shard/atlas/import` | re-import without restarting; `{force}` ignores the hash gate. **An unreadable tree answers 200 with `status:"unavailable"`, not 500** — `refresh()` reports outcomes rather than throwing (the boot path must never be blocked by a bad tree) and that contract is preserved at the API. | +| POST | `/shard/atlas/approve` · `/shard/atlas/reject` | answer a refresh staged because it would REMOVE a facet. Approving **re-parses** the tree, so what lands matches it at approval time; rejecting is remembered against those source hashes so it does not re-prompt every restart. 404 when nothing is staged. | +| PUT | `/shard/atlas/path` | point the atlas at a different tree (persisted as `spawn_atlas_servuo_path`, which wins over `SERVUO_PATH`). Blank clears it. Deliberately **does not import** — moving the mount and reloading the world are separate decisions — and returns fresh status so the panel can offer the import next. | +| GET | `/shard/clilocs` | cliloc-table status (`adminOnly`): every source found now (base first, then `custom/` overlays in merge order), what each contributed at the last import, readability, drift across the set, the entry count, and `missingSources`. `configured:false` is a supported state — item names then render as ids. No public counterpart: the table is never served *as* a table. | +| POST | `/shard/clilocs/import` | reload after a client patch or an overlay edit; `{force}` ignores the hash gate, `{approve}` accepts a **vanished** source (refused by default — see the table notes above). **A missing path — or the likely mistake of pointing at the client's own COMPRESSED `Cliloc.enu` — answers 200 with `status:"unavailable"` and a `code`, not 500.** `COMPRESSED` is called out by name: a 500 would say only "something broke", and the operator needs to be told which file to convert. | +| PUT | `/shard/clilocs/path` | point the site at a different cliloc base file or directory (persisted as `cliloc_client_path`, which wins over `UO_CLIENT_PATH`). Overlays are read from `custom/` beside it either way. Blank clears it. Deliberately **does not import**, same reasoning as the atlas path. | + +## 4. Shard visibility — the audience boundary (Protocol 3.0) + +Every shard-derived surface is gated by an **admin-configurable, per-feature and per-field** audience +setting. This **replaces** the static `PUBLIC_KINDS` allowlist that used to be the whole boundary. +Policy lives in `utils/shardVisibility.js`; rows live in `shard_feature_visibility`; the admin surface +is `GET`/`PUT /admin/shard/visibility` (`adminOnly`). Admin-facing guide: +[`SHARD_VISIBILITY.md`](../../website/SHARD_VISIBILITY.md). Design: [`link/v3.md`](../../link/v3.md) §3. + +**The ladder.** `anonymous < logged_in < player < staff < admin`, each rung implying the ones below. +`viewerLevel(req)` resolves it: no session ⇒ `anonymous`; authenticated ⇒ `logged_in`; authenticated +with a linked game account ⇒ `player`; moderator ⇒ `staff`; admin ⇒ `admin`. **Staff satisfy the +`player` rung without a linked account** (consistent with `/player/*` being role-agnostic). +**`editor` gets no shard privilege** — it is a content role, and mapping it to `staff` would silently +widen what editors see. + +**Two invariants that are code, not configuration.** Both are enforced server-side and both reject +rather than silently ignore: + +1. **`acct` and `webId` are admin-only, always.** They are not exposed as configurable fields, and a + stored row attempting to loosen them is discarded on read as well as rejected on write. A character + name is visible in game; the account behind it and the website user it links to are not. + The lock is on the field's **meaning, not one spelling**: `isLockedField(key)` matches a key that + *is* or *ends in* `acct`/`webId`, case-insensitively, so the flattened forms the read models emit + (`shapeHouse` → `ownerAcct`, `shapeGuild` → `leaderWebId`) are covered too. An exact-key check was + the original implementation and it let `GET /public/shard/idoc` serve `ownerAcct` anonymously. +2. **A kind absent from `KIND_FEATURE` is never broadcast below `admin`.** Fail closed. This is what + keeps the kind map a security boundary rather than a convenience filter, and it means a shard that + starts emitting an unknown event degrades to staff-only, never to public. + +**Fail-closed everywhere else too.** An unreadable visibility config withholds every public frame; a +DB failure falls back to the compiled defaults (pre-3.0 behavior), not to open; an unresolvable viewer +subscribes as `anonymous`. The ladder comparison uses **asymmetric** fallbacks by design — an unknown +*viewer* level floors to the bottom rung and an unknown *requirement* ceils to admin, so an +unrecognised value loses on both sides. (A single shared fallback cannot do that: whichever direction +it picks, it fails open on one side.) + +**Three enforcement points, one config:** + +| Where | Mechanism | +|---|---| +| Routes | `requireFeature(name)` — **404** when the feature is disabled (don't leak that it exists), **403** when the caller is below its audience. `projectFeature` then strips out-of-rung fields from the body. | +| SSE (`utils/shardBroadcast.js`) | Per-connection filtering. A subscriber's rung is resolved **once at subscribe time and frozen** for that connection, so a long-lived stream can't gain privilege; each frame is then mapped kind→feature, gated, and field-projected per viewer. Two subscribers can legitimately receive different versions of one event, or one of them nothing. | +| Nav | `GET /public/shard/features` returns only what the caller may reach, so the SPA never renders a link that would 403. Presentation only. | + +Config reads are cached ~5s, so admin changes take effect within seconds **including on already-open +streams**. `PUBLIC_KINDS` still exists and is still exported (`utils/shardBroadcast.js`) but is now +**derived** from the kind map rather than hand-maintained, so the two cannot drift. + +**`PUBLIC_KINDS` is a module-load constant and must not be used to answer "may this caller read this +kind?"** — it is computed from the compiled *defaults*, so it cannot see an admin's changes. Use +`visibleKinds(level, config)`, which resolves against the live config. `/feed` uses it; it originally +used `PUBLIC_KINDS` and consequently kept serving `guild.join` to anonymous callers after an admin had +moved `guilds` to `staff`. `visibleKinds` deliberately ignores the `stream` flag: that governs SSE +fan-out only, so a feature whose live firehose ships off (market) stays readable from stored history. + +**Every read path that returns shard data must call `projectFeature`.** The stored-history endpoints +are not exempt — `/feed` returns the same events the stream does, and returning them unprojected +reopens on the REST side exactly what the stream closes. Relatedly, `shardEvents.db.list` treats an +**empty** `kinds` array as "serve nothing", never "no filter"; the fall-through it used to take would +have turned a fully-gated config into a dump of the entire event log. + +`projectFeature` walks **arrays and plain objects only**. A `Date`, `Buffer` or other class instance +is passed through as a value — rebuilding one key-by-key yields `{}`, which is the difference between +the pure-JSON wire frames and the DB-backed read models whose rows carry real `Date` columns. + +**Defaults reproduce pre-3.0 behavior exactly**, so installing the framework is a no-op until an admin +changes something — with deliberate exceptions, which are the leaks it was written to close. +`/public/shard/guilds`, `/public/shard/governors` and `/public/shard/feed` previously returned the raw +stored payload, whose actors carry `acct` and `webId`; `/public/shard/idoc` returned the flattened +`ownerAcct`. All are now stripped for every caller below admin. diff --git a/modules/uo/README.md b/modules/uo/README.md index 85a70f1..9de2ec9 100644 --- a/modules/uo/README.md +++ b/modules/uo/README.md @@ -15,6 +15,8 @@ these routes *mean* are the ones that already existed and did not move: | Doc | What it covers | |---|---| +| [`API.md`](API.md) | **This module's 72 URLs** and the audience ladder that gates them — moved out of core's `BACKEND_DESIGN.md` §4/§6.5 | +| [`SCHEMA.md`](SCHEMA.md) | **The 27 tables it owns** and why they are shaped that way — moved out of core's `BACKEND_DESIGN.md` §3 | | [`SHARD_VISIBILITY.md`](../../website/SHARD_VISIBILITY.md) | Who sees which shard data — the admin-configurable audience framework | | [`SPAWN_ATLAS.md`](../../website/SPAWN_ATLAS.md) | The bestiary / spawn atlas, parsed from the shard's own ServUO tree | | [`MARKETPLACE.md`](../../website/MARKETPLACE.md) | The player-vendor index | @@ -30,7 +32,8 @@ these routes *mean* are the ones that already existed and did not move: [`routes.manifest.json`](https://gitea.whitlocktech.com/RunicGateway/Module-uo/src/branch/main/routes.manifest.json) and documented in its [`swagger-fragment.json`](https://gitea.whitlocktech.com/RunicGateway/Module-uo/src/branch/main/swagger-fragment.json), -which core merges into `/api/docs.json` while the module is running. +which core merges into `/api/docs.json` while the module is running. Route counts, gates and the +notes that used to sit in core's contract are in [`API.md`](API.md). | Mount | Tier | What | |---|---|---| @@ -52,7 +55,8 @@ into core's nav so an operator can reorder, relabel or hide them like any other ## What it owns -- **27 database tables** — 26 `shard_*` plus `uo_link_config`. Created by an idempotent +- **27 database tables** — 26 `shard_*` plus `uo_link_config`, documented one by one in + [`SCHEMA.md`](SCHEMA.md). Created by an idempotent `schema.sql` fragment core replays on every boot, after its own schema. The `shard_`/`uo_link_` prefixes are **grandfathered** ([`MODULE_API.md`](../../website/MODULE_API.md) §6.5): the rule for a new module is `_`, and these predate it. @@ -65,10 +69,35 @@ into core's nav so an operator can reorder, relabel or hide them like any other ## For an operator -**Installing.** A release is `module-uo-.tar.gz` plus a manifest carrying its `sha256`. -Unpack it as `modules/uo/` on the website's modules volume (or use the admin Modules screen when -phase 4 lands) and restart. **You never build anything** — the client chunk is prebuilt and the one -runtime dependency ships inside the tarball. +**Installing — three ways in, and none of them is a build.** A release publishes +`module-uo-.tar.gz`, an install manifest `module-uo-.json` carrying its `sha256`, +and a `SHA256SUMS`. **You never build anything**: the client chunk is prebuilt and the one runtime +dependency ships inside the tarball. + +| | How | Where it fits | +|---|---|---| +| **Admin panel** | **Admin → Modules**, paste the URL of the release's `module-uo-.json`, then press Restart when it asks | The click path — no shell on the box. Core downloads the artifact the manifest names, verifies the declared `sha256`, inspects the whole archive before writing anything, and unpacks it as `modules/uo/` | +| **`MODULES`** | Declare it in the environment and the container resolves it at every start:
`MODULES=uo@0.3.0=https://…/module-uo-0.3.0.json` | The compose-managed host. The running set is a line in a file you version-control. Already at that version ⇒ no network at all, so a restart with the internet down comes up unchanged | +| **By hand** | `tar -xf module-uo-0.3.0.tar.gz -C ./modules && mv modules/module-uo-0.3.0 modules/uo`, then restart | Development, and any host where the other two do not fit. The bundle's top-level directory is named after the release, not after the module id — rename it to `uo` | + +The install source must be an `https` host on the allowlist (seeded from `MODULE_SOURCE_HOSTS`, +editable in the panel from then on). The `sha256` in the manifest is the trust anchor; the allowlist +is what stops a pasted URL from being an SSRF primitive as well. + +**Uninstalling, and the one destructive choice.** Uninstall removes the module's directory and +leaves its row `disabled` — **your data is kept**, and reinstalling picks it up exactly where it was. +Deleting the data is a separate, opt-in tick box *inside* the uninstall dialog, and it has to be +there rather than after: `purge.sql` is a file inside the directory being deleted. There is also a +standalone **Purge** action on a module that is still installed but disabled. Purging drops all 27 +tables; it does **not** touch the two `settings` rows (renaming or deleting +`uo_link_protocol_3_migrated` would re-arm a protocol migration against tables that no longer +exist), and what the module can re-derive from your ServUO tree — the atlas and the cliloc table — is +rebuilt at the next boot. Everything the shard and your players produced is gone. + +**Disabling is a kill switch, not a visibility flag.** Disable runs the module's `onShutdown` +immediately: the uo-link WebSocket closes, the SSE streams end, and its routes, nav rows and client +chunk answer 404. Re-enabling flips the row and asks for a restart, because there is no `onBoot` +re-dispatch — the hooks have never been promised to be re-entrant. **Connecting it to a shard.** The module needs the [uo-link sidecar](https://gitea.whitlocktech.com/RunicGateway/link) running next to the ServUO diff --git a/modules/uo/SCHEMA.md b/modules/uo/SCHEMA.md new file mode 100644 index 0000000..2fa85ba --- /dev/null +++ b/modules/uo/SCHEMA.md @@ -0,0 +1,253 @@ +# module-uo — the tables it owns + +The 27 tables `module-uo` creates and owns, and the reasoning behind their shapes. **26 `shard_*` +plus `uo_link_config`**, all created by the idempotent `server/db/schema.sql` fragment core replays +after its own schema on every boot, and all dropped by `server/db/purge.sql` when an operator purges +the module's data. + +This page moved out of [`BACKEND_DESIGN.md`](../../website/BACKEND_DESIGN.md) §3 when Phase 4 closed +([`MODULE_SYSTEM.md`](../../website/MODULE_SYSTEM.md) §2.7.2): core's schema reference describes +core's tables, and these are not core's. The text is unchanged — where it says "the shard", it means +the Ultima Online shard this module fronts. Core's own module bookkeeping table, +`installed_modules`, stays in `BACKEND_DESIGN.md` where it belongs. + +The `shard_` and `uo_link_` prefixes are **grandfathered** +([`MODULE_API.md`](../../website/MODULE_API.md) §6.5). A new module's tables are prefixed with its +own id; these predate the rule, and they are read by the shipped Android app, so renaming them would +be a data migration plus a client break. + +**Not every table has a section here.** The pre-3.0 ingest tables (`shard_events`, `shard_online`, +`shard_economy`, `shard_houses`, `shard_champs`, `shard_guilds`, `shard_governors`, +`shard_governor_terms`, `shard_presence`, `shard_pages`, `shard_account_links`, `uo_link_config`) +are described where their wire frames are — [`link/PLAN.md`](../../link/PLAN.md) §5 and +[`link/INTEGRATION.md`](../../link/INTEGRATION.md). What follows is everything that carried a +design note worth keeping. + +--- + +## shard_ruleset — the shard's published ruleset (Protocol 3.0) + +Singleton row (`id = 1`, CHECK-constrained) holding the latest `world.ruleset` frame: `rev`, +`expansion`, `payload` JSON (the whole frame), `t`, `updated_at`. The shard re-emits the complete +ruleset on every sidecar connect, so this is an **overwrite, not an append** — and the kind is +deliberately **not** in `LOGGED_KINDS`, since logging it would put a duplicate row in `shard_events` +on every reconnect while `server.hello` already marks each of those. + +The frame is stored whole rather than normalized into columns: it is a flat description of server +config that is read as one page, so splitting it up would mean a schema change every time the shard +grows a new block. `rev` (the shard's FNV-1a of the body) and `expansion` are hoisted only because +they are cheap to display — the same payload-plus-hoisted-columns shape `shard_champs` uses. + +**No row means the shard has never published one** (an older plugin, or `Bridge.RulesetEnabled=false`), +served as `null` rather than `{}`: "not published yet" and "published, everything off" are different +answers and the page renders them differently. + +## shard_points_boards — points / loyalty leaderboards (Protocol 3.0) + +One row per point system, keyed by the shard's own `PointsType` name (`QueensLoyalty`, +`CleanUpBritannia`, …). The shard carries ~25 of these, each a standing players build over months. +Columns: `system` (PK), `name`, `name_cliloc`, `max_points`, `players`, `show_on_gump`, `payload` JSON +(the whole `points.board` frame), `t`, `updated_at`. + +**The top-N list stays inside `payload`** rather than being normalized into a `shard_points_entries` +table. It is a fixed-size list (10 by default) that is only ever read whole — exactly like +`shard_governors.candidates` — so normalizing buys nothing until something needs a per-character +reverse lookup, and a character's own standings already ride inside `char.profile` instead. + +Board state, not events: `points.board` is **not** in `LOGGED_KINDS`, for the same reason +`guild.update` isn't. The shard emits a frame every time anyone's score moves a top ten, so logging +would grow `shard_events` without bound for something whose only interesting value is its latest +version. There is also **no delete path** — the shard's set of systems is fixed at startup, so there is +no `points.remove` to mirror. + +Two values carry non-obvious meanings, both set by the plugin and both documented in +[`link/INTEGRATION.md`](../../link/INTEGRATION.md) §4: + +- **`max_points = 0` means uncapped**, and on a real shard that is the *common* case (ServUO's + uncapped idiom is `double.MaxValue`, which the plugin normalises to 0). Anything rendering + `points / max_points` must special-case it. +- **`name` is usually NULL**, with `name_cliloc` set instead — most systems name themselves with a + cliloc rather than a literal. Listing therefore orders by `COALESCE(name, system)`, so boards + awaiting cliloc resolution sort by their own key rather than clumping together under NULL. + +## shard_vendors / shard_vendor_items — the player-vendor marketplace (Protocol 3.0) + +The shard-wide shop index, fed by `vendor.listing` / `vendor.listing.remove`. One row per player +vendor and one per priced listing. Full operator detail in [`MARKETPLACE.md`](../../website/MARKETPLACE.md); the +design is `docs/link/v3.md` §8. + +| Table | Shape | +|---|---| +| `shard_vendors` | `serial` (PK), `shop_name`, `owner_serial`, `owner_name`, `map`/`x`/`y`/`z`, `region`, `house`, `item_count`, `item_total`, `truncated`, `t`, `updated_at`. Indexes on owner, map, region and `updated_at`. | +| `shard_vendor_items` | `id` (PK), `vendor_serial`, `serial`, `item_id`, `hue`, `amount`, `price`, `name`, `cliloc`, `display_name`, `child`. Indexes on `vendor_serial`, `price`, `item_id`, `display_name`, and `(display_name, price)`. | + +**Ingest is per-vendor and authoritative**: the frame is the whole shop, so ingest is +delete-then-insert of that vendor's listings inside one transaction. All-or-nothing matters +specifically because the two writes are "the shop" and "what is in it" — a failure between them +leaves a shop advertising an inventory it no longer has, which is visibly wrong and indistinguishable +from a genuinely empty shop. No foreign keys, consistent with every other `shard_*` table. + +**There is deliberately no `payload` column**, unlike `shard_points_boards` directly above. The +board's top-N is a fixed-size list read whole, so it lives in JSON; here the items *are* the +searchable rows, so they are normalized and nothing is left worth duplicating. The sidecar keeps the +whole blob — outage resilience is its job, search is ours. + +Market state, not events: neither kind is in `LOGGED_KINDS`, and this is the strongest case of the +three v3 kinds. One frame carries up to 250 listings and the sweep re-emits a shop on any price +change, so logging would turn `shard_events` into a price history nobody reads. + +Two columns carry non-obvious meanings: + +- **`item_count` vs `item_total`.** `item_count` is what the frame published; `item_total` is what + the shop actually holds. They differ when `truncated` — the shard caps listings per frame + (`Bridge.MarketMaxListings`, 250 by default), and a commodity reseller with thousands of stacks + genuinely exceeds it. Any UI must show both or it presents a partial shop as complete. +- **`display_name` is denormalized at ingest**, resolved from the item's literal `name` (preferred — + a player set it, so it is more specific) else its `cliloc` against `shard_clilocs`. Resolving at + query time would put the cliloc table on the hot path and make search-by-name impossible. Because + the shard's diff sweep will not re-send an unchanged shop just because the site learned what its + items are called, **a cliloc import triggers a bulk re-resolution** of this column (after a boot + import and after an admin import; ~50 ms per thousand rows, never throws). + +`updated_at` is written explicitly on every upsert rather than left to `ON UPDATE CURRENT_TIMESTAMP`, +which MariaDB does not fire when every column is written back unchanged. A shop re-published +identically is still *freshly confirmed*, and without this the staleness banner would age a perfectly +current shop forever. + +## shard_feature_visibility — per-feature audience config (Protocol 3.0) + +One row per shard feature: `feature` (PK), `enabled`, `audience` (a rung on the ladder in [`API.md`](API.md)), +`stream` (whether the feature's kinds fan out over SSE at all), `field_rules` JSON (`{field: rung}` +for the sensitive fields only), `updated_by`, `updated_at`. + +**An absent row means "use the compiled default", and the compiled defaults reproduce pre-3.0 +behavior — so an empty table is a no-op and there is nothing to seed.** Stored rows are merged over +the defaults on read, which is also where the invariants are re-applied: a row naming an unknown +feature is ignored (a stale row must not resurrect a removed feature), an invalid rung falls back to +the default rather than failing open, and a rule touching a locked field (`acct` / `webId`) is +discarded. See [`API.md`](API.md). + +## shard_spawn_* / shard_regions / shard_landmarks / shard_champion_spawns / shard_atlas_meta — the spawn atlas (Protocol 3.0) + +Static shard **content**, not live shard state. Nothing here comes from the sidecar: the atlas is +derived from the shard's own ServUO tree, re-read on **every server boot** and hash-gated so an +unchanged tree costs one read pass and no write. Nothing is precomputed and committed — a shard's +maps change over its life, and a snapshot in the repo would silently drift from the world players +actually see. These tables stay populated whether the shard is up or not. Full operator detail in +[`SPAWN_ATLAS.md`](../../website/SPAWN_ATLAS.md); the design is `docs/link/v3.md` §6. + +**No facet name appears anywhere in the code.** A shard may add facets, replace them, or rename them +when its maps are updated; the facet set is discovered from the tree, and the loose spellings in +`Data/Locations` are matched against it rather than looked up in a table. + +| Table | Key columns | +|---|---| +| `shard_spawn_creatures` | `slug` PK, `name`, `total`, `points`, `facets` JSON, `art` NULL | +| `shard_spawn_points` | `id` PK, `facet`, `name`, `x`, `y`, `width`, `height`, `spawn_range`, `max_count`, `min_delay`, `max_delay`, `tod_start/end/mode`, `region`, `landmark`, `label` | +| `shard_spawn_point_types` | `(point_id, slug)` PK, `max_count` | +| `shard_regions` | `facet`, `name`, `type`, `priority`, `parent`, `rects` JSON | +| `shard_landmarks` | `facet`, `name`, `grp`, `x`, `y`, `z` | +| `shard_champion_spawns` | `slug` PK, `name`, `grp`, `type`, `random_type`, `facet`, `x`, `y`, `z`, `radius`, `label` | +| `shard_atlas_meta` | Singleton (`id = 1`), `payload` JSON (counts, a sha256 per source file, `parserVersion`), `imported_at` | +| `shard_atlas_pending` | Singleton (`id = 1`), `status` (`pending`/`rejected`), `payload` JSON, `detected_at` | + +The first seven are **import-owned**: a refresh empties and reloads every one inside a single +transaction, so a failed reload leaves the previous atlas intact rather than a half-loaded world. +Nothing else writes to them, and nothing holds a foreign key to them — no FKs at all, consistent with +every other `shard_*` table. + +**`shard_atlas_pending` is the security-relevant one.** A refresh that would REMOVE a facet is never +applied automatically: facet loss is indistinguishable at boot from a half-copied or mid-update tree, +so it is staged here for an admin to approve or reject, and **startup is never blocked by it**. Only +the decision is stored — source hashes plus the facet diff, a few KB — and approving re-parses the +tree, so a multi-megabyte blob never lands in the database and what gets applied matches the tree at +approval time. A rejection is remembered against those exact hashes so a declined refresh does not +re-prompt on every restart. Everything else (new facets, renamed regions, changed spawns) applies +immediately, since none of it can destroy data an operator would miss. + +The boot refresh is **best-effort by contract**: no configured path, an unreadable mount, a malformed +file or a database error is caught and logged, and the site comes up serving whatever atlas it had. +The tree path comes from the `spawn_atlas_servuo_path` setting, falling back to `SERVUO_PATH`. + +**A refresh re-derives when the tree changed OR the parser did.** `spawnAtlasSource.PARSER_VERSION` +is stored in `shard_atlas_meta` beside the source hashes and bumped whenever the parser produces +different data from identical files. Hashing the tree alone would strand an install whose maps never +change on whatever an older build derived — a corrected parse would ship and never reach the data. + +Four column choices worth stating, because each one is a trap: + +- **`spawn_range`, not `range`**, and **`grp`, not `group`** — both are reserved words. +- **`DELETE`, not `TRUNCATE`.** `TRUNCATE` is DDL in MariaDB and implicitly commits, which would + defeat the all-or-nothing reload. At ~7k rows the difference does not matter. +- **Point ids are assigned explicitly**, not left to `AUTO_INCREMENT`: the `shard_spawn_point_types` + rows need to know them, and `conn.batch()` reports no usable `insertId` for a multi-row insert. +- **Plain `INDEX` on `name`, deliberately not `FULLTEXT`.** ~800 creature rows makes a `LIKE` scan + free, and FULLTEXT's minimum token length would break searches for names like "orc". + +`shard_champion_spawns` is the *configured* altar roster ("there is an Unholy Terror altar in +Deceit"). The live `champ.update` feed in `shard_champs` is the separate answer to "it is on level 3 +right now". Both exist; they are not the same data. + +**`shard_spawn_creatures.art` is always NULL on a fresh import.** The project ships no creature +artwork: sprites live in the operator's own client `.mul`/`.uop` files and are theirs, not ours to +redistribute. An operator supplies art via a gitignored map plus images under the (already +gitignored) `server/uploads/atlas/`. Text-only is the normal, supported state. + +## shard_clilocs / shard_cliloc_meta — UO's localization table (Protocol 3.0) + +Items on the wire carry a `LabelNumber`, not a name. The bridge has always sent it — +`char.profile.equipment.cliloc`, reward titles as a cliloc number in string form, and one per +marketplace listing — but with no table to resolve it against, the character sheet could only render +`id 1023721` where the game renders "quarter staff". + +| Table | Shape | +|---|---| +| `shard_clilocs` | `number` INT PK, `flag`, `text` TEXT | +| `shard_cliloc_meta` | Singleton (`id = 1`), `payload` JSON (source file, sha256, count, `parserVersion`), `imported_at` | + +Import-owned and all-or-nothing in one transaction, same contract as the atlas — including **`DELETE`, +not `TRUNCATE`**, for the same reason. + +**Sourced from files the operator supplies**, at a path from the `cliloc_client_path` setting falling +back to `UO_CLIENT_PATH`. Nothing client-derived is committed: UO's strings are EA's, exactly as the +creature sprites are. A shard with nothing configured is fully supported — names render as ids. Full +design and operator guide: [`CLILOCS.md`](../../website/CLILOCS.md). + +**It reads a SET of sources, not one file**, because shards edit items and add new ones and those +carry cliloc ids no stock client table has. A base (the converted client table) plus every overlay +under `custom/` are re-read on every boot and hash-gated **together**, exactly as the atlas re-reads +`Regions.xml` + `Locations/*.xml` + `Spawns/*.xml` + `ChampionSpawns.xml`. Later sources win, so an +overlay both adds ids and overrides stock ones, and adding one custom item never means re-exporting a +5 MB client file. Scale, measured on the live shard: its script tree references 16,434 cliloc ids and +only 37 are absent from stock — tens of entries against a 67k base, which is why this is an overlay +and not a second table. + +The conversion step is not avoidable: **every current client ships its cliloc files compressed** +(first DWORD's high byte `0x8E`), and ServUO's own bundled `Ultima.StringList` cannot read that +either — so the shard cannot supply names on our behalf. The plain layout and a delimited text export +are both accepted, sniffed by header rather than extension. + +Three decisions worth stating: + +- **`text` is TEXT, not VARCHAR.** Long property descriptions reach 12 KB. The index that matters for + marketplace search is the denormalized `shard_vendor_items.display_name`, not this table. +- **Blank entries are dropped at import** — 123,490 parsed → **67,496** stored. Roughly half a cliloc + table is empty strings for ids the client reserves and never uses; a row that resolves to no name is + indistinguishable from no row at all, and dropping them makes the binary and text imports converge + on identical content. +- **Two refusals, one of them the atlas's.** A corrupt source fails the parse on a truncated record, + so it is caught outright and leaves the previous table serving. But a source that has **vanished** + parses perfectly and imports a table quietly missing everything it contributed — an unmounted volume + and a deliberate deletion are indistinguishable from here, which is precisely the ambiguity the + atlas stages a facet removal for. So it is escalated: `status: 'needsReview'`, nothing applied, + `missingSources` reported by both the import and `status()`, and an admin accepts it with + `{approve:true}`. That is a flag rather than the atlas's approve/reject pair because the atlas + stores a pending decision so that approving **re-parses** the tree; here nothing is stored, so + re-reading at approval time is automatic. + +**Resolution is server-side and there is no public route.** The table is never served *as* a table: +67k rows would dwarf any page using them, and the Android client consumes the same already-resolved +JSON. `resolveMany()` returns only ids that resolved to something displayable — placeholders like +`~1_val~` are stripped, since the bridge sends the id and never the property packet that carries the +arguments — and it never throws, because a cliloc lookup is decoration on a character sheet. + diff --git a/website/BACKEND_DESIGN.md b/website/BACKEND_DESIGN.md index f9d341f..ed1b5b9 100644 --- a/website/BACKEND_DESIGN.md +++ b/website/BACKEND_DESIGN.md @@ -100,12 +100,6 @@ server/ pages.router.js (2) /public/pages — the draft-preview route precedes /:slug and is deliberately not site-mode gated - shard.router.js (14) /public/shard/* incl. the anonymous - SSE stream; never site-mode gated - atlas.router.js (6) /public/atlas/* — the spawn atlas. - NOT under /shard: nothing here - touches the sidecar, and unlike - /shard it IS site-mode gated modules.router.js (1) /public/modules — the installed-module list a client feature-detects against. A real prefix and not a fifth singleton @@ -115,17 +109,18 @@ server/ site.router.js (4) /settings /status /version /contact — the group-root singletons; declares no router-level middleware - public.controller.js + shard.controller.js + public.controller.js + (/public/shard and /public/atlas are module-uo's — see + ../modules/uo/API.md) player/ index.js owns the shared `noindex, requireAuth` gate (authenticated, ANY role — staff are a superset of players) and the mount table account.router.js (8) /player/account — credentials, TOTP, linked identities; handlers shared with /admin/account and /auth/me - shard.router.js (8) /player/shard — linking + own roster, - vendors, chars, sales, houses appeals.router.js (4) /player/appeals - shard.controller.js + appeals.controller.js + appeals.controller.js + (/player/shard is module-uo's) settings/ index.js owns the shared `noindex, requireAuth` gate (authenticated, ANY role) and the mount table. A fifth group, for site-wide settings that @@ -145,7 +140,10 @@ server/ `noindex, isLoggedIn, staffOnly` gate and declares no routes itself account.router.js (6) /admin/account — self-service, no adminOnly - users.router.js (15) /admin/users — adminOnly + users.router.js (9) /admin/users — adminOnly. The six + /users/:id/shard/* routes are a + MODULE's, reached through the + admin.users.detail extension slot invites.router.js (3) /admin/invites — adminOnly authProviders.router.js (4) /admin/auth — adminOnly moderation.router.js (15) /admin/moderation — modAccess @@ -162,13 +160,11 @@ server/ pages.router.js (7) /admin/pages — CMS page builder imageUpload.js shared multer config for the two upload routes above (not a router) - shard.router.js (16) /admin/shard — 7 self-service - account-linking routes (no extra - gate, handlers shared with - /player/shard) + 9 in-game staff - ops on modAccess, per route - uoLink.router.js (5) /admin/uo-link — sidecar config, - town crier, admin SSE — adminOnly + modules.router.js (8) /admin/modules — adminOnly, the + module delivery surface: install + from a manifest URL, enable, + disable, uninstall, purge, restart + and the source allowlist email.router.js (6) /admin/email — Gmail OAuth2 delivery — adminOnly discordBot.router.js (2) /admin/discord-bot — adminOnly @@ -177,7 +173,7 @@ server/ and carries its own key allowlist (theming/nav keys + the hero draft) so it can never drop site_mode or - the uo-link config; POST + a module's own seeded row; POST /brand-asset/:slot uploads a logo/hero/favicon and writes the brand_assets row in the same call @@ -191,6 +187,7 @@ server/ admin.controller.js + the per-capability controllers (already domain-split; the split PRs re-wire routes, not logic) + (/admin/shard and /admin/uo-link are module-uo's) model/ users/ users.model.js + users.db.js posts/ posts.model.js + posts.db.js (news/five-on-friday/newsletter/screenshots) @@ -326,8 +323,10 @@ base `NAV` arrays are client constants, and duplicating them server-side would create a second source of truth for navigation that drifts the first time a route is added. `client/src/lib/navOverrides.js` drops an unknown `to` at merge time instead, which is also what makes deleting a route in code safe. The merge runs -*before* the role and shard-feature filters in `SiteHeader.jsx` / -`AdminLayout.jsx`, which are unchanged and remain the boundary — a stored +*before* the role and feature filters in `SiteHeader.jsx` / `AdminLayout.jsx` — +a `feature` on a nav row is resolved by the module that **registered** the row +(`client/src/modules/featureGate.js`), so no flag string carries a parsed prefix +and core learns nothing about a game — and those filters remain the boundary: a stored `hidden: false` on a gated item shows nobody anything. `hidden: false` is accepted (the editor sends it mid-edit) but never stored, so hiding stays subtractive. `hidden` on `/admin/navigation` is dropped for `nav_admin`, because @@ -471,231 +470,15 @@ analogue to a password — and there is no hash-lookup constraint (verification unused rows and `bcrypt.compare`s each, like password verification). `used_at` is the single-use marker. Cleared wholesale on TOTP disable / password change / password reset. -### shard_ruleset — the shard's published ruleset (Protocol 3.0) +### The 27 shard tables — module-owned (module system) -Singleton row (`id = 1`, CHECK-constrained) holding the latest `world.ruleset` frame: `rev`, -`expansion`, `payload` JSON (the whole frame), `t`, `updated_at`. The shard re-emits the complete -ruleset on every sidecar connect, so this is an **overwrite, not an append** — and the kind is -deliberately **not** in `LOGGED_KINDS`, since logging it would put a duplicate row in `shard_events` -on every reconnect while `server.hello` already marks each of those. +`shard_*` and `uo_link_config` are **not core's**. They are created and dropped by `module-uo`'s own +schema fragment, and a core running without that module has none of them. Their shapes and the +reasoning behind them live with the module: +[`../modules/uo/SCHEMA.md`](../modules/uo/SCHEMA.md). -The frame is stored whole rather than normalized into columns: it is a flat description of server -config that is read as one page, so splitting it up would mean a schema change every time the shard -grows a new block. `rev` (the shard's FNV-1a of the body) and `expansion` are hoisted only because -they are cheap to display — the same payload-plus-hoisted-columns shape `shard_champs` uses. - -**No row means the shard has never published one** (an older plugin, or `Bridge.RulesetEnabled=false`), -served as `null` rather than `{}`: "not published yet" and "published, everything off" are different -answers and the page renders them differently. - -### shard_points_boards — points / loyalty leaderboards (Protocol 3.0) - -One row per point system, keyed by the shard's own `PointsType` name (`QueensLoyalty`, -`CleanUpBritannia`, …). The shard carries ~25 of these, each a standing players build over months. -Columns: `system` (PK), `name`, `name_cliloc`, `max_points`, `players`, `show_on_gump`, `payload` JSON -(the whole `points.board` frame), `t`, `updated_at`. - -**The top-N list stays inside `payload`** rather than being normalized into a `shard_points_entries` -table. It is a fixed-size list (10 by default) that is only ever read whole — exactly like -`shard_governors.candidates` — so normalizing buys nothing until something needs a per-character -reverse lookup, and a character's own standings already ride inside `char.profile` instead. - -Board state, not events: `points.board` is **not** in `LOGGED_KINDS`, for the same reason -`guild.update` isn't. The shard emits a frame every time anyone's score moves a top ten, so logging -would grow `shard_events` without bound for something whose only interesting value is its latest -version. There is also **no delete path** — the shard's set of systems is fixed at startup, so there is -no `points.remove` to mirror. - -Two values carry non-obvious meanings, both set by the plugin and both documented in -[`link/INTEGRATION.md`](../link/INTEGRATION.md) §4: - -- **`max_points = 0` means uncapped**, and on a real shard that is the *common* case (ServUO's - uncapped idiom is `double.MaxValue`, which the plugin normalises to 0). Anything rendering - `points / max_points` must special-case it. -- **`name` is usually NULL**, with `name_cliloc` set instead — most systems name themselves with a - cliloc rather than a literal. Listing therefore orders by `COALESCE(name, system)`, so boards - awaiting cliloc resolution sort by their own key rather than clumping together under NULL. - -### shard_vendors / shard_vendor_items — the player-vendor marketplace (Protocol 3.0) - -The shard-wide shop index, fed by `vendor.listing` / `vendor.listing.remove`. One row per player -vendor and one per priced listing. Full operator detail in [`MARKETPLACE.md`](MARKETPLACE.md); the -design is `docs/link/v3.md` §8. - -| Table | Shape | -|---|---| -| `shard_vendors` | `serial` (PK), `shop_name`, `owner_serial`, `owner_name`, `map`/`x`/`y`/`z`, `region`, `house`, `item_count`, `item_total`, `truncated`, `t`, `updated_at`. Indexes on owner, map, region and `updated_at`. | -| `shard_vendor_items` | `id` (PK), `vendor_serial`, `serial`, `item_id`, `hue`, `amount`, `price`, `name`, `cliloc`, `display_name`, `child`. Indexes on `vendor_serial`, `price`, `item_id`, `display_name`, and `(display_name, price)`. | - -**Ingest is per-vendor and authoritative**: the frame is the whole shop, so ingest is -delete-then-insert of that vendor's listings inside one transaction. All-or-nothing matters -specifically because the two writes are "the shop" and "what is in it" — a failure between them -leaves a shop advertising an inventory it no longer has, which is visibly wrong and indistinguishable -from a genuinely empty shop. No foreign keys, consistent with every other `shard_*` table. - -**There is deliberately no `payload` column**, unlike `shard_points_boards` directly above. The -board's top-N is a fixed-size list read whole, so it lives in JSON; here the items *are* the -searchable rows, so they are normalized and nothing is left worth duplicating. The sidecar keeps the -whole blob — outage resilience is its job, search is ours. - -Market state, not events: neither kind is in `LOGGED_KINDS`, and this is the strongest case of the -three v3 kinds. One frame carries up to 250 listings and the sweep re-emits a shop on any price -change, so logging would turn `shard_events` into a price history nobody reads. - -Two columns carry non-obvious meanings: - -- **`item_count` vs `item_total`.** `item_count` is what the frame published; `item_total` is what - the shop actually holds. They differ when `truncated` — the shard caps listings per frame - (`Bridge.MarketMaxListings`, 250 by default), and a commodity reseller with thousands of stacks - genuinely exceeds it. Any UI must show both or it presents a partial shop as complete. -- **`display_name` is denormalized at ingest**, resolved from the item's literal `name` (preferred — - a player set it, so it is more specific) else its `cliloc` against `shard_clilocs`. Resolving at - query time would put the cliloc table on the hot path and make search-by-name impossible. Because - the shard's diff sweep will not re-send an unchanged shop just because the site learned what its - items are called, **a cliloc import triggers a bulk re-resolution** of this column (after a boot - import and after an admin import; ~50 ms per thousand rows, never throws). - -`updated_at` is written explicitly on every upsert rather than left to `ON UPDATE CURRENT_TIMESTAMP`, -which MariaDB does not fire when every column is written back unchanged. A shop re-published -identically is still *freshly confirmed*, and without this the staleness banner would age a perfectly -current shop forever. - -### shard_feature_visibility — per-feature audience config (Protocol 3.0) - -One row per shard feature: `feature` (PK), `enabled`, `audience` (a rung on the ladder in §6.5), -`stream` (whether the feature's kinds fan out over SSE at all), `field_rules` JSON (`{field: rung}` -for the sensitive fields only), `updated_by`, `updated_at`. - -**An absent row means "use the compiled default", and the compiled defaults reproduce pre-3.0 -behavior — so an empty table is a no-op and there is nothing to seed.** Stored rows are merged over -the defaults on read, which is also where the invariants are re-applied: a row naming an unknown -feature is ignored (a stale row must not resurrect a removed feature), an invalid rung falls back to -the default rather than failing open, and a rule touching a locked field (`acct` / `webId`) is -discarded. See §6.5. - -### shard_spawn_* / shard_regions / shard_landmarks / shard_champion_spawns / shard_atlas_meta — the spawn atlas (Protocol 3.0) - -Static shard **content**, not live shard state. Nothing here comes from the sidecar: the atlas is -derived from the shard's own ServUO tree, re-read on **every server boot** and hash-gated so an -unchanged tree costs one read pass and no write. Nothing is precomputed and committed — a shard's -maps change over its life, and a snapshot in the repo would silently drift from the world players -actually see. These tables stay populated whether the shard is up or not. Full operator detail in -[`SPAWN_ATLAS.md`](SPAWN_ATLAS.md); the design is `docs/link/v3.md` §6. - -**No facet name appears anywhere in the code.** A shard may add facets, replace them, or rename them -when its maps are updated; the facet set is discovered from the tree, and the loose spellings in -`Data/Locations` are matched against it rather than looked up in a table. - -| Table | Key columns | -|---|---| -| `shard_spawn_creatures` | `slug` PK, `name`, `total`, `points`, `facets` JSON, `art` NULL | -| `shard_spawn_points` | `id` PK, `facet`, `name`, `x`, `y`, `width`, `height`, `spawn_range`, `max_count`, `min_delay`, `max_delay`, `tod_start/end/mode`, `region`, `landmark`, `label` | -| `shard_spawn_point_types` | `(point_id, slug)` PK, `max_count` | -| `shard_regions` | `facet`, `name`, `type`, `priority`, `parent`, `rects` JSON | -| `shard_landmarks` | `facet`, `name`, `grp`, `x`, `y`, `z` | -| `shard_champion_spawns` | `slug` PK, `name`, `grp`, `type`, `random_type`, `facet`, `x`, `y`, `z`, `radius`, `label` | -| `shard_atlas_meta` | Singleton (`id = 1`), `payload` JSON (counts, a sha256 per source file, `parserVersion`), `imported_at` | -| `shard_atlas_pending` | Singleton (`id = 1`), `status` (`pending`/`rejected`), `payload` JSON, `detected_at` | - -The first seven are **import-owned**: a refresh empties and reloads every one inside a single -transaction, so a failed reload leaves the previous atlas intact rather than a half-loaded world. -Nothing else writes to them, and nothing holds a foreign key to them — no FKs at all, consistent with -every other `shard_*` table. - -**`shard_atlas_pending` is the security-relevant one.** A refresh that would REMOVE a facet is never -applied automatically: facet loss is indistinguishable at boot from a half-copied or mid-update tree, -so it is staged here for an admin to approve or reject, and **startup is never blocked by it**. Only -the decision is stored — source hashes plus the facet diff, a few KB — and approving re-parses the -tree, so a multi-megabyte blob never lands in the database and what gets applied matches the tree at -approval time. A rejection is remembered against those exact hashes so a declined refresh does not -re-prompt on every restart. Everything else (new facets, renamed regions, changed spawns) applies -immediately, since none of it can destroy data an operator would miss. - -The boot refresh is **best-effort by contract**: no configured path, an unreadable mount, a malformed -file or a database error is caught and logged, and the site comes up serving whatever atlas it had. -The tree path comes from the `spawn_atlas_servuo_path` setting, falling back to `SERVUO_PATH`. - -**A refresh re-derives when the tree changed OR the parser did.** `spawnAtlasSource.PARSER_VERSION` -is stored in `shard_atlas_meta` beside the source hashes and bumped whenever the parser produces -different data from identical files. Hashing the tree alone would strand an install whose maps never -change on whatever an older build derived — a corrected parse would ship and never reach the data. - -Four column choices worth stating, because each one is a trap: - -- **`spawn_range`, not `range`**, and **`grp`, not `group`** — both are reserved words. -- **`DELETE`, not `TRUNCATE`.** `TRUNCATE` is DDL in MariaDB and implicitly commits, which would - defeat the all-or-nothing reload. At ~7k rows the difference does not matter. -- **Point ids are assigned explicitly**, not left to `AUTO_INCREMENT`: the `shard_spawn_point_types` - rows need to know them, and `conn.batch()` reports no usable `insertId` for a multi-row insert. -- **Plain `INDEX` on `name`, deliberately not `FULLTEXT`.** ~800 creature rows makes a `LIKE` scan - free, and FULLTEXT's minimum token length would break searches for names like "orc". - -`shard_champion_spawns` is the *configured* altar roster ("there is an Unholy Terror altar in -Deceit"). The live `champ.update` feed in `shard_champs` is the separate answer to "it is on level 3 -right now". Both exist; they are not the same data. - -**`shard_spawn_creatures.art` is always NULL on a fresh import.** The project ships no creature -artwork: sprites live in the operator's own client `.mul`/`.uop` files and are theirs, not ours to -redistribute. An operator supplies art via a gitignored map plus images under the (already -gitignored) `server/uploads/atlas/`. Text-only is the normal, supported state. - -### shard_clilocs / shard_cliloc_meta — UO's localization table (Protocol 3.0) - -Items on the wire carry a `LabelNumber`, not a name. The bridge has always sent it — -`char.profile.equipment.cliloc`, reward titles as a cliloc number in string form, and one per -marketplace listing — but with no table to resolve it against, the character sheet could only render -`id 1023721` where the game renders "quarter staff". - -| Table | Shape | -|---|---| -| `shard_clilocs` | `number` INT PK, `flag`, `text` TEXT | -| `shard_cliloc_meta` | Singleton (`id = 1`), `payload` JSON (source file, sha256, count, `parserVersion`), `imported_at` | - -Import-owned and all-or-nothing in one transaction, same contract as the atlas — including **`DELETE`, -not `TRUNCATE`**, for the same reason. - -**Sourced from files the operator supplies**, at a path from the `cliloc_client_path` setting falling -back to `UO_CLIENT_PATH`. Nothing client-derived is committed: UO's strings are EA's, exactly as the -creature sprites are. A shard with nothing configured is fully supported — names render as ids. Full -design and operator guide: [`CLILOCS.md`](CLILOCS.md). - -**It reads a SET of sources, not one file**, because shards edit items and add new ones and those -carry cliloc ids no stock client table has. A base (the converted client table) plus every overlay -under `custom/` are re-read on every boot and hash-gated **together**, exactly as the atlas re-reads -`Regions.xml` + `Locations/*.xml` + `Spawns/*.xml` + `ChampionSpawns.xml`. Later sources win, so an -overlay both adds ids and overrides stock ones, and adding one custom item never means re-exporting a -5 MB client file. Scale, measured on the live shard: its script tree references 16,434 cliloc ids and -only 37 are absent from stock — tens of entries against a 67k base, which is why this is an overlay -and not a second table. - -The conversion step is not avoidable: **every current client ships its cliloc files compressed** -(first DWORD's high byte `0x8E`), and ServUO's own bundled `Ultima.StringList` cannot read that -either — so the shard cannot supply names on our behalf. The plain layout and a delimited text export -are both accepted, sniffed by header rather than extension. - -Three decisions worth stating: - -- **`text` is TEXT, not VARCHAR.** Long property descriptions reach 12 KB. The index that matters for - marketplace search is the denormalized `shard_vendor_items.display_name`, not this table. -- **Blank entries are dropped at import** — 123,490 parsed → **67,496** stored. Roughly half a cliloc - table is empty strings for ids the client reserves and never uses; a row that resolves to no name is - indistinguishable from no row at all, and dropping them makes the binary and text imports converge - on identical content. -- **Two refusals, one of them the atlas's.** A corrupt source fails the parse on a truncated record, - so it is caught outright and leaves the previous table serving. But a source that has **vanished** - parses perfectly and imports a table quietly missing everything it contributed — an unmounted volume - and a deliberate deletion are indistinguishable from here, which is precisely the ambiguity the - atlas stages a facet removal for. So it is escalated: `status: 'needsReview'`, nothing applied, - `missingSources` reported by both the import and `status()`, and an admin accepts it with - `{approve:true}`. That is a flag rather than the atlas's approve/reject pair because the atlas - stores a pending decision so that approving **re-parses** the tree; here nothing is stored, so - re-reading at approval time is automatic. - -**Resolution is server-side and there is no public route.** The table is never served *as* a table: -67k rows would dwarf any page using them, and the Android client consumes the same already-resolved -JSON. `resolveMany()` returns only ids that resolved to something displayable — placeholders like -`~1_val~` are stripped, since the bridge sends the id and never the property packet that carries the -arguments — and it never throws, because a cliloc lookup is decoration on a character sheet. +The prefixes are grandfathered ([`MODULE_API.md`](MODULE_API.md) §6.5) — a new module prefixes its +tables with its own id. ### installed_modules — what is installed, and what happened to it (module system) @@ -863,12 +646,12 @@ for web back-compat. **The `/player/*` group is self-service, not player-only.** Staff are a **superset** of players — every player ability plus their staff tools on top — so the whole group (`account.router.js`, -`shard.router.js`, `appeals.router.js`, mounted by `player/index.js`) sits behind the shared -`noindex, requireAuth` gate **only**, never `requireRole('player')`. Every handler is self-scoped to the caller by `req.user.id`, so an admin/editor/ +`appeals.router.js`, mounted by `player/index.js`, plus whatever a module mounts here) sits behind +the shared `noindex, requireAuth` gate **only**, never `requireRole('player')`. Every handler is self-scoped to the caller by `req.user.id`, so an admin/editor/ moderator using it sees only their **own** linked accounts and characters (with the pre-existing -`isAdmin` bypass still letting a genuine admin read *any* character). Staff also reach the identical -self-scoped handlers under `/admin/shard/*` (same controller) for the web admin surface; the two are -interchangeable. This is why a staff account with linked game characters gets its "My characters" and +`isAdmin` bypass still letting a genuine admin read *any* character). `module-uo` inherits the rule +and relies on it: its `/player/shard/*` handlers are the identical self-scoped ones it also serves +under `/admin/shard/*`, so the two are interchangeable. This is why a staff account with linked game characters gets its "My characters" and personal notification streams on the mobile client — the group no longer 403s a non-`player` role. **Password reset.** Uses the same audited pattern as `user_invites`: an opaque 32-byte token @@ -886,16 +669,17 @@ checked content over the authenticated API. Two producers fan out through the on ingest dispatcher (`utils/shardIngest`, beside the SSE broadcast) for shard-derived streams, and the create/publish-post path for `news.post`. The catalog is assembled at boot by `modules/registries.js` from core's own streams (`config/coreStreams.js` — just `news.post`) plus -each installed module's; the shard streams and their event→stream mapping are -`config/shardStreams.js`, which belongs to module-uo and moves out with it -(MODULE_SYSTEM.md §1.8). Security invariants: -- **Same public/admin split as the SSE feed.** Public streams are drawn *only* from the SSE - `PUBLIC_KINDS` allowlist; a sensitive kind (audit/cheat/IP/login-attempt) can never produce a public - push. -- **Personal streams are owner-keyed.** `vendor.sale` / `house.idoc` / `account.login` are delivered - only to the *owning* user's devices, resolved via `shardLinks` (the same ownership check as - `/player/shard/*`) in `utils/shardPush.js`. `utils/pushDispatch.js` itself only publishes to a - stream id someone else resolved — it has no idea what a shard event is. +each installed module's. The seven shard streams and their event→stream mapping left with +`module-uo` in Phase 3 and are registered by it; their ids are grandfathered to that module +([`MODULE_API.md`](MODULE_API.md) §6.5) because they are stored in `notification_subs` and read by +the shipped Android app. Security invariants: +- **Whether a stream is safe to publish is the registering module's decision, and it stays inside + that module.** `module-uo` applies the same public/admin split as its SSE feed — public streams are + drawn only from its own allowlist, so a sensitive kind (audit/cheat/IP/login-attempt) can never + produce a public push — and resolves personal streams (`vendor.sale`, `house.idoc`, + `account.login`) to the *owning* user's devices through its own ownership check. Core never sees a + shard event. **`utils/pushDispatch.js` publishes to a stream id someone else resolved** and knows + nothing about what produced it, which is what lets a second game's module reuse the whole pipe. - **SSRF guard.** A device `endpoint` is a client-supplied URL the server POSTs to, so registration and every publish validate it is HTTPS, non-private/loopback, and (when configured) on the shard's ntfy allow-set (`NTFY_BASE_URL` / `NTFY_ALLOWED_ORIGINS`). @@ -973,7 +757,8 @@ expiry (~5 min); `/exchange` is rate-limited per-IP. The bridge tables self-prun **No group gate, deliberately.** This surface is anonymous by design: the SPA renders it logged-out, the Discord bot reads it with no credentials, and the Android `ShardStreamClient` consumes -`/public/shard/stream` without an `Authorization` header. Content visibility during maintenance comes +module-uo's `/public/shard/stream` without an `Authorization` header — a module mounting here +inherits the same "no gate" and owns whatever gate it adds. Content visibility during maintenance comes from the per-route **siteMode** middleware (§5), never from an auth gate. | Method | Path | Notes | @@ -987,19 +772,7 @@ from the per-route **siteMode** middleware (§5), never from an auth gate. | GET | `/wiki` | list of pages (slug + title) | | GET | `/wiki/:slug` | single page | | POST | `/contact` | (rate-limited) send mail via SMTP; if unconfigured, respond `{fallback:"mailto", email}` | -| GET | `/shard/ruleset` | the shard's own published ruleset (Protocol 3.0 `world.ruleset`): expansion, which optional systems are on, skill/stat caps, account and house limits, champion scroll rules, the save/restart schedule. Served from `shard_ruleset`, so it renders while the shard is down; live via `world.ruleset` on `/shard/stream`. Behind `requireFeature('ruleset')`. **`null`** means the shard has never published one — a real answer, distinct from a published ruleset. `caps.skill` / `caps.totalSkill` are in **tenths** (1000 = 100.0). | -| GET | `/shard/points` | every points/loyalty leaderboard the shard publishes (Protocol 3.0 `points.board`) — Queen's Loyalty, Void Pool, the nine city loyalties, Clean Up Britannia, … Served from `shard_points_boards`, so it renders while the shard is down; live via `points.board` on `/shard/stream`. Behind `requireFeature('leaderboards')`, ordered by display name. **`maxPoints: 0` means uncapped** (the common case), and `nameString` is usually `null` with `nameNumber` holding a cliloc — resolve client-side or humanise the `system` key. | -| GET | `/shard/points/:system` | one board by the shard's `PointsType` name (e.g. `QueensLoyalty`); `:system` must match `/^[A-Za-z][A-Za-z0-9_]{0,47}$/` or **400** before any query runs. **404** = the shard has never published that system, which is distinct from a published board nobody has scored in yet (**200** with an empty `top`). | -| GET | `/shard/market?q=&minPrice=&maxPrice=&itemId=&map=®ion=&sort=&limit=&offset=` | search the player-vendor marketplace (Protocol 3.0 `vendor.listing`). Returns **listings**, not vendors — "who sells X and for how much" is the question, and a vendor-shaped result would make every caller flatten the shops back out. Served from `shard_vendors` + `shard_vendor_items`, so it renders while the shard is down. Behind `requireFeature('market')` **and rate-limited** — the first genuinely expensive public read on the site (a `LIKE` scan plus a `COUNT` over what is typically the largest `shard_*` table, reachable with no session). `sort ∈ {price_asc, price_desc, recent}`. `q` matches the resolved display name **or** the item's literal name, with `%`/`_` escaped: they are `LIKE` metacharacters, not SQL ones, so parameterization alone would let `?q=%` match every listing on the shard. Every response repeats `staleAt` (the oldest vendor row) because the shard sweeps round-robin — a banner that ages with the results it labels, not one fetched once. | -| GET | `/shard/market/meta` | index size, staleness (`staleAt`/`freshAt`) and which facets and regions actually hold vendors, so a client builds its filters without running a search it will discard. | -| GET | `/shard/market/vendors/:serial` | one shop and its listings; `:serial` must match `/^0x[0-9A-Fa-f]{1,16}$/` or **400** before any query runs. **404** = a serial the index has never seen, which also covers a vendor since dismissed or hidden — to an anonymous caller those are the same answer, and distinguishing them would leak that a hidden vendor exists. `truncated` (with `total` exceeding `count`) means the shop holds more than the shard publishes per frame. | -| GET | `/shard/features` | the shard features **this caller** may reach plus the audience rung they resolved to (§6.5), so a client hides nav it can't follow. Reports only what the caller can see — the list itself never discloses a gated feature. Consumed by the SPA header and (pending) the Android nav. | -| GET | `/atlas/creatures?q=&facet=&limit=&offset=` | the bestiary, most numerous first, with an unpaginated `total`. Static content parsed from the shard's ServUO tree — **not** sidecar-backed, which is why the atlas sits outside `/shard`, and unlike `/shard/*` it **is** site-mode gated. Behind `requireFeature('atlas')`. `?facet=` is matched exactly and never validated against a list (no facet name exists in the code); the filter is an `EXISTS` over the points rather than a JSON path or `JSON_SEARCH` built from caller input, whose `%`/`_` wildcards would make `?facet=%` match everything. | -| GET | `/atlas/creatures/:slug` | one creature: `places` (the point-in-rect aggregate — "lizardman → Shrines, Isamu-Jima, Yew"), `spawners` (the bounded raw list, with `spawnersTruncated`), `alsoHere`. **`points` is a COUNT and `spawners` is the LIST** — named apart so one key never means a number on one route and an array on another. `minDelay`/`maxDelay` are in **seconds**, normalised at parse time from the source's per-record minutes-or-seconds. 404 = no such creature in this atlas. | -| GET | `/atlas/regions?facet=&q=` | named regions and the rectangles that placed each spawner | -| GET | `/atlas/landmarks?facet=&q=` | points of interest, labelled by `group` ("Covetous", not "Level 1") | -| GET | `/atlas/champions?facet=` | the **configured** altar roster. Not `/shard/champs`, which is the live board. | -| GET | `/atlas/meta` | facets, counts and when the atlas was parsed. Game-world facts only — the ServUO path, source hashes and any pending refresh are operator detail and live on the admin route. | +| — | `/shard/*` · `/atlas/*` | **Served by `module-uo`, not by core** (25 routes). Documented in [`../modules/uo/API.md`](../modules/uo/API.md); absent entirely when the module is not installed, which is a 404 and not an error. | Public content GETs pass through the **siteMode** gate (§5). @@ -1021,14 +794,18 @@ These rows are configuration that happens to need a login. `users`, `invites`, `auth/providers` and `bot-activity` add `adminOnly` on top, and `moderation` adds `modAccess` (admin + moderator, so editors are excluded). The content capabilities — `posts`, `uploads`, `wiki`, `pages` — add nothing: managing content is the editor tier's job, so `staffOnly` is -the whole gate. The ops/config capabilities — `uo-link`, `email`, `discord-bot`, `settings`, and -`PUT /site-mode` — are `adminOnly`; `shard` is the one mixed prefix, where self-service account -linking carries no extra gate and the in-game staff operations carry `modAccess`. There is no residual -file: every admin route is declared in a capability router. +the whole gate. The ops/config capabilities — `modules`, `email`, `discord-bot`, `settings`, and +`PUT /site-mode` — are `adminOnly`. There is no residual file: every admin route is declared in a +capability router. -`GET`/`PUT /admin/shard/visibility` are the third tier on that mixed prefix: **`adminOnly`**, because -they decide what *anonymous* visitors can see (§6.5). They sit above `modAccess` deliberately — a -moderator can ban a player but cannot decide what the public internet reads. +**A module mounts into this group as a peer**, at a prefix it claims and core has verified nothing +else owns; the shared gate above applies to it, and any gate beyond that is the module's own. So the +mixed-tier prefixes here are `module-uo`'s `/admin/shard` and `/admin/uo-link`, documented in +[`../modules/uo/API.md`](../modules/uo/API.md) — not core's, and absent from this table. + +`/admin/modules` is `adminOnly` rather than `staffOnly` for the reason the endpoint exists: it +installs code that will run inside the server process at the next boot. An editor or a moderator has +no business doing that, and the group gate alone would let them. `GET /dashboard` and `PUT /site-mode` are the one place where a **single screen spans two tiers**: the dashboard is staff-wide, but the site-mode toggle on it is `adminOnly`. The client must therefore gate @@ -1057,13 +834,15 @@ file a route sits in — that is the property the route manifest freezes. | GET | `/users/:id/trusted-devices` | list a user's active trusted devices (never tokens) | | DELETE | `/users/:id/trusted-devices` · `…/:deviceId` | revoke all / one of a user's trusted devices (logs `admin.trusted_device.revoke[_all]`) | | POST | `/users/:id/mfa/reset` | recover a locked-out user: disable TOTP + revoke all trusted devices + clear recovery codes (logs `admin.user.totp.reset`) | -| GET | `/shard/atlas` | spawn-atlas status (`adminOnly`): the ServUO path, whether the tree is readable, whether it has drifted from what is loaded, counts, facets, and any refresh staged for review. The public `/atlas/meta` reports the game world only; the filesystem detail is here. | -| POST | `/shard/atlas/import` | re-import without restarting; `{force}` ignores the hash gate. **An unreadable tree answers 200 with `status:"unavailable"`, not 500** — `refresh()` reports outcomes rather than throwing (the boot path must never be blocked by a bad tree) and that contract is preserved at the API. | -| POST | `/shard/atlas/approve` · `/shard/atlas/reject` | answer a refresh staged because it would REMOVE a facet. Approving **re-parses** the tree, so what lands matches it at approval time; rejecting is remembered against those source hashes so it does not re-prompt every restart. 404 when nothing is staged. | -| PUT | `/shard/atlas/path` | point the atlas at a different tree (persisted as `spawn_atlas_servuo_path`, which wins over `SERVUO_PATH`). Blank clears it. Deliberately **does not import** — moving the mount and reloading the world are separate decisions — and returns fresh status so the panel can offer the import next. | -| GET | `/shard/clilocs` | cliloc-table status (`adminOnly`): every source found now (base first, then `custom/` overlays in merge order), what each contributed at the last import, readability, drift across the set, the entry count, and `missingSources`. `configured:false` is a supported state — item names then render as ids. No public counterpart: the table is never served *as* a table. | -| POST | `/shard/clilocs/import` | reload after a client patch or an overlay edit; `{force}` ignores the hash gate, `{approve}` accepts a **vanished** source (refused by default — see the table notes above). **A missing path — or the likely mistake of pointing at the client's own COMPRESSED `Cliloc.enu` — answers 200 with `status:"unavailable"` and a `code`, not 500.** `COMPRESSED` is called out by name: a 500 would say only "something broke", and the operator needs to be told which file to convert. | -| PUT | `/shard/clilocs/path` | point the site at a different cliloc base file or directory (persisted as `cliloc_client_path`, which wins over `UO_CLIENT_PATH`). Overlays are read from `custom/` beside it either way. Blank clears it. Deliberately **does not import**, same reasoning as the atlas path. | +| GET | `/modules` | installed modules reconciled across all four sources of truth — the `installed_modules` row, the live loader record, the modules volume, and the `MODULES` declaration — plus the install-source allowlist. They are allowed to disagree, and the screen renders the disagreement rather than picking one (module system, [`MODULE_SYSTEM.md`](MODULE_SYSTEM.md) §2.4) | +| POST | `/modules` | install or upgrade from a release **install-manifest URL**: allowlisted `https` host, declared `sha256`, whole-archive inspection, unpack into a scratch dir, move into place last. Takes effect at the next restart. `adminOnly`, rate-limited, audit-logged — this endpoint installs code that will run in the server process | +| PUT | `/modules/sources` | replace the host allowlist. Seeded from `MODULE_SOURCE_HOSTS` on a fresh install and DB-owned from then on, so changing the variable never overwrites an operator's choice. An **empty list forbids every install**, never permits all | +| POST | `/modules/restart` | graceful shutdown so module changes take effect; the supervisor brings the process back (`restart: unless-stopped` on the shipped compose). Emits `SIGTERM` **as an event** rather than signalling the pid — `process.kill` is unconditional termination on Windows | +| POST | `/modules/:id/enable` | move the row to `enabled`. Deliberately does **not** touch the loader: there is no `onBoot` re-dispatch, so the screen asks for a restart | +| POST | `/modules/:id/disable` | the one module action that takes effect immediately — dispatches that module's `onShutdown`, then its routes, nav and client chunk answer 404. A real kill switch, not a visibility flag | +| POST | `/modules/:id/purge` | run a **disabled** module's `purge.sql`, dropping its tables and data. `409` while it is still running; `400` if it ships no `purge.sql` | +| DELETE | `/modules/:id[?purge=true]` | uninstall: stop, then (with `purge=true`) drop its data, then delete its directory. Non-destructive by default — the row stays `disabled` and the data is left for a reinstall to pick up. The purge option lives here because it cannot live after: `purge.sql` is a file inside the directory being deleted | +| — | `/shard/*` · `/uo-link/*` | **Served by `module-uo`, not by core** (33 routes). Documented in [`../modules/uo/API.md`](../modules/uo/API.md) | Every admin write logs to `activity_log`. @@ -1133,11 +912,11 @@ who"; `activity_log` provides the history feed. - **Cookie**: `httpOnly`, `sameSite=Lax`, `path=/`, and **`secure` decided per-request** (`COOKIE_SECURE=auto` → `secure: req.secure`). - **Trusted-device MFA.** A second, separate httpOnly cookie (`rg_trust`, default 30d) — opaque, sha256-hashed server-side in `trusted_devices` — lets a browser/app **skip the TOTP step** (never the password) on future logins. It is a server-side, per-row-revocable record (never a JWT claim), so the stateless session JWT is unchanged and trust stays revocable. It only ever gates the **second factor**; it deliberately outlives logout, and is cleared on untrust / password change / password reset / TOTP disable. **Recovery codes** (bcrypt, single-use) are the 2FA-lockout fallback. All admin trusted-device/MFA actions and the self actions (`auth.login.trusted_device`, `account.trusted_device.*`, `account.recovery_code*`, `admin.trusted_device.*`, `admin.user.totp.reset`) are audit-logged. See `docs/website/TRUSTED_DEVICES_MFA.md`. This is the key to dual access: the cookie is `Secure` when reached through Pangolin (HTTPS, `X-Forwarded-Proto: https`) but **not** `Secure` when reached directly over the LAN IP on plain HTTP — so login works in both. `COOKIE_SECURE=true|false` can force it. Requires `trust proxy` (below). `localhost:5173` (Vite) and `localhost:3000` are same-site, so the cookie flows in dev too. - **bcrypt** hashing (cost 10+); plaintext passwords never stored, logged, or returned. -- **Rate limiting** (`express-rate-limit`) on `/auth/login`, `/public/contact`, and — the only limited - *read* — `/public/shard/market` and `/public/shard/market/vendors/:serial` (60/min/IP). Every other - public read is an indexed lookup of bounded size; the marketplace search is a `LIKE` scan plus a - `COUNT` over the largest `shard_*` table, anonymous by default, so it is the one public GET that is - worth money to serve. +- **Rate limiting** (`express-rate-limit`) on `/auth/login`, `/public/contact`, and the account-change + routes. Every core public read is an indexed lookup of bounded size, so none is limited. A module + gets the same factory through `ctx.middleware.rateLimit` and is expected to use it on any read that + is expensive to serve — `module-uo` limits its marketplace search (60/min/IP), the one public GET + in the system that costs real money to answer. - **Validation** (`express-validator`) on all writes; centralized error handler. - **helmet** with a Content-Security-Policy tuned for the built React SPA. The policies now live in **`server/src/config/csp.js`** (`app.js` only wires them up): @@ -1181,76 +960,17 @@ who"; `activity_log` provides the history feed. - **`app.set('trust proxy', 1)`** so secure cookies, `req.ip`, and rate-limiting work behind Pangolin. - **CORS**: same-origin in prod (SPA served by Express). Dev only: allow `CLIENT_ORIGIN` (Vite, `http://localhost:5173`) with `credentials:true`. -### 6.5 Shard visibility — the audience boundary (Protocol 3.0) +### 6.5 Module-owned audience boundaries -Every shard-derived surface is gated by an **admin-configurable, per-feature and per-field** audience -setting. This **replaces** the static `PUBLIC_KINDS` allowlist that used to be the whole boundary. -Policy lives in `utils/shardVisibility.js`; rows live in `shard_feature_visibility`; the admin surface -is `GET`/`PUT /admin/shard/visibility` (`adminOnly`). Admin-facing guide: -[`SHARD_VISIBILITY.md`](SHARD_VISIBILITY.md). Design: [`../link/v3.md`](../link/v3.md) §3. +Core's security boundaries end at authentication, roles and the session. A module that serves +game data brings its own audience rules, and core does not police them beyond the gates it hands +over (`requireAuth`, `requireRole`, the tier group gates). -**The ladder.** `anonymous < logged_in < player < staff < admin`, each rung implying the ones below. -`viewerLevel(req)` resolves it: no session ⇒ `anonymous`; authenticated ⇒ `logged_in`; authenticated -with a linked game account ⇒ `player`; moderator ⇒ `staff`; admin ⇒ `admin`. **Staff satisfy the -`player` rung without a linked account** (consistent with `/player/*` being role-agnostic). -**`editor` gets no shard privilege** — it is a content role, and mapping it to `staff` would silently -widen what editors see. - -**Two invariants that are code, not configuration.** Both are enforced server-side and both reject -rather than silently ignore: - -1. **`acct` and `webId` are admin-only, always.** They are not exposed as configurable fields, and a - stored row attempting to loosen them is discarded on read as well as rejected on write. A character - name is visible in game; the account behind it and the website user it links to are not. - The lock is on the field's **meaning, not one spelling**: `isLockedField(key)` matches a key that - *is* or *ends in* `acct`/`webId`, case-insensitively, so the flattened forms the read models emit - (`shapeHouse` → `ownerAcct`, `shapeGuild` → `leaderWebId`) are covered too. An exact-key check was - the original implementation and it let `GET /public/shard/idoc` serve `ownerAcct` anonymously. -2. **A kind absent from `KIND_FEATURE` is never broadcast below `admin`.** Fail closed. This is what - keeps the kind map a security boundary rather than a convenience filter, and it means a shard that - starts emitting an unknown event degrades to staff-only, never to public. - -**Fail-closed everywhere else too.** An unreadable visibility config withholds every public frame; a -DB failure falls back to the compiled defaults (pre-3.0 behavior), not to open; an unresolvable viewer -subscribes as `anonymous`. The ladder comparison uses **asymmetric** fallbacks by design — an unknown -*viewer* level floors to the bottom rung and an unknown *requirement* ceils to admin, so an -unrecognised value loses on both sides. (A single shared fallback cannot do that: whichever direction -it picks, it fails open on one side.) - -**Three enforcement points, one config:** - -| Where | Mechanism | -|---|---| -| Routes | `requireFeature(name)` — **404** when the feature is disabled (don't leak that it exists), **403** when the caller is below its audience. `projectFeature` then strips out-of-rung fields from the body. | -| SSE (`utils/shardBroadcast.js`) | Per-connection filtering. A subscriber's rung is resolved **once at subscribe time and frozen** for that connection, so a long-lived stream can't gain privilege; each frame is then mapped kind→feature, gated, and field-projected per viewer. Two subscribers can legitimately receive different versions of one event, or one of them nothing. | -| Nav | `GET /public/shard/features` returns only what the caller may reach, so the SPA never renders a link that would 403. Presentation only. | - -Config reads are cached ~5s, so admin changes take effect within seconds **including on already-open -streams**. `PUBLIC_KINDS` still exists and is still exported (`utils/shardBroadcast.js`) but is now -**derived** from the kind map rather than hand-maintained, so the two cannot drift. - -**`PUBLIC_KINDS` is a module-load constant and must not be used to answer "may this caller read this -kind?"** — it is computed from the compiled *defaults*, so it cannot see an admin's changes. Use -`visibleKinds(level, config)`, which resolves against the live config. `/feed` uses it; it originally -used `PUBLIC_KINDS` and consequently kept serving `guild.join` to anonymous callers after an admin had -moved `guilds` to `staff`. `visibleKinds` deliberately ignores the `stream` flag: that governs SSE -fan-out only, so a feature whose live firehose ships off (market) stays readable from stored history. - -**Every read path that returns shard data must call `projectFeature`.** The stored-history endpoints -are not exempt — `/feed` returns the same events the stream does, and returning them unprojected -reopens on the REST side exactly what the stream closes. Relatedly, `shardEvents.db.list` treats an -**empty** `kinds` array as "serve nothing", never "no filter"; the fall-through it used to take would -have turned a fully-gated config into a dump of the entire event log. - -`projectFeature` walks **arrays and plain objects only**. A `Date`, `Buffer` or other class instance -is passed through as a value — rebuilding one key-by-key yields `{}`, which is the difference between -the pure-JSON wire frames and the DB-backed read models whose rows carry real `Date` columns. - -**Defaults reproduce pre-3.0 behavior exactly**, so installing the framework is a no-op until an admin -changes something — with deliberate exceptions, which are the leaks it was written to close. -`/public/shard/guilds`, `/public/shard/governors` and `/public/shard/feed` previously returned the raw -stored payload, whose actors carry `acct` and `webId`; `/public/shard/idoc` returned the flattened -`ownerAcct`. All are now stripped for every caller below admin. +`module-uo`'s is the worked example, and it is a real boundary rather than a convenience filter: an +admin-configurable, per-feature and per-field audience ladder with fail-closed defaults, applied at +routes, at SSE subscribe time and at the nav. It used to be documented here as core's; it moved to +[`../modules/uo/API.md`](../modules/uo/API.md) §4 when Phase 4 closed, with the admin-facing guide +still at [`SHARD_VISIBILITY.md`](SHARD_VISIBILITY.md). --- diff --git a/website/MODULE_SYSTEM.md b/website/MODULE_SYSTEM.md index e1cd671..db24075 100644 --- a/website/MODULE_SYSTEM.md +++ b/website/MODULE_SYSTEM.md @@ -1397,10 +1397,12 @@ Six slices, 2026-08-11. Core is 158 routes and knows nothing about any game; mod | 3 | Route manifest diff is only the deliberate move | core 158 + module 72, **0 core routes removed** | | 4 | A written `module-rust` dry run | [`../modules/rust-dryrun.md`](../modules/rust-dryrun.md) | -**Phase 4 — Delivery.** The admin-panel Modules screen (install, enable, disable, retry, purge, -`startup_failed` with its recorded reason) and the Docker-environment path from §2.5. Deliberately -last, so loader, packaging, schema and chunk-loading problems are not all being debugged at once. -Its shape, and the four decisions it turned on, are in §2.7.2 below. +**Phase 4 — Delivery. COMPLETE 2026-08-12**, in five slices. The admin-panel Modules screen (install, +enable, disable, retry, purge, `startup_failed` with its recorded reason) and the Docker-environment +path from §2.5. Deliberately last, so loader, packaging, schema and chunk-loading problems are not +all being debugged at once. Its shape, the six decisions it turned on, the per-slice record and the +acceptance table with its results are in §2.7.2 below — all four criteria met, each against the real +published `module-uo` release rather than a fixture. **Phase 5 — The Integration Kit.** `RunicGateway/Integration-kit`, the instruction book for building a module for a game that is not UO — the website module, the sidecar and why it exists, and the @@ -1542,12 +1544,12 @@ rather than merely depended on.)* ##### What Phase 4 must prove -| # | Criterion | How | -| --- | --- | --- | -| 1 | A module installs, starts and serves with no shell access and nothing built | Admin panel install from a release URL, restart, module `started`, its pages render | -| 2 | Uninstall leaves the data, purge removes it | Uninstall then reinstall recovers the rows; uninstall **with the purge box ticked** then reinstall does not | -| 3 | A hostile archive cannot write outside `modules//` | Unit tests over the rejection list above, each with a real crafted tar | -| 4 | A compose-declared module resolves at container start, offline | Container boots with the network down and an already-unpacked module, and comes up unchanged | +| # | Criterion | How | Result | +| --- | --- | --- | --- | +| 1 | A module installs, starts and serves with no shell access and nothing built | Admin panel install from a release URL, restart, module `started`, its pages render | **Met** (slice 2). `module-uo` v0.3.0 installed by pasting its manifest URL: five mounts, seven streams, 37 schema statements, `started`, its own nav rows in the sidebar, its pages rendering | +| 2 | Uninstall leaves the data, purge removes it | Uninstall then reinstall recovers the rows; uninstall **with the purge box ticked** then reinstall does not | **Met** (slice 4). On an empty database: uninstall → reinstall kept a marker row and all 27 tables; uninstall with purge → `purged: 27`, every table gone; reinstall → tables recreated **empty**. Standalone purge: `409` while started, `200` while disabled. Found the [ordering defect](#the-defect-criterion-2-found) below | +| 3 | A hostile archive cannot write outside `modules//` | Unit tests over the rejection list above, each with a real crafted tar | **Met** (slice 1). 16 tests over real crafted tars — absolute, `..`, NUL, backslash, symlink, hardlink, device, multiple top-level entries, oversize, over-count — plus the measured proof that node-tar rejects an escaping member *late*, which is why the inspection is a separate pass | +| 4 | A compose-declared module resolves at container start, offline | Container boots with the network down and an already-unpacked module, and comes up unchanged | **Met** (slice 3). `already at the declared version 0.3.0`, **zero requests**, module up. Two unresolvable declarations logged as errors without stopping the site; uninstall + restart returned the files with the row still `disabled` | #### Slice 1 — the install service and the admin API (website#142, 2026-08-12) @@ -1784,6 +1786,82 @@ matter (a running module whose declared upgrade failed, and a declared module wi volume); the rendering above that is one derived line, covered by the client suite. It is the one thing in this slice not proved in a browser. +#### Slice 4 — closing the phase (website#146, 2026-08-12) + +Documentation, the acceptance table above, and one defect the acceptance run turned up. Core's code +change is a three-line reorder; the bulk of the slice is that **`BACKEND_DESIGN.md` had never been +de-UO'd**. Phase 3 rewrote core's *code* and, in slice 5, core's README and OpenAPI metadata — but +§5.2's identifier check reads code, not prose, so nothing ever looked at the design document. It was +still describing 27 `shard_*` tables, 20 UO route rows and the shard visibility ladder as **core's**, +a phase after core stopped being able to serve any of them. + +##### What moved, and the rule it follows + +| Was | Now | +| --- | --- | +| `BACKEND_DESIGN.md` §3 — six `shard_*` schema sections, 226 lines | [`../modules/uo/SCHEMA.md`](../modules/uo/SCHEMA.md) | +| `BACKEND_DESIGN.md` §4 — 13 public + 7 admin UO route rows, the `/admin/shard` tier prose | [`../modules/uo/API.md`](../modules/uo/API.md) | +| `BACKEND_DESIGN.md` §6.5 — the audience ladder, 70 lines | [`../modules/uo/API.md`](../modules/uo/API.md) §4 | + +The text is moved, not rewritten — a relocation that also reworded would make it impossible to tell +which parts changed meaning. What core keeps is the **seam**: `installed_modules`, `/public/modules`, +the eight `/admin/modules` routes, the extension slot, and — new in this slice — a paragraph in each +place saying *a module mounts here as a peer, inherits this group's gate, and owns whatever gate it +adds on top*. §6.5 became "Module-owned audience boundaries", which is core's half of that sentence: +core's security boundary ends at authentication, roles and the session, and a module that serves game +data brings its own. + +The general rule, and it is worth stating because the next module will need it: **core's reference +documents core's surface; §2.10 already said module documentation aggregates in `docs/modules//`, +and that has to include the parts core used to own.** A design doc that keeps describing a module is +a doc that silently becomes wrong the first time an operator runs core without it. + +Two stale things fell out on the way, both invisible until the tree was read against the manifest: +`users.router.js` was still listed as 15 routes (it is **9** — six went to the extension slot), and +the push-notification section still promised `config/shardStreams.js` "belongs to module-uo and moves +out with it", in the future tense, four slices after it left. + +##### The defect criterion 2 found + +Proving criterion 2 meant running the one destructive path nobody had run: install `module-uo` v0.3.0 +from the real release onto a **brand-new empty database**, let it build its 27 tables and import a +real atlas and a 67,496-row cliloc table, write a marker row, and then take it all away twice — once +keeping the data and once not. + +Both directions were correct. What the run exposed is the **order** the uninstall does it in: + +``` +purge → stop → removeDir ← was +stop → purge → removeDir ← is +``` + +`purge.sql` dropped 27 tables while the module was still `started`, and it stayed started until +`lifecycle.stop()` finished — up to the five-second `onShutdown` budget. In that window the module is +serving and ingesting against a schema that no longer exists: module-uo's uo-link WebSocket keeps +writing shard events into dropped tables, and requests in flight answer **500** where a stopped module +answers **404**. The comment justified the old order as *"purge while the SQL is still readable"* — +but `removeDir` is the only step that touches the filesystem, so the file was readable either way. +The dependency chain the comment described was real for one link and imagined for the other. + +The trade-off is now stated where the code is: if the purge fails, the module is left +stopped-and-disabled rather than untouched. That is recoverable from the panel; a live module on a +half-dropped schema is not. The 400 for a module shipping no `purge.sql` moved above the stop, so a +refused request stops nothing. + +**Generalises: an ordering that is only wrong for a few seconds is invisible to every test and to any +smoke that does not have a live producer.** This one needed a module whose `onBoot` opens a socket +and whose tables are being written continuously — which is to say, it needed the real module against +a real release, not a fixture. + +##### What a purge does and does not take + +Worth writing down because the answer surprised the run: after purge-and-reinstall the module's +tables come back and the **atlas and cliloc content comes back with them** — 67,496 rows — because +that content is re-derived from the operator's own ServUO tree at the next `onBoot`. What is gone is +everything the shard and its players produced. The two `settings` rows survive on purpose, which +`purge.sql` explains at the top: `uo_link_protocol_3_migrated` is a one-shot migration marker, and +deleting it would re-arm a protocol bump against tables that no longer exist. + ### 2.8 SPA URL namespacing — a deliberate break **Decision: module pages are namespaced, and old paths are not redirected.** The site is not public @@ -1930,3 +2008,10 @@ row for it — when it has content, not while it is an empty repo. | 18 | A module's frozen manifest is the **difference** between a core without it and the same core with it — never a prefix filter | API §5.3 | | 19 | A module's release version is **declared** in `module.json`, not computed from commit subjects; the workflow tags and publishes and never writes to a branch | §2.7.1 | | 20 | A module namespaces the schemas it **defines** and references core's shared ones by core's name | API §2.8, §6.1a | +| 21 | Restart is a **button** in the panel, not an instruction — the endpoint raises `SIGTERM` at itself and the supervisor brings the process back | §2.7.2 d1 | +| 22 | The install source is a **pasted install-manifest URL** against a host allowlist, never a catalog | §2.7.2 d2 | +| 23 | Disable dispatches that module's `onShutdown` and is a real kill switch; enable is **not** its mirror and asks for a restart | §2.7.2 d3 | +| 24 | The declarative Docker set is an **environment variable** (`MODULES`), resolved in the server process before `app.js` is required | §2.7.2 d4 | +| 25 | Purge is offered **inside** the uninstall flow, because `purge.sql` lives in the directory being deleted | §2.7.2 d5 | +| 26 | The host allowlist **bootstraps from `MODULE_SOURCE_HOSTS`** into a settings row and is DB-owned thereafter | §2.7.2 d6 | +| 27 | Documentation follows the code out: what core's reference described and no longer serves moves to `docs/modules//`, text unchanged | §2.7.2 slice 4 | diff --git a/website/SHARD_VISIBILITY.md b/website/SHARD_VISIBILITY.md index 31254f9..84c3a86 100644 --- a/website/SHARD_VISIBILITY.md +++ b/website/SHARD_VISIBILITY.md @@ -3,7 +3,9 @@ **Status:** Built (Protocol 3.0 Part A). Admin → Shard Visibility. **Audience:** shard owners and admins. **Companion to** [`../link/v3.md`](../link/v3.md) §3 (the design) and -[`BACKEND_DESIGN.md`](BACKEND_DESIGN.md) §6 (the security contract). +[`../modules/uo/API.md`](../modules/uo/API.md) §4 (the enforcement contract — it moved there with the +rest of module-uo's surface; core's [`BACKEND_DESIGN.md`](BACKEND_DESIGN.md) §6 keeps the security +contract core still owns). The website surfaces a lot of live shard data. What your players, your staff and the anonymous internet may each see is **yours to decide**, per feature, from Admin → Shard Visibility. diff --git a/website/SPAWN_ATLAS.md b/website/SPAWN_ATLAS.md index 343ad62..4c51e87 100644 --- a/website/SPAWN_ATLAS.md +++ b/website/SPAWN_ATLAS.md @@ -191,7 +191,7 @@ All are **import-owned**: a refresh empties and reloads them in one transaction, so a failed reload leaves the previous atlas intact rather than a half-loaded world. Nothing else writes to them and nothing holds a foreign key to them — no FKs at all, consistent with every other `shard_*` table. Full column listings in -[`BACKEND_DESIGN.md`](BACKEND_DESIGN.md). +[`../modules/uo/SCHEMA.md`](../modules/uo/SCHEMA.md) — these are module-uo's tables, not core's. | Table | Rows (stock) | Notes | |---|---|---|