Closes Phase 2. Modules live on a mount, never in the image — that is what
lets an operator add one to a pull-only deployment without building anything.
`./modules` is a bind mount rather than a named volume: placing a module
directory by hand is a supported install (MODULE_SYSTEM.md §2.5), and that has
to be doable from the host rather than through `docker cp`. Read-write, because
the admin panel's install/uninstall unpacks and removes directories there.
The directory is tracked via its README so it exists in the checkout with the
operator's own ownership — Docker recreates a missing bind-mount source as
root:root, which the container user could not then write. `.dockerignore`
excludes it so a module in the builder's working tree can never ship inside an
image.
Also corrects the route-manifest generator's list of filesystem-conditional
mounts, which never picked up `/modules` when PR 7 added it. Comment only; the
generator filters on an allowlist, so its behaviour was already right.
Verified against a real container, not just a parsed compose file: image
carries an empty node-owned /app/modules despite a module in the build context;
a module on the bind mount loads, mounts, replays and reaches `started`;
`/api/v1/public/modules` lists it; the chunk serves from the entry's directory
only (server source and module.json 404) with `no-cache`; the injected tag
follows core's bundle; and in Chrome the page renders on first paint inside
core's PublicLayout with its nav row interleaved into core's public nav, under
enforced `script-src 'self'` with zero CSP reports and no console errors.
Removing the directory by hand reconciles the row to `startup_failed`/`require`
and leaves core healthy with no injection.
933 server + 160 client tests pass, manifest unchanged at 230 routes, swagger
regenerates byte-identical.
Co-Authored-By: Claude <noreply@anthropic.com>
Phase 2, PR 8 of docs/website/MODULE_SYSTEM.md 2.7 - the nav half PR 7
deferred, plus the two seams 1.4 and 1.5 asked for.
withModuleNav (client/src/modules/nav.js) merges an installed module's rows
into core's three navs BEFORE the admin-override merge, and that ordering is
the design. applyNavOverrides and buildPublicNav are keyed by `to` and drop
any key their base array does not declare, so rows appended after the merge
would be unorderable, unrelabellable and unhideable in Admin - Navigation.
Today's UO rows are all three of those things, so appending would make the
extraction a visible regression for anyone who has ever edited their nav.
Merging first means a module row is an ordinary row downstream: nothing in
navOverrides.js, NavEditor.jsx or the layouts knows a module exists.
MOD_PATHS is gone. Moderator visibility and the redirect that confines a
moderator both derive from each row's own `roles`, in the new plain-JS
lib/adminNav.js (plain so the DOM-less runner can reach it). Two rows move,
both toward what the server already permitted: Dashboard, whose roles had
always named moderator, and My Characters, which is ungated self-service.
That also fixes a defect predating the module system. The redirect was a
THIRD hardcoded list - three path prefixes against MOD_PATHS' five paths -
and they disagreed about /admin/houses, so a moderator who clicked Houses in
their own sidebar was bounced back to Moderation. The derived allow-list is
computed from the BASE nav, never the override-merged one: an override is
presentation and must not move an authorization boundary either way.
The feature seam (modules/features.jsx + modules/featureGate.js) resolves a
row's `feature` against the provider its OWN module registered, so the
namespace comes from the registration and no string carries a parsed prefix.
Core registers useShardFlags under the owner id `core` - the client twin of
registries.registerCore() - so the ten shard-gated header rows already run
through the seam and Phase 3 deletes a registration instead of rewriting
SiteHeader. Every unknown fails open: no provider, a null answer while a
fetch is in flight, or a junk return all show the link, because the server is
the gate and hiding a page from someone entitled to it is the worse mistake.
933 server tests (unchanged - this PR is client-only), 160 client tests
(+37). routes.manifest.json unchanged at 230 routes; the OpenAPI spec
regenerates byte-identical.
Re-ran the MODULE_API.md 7.7 browser smoke, since this is the seam that rule
exists for. A throwaway module registering nav in all three areas and a
provider granting one flag and withholding another: the row lands inside
core's Moderation group rather than an appended block, the withheld row does
not render, a moderator reaches both /admin/houses and the module's admin
page, and an admin can relabel a module row and have it persist and apply.
Zero CSP reports, zero console errors.
Co-Authored-By: Claude <noreply@anthropic.com>
Phase 2, PR 7 of docs/website/MODULE_SYSTEM.md 2.7 — the client half's
delivery. A module's prebuilt chunk is served, injected, handed core's React
and its UI kit, and its routes are rendered by App.jsx. The registry is empty
on a bare core, so nothing an operator can see changes.
Client:
- modules/registry.js — registerRoutes/registerNav/registerFeatureProvider,
with the URL namespace written by core, never by the module
- modules/shared.js — window.__rg: React, react-dom/client, react-router-dom,
react/jsx-runtime, the registry, the seven-member UI kit and the request
primitive, frozen
- App.jsx reads routesFor for all three areas; nav consumption is PR 8
- main.jsx publishes the global, then mounts on DOMContentLoaded
Server:
- the loader validates client.entry and publishes clientChunks() and
clientEntryUrls(); an entry in the module root is rejected, because the
directory it sits in is what gets served
- app.js mounts each chunk at /modules/<id>/ behind the module's state guard
with no-cache; anything else under /modules is a 404, not the SPA shell
- htmlShell injects the tag before </body>, so core's bundle runs first
wherever a bundler puts it
Found by loading a real chunk in a browser, and fixed here: core mounted before
any module chunk had evaluated, because document.readyState during a deferred
script is 'interactive', not 'loading'. Every test passed against that build.
The smoke is written down in MODULE_API.md 7.7.
933 server tests (+23), 123 client tests (+14). routes.manifest.json unchanged
at 230 routes; the OpenAPI spec regenerates byte-identical.
Co-Authored-By: Claude <noreply@anthropic.com>
Phase 2, PR 6 of docs/website/MODULE_SYSTEM.md 2.7 — the first module-system
URL a client can see. The SPA and the Android app feature-detect against the
capabilities a module declares; the shape is settled in MODULE_API.md 2.9.
Four decisions, and what is absent from the payload is most of the design:
* started modules only. A module that is disabled or failed to load is
ABSENT, exactly as 4.4 already leaves its routes and its nav absent, so a
client renders a site without that capability rather than advertising one
that 503s.
* no state, failure_stage or failure_reason. Where a module broke belongs to
the admin Modules screen, and the reason is an exception string from inside
core — not anonymous-visitor business.
* no client chunk URL. htmlShell injects a script tag per started module
(3.1.3), so the browser is handed the tag rather than a URL to fetch. This
endpoint feature-detects; it does not load. MODULE_SYSTEM 2.6 step 4 is
amended to match (API 6.7).
* no siteMode gate and no database — the same class as /public/status and
/public/version, so a client can still feature-detect during maintenance.
It is a capability router of its own rather than a fifth singleton in
site.router.js, and that is load-bearing: the loader's prefix-collision probe
reads the live tier stack and skips root-mounted layers, because a use('/', ...)
matches every path. A route inside the root-mounted site router would be
invisible to it — mounting use('/modules', ...) is what makes "no module may
claim /modules" a rule the loader enforces.
910 tests pass (+9, every one on the boundary — what must NOT appear).
routes.manifest.json gains exactly the one route and routes.guards.json records
it with an empty gates list, which is itself the assertion that it is ungated.
Co-Authored-By: Claude <noreply@anthropic.com>
`npm test` never terminated. Twenty-two test files omitted the two lines that
point the pool at a dead port, so utils/db.js -- which builds its mariadb pool at
require time and calls dotenv.config() itself -- picked up server/.env and opened
five live connections to the developer's MariaDB. The tests still passed, because
they stub their models and never issue a query; the only symptoms were a process
that never exited and five connections held for as long as it lived. Thirty
stranded workers is 150 connections, which is the whole server's limit, and that
is the "too many connections" this workspace has hit before.
The convention was right and only ever as good as the next test file's memory of
it, so it moves into the harness: test/_setup.js is loaded with --require by the
npm script, ahead of the test file it hosts, which is the only moment early
enough to matter. It pins the dead port -- dotenv does not overwrite an existing
variable, so an explicit DB_PORT= still wins for anyone who wants a live database
-- and closes the pool after the file's tests, so the process exits at once
instead of waiting out the driver's connect retries. The per-file preambles stay:
they keep `node --test test/one.test.js` safe on its own.
Two supporting fixes:
- db.close() is idempotent. pool.end() throws "pool is already closed" on a
second call, and closing twice is now normal rather than exceptional -- the
harness closes the pool for every file on top of the suites that close it
themselves, and a SIGINT followed by a SIGTERM already reached the shutdown
handler twice.
- test/_helper.js's close() destroys open connections. server.close() only stops
accepting and waits for existing connections to end, and node's global fetch
keeps its sockets alive, so the listener outlived the test that created it --
invisible until now, because the pool was holding the process open anyway.
announceJobs.test.js alone: 120s+ hang -> 0.35s. The whole suite now finishes in
~75s where it previously did not finish at all: 901 tests, 901 pass, verified
three times on CI's exact platform (node:20 on Linux, via Docker).
Co-Authored-By: Claude <noreply@anthropic.com>
Phase 2, PR 5 of docs/website/MODULE_SYSTEM.md 2.7. api.onBoot/api.onShutdown
stop throwing, server.js gains one call on each side, and the 2.4 state machine
finally runs against real outcomes -- which is what makes 4.5's `disabled` 404
leg reachable for the first time.
Dispatch and reconcile live in src/modules/lifecycle.js rather than in the
loader, for the reason the schema replay does: routeManifest.js and swagger.js
both require app.js against a dead pool, so the loader may not reach the
database. The two halves meet at exactly one function, loader.setState(), so the
in-memory record the dispatch guard reads and the row the admin panel reads are
moved together and cannot disagree.
Four decisions, all recorded in MODULE_API.md 2.5 and 4.4:
- The loader classifies its failures by 4.3 step, so failure_stage says where a
module broke instead of being a column nothing ever filled. The four steps
readManifest covers in one pass label themselves; the rest are inferred from
how far load() had got, and an unlabelled throw is recorded against the step
that was running rather than guessed at.
- A row whose directory is gone is marked startup_failed rather than left
claiming `enabled` -- the boot reset has just moved it there, and a row
claiming to be enabled for a module that is not on the volume is the one state
that is simply untrue. An uninstall leaves `disabled`, which the reset never
touches, so this catches only a hand-deleted directory.
- Core's eight UO boot call sites stay in server.js until Phase 3. Unlike a
registered announce leg, a boot call site already has somewhere to live, so
moving it now would be extraction done early in a phase whose exit criterion
is that nothing changes.
- onBoot gets no timeout. Shutdown races a SIGKILL and boot does not, and a slow
onBoot delaying the listener is the contract's promise to a module that must
warm up before it serves.
The operator's switch wins over everything: a disabled module is guarded, not
booted, and does not have its failure re-recorded, or an outcome would silently
switch it back on next boot. Every database write in the reconcile is
individually caught -- a row that will not update is worse reporting, never a
failed boot.
900 tests pass (17 new). routes.manifest.json is unchanged at 229 routes and the
OpenAPI spec regenerates byte-identical.
Co-Authored-By: Claude <noreply@anthropic.com>
Phase 2 PR 4 of docs/website/MODULE_SYSTEM.md §2.7. Adds server/src/modules/registries.js
and moves core's own notification streams, announce leg and users-detail routes
behind it, so the three seams §1.8 and §1.9 named are exercised on every boot
before any module depends on them.
Registering is validate-then-commit per registrant: the loader stages what a
module claims and the second pass commits it, so a module that throws halfway
through register() — or fails checkDeclared after it — leaves nothing behind.
That is the registry-side twin of PR 2's second-pass mount rule.
Four decisions, all the recommended option:
- announce legs became a child table. `announce_job_legs` replaces the
towncrier_*/discord_* column groups, so the leg set is data: core registers
`discord`, module-uo will register `towncrier`, and a module cannot ALTER a
core table to add its own. Backfill is guarded on information_schema (a
SELECT of a dropped column is a parse error, not a runtime one) and the
columns go with DROP COLUMN IF EXISTS. Verified against the live dev DB:
three legacy jobs migrated faithfully, three replays, no duplicates.
- `mapEvent` dropped from registerNotificationStreams. §1.8 already inverts the
push path so a module owns fromShardEvent and calls core's publish() with a
stream id it resolved; a second mapping mechanism was a leftover. The public
safety filter, the kinds it reads and the streams it protects now live in one
file and move together.
- core registers through the same staging area a module uses, via an explicit
registries.registerCore() in app.js before modules.load().
- core's six /admin/users/:id/shard/* paths now go through the
`admin.users.detail` slot, and getUser moved back to admin.controller.js.
Found on the way, and the reason two build tools changed:
- scripts/routeManifest.js could not decode a parameterised mount. Its
unwinder expected `(?:([^\/]+?))`; express 4.22 emits `(?:\/([^/]+?))` with
the separator inside the group. The branch had never run. It threw rather
than guessing, which is what it is for.
- swagger-autogen cannot follow a route into an extension slot — the slot's
router is created by declareSlot() and filled later, so there is no literal
mount for a static parse. Regenerating deleted 407 lines and printed
`Swagger-autogen: Success`, the spike's exact failure (MODULE_API.md §7.4).
swagger/slotSpecs.js generates a fragment per filled slot and re-roots it at
the prefix the router actually hangs at in the live app — read from the
express stack via routeManifest's own mountPath, so the manifest and the spec
cannot disagree. swagger/mergeSpec.js is the merge helper core owes for
module fragments anyway (§6.1a), proved here against core's own slot first.
884 tests pass (856 before). routes.manifest.json is unchanged at 229 routes.
The OpenAPI spec diff is two lines of intent: the retry endpoint's summary, and
its `leg` no longer being a fixed enum.
Co-Authored-By: Claude <noreply@anthropic.com>
Phase 2, PR 3 of docs/website/MODULE_SYSTEM.md 2.7. ensureSchema() now replays
every installed module's schema fragment immediately after core's schema.sql,
per MODULE_API.md 2.6.
The work splits across two files on the line of whether a database is needed to
know the answer. loader.js VALIDATES a fragment at load time, before anything is
mounted, because every rule 2.6 states about the SQL is knowable by reading it;
a module that breaks one never mounts (4.4, left column). modules/schema.js
EXECUTES it, so the only failures there are the ones the database alone could
report, and those are post-mount and answer 503 (4.4, right column).
Validation is a leading-verb allowlist -- CREATE, ALTER, INSERT, UPDATE, the
four core's own schema.sql uses -- rather than the DROP denylist 2.6 words it
as. A fragment is replayed on every boot, so TRUNCATE and DELETE would empty a
table at each restart and RENAME would fail at the second one; a denylist only
ever bans what somebody thought of. A CREATE TABLE missing IF NOT EXISTS is
rejected for the same reason: it works once and fails every boot after, which
presents to an operator as a module that broke on restart.
The splitter moves to utils/sqlStatements.js so core's schema and a fragment are
split by literally the same code, which is what 2.6 promises. It is its own file
rather than an export of utils/db.js because the loader validates fragments at
require time and must not drag the mariadb pool into app.js's require chain.
The replay sits outside ensureSchema's wait-for-the-database retry loop: a
fragment that throws is one module's failure, not a signal the database is
coming up, and retrying core's whole schema nine more times over one module's
bad SQL would turn a 503'd module into a two-minute boot.
Found while wiring it: db/seed.js calls ensureSchema() standalone for
`npm run seed`, without ever requiring app.js, so the loader has not scanned and
fragments()'s 7.6 throw would have broken seeding outright. The replay asks
isLoaded() and logs the skip rather than swallowing it -- a booting server
quietly getting no module tables is the thing 7.6 exists to prevent.
Verification:
- 856 server tests pass, 14 new. moduleSchema.test.js injects the query fn, so
the exact statements and their order are asserted with the pool at a dead port
like every other suite.
- routes.manifest.json and routes.guards.json diffs are zero lines, 229 routes
-- the phase 2 exit criterion. swagger-output.json regenerates byte-identical.
- Run for real against the local MariaDB with two fixture modules: a good
fragment created its table, applied its ALTER and seeded its row; a fragment
whose SQL passes validation but the server rejects (`id NOTATYPE`) marked only
that module startup_failed, its route answering 503 while the other answered
200; a second ensureSchema on the same database was a clean no-op.
Co-Authored-By: Claude <noreply@anthropic.com>
Phase 2 PR 2 of docs/website/MODULE_SYSTEM.md 2.7. Adds
server/src/modules/{loader,semver,version}.js: the synchronous scan of
MODULES_DIR, manifest validation, prefix and table-name collision
rejection, per-module try/catch and the mount into the three tier
routers behind the MODULE_API.md 4.5 dispatch guard.
Two decisions the contract left open, both now written up there:
- The load trigger is one explicit modules.load(tierRouters) call in
app.js, not a lazy scan (API 7.6). Accessors throw until it has run,
because "no modules installed" is a real answer a caller must not be
handed by accident.
- Whether core owns a prefix is asked of the live tier routers via
express's own layer.match(), skipping root-mounted layers, rather than
a hardcoded table -- the spike's was already stale when written
(API 4.3).
Mounting is a second pass after every module is validated. Doing it
inside the scan loop makes the first module's layers indistinguishable
from core's, so the second module claiming a taken prefix is told it
collided with core and the module-versus-module check is unreachable.
registerExtension/NotificationStreams/AnnounceLeg and onBoot/onShutdown
throw "not available until phase 2 PR 4/5" rather than no-op; an
accepting stub would let a module believe it had registered something.
No schema replay, no boot dispatch, no installed_modules reconcile --
those are PRs 3 and 5, and until PR 5 a record's state is in memory only.
No module ships on the volume, so nothing an operator or client can see
changes: 842 tests pass, routes.manifest.json is unchanged at 229 routes
and swagger-output.json regenerates byte-identical.
Co-Authored-By: Claude <noreply@anthropic.com>
Phase 2 PR 1 of the module system (docs/website/MODULE_SYSTEM.md 2.7). The
table and the state machine only: no loader, no routes, no boot wiring, so
nothing an operator or a client can see changes and the route manifest diff
is zero lines.
The five states of 2.4 live in one `state` column: installed -> enabled ->
started, with disabled and startup_failed as the recoverable ones. The row is
a record of what happened, never the source of truth for what is mounted --
the loader scans the filesystem at require time, before the database is
reachable (MODULE_API.md 4.1), which is what keeps routes.manifest.json
generatable against a dead database.
Two rules the model owns and the boot path will lean on:
- Every boot resets each non-disabled row to `enabled` and clears its
recorded failure, so a startup_failed module is retried on the next restart
and a fixed one recovers with no admin-panel visit. `disabled` is the one
operator decision rather than outcome, so it survives untouched -- and a
disabled module's failure is a no-op, never a re-enable.
- A failure carries the stage it happened at, and every non-failing
transition clears it, so a running module can never show a stale reason.
An illegal move throws instead of writing a row that misrepresents the state,
except on the two boot-path softenings noted above, because one module's
failure must never become everybody's.
22 model tests over an in-memory fake; the SQL and the DDL were round-tripped
against a real MariaDB separately.
Co-Authored-By: Claude <noreply@anthropic.com>
Long workstreams land phase by phase on `edge` and reach `main` as a single
cutover. With `branches: [main]` alone, every one of those phase PRs merges
with no checks at all -- no server tests, no client build, no bot install --
and the whole workstream runs blind until the cutover, where the breakage
arrives all at once and un-bisected.
This is not hypothetical: it is what happened to all nine Android M12 phase
PRs in the Android-app repo, whose pr-checks.yml carries the same trigger.
It matters for the module system specifically because Phase 2's exit
criterion IS a CI result -- a zero-line routes.manifest.json diff and a
passing suite -- and that manifest is the frozen URL surface protecting
three shipped clients. A branch accumulating work for weeks needs the gate
more than main does, not less.
Landing before the module-system edge branch is cut, so the first phase PR
is checked. build-images.yml is deliberately untouched: it triggers on push
to main, so images publish and production rolls at the cutover and never
before.
Co-Authored-By: Claude <noreply@anthropic.com>
The public header's dropdown trigger is a <button class="pill"> sitting in a row
of <a class="pill"> links, and it rendered ~7px shorter.
It was not failing to pick up the theme: font-size, font-family, padding, border
and box-sizing all matched exactly. The one property that differed was
line-height, because form controls do not inherit it — the UA stylesheet gives
<button> `line-height: normal` (~1.15), while the anchors inherited body's 1.6.
38.02px against 31px, which is precisely 22.016 - 15.8.
Stating it on .pill fixes it at the source rather than patching the one button:
every other property in that rule is already explicit for the same reason, and
this was the remaining gap. The value matches body's 1.6, so no link pill
changes. The ~70 <button class="pill"> elsewhere in the admin gain the same 7px
and now line up with the .btn buttons they sit beside.
.btn has the same latent difference and is deliberately left alone: it is used on
80 buttons and 2 anchors, they never appear on the same row, so nothing is
visibly wrong and the blast radius is not worth it.
Co-Authored-By: Claude <noreply@anthropic.com>
Phase 10 of docs/website/THEMING_AND_NAV.md, asked for before the edge -> main
cutover. An admin can now create dropdown sections in the public header, organise
the coded entries into them, and add links of their own.
This deliberately amends §7, which said the override layer "cannot introduce a
`to` that is not already in the hardcoded NAV array". That stays true of every
CODED entry; an admin may now also add a link, restricted to a same-origin path —
no scheme, no protocol-relative //host. A link carries no gate of its own and
needs none: the page behind it enforces its own access, so an added link
advertises a route and never grants one.
The invariant is kept structurally rather than by vigilance. Coded entries live
in an `items` map whose keys must be routes the base array declares, so that map
cannot invent a route; everything that CAN name an arbitrary path lives in
`links`, which is the one place the path rule is applied — on both the write and
the read path.
nav_public therefore grew a { items, sections, links } wrapper. A bare map still
reads as the items map, and a nav with no sections still stores one, so this
changed nothing for a nav that does not use it. Free to do now because nothing
has shipped; after the cutover it would have needed a migration.
The Public tab gets its own editor. A public section is an entry in the
top-level order that the admin created and can drag among the pills, unlike the
admin sidebar's four coded sections, where only membership moves — that is a tree
rather than a list of groups. Deleting a section returns its entries to the top
level rather than removing them, which is the one destructive act this screen
could otherwise commit.
The dropdown opens on click and never on hover, and its trigger is not a link: a
hover menu is unusable on touch, and a trigger that navigates means tapping to
open takes you somewhere instead. Escape closes and returns focus, an outside
press closes, navigating closes, and Arrow Up/Down walk the items.
pruneNav applies the shard-feature gate inside a section and drops one it leaves
empty, so a dropdown never opens onto nothing.
Also fixes a bug this surfaced in the phase 6-8 code: the save path judged "does
this route still exist?" against the palette — the base array already filtered to
what the editing admin can see — so on the public header a feature-gated row's
override could never be carried through and would have been silently reset.
Membership is now judged against the full coded nav while the rows still come
from the palette.
Co-Authored-By: Claude <noreply@anthropic.com>
Phases 6-8 of docs/website/THEMING_AND_NAV.md. The public header, the admin
sidebar and the player portal now read their override row, and /admin/navigation
writes them: rename, reorder by drag, hide, and — on the admin sidebar — move a
row into another existing section.
The merge always runs BEFORE the role and shard-feature filters in the layouts,
which are unchanged and remain the boundary. An override is presentation: it
cannot introduce a route, cannot touch a `roles` or `feature` gate, and a stored
`hidden: false` on a gated item shows nobody anything.
The design scoped these phases as client work, but the server had no way to
store a nav row: updateSettings validates and stringifies theme_visual and
brand_assets and lets everything else through, so a nav object would have been
written as "[object Object]" and read as absent for ever. utils/navOverrides.js
mirrors utils/brandAssets.js — strict on write with the offending key named,
forgiving on read. It validates shape only; whether a `to` exists is settled
client-side at merge time, because the base NAV arrays are client constants and
a server-side copy would be a second source of truth that drifts.
The nav editor cannot be hidden — its own toggle is disabled, the write path
drops `hidden` on that one `to`, and AdminLayout strips it again before merging,
which also covers a row edited straight in the database.
Orders are written only when the sequence actually differs from the code's, and
the comparison is restricted to the rows the editing admin can see, so renaming
one item does not pin the position of every other one and a role- or
feature-gated item missing from their palette is not mistaken for a reorder.
Co-Authored-By: Claude <noreply@anthropic.com>
Phase 5 of docs/website/THEMING_AND_NAV.md: uploaded logo/hero/favicon
overrides on top of the BRAND_* env defaults, delivered through an HTML
shell that is no longer built once at boot.
- utils/htmlShell.js owns the shell lifecycle: rendered lazily, cached per
process, invalidated on a brand_assets/theme_visual write with a 5-minute
TTL so other workers converge. A settings-read failure renders the
env-only shell and caches that, so a DB outage is not a failing query per
page view, and with no rows the output is byte-identical to what app.js
served before.
- POST /admin/settings/brand-asset/:slot uploads one asset and writes the
row in the same call, so an upload never leaves an unreferenced file. It
reuses the shared multer allowlist and only tightens it per slot: favicons
are PNG-only and capped at 512 KB, logos at 1 MB, heroes at 8 MB. Refused
files are unlinked before the response.
- utils/brandAssets.js constrains a stored asset to a same-origin path under
/uploads, /brand or /assets — these are the only settings values written
straight into the page as a URL. Strict on write, forgiving on read.
- The shell also carries the resolved theme as a <style id="theme-boot">
block, removing the first-paint flash phases 3-4 deferred; SiteContext
drops that block once a successful settings fetch has been applied.
- BrandLogo renders beside the MoonDot on all six shells and renders nothing
when no logo is set, which is the shipped default.
Co-Authored-By: Claude <noreply@anthropic.com>
Phases 3-4 of docs/website/THEMING_AND_NAV.md. Three presets, the curated font
shortlist, and /admin/appearance to drive them.
The design put the presets in theme.css as [data-theme] blocks. That does not
work: SiteContext writes --accent as an inline style on <html>, which beats any
attribute-selector block, so a preset's accent would have been painted over by
BRAND_ACCENT_COLOR while getPublic().brand.accent -- the value the Android app
themes itself from -- reported the other one.
Presets now live in server/src/config/themePresets.js. themeResolve.js layers
:root <- preset <- custom per field into a token map, getPublic() returns it as
`theme`, and the client writes it onto <html>. One authority for the merge, and
brand.accent is by construction the accent the site paints. theme.css's :root is
untouched, so an instance with no row gets no theme block and renders as today.
Also: presets carry the full 15-token palette (eight would have left Fantasy
with blue-grey borders); the option catalog is served from
GET /settings/theme/options so the form cannot offer what the server rejects;
validation is strict on write and forgiving on read; and the Discord bot now
fetches the effective accent instead of its boot-time env copy.
Fixes a Phase 0 bug in passing: settings/nav.controller.js imported the logger
factory rather than calling it, so a DB fault would have thrown a TypeError
inside the catch instead of returning 500.
Co-Authored-By: Claude <noreply@anthropic.com>
Phases 0-2 of docs/website/THEMING_AND_NAV.md. Groundwork only: no admin UI,
no consumer wiring, and an instance that never touches the new settings keys
renders exactly as it does today.
Phase 0 - settings store:
- settingsDb.remove() and DELETE /api/v1/admin/settings/:key, the "reset to
default" primitive. Defaults for these keys live in BRAND_* env, theme.css
and the hardcoded NAV arrays, so reset has to delete the row rather than
store a copy of the default. Allowlisted to the five theming/nav keys plus
hero_layout_draft, admin-only, idempotent.
- GET /api/v1/settings/nav behind requireAuth with no role gate. AdminLayout
renders for editors and moderators and PlayerPortalLayout for players, and
none of them can read GET /admin/settings, so without this their nav
override would silently never apply.
- A fifth router group for it: /public is anonymous, /admin/settings is
adminOnly, /player is self-scoped data. This is configuration that needs a
login.
- parseJsonSetting() in utils/settingsJson.js. settings.value is TEXT, so
every JSON key arrives as a string; malformed or wrong-shaped reads as
absent, never as an error and never half-applied.
- theme_visual / brand_assets / nav_public join PUBLIC_KEYS; nav_admin and
nav_player deliberately do not.
Phase 1 - client/src/lib/navOverrides.js, the pure merge util. Presentation
only: it can set label/order/hidden and (grouped navs) group, and nothing
else. It cannot introduce a `to`, cannot touch roles/feature, and hidden:false
cannot un-hide anything - the existing filters run afterward, unchanged, and
remain the boundary.
Phase 2 - promoted 23 border-radius literals in theme.css to four tokens at
today's values (14x8px, 4x999px, 4x10px, 1x12px). The 7px/6px editor chrome
and the two 50% circles stay literal. --shadow-card and --panel-grad were
already tokens.
Tests: 16 new server tests, 20 new client tests. The route-manifest guard now
also asserts /settings/** sits behind requireAuth. Swagger and both route
artifacts regenerated.
Co-Authored-By: Claude <noreply@anthropic.com>
The shard integration section documented the contract in detail but never
told an admin where the base URL, WS URL, protocol version and token come
from. They come from the installer, which prints them at the end of a run.
Co-Authored-By: Claude <noreply@anthropic.com>
ServUO ships Server.cfg with `Name=My Shard`. An operator who never edited it
publishes that verbatim, so the rules page read "My Shard" under a header
carrying the real name. That value is the shard saying *unnamed* rather than
naming anything, so the site now answers with its own.
`settings.getInstanceName()` resolves `site_title || BRAND_NAME` — the same
resolution `getPublic().brand.name` already uses, so an install that set only
the site title can never show two different names on two pages. Bare
`brand.name` would have been wrong for exactly that case.
Substituted at INGEST rather than on read: world.ruleset is also broadcast
live, and the same object is handed to the SSE fan-out, so a read-time fix
would be undone by the next reconnect's frame. Matched case- and
padding-insensitively but only as a whole value, so a shard genuinely called
"My Shard Reborn" keeps its name.
Fixes a second ruleset writer found on the way: uoLinkSocket.backfill() called
shardState.setRuleset directly instead of going through the dispatcher as
ingestEach does, so the boot/reconnect snapshot silently skipped this
normalization. The two arrival orders have to produce the same stored frame.
Also renders a placeholder row on an unscored leaderboard — the instance name
with an em dash where a score goes, deliberately not shaped like an entry (no
medal, no bar) because a placeholder that looked like a real standing would be
a fabricated one. Presentation only; the API still sends an empty `top`.
Verified live against the shard + sidecar: rules page and leaderboards on web
and Android both correct.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U7CBg11prhLimL9iHSX1bP
The site's declared version is the admin-set uo_link_config.protocol column, so
the sidecar's PROTOCOL_VERSION 2 -> 3 bump has to be matched here or every REST
call 409s and uoLinkSocket closes the WS on the ws.hello mismatch. Five places
carry the number and all five move together: the column default, the model's
DEFAULT_PROTOCOL (what a site with nothing saved yet declares), the two
`config.protocol || 1` fallbacks in uoLinkClient/uoLinkSocket -- unreachable
today, but an unset value quietly sending 1 is exactly the confusing 409 the
version check exists to prevent -- the admin form's initial value, and the
documented env default.
The boot migration is the only subtle part. schema.sql is re-run on EVERY boot,
and `protocol` is admin-editable, so a bare UPDATE would silently un-pin an
operator who had deliberately pinned an older sidecar in Admin -> Shard. It is
therefore gated on a marker row in `settings`, written after the UPDATE: the
first boot on this build migrates, every later boot is a no-op. `protocol < 3`
rather than `= 2` picks up an install still on the old default of 1, which could
not have been talking to a v2 sidecar anyway. A fresh install has no row to
update and just gets the marker plus the new column default.
Verified against the local MariaDB through ensureSchema (the production path):
2 -> 3 with the marker written and the column default now 3; pinned back to 2 by
hand, re-ran, and it STAYED 2 -- the one-shot property holds. 673 server tests,
47 client tests, client build green.
Co-Authored-By: Claude <noreply@anthropic.com>
Protocol 3.0 §8, the website half. Ingests vendor.listing / vendor.listing.remove
into shard_vendors + shard_vendor_items, serves a searchable public API over
them, and ships /site/market and /site/market/vendors/:serial.
Three things the pages have to say out loud, all consequences of how the data is
gathered:
- The prices are NOT live. The shard sweeps vendors round-robin, so a shop can be
a full cycle behind. The banner is driven by the OLDEST vendor row, not the
newest — the one stale shop is the one that wastes somebody's trip.
- A shop can be truncated. `total` exceeding `count` means the shop holds more
than the shard publishes per frame; the vendor page says "showing 250 of 3,104"
rather than presenting a partial shop as complete.
- An item may have no name. On a shard with no cliloc table the honest render is
the item id, never an invented label.
## The pre-wired visibility rules, re-checked
Part A pre-wired market.ownerName and market.location before the frame existed,
and the sibling rule it pre-wired for leaderboards (`characterName`) turned out
to be INERT because projectValue matches literal JSON keys. Both market rules
were checked against the real frame this time:
- `ownerName` is a real key. Kept.
- `location` is a real key ONLY because the frame nests it. Flat map/x/y/region
would have made the rule match nothing — the same failure, one part later. It
is nested on the wire and on the read model so one rule hides the facet, the
coordinates, the region and the house together; five flat keys would be five
rules that drift apart.
- `ownerSerial` was ADDED. An admin who hides the owner's name and leaves a
serial that the leaderboards and guild boards resolve back to that same name
has not hidden anything.
Tests assert all three bite, on the stored read model AND on the raw frame —
the market's SSE stream is off by default but an admin can turn it on, and a rule
that worked on only one path is exactly the leak §3.6.1 records.
## Notable
- **No payload column on shard_vendors**, unlike shard_points_boards next door.
The board's top-N is a fixed-size list read whole; here the items ARE the
searchable rows, so they are normalized and nothing is left worth duplicating.
- **display_name is denormalized at ingest** (literal name preferred over the
cliloc — a player set it, so it is more specific). Resolving at query time
would put the cliloc table on the hot path and make search-by-name impossible.
Because the shard's diff sweep will not re-send an unchanged shop just because
the site learned what its items are called, a cliloc import now triggers a bulk
re-resolution — 50 ms per thousand rows, never throws.
- **updated_at is written explicitly** on every upsert. MariaDB does not fire ON
UPDATE CURRENT_TIMESTAMP when every column is written back unchanged, and a
shop re-published identically is still freshly confirmed — without this the
staleness banner would age a perfectly current shop forever.
- **LIKE wildcards in `q` are escaped.** `%` and `_` are LIKE metacharacters, not
SQL ones, so parameterization does not neutralize them: `?q=%` would otherwise
match every listing on the shard.
- **Rate-limited** (60/min/IP), the only limited public read. Every other public
GET is an indexed lookup of bounded size; this is a LIKE scan plus a COUNT over
the largest shard_* table, anonymous by default.
- Reconnect backfill pages /market, bounded by MARKET_SNAPSHOT_MAX = 5000 and
stopping on a short page as well as on `total`, so a concurrent sweep shrinking
the index cannot spin the walk.
## How it was tested
673 server tests pass (27 new). Client builds clean; swagger-output.json,
routes.manifest.json and routes.guards.json regenerated.
Verified full-stack against the live MariaDB and a real shard, not only units:
- 27 real vendors / 1,040 listings swept off the ServUO tree, through the Rust
sidecar, into the site — names resolving through the cliloc table ("longsword",
"katana"), real facets and regions in the filters.
- `?q=sword` 682, `?q=%` and `?q=_` **0** (the escape), map/region/price/sort
filters, paging, and the vendor detail route.
- Visibility live: fields gated to staff vanish for an anonymous caller while
shopName and price survive; audience=player 403s; enabled=0 404s; and
/shard/features correctly drops `market` so the nav hides it.
- Re-publishing a shop smaller leaves no orphan items; an identical re-publish
moves updated_at.
- The limiter fires (38x200 then 32x429 on a 70-request burst).
Not covered by an automated test: the two React pages are presentational and this
repo's client suite covers pure-logic modules only. They were driven against the
live API above, but not rendered in a DOM harness.
Co-Authored-By: Claude <noreply@anthropic.com>
Shards edit items and add new ones, and those carry cliloc ids no stock client
table has. Reading exactly one converted file meant an operator had to
re-export 5 MB every time they added one item — friction enough that the table
would simply go stale, which is the failure the spawn atlas was redesigned to
avoid in the first place.
So this mirrors spawnAtlasSource.readSources(): a BASE (the converted client
table) plus every operator-maintained overlay under `custom/`, all re-read on
every boot and hash-gated as a SET. Later sources win, so an overlay both adds
ids the client never had and overrides stock ones the shard re-purposed.
Adding, editing or removing any overlay counts as drift.
`custom/` is the one convention here that is ours rather than the shard's, and
deliberately so: ServUO has no server-side notion of a custom cliloc — they
live in the patched client a shard distributes, and nothing in the tree
declares them. There is nothing to discover. (An operator who does patch their
client cliloc needs no overlay: convert the patched file and the edits are in
the base.) Scale, measured on the live shard: its script tree references 16,434
cliloc ids and only 37 are absent from stock — tens against a 67k base, which
is why this is an overlay and not a second table.
The set brings back a hazard a single file did not have, and it gets the
atlas's answer. A corrupt source fails the parse loudly, but a source that has
VANISHED parses perfectly and imports a table quietly missing everything it
contributed — an unmounted volume is indistinguishable from a deliberate
deletion. So it is staged, not applied (`needsReview`), reported by both the
import and status(), and accepted with `{approve:true}`. That is a flag rather
than the atlas's approve/reject pair because the atlas stores a pending
decision SO THAT approving re-parses; here nothing is stored, so re-reading at
approval time is automatic.
Also reports a per-source breakdown (entries/added/overrode) on import and in
status, which is how an operator confirms an overlay took effect — "overrode: 0"
on a file meant to re-label stock items says it did not.
Two bugs this surfaced, both found by running a shard-style overlay rather than
by another stock-table fixture:
- displayText tidied punctuation unconditionally, so a custom
"Runic Gateway Sigil (v2)" rendered as "(v2". Stripping leftover brackets is
right after a placeholder is removed and wrong otherwise — the same condition
the `%` rule already had.
- CANDIDATE_NAMES did not include `clilocs.plain`, which is the exact filename
CLILOCS.md and the export tool's README tell operators to write. Pointing at
the directory they were told to create failed with NO_FILE.
Verified end to end against the live MariaDB and a real server boot: base-only
import, overlay adding one id and overriding another (per-source breakdown
correct), unchanged set as a no-op, an edited overlay re-importing and
withdrawing its override, a vanished overlay refused with the table intact,
status reporting missingSources, approve applying it, and a file-path
configuration still finding overlays beside it. All three resolve correctly
through the running server: shard-added, overridden and stock. 646 server tests
pass (16 new in clilocSource.test.js, 3 new in clilocParse.test.js); swagger,
routes.manifest.json and routes.guards.json regenerated.
Co-Authored-By: Claude <noreply@anthropic.com>
Protocol 3.0 §8.6 (docs/link/v3.md), the dependency order 5 was sequenced
behind. Items on the wire carry a LabelNumber, not a name — the bridge has
always sent it (char.profile.equipment.cliloc, reward titles as a cliloc
number in string form, and one per marketplace listing) but the site had no
table to resolve it against, so a character sheet could only render
`id 1023721` where the game renders "quarter staff".
The number was never the missing piece. The table was.
Sourced from a file the operator converts once from their own client, at a
path from the `cliloc_client_path` setting falling back to UO_CLIENT_PATH.
Nothing client-derived is committed: UO's strings are EA's, exactly as the
creature sprites are. A shard with nothing configured is fully supported —
names render as ids, as they did before.
The conversion step is not avoidable, and that is the substantive finding
here: every current client ships its cliloc files COMPRESSED (first DWORD's
high byte 0x8E, the Mythic container), and ServUO's own bundled
Ultima.StringList cannot read that either — so VendorSearch.GetItemName is
already inert on such a shard and the plugin could not supply names instead.
v3.md's original "read the client's Cliloc.enu" recommendation was therefore
not implementable as written, and its committed db/data/clilocs.json artifact
also predates the Part C corrections (no committed derived snapshots, nothing
EA-derived shipped). Replaced with the spawn-atlas pattern: parse on boot from
an operator-configured path, hash-gated, output gitignored.
- utils/clilocParse.js — pure parsers, fs-free so the suite runs in CI.
Accepts the plain binary layout and delimited text, sniffed by header rather
than extension. Rejects a compressed file BY NAME: without that check the
plain parser reads it as ~19k records of negative ids and 60 KB "strings"
before dying mid-file, and the resulting error names the wrong problem.
displayText() drops the ~1_val~ arguments the bridge never sends.
- utils/clilocSource.js — the fs layer. hashSource reports `compressed` so the
admin panel can flag an unconverted file WITHOUT parsing 5 MB per poll;
otherwise pointing at a client directory reports a healthy file with pending
drift ("ready to import") and the operator only finds out on failure.
- model/shardClilocs — refresh/status/lookup. All-or-nothing replace (DELETE,
not TRUNCATE — TRUNCATE is DDL in MariaDB and implicitly commits). Batched
server-side resolution behind a capped cache; never throws, because a cliloc
lookup is decoration on a character sheet.
- Deliberately NO staged-approval flow, unlike the atlas: the atlas escalates
facet loss because a half-copied tree and a real map change are
indistinguishable from inside the process, whereas a partial cliloc copy
makes the parser fail on a truncated record. The ambiguity the atlas must
escalate is one this parser simply detects.
- No public route. The table is never served AS a table: 67k rows would dwarf
any page using them, and the Android client consumes the same resolved JSON.
Two parser bugs found by building it, both now covered by tests: trimming a
text line before splitting ate the trailing separator on empty-text entries
and silently dropped 55,994 of 123,490 while still reporting success; and
Number('') is 0, not NaN, so a line starting with a separator imported as a
bogus cliloc 0.
Verified against the real client table (123,490 entries) and the live MariaDB:
import 663 ms, hash-gated boot no-op 14 ms, cold resolve 4.2 ms / warm 0.015 ms.
Binary and TSV imports converge on the same 67,496 rows with identical keys
(blank entries — half the table — are dropped at import). A file truncated to
half its length is refused with TRUNCATED and leaves the previous table
serving. Boot logs verified for both the import and the compressed-file
warning; neither blocks startup. All three admin routes exercised over HTTP
with a real session. 629 server tests pass; client builds clean; swagger,
routes.manifest.json and routes.guards.json regenerated.
Not covered by an automated test: the character sheet renders resolved names
in presentational React with no DOM test harness in this repo, and was not
rendered against a live linked-player profile — that needs a logged-in player
with a linked game account and a shard answering a profile RPC.
Co-Authored-By: Claude <noreply@anthropic.com>
Protocol 3.0 §7 (docs/link/v3.md). The shard publishes ~25 points/loyalty
leaderboards — Queen's Loyalty, Void Pool, the nine city loyalties, Clean Up
Britannia — and the site renders them, plus each character's own standings on
their sheet.
Server
- shard_points_boards: one row per system, keyed by the shard's PointsType
name. The top-N list stays inside `payload` — a fixed-size list read whole,
exactly like shard_governors.candidates. Normalizing into an entries table
buys nothing until something needs a per-character reverse lookup, and a
character's own standings already ride inside char.profile.
- shardIngest routes points.board to upsertPointsBoard and deliberately does
NOT log it: this is board state like guild.update, and the shard emits a
frame every time anyone's score moves a top ten.
- uoLinkSocket backfills /points through snapshot() with ingestEach rather
than a replace*: there is no points.remove and the system set is fixed, so
upserting IS the reconciliation, and a system the operator later excludes
keeps its last-known board rather than vanishing.
- GET /public/shard/points and /points/:system behind
requireFeature('leaderboards'), both projected per §3.6.1. :system is
constrained to an identifier before any query runs; 404 for a system never
published, distinct from a published board nobody has scored in (200, empty
top).
The leaderboards field rule now keys on `name`, not `characterName`
Part A pre-wired FEATURES.leaderboards.fields = { characterName: ... }, but
projectValue matches on the LITERAL JSON key and the wire key is `name`. As
written the rule was inert: an admin tightening character names would have got
no enforcement and no error — precisely the failure §3.6.1 records for the
flattened `ownerAcct` spelling. Fixed, with a test that fails if it is renamed
back, and the admin panel's FIELD_LABEL carries the meaning instead.
Client
- routes/public/Leaderboards.jsx at /site/leaderboards. A points.board frame
describes ONE system, so live frames merge over the fetched set by system
key rather than replacing it wholesale the way the ruleset does. Filter
matches board name, system key, or any ranked player — the last is what
makes it useful ("where do I appear?").
- A "Loyalty & Points" section in CharacterSheet.jsx, one edit serving both
PlayerCharacter and AdminCharacter.
- Both treat maxPoints: 0 as UNCAPPED and both fall back to humanising the
system key when nameString is null. Neither is defensive padding: on a real
shard uncapped and cliloc-only names are the majority case.
Verified end to end against the local MariaDB, the Rust sidecar, and the real
ServUO shard: backfill from /points, live SSE delivery (a board absent from the
initial fetch appearing without a reload, and an existing one updating in
place), REST reflecting the overwrite, and the gate at every rung — 200 by
default with names, names stripped but points kept at fieldRules name=staff, 403
plus dropped from /features at audience=staff, 404 when disabled. Page rendered
clean, no console errors beyond the pre-existing React Router v7 warnings.
605 server tests pass; routes.manifest.json, routes.guards.json and the OpenAPI
spec regenerated.
Co-Authored-By: Claude <noreply@anthropic.com>
Protocol 3.0 order 3 (Part C), second of two website PRs. #112 built the data
pipeline; this makes it reachable — six public routes, five admin ones, two
public pages and an admin panel. Still website-only: no plugin, no sidecar, no
new event kinds, no wire change.
The API sits at /api/v1/public/atlas, not under /public/shard. Nothing here
touches the sidecar, so the pages stay complete while the shard is down, and a
/shard prefix would imply a dependency the atlas does not have. Unlike /shard/*
it IS site-mode gated, like /posts and /wiki: a bestiary is site content.
Every route carries requireFeature('atlas') and projects its response. The atlas
feature declares no sensitive fields, so the projection is a no-op today — the
call is there because v3.md 3.6.1's rule is that the FIRST field needing a gate
should be covered by construction rather than by a retrofit.
Two bugs the UI surfaced, both fixed here:
Respawn delays were stored in the wrong unit, sometimes. XmlSpawner writes
MinDelay/MaxDelay in minutes and switches to seconds only when a delay does not
divide into whole minutes, flagging it per record with DelayInSec. A `5` means
five minutes on one spawner and five seconds on the next, both plausible, and
the pipeline stored the raw number. 170 of 6,455 stock spawners are second
flagged. The parser normalises to seconds; the API and UI carry seconds.
That exposed the hash gate as a trap. "Has the tree changed?" is the wrong
question on its own: an install whose maps never change would have kept serving
the old readings forever, because the only thing compared was the tree.
PARSER_VERSION is now stored beside the source hashes and a mismatch counts as
drift, so any future parse correction lands on the next boot.
Also renamed the detail route's spawn-point array to `spawners` — it was
`points`, which is the COUNT on the search route, so one key meant a number in
one place and an array in the other.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U7CBg11prhLimL9iHSX1bP
