docs/website/MODULE_SYSTEM.md

The Module System — design of record

Status: approved design, not yet implemented. Every decision in Part 3 has been settled with the org lead; Part 1 records what was verified against the working trees on 2026-08-10, including the places the original draft was wrong.

Goal. Turn Runic Gateway from a UO/ServUO-specific platform into a game-agnostic one. The core architecture is unchanged — sidecar → website → browser. What changes is that game-specific behaviour (routes, tables, screens, nav) leaves the core website and becomes an installable module. An operator installs the base site, installs the module for their game, and restarts.

The model is WordPress plugins, not a build system. An operator never compiles anything to deploy a module. The module's own CI publishes it prebuilt; the operator drops it in and enables it. This single constraint drives most of Part 2.

Out of scope. The sidecar's per-game protocol adapters. link/, servuo-plugins/ and installer/ are the shard side and stay independent of this work — the installer runs on the shard host and by design never contacts the website (installer/src/cli.rs:162). Also out of scope: the Android app, which gets its own plan covering module discovery and multi-server profiles; this plan only owes it the capability endpoint in §2.5.

---

Part 1 — What is actually there

1.1 The parts that are already clean

The extraction is closer to a folder move than a teardown, and that is not an assumption:

• Models. server/src/model/ holds 31 directories; exactly 8 are UO — shardAtlas/, shardClilocs/, shardEvents/, shardLinks/, shardMarket/, shardState/, shardVisibility/, uoLinkConfig/. No mixing with users/, posts/, pages/, wiki/, settings/.
• Routers. All 13 UO router/controller files are single-purpose, with no shared code: admin/shard.router.js, admin/shardAtlas|shardClilocs|shardOps|shardVisibility.controller.js, admin/uoLink.router.js + .controller.js, public/atlas.router.js + .controller.js, public/shard.router.js + .controller.js, player/shard.router.js + .controller.js.
• Mount points. router/v1/{public,admin,player}/index.js are pure mount tables that declare no routes of their own. Module mounting drops straight in with no restructuring.
• The API surface is small and knowable. The nine UO utils/ files import only four things from core: settings.model, logger, auth, pushDispatch. That is the empirical basis for §2.1 — the contract is derived from what the real code uses, not designed speculatively.

1.2 Route prefixes: one flat module prefix is impossible

A module cannot be handed a single pre-scoped router at, say, /api/v1/game/uo, because the existing UO URLs live under three different access tiers — /api/v1/public/shard/*, /api/v1/admin/shard/*, /api/v1/player/shard/* — and those URLs are protected by routeManifest.test.js and consumed by three shipped clients (SPA, Android app, Discord bot).

Resolved: a module owns a named slot inside each tier. It still only ever holds a pre-scoped express.Router() and structurally cannot reach above its mount point; it simply holds one per tier.

"mounts": {
  "public": ["/shard", "/atlas"],
  "admin":  ["/shard", "/uo-link"],
  "player": ["/shard"]
}

The loader rejects a prefix collision between two modules, or between a module and core, at registration time. That check is unavoidable; the per-request routing boundary is not left to the module's good behaviour.

1.3 There is no server-side nav list, and there must not be one

server/src/utils/navOverrides.js (lines 13–21) refuses this explicitly:

What this module cannot check, deliberately: whether a to exists. The three base NAV arrays are client constants (SiteHeader.jsx, AdminLayout.jsx, PlayerPortalLayout.jsx). Shipping a copy of them to the server would create a second source of truth for navigation that drifts the first time a route is added…

Confirmed: export const NAV lives in client/src/components/SiteHeader.jsx:23, client/src/routes/admin/AdminLayout.jsx:56, client/src/routes/player/PlayerPortalLayout.jsx:41. The server validates override shape and nothing else.

Resolved: nav registration is client-side, performed by the module's own client bundle against a core-provided registry. No server nav API is introduced, and THEMING_AND_NAV.md's override model is untouched. The resulting pipeline is:

registered defaults (core + modules) → role/feature filtering → admin overrides → rendered nav

1.4 Module nav items interleave into core groups

Appending a "UO" group is not enough. Today's UO items sit inside core groups in AdminLayout.jsx: group Moderation holds /admin/shard-ops and /admin/houses; group System holds /admin/shard, /admin/shard-visibility, /admin/shard-atlas; the unnamed footer group holds /admin/characters. MOD_PATHS (line 109) additionally hardcodes two UO paths as moderator-visible.

Resolved: nav registration takes a target group and order ({ group: 'Moderation', order: 30 }), and MOD_PATHS becomes a roles-derived computation rather than a path allowlist.

1.5 The public nav's feature-gating mechanism is itself a shard system

Ten of the sixteen entries in SiteHeader.jsx's NAV carry a feature: key (status, champs, guilds, governors, houses, ruleset, atlas, leaderboards, market), resolved by useShardFeatures() against /api/v1/public/shard/features — the shard visibility system. Extracting the module removes the provider that core's own nav filter depends on.

Resolved: core keeps a generic feature-flag context with a registerable provider; the module registers its useShardFeatures for its own namespace. No core nav item carries a feature today, so with no module installed the filter is a correct no-op.

1.6 There is no migration system to model a module migration runner on

server/db/schema.sql is a single idempotent file — 1,380 lines, 67 tables — replayed in full on every boot by ensureSchema() (src/utils/db.js:48), split on ; and executed statement by statement. Schema evolution uses ALTER TABLE … ADD COLUMN IF NOT EXISTS / MODIFY COLUMN (from line 1322). There is no version table, no runner, no migrations directory.

A module-scoped migration runner would therefore be the first migration system in the codebase, and would leave core and modules on two different schema models.

Resolved: modules ship a schema.sql fragment, replayed idempotently by the same ensureSchema() immediately after core's. Forward-only falls out for free — it is all an idempotent replay can be. Install and upgrade become the same operation. Uninstall stops the fragment being replayed; purge is a separate, explicit, destructive admin action that runs the module's purge.sql. A real migration runner, covering core and modules together, is a legitimate future workstream; it is not a prerequisite for this one.

Twenty-five of the 67 tables move with the module: the 24 shard_* tables plus uo_link_config.

1.7 Boot and shutdown is a lifecycle gap

server/src/server.js holds eight UO call sites that routes, nav and schema do not cover:

Line	Call
92	shardAtlas.refreshOnBoot()
99	shardClilocs.refreshOnBoot()
107	shardMarket.refreshDisplayNames() (conditional on the cliloc import result)
130	uoLinkSocket.start()
131	checkUoLink() — plus the whole function at 147–169
179	uoLinkSocket.stop()
180	shardBroadcast.closeAll()
7–19	five top-level requires of UO modules

Resolved: the API surface includes onBoot(ctx) and onShutdown(), each individually try/caught by the loader per §2.4.

1.8 Three core files are genuinely entangled

Everything else is a folder move. These are not:

1. src/config/notificationStreams.js — the push stream catalog. mapShardEvent() and most of STREAMS are shard-derived, and it imports PUBLIC_KINDS from utils/shardBroadcast. Push infrastructure is core; this catalog is module content. → registerNotificationStreams({ streams, mapEvent }).
2. src/utils/pushDispatch.js — core infrastructure, but fromShardEvent() (line 112) requires the shardLinks model (line 21) and mapShardEvent (line 23). → invert: publish() stays core, fromShardEvent moves into the module and calls it.
3. src/utils/announceWorker.js — the news dispatcher, with two delivery legs: Discord (core) and town crier (module, via uoLinkClient.postTownCrier, line 36). → registerAnnounceLeg({ leg, dispatch, classify }).

src/utils/newsGump.js is module-side (news → in-game gump) and moves whole.

1.9 A fourth mount shape: module routes under a core resource

router/v1/admin/users.router.js mounts usersShard.controller.js at six UO sub-paths of a core resource — /:id/shard/accounts|sales|houses|online|standing and DELETE /:id/shard/link/:account. And GET /api/v1/admin/users/:id (line 159) is itself served by usersShard.getUser, which is core semantics that ended up in the UO controller by proximity.

Resolved, two parts: (a) getUser moves back into admin.controller.js; (b) core declares a narrow extension slot on /admin/users/:id that the module mounts into, so core never learns what "shard" means and all six URLs are preserved. Only core may declare an extension slot; a module may not invent one.

1.10 The Discord bot has no UO logic

The draft listed the bot's "UO-specific event/moderation logic" as an extraction candidate. Grepping website/bot/src for uo|ultima|shard|towncrier|governor|vendor returns zero matches. The bot's only site coupling is src/site/siteApiClient.js. There is nothing to extract.

1.11 The installer is not, and will not become, the delivery path

Two independent reasons, and the decision is that website and installer stay independent:

1. The installer runs on the shard host and never contacts the website — src/cli.rs:162: "The installer never contacts your website, never deletes anything from your…". A website module is a website-host artifact.
2. Bundle is hardcoded to exactly two components. installer/src/bundle.rs declares pub link: LinkComponent and pub overlay: OverlayComponent, both non-Option, alongside a single top-level protocol: u32 and SUPPORTED_SCHEMA: u32 = 1. A third artifact type would be a schema-2 bump — and the sidecar/overlay protocol number has nothing to say about a website module anyway.

Note also that there is no SHA256SUMS trust anchor anywhere in the installer, contrary to the draft. The real model is a per-asset sha256 field inside a bundle JSON fetched anonymously over HTTPS from the bundles branch. No signatures. The shape is worth reusing; the name was wrong.

1.12 Modules must mount synchronously, from the filesystem

server/scripts/routeManifest.js:38 and server/swagger/swagger.js:29 both walk the Express stack by require-ing src/app.js with no database connection — the manifest script deliberately points the pool at a dead port. A DB-driven async loader would make module routes invisible to both, silently breaking the frozen-URL-surface test and shipping undocumented routes.

Resolved: the filesystem is the mounting source of truth. app.js synchronously scans modules/*/module.json at require time and mounts what it finds. The installed_modules row carries state and metadata (version, installed-at, startup_failed reason, admin enable/disable) and is reconciled against the filesystem once the DB is up. A module disabled in the DB is skipped by a one-line dispatch guard rather than being unmounted, so the URL surface stays deterministic and generatable.

1.13 The client seam is a route registry, not an admin-panel loader

client/src/App.jsx is a flat 235-line static route table, and UO routes appear in all three areas — public (/site/shard, /site/shard/activity, /site/governors, /site/houses, /site/atlas, /site/atlas/:slug, /site/market, /site/market/vendors/:serial, plus champs, guilds, rules, leaderboards), admin (shard, shard-visibility, shard-atlas, shard-ops, houses, characters, characters/:serial) and player. Nav is one consumer of that registry, not the mechanism itself.

1.14 Production is a prebuilt, pull-only image — and that is the binding constraint

website/Dockerfile bakes client/dist at image build time, and docker-compose.yml has no build: stanza at all (deliberately: "a production host can only ever pull, never accidentally build"). Combined with the requirement that an operator must never build anything to deploy a module, this rules out build-time inclusion of module client code, which the draft had as its default.

It also rules out import maps as the shared-dependency mechanism: config/csp.js:49 sets 'script-src': ["'self'"] with no 'unsafe-inline', and an import map must be an inline <script type="importmap">.

Resolved — see §2.6. The path that survives all three constraints is: the module's CI ships a prebuilt ESM chunk, core hands it React through a global rather than an import map, and htmlShell.js:111 injects a same-origin <script type="module" src>, which 'self' already allows.

---

Part 2 — The plan

2.0 Scope and non-goals

Out of scope unless Phase 1 turns up a concrete reason otherwise: hot module reload; sandboxing beyond the boundary stated in §2.2; inter-module dependency resolution; a module marketplace or discovery UI; automatic data rollback beyond the forward-only model in §1.6. Install and uninstall require a controlled restart — never a rebuild.

Added: no installer changes at all (§1.11), and no Android changes in this workstream beyond the one consequence recorded in §2.8.

2.1 The API surface, derived from real dependencies

Taken from what the UO code actually imports today. Nothing speculative — if module-uo does not use it, it is not on the list.

Server — the ctx handed to a module's entry point

Member	Backed by	Why it is here
ctx.db	utils/db (query, pool)	every *.db.js
ctx.settings	model/settings/settings.model	shardIngest.js:20
ctx.log(namespace)	utils/logger	all nine UO utils
ctx.auth	utils/auth	shardVisibility.js:26
ctx.push.publish()	utils/pushDispatch	shardIngest.js:22
ctx.secretBox	utils/secretBox	uoLinkConfig model
ctx.middleware	requireAuth, requireRole, siteMode, validate	every UO router
ctx.uploads	admin/imageUpload.js	atlas art import
ctx.posts	model/posts/posts.model	newsGump.js, announce legs

Server — what a module registers

registerRoutes(mounts) (§1.2) · registerExtension(slot, router) (§1.9) · registerNotificationStreams({ streams, mapEvent }) (§1.8) · registerAnnounceLeg({ leg, dispatch, classify }) (§1.8) · onBoot(ctx) / onShutdown() (§1.7).

Client — what a module registers

registerRoutes({ public, admin, player }) (§1.13) · registerNav({ nav, group, order, feature }) (§1.3, §1.4) · registerFeatureProvider(namespace, hook) (§1.5).

The acceptance test for the whole contract: module-uo runs with zero require/import reaching outside its own directory. Any gap extends the surface before extraction proceeds.

2.2 What the module boundary is, and is not

A module runs in the same Node process with full access. The boundary is a code-organisation and distribution boundary, not a security boundary — which is fine for a self-hosted operator installing software they chose, the same trust category as running its schema fragment. What makes it worth having is that modules interact with core through a defined surface, so a core refactor cannot silently break a module. Hence the zero-internal-imports rule above, enforced in CI rather than by review.

2.3 Module packaging — one repo, one bundle

RunicGateway/module-uo, GPL-3.0-or-later, CI shaped like the other repos. Server and client halves live side by side and version together, so a route and the screen that calls it can never be mismatched:

RunicGateway/module-uo
  module.json          id, version, coreApi range, mounts, extensions
  server/              routers, controllers, models, utils
  server/db/schema.sql fragment replayed by ensureSchema()
  server/db/purge.sql  destructive, only ever run by an explicit purge
  client/src/          route components, nav registrations, feature provider
  client/dist/         PREBUILT ESM chunk, published by module CI

Release artifact: module-uo-<version>.tar.gz plus a manifest carrying its sha256.

module.json declares a coreApi semver range, checked at boot against a MODULE_API_VERSION constant in core; a mismatch fails loudly rather than silently. This is a separate number from PROTOCOL_VERSION, which versions the shard wire and says nothing about a website module.

2.4 The module state machine

installed → enabled → started, with disabled and startup_failed as recoverable states.

A module that fails to load must never take the site down. The loader catches failures across the module's entire lifecycle — require, schema fragment, router construction, registration calls, onBoot — not merely those that surface after a router object was returned. Any failure at any point marks that one module startup_failed, records the reason, and the site comes up with that module's routes and nav absent. startup_failed is recoverable from the admin panel — disable, retry, or roll back to the previous version — with no shell access to the box.

2.5 Install, uninstall, purge

Modules live on a mounted volume, not in the image — the same treatment uploads already gets in docker-compose.yml. That is what makes the WordPress model work against a pull-only image.

Install: admin selects the module → bundle downloaded from the module repo's release and verified against its sha256 → unpacked into modules/<id>/ on the volume → installed_modules row written → restart. On boot the loader scans the filesystem (§1.12), validates prefixes, mounts, replays the schema fragment, runs onBoot, and each module reaches started or startup_failed.

Nothing is compiled at any point. The operator restarts; they never build.

Uninstall (default, non-destructive): row set to disabled, directory removed, restart. The module's tables and data are retained. Purge is a separate, explicit, destructive action that runs purge.sql; it is never bundled into uninstall.

Surfaces: the admin panel, and the Docker environment under website/ — a declarative module set resolved at container start from the mounted volume, so a compose-managed host is not driven by clicking. Both paths write the same installed_modules row and neither requires a build step.

2.6 How the client half loads

This is the piece §1.14 constrains hardest. Three requirements had to hold at once: the operator builds nothing, production pulls a prebuilt image, and script-src 'self' forbids inline script.

1. The module's CI builds its client half with Vite in library mode, declaring react, react-dom and react-router-dom as externals. The module never bundles its own React — there is exactly one React instance, owned by core.
2. Core exposes the shared dependencies on a global before mount — window.__rg = { react, reactDom, router, registry } — and the module's externals resolve to it. A global, not an import map, precisely because an import map must be inline and CSP forbids that.
3. htmlShell.js injects the module's entry script. It already rewrites </head> (utils/htmlShell.js:111), so this is an extension of a working mechanism, not a new one. The tag is <script type="module" src="/modules/uo/entry.js"> — same-origin, so 'self' passes with no nonce and no inline.
4. The SPA reads /api/v1/public/modules to learn what to load, then registers routes, nav and its feature provider through window.__rg.registry.

Phase 1 prototypes exactly this before anything is committed to it (§2.7).

2.7 Phases

Phase 1 — API contract + spike (blocking). Merge this document. Write the contract at docs/website/MODULE_API.md. Then a throwaway spike on an unmerged branch moving /api/v1/public/atlas/* behind the proposed surface — the smallest honest test: five routes, DB-backed, no sidecar, no SSE, one boot hook. The spike must also prove the §2.6 chunk load end to end, since that is the highest-risk decision in the plan. Exit criteria: no internal-file imports, npm run routes:manifest produces a zero-line diff, and the chunk loads under the enforced CSP.

Phase 2 — Core scaffolding, no behaviour change. One PR each, in order:

1. installed_modules table + the §2.4 state machine.
2. src/modules/loader.js — synchronous filesystem scan, manifest validation, prefix-collision rejection, per-module try/catch across the whole load path, mounting into the tier routers.
3. ensureSchema() extended to replay module fragments after core's.
4. The three de-entanglement registries (§1.8), with core still the only registrant.
5. Boot/shutdown hook dispatch in server.js, likewise.
6. GET /api/v1/public/modules — installed ids, versions and capabilities, shaped like the existing branding/site-settings endpoint. The SPA needs it to know what to load; the Android plan consumes the same endpoint.
7. Client src/modules/registry.js, the window.__rg shared-dependency global, and the htmlShell script injection — empty registry, no visible change.
8. MOD_PATHS → roles-derived (§1.4); the generic feature-provider seam (§1.5).
9. docker-compose.yml gains the modules volume.

Exit criterion: routes.manifest.json diff is zero lines and every existing test passes. If Phase 2 changes one URL, it is wrong.

Phase 3 — Extract module-uo. Moves out of website/: the 8 model directories and their 25 tables; the nine UO utils/ files plus newsGump.js; the 13 router/controller files; scripts/importSpawnAtlas.js and db/spawnAtlas.art.json; usersShard.controller.js minus getUser (§1.9); the shard-derived half of notificationStreams.js and the town-crier leg of announceWorker.js; and on the client, roughly twenty route components, their nav registrations and useShardFeatures.

Acceptance, all four required:

1. Zero UO identifiers in core — no shard, uoLink, cliloc, atlas or towncrier outside modules/. Enforced by a CI grep test, not by review.
2. Zero internal-file imports from module-uo into core.
3. routes.manifest.json API diff is zero lines, except the deliberate GET /admin/users/:id ownership move, which changes no URL. After extraction the core manifest no longer contains UO routes — module-uo generates and freezes its own in its own repo.
4. A written module-rust dry run — manifest, mounts, nav entries, one notification stream — not implemented, to prove the contract generalises before more is built on it.

Phase 4 — Delivery. The admin-panel Modules screen (install, enable, disable, retry, purge, startup_failed with its recorded reason) and the Docker-environment path from §2.5. Deliberately last, so loader, packaging, schema and chunk-loading problems are not all being debugged at once.

2.8 SPA URL namespacing — a deliberate break

Decision: module pages are namespaced, and old paths are not redirected. The site is not public yet, so bookmarks, inbound links and configured nav overrides carry no real weight. This buys a visible boundary in the URL rather than a hidden one.

The rule is that a module owns one path segment wherever it appears:

Today	After
/site/shard, /site/atlas, /site/market, /site/governors, …	/uo/shard, /uo/atlas, /uo/market, /uo/governors, …
/admin/shard-ops, /admin/shard, /admin/shard-visibility	/admin/uo/shard-ops, /admin/uo/link, /admin/uo/visibility
player shard screens	/player/uo/…

API URLs are not affected — they keep their exact paths per §1.2, so the Android app and the Discord bot need no change for the API.

Two consequences, both accepted:

• Saved nav-override rows are keyed by to (utils/navOverrides.js), so any stored nav_public / nav_admin / nav_player customisation stops applying and must be redone. No migration is written.
• android-app/.../ui/navigation/NavPaths.kt maps SPA paths to native screens and holds ten /site/* constants that will no longer resolve. That is one small Android PR, folded into the separate Android module plan. App Links verification itself is unaffected — the manifest's intent filters only cover /mobile/callback and auth/callback.

2.9 Process obligations

Every server-side PR runs npm run swagger, npm run routes:manifest (the diff is reviewed, not merely regenerated) and npm test, and carries a matching edit to BACKEND_DESIGN.md. Module documentation aggregates in this repo under docs/modules/<id>/ rather than living in module repos. Conventional Commits, the AI-disclosure trailer, branches cut from an up-to-date main.

---

Part 3 — Settled decisions

#	Decision	Where
1	Modules ship idempotent schema.sql fragments; no migration runner is built	§1.6
2	Website and installer stay independent; delivery is website-side only	§1.11, §2.5
3	Core declares an extension slot on /admin/users/:id; all six URLs preserved	§1.9
4	Phase 1 spike targets /api/v1/public/atlas/*	§2.7
5	Install surfaces: admin panel and the Docker environment; never a build step	§2.5
6	One repo, one bundle — server and client halves version together	§2.3
7	Android app is a separate plan; core owes it /api/v1/public/modules	§2.5, §2.7
8	SPA pages namespaced: /uo/*, /admin/uo/*, /player/uo/*	§2.8
9	Clean break — no redirects, no nav-override migration; site is not public yet	§2.8
10	Client half loads as a prebuilt ESM chunk with React shared via a core global	§2.6