Slice 0 built the module bundle skeleton against the contract and found that §3.6 does not work as written. It shows Rollup's `external` alongside the resolve aliases, and the two do not compose: Rollup asks `external` BEFORE Vite's alias resolver runs, so a specifier in both is marked external and never aliased. The chunk then emits bare `import "react"`, which no browser can resolve without an import map, and CSP forbids the inline script an import map has to be. It built cleanly and emitted exactly that. §3.6 is corrected: alias only, `external` empty, with the alias table shown in full because the anchoring is what stops `react` also capturing `react/jsx-runtime`. What `external` was guarding -- a missed alias welding a second React into the chunk -- moves to a resolution-time build plugin, and two properties of that plugin are now contract because both were wrong first: it hooks `transform` rather than `load` (first-wins, so it never ran), and its forbidden-package list is stated rather than derived from the alias list (deriving it means deleting an alias also deletes the guard). Also records slice 0's outcome in §2.7.1, including the finding that generalises past this repo: the boundary check failed on its own documentation, because the comments describing what it catches are written in the syntax it catches. Slice 8's §5.2 grep has the same problem waiting for it. And the loader skips a SYMLINKED module directory silently, which is the first thing to check when a module fails to appear locally. Co-Authored-By: Claude <noreply@anthropic.com>
991 lines
67 KiB
Markdown
991 lines
67 KiB
Markdown
# The Module System — design of record
|
||
|
||
**Status:** approved design, **in implementation** — Phase 2's core scaffolding is landing on the
|
||
website `edge` branch, PRs 1–7 of 9 done (§2.7 tracks what each settled). Every decision in Part 3
|
||
has been settled with the org lead; Part 1 records what was verified against the working trees on
|
||
2026-08-10, including the places the original draft was wrong.
|
||
|
||
**The normative contract is [`MODULE_API.md`](MODULE_API.md)** (Phase 1). This document decides what
|
||
the module system *is*; that one decides exactly what a module may call. Where the two differ, that
|
||
one wins — its Part 6 lists every place it amends this document.
|
||
|
||
**Goal.** Turn Runic Gateway from a UO/ServUO-specific platform into a game-agnostic one. The core
|
||
architecture is unchanged — sidecar → website → browser. What changes is that game-specific
|
||
behaviour (routes, tables, screens, nav) leaves the core website and becomes an installable
|
||
**module**. An operator installs the base site, installs the module for their game, and restarts.
|
||
|
||
**The model is WordPress plugins, not a build system.** An operator never compiles anything to
|
||
deploy a module. The module's own CI publishes it prebuilt; the operator drops it in and enables it.
|
||
This single constraint drives most of Part 2.
|
||
|
||
**Out of scope.** The sidecar's per-game protocol adapters. `link/`, `servuo-plugins/` and
|
||
`installer/` are the shard side and stay independent of this work — the installer runs on the shard
|
||
host and by design never contacts the website (`installer/src/cli.rs:162`). Also out of scope: the
|
||
Android app, which gets its own plan covering module discovery and multi-server profiles; this plan
|
||
only owes it the capability endpoint in §2.5.
|
||
|
||
---
|
||
|
||
## Part 1 — What is actually there
|
||
|
||
### 1.1 The parts that are already clean
|
||
|
||
The extraction is closer to a folder move than a teardown, and that is not an assumption:
|
||
|
||
- **Models.** `server/src/model/` holds 31 directories; exactly 8 are UO — `shardAtlas/`,
|
||
`shardClilocs/`, `shardEvents/`, `shardLinks/`, `shardMarket/`, `shardState/`, `shardVisibility/`,
|
||
`uoLinkConfig/`. No mixing with `users/`, `posts/`, `pages/`, `wiki/`, `settings/`.
|
||
- **Routers.** All 13 UO router/controller files are single-purpose, with no shared code:
|
||
`admin/shard.router.js`, `admin/shardAtlas|shardClilocs|shardOps|shardVisibility.controller.js`,
|
||
`admin/uoLink.router.js` + `.controller.js`, `public/atlas.router.js` + `.controller.js`,
|
||
`public/shard.router.js` + `.controller.js`, `player/shard.router.js` + `.controller.js`.
|
||
- **Mount points.** `router/v1/{public,admin,player}/index.js` are pure mount tables that declare no
|
||
routes of their own. Module mounting drops straight in with no restructuring.
|
||
- **The API surface is small and knowable.** The nine UO `utils/` files import only four things from
|
||
core: `settings.model`, `logger`, `auth`, `pushDispatch`. That is the empirical basis for §2.1 —
|
||
the contract is derived from what the real code uses, not designed speculatively.
|
||
|
||
### 1.2 Route prefixes: one flat module prefix is impossible
|
||
|
||
A module cannot be handed a single pre-scoped router at, say, `/api/v1/game/uo`, because the existing
|
||
UO URLs live under three different access tiers — `/api/v1/public/shard/*`,
|
||
`/api/v1/admin/shard/*`, `/api/v1/player/shard/*` — and those URLs are protected by
|
||
`routeManifest.test.js` and consumed by three shipped clients (SPA, Android app, Discord bot).
|
||
|
||
**Resolved:** a module owns a *named slot inside each tier*. It still only ever holds a pre-scoped
|
||
`express.Router()` and structurally cannot reach above its mount point; it simply holds one per tier.
|
||
|
||
```json
|
||
"mounts": {
|
||
"public": ["/shard", "/atlas"],
|
||
"admin": ["/shard", "/uo-link"],
|
||
"player": ["/shard"]
|
||
}
|
||
```
|
||
|
||
The loader rejects a prefix collision between two modules, or between a module and core, at
|
||
registration time. That check is unavoidable; the per-request routing boundary is not left to the
|
||
module's good behaviour.
|
||
|
||
### 1.3 There is no server-side nav list, and there must not be one
|
||
|
||
`server/src/utils/navOverrides.js` (lines 13–21) refuses this explicitly:
|
||
|
||
> **What this module cannot check, deliberately: whether a `to` exists.** The three base NAV arrays
|
||
> are client constants (SiteHeader.jsx, AdminLayout.jsx, PlayerPortalLayout.jsx). Shipping a copy of
|
||
> them to the server would create a second source of truth for navigation that drifts the first time
|
||
> a route is added…
|
||
|
||
Confirmed: `export const NAV` lives in `client/src/components/SiteHeader.jsx:23`,
|
||
`client/src/routes/admin/AdminLayout.jsx:56`, `client/src/routes/player/PlayerPortalLayout.jsx:41`.
|
||
The server validates override *shape* and nothing else.
|
||
|
||
**Resolved:** nav registration is **client-side**, performed by the module's own client bundle
|
||
against a core-provided registry. No server nav API is introduced, and
|
||
`THEMING_AND_NAV.md`'s override model is untouched. The resulting pipeline is:
|
||
|
||
> registered defaults (core + modules) → role/feature filtering → admin overrides → rendered nav
|
||
|
||
### 1.4 Module nav items interleave into *core* groups
|
||
|
||
Appending a "UO" group is not enough. Today's UO items sit inside core groups in `AdminLayout.jsx`:
|
||
group **Moderation** holds `/admin/shard-ops` and `/admin/houses`; group **System** holds
|
||
`/admin/shard`, `/admin/shard-visibility`, `/admin/shard-atlas`; the unnamed footer group holds
|
||
`/admin/characters`. `MOD_PATHS` (line 109) additionally hardcodes two UO paths as
|
||
moderator-visible.
|
||
|
||
**Resolved:** nav registration takes a target group and order (`{ group: 'Moderation', order: 30 }`),
|
||
and `MOD_PATHS` becomes a `roles`-derived computation rather than a path allowlist.
|
||
|
||
Built in Phase 2 PR 8. Two things it turned up that this section did not predict. The interleave has
|
||
to happen **before** the admin-override merge and not after it, because that merge drops any `to`
|
||
its base array does not declare — appending module rows afterwards would leave them uneditable in
|
||
Admin → Navigation, which today's UO rows are not. And there was a **third** hardcoded list: the
|
||
redirect that confines a moderator checked three path prefixes, while `MOD_PATHS` listed five paths,
|
||
and they disagreed about `/admin/houses` — a moderator who clicked Houses in their own sidebar was
|
||
bounced straight back to Moderation. One derivation cannot disagree with itself.
|
||
|
||
### 1.5 The public nav's feature-gating mechanism is itself a shard system
|
||
|
||
Ten of the sixteen entries in `SiteHeader.jsx`'s NAV carry a `feature:` key (`status`, `champs`,
|
||
`guilds`, `governors`, `houses`, `ruleset`, `atlas`, `leaderboards`, `market`), resolved by
|
||
`useShardFeatures()` against `/api/v1/public/shard/features` — the shard visibility system.
|
||
Extracting the module removes the provider that core's own nav filter depends on.
|
||
|
||
**Resolved:** core keeps a generic feature-flag context with a **registerable provider**; the module
|
||
registers its `useShardFeatures` for its own namespace. No core nav item carries a `feature` today,
|
||
so with no module installed the filter is a correct no-op.
|
||
|
||
Built in Phase 2 PR 8, and core registers into it **now** rather than at extraction: `useShardFlags`
|
||
goes in under the owner id `core`, so the ten rows above are already resolved through the seam and
|
||
`SiteHeader` runs one mechanism instead of two. Which provider answers a row is decided by the
|
||
module that registered it, not by a prefix parsed out of the flag name, so those ten keep the exact
|
||
strings they carry today and Phase 3 moves them without a rename.
|
||
|
||
### 1.6 There is no migration system to model a module migration runner on
|
||
|
||
`server/db/schema.sql` is a single idempotent file — 1,380 lines, 67 tables — replayed in full on
|
||
every boot by `ensureSchema()` (`src/utils/db.js:48`), split on `;` and executed statement by
|
||
statement. Schema evolution uses `ALTER TABLE … ADD COLUMN IF NOT EXISTS` / `MODIFY COLUMN`
|
||
(from line 1322). There is **no version table, no runner, no migrations directory**.
|
||
|
||
A module-scoped migration runner would therefore be the *first* migration system in the codebase,
|
||
and would leave core and modules on two different schema models.
|
||
|
||
**Resolved:** modules ship a `schema.sql` fragment, replayed idempotently by the same
|
||
`ensureSchema()` immediately after core's. Forward-only falls out for free — it is all an idempotent
|
||
replay can be. Install and upgrade become the same operation. Uninstall stops the fragment being
|
||
replayed; **purge** is a separate, explicit, destructive admin action that runs the module's
|
||
`purge.sql`. A real migration runner, covering core *and* modules together, is a legitimate future
|
||
workstream; it is not a prerequisite for this one.
|
||
|
||
Twenty-seven of the tables move with the module: the 26 `shard_*` tables plus `uo_link_config`.
|
||
(This said 25 when written; the working tree was recounted in Phase 1 — see
|
||
[`MODULE_API.md`](MODULE_API.md) §6.4.)
|
||
|
||
### 1.7 Boot and shutdown is a lifecycle gap
|
||
|
||
`server/src/server.js` holds eight UO call sites that routes, nav and schema do not cover:
|
||
|
||
| Line | Call |
|
||
| --- | --- |
|
||
| 92 | `shardAtlas.refreshOnBoot()` |
|
||
| 99 | `shardClilocs.refreshOnBoot()` |
|
||
| 107 | `shardMarket.refreshDisplayNames()` (conditional on the cliloc import result) |
|
||
| 130 | `uoLinkSocket.start()` |
|
||
| 131 | `checkUoLink()` — plus the whole function at 147–169 |
|
||
| 179 | `uoLinkSocket.stop()` |
|
||
| 180 | `shardBroadcast.closeAll()` |
|
||
| 7–19 | five top-level `require`s of UO modules |
|
||
|
||
**Resolved:** the API surface includes `onBoot(ctx)` and `onShutdown()`, each individually
|
||
try/caught by the loader per §2.4.
|
||
|
||
### 1.8 Three core files are genuinely entangled
|
||
|
||
Everything else is a folder move. These are not:
|
||
|
||
1. **`src/config/notificationStreams.js`** — the push stream catalog. `mapShardEvent()` and most of
|
||
`STREAMS` are shard-derived, and it imports `PUBLIC_KINDS` from `utils/shardBroadcast`. Push
|
||
*infrastructure* is core; this *catalog* is module content.
|
||
→ `registerNotificationStreams(streams)`.
|
||
2. **`src/utils/pushDispatch.js`** — core infrastructure, but `fromShardEvent()` (line 112) requires
|
||
the `shardLinks` model (line 21) and `mapShardEvent` (line 23).
|
||
→ invert: `publish()` stays core, `fromShardEvent` moves into the module and calls it.
|
||
3. **`src/utils/announceWorker.js`** — the news dispatcher, with two delivery legs: Discord (core)
|
||
and town crier (module, via `uoLinkClient.postTownCrier`, line 36).
|
||
→ `registerAnnounceLeg({ leg, label, dispatch, classify })`.
|
||
|
||
`src/utils/newsGump.js` is module-side (news → in-game gump) and moves whole.
|
||
|
||
**Done in Phase 2 PR 4**, with core still the only registrant — the registries are
|
||
`src/modules/registries.js` and core goes through them by the same door a module will
|
||
(`registerCore()`, called explicitly from `app.js` before `modules.load()`). What each of the three
|
||
became:
|
||
|
||
1. Split in two. `config/coreStreams.js` is core's one stream (`news.post`, produced by the website's
|
||
own posts path); `config/shardStreams.js` is the other seven plus `mapShardEvent` and the
|
||
public-safety filter, and moves to module-uo whole. `registerNotificationStreams` lost its
|
||
`mapEvent` half — see [`MODULE_API.md`](MODULE_API.md) §2.4 for why that was a leftover, and what
|
||
follows for the public/personal split.
|
||
2. Inverted. `pushDispatch.js` is `publish` + `isAllowedEndpoint` and nothing else;
|
||
`utils/shardPush.js` holds `fromShardEvent` and is what `shardIngest` now calls.
|
||
3. Legs became registrations, and per-leg **rows**. The `towncrier_*` / `discord_*` column groups on
|
||
`announce_jobs` could never have held a module's leg — a module cannot `ALTER` a core table — so
|
||
they became `announce_job_legs`, backfilled and dropped in the same idempotent replay. The worker
|
||
no longer contains the word "towncrier": it iterates whatever is registered.
|
||
|
||
The residue in core is a one-time backfill block in `schema.sql`, deletable once every deployment has
|
||
booted it, and the two lines of `registerCore()` that Phase 3 turns into module-uo's `register()`.
|
||
|
||
### 1.9 A fourth mount shape: module routes under a core resource
|
||
|
||
`router/v1/admin/users.router.js` mounts `usersShard.controller.js` at six UO sub-paths of a **core**
|
||
resource — `/:id/shard/accounts|sales|houses|online|standing` and `DELETE /:id/shard/link/:account`.
|
||
And `GET /api/v1/admin/users/:id` (line 159) is itself served by `usersShard.getUser`, which is core
|
||
semantics that ended up in the UO controller by proximity.
|
||
|
||
**Resolved, two parts:** (a) `getUser` moves back into `admin.controller.js`; (b) core declares a
|
||
narrow **extension slot** on `/admin/users/:id` that the module mounts into, so core never learns
|
||
what "shard" means and all six URLs are preserved. Only core may declare an extension slot; a module
|
||
may not invent one.
|
||
|
||
**Both done in Phase 2 PR 4**, with core filling its own slot: the six paths are
|
||
`router/v1/admin/usersShard.router.js`, registered into `admin.users.detail` by `registerCore()`, and
|
||
Phase 3 changes the registrant rather than the routes. The slot's router is created at declare time
|
||
and filled later, because `users.router.js` is required while `app.js` is still being built. It is
|
||
mounted **last** on the resource, so core wins any path conflict by first-match.
|
||
|
||
One consequence was not foreseen and is worth the warning: **a slot is invisible to static analysis.**
|
||
There is no literal mount for `swagger-autogen` to follow, so the move silently deleted all six paths
|
||
from `swagger-output.json` while printing `Success`. The OpenAPI build now merges a generated
|
||
fragment per filled slot — [`MODULE_API.md`](MODULE_API.md) §6.6.
|
||
|
||
### 1.10 The Discord bot has no UO logic
|
||
|
||
The draft listed the bot's "UO-specific event/moderation logic" as an extraction candidate. Grepping
|
||
`website/bot/src` for `uo|ultima|shard|towncrier|governor|vendor` returns **zero matches**. The bot's
|
||
only site coupling is `src/site/siteApiClient.js`. There is nothing to extract.
|
||
|
||
### 1.11 The installer is not, and will not become, the delivery path
|
||
|
||
Two independent reasons, and the decision is that website and installer stay independent:
|
||
|
||
1. **The installer runs on the shard host and never contacts the website** — `src/cli.rs:162`: *"The
|
||
installer never contacts your website, never deletes anything from your…"*. A website module is a
|
||
website-host artifact.
|
||
2. **`Bundle` is hardcoded to exactly two components.** `installer/src/bundle.rs` declares
|
||
`pub link: LinkComponent` and `pub overlay: OverlayComponent`, both non-`Option`, alongside a
|
||
single top-level `protocol: u32` and `SUPPORTED_SCHEMA: u32 = 1`. A third artifact type would be
|
||
a schema-2 bump — and the sidecar/overlay protocol number has nothing to say about a website
|
||
module anyway.
|
||
|
||
Note also that **there is no SHA256SUMS trust anchor** anywhere in the installer, contrary to the
|
||
draft. The real model is a per-asset `sha256` field inside a bundle JSON fetched anonymously over
|
||
HTTPS from the `bundles` branch. No signatures. The *shape* is worth reusing; the name was wrong.
|
||
|
||
### 1.12 Modules must mount synchronously, from the filesystem
|
||
|
||
`server/scripts/routeManifest.js:38` and `server/swagger/swagger.js:29` both walk the Express stack by
|
||
`require`-ing `src/app.js` **with no database connection** — the manifest script deliberately points
|
||
the pool at a dead port. A DB-driven async loader would make module routes invisible to both,
|
||
silently breaking the frozen-URL-surface test and shipping undocumented routes.
|
||
|
||
**Resolved:** the **filesystem is the mounting source of truth.** `app.js` synchronously scans
|
||
`modules/*/module.json` at require time and mounts what it finds. The `installed_modules` row carries
|
||
state and metadata (version, installed-at, `startup_failed` reason, admin enable/disable) and is
|
||
reconciled against the filesystem once the DB is up. A module disabled in the DB is skipped by a
|
||
one-line dispatch guard rather than being unmounted, so the URL surface stays deterministic and
|
||
generatable.
|
||
|
||
### 1.13 The client seam is a route registry, not an admin-panel loader
|
||
|
||
`client/src/App.jsx` is a flat 235-line static route table, and UO routes appear in all three areas —
|
||
public (`/site/shard`, `/site/shard/activity`, `/site/governors`, `/site/houses`, `/site/atlas`,
|
||
`/site/atlas/:slug`, `/site/market`, `/site/market/vendors/:serial`, plus champs, guilds, rules,
|
||
leaderboards), admin (`shard`, `shard-visibility`, `shard-atlas`, `shard-ops`, `houses`,
|
||
`characters`, `characters/:serial`) and player. Nav is one consumer of that registry, not the
|
||
mechanism itself.
|
||
|
||
### 1.14 Production is a prebuilt, pull-only image — and that is the binding constraint
|
||
|
||
`website/Dockerfile` bakes `client/dist` at image build time, and `docker-compose.yml` has no
|
||
`build:` stanza at all (deliberately: *"a production host can only ever pull, never accidentally
|
||
build"*). Combined with the requirement that **an operator must never build anything to deploy a
|
||
module**, this rules out build-time inclusion of module client code, which the draft had as its
|
||
default.
|
||
|
||
It also rules out import maps as the shared-dependency mechanism: `config/csp.js:49` sets
|
||
`'script-src': ["'self'"]` with no `'unsafe-inline'`, and an import map must be an inline
|
||
`<script type="importmap">`.
|
||
|
||
**Resolved** — see §2.6. The path that survives all three constraints is: the module's CI ships a
|
||
**prebuilt ESM chunk**, core hands it React through a **global** rather than an import map, and
|
||
`htmlShell.js` injects a **same-origin** `<script type="module" src>`, which `'self'` already
|
||
allows. Verified in a browser against the enforced policy in Phase 2 PR 7, not only reasoned about
|
||
([`MODULE_API.md`](MODULE_API.md) §7.7).
|
||
|
||
---
|
||
|
||
## Part 2 — The plan
|
||
|
||
### 2.0 Scope and non-goals
|
||
|
||
Out of scope unless Phase 1 turns up a concrete reason otherwise: hot module reload; sandboxing
|
||
beyond the boundary stated in §2.2; inter-module dependency resolution; a module marketplace or
|
||
discovery UI; automatic data rollback beyond the forward-only model in §1.6. Install and uninstall
|
||
require a controlled **restart** — never a rebuild.
|
||
|
||
Added: **no installer changes at all** (§1.11), and **no Android changes in this workstream** beyond
|
||
the one consequence recorded in §2.8.
|
||
|
||
### 2.1 The API surface, derived from real dependencies
|
||
|
||
Taken from what the UO code actually imports today. Nothing speculative — if module-uo does not use
|
||
it, it is not on the list.
|
||
|
||
**Server — the `ctx` handed to a module's entry point**
|
||
|
||
| Member | Backed by | Why it is here |
|
||
| --- | --- | --- |
|
||
| `ctx.db` | `utils/db` (`query`, `pool`) | every `*.db.js` |
|
||
| `ctx.settings` | `model/settings/settings.model` | `shardIngest.js:20` |
|
||
| `ctx.log(namespace)` | `utils/logger` | all nine UO utils |
|
||
| `ctx.auth` | `utils/auth` | `shardVisibility.js:26` |
|
||
| `ctx.push.publish()` | `utils/pushDispatch` | `shardIngest.js:22` |
|
||
| `ctx.secretBox` | `utils/secretBox` | `uoLinkConfig` model |
|
||
| `ctx.middleware` | `requireAuth`, `requireRole`, `siteMode`, `validate` | every UO router |
|
||
| `ctx.uploads` | `admin/imageUpload.js` | atlas art import |
|
||
| `ctx.posts` | `model/posts/posts.model` | `newsGump.js`, announce legs |
|
||
|
||
**Server — what a module registers**
|
||
|
||
`registerRoutes(mounts)` (§1.2) · `registerExtension(slot, router)` (§1.9) ·
|
||
`registerNotificationStreams({ streams, mapEvent })` (§1.8) ·
|
||
`registerAnnounceLeg({ leg, dispatch, classify })` (§1.8) · `onBoot(ctx)` / `onShutdown()` (§1.7).
|
||
|
||
**Client — what a module registers**
|
||
|
||
`registerRoutes({ public, admin, player })` (§1.13) ·
|
||
`registerNav({ nav, group, order, feature })` (§1.3, §1.4) ·
|
||
`registerFeatureProvider(namespace, hook)` (§1.5).
|
||
|
||
**The acceptance test for the whole contract:** `module-uo` runs with **zero** `require`/`import`
|
||
reaching outside its own directory. Any gap extends the surface *before* extraction proceeds.
|
||
|
||
### 2.2 What the module boundary is, and is not
|
||
|
||
A module runs in the same Node process with full access. The boundary is a **code-organisation and
|
||
distribution boundary, not a security boundary** — which is fine for a self-hosted operator
|
||
installing software they chose, the same trust category as running its schema fragment. What makes
|
||
it worth having is that modules interact with core through a *defined* surface, so a core refactor
|
||
cannot silently break a module. Hence the zero-internal-imports rule above, enforced in CI rather
|
||
than by review.
|
||
|
||
### 2.3 Module packaging — one repo, one bundle
|
||
|
||
**`RunicGateway/Module-uo`** — `https://gitea.whitlocktech.com/RunicGateway/Module-uo.git`, note the
|
||
capital `M`, matching `Android-app`'s casing rather than the lowercase directory name. The repo
|
||
exists but is **empty** as of 2026-08-10: no branches, no initial commit. Its first commit needs the
|
||
usual scaffolding — `README.md`, `LICENSE.md` (GPL-3.0-or-later), `CONTRIBUTING.md` with the
|
||
AI-disclosure clause, the PR template, and CI.
|
||
|
||
Server and client halves live side by side and version together, so a route and the screen that
|
||
calls it can never be mismatched:
|
||
|
||
```
|
||
RunicGateway/Module-uo
|
||
module.json id, version, coreApi range, mounts, extensions
|
||
server/ routers, controllers, models, utils
|
||
server/db/schema.sql fragment replayed by ensureSchema()
|
||
server/db/purge.sql destructive, only ever run by an explicit purge
|
||
client/src/ route components, nav registrations, feature provider
|
||
client/dist/ PREBUILT ESM chunk, published by module CI
|
||
```
|
||
|
||
Release artifact: `module-uo-<version>.tar.gz` plus a manifest carrying its `sha256`.
|
||
|
||
The module's **id** is `uo` — that is what appears in `module.json`, in `installed_modules`, in the
|
||
`modules/<id>/` path and in the URL segment. `Module-uo` is the repository; `module-uo` elsewhere in
|
||
this document names the module and its artifact, not the repo.
|
||
|
||
`module.json` declares a `coreApi` semver range, checked at boot against a `MODULE_API_VERSION`
|
||
constant in core; a mismatch fails **loudly** rather than silently. This is a separate number from
|
||
`PROTOCOL_VERSION`, which versions the shard wire and says nothing about a website module.
|
||
|
||
### 2.4 The module state machine
|
||
|
||
`installed → enabled → started`, with `disabled` and `startup_failed` as recoverable states.
|
||
|
||
**A module that fails to load must never take the site down.** The loader catches failures across the
|
||
module's entire lifecycle — require, schema fragment, router construction, registration calls,
|
||
`onBoot` — not merely those that surface after a router object was returned. Any failure at any point
|
||
marks that one module `startup_failed`, records the reason, and the site comes up with that module's
|
||
routes and nav absent. `startup_failed` is recoverable from the admin panel — disable, retry, or roll
|
||
back to the previous version — with no shell access to the box.
|
||
|
||
**Where the states live.** One `installed_modules` row per module, keyed by its id, with the machine
|
||
held in a single `state` column carrying all five values — the shape this section already describes,
|
||
rather than a policy flag beside a runtime one. The table also carries `name`/`version` for the admin
|
||
screen, `failure_stage` + `failure_reason` for [`MODULE_API.md`](MODULE_API.md) §4.4's recorded
|
||
reason, `source` + `sha256` for the
|
||
install provenance of §2.5 below (both null for a directory placed on the volume by hand, which stays
|
||
supported), and `installed_at` / `started_at` / `updated_at`. Full column list in
|
||
[`BACKEND_DESIGN.md`](BACKEND_DESIGN.md) §3.
|
||
|
||
**The row is a record of what happened, never the source of truth for what is mounted.** The loader
|
||
scans the filesystem at require time, before the database is reachable (API §4.1), so the URL surface
|
||
is a property of the volume and not of a row here. What the row decides is whether a mounted module
|
||
*answers* (`disabled` ⇒ its guard 404s, API §4.5) and what the admin panel shows after a failure.
|
||
This is also why
|
||
`routes.manifest.json` can be generated against a dead database.
|
||
|
||
**`disabled` is the only state a boot leaves alone.** Every boot resets each non-disabled row to
|
||
`enabled`, clearing any recorded failure, and the load that follows writes this boot's outcome —
|
||
`started` or `startup_failed`. Three consequences, all deliberate:
|
||
|
||
- **A `startup_failed` module is retried on every restart.** An operator who fixes the underlying
|
||
cause — a truncated file, a missing dependency, a database that was not up yet — gets the module
|
||
back by restarting, with no admin-panel visit. The cost is that a deterministically broken module
|
||
re-records its failure each boot, which is the honest thing for it to do.
|
||
- **A stale reason can never be shown against a running module**, because every non-failing
|
||
transition clears the failure columns.
|
||
- **Disabling is an operator decision, not an outcome**, so it survives restarts untouched — and a
|
||
module the operator switched off is neither started nor re-recorded as failed if it happens to be
|
||
broken. `installed` is likewise transient: it is the gap between an install writing the row and the
|
||
restart that resolves it.
|
||
|
||
A re-install or an upgrade refreshes `name`/`version`/provenance and deliberately leaves `state`
|
||
alone: upgrading an enabled module must not silently switch it off, and re-installing a disabled one
|
||
must not silently switch it on.
|
||
|
||
**A row whose directory is gone is marked `startup_failed`** (stage `require`, reason "module
|
||
directory not present on the volume"), settled with PR 5. The boot reset above has just moved it to
|
||
`enabled`, and a row claiming to be enabled for a module that is not on the volume is the one state
|
||
that is simply untrue — it would be read that way by the admin panel and by
|
||
`GET /api/v1/public/modules` alike. This catches only a directory deleted by hand: an uninstall
|
||
leaves the row `disabled`, which the reset never touches.
|
||
|
||
### 2.5 Install, uninstall, purge
|
||
|
||
Modules live on a **mounted volume**, not in the image. That is what makes the WordPress model work
|
||
against a pull-only image. *(Settled in Phase 2 PR 9: it is a **bind mount** of `./modules`, not the
|
||
named volume this sentence originally reached for by analogy with `uploads` — hand-placing a module
|
||
directory is a supported install below, and a named volume would route it through `docker cp`. The
|
||
image's copy is excluded by `.dockerignore`, so a module in a builder's working tree can never ship
|
||
inside an image; see `modules/README.md` in the website repo.)*
|
||
|
||
**Install:** admin selects the module → bundle downloaded from the module repo's release and verified
|
||
against its `sha256` → unpacked into `modules/<id>/` on the volume → `installed_modules` row written →
|
||
**restart**. On boot the loader scans the filesystem (§1.12), validates prefixes, mounts, replays the
|
||
schema fragment, runs `onBoot`, and each module reaches `started` or `startup_failed`.
|
||
|
||
Nothing is compiled at any point. The operator restarts; they never build.
|
||
|
||
**Uninstall** (default, non-destructive): row set to `disabled`, directory removed, restart. The
|
||
module's tables and data are **retained**. **Purge** is a separate, explicit, destructive action that
|
||
runs `purge.sql`; it is never bundled into uninstall.
|
||
|
||
**Surfaces:** the admin panel, and the Docker environment under `website/` — a declarative module set
|
||
resolved at container start from the mounted volume, so a compose-managed host is not driven by
|
||
clicking. Both paths write the same `installed_modules` row and neither requires a build step.
|
||
|
||
### 2.6 How the client half loads
|
||
|
||
This is the piece §1.14 constrains hardest. Three requirements had to hold at once: the operator
|
||
builds nothing, production pulls a prebuilt image, and `script-src 'self'` forbids inline script.
|
||
|
||
1. **The module's CI builds its client half** with Vite in library mode, declaring `react`,
|
||
`react-dom` and `react-router-dom` as **externals**. The module never bundles its own React —
|
||
there is exactly one React instance, owned by core.
|
||
2. **Core exposes the shared dependencies on a global** before mount — `window.__rg = { react,
|
||
reactDom, router, registry }` — and the module's externals resolve to it. A global, not an import
|
||
map, precisely because an import map must be inline and CSP forbids that.
|
||
3. **`htmlShell.js` injects the module's entry script.** It already rewrites the shell it serves, so
|
||
this is an extension of a working mechanism, not a new one. The tag is
|
||
`<script type="module" src="/modules/uo/entry.js">` — same-origin, so `'self'` passes with no
|
||
nonce and no inline. *(Amended in Phase 2 PR 7: the tag is injected before `</body>`, not at the
|
||
`</head>` rewrite this step assumed. Module scripts execute in document order and core's bundle
|
||
has to run first, so the injection must be after core's own script tag wherever a bundler chooses
|
||
to put it — [`MODULE_API.md`](MODULE_API.md) §3.1, which also states the static mount's root, its
|
||
state guard and its cache policy.)*
|
||
4. **The SPA reads `/api/v1/public/modules`** to feature-detect against what this backend is
|
||
serving. Registration happens when the injected chunk executes and calls `window.__rg.registry` —
|
||
it is not gated on this call. *(Amended by [`MODULE_API.md`](MODULE_API.md) §6.7: this step
|
||
originally said the SPA reads the endpoint "to learn what to load", which step 3 above had
|
||
already answered a different way. Nothing waits on an API round trip to start loading. The
|
||
endpoint's shape is API §2.9.)*
|
||
|
||
Phase 1 prototypes exactly this before anything is committed to it (§2.7).
|
||
|
||
### 2.7 Phases
|
||
|
||
**Phase 0 — unblock CI and scaffold the repo.** Land the one-line `pr-checks.yml` trigger fix on
|
||
`website` `main` (§2.9), cut `edge` from `main`, and give `Module-uo` its initial commit (§2.3).
|
||
Nothing else can be trusted until the first of these is done.
|
||
|
||
**Phase 1 — API contract + spike (blocking).** Merge this document. Write the contract at
|
||
[`docs/website/MODULE_API.md`](MODULE_API.md) — **done**; the places it amends this document are
|
||
listed in its Part 6, one of which (OpenAPI generation, §6.1 there) needs a decision before Phase 2
|
||
starts. Then a throwaway spike on an unmerged branch moving **`/api/v1/public/atlas/*`** behind the
|
||
proposed surface — the smallest honest test: six routes, DB-backed, no sidecar, no SSE, one boot
|
||
hook. The spike must *also* prove the §2.6 chunk load end to end, since that is the highest-risk
|
||
decision in the plan. Exit criteria: no internal-file imports, `npm run routes:manifest` produces a
|
||
zero-line diff, and the chunk loads under the enforced CSP.
|
||
|
||
**Phase 1 is complete.** The spike ran on `website` branch `spike/module-atlas` (cut from `edge`,
|
||
never merged) and **met all three exit criteria** — see [`MODULE_API.md`](MODULE_API.md) Part 7. §2.6
|
||
survives intact: the prebuilt chunk loads and renders under `script-src 'self'` with zero violation
|
||
reports. The one thing it changed is that §2.6's one-React rule turns out to have a server-side twin
|
||
nobody had written down — a module cannot resolve core's `express` either, so core hands that over
|
||
too (API §7.2).
|
||
|
||
**Phase 2 — Core scaffolding, no behaviour change.** One PR each, in order:
|
||
|
||
1. `installed_modules` table + the §2.4 state machine.
|
||
2. `src/modules/loader.js` — synchronous filesystem scan, manifest validation, prefix-collision
|
||
rejection, per-module try/catch across the whole load path, mounting into the tier routers.
|
||
3. `ensureSchema()` extended to replay module fragments after core's.
|
||
4. The three de-entanglement registries (§1.8), with core still the only registrant.
|
||
5. Boot/shutdown hook dispatch in `server.js`, likewise.
|
||
6. `GET /api/v1/public/modules` — ids, names, versions and capabilities of the modules currently
|
||
**serving**, shaped like the existing branding/site-settings endpoints (anonymous, database-free,
|
||
not site-mode gated). The SPA and the Android plan both feature-detect against it; it is not what
|
||
loads a client chunk ([`MODULE_API.md`](MODULE_API.md) §2.9 and §6.7).
|
||
7. Client `src/modules/registry.js`, the `window.__rg` shared-dependency global, the chunk's static
|
||
mount and the `htmlShell` script injection — empty registry, no visible change.
|
||
8. `MOD_PATHS` → `roles`-derived (§1.4); the generic feature-provider seam (§1.5).
|
||
9. `docker-compose.yml` gains the `modules` mount.
|
||
|
||
Exit criterion: no **existing** URL moves and every existing test passes. If Phase 2 changes one URL,
|
||
it is wrong. PR 6 is the single deliberate exception in the phase and it *adds*: `routes.manifest.json`
|
||
gains exactly one line, `GET /api/v1/public/modules`, and nothing else in the file moves. Every other
|
||
PR in Phase 2 produces a zero-line diff.
|
||
|
||
**Progress: complete — all nine PRs landed.**
|
||
|
||
- **PR 1** — `installed_modules` and the state machine, with the stored shape and the boot rules
|
||
settled in §2.4 above.
|
||
- **PR 2** — `server/src/modules/loader.js`: the filesystem scan, manifest validation, prefix and
|
||
table-name collision rejection, per-module try/catch and the tier mount, behind the §4.5 dispatch
|
||
guard. Two decisions landed with it, both recorded in [`MODULE_API.md`](MODULE_API.md): the load
|
||
trigger is **one explicit `modules.load(tierRouters)` call in `app.js`**, never a lazy scan
|
||
(API §7.6); and the "does core own this prefix" check **probes the live tier routers** rather than
|
||
a hardcoded table, so it cannot drift when core adds a capability router (API §4.3). The three
|
||
de-entanglement registries and the two lifecycle hooks throw `not available until phase 2 PR 4/5`
|
||
rather than no-op — an accepting stub would let a module believe it had registered something.
|
||
28 tests, all on the failure paths.
|
||
- **PR 3** — schema fragment replay. `ensureSchema()` replays each installed module's fragment after
|
||
core's, with the statement splitter extracted to `utils/sqlStatements.js` so both are split by the
|
||
same code. The decision that shaped it, recorded in [`MODULE_API.md`](MODULE_API.md) §2.6: the
|
||
fragment is **validated at load time and executed later**, split on whether a database is needed to
|
||
know the answer — a fragment breaking a stated rule never mounts, while a failure only the server
|
||
could report (a bad column type) is post-mount and 503s. The rules are enforced as a **leading-verb
|
||
allowlist** (`CREATE`, `ALTER`, `INSERT`, `UPDATE`) rather than the `DROP` denylist §2.6 words them
|
||
as, because the file is replayed on **every boot**. Found while wiring it: `npm run seed` calls
|
||
`ensureSchema()` without ever requiring `app.js`, so the replay has to tolerate an unscanned loader.
|
||
|
||
- **PR 4** — the three de-entanglement registries, `src/modules/registries.js`. Core's own streams,
|
||
its Discord announce leg and its users-detail routes all go through them, so the seams are
|
||
exercised on every boot before a module depends on them; §1.8 and §1.9 above record what each
|
||
became. Four decisions landed with it, all recorded in [`MODULE_API.md`](MODULE_API.md): announce
|
||
legs became a **child table** rather than waiting for Phase 3 (§2.4 — a module cannot alter a core
|
||
table, so a registered leg had nowhere to live); **`mapEvent` dropped** from the stream registry
|
||
(§2.4 — a leftover from before the push inversion was settled); **core registers through the same
|
||
staging area a module uses**; and core's six shard sub-paths **moved behind the slot now** rather
|
||
than in Phase 3.
|
||
|
||
Registering is **validate-then-commit**: the loader stages a module's claims and the second pass
|
||
commits them, so a module that throws halfway through `register()` — or fails a later validation
|
||
step — leaves nothing behind. That is the registry-side twin of PR 2's second-pass mount rule.
|
||
|
||
Two build tools needed teaching, both because a mechanism this PR introduced is one they had never
|
||
seen. `scripts/routeManifest.js` could not decode a **parameterised mount**: its unwinder expected
|
||
a group shape express does not emit, and the branch had never run. It threw rather than guessing,
|
||
which is exactly what it is for. And `swagger-autogen` could not follow a route into an extension
|
||
**slot**, deleting 407 lines while reporting success; the fix is the fragment merge core owed
|
||
anyway ([`MODULE_API.md`](MODULE_API.md) §6.6).
|
||
|
||
There is still no module on the volume and no boot wiring, so this changes nothing an operator or a
|
||
client can see: **884 tests pass** and `routes.manifest.json` is unchanged at 229 routes. The two
|
||
lines of OpenAPI that do move are the retry endpoint's summary and its `leg`, which is no longer a
|
||
fixed enum because the leg set is whatever has been registered.
|
||
|
||
- **PR 5** — boot/shutdown dispatch and the `installed_modules` reconcile, `src/modules/lifecycle.js`.
|
||
`api.onBoot`/`api.onShutdown` stop throwing, `server.js` gains one call on each side, and the
|
||
§2.4 machine finally runs against real outcomes — which is what makes §4.5's `disabled` 404 leg
|
||
reachable for the first time. Four decisions landed with it, all recorded in
|
||
[`MODULE_API.md`](MODULE_API.md) §2.5 and §4.4: **the loader classifies its failures** by §4.3 step,
|
||
so `failure_stage` says where a module broke instead of being a column nothing filled; **a row whose
|
||
directory is gone is marked failed** rather than left claiming `enabled` (§2.4 above); **core's eight
|
||
UO boot call sites stay in `server.js`** until Phase 3, because unlike a registered announce leg a
|
||
boot call site already has somewhere to live and moving it now would be extraction done early in a
|
||
phase whose exit criterion is that nothing changes; and **`onBoot` gets no timeout** — shutdown races
|
||
a SIGKILL and boot does not, and a slow `onBoot` delaying the listener is the contract's promise to
|
||
a module that must warm up before it serves.
|
||
|
||
The dispatch lives outside the loader for the reason the schema replay does: the loader is required
|
||
by `app.js` against a dead pool, and this half is database-first. They meet at one function,
|
||
`loader.setState()`, so the in-memory record the dispatch guard reads and the row the admin panel
|
||
reads cannot drift apart.
|
||
|
||
Still nothing on the volume: **900 tests pass**, `routes.manifest.json` is unchanged at 229 routes
|
||
and the OpenAPI spec regenerates byte-identical.
|
||
|
||
- **PR 6** — `GET /api/v1/public/modules`, the first module-system URL a client can see. Four
|
||
decisions, all recorded in [`MODULE_API.md`](MODULE_API.md) §2.9: **`started` modules only**, so a
|
||
disabled or failed module is absent exactly as its routes and nav already are, and no visitor is
|
||
told that something is broken; **no `state`, `failure_stage` or `failure_reason`** on the public
|
||
surface — those are the admin screen's, and the reason is an exception string from inside core;
|
||
**no `client` chunk URL**, because `htmlShell` hands the browser the tag rather than a URL to fetch
|
||
(which amends §2.6 step 4 above — see API §6.7); and **no `siteMode` gate and no database**, the
|
||
same class as `/public/status` and `/public/version`, so a client can still feature-detect while
|
||
the site is in maintenance.
|
||
|
||
It is a **capability router of its own** rather than a fifth singleton in `site.router.js`, and that
|
||
is the load-bearing part. The loader's prefix-collision probe reads the live tier stack and skips
|
||
root-mounted layers, because a `use('/', …)` matches every path — so a route declared inside the
|
||
root-mounted site router is invisible to it. Mounting `use('/modules', …)` is what makes "no module
|
||
may ever claim `/modules`" a rule the loader enforces rather than a convention a reviewer has to
|
||
remember.
|
||
|
||
**910 tests pass** (+9, every one of them on the boundary: what must *not* appear). The route
|
||
inventory goes 229 → 230 (228 public + 2 internal) and moves by exactly the one added route;
|
||
`routes.guards.json` records it with an empty `gates` list, which is itself the assertion that the
|
||
endpoint is ungated. The OpenAPI spec gains the operation and the `PublicModules`/`PublicModule`
|
||
schemas. The published mirror [`api-route-inventory.json`](./api-route-inventory.json) is refreshed
|
||
to match.
|
||
|
||
- **PR 7** — the client half's delivery: `client/src/modules/registry.js`, `window.__rg`
|
||
(`modules/shared.js`), the chunk's static mount and the `htmlShell` injection, with `App.jsx`
|
||
reading `routesFor` for all three areas. The registry is empty on a bare core, so nothing an
|
||
operator can see changes. Four decisions, all recorded in [`MODULE_API.md`](MODULE_API.md) §3.1 and
|
||
§3.4: **routes now, nav in PR 8** — PR 7 is "a module chunk loads and renders its page", PR 8 is
|
||
"it appears in the nav", which keeps the nav interleave and its override merge in one reviewable
|
||
change; **the script tag is injected before `</body>`**, not into `</head>`, so the ordering that
|
||
the whole client contract rests on comes from document structure rather than from Vite's choice to
|
||
hoist core's entry into `<head>`; **the static mount is rooted at the entry's directory, behind the
|
||
module's state guard, with `no-cache`** — a mount rooted at the module root would publish server
|
||
source, `module.json` and the schema fragment, so an entry in the module root is rejected outright;
|
||
and **the UI kit ships its seven real members**, with `AdminPage` struck from the contract rather
|
||
than invented in core to satisfy a table.
|
||
|
||
The verification that mattered was **not a test**. Everything passed against a build that did not
|
||
work in a browser: core mounted before any module chunk had evaluated, because `document.readyState`
|
||
during a deferred script is `'interactive'`, not `'loading'`. A module's routes were missing from
|
||
the first render and its URL redirected home — indistinguishable from a module that failed to load,
|
||
and with nothing logged anywhere. It was found by loading a hand-written chunk in Chrome, and that
|
||
smoke is now written down as part of the contract ([`MODULE_API.md`](MODULE_API.md) §7.7) because no
|
||
test in this repo can see it. The same run confirmed the property §3.6 called the highest-risk
|
||
detail in the plan: the chunk executes under the **enforced** `script-src 'self'`, resolving core's
|
||
React and UI kit off the global, with zero CSP reports.
|
||
|
||
**933 server tests** (+23) and **123 client tests** (+14) pass; `routes.manifest.json` is unchanged
|
||
at 230 routes and the OpenAPI spec regenerates byte-identical — `/modules/<id>/` is a
|
||
filesystem-conditional static mount, not API surface, for the same reason `/uploads` and `/brand`
|
||
are not in the manifest.
|
||
|
||
- **PR 8** — the nav half PR 7 deferred, and the two seams §1.4 and §1.5 asked for: `withModuleNav`
|
||
(`client/src/modules/nav.js`) interleaving module rows into core's three navs, `MOD_PATHS` and the
|
||
moderator redirect replaced by a `roles`-derived computation in `client/src/lib/adminNav.js`, and
|
||
the generic feature-provider seam (`modules/features.jsx` + `modules/featureGate.js`) that core
|
||
registers its own `useShardFlags` into. Four decisions, all recorded in
|
||
[`MODULE_API.md`](MODULE_API.md) §3.3.
|
||
|
||
**The interleave happens before the admin-override merge**, which is the decision the rest follow
|
||
from: the merge is keyed by `to` and drops any key its base array does not declare, so module rows
|
||
appended after it would be unorderable, unrelabellable and unhideable — and today's UO rows are
|
||
all three of those things, so appending would make the extraction a visible regression for every
|
||
operator who has ever edited their nav. Doing it first means a module row is an ordinary row to
|
||
everything downstream: nothing in `navOverrides.js`, `NavEditor.jsx` or the layouts knows a module
|
||
exists. **Moderator visibility derives purely from `roles`**, which moves two rows the old
|
||
allowlist withheld — Dashboard, whose `roles` had always named moderator, and My Characters, which
|
||
is ungated self-service — both toward what the server already permitted. **A row's `feature` is
|
||
resolved by the provider its own module registered**, so the namespace comes from the registration
|
||
rather than from a parsed string prefix. And **core registers through the same seam**, under the
|
||
owner id `core`, so `SiteHeader` holds one mechanism instead of two and Phase 3 is a deletion.
|
||
|
||
The PR also fixed a defect that predates the module system: the moderator redirect was a **third**
|
||
hardcoded list, and it disagreed with `MOD_PATHS` about `/admin/houses`, so a moderator who
|
||
clicked Houses in their own sidebar was bounced back to Moderation. The derived allow-list is
|
||
computed from the **base** nav rather than the merged one, so an override — which is presentation
|
||
— cannot move that boundary in either direction.
|
||
|
||
**933 server tests** (unchanged — this PR is client-only) and **160 client tests** (+37) pass;
|
||
`routes.manifest.json` is unchanged at 230 routes and the OpenAPI spec regenerates byte-identical.
|
||
The [§7.7](MODULE_API.md#77-the-client-half-has-to-be-verified-in-a-browser--the-timing-bug-no-test-could-see)
|
||
browser smoke was re-run, since this is the seam that rule exists for: a throwaway module
|
||
registering nav in all three areas and a provider granting one flag and withholding another. It
|
||
confirmed, in Chrome with the console open, that the row lands inside core's Moderation group
|
||
rather than in an appended block, that the withheld row does not render while the granted one
|
||
does, that a moderator reaches both `/admin/houses` and the module's own admin page, and that an
|
||
admin can relabel a module row in Admin → Navigation and have it persist and apply — the whole
|
||
point of merging before the override layer. Zero CSP reports, zero console errors.
|
||
|
||
- **PR 9** — the mount itself, which closes the phase: `docker-compose.yml` gains `./modules` at
|
||
`/app/modules`, the `Dockerfile` creates that directory node-owned, `.dockerignore` keeps any local
|
||
one out of the image and `modules/README.md` documents the directory for whoever opens it.
|
||
|
||
It is a **bind mount, not the named volume** §2.5 assumed by analogy with `uploads`. Placing a
|
||
module directory by hand is a supported install, and a named volume makes that a `docker cp` into a
|
||
running container — the one install path an operator without the admin panel has, routed through
|
||
the least discoverable mechanism Docker offers. A bind mount makes it `tar -xf … -C ./modules`, and
|
||
makes the installed set something an operator can *see*. §2.5 is amended above; nothing else about
|
||
install, uninstall or purge changes.
|
||
|
||
Two consequences worth stating, because both are silent failures rather than errors. The directory
|
||
is **tracked** — via its README, the same shape `brand/` already uses — because Docker recreates a
|
||
missing bind-mount source as `root:root`, and the container runs as uid 1000: delete `modules/` from
|
||
a checkout and the next install fails on a permission error that names no cause. And the mount is
|
||
**read-write**, since §2.5's install unpacks into it from inside the container; deferring that to
|
||
Phase 4 would have bought nothing, as a mount-mode change is a redeploy either way.
|
||
|
||
`.dockerignore` matters more than it looks. `COPY . .` would otherwise bake whatever module the
|
||
builder had checked out into every image — and because Docker seeds a *fresh* named volume from
|
||
the image's contents, that module could have appeared on a production deployment that never
|
||
installed it. The exclusion is what makes "modules live on a mount, never in the image" true rather
|
||
than merely intended.
|
||
|
||
Verified against a **running container**, which is the only thing that can check any of the above —
|
||
a compose file that parses proves nothing about ownership, and nothing about what the image
|
||
contains. The image carries an empty, node-owned `/app/modules` despite a module sitting in the
|
||
build context. A module on the mount loads, mounts, runs `onBoot` and reaches `started`;
|
||
`/api/v1/public/modules` lists it; its chunk serves from the entry's directory alone, with the
|
||
module's own server source and `module.json` both `404`. The [§7.7](MODULE_API.md#77-the-client-half-has-to-be-verified-in-a-browser--the-timing-bug-no-test-could-see)
|
||
browser smoke was re-run against the containerised stack rather than a working tree: in Chrome, the
|
||
page renders on first paint inside core's `PublicLayout`, drawing React and the UI kit off
|
||
`window.__rg`, with its nav row interleaved into core's public nav — under the enforced
|
||
`script-src 'self'`, with zero CSP reports and no console errors. Removing the directory by hand and
|
||
restarting reconciles the row to `startup_failed`/`require` exactly as §2.4 says, and leaves core
|
||
healthy with no script injected and `{"modules":[]}` published.
|
||
|
||
**933 server tests** and **160 client tests** pass, both unchanged — this PR ships no application
|
||
code. `routes.manifest.json` stays at 230 routes and the OpenAPI spec regenerates byte-identical.
|
||
The one source change is a comment: `scripts/routeManifest.js` enumerated the filesystem-conditional
|
||
mounts it excludes and had never been told about `/modules`. Its filter is an allowlist, so the
|
||
behaviour was always right and only the explanation was stale.
|
||
|
||
**Phase 2 is complete.** Core can discover, validate, mount, migrate, boot, publish, serve and
|
||
navigate a module it does not contain, on a deployment that builds nothing — and it does all of that
|
||
while no existing URL has moved. The exit criterion held: `routes.manifest.json` went 229 → 230 across
|
||
the whole phase, and the one added line is PR 6's deliberate `GET /api/v1/public/modules`.
|
||
|
||
**Phase 3 — Extract `module-uo`.** Moves out of `website/`: the 8 model directories and their 25
|
||
tables; the nine UO `utils/` files plus `newsGump.js`; the 13 router/controller files;
|
||
`scripts/importSpawnAtlas.js` and `db/spawnAtlas.art.json`; `usersShard.controller.js` **minus
|
||
`getUser`** (§1.9); the shard-derived half of `notificationStreams.js` and the town-crier leg of
|
||
`announceWorker.js`; and on the client, roughly twenty route components, their nav registrations and
|
||
`useShardFeatures`.
|
||
|
||
Acceptance, all four required:
|
||
|
||
1. **Zero UO identifiers in core** — no `shard`, `uoLink`, `cliloc`, `atlas` or `towncrier` outside
|
||
`modules/`. Enforced by a CI grep test, not by review.
|
||
2. **Zero internal-file imports** from `module-uo` into core.
|
||
3. **`routes.manifest.json` API diff is zero lines**, except the deliberate `GET /admin/users/:id`
|
||
ownership move, which changes no URL. After extraction the core manifest no longer contains UO
|
||
routes — `module-uo` generates and freezes its own in its own repo.
|
||
4. **A written `module-rust` dry run** — manifest, mounts, nav entries, one notification stream — not
|
||
implemented, to prove the contract generalises before more is built on it. It lands as
|
||
`docs/modules/rust-dryrun.md`, where §2.10 already aggregates module documentation; Phase 5's
|
||
Integration Kit links to it rather than copying it, per the kit's own never-re-specify rule
|
||
(§2.11).
|
||
|
||
#### 2.7.1 Phase 3's shape — settled 2026-08-11
|
||
|
||
Measured against `edge` at the close of Phase 2, the surface is **72 server files / ~9,700 lines**,
|
||
**51 client files / ~3,700 lines**, and **32 of core's 82 server test files**. The counts in the
|
||
paragraph above were written in Phase 0 against a smaller tree and are superseded by the slice table
|
||
below.
|
||
|
||
**The finding that sets the order: the two halves are independent.** Because §1.2 preserves API URLs
|
||
exactly, core's client keeps calling `/api/v1/public/shard/status` after that route is served by the
|
||
module, and a module page calls the same URL while core still serves it. Nothing forces a feature's
|
||
server and client halves to move together, so the extraction is **server-first, then client**, sliced
|
||
by feature — which keeps each PR inside one layer and one review's worth of context.
|
||
|
||
**Merge order across the two repos: `module-uo` first, then `website`.** The loader's `ownedByCore`
|
||
probe means a module cannot *load* while core still owns its prefix — but `module-uo`'s own CI never
|
||
loads it into core, so its PR merges perfectly well beforehand. Taking that order means `edge` serves
|
||
the feature from core right up to the moment core drops it, and there is never a window where the
|
||
branch is missing a feature outright. The reverse order would break `edge` at every slice boundary
|
||
for the length of a review. Verification is unaffected either way: a slice is proved by running the
|
||
*pair* together locally — the module branch checked out into `website/modules/uo`, the deletion
|
||
branch checked out in `website/` — before either merges.
|
||
|
||
Each slice is one `module-uo` PR (adds), one `website` PR (deletes), and one `docs` PR:
|
||
|
||
| # | Slice | Moves |
|
||
| --- | --- | --- |
|
||
| 0 | **The bundle skeleton** | `module.json`, both `package.json`s, `server/index.js` registering nothing, the Vite library build + the four shared-dep shims, an empty `schema.sql`/`purge.sql`, CI armed, the §5.1 zero-internal-imports check. `website` untouched. |
|
||
| 1 | **Atlas + clilocs** | `shardAtlas`/`shardClilocs` models, `clilocParse`/`clilocSource`/`spawnAtlasParse`/`spawnAtlasSource`, `public/atlas.*`, `admin/shardAtlas.controller`/`shardClilocs.controller`, `scripts/importSpawnAtlas.js`, the art JSON |
|
||
| 2 | **The live shard** | `shardState`/`shardEvents`/`shardVisibility`/`uoLinkConfig` models, `uoLinkClient`/`uoLinkSocket`/`shardIngest`/`shardBroadcast`/`shardPush`/`shardVisibility` utils, `config/shardStreams.js`, `public/shard.*`, `admin/shard.router`/`shardOps`/`shardVisibility`/`uoLink.*` |
|
||
| 3 | **Market** | `shardMarket` model, `shardSales` util and their routes |
|
||
| 4 | **Account links + the extension slot** | `shardLinks` model, `player/shard.*`, `admin/usersShard.*` minus `getUser` — the first real filling of `admin.users.detail` |
|
||
| 5 | **The news leg** | `newsGump.js`, `shardAnnounce.js`, the town-crier `registerAnnounceLeg` |
|
||
| 6 | **Public pages** | `Shard`, `ShardActivity`, `Rules`, `Atlas`, `AtlasCreature`, `ChampSpawns`, `Market`, `MarketVendor`, `Governors`, `Guilds`, `Houses`, `Leaderboards`, `PlayersOnline`, `VendorSales`, `data/cityCrests.js`, `lib/shardEvents.js`, `lib/useShardFeed.js`, and their public nav rows — under `/uo/*` per §2.8 |
|
||
| 7 | **Admin + player pages** | `ShardAdmin`, `ShardOps`, `ShardVisibility`, `SpawnAtlas`, `HousesAdmin`, `AdminCharacter(s)`, `PlayerCharacter(s)`, `GameAccounts`, `CharacterSheet`, `CharacterStats`, `ShardAccountActions`, `CreateGameAccountForm`, `useShardFeatures` and the `useShardFlags` feature provider — under `/admin/uo/*` and `/player/uo/*` |
|
||
| 8 | **De-UO core's copy** | `About`, `Screenshots`, `Website`, `SiteFooter`, `heroLayout`'s defaults, `api/client.js`'s `shard`/`atlas` namespaces, and the comments in `navOverrides.js` — plus the §5.2 CI grep that keeps them out |
|
||
| 9 | **Close the phase** | `module-uo`'s frozen route manifest and release workflow; `docs/modules/uo/` and `docs/modules/rust-dryrun.md` |
|
||
|
||
**Criterion 1 is a grep over code, not over prose** — see [API §5.2](MODULE_API.md#52-zero-uo-identifiers-in-core-ci-website-repo)
|
||
for what that means precisely. Core's marketing copy says "shard" in a dozen places, and a literal
|
||
word grep would have made every one of them a CI failure while proving nothing about the boundary.
|
||
Slice 8 rewrites that copy anyway, because a core that still reads as a UO site is not the
|
||
game-agnostic platform this workstream is for — but it is a deliberate piece of work with its own
|
||
review, not an exemption hidden in a grep pattern.
|
||
|
||
**One small gap in the kit, deliberately not closed.** The UO client views import almost exactly the
|
||
seven §3.4 members — plus `lib/format.js`, a pure leaf formatter. The module **vendors a copy**
|
||
rather than core adding an eighth member: the kit is closed on purpose, and a function with no
|
||
props and no layout cannot drift the way a component can. The same is not true of `PublicLayout`,
|
||
which is why that one is in the kit.
|
||
|
||
#### Slice 0 — the bundle skeleton (Module-uo#2, 2026-08-11)
|
||
|
||
`module.json`, an entry point taking `(ctx, api)`, the Vite library build, four shims, and both
|
||
boundary checks. It **registers nothing**, and core is untouched — what it proves is the delivery
|
||
path itself, before a single UO file moves into it. 29 server tests and 9 client tests, both new.
|
||
|
||
Verified against a real core rather than asserted: the module loads, mounts its zero routes, reaches
|
||
`started` and is published by `/api/v1/public/modules`; its chunk serves from the entry's directory
|
||
with `Cache-Control: no-cache` while its server source, `module.json` and `package.json` all 404;
|
||
and in Chrome, under the enforced `script-src 'self'`, the chunk reports every shared dependency
|
||
**identity-equal** to core's, with zero CSP reports.
|
||
|
||
**Three findings, each of which had produced a green build that was wrong.** The first amends the
|
||
contract and is written up at [API §3.6](MODULE_API.md#36-vite-library-mode-build): `external` and
|
||
the aliases do not compose, so `external` is now empty and a resolution-time build guard replaces
|
||
it. The second is that guard's own two failures — hooking `load` (first-wins, so it never ran) and
|
||
deriving its forbidden list from the alias list (so deleting an alias deleted the guard). Both were
|
||
found by breaking an alias on purpose and checking the build actually went red, which is the only
|
||
way a guard's absence is visible.
|
||
|
||
The third is about the boundary check itself and generalises past this repo. **`checkImports.js`
|
||
failed on its own documentation** — the comment naming `require("../../etc/passwd")` as an example
|
||
of what to catch, and the entry point's comment explaining why a module must never
|
||
`require('express')`. A check that cannot survive being described is one people stop writing
|
||
comments around, so it strips comments and template literals with a character walk rather than a
|
||
regexp (a URL in a string contains a comment opener; a comment contains quotes) and carries its own
|
||
test suite. The same applies to slice 8's §5.2 grep, which will be read by a codebase that discusses
|
||
modules constantly.
|
||
|
||
**One thing to know before running a module locally: the loader skips a *symlinked* module directory
|
||
silently.** `readdirSync(…, { withFileTypes: true }).filter(e => e.isDirectory())` reports a Windows
|
||
junction as a symlink, so a module linked rather than copied into `modules/` is simply not there,
|
||
with nothing logged. Not a defect for a real install — `modules/` is a bind mount of real
|
||
directories (§2.5) — but it is the first thing to check when a module fails to appear.
|
||
|
||
**Phase 4 — Delivery.** The admin-panel Modules screen (install, enable, disable, retry, purge,
|
||
`startup_failed` with its recorded reason) and the Docker-environment path from §2.5. Deliberately
|
||
last, so loader, packaging, schema and chunk-loading problems are not all being debugged at once.
|
||
|
||
**Phase 5 — The Integration Kit.** `RunicGateway/Integration-kit`, the instruction book for building
|
||
a module for a game that is not UO — the website module, the sidecar and why it exists, and the
|
||
game-side plugin that feeds it. Scaffolded when Phase 2 lands, written against Phase 3's extraction,
|
||
finished alongside Phase 4. Full shape and its acceptance test in §2.11.
|
||
|
||
### 2.8 SPA URL namespacing — a deliberate break
|
||
|
||
**Decision: module pages are namespaced, and old paths are not redirected.** The site is not public
|
||
yet, so bookmarks, inbound links and configured nav overrides carry no real weight. This buys a
|
||
visible boundary in the URL rather than a hidden one.
|
||
|
||
The rule is that a module owns one path segment wherever it appears:
|
||
|
||
| Today | After |
|
||
| --- | --- |
|
||
| `/site/shard`, `/site/atlas`, `/site/market`, `/site/governors`, … | `/uo/shard`, `/uo/atlas`, `/uo/market`, `/uo/governors`, … |
|
||
| `/admin/shard-ops`, `/admin/shard`, `/admin/shard-visibility` | `/admin/uo/shard-ops`, `/admin/uo/link`, `/admin/uo/visibility` |
|
||
| player shard screens | `/player/uo/…` |
|
||
|
||
**API URLs are not affected** — they keep their exact paths per §1.2, so the Android app and the
|
||
Discord bot need no change for the API.
|
||
|
||
Two consequences, both accepted:
|
||
|
||
- **Saved nav-override rows are keyed by `to`** (`utils/navOverrides.js`), so any stored
|
||
`nav_public` / `nav_admin` / `nav_player` customisation stops applying and must be redone. No
|
||
migration is written.
|
||
- **`android-app/.../ui/navigation/NavPaths.kt` maps SPA paths to native screens** and holds ten
|
||
`/site/*` constants that will no longer resolve. That is one small Android PR, folded into the
|
||
separate Android module plan. App Links verification itself is unaffected — the manifest's intent
|
||
filters only cover `/mobile/callback` and `auth/callback`.
|
||
|
||
### 2.9 Branch strategy — `edge`, then one cutover
|
||
|
||
All website work lands on an **`edge`** branch and reaches `main` as a single cutover at the end,
|
||
the same shape used for [protocol v3](../link/v3.md) and the Android theming workstream. Nothing
|
||
half-extracted is ever on `main`: a core that has grown a module loader but not yet lost its UO code
|
||
is a coherent state, and a core mid-extraction is not.
|
||
|
||
`edge` does not exist on `website` today — the protocol v3 cutover landed and the branch was cleaned
|
||
up, so it is cut fresh from `main`. `Module-uo` develops on its own `main` from its first commit;
|
||
it has no cutover to perform, since nothing depends on it until the website cutover lands.
|
||
|
||
**Phase 0, and it blocks everything: the CI trigger.** `website/.gitea/workflows/pr-checks.yml`
|
||
declares:
|
||
|
||
```yaml
|
||
on:
|
||
pull_request:
|
||
branches: [main]
|
||
```
|
||
|
||
So a PR into `edge` runs **no checks at all** — no server tests, no client build, no bot install.
|
||
This is the same trap that let all nine Android M12 phase PRs merge with zero CI. It matters more
|
||
here than it did there, because Phase 2's exit criterion *is* a CI result: a zero-line
|
||
`routes.manifest.json` diff and a passing test suite. Running the whole workstream blind and
|
||
discovering the breakage at cutover is the expensive version of this.
|
||
|
||
The fix is one line — `branches: [main, edge]` — and it must land on `website` `main` **before** the
|
||
first module PR, not alongside it. `build-images.yml` is untouched: it triggers on push to `main`, so
|
||
images are published and production rolls at the cutover and at no point before it, which is correct.
|
||
|
||
### 2.10 Process obligations
|
||
|
||
Every server-side PR runs `npm run swagger`, `npm run routes:manifest` (the diff is reviewed, not
|
||
merely regenerated) and `npm test`, and carries a matching edit to `BACKEND_DESIGN.md`. Module
|
||
documentation aggregates in this repo under `docs/modules/<id>/` rather than living in module repos.
|
||
Conventional Commits, the AI-disclosure trailer, branches cut from an up-to-date `main`.
|
||
|
||
### 2.11 The Integration Kit — the instruction book for building a module
|
||
|
||
**`RunicGateway/Integration-kit`** — `https://gitea.whitlocktech.com/RunicGateway/Integration-kit.git`,
|
||
**empty as of 2026-08-10**: no branches, no initial commit, exactly where `Module-uo` was at the start
|
||
of Phase 0. Its first commit needs the same scaffolding as any other repo here — `README.md`,
|
||
`LICENSE.md` (GPL-3.0-or-later), `CONTRIBUTING.md` with the AI-disclosure clause, the PR template.
|
||
|
||
**Who it is for.** Everything else in this plan is written for someone changing *this* system. The kit
|
||
is written for someone building a **new** one: a person who wants Runic Gateway to front a game that
|
||
is not Ultima Online, starting from nothing. It is the only document in the project whose audience is
|
||
outside the org, and that changes how it is written — it explains and motivates rather than records
|
||
decisions.
|
||
|
||
The job spans all three layers of the data path, which is why it is one book and not a page in each
|
||
repo:
|
||
|
||
1. **The website module.** `module.json`, the server entry point and what `ctx` hands you, the
|
||
`register*` calls, the schema fragment, the prebuilt client chunk and the shared-dependency rule,
|
||
packaging and release CI. The bulk of it.
|
||
2. **The sidecar** — what it is and, more importantly, *why*. The shard is never network-reachable;
|
||
the shard dials **out** and the sidecar is the listener; the wire is a versioned compatibility
|
||
contract rather than a build dependency; only the website's backend talks to it. A new game needs
|
||
its own sidecar or an adapter into the existing one, and neither can be designed by someone who has
|
||
been handed the message list and none of the reasoning.
|
||
3. **The game-side plugin** — how a shard feeds the sidecar without ever letting the sidecar stall the
|
||
game: the bounded drop-oldest queue, the dedicated writer thread, world reads only on the game's
|
||
own thread. `servuo-plugins/` is the worked example; the constraints are general, and a plugin that
|
||
ignores them takes the game down when the sidecar wedges.
|
||
|
||
**The rule that keeps it from rotting: the kit never re-specifies a contract.**
|
||
[`MODULE_API.md`](MODULE_API.md) stays normative for the module surface, and
|
||
[`../link/PLAN.md`](../link/PLAN.md) + [`../link/INTEGRATION.md`](../link/INTEGRATION.md) for the wire
|
||
protocol. The kit *teaches* — worked examples, the reasoning, the order to do things in, the mistakes
|
||
that cost time — and links out for the authority. Where it must show a member list it quotes with a
|
||
pointer, never a copy. A guide that restates a contract diverges from it silently, and a reader who
|
||
follows the divergent copy gets a module that fails validation for reasons the guide cannot explain.
|
||
|
||
**It cannot be written before the contract is proven**, so it trails the implementation rather than
|
||
leading it:
|
||
|
||
- **Scaffolded once Phase 2 lands** — repo, license, CI, and an outline. By then a loader exists to
|
||
describe and a real module to point at.
|
||
- **Written against Phase 3's extraction**, using `Module-uo` as the worked example throughout. A kit
|
||
whose examples are invented is a kit whose examples do not compile.
|
||
- Phase 3's fourth acceptance criterion, the written `module-rust` dry run, is really this book's
|
||
first chapter — and doubles as the honest test that the contract generalises past its first module.
|
||
|
||
**Acceptance:** someone builds a trivial working module for a second game by following the kit alone,
|
||
without reading core's source. Until that has happened it is a draft, however finished it looks.
|
||
|
||
The org landing page (`RunicGateway/.profile`) and the workspace's `CLAUDE.md` repo table both gain a
|
||
row for it — when it has content, not while it is an empty repo.
|
||
|
||
---
|
||
|
||
## Part 3 — Settled decisions
|
||
|
||
| # | Decision | Where |
|
||
| --- | --- | --- |
|
||
| 1 | Modules ship idempotent `schema.sql` fragments; no migration runner is built | §1.6 |
|
||
| 2 | Website and installer stay independent; delivery is website-side only | §1.11, §2.5 |
|
||
| 3 | Core declares an extension slot on `/admin/users/:id`; all six URLs preserved | §1.9 |
|
||
| 4 | Phase 1 spike targets `/api/v1/public/atlas/*` | §2.7 |
|
||
| 4a | The contract lives in [`MODULE_API.md`](MODULE_API.md); it is normative where the two differ | §2.7 |
|
||
| 4b | Modules ship an OpenAPI **fragment**; core merges started modules' fragments into `/api/docs.json` | API §6.1 |
|
||
| 4c | Core exposes a **curated, closed** UI kit + request primitive on `window.__rg`, versioned by `MODULE_API_VERSION` | API §3.4 |
|
||
| 5 | Install surfaces: admin panel and the Docker environment; never a build step | §2.5 |
|
||
| 6 | One repo, one bundle — server and client halves version together | §2.3 |
|
||
| 7 | Android app is a separate plan; core owes it `/api/v1/public/modules` | §2.5, §2.7 |
|
||
| 8 | SPA pages namespaced: `/uo/*`, `/admin/uo/*`, `/player/uo/*` | §2.8 |
|
||
| 9 | Clean break — no redirects, no nav-override migration; site is not public yet | §2.8 |
|
||
| 10 | Client half loads as a prebuilt ESM chunk with React shared via a core global | §2.6 |
|
||
| 11 | Website work lands on `edge` and reaches `main` as one cutover at the end | §2.9 |
|
||
| 12 | The module repo is `RunicGateway/Module-uo`; the module id is `uo` | §2.3 |
|
||
| 13 | `RunicGateway/Integration-kit` is the module-builder's instruction book — module + sidecar + game plugin, teaching only, never re-specifying a contract | §2.11 |
|
||
| 14 | Phase 3 extracts **server-first, then client**, sliced by feature; `module-uo` merges before `website` in each pair | §2.7.1 |
|
||
| 15 | Criterion 1's grep reads **code, not prose**; core's UO copy is rewritten in its own slice instead | API §5.2, §2.7.1 |
|
||
| 16 | `module-uo`'s CI checks core out at a **pinned ref** to generate its frozen route manifest | API §5.3 |
|
||
| 17 | The `module-rust` dry run lands as `docs/modules/rust-dryrun.md`; the Integration Kit links to it | §2.7.1, §2.11 |
|