feat(modules): the filesystem module loader
All checks were successful
PR Checks / bot-install (pull_request) Successful in 16s
PR Checks / client-build (pull_request) Successful in 24s
PR Checks / server-tests (pull_request) Successful in 10m4s

Phase 2 PR 2 of docs/website/MODULE_SYSTEM.md 2.7. Adds
server/src/modules/{loader,semver,version}.js: the synchronous scan of
MODULES_DIR, manifest validation, prefix and table-name collision
rejection, per-module try/catch and the mount into the three tier
routers behind the MODULE_API.md 4.5 dispatch guard.

Two decisions the contract left open, both now written up there:

- The load trigger is one explicit modules.load(tierRouters) call in
  app.js, not a lazy scan (API 7.6). Accessors throw until it has run,
  because "no modules installed" is a real answer a caller must not be
  handed by accident.
- Whether core owns a prefix is asked of the live tier routers via
  express's own layer.match(), skipping root-mounted layers, rather than
  a hardcoded table -- the spike's was already stale when written
  (API 4.3).

Mounting is a second pass after every module is validated. Doing it
inside the scan loop makes the first module's layers indistinguishable
from core's, so the second module claiming a taken prefix is told it
collided with core and the module-versus-module check is unreachable.

registerExtension/NotificationStreams/AnnounceLeg and onBoot/onShutdown
throw "not available until phase 2 PR 4/5" rather than no-op; an
accepting stub would let a module believe it had registered something.
No schema replay, no boot dispatch, no installed_modules reconcile --
those are PRs 3 and 5, and until PR 5 a record's state is in memory only.

No module ships on the volume, so nothing an operator or client can see
changes: 842 tests pass, routes.manifest.json is unchanged at 229 routes
and swagger-output.json regenerates byte-identical.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2026-08-10 14:35:27 -05:00
parent dcf6ef1886
commit ec1ca7e794
5 changed files with 1001 additions and 0 deletions

View File

@@ -10,6 +10,7 @@ require('dotenv').config()
const swaggerUi = require('swagger-ui-express')
const apiRouter = require('./router/api.router')
const modules = require('./modules/loader')
const wellKnown = require('./router/wellKnown.controller')
const cspReport = require('./router/cspReport.controller')
const brand = require('./config/brand')
@@ -154,6 +155,28 @@ app.get(
app.post(csp.REPORT_PATH, cspReportLimiter, ...cspReport.parsers, cspReport.receive)
app.use('/api', apiRouter)
// ── Installed modules ─────────────────────────────────────────────────
// Discover, validate and mount whatever is on the modules volume
// (docs/website/MODULE_API.md Part 4). One explicit call, here and nowhere else:
// the loader has no lazy self-scan, so there is exactly one place that decides
// when modules are discovered, and reading the module list before this line is
// an error rather than a silent empty answer (§7.6).
//
// Position is load-bearing, in both directions. It is AFTER `/api` is mounted,
// so every core prefix is already on the tier routers when the collision check
// asks them what core owns — and so first-match-wins means a module physically
// cannot shadow a core route. It is BEFORE the `/api` 404 below, so a module
// route reaches its handler instead of the catch-all.
//
// The three requires resolve from cache to the very routers v1.router.js
// mounted; this is a reference to them, not a second copy.
modules.load({
public: require('./router/v1/public'),
admin: require('./router/v1/admin'),
player: require('./router/v1/player'),
})
app.use('/api', (req, res) => res.status(404).json({ message: 'Not found' }))
// ── /.well-known ──────────────────────────────────────────────────────

View File

@@ -0,0 +1,473 @@
// ── The module loader ──────────────────────────────────────────────────────
//
// Phase 2, PR 2 of docs/website/MODULE_SYSTEM.md §2.7. The normative contract is
// docs/website/MODULE_API.md Part 4; where the two disagree, the contract wins.
//
// The one property this file exists to guarantee, and the reason it looks the
// way it does:
//
// **The filesystem is the mounting source of truth, and mounting is
// SYNCHRONOUS.** `scripts/routeManifest.js:38` and `swagger/swagger.js:29`
// both require app.js with the pool pointed at a dead port. A loader that
// awaited a database row before mounting would make every module route
// invisible to the frozen-URL-surface test (§1.12). So: readdirSync at require
// time, no database, no promises (§4.1).
//
// A module that fails ANYWHERE in this file fails alone. Nothing here may throw
// past its own try/catch — a bad module must cost the site its routes, never its
// boot (§4.4).
//
// What is deliberately NOT here yet, each landing with the PR that first calls
// it (§2.7): schema-fragment replay (PR 3), the three de-entanglement registries
// (PR 4), boot/shutdown hook dispatch and the `installed_modules` reconcile
// (PR 5), GET /api/v1/public/modules and the client chunk's static mount
// (PRs 6-7). Until PR 5 the state a module carries is in memory only.
const fs = require('fs')
const path = require('path')
const { MODULE_API_VERSION } = require('./version')
const semver = require('./semver')
const log = require('../utils/logger')('modules')
const REPO_ROOT = path.join(__dirname, '..', '..', '..')
const MODULES_DIR = process.env.MODULES_DIR || path.join(REPO_ROOT, 'modules')
// One segment, lowercase, no parameters. A module prefix that could contain a
// `/` or a `:` would let a module reach outside the slot it was given.
const ID = /^[a-z][a-z0-9-]{1,31}$/
const PREFIX = /^\/[a-z0-9][a-z0-9-]*$/
const TIERS = ['public', 'admin', 'player']
const MANIFEST_KEYS = new Set([
'id', 'name', 'version', 'coreApi', 'server', 'client',
'schema', 'purge', 'mounts', 'extensions', 'capabilities',
])
// Extension slots core declares (§2.4). Only core may declare one; a module may
// only fill one. Validation rejects a manifest naming a slot that does not
// exist — `registerExtension` itself arrives with PR 4.
const CORE_SLOTS = new Set(['admin.users.detail'])
// id → record. Populated by load(), read by list().
const modules = new Map()
let loaded = false
// ── ctx ────────────────────────────────────────────────────────────────────
// Everything a module may reach in core, and nothing else (§2.3). Required
// lazily inside the factory rather than at file scope: this file is required by
// app.js, and hoisting these to the top would make the DB pool, the settings
// model and the upload directory startup-time dependencies of the loader itself.
function buildCtx(id, moduleRoot) {
/* eslint-disable global-require */
// The shared SERVER dependencies — the exact counterpart of window.__rg's
// react/react-dom/react-router on the client, and load-bearing for the same
// two reasons (§7.2).
//
// 1. A module lives at <repo>/modules/<id>/, OUTSIDE server/, so Node's
// resolver walks up from there and never sees server/node_modules. A
// module that required 'express' itself would fail to load — which is
// exactly how this was discovered.
// 2. Even if it resolved, a second copy of express in the process is a
// second Router prototype and a second set of instanceof checks. One
// express, owned by core, is the same rule as one React.
//
// The consequence for a module author is the same on both sides: declare these
// external, never bundle them, take them from what core hands you.
const express = require('express')
const validator = require('express-validator')
const db = require('../utils/db')
const settings = require('../model/settings/settings.model')
const posts = require('../model/posts/posts.model')
const auth = require('../utils/auth')
const pushDispatch = require('../utils/pushDispatch')
const secretBox = require('../utils/secretBox')
const createLogger = require('../utils/logger')
const { requireAuth, requireRole } = require('../auth/session.middleware')
const siteMode = require('../middleware/siteMode')
const validate = require('../middleware/validate')
const noindex = require('../middleware/noindex')
const uploads = require('../router/v1/admin/imageUpload')
/* eslint-enable global-require */
// Narrowed on purpose (§2.3): utils/auth also re-exports signToken,
// setAuthCookie and the TOTP challenge primitives, and minting a session is
// core's job. A module that needs an identity needs to READ one.
const ctx = {
moduleId: id,
paths: { moduleRoot },
express,
validator,
db: { query: db.query, pool: db.pool },
log: (namespace) => createLogger(namespace ? `${id}:${namespace}` : id),
settings: {
get: settings.get,
set: settings.set,
getInstanceName: settings.getInstanceName,
},
auth: { getUserFromRequest: auth.getUserFromRequest },
push: { publish: pushDispatch.publish },
secretBox: { encrypt: secretBox.encrypt, decrypt: secretBox.decrypt },
middleware: { requireAuth, requireRole, siteMode, validate, noindex },
uploads,
posts: {
listAll: posts.listAll,
getById: posts.getById,
linkAnnounceJob: posts.linkAnnounceJob,
markAnnounced: posts.markAnnounced,
},
}
// A guard against accident, not against a hostile module — the boundary is
// organisational, not a security boundary (MODULE_SYSTEM.md §2.2).
for (const value of Object.values(ctx)) {
if (value && typeof value === 'object') Object.freeze(value)
}
return Object.freeze(ctx)
}
// ── The registration api ───────────────────────────────────────────────────
// Collects what the module registers so validation can compare it against what
// module.json DECLARED. Declaration is the contract; a module that registers a
// prefix it did not declare is rejected, because module.json is what the admin
// panel, the collision check and the reviewer all read.
function buildApi(record) {
const once = (name) => {
if (record.called.has(name)) throw new Error(`${name}() called twice`)
record.called.add(name)
}
// PR 4 brings the three de-entanglement registries and PR 5 the boot hooks.
// They throw rather than no-op: an accepting stub would let a module believe
// it had registered something and fail silently at the far end.
const notYet = (name, pr) => () => {
throw new Error(`${name}: not available until phase 2 PR ${pr}`)
}
return {
registerRoutes(mounts) {
once('registerRoutes')
if (!mounts || typeof mounts !== 'object') throw new Error('registerRoutes: expected an object')
for (const [tier, byPrefix] of Object.entries(mounts)) {
if (!TIERS.includes(tier)) throw new Error(`registerRoutes: unknown tier "${tier}"`)
for (const [prefix, router] of Object.entries(byPrefix)) {
if (!PREFIX.test(prefix)) throw new Error(`registerRoutes: bad prefix "${prefix}"`)
if (typeof router !== 'function') throw new Error(`registerRoutes: ${tier}${prefix} is not a router`)
record.routes[tier].set(prefix, router)
}
}
},
registerExtension: notYet('registerExtension', 4),
registerNotificationStreams: notYet('registerNotificationStreams', 4),
registerAnnounceLeg: notYet('registerAnnounceLeg', 4),
onBoot: notYet('onBoot', 5),
onShutdown: notYet('onShutdown', 5),
}
}
// ── Validation ─────────────────────────────────────────────────────────────
// Table names a module may create despite not carrying its own id as a prefix.
//
// module-uo's twenty-seven tables predate the module system by two years, and
// renaming live tables is a data migration this workstream deliberately does not
// do (MODULE_SYSTEM.md §1.6). Grandfathering them by an explicit, per-module
// allowlist keeps the prefix rule real for every module written after this one —
// the alternative, dropping the rule, would leave the first name collision to be
// discovered by a module silently adopting someone else's table.
const LEGACY_TABLE_PREFIXES = { uo: ['shard_', 'uo_link_'] }
const CREATE_TABLE = /CREATE\s+TABLE\s+(?:IF\s+NOT\s+EXISTS\s+)?[`"]?(\w+)[`"]?/gi
/** Table names core's own schema.sql declares — a module may not touch these. */
let coreTables = null
function coreTableNames() {
if (coreTables) return coreTables
coreTables = new Set()
try {
const sql = fs.readFileSync(path.join(__dirname, '..', '..', 'db', 'schema.sql'), 'utf8')
for (const m of sql.matchAll(CREATE_TABLE)) coreTables.add(m[1].toLowerCase())
} catch (err) {
log.warn('could not read core schema for the table-collision check', { message: err.message })
}
return coreTables
}
/** Every table name a schema fragment declares. Throws if the file is unreadable. */
function tablesOf(dir, manifest) {
if (!manifest.schema) return new Set()
const sql = fs.readFileSync(path.join(dir, manifest.schema), 'utf8')
return new Set([...sql.matchAll(CREATE_TABLE)].map((m) => m[1].toLowerCase()))
}
function checkTableNames(id, tables) {
const allowed = LEGACY_TABLE_PREFIXES[id] || []
const core = coreTableNames()
for (const table of tables) {
if (core.has(table)) throw new Error(`schema fragment declares core table "${table}"`)
for (const other of modules.values()) {
if (other.tables.has(table)) {
throw new Error(`schema fragment declares "${table}", already owned by module "${other.id}"`)
}
}
const prefixed = table.startsWith(`${id}_`) || allowed.some((p) => table.startsWith(p))
if (!prefixed) throw new Error(`schema fragment table "${table}" is not prefixed "${id}_"`)
}
}
/**
* Does core already own this prefix in this tier?
*
* Asked of the LIVE tier router rather than a hardcoded list, so the check
* cannot drift the first time core adds a capability router — the spike's
* hardcoded table was already one prefix stale when it was written. Modules are
* loaded after every core mount, so the stack is complete by the time this runs,
* and `layer.match` is express's own matcher rather than a second-guess at its
* regexp grammar.
*
* Root-mounted layers are skipped: `use(noindex, requireAuth)` and the two
* `use('/', singletonRouter)` mounts match every path, and counting them would
* report every prefix as taken.
*/
function ownedByCore(tierRouter, prefix) {
return (tierRouter.stack || []).some(
(layer) => layer.regexp && !layer.regexp.fast_slash && layer.match(prefix),
)
}
function readManifest(dir, id, tierRouters) {
const file = path.join(dir, 'module.json')
const manifest = JSON.parse(fs.readFileSync(file, 'utf8'))
for (const key of Object.keys(manifest)) {
// Rejected, not ignored: a typo'd key must be a loud failure rather than a
// silently inert setting the operator believes they configured.
if (!MANIFEST_KEYS.has(key)) throw new Error(`unknown key "${key}" in module.json`)
}
if (!ID.test(manifest.id || '')) throw new Error(`invalid id "${manifest.id}"`)
if (manifest.id !== id) throw new Error(`id "${manifest.id}" does not match directory "${id}"`)
if (!manifest.version) throw new Error('missing version')
if (!manifest.coreApi) throw new Error('missing coreApi')
if (!semver.satisfies(MODULE_API_VERSION, manifest.coreApi)) {
throw new Error(`needs core API ${manifest.coreApi}, this core is ${MODULE_API_VERSION}`)
}
for (const [tier, prefixes] of Object.entries(manifest.mounts || {})) {
if (!TIERS.includes(tier)) throw new Error(`unknown tier "${tier}" in mounts`)
for (const prefix of prefixes) {
if (!PREFIX.test(prefix)) throw new Error(`bad prefix "${prefix}" in mounts.${tier}`)
if (ownedByCore(tierRouters[tier], prefix)) {
throw new Error(`prefix ${tier}${prefix} is owned by core`)
}
for (const other of modules.values()) {
if ((other.manifest.mounts?.[tier] || []).includes(prefix)) {
throw new Error(`prefix ${tier}${prefix} already registered by module "${other.id}"`)
}
}
}
}
for (const slot of manifest.extensions || []) {
if (!CORE_SLOTS.has(slot)) throw new Error(`unknown extension slot "${slot}"`)
}
if (manifest.schema && !manifest.purge) {
// A module that can create tables and cannot drop them leaves an operator
// with orphaned data and no supported way to remove it.
throw new Error('declares schema but no purge')
}
if (manifest.purge && !fs.existsSync(path.join(dir, manifest.purge))) {
throw new Error(`purge file "${manifest.purge}" is missing`)
}
return manifest
}
// What the module registered must equal what it declared — in both directions.
function checkDeclared(record) {
const declared = record.manifest.mounts || {}
for (const tier of TIERS) {
const want = new Set(declared[tier] || [])
const got = new Set(record.routes[tier].keys())
for (const p of got) if (!want.has(p)) throw new Error(`registered ${tier}${p} without declaring it`)
for (const p of want) if (!got.has(p)) throw new Error(`declared ${tier}${p} but never registered it`)
}
}
// ── Load ───────────────────────────────────────────────────────────────────
/**
* Discover, validate, register and mount every module under MODULES_DIR.
*
* **Called exactly once, explicitly, from app.js**, after the three tier routers
* are required and before the app is exported. There is no lazy self-scan: the
* spike's was lazy and silent, so requiring the loader and reading the module
* list gave an empty array and no error (MODULE_API.md §7.6). Everything that
* reads the module list now throws until this has run.
*
* The ordering is not incidental. Core's mounts must already be on the tier
* routers, because that is what the prefix-collision check is asked about; and
* modules mount after them, so first-match-wins means a module could not shadow
* a core prefix even if the check were bypassed.
*
* Safe to call when the modules directory does not exist — that is the normal
* case for a bare core, and it is the state this PR ships in.
*
* @param {{public: Router, admin: Router, player: Router}} tierRouters
*/
function load(tierRouters) {
if (loaded) return
for (const tier of TIERS) {
if (!tierRouters || typeof tierRouters[tier] !== 'function') {
throw new Error(`modules.load: missing the "${tier}" tier router`)
}
}
loaded = true
let entries = []
try {
entries = fs.readdirSync(MODULES_DIR, { withFileTypes: true })
.filter((e) => e.isDirectory())
.map((e) => e.name)
.sort() // alphabetical: there is no dependency resolution, and any other
// order would imply a precedence nothing computes (§4.2)
} catch {
return // no modules directory is the normal case for a bare core
}
for (const id of entries) {
const dir = path.join(MODULES_DIR, id)
if (!fs.existsSync(path.join(dir, 'module.json'))) continue
const record = {
id,
dir,
manifest: null,
routes: { public: new Map(), admin: new Map(), player: new Map() },
tables: new Set(),
called: new Set(),
state: 'installed',
reason: null,
}
try {
record.manifest = readManifest(dir, id, tierRouters)
record.tables = tablesOf(dir, record.manifest)
checkTableNames(id, record.tables)
if (record.manifest.server) {
const entry = path.join(dir, record.manifest.server)
// eslint-disable-next-line global-require, import/no-dynamic-require
const register = require(entry)
if (typeof register !== 'function') throw new Error(`${record.manifest.server} does not export a function`)
register(buildCtx(id, dir), buildApi(record))
checkDeclared(record)
}
record.state = 'registered'
modules.set(id, record)
log.info(`registered module "${id}" v${record.manifest.version}`, {
mounts: record.manifest.mounts,
})
} catch (err) {
// A failure here is BEFORE any route was mounted, so this module's routes
// and nav are simply absent and the site comes up without it (§4.4).
record.state = 'startup_failed'
record.reason = err.message
record.manifest = record.manifest || { id, version: 'unknown' }
modules.set(id, record)
log.error(`module "${id}" failed to load — continuing without it`, { reason: err.message })
}
}
// Mounting is a SECOND pass, after every module has been validated, and not
// because it reads better. `ownedByCore` asks the live tier router what is
// already on it, so mounting inside the loop would make the first module's
// layers indistinguishable from core's — the second module claiming a taken
// prefix would be told it collided with core, naming the wrong culprit, and
// the module-versus-module check below it could never be reached.
for (const record of modules.values()) {
if (record.state === 'registered') mount(record, tierRouters)
}
}
/**
* Mount one module's routers onto the tier routers, behind the dispatch guard.
*
* The guard is the other half of §4.4. A module that fails BEFORE this point has
* no routes at all; one that fails after — schema replay (PR 3), `onBoot`
* (PR 5) — keeps its URLs and answers 503, so `routes.manifest.json` never
* depends on whether a boot hook happened to succeed on the machine that
* generated it. `disabled` is 404 and unreachable until PR 5 wires
* `installed_modules` in; it is written here because the guard is the contract's
* §4.5, not a later addition.
*/
function mount(record, tierRouters) {
for (const tier of TIERS) {
for (const [prefix, router] of record.routes[tier]) {
tierRouters[tier].use(prefix, (req, res, next) => {
if (record.state === 'startup_failed') {
return res.status(503).json({ message: 'Module unavailable' })
}
if (record.state === 'disabled') return res.status(404).json({ message: 'Not found' })
return next()
}, router)
}
}
}
// ── State ──────────────────────────────────────────────────────────────────
// The states a loaded record may hold, deliberately a hardcoded subset rather
// than an import of model/modules/modules.model.js's STATES: that model reaches
// the database, and this file must stay require-able against a dead one.
// `installed` is not here because a record leaves load() resolved either way.
const RECORD_STATES = new Set(['registered', 'started', 'disabled', 'startup_failed'])
/**
* Move a loaded module to a new state — the POST-mount transitions.
*
* Called by whoever ran the step that failed or the step that succeeded, because
* only they can know: PR 3's `ensureSchema()` replays the fragments, PR 5's boot
* dispatch runs `onBoot` and reconciles `installed_modules` (whose `disabled`
* rows are what first make the guard's 404 leg reachable).
*
* Unknown ids are ignored rather than thrown on: a module can be absent from the
* volume and still have a row, and a caller on the boot path must not turn that
* into everyone's failure.
*/
function setState(id, state, reason = null) {
if (!RECORD_STATES.has(state)) throw new Error(`unknown module state "${state}"`)
const record = modules.get(id)
if (!record) return
record.state = state
record.reason = reason
}
// ── Introspection ──────────────────────────────────────────────────────────
function assertLoaded(caller) {
if (!loaded) throw new Error(`modules.${caller}() before modules.load()`)
}
/**
* Every module found on the volume, loaded or failed, in scan order.
*
* Throws rather than returning `[]` when load() has not run — the empty list is
* a real answer for a core with no modules installed, and a caller cannot tell
* the two apart (§7.6).
*/
function list() {
assertLoaded('list')
return [...modules.values()].map((r) => ({
id: r.id,
name: r.manifest.name || r.id,
version: r.manifest.version,
state: r.state,
reason: r.reason,
capabilities: r.manifest.capabilities || [],
}))
}
/** Absolute path of the modules directory. */
const dir = () => MODULES_DIR
module.exports = { load, list, setState, dir }

View File

@@ -0,0 +1,47 @@
// A deliberately tiny semver range check — enough for `coreApi` and no more.
//
// Supports `*`, an exact `x.y.z`, `^x.y.z` and `~x.y.z`. That is the whole
// grammar a module manifest is allowed to use (MODULE_API.md §1.1), so pulling
// in the `semver` package for it would add a dependency to the server for a
// twenty-line job. A range this parser does not understand is REJECTED rather
// than assumed to match — an unparseable range must not silently load a module
// against an API it was never tested on.
const PARTS = /^(\d+)\.(\d+)\.(\d+)$/
function parse(version) {
const m = PARTS.exec(String(version).trim())
if (!m) return null
return { major: Number(m[1]), minor: Number(m[2]), patch: Number(m[3]) }
}
const gte = (a, b) => {
if (a.major !== b.major) return a.major > b.major
if (a.minor !== b.minor) return a.minor > b.minor
return a.patch >= b.patch
}
/**
* Does `version` satisfy `range`?
* @param {string} version an exact x.y.z
* @param {string} range `*` | `x.y.z` | `^x.y.z` | `~x.y.z`
* @returns {boolean} false for anything unparseable, on either side
*/
function satisfies(version, range) {
const v = parse(version)
if (!v) return false
const raw = String(range).trim()
if (raw === '*') return true
const op = raw[0] === '^' || raw[0] === '~' ? raw[0] : ''
const b = parse(op ? raw.slice(1) : raw)
if (!b) return false
if (op === '') return v.major === b.major && v.minor === b.minor && v.patch === b.patch
if (!gte(v, b)) return false
// ^ allows minor+patch within the same major; ~ allows patch within the same minor.
if (op === '^') return v.major === b.major
return v.major === b.major && v.minor === b.minor
}
module.exports = { satisfies, parse }

View File

@@ -0,0 +1,14 @@
// The module API version — the single number a module's `coreApi` range is
// checked against (docs/website/MODULE_API.md §1.1).
//
// Bump minor when a member is ADDED to ctx or a new register* call appears;
// major when one is removed, its signature changes, or its behaviour changes
// without a signature change. A core-internal refactor behind an unchanged
// member is not a bump.
//
// Deliberately separate from PROTOCOL_VERSION (which versions the shard wire and
// has nothing to say about a website module) and from any module's own version.
const MODULE_API_VERSION = '1.0.0'
module.exports = { MODULE_API_VERSION }