feat(modules): the declarative Docker path (phase 4, slice 3)

MODULES declares the module set a deployment runs, one entry per module as
`<id>@<version>=<install manifest URL>`, and the container arrives at it by
itself (MODULE_SYSTEM.md §2.7.2 decision 4). A module already unpacked at the
declared version is a no-op that makes NO network call, so a restart with the
network down comes up unchanged; anything else goes through install.js — same
allowlist, same sha256, same inspect-then-extract — and install() now takes an
`expect: {id, version}` so a URL resolving to another module or version is
refused while it is still only a manifest.

Resolution runs inside start(), between the seed and the require of app.js: the
seed is where the host allowlist setting comes from, and the require is what
scans the volume. That buys it the database, so a compose-installed module gets
the same provenance columns an admin install writes.

A failure is logged and carried, never fatal — an unreachable release host must
not take the site down. The declaration owns what is on the volume; the row owns
whether a module runs, so uninstalling a declared module returns its files at
the next start and leaves it disabled. The admin list gains that as a fourth
source (declared / declaredVersion / declaredError), because a declared module
that failed to resolve has no row, no directory and nothing mounted.

Deferring the app require moved core's schema ahead of the volume scan, and the
module schema-fragment replay was wired to core's schema — so every installed
module silently got no tables. Invisible to the suite (each one stubs the loader
or the pool) and to a smoke on a database that already had the tables; found by
booting against an empty one. ensureSchema() now takes `replayModules: false`
for the one caller that scans later, server.js replays them itself after the
require, and a bootOrder test pins the five steps in the only order they work in.

741 server tests (+18), 187 client (+5); manifest unchanged at 166 public + 2
internal, OpenAPI byte-identical.

Co-Authored-By: Claude <noreply@anthropic.com>

server/src/modules/declared.js

@@ -0,0 +1,220 @@
+// ── The declared module set ────────────────────────────────────────────────
+//
+// Phase 4, slice 3 of docs/website/MODULE_SYSTEM.md §2.7.2 — decision 4. A
+// compose-managed host is not driven by clicking: it declares which modules it
+// runs, in the file it already edits and version-controls, and the container
+// arrives at that set by itself.
+//
+//   MODULES: uo@0.3.0=https://<host>/…/module-uo-0.3.0.json
+//
+// One entry per module, `<id>@<version>=<install manifest URL>`, separated by
+// whitespace or commas. The id and the version are written out rather than left
+// to be discovered inside the manifest for one reason: **the no-op case must not
+// need the network.** A module already unpacked at the declared version is
+// answered by reading its own `module.json` off the volume, so a restart with
+// the network down brings the site up exactly as it was. Only a module that is
+// missing, or unpacked at some other version, reaches out — and it reaches out
+// through modules/install.js, the same fetch-verify-unpack path the admin panel
+// uses, under the same host allowlist.
+//
+// Three things this file deliberately does not do:
+//
+//   - **It does not decide whether a module RUNS.** Resolution owns what is on
+//     the volume; `installed_modules` owns whether a mounted module answers. An
+//     admin who uninstalls a declared module gets its directory back at the next
+//     boot with the row still `disabled`, so it stays off until they enable it.
+//     The two never fight because they are not answering the same question.
+//   - **It does not fail a boot.** A module publisher's host being unreachable
+//     must not take a shard's website down with it; core is built to serve with
+//     a module absent (§1.6). Every failure is logged loudly and kept for the
+//     admin screen, and the site comes up.
+//   - **It does not mount anything.** §1.12 makes the volume the mounting source
+//     of truth, read once at require time — which is why this runs before
+//     `require('./app')` in server.js and not from inside it.
+//
+// It runs on every boot, not only in Docker: a bare `npm start` with MODULES set
+// resolves the same way. The Docker path is the reason it exists, not a special
+// case in it.
+
+const fs = require('fs')
+const path = require('path')
+
+const install = require('./install')
+
+const log = require('../utils/logger')('modules')
+
+// The variable an operator sets. Named next to MODULES_DIR, which is the other
+// half of the same story: one says where modules live, the other says which.
+const VAR = 'MODULES'
+
+// Same id rule the loader enforces when scanning and install.js enforces when
+// placing, restated here so a declaration cannot name something neither would
+// accept.
+const ID = /^[a-z][a-z0-9-]{1,31}$/
+
+// The outcome of the last resolution, in memory, for the admin screen. Not a
+// database row: a declaration is a fact about this process's environment, and
+// writing it down would put it in front of the boot reconcile, which resets
+// every non-disabled row (§2.4). The screen merges it as a fourth source
+// alongside the row, the loader and the volume.
+let results = []
+
+/**
+ * Parse the declaration into entries.
+ *
+ * Malformed entries are collected rather than thrown: one operator typo should
+ * cost that module, not every module on the host. A duplicate id keeps the
+ * first — there is no sensible way to run two versions of one module, and
+ * silently preferring the last would make the outcome depend on the order of a
+ * list nobody reads as ordered.
+ *
+ * @param {string} value the raw variable
+ * @returns {{entries: Array<{id,version,url}>, errors: string[]}}
+ */
+function parse(value) {
+  const entries = []
+  const errors = []
+  const seen = new Set()
+
+  for (const token of String(value || '').split(/[,\s]+/).filter(Boolean)) {
+    const at = token.indexOf('@')
+    const eq = token.indexOf('=')
+    if (at < 1 || eq < at + 2) {
+      errors.push(`"${token}" is not <id>@<version>=<url>`)
+      continue
+    }
+    const id = token.slice(0, at)
+    const version = token.slice(at + 1, eq)
+    const url = token.slice(eq + 1)
+
+    if (!ID.test(id)) {
+      errors.push(`"${token}" names an invalid module id "${id}"`)
+      continue
+    }
+    if (!url) {
+      errors.push(`"${token}" has no install manifest URL`)
+      continue
+    }
+    if (seen.has(id)) {
+      errors.push(`"${id}" is declared more than once — keeping the first`)
+      continue
+    }
+    seen.add(id)
+    entries.push({ id, version, url })
+  }
+
+  return { entries, errors }
+}
+
+/**
+ * The version currently unpacked on the volume, or null.
+ *
+ * Read straight out of the module's own `module.json`, which is the same file
+ * the loader trusts for the same fact — and never from `installed_modules`,
+ * because the row records what was installed and this has to answer what is
+ * actually there. An unreadable manifest counts as absent: whatever is in that
+ * directory, it is not a module at the declared version.
+ */
+function installedVersion(id) {
+  try {
+    const manifest = JSON.parse(
+      fs.readFileSync(path.join(install.moduleDir(id), 'module.json'), 'utf8'),
+    )
+    return manifest && manifest.version ? String(manifest.version) : null
+  } catch {
+    return null
+  }
+}
+
+/**
+ * Bring the volume in line with the declaration.
+ *
+ * Never throws and never rejects. Returns one outcome per declared entry, and
+ * remembers them for `state()`.
+ *
+ * @param {object} args
+ * @param {string} [args.value] the raw variable (defaults to the environment)
+ * @param {string[]} args.hosts the install allowlist, already parsed
+ * @param {object} args.model  modules.model, for recording provenance
+ * @param {object} [args.installImpl] injection seam, as everywhere else here
+ * @returns {Promise<Array<{id,version,url,action,message}>>}
+ */
+async function resolve({ value = process.env[VAR], hosts = [], model, installImpl = install } = {}) {
+  const { entries, errors } = parse(value)
+  results = []
+
+  for (const message of errors) log.error(`${VAR}: ${message}`)
+
+  if (!entries.length) return results
+
+  log.info(`${VAR} declares ${entries.length} module(s)`, {
+    modules: entries.map((e) => `${e.id}@${e.version}`).join(' '),
+  })
+
+  for (const entry of entries) {
+    const present = installedVersion(entry.id)
+    if (present === entry.version) {
+      // The offline path, and the common one: nothing is fetched, nothing is
+      // written, and a host with no route to the internet boots unchanged.
+      log.info(`module "${entry.id}" is already at the declared version ${entry.version}`)
+      results.push({ ...entry, action: 'noop', message: null })
+      continue
+    }
+
+    try {
+      // `expect` is the declaration itself, handed down so install.js can refuse
+      // a URL that resolves to another module or another version while it is
+      // still only a manifest — a check made after the unpack would be made with
+      // the undeclared module already on the volume.
+      const result = await installImpl.install({
+        url: entry.url,
+        hosts,
+        expect: { id: entry.id, version: entry.version },
+      })
+
+      // Provenance, written exactly as the admin route writes it — the whole
+      // point of resolving in-process rather than from a script that cannot
+      // reach the database. A module installed by the compose file and one
+      // installed by an admin are then indistinguishable on the screen, which
+      // is what makes this one feature and not two.
+      if (model) {
+        await model.recordInstalled({
+          id: result.id,
+          name: result.name,
+          version: result.version,
+          source: entry.url,
+          sha256: result.sha256,
+        })
+      }
+
+      log.warn(`installed declared module "${entry.id}" v${entry.version}`, {
+        from: present || 'nothing',
+        source: entry.url,
+        sha256: result.sha256,
+      })
+      results.push({ ...entry, action: 'installed', message: null })
+    } catch (err) {
+      // Loud, and then onward. The site serves; this module does not, or serves
+      // the version that was already there.
+      log.error(
+        `could not resolve declared module "${entry.id}@${entry.version}": ${err.message}` +
+          (present ? ` — leaving version ${present} in place` : ''),
+      )
+      results.push({ ...entry, action: 'failed', message: err.message })
+    }
+  }
+
+  return results
+}
+
+/** What the last resolution decided, for the admin screen. */
+function state() {
+  return results.map((r) => ({ ...r }))
+}
+
+/** Test seam: forget the last resolution. */
+function reset() {
+  results = []
+}
+
+module.exports = { VAR, parse, installedVersion, resolve, state, reset }

server/src/modules/install.js

@@ -79,6 +79,13 @@ class InstallError extends Error {
 
 // ── The allowlist ──────────────────────────────────────────────────────────
 
+// The settings row the allowlist lives in (decision 6): seeded from
+// MODULE_SOURCE_HOSTS on a fresh install and admin-managed from then on. The KEY
+// lives here rather than in the admin controller because it is now read from two
+// places — the controller, and the boot-time resolution of the declared module
+// set (modules/declared.js), which has no route and no request.
+const HOSTS_SETTING = 'module_source_hosts'
+
 /**
  * Parse the stored allowlist setting into hostnames.
  *
@@ -313,13 +320,34 @@ function purgeFile(id) {
  * scratch directory that is removed on any failure, and the move into place is
  * the last step.
  *
+ * `expect` is what the CALLER was promised, as opposed to what the manifest
+ * promises about itself — the declared module set (modules/declared.js) pins an
+ * id and a version in the environment, and a URL that resolves to something else
+ * has to be refused rather than installed. Checked against the manifest, before
+ * a byte is downloaded: catching it after the unpack would mean the undeclared
+ * module is already on the volume when the objection is raised. The admin panel
+ * passes nothing, because there a URL is the whole of what was asked for.
+ *
  * @param {object} args
  * @param {string} args.url    the install manifest URL the admin pasted
  * @param {string[]} args.hosts the allowlist, already parsed
+ * @param {{id?: string, version?: string}} [args.expect] what the caller pinned
  * @returns {Promise<{id,name,version,sha256,source,bytes,replaced}>}
  */
-async function install({ url, hosts, fetchImpl = fetch }) {
+async function install({ url, hosts, expect = null, fetchImpl = fetch }) {
   const manifest = await fetchManifest(url, hosts, fetchImpl)
+
+  if (expect && expect.id && manifest.id !== expect.id) {
+    throw new InstallError(
+      `that URL installs the module "${manifest.id}", but "${expect.id}" was asked for`,
+    )
+  }
+  if (expect && expect.version && manifest.version !== expect.version) {
+    throw new InstallError(
+      `that URL installs ${manifest.id} v${manifest.version}, but v${expect.version} was asked for`,
+    )
+  }
+
   const target = moduleDir(manifest.id)
   const scratch = await fsp.mkdtemp(path.join(loader.dir(), `.install-${manifest.id}-`))
   const tarball = path.join(scratch, 'bundle.tar.gz')
@@ -407,6 +435,7 @@ async function removeDir(id) {
 
 module.exports = {
   InstallError,
+  HOSTS_SETTING,
   parseHosts,
   checkUrl,
   get,

server/src/router/v1/admin/modules.controller.js

@@ -30,6 +30,7 @@ const settings = require('../../../model/settings/settings.model')
 const loader = require('../../../modules/loader')
 const lifecycle = require('../../../modules/lifecycle')
 const install = require('../../../modules/install')
+const declared = require('../../../modules/declared')
 // A namespace import, like every other require in this file, and not
 // `const { runPurge } = …`: destructuring at require time captures the function
 // rather than the module, which makes it the one dependency here that cannot be
@@ -42,8 +43,9 @@ const log = require('../../../utils/logger')('admin-modules')
 // The allowlist setting. Seeded from MODULE_SOURCE_HOSTS on first boot and
 // admin-managed from then on (decision 6) — db/seed.js writes it once and never
 // overwrites it, so changing the variable later does not silently reach in and
-// undo an operator's choice.
-const HOSTS_KEY = 'module_source_hosts'
+// undo an operator's choice. The key itself lives in install.js, which is now
+// read by boot-time declared-set resolution as well as by this controller.
+const HOSTS_KEY = install.HOSTS_SETTING
 
 // A hostname, not a URL: no scheme, no path, no port, no wildcard. Deliberately
 // strict — every character allowed here is a character that can appear in the
@@ -57,24 +59,29 @@ async function allowedHosts() {
 /**
  * One module, as the admin screen needs it.
  *
- * Three sources have to be reconciled, and which one answers which question is
+ * Four sources have to be reconciled, and which one answers which question is
  * the whole of §2.4:
  *
  *   - the ROW says what the operator decided and what the last boot recorded;
  *   - the LOADER says what is mounted and answering right now;
- *   - the VOLUME says whether there is still a directory there at all.
+ *   - the VOLUME says whether there is still a directory there at all;
+ *   - the DECLARATION (slice 3) says what this container's environment asks for,
+ *     which is the only one of the four an admin cannot change from this screen.
  *
  * They can legitimately disagree, and the screen has to show that rather than
  * pick a winner. A row `enabled` with a loader state of `disabled` is a module
  * the operator has just switched back on and which is waiting for a restart —
  * exactly the case decision 3 creates, and it would be a lie to render it as
- * either "running" or "off".
+ * either "running" or "off". A declared module with no row and no directory is
+ * the newest of those disagreements: MODULES asked for it and resolution could
+ * not get it, so the screen carries the reason rather than showing nothing.
  */
-function present(row, live, onVolume) {
+function present(row, live, onVolume, declaration = null) {
+  const id = row ? row.id : live ? live.id : declaration.id
   return {
-    id: row ? row.id : live.id,
-    name: row ? row.name : live.name,
-    version: row ? row.version : live.version,
+    id,
+    name: row ? row.name : live ? live.name : id,
+    version: row ? row.version : live ? live.version : null,
     // What the database records.
     state: row ? row.state : null,
     failureStage: row ? row.failureStage : (live && live.stage) || null,
@@ -93,7 +100,16 @@ function present(row, live, onVolume) {
     capabilities: live ? live.capabilities : [],
     // What is on the volume.
     onVolume,
-    canPurge: onVolume && Boolean(install.purgeFile(row ? row.id : live.id)),
+    canPurge: onVolume && Boolean(install.purgeFile(id)),
+    // What the environment declares. `declaredVersion` is what MODULES pins, not
+    // what is installed — they differ exactly while a resolution is failing, and
+    // `declaredError` says why. Uninstalling a declared module from this screen
+    // removes its directory and disables its row; the next boot puts the
+    // directory back and leaves the row disabled, so the screen says so rather
+    // than letting the files reappear unexplained.
+    declared: Boolean(declaration),
+    declaredVersion: declaration ? declaration.version : null,
+    declaredError: declaration && declaration.action === 'failed' ? declaration.message : null,
   }
 }
 
@@ -107,18 +123,31 @@ async function list(req, res) {
     // `npm run seed` never gets here, but a test harness might.
     const live = loader.isLoaded() ? loader.list() : []
     const byId = new Map(live.map((m) => [m.id, m]))
+    const declaredById = new Map(declared.state().map((d) => [d.id, d]))
 
     const seen = new Set()
     const out = []
     for (const row of rows) {
       seen.add(row.id)
-      out.push(present(row, byId.get(row.id) || null, install.isInstalled(row.id)))
+      out.push(
+        present(row, byId.get(row.id) || null, install.isInstalled(row.id), declaredById.get(row.id)),
+      )
     }
     // A directory on the volume that has no row yet — a hand-placed install
     // before its first boot. It has to be listed, or the screen would show
     // nothing for a module whose routes are already being served.
     for (const m of live) {
-      if (!seen.has(m.id)) out.push(present(null, m, true))
+      if (!seen.has(m.id)) {
+        seen.add(m.id)
+        out.push(present(null, m, true, declaredById.get(m.id)))
+      }
+    }
+    // A module MODULES declares that has neither. Resolution failed and left
+    // nothing behind — the case an operator most needs told, because from the
+    // screen's other three sources it is indistinguishable from never having
+    // asked for it.
+    for (const d of declaredById.values()) {
+      if (!seen.has(d.id)) out.push(present(null, null, install.isInstalled(d.id), d))
     }
 
     return res.json({ modules: out, sourceHosts: await allowedHosts() })

server/src/server.js

@@ -1,8 +1,12 @@
 require('dotenv').config()
 const http = require('http')
 
-const app = require('./app')
-const internalApp = require('./internalApp')
+// NOTE: `./app` and `./internalApp` are deliberately NOT required here. Requiring
+// app.js runs `modules.load()`, which scans the volume and mounts whatever is on
+// it (MODULE_API.md §4.1) — so the declared module set has to be resolved before
+// that require, not before the listener. They are required inside start(), after
+// resolveDeclaredModules(); everything else this file needs is safe to pull in
+// now because none of it reaches the loader's scan.
 const botScore = require('./middleware/botScore')
 const announceWorker = require('./utils/announceWorker')
 const { ensureSchema, close } = require('./utils/db')
@@ -11,6 +15,9 @@ const settings = require('./model/settings/settings.model')
 const revokedSessions = require('./model/revokedSessions/revokedSessions.model')
 const mobileAuthBridge = require('./model/mobileAuthBridge/mobileAuthBridge.model')
 const moduleLifecycle = require('./modules/lifecycle')
+const declaredModules = require('./modules/declared')
+const moduleInstall = require('./modules/install')
+const moduleModel = require('./model/modules/modules.model')
 const createLogger = require('./utils/logger')
 const { evaluateBotInternalKey } = require('./utils/botInternalKey')
 const brand = require('./config/brand')
@@ -51,7 +58,9 @@ async function start() {
   }
 
   log.info('ensuring database schema...')
-  await ensureSchema()
+  // Core's schema only. Each installed module's fragment is replayed further
+  // down, after the volume has been scanned — see the require of ./app below.
+  await ensureSchema({ replayModules: false })
   log.info('seeding defaults...')
   await seedDefaults()
   await createInitialAdminFromEnv()
@@ -77,6 +86,36 @@ async function start() {
   const mode = await settings.get('site_mode')
   log.info(`site mode: ${String(mode || 'live').toUpperCase()}`)
 
+  // Bring the modules volume in line with what MODULES declares (§2.7.2
+  // decision 4), and only then require the app — the loader scans and mounts at
+  // require time, so this is the last moment at which a module can be put on the
+  // volume and still be part of this process.
+  //
+  // After the schema and the seed, because the host allowlist it installs under
+  // is a settings row that the seed creates on a fresh instance. Never throws:
+  // an unreachable release host leaves the site serving without that module
+  // rather than taking the site down with it.
+  await declaredModules.resolve({
+    hosts: moduleInstall.parseHosts(await settings.get(moduleInstall.HOSTS_SETTING)),
+    model: moduleModel,
+  })
+
+  // Requiring app.js is what scans the volume and mounts what is on it. Every
+  // line above this one runs against a core that has no modules in it yet.
+  // eslint-disable-next-line global-require
+  const app = require('./app')
+  // eslint-disable-next-line global-require
+  const internalApp = require('./internalApp')
+
+  // Now that the scan has happened, replay each module's schema fragment
+  // (MODULE_API.md §2.6). This used to ride inside ensureSchema() and could,
+  // because app.js was required at the top of this file; resolving the declared
+  // set first moved the scan after it, and a booting server quietly getting no
+  // module tables is precisely what §7.6 warns about. Caught by the browser
+  // smoke rather than by a test: every suite here stubs one side or the other.
+  // eslint-disable-next-line global-require
+  await require('./modules/schema').replayFragments()
+
   // Reconcile installed_modules with what the loader found on the volume at
   // require time, then run each module's onBoot (MODULE_API.md §2.5).
   //

server/src/utils/db.js

@@ -55,11 +55,18 @@ const SCHEMA_PATH = path.join(__dirname, '..', '..', 'db', 'schema.sql')
  * below — the discovery, splitting and per-module failure handling all live in
  * modules/schema.js, required lazily so that requiring the pool never drags the
  * loader in with it.
+ *
+ * `replayModules: false` is for a caller that has not scanned the volume YET and
+ * intends to. server.js is the one: since slice 3 it resolves the declared
+ * module set before requiring app.js, which puts core's schema *before* the scan
+ * — so it replays the fragments itself, in the one place that knows the scan has
+ * happened. Left true everywhere else, so the ordinary caller cannot get module
+ * tables by accident and lose them by refactor.
  */
-async function ensureSchema({ retries = 10, delayMs = 2000 } = {}) {
+async function ensureSchema({ retries = 10, delayMs = 2000, replayModules = true } = {}) {
   await ensureCoreSchema({ retries, delayMs })
   // eslint-disable-next-line global-require
-  await require('../modules/schema').replayFragments()
+  if (replayModules) await require('../modules/schema').replayFragments()
 }
 
 /** Core's own schema.sql, with the wait-for-the-database retry. */