feat(modules): the declarative Docker path (phase 4, slice 3)

MODULES declares the module set a deployment runs, one entry per module as
`<id>@<version>=<install manifest URL>`, and the container arrives at it by
itself (MODULE_SYSTEM.md §2.7.2 decision 4). A module already unpacked at the
declared version is a no-op that makes NO network call, so a restart with the
network down comes up unchanged; anything else goes through install.js — same
allowlist, same sha256, same inspect-then-extract — and install() now takes an
`expect: {id, version}` so a URL resolving to another module or version is
refused while it is still only a manifest.

Resolution runs inside start(), between the seed and the require of app.js: the
seed is where the host allowlist setting comes from, and the require is what
scans the volume. That buys it the database, so a compose-installed module gets
the same provenance columns an admin install writes.

A failure is logged and carried, never fatal — an unreachable release host must
not take the site down. The declaration owns what is on the volume; the row owns
whether a module runs, so uninstalling a declared module returns its files at
the next start and leaves it disabled. The admin list gains that as a fourth
source (declared / declaredVersion / declaredError), because a declared module
that failed to resolve has no row, no directory and nothing mounted.

Deferring the app require moved core's schema ahead of the volume scan, and the
module schema-fragment replay was wired to core's schema — so every installed
module silently got no tables. Invisible to the suite (each one stubs the loader
or the pool) and to a smoke on a database that already had the tables; found by
booting against an empty one. ensureSchema() now takes `replayModules: false`
for the one caller that scans later, server.js replays them itself after the
require, and a bootOrder test pins the five steps in the only order they work in.

741 server tests (+18), 187 client (+5); manifest unchanged at 166 public + 2
internal, OpenAPI byte-identical.

Co-Authored-By: Claude <noreply@anthropic.com>

server/src/modules/declared.js

@@ -0,0 +1,220 @@
+// ── The declared module set ────────────────────────────────────────────────
+//
+// Phase 4, slice 3 of docs/website/MODULE_SYSTEM.md §2.7.2 — decision 4. A
+// compose-managed host is not driven by clicking: it declares which modules it
+// runs, in the file it already edits and version-controls, and the container
+// arrives at that set by itself.
+//
+//   MODULES: uo@0.3.0=https://<host>/…/module-uo-0.3.0.json
+//
+// One entry per module, `<id>@<version>=<install manifest URL>`, separated by
+// whitespace or commas. The id and the version are written out rather than left
+// to be discovered inside the manifest for one reason: **the no-op case must not
+// need the network.** A module already unpacked at the declared version is
+// answered by reading its own `module.json` off the volume, so a restart with
+// the network down brings the site up exactly as it was. Only a module that is
+// missing, or unpacked at some other version, reaches out — and it reaches out
+// through modules/install.js, the same fetch-verify-unpack path the admin panel
+// uses, under the same host allowlist.
+//
+// Three things this file deliberately does not do:
+//
+//   - **It does not decide whether a module RUNS.** Resolution owns what is on
+//     the volume; `installed_modules` owns whether a mounted module answers. An
+//     admin who uninstalls a declared module gets its directory back at the next
+//     boot with the row still `disabled`, so it stays off until they enable it.
+//     The two never fight because they are not answering the same question.
+//   - **It does not fail a boot.** A module publisher's host being unreachable
+//     must not take a shard's website down with it; core is built to serve with
+//     a module absent (§1.6). Every failure is logged loudly and kept for the
+//     admin screen, and the site comes up.
+//   - **It does not mount anything.** §1.12 makes the volume the mounting source
+//     of truth, read once at require time — which is why this runs before
+//     `require('./app')` in server.js and not from inside it.
+//
+// It runs on every boot, not only in Docker: a bare `npm start` with MODULES set
+// resolves the same way. The Docker path is the reason it exists, not a special
+// case in it.
+
+const fs = require('fs')
+const path = require('path')
+
+const install = require('./install')
+
+const log = require('../utils/logger')('modules')
+
+// The variable an operator sets. Named next to MODULES_DIR, which is the other
+// half of the same story: one says where modules live, the other says which.
+const VAR = 'MODULES'
+
+// Same id rule the loader enforces when scanning and install.js enforces when
+// placing, restated here so a declaration cannot name something neither would
+// accept.
+const ID = /^[a-z][a-z0-9-]{1,31}$/
+
+// The outcome of the last resolution, in memory, for the admin screen. Not a
+// database row: a declaration is a fact about this process's environment, and
+// writing it down would put it in front of the boot reconcile, which resets
+// every non-disabled row (§2.4). The screen merges it as a fourth source
+// alongside the row, the loader and the volume.
+let results = []
+
+/**
+ * Parse the declaration into entries.
+ *
+ * Malformed entries are collected rather than thrown: one operator typo should
+ * cost that module, not every module on the host. A duplicate id keeps the
+ * first — there is no sensible way to run two versions of one module, and
+ * silently preferring the last would make the outcome depend on the order of a
+ * list nobody reads as ordered.
+ *
+ * @param {string} value the raw variable
+ * @returns {{entries: Array<{id,version,url}>, errors: string[]}}
+ */
+function parse(value) {
+  const entries = []
+  const errors = []
+  const seen = new Set()
+
+  for (const token of String(value || '').split(/[,\s]+/).filter(Boolean)) {
+    const at = token.indexOf('@')
+    const eq = token.indexOf('=')
+    if (at < 1 || eq < at + 2) {
+      errors.push(`"${token}" is not <id>@<version>=<url>`)
+      continue
+    }
+    const id = token.slice(0, at)
+    const version = token.slice(at + 1, eq)
+    const url = token.slice(eq + 1)
+
+    if (!ID.test(id)) {
+      errors.push(`"${token}" names an invalid module id "${id}"`)
+      continue
+    }
+    if (!url) {
+      errors.push(`"${token}" has no install manifest URL`)
+      continue
+    }
+    if (seen.has(id)) {
+      errors.push(`"${id}" is declared more than once — keeping the first`)
+      continue
+    }
+    seen.add(id)
+    entries.push({ id, version, url })
+  }
+
+  return { entries, errors }
+}
+
+/**
+ * The version currently unpacked on the volume, or null.
+ *
+ * Read straight out of the module's own `module.json`, which is the same file
+ * the loader trusts for the same fact — and never from `installed_modules`,
+ * because the row records what was installed and this has to answer what is
+ * actually there. An unreadable manifest counts as absent: whatever is in that
+ * directory, it is not a module at the declared version.
+ */
+function installedVersion(id) {
+  try {
+    const manifest = JSON.parse(
+      fs.readFileSync(path.join(install.moduleDir(id), 'module.json'), 'utf8'),
+    )
+    return manifest && manifest.version ? String(manifest.version) : null
+  } catch {
+    return null
+  }
+}
+
+/**
+ * Bring the volume in line with the declaration.
+ *
+ * Never throws and never rejects. Returns one outcome per declared entry, and
+ * remembers them for `state()`.
+ *
+ * @param {object} args
+ * @param {string} [args.value] the raw variable (defaults to the environment)
+ * @param {string[]} args.hosts the install allowlist, already parsed
+ * @param {object} args.model  modules.model, for recording provenance
+ * @param {object} [args.installImpl] injection seam, as everywhere else here
+ * @returns {Promise<Array<{id,version,url,action,message}>>}
+ */
+async function resolve({ value = process.env[VAR], hosts = [], model, installImpl = install } = {}) {
+  const { entries, errors } = parse(value)
+  results = []
+
+  for (const message of errors) log.error(`${VAR}: ${message}`)
+
+  if (!entries.length) return results
+
+  log.info(`${VAR} declares ${entries.length} module(s)`, {
+    modules: entries.map((e) => `${e.id}@${e.version}`).join(' '),
+  })
+
+  for (const entry of entries) {
+    const present = installedVersion(entry.id)
+    if (present === entry.version) {
+      // The offline path, and the common one: nothing is fetched, nothing is
+      // written, and a host with no route to the internet boots unchanged.
+      log.info(`module "${entry.id}" is already at the declared version ${entry.version}`)
+      results.push({ ...entry, action: 'noop', message: null })
+      continue
+    }
+
+    try {
+      // `expect` is the declaration itself, handed down so install.js can refuse
+      // a URL that resolves to another module or another version while it is
+      // still only a manifest — a check made after the unpack would be made with
+      // the undeclared module already on the volume.
+      const result = await installImpl.install({
+        url: entry.url,
+        hosts,
+        expect: { id: entry.id, version: entry.version },
+      })
+
+      // Provenance, written exactly as the admin route writes it — the whole
+      // point of resolving in-process rather than from a script that cannot
+      // reach the database. A module installed by the compose file and one
+      // installed by an admin are then indistinguishable on the screen, which
+      // is what makes this one feature and not two.
+      if (model) {
+        await model.recordInstalled({
+          id: result.id,
+          name: result.name,
+          version: result.version,
+          source: entry.url,
+          sha256: result.sha256,
+        })
+      }
+
+      log.warn(`installed declared module "${entry.id}" v${entry.version}`, {
+        from: present || 'nothing',
+        source: entry.url,
+        sha256: result.sha256,
+      })
+      results.push({ ...entry, action: 'installed', message: null })
+    } catch (err) {
+      // Loud, and then onward. The site serves; this module does not, or serves
+      // the version that was already there.
+      log.error(
+        `could not resolve declared module "${entry.id}@${entry.version}": ${err.message}` +
+          (present ? ` — leaving version ${present} in place` : ''),
+      )
+      results.push({ ...entry, action: 'failed', message: err.message })
+    }
+  }
+
+  return results
+}
+
+/** What the last resolution decided, for the admin screen. */
+function state() {
+  return results.map((r) => ({ ...r }))
+}
+
+/** Test seam: forget the last resolution. */
+function reset() {
+  results = []
+}
+
+module.exports = { VAR, parse, installedVersion, resolve, state, reset }

server/src/modules/install.js

@@ -79,6 +79,13 @@ class InstallError extends Error {
 
 // ── The allowlist ──────────────────────────────────────────────────────────
 
+// The settings row the allowlist lives in (decision 6): seeded from
+// MODULE_SOURCE_HOSTS on a fresh install and admin-managed from then on. The KEY
+// lives here rather than in the admin controller because it is now read from two
+// places — the controller, and the boot-time resolution of the declared module
+// set (modules/declared.js), which has no route and no request.
+const HOSTS_SETTING = 'module_source_hosts'
+
 /**
  * Parse the stored allowlist setting into hostnames.
  *
@@ -313,13 +320,34 @@ function purgeFile(id) {
  * scratch directory that is removed on any failure, and the move into place is
  * the last step.
  *
+ * `expect` is what the CALLER was promised, as opposed to what the manifest
+ * promises about itself — the declared module set (modules/declared.js) pins an
+ * id and a version in the environment, and a URL that resolves to something else
+ * has to be refused rather than installed. Checked against the manifest, before
+ * a byte is downloaded: catching it after the unpack would mean the undeclared
+ * module is already on the volume when the objection is raised. The admin panel
+ * passes nothing, because there a URL is the whole of what was asked for.
+ *
  * @param {object} args
  * @param {string} args.url    the install manifest URL the admin pasted
  * @param {string[]} args.hosts the allowlist, already parsed
+ * @param {{id?: string, version?: string}} [args.expect] what the caller pinned
  * @returns {Promise<{id,name,version,sha256,source,bytes,replaced}>}
  */
-async function install({ url, hosts, fetchImpl = fetch }) {
+async function install({ url, hosts, expect = null, fetchImpl = fetch }) {
   const manifest = await fetchManifest(url, hosts, fetchImpl)
+
+  if (expect && expect.id && manifest.id !== expect.id) {
+    throw new InstallError(
+      `that URL installs the module "${manifest.id}", but "${expect.id}" was asked for`,
+    )
+  }
+  if (expect && expect.version && manifest.version !== expect.version) {
+    throw new InstallError(
+      `that URL installs ${manifest.id} v${manifest.version}, but v${expect.version} was asked for`,
+    )
+  }
+
   const target = moduleDir(manifest.id)
   const scratch = await fsp.mkdtemp(path.join(loader.dir(), `.install-${manifest.id}-`))
   const tarball = path.join(scratch, 'bundle.tar.gz')
@@ -407,6 +435,7 @@ async function removeDir(id) {
 
 module.exports = {
   InstallError,
+  HOSTS_SETTING,
   parseHosts,
   checkUrl,
   get,