feat(modules): boot/shutdown hook dispatch and the installed_modules reconcile

Phase 2, PR 5 of docs/website/MODULE_SYSTEM.md 2.7. api.onBoot/api.onShutdown
stop throwing, server.js gains one call on each side, and the 2.4 state machine
finally runs against real outcomes -- which is what makes 4.5's `disabled` 404
leg reachable for the first time.

Dispatch and reconcile live in src/modules/lifecycle.js rather than in the
loader, for the reason the schema replay does: routeManifest.js and swagger.js
both require app.js against a dead pool, so the loader may not reach the
database. The two halves meet at exactly one function, loader.setState(), so the
in-memory record the dispatch guard reads and the row the admin panel reads are
moved together and cannot disagree.

Four decisions, all recorded in MODULE_API.md 2.5 and 4.4:

- The loader classifies its failures by 4.3 step, so failure_stage says where a
  module broke instead of being a column nothing ever filled. The four steps
  readManifest covers in one pass label themselves; the rest are inferred from
  how far load() had got, and an unlabelled throw is recorded against the step
  that was running rather than guessed at.
- A row whose directory is gone is marked startup_failed rather than left
  claiming `enabled` -- the boot reset has just moved it there, and a row
  claiming to be enabled for a module that is not on the volume is the one state
  that is simply untrue. An uninstall leaves `disabled`, which the reset never
  touches, so this catches only a hand-deleted directory.
- Core's eight UO boot call sites stay in server.js until Phase 3. Unlike a
  registered announce leg, a boot call site already has somewhere to live, so
  moving it now would be extraction done early in a phase whose exit criterion
  is that nothing changes.
- onBoot gets no timeout. Shutdown races a SIGKILL and boot does not, and a slow
  onBoot delaying the listener is the contract's promise to a module that must
  warm up before it serves.

The operator's switch wins over everything: a disabled module is guarded, not
booted, and does not have its failure re-recorded, or an outcome would silently
switch it back on next boot. Every database write in the reconcile is
individually caught -- a row that will not update is worse reporting, never a
failed boot.

900 tests pass (17 new). routes.manifest.json is unchanged at 229 routes and the
OpenAPI spec regenerates byte-identical.

Co-Authored-By: Claude <noreply@anthropic.com>

server/src/modules/lifecycle.js

@@ -0,0 +1,208 @@
+// ── Module lifecycle dispatch ──────────────────────────────────────────────
+//
+// Phase 2, PR 5 of docs/website/MODULE_SYSTEM.md §2.7. Normative contract:
+// docs/website/MODULE_API.md §2.5 (the hooks and when they run) and §4.4
+// (failure is a state), plus MODULE_SYSTEM.md §2.4 (what a boot does to
+// `installed_modules`).
+//
+// This is the database half of the loader, and it is a separate file for the
+// same reason modules/schema.js is: `scripts/routeManifest.js` and
+// `swagger/swagger.js` both require app.js with the pool pointed at a dead port,
+// so loader.js may not reach the database. Everything here runs from server.js,
+// after ensureSchema() has proved the database is up.
+//
+// The two halves meet at exactly one place — `loader.setState()` — so the
+// in-memory record that the §4.5 dispatch guard reads and the row the admin
+// panel reads are moved together and cannot disagree.
+//
+// Two properties this file exists to keep:
+//
+//   1. **A module's boot failure costs that module and nothing else** (§4.4).
+//      Its routes stay mounted and answer 503; the site comes up; the next
+//      module boots as if nothing happened.
+//   2. **The row is a record of what happened, never the source of truth for
+//      what is mounted** (§2.4). Nothing here mounts, unmounts or re-scans. It
+//      reads the outcome of a scan that already happened and writes it down.
+
+const log = require('../utils/logger')('modules')
+
+// §2.5: shutdown races the process being killed, so a module that will not let
+// go is logged and skipped rather than allowed to hang the exit. `onBoot` has no
+// such budget on purpose — it delays the listener binding, which is the feature.
+const SHUTDOWN_BUDGET_MS = 5000
+
+/**
+ * Run one database call for one module without letting it become everyone's
+ * failure. Returns null on failure, having logged it.
+ *
+ * The boot path is the whole reason this exists. A row that will not update is
+ * bad — the admin panel shows the wrong thing — but it is strictly less bad than
+ * a site that will not start, and it must not stop the modules after it from
+ * booting.
+ */
+async function safe(what, fn) {
+  try {
+    return await fn()
+  } catch (err) {
+    log.error(`module bookkeeping failed: ${what}`, { error: err.message })
+    return null
+  }
+}
+
+/**
+ * Reconcile `installed_modules` with what the loader found, then run every
+ * surviving module's `onBoot`.
+ *
+ * Called once from server.js, after `ensureSchema()` (so a module's own tables
+ * exist) and `seedDefaults()`, and **before the HTTP listener binds** — a module
+ * that must not serve traffic until it has warmed a cache gets that for free
+ * (§2.5).
+ *
+ * The order of the four steps is the whole design:
+ *
+ *   1. `beginBoot()` clears the last boot's outcomes, so what is on display
+ *      afterwards is what THIS boot did. `disabled` rows are left alone: that is
+ *      an operator decision, not an outcome (§2.4).
+ *   2. Every module on the volume gets a row, written with null provenance if it
+ *      does not have one — a directory placed on the volume by hand is a
+ *      supported install (§2.4/§2.5), and without a row it could never be
+ *      disabled or shown as failed.
+ *   3. Rows with no directory are marked failed, because step 1 has just reset
+ *      them to `enabled` and a row claiming to be enabled for a module that is
+ *      not there is the one state that is simply untrue. (A plain uninstall
+ *      leaves `disabled`, which step 1 never touches, so this only catches a
+ *      directory deleted by hand.)
+ *   4. The outcome each module already carries — disabled by the operator,
+ *      failed during load or schema replay, or ready — is written down, and only
+ *      then is `onBoot` dispatched.
+ *
+ * Never throws. @param {object} [deps] injection seam for tests.
+ */
+async function boot({ modules, model } = {}) {
+  /* eslint-disable global-require */
+  const loader = modules || require('./loader')
+  const rows = model || require('../model/modules/modules.model')
+  /* eslint-enable global-require */
+
+  // Not an error, and the same guard replayFragments carries: a process that
+  // never required app.js has no scan to reconcile against, and writing rows
+  // from an empty list would mark every installed module as missing.
+  if (!loader.isLoaded()) {
+    log.info('no module scan in this process — skipping module boot')
+    return
+  }
+
+  const scanned = loader.list()
+
+  await safe('resetting last boot\'s outcomes', () => rows.beginBoot())
+
+  for (const m of scanned) {
+    await safe(`recording module "${m.id}"`, () => rows.recordInstalled({
+      id: m.id,
+      name: m.name,
+      version: m.version,
+      // Null provenance is what a hand-placed directory looks like. An install
+      // performed through the admin panel (§2.5, a later phase) writes the row
+      // with its source and hash first; this refresh deliberately does not
+      // overwrite either, because recordInstalled leaves what it is not given.
+    }))
+  }
+
+  const stored = (await safe('reading module rows', () => rows.list())) || []
+  const onVolume = new Set(scanned.map((m) => m.id))
+
+  for (const row of stored) {
+    if (onVolume.has(row.id) || row.state === 'disabled') continue
+    await safe(`marking module "${row.id}" missing`, () => rows.markStartupFailed(row.id, {
+      stage: 'require',
+      reason: 'module directory not present on the volume',
+    }))
+  }
+
+  const disabled = new Set(stored.filter((r) => r.state === 'disabled').map((r) => r.id))
+
+  for (const m of scanned) {
+    // The operator's switch wins over everything, including a failure. Its
+    // routes 404 from here on (§4.5 — the leg that was unreachable until this
+    // PR), it is not booted, and its failure is not re-recorded: overwriting a
+    // deliberate `disabled` with an outcome would silently re-enable it on the
+    // next boot.
+    if (disabled.has(m.id)) {
+      loader.setState(m.id, 'disabled')
+      continue
+    }
+    if (m.state === 'startup_failed') {
+      // Already failed in load() or the schema replay — both of which ran before
+      // the database was available to write it down. This is where it lands.
+      await safe(`recording failure for "${m.id}"`, () => rows.markStartupFailed(m.id, {
+        stage: m.stage,
+        reason: m.reason,
+      }))
+    }
+  }
+
+  for (const { id, hook, ctx } of loader.bootable()) {
+    try {
+      // Awaited without a timeout, deliberately (§2.5): a slow onBoot delays the
+      // listener, which is the contract's promise to a module that must warm up
+      // before it serves. Core's own boot steps are awaited the same way.
+      if (hook) await hook(ctx)
+      loader.setState(id, 'started')
+      await safe(`marking module "${id}" started`, () => rows.markStarted(id))
+      log.info(`module "${id}" started`)
+    } catch (err) {
+      // §4.4's second column: the routes are already mounted, so they stay
+      // mounted and answer 503. A module that failed to warm up serving
+      // half-initialised data is worse than one that says it is down.
+      loader.setState(id, 'startup_failed', { stage: 'boot', reason: err.message })
+      await safe(`recording boot failure for "${id}"`, () => rows.markStartupFailed(id, {
+        stage: 'boot',
+        reason: err.message,
+      }))
+      log.error(`module "${id}" onBoot failed — its routes will answer 503`, {
+        reason: err.message,
+      })
+    }
+  }
+}
+
+/** Reject if `fn`'s promise has not settled within `ms`. */
+function withBudget(fn, ms) {
+  let timer
+  const budget = new Promise((_, reject) => {
+    timer = setTimeout(() => reject(new Error(`onShutdown exceeded its ${ms}ms budget`)), ms)
+  })
+  return Promise.race([Promise.resolve().then(fn), budget]).finally(() => clearTimeout(timer))
+}
+
+/**
+ * Run every started module's `onShutdown`, in reverse registration order.
+ *
+ * Called from server.js's signal handler before anything core owns is closed, so
+ * a module still has a working database pool and push dispatcher to flush
+ * through. Reverse order is the mirror of boot order: a module that booted after
+ * another may be holding something the earlier one handed it.
+ *
+ * Never throws, and never hangs: each hook gets `SHUTDOWN_BUDGET_MS`, after
+ * which it is logged and abandoned. Abandoned, not cancelled — nothing can stop
+ * a promise that is still running — but the process is exiting anyway, and the
+ * alternative is a shard host where `systemctl stop` hangs until SIGKILL.
+ */
+async function shutdown({ modules, budgetMs = SHUTDOWN_BUDGET_MS } = {}) {
+  // eslint-disable-next-line global-require
+  const loader = modules || require('./loader')
+  if (!loader.isLoaded()) return
+
+  for (const { id, hook } of loader.shutdownHooks()) {
+    try {
+      await withBudget(hook, budgetMs)
+      log.info(`module "${id}" shut down`)
+    } catch (err) {
+      log.warn(`module "${id}" onShutdown failed or timed out — continuing`, {
+        error: err.message,
+      })
+    }
+  }
+}
+
+module.exports = { boot, shutdown, SHUTDOWN_BUDGET_MS }