feat(modules): the three de-entanglement registries, with core as the registrant

Phase 2 PR 4 of docs/website/MODULE_SYSTEM.md §2.7. Adds server/src/modules/registries.js
and moves core's own notification streams, announce leg and users-detail routes
behind it, so the three seams §1.8 and §1.9 named are exercised on every boot
before any module depends on them.

Registering is validate-then-commit per registrant: the loader stages what a
module claims and the second pass commits it, so a module that throws halfway
through register() — or fails checkDeclared after it — leaves nothing behind.
That is the registry-side twin of PR 2's second-pass mount rule.

Four decisions, all the recommended option:

- announce legs became a child table. `announce_job_legs` replaces the
  towncrier_*/discord_* column groups, so the leg set is data: core registers
  `discord`, module-uo will register `towncrier`, and a module cannot ALTER a
  core table to add its own. Backfill is guarded on information_schema (a
  SELECT of a dropped column is a parse error, not a runtime one) and the
  columns go with DROP COLUMN IF EXISTS. Verified against the live dev DB:
  three legacy jobs migrated faithfully, three replays, no duplicates.
- `mapEvent` dropped from registerNotificationStreams. §1.8 already inverts the
  push path so a module owns fromShardEvent and calls core's publish() with a
  stream id it resolved; a second mapping mechanism was a leftover. The public
  safety filter, the kinds it reads and the streams it protects now live in one
  file and move together.
- core registers through the same staging area a module uses, via an explicit
  registries.registerCore() in app.js before modules.load().
- core's six /admin/users/:id/shard/* paths now go through the
  `admin.users.detail` slot, and getUser moved back to admin.controller.js.

Found on the way, and the reason two build tools changed:

- scripts/routeManifest.js could not decode a parameterised mount. Its
  unwinder expected `(?:([^\/]+?))`; express 4.22 emits `(?:\/([^/]+?))` with
  the separator inside the group. The branch had never run. It threw rather
  than guessing, which is what it is for.
- swagger-autogen cannot follow a route into an extension slot — the slot's
  router is created by declareSlot() and filled later, so there is no literal
  mount for a static parse. Regenerating deleted 407 lines and printed
  `Swagger-autogen: Success`, the spike's exact failure (MODULE_API.md §7.4).
  swagger/slotSpecs.js generates a fragment per filled slot and re-roots it at
  the prefix the router actually hangs at in the live app — read from the
  express stack via routeManifest's own mountPath, so the manifest and the spec
  cannot disagree. swagger/mergeSpec.js is the merge helper core owes for
  module fragments anyway (§6.1a), proved here against core's own slot first.

884 tests pass (856 before). routes.manifest.json is unchanged at 229 routes.
The OpenAPI spec diff is two lines of intent: the retry endpoint's summary, and
its `leg` no longer being a fixed enum.

Co-Authored-By: Claude <noreply@anthropic.com>

server/swagger/mergeSpec.js

@@ -0,0 +1,115 @@
+// ── Merging an OpenAPI fragment into a spec ────────────────────────────────
+//
+// The merge half of docs/website/MODULE_API.md §6.1's settled decision: routes
+// that reach the app through something swagger-autogen cannot statically follow
+// contribute a FRAGMENT, and core merges it.
+//
+// Two callers, one function, deliberately:
+//   • build time — swagger/slotSpecs.js, for core's own extension-slot routers
+//     (§1.9). Those are core routes that a static parse of app.js cannot see,
+//     because the slot's router is created by registries.declareSlot() and filled
+//     later. They belong in the committed `swagger-output.json`.
+//   • request time (Phase 2 PR 6+) — an installed module's `swagger-fragment.json`,
+//     merged over the committed spec for `/api/docs.json`.
+//
+// **Core always wins a key collision** (§6.1a). A fragment cannot redefine a path,
+// a tag or a schema core already declares; the collision is reported and the
+// fragment's version dropped. Merging is shallow-per-section — `paths`, `tags`
+// and `components.schemas` — because those are the only three sections a fragment
+// is allowed to carry, and a deeper merge would let a fragment reach into
+// `info`, `servers` or the security schemes.
+
+/**
+ * Merge `fragment` into `spec`, in place, with core winning every collision.
+ *
+ * @param {object} spec      the base spec — mutated
+ * @param {object} fragment  `{ paths?, tags?, components?: { schemas? } }`
+ * @param {string} source    who the fragment came from, for the collision message
+ * @returns {string[]} the collisions that were dropped (empty when clean)
+ */
+function mergeFragment(spec, fragment, source) {
+  const dropped = []
+
+  for (const [path, item] of Object.entries(fragment.paths || {})) {
+    if (spec.paths[path]) {
+      // Not a merge of the two path items: a fragment adding a METHOD to a core
+      // path is the same overreach as replacing it, and the extension-slot
+      // contract already says core owns the resource (§2.4).
+      dropped.push(`path ${path}`)
+      continue
+    }
+    spec.paths[path] = item
+  }
+
+  const tagNames = new Set((spec.tags || []).map((t) => t.name))
+  for (const tag of fragment.tags || []) {
+    if (tagNames.has(tag.name)) continue // same tag, not a collision worth reporting
+    spec.tags.push(tag)
+    tagNames.add(tag.name)
+  }
+
+  const schemas = (fragment.components && fragment.components.schemas) || {}
+  for (const [name, schema] of Object.entries(schemas)) {
+    if (spec.components.schemas[name]) {
+      dropped.push(`schema ${name}`)
+      continue
+    }
+    spec.components.schemas[name] = schema
+  }
+
+  if (dropped.length > 0) {
+    process.stderr.write(
+      `swagger: dropped ${dropped.length} colliding key(s) from ${source} — core wins: ${dropped.join(', ')}\n`,
+    )
+  }
+  return dropped
+}
+
+/**
+ * Re-root a fragment's paths under the prefix its router is actually mounted at.
+ *
+ * A fragment generated by pointing swagger-autogen at a router file alone has
+ * paths relative to that router (`/shard/accounts`), because nothing in the file
+ * says where it hangs. The prefix comes from the LIVE express stack rather than a
+ * table, so it cannot drift the way a hand-written mount list would.
+ *
+ * Express path params (`:id`) become OpenAPI's (`{id}`) — swagger-autogen already
+ * does that for the paths it generates, so the prefix has to match.
+ */
+function prefixPaths(fragment, prefix) {
+  const oas = prefix.replace(/:([A-Za-z0-9_]+)/g, '{$1}').replace(/\/+$/, '')
+  const outer = [...oas.matchAll(/\{([A-Za-z0-9_]+)\}/g)].map((m) => m[1])
+  const paths = {}
+  for (const [p, item] of Object.entries(fragment.paths || {})) {
+    paths[`${oas}${p}`] = orderParams(item, outer)
+  }
+  return { ...fragment, paths }
+}
+
+/**
+ * Put the prefix's own path parameters first, in prefix order.
+ *
+ * swagger-autogen orders parameters by where they appear in the path it saw, and
+ * the fragment's path is only the tail — so `/{id}/shard/link/{account}` comes
+ * out as (account, id) rather than (id, account). Re-rooting the path has to
+ * re-root the parameter order with it, or every slot route churns the committed
+ * spec by a reorder that means nothing.
+ */
+function orderParams(item, outer) {
+  for (const operation of Object.values(item)) {
+    const params = operation && operation.parameters
+    if (!Array.isArray(params)) continue
+    const rank = (p) => {
+      const i = outer.indexOf(p && p.name)
+      return i === -1 ? outer.length : i
+    }
+    // Stable: only the prefix params move, and only ahead of the rest.
+    operation.parameters = params
+      .map((p, i) => ({ p, i }))
+      .sort((a, b) => rank(a.p) - rank(b.p) || a.i - b.i)
+      .map(({ p }) => p)
+  }
+  return item
+}
+
+module.exports = { mergeFragment, prefixPaths }

server/swagger/slotSpecs.js

@@ -0,0 +1,120 @@
+// ── OpenAPI for core's extension-slot routers ──────────────────────────────
+//
+// The second half of `npm run swagger`. It exists because of a failure mode that
+// announces itself as a success.
+//
+// `swagger/swagger.js` is STATIC analysis: swagger-autogen parses `src/app.js` as
+// text and follows the literal `app.use(...)` mount chain. An extension slot
+// (MODULE_SYSTEM.md §1.9) breaks that chain on purpose — the slot's router is
+// created by `registries.declareSlot()` and filled later, so there is no literal
+// require for the parser to follow. When PR 4 moved the six `/admin/users/:id/shard/*`
+// routes behind the `admin.users.detail` slot, regenerating the spec printed
+// `Swagger-autogen: Success` and deleted 407 lines. Nothing failed. The spike hit
+// the identical thing (MODULE_API.md §7.4) and it is why the fragment merge is
+// the settled answer (§6.1).
+//
+// So: generate a fragment per filled slot by pointing swagger-autogen at that
+// router's own file, re-root its paths at the prefix the router is ACTUALLY
+// mounted at in the live app, and merge. Two things are deliberately derived
+// rather than written down, because a written-down copy is a copy that drifts:
+//
+//   • WHICH slots — from `registries.filledSlots()`, not a list here.
+//   • WHERE each hangs — by finding the slot's own router object in the live
+//     express stack and accumulating the mount prefixes above it, using
+//     `scripts/routeManifest.js`'s `mountPath` so the manifest and the spec can
+//     never disagree about what a mount decodes to.
+//
+// This is core's own slot fill only. A MODULE ships a prebuilt
+// `swagger-fragment.json` in its bundle and core merges it at request time
+// (§6.1a) — core never has a module's sources to analyse.
+
+const fs = require('fs')
+const os = require('os')
+const path = require('path')
+
+const swaggerAutogen = require('swagger-autogen')({ openapi: '3.0.0' })
+
+const { mergeFragment, prefixPaths } = require('./mergeSpec')
+const { mountPath } = require('../scripts/routeManifest')
+
+const SERVER_ROOT = path.join(__dirname, '..')
+
+/**
+ * Find `target` in an express stack and return the path prefix it is mounted at.
+ *
+ * Depth-first, accumulating each enclosing mount. Returns null when the router is
+ * not on the stack at all — which for a filled slot means core declared it and
+ * never mounted it, a bug worth failing the build over rather than papering over
+ * with an unprefixed path.
+ */
+function findMountPrefix(stack, target, prefix = '') {
+  for (const layer of stack || []) {
+    if (!layer.handle || !Array.isArray(layer.handle.stack)) continue
+    const here = prefix + mountPath(layer)
+    if (layer.handle === target) return here
+    const found = findMountPrefix(layer.handle.stack, target, here)
+    if (found !== null) return found
+  }
+  return null
+}
+
+/**
+ * Generate one fragment by running swagger-autogen over a single router file.
+ *
+ * Its paths come out relative to that router (`/shard/accounts`) because nothing
+ * in the file says where it hangs; `prefixPaths` supplies the rest.
+ */
+async function fragmentFor(specFile) {
+  const out = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'rg-swagger-')), 'fragment.json')
+  await swaggerAutogen(out, [path.relative(SERVER_ROOT, specFile).split(path.sep).join('/')], {
+    info: { title: 'slot fragment', version: '0' },
+  })
+  const fragment = JSON.parse(fs.readFileSync(out, 'utf8'))
+  fs.rmSync(path.dirname(out), { recursive: true, force: true })
+  return fragment
+}
+
+/**
+ * Merge every filled core slot's routes into the generated spec file, in place.
+ *
+ * @param {string} outputFile  the swagger-output.json swagger.js just wrote
+ * @returns {Promise<number>} how many paths were added
+ */
+async function mergeSlotSpecs(outputFile) {
+  /* eslint-disable global-require */
+  const app = require('../src/app') // builds the app: declares and fills the slots
+  const registries = require('../src/modules/registries')
+  /* eslint-enable global-require */
+
+  const filled = registries.filledSlots().filter((s) => s.specFile)
+  if (filled.length === 0) return 0
+
+  const spec = JSON.parse(fs.readFileSync(outputFile, 'utf8'))
+  let added = 0
+
+  for (const slot of filled) {
+    const prefix = findMountPrefix(app._router.stack, slot.router)
+    if (prefix === null) {
+      throw new Error(
+        `swagger: extension slot "${slot.slot}" is filled but its router is not mounted on the app — ` +
+          'declareSlot() returned a router nobody use()d.',
+      )
+    }
+    const fragment = prefixPaths(await fragmentFor(slot.specFile), prefix)
+    const paths = Object.keys(fragment.paths || {}).length
+    if (paths === 0) {
+      throw new Error(
+        `swagger: extension slot "${slot.slot}" generated an EMPTY fragment from ${slot.specFile}. ` +
+          'That is the silent-drop failure this step exists to catch, not a slot with no routes.',
+      )
+    }
+    mergeFragment(spec, fragment, `slot ${slot.slot}`)
+    added += paths
+    process.stdout.write(`merged ${paths} path(s) from slot ${slot.slot} at ${prefix}\n`)
+  }
+
+  fs.writeFileSync(outputFile, `${JSON.stringify(spec, null, 2)}\n`)
+  return added
+}
+
+module.exports = { mergeSlotSpecs, findMountPrefix }

server/swagger/swagger-output.json

@@ -3249,7 +3249,7 @@
         "tags": [
           "Admin · Posts"
         ],
-        "summary": "Retry one announcement delivery leg (town crier or Discord)",
+        "summary": "Retry one announcement delivery leg",
         "description": "",
         "parameters": [
           {
@@ -3308,10 +3308,7 @@
                 "properties": {
                   "leg": {
                     "type": "string",
-                    "enum": [
-                      "towncrier",
-                      "discord"
-                    ]
+                    "description": "A registered delivery leg id, as returned by GET /announce."
                   }
                 },
                 "required": [

server/swagger/swagger.js

@@ -1513,9 +1513,31 @@ function normalizePaths(spec) {
   return spec
 }
 
-swaggerAutogen(outputFile, routes, doc).then(() => {
-  const written = JSON.parse(fs.readFileSync(outputFile, 'utf8'))
-  fs.writeFileSync(outputFile, `${JSON.stringify(normalizePaths(written), null, 2)}\n`)
-  // eslint-disable-next-line no-console
-  console.log('swagger-output.json generated.')
-})
+// Static analysis cannot follow a route into an extension slot, so the slot
+// routers contribute a generated fragment afterwards — see swagger/slotSpecs.js
+// for what goes wrong without it. Merged BEFORE normalizePaths, so the merged-in
+// paths are sorted and trailing-slash-checked with everything else.
+//
+// The pool is pointed at a closed port here for the same reason
+// scripts/routeManifest.js does it: the merge step requires src/app.js to find
+// where each slot router is mounted, and requiring app.js builds the models. No
+// query is ever run.
+process.env.DB_HOST = process.env.DB_HOST || '127.0.0.1'
+process.env.DB_PORT = process.env.DB_PORT || '59999'
+
+/* eslint-disable global-require */
+swaggerAutogen(outputFile, routes, doc)
+  .then(() => require('./slotSpecs').mergeSlotSpecs(outputFile))
+  .then(() => {
+    const written = JSON.parse(fs.readFileSync(outputFile, 'utf8'))
+    fs.writeFileSync(outputFile, `${JSON.stringify(normalizePaths(written), null, 2)}\n`)
+    // eslint-disable-next-line no-console
+    console.log('swagger-output.json generated.')
+    // The mariadb pool keeps the loop alive even pointed at a dead port.
+    return require('../src/utils/db').close()
+  })
+  .catch((err) => {
+    process.stderr.write(`${err.stack || err.message}\n`)
+    process.exit(1)
+  })
+/* eslint-enable global-require */