feat(modules): install, uninstall, purge and restart (phase 4, slice 1)
The consumer half of a release module-uo's CI has been publishing since
phase 3 closed. Before this, core had the installed_modules provenance
columns and no code that could ever fill them: nothing fetched, verified,
unpacked, removed or purged anything, and there was no admin route at all.
Adds modules/archive.js, modules/install.js, schema.runPurge(),
lifecycle.stop(), loader.stopHook(), and /api/v1/admin/modules with eight
routes. 797 server tests (+76), manifest 158 -> 166 + 2 internal, OpenAPI
gains 8 operations and loses nothing.
Reject, never sanitise
----------------------
The download is the easy part: an https-only allowlist re-checked on every
redirect hop, a declared sha256 compared against the bytes that arrived, and
a byte cap. Unpacking is where the archive chooses the filenames, and core
writes into a directory bind-mounted from the host, so an escape is not
confined to the container.
archive.js inspects the whole archive before a byte is unpacked and refuses
absolute and drive-absolute paths, `..` segments, NUL bytes, backslashes,
anything that is not a regular file or a directory, more than one top-level
entry, and anything over the entry or byte caps. Refusing symlinks and
hardlinks outright is what keeps this off the majority of node-tar's
published advisories rather than depending on the library to contain them.
That two-pass shape is load-bearing, and it was measured rather than assumed:
extracting an archive whose fourth member escapes upward throws under
node-tar 7.5.22 -- and leaves the first three members on disk. The loader
scans that directory at require time on the next boot, so a half-unpacked
module is a module. Everything therefore happens in a scratch directory that
is removed on any failure, and the move into place is the last step.
`tar` is pinned to ^7.5.22 rather than the ^6 that installs by default: 6.x
is flagged critical, and reading the advisory list is what the file's header
now says out loud -- almost all of it is hardlink or symlink traversal and
PAX header interpretation differentials, which is exactly this feature's
threat model.
Two things the plan had wrong
-----------------------------
The bundle's top-level directory is `module-uo-<version>`, not the module id
-- so "the top-level name must equal the id" was checked against nothing real.
The extractor strips that level instead, because its name belongs to whoever
published the bundle and the directory it lands in is core's. What is checked
instead is the unpacked module.json: a manifest promising `uo` and delivering
something else is refused rather than installed under the name it promised.
And purge cannot be a follow-up action (decision 5): purge.sql lives inside
the directory uninstall deletes. It is offered in the uninstall flow and as a
standalone action on a still-installed module, and the standalone one refuses
unless the module is already disabled -- dropping tables under something that
is still serving leaves it answering out of a world that no longer exists.
Disable now means stopped
-------------------------
lifecycle.stop() dispatches that one module's onShutdown before flipping the
guard, so a module an operator switches off actually releases its sockets and
closes its streams instead of merely becoming unreachable. The hook runs
first and the state moves after it, because while onShutdown runs the module
is still `started` and that is the only state in which its routes and the
world it is tearing down agree. A hook that throws does not stop the disable
-- the opposite of the boot path's rule, and deliberately.
Enable is not its mirror and there is no start(id) beside it. There is no
onBoot re-dispatch and the hooks were never promised re-entrant, so enable
moves the row and the restart route starts it. A test pins that enable does
not touch the loader, because "fixing" it is a one-line change that would put
a module with closed sockets back on the nav.
Restart raises SIGTERM against its own process rather than calling the
shutdown path directly, so server.js's handler stays the one graceful-shutdown
path and this route cannot drift from it.
The allowlist bootstraps from MODULE_SOURCE_HOSTS into a settings row and is
admin-managed after that (decision 6); seedDefault is INSERT IGNORE, so
changing the variable on an existing deployment is a no-op by design. An empty
list forbids every install rather than allowing every host -- the safe
direction for a value someone might blank by accident.
Verified against the real v0.3.0 release
----------------------------------------
Not a fixture: fetched the published install manifest over the real Gitea
host and its redirect chain, verified the sha256, inspected and unpacked the
252,517-byte artifact to 82 files, and then booted core against the result --
the module registered its five mounts, seven streams and eight capabilities
and resolved its client chunk, with no scratch directory left behind.
Two defects this slice's own tooling caught, both of which had already been
written down as classes:
- the controller destructured runPurge at require time, capturing the
function rather than the module, which made the one dependency whose
ORDER matters the one that could not be substituted;
- two swagger annotations carried an apostrophe inside a quoted string,
dropped silently by swagger-autogen before slice 5 taught it to fail loudly.
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
221
server/src/modules/archive.js
Normal file
221
server/src/modules/archive.js
Normal file
@@ -0,0 +1,221 @@
|
||||
// ── The hardened bundle extractor ──────────────────────────────────────────
|
||||
//
|
||||
// Phase 4, slice 1 of docs/website/MODULE_SYSTEM.md §2.7.2. This is the part of
|
||||
// the install path that handles input an attacker chose, and it is separated
|
||||
// from install.js so that it can be tested against crafted archives without a
|
||||
// network, a database or a filesystem layout.
|
||||
//
|
||||
// **The download is not the dangerous part.** An allowlisted host, a declared
|
||||
// sha256 and a size cap between them settle where the bytes came from and that
|
||||
// they are the bytes that were published. Unpacking is different: the ARCHIVE
|
||||
// chooses the filenames, and core writes into a directory bind-mounted from the
|
||||
// host (MODULE_SYSTEM.md §2.5), so an escape is not confined to the container.
|
||||
//
|
||||
// The rule this file follows is **reject, never sanitise.** node-tar will
|
||||
// happily strip a leading `/` and drop a `..` for you, which turns a hostile
|
||||
// archive into a slightly different archive that then gets installed. An archive
|
||||
// that needs correcting is an archive that should not be trusted, so anything on
|
||||
// the list below fails the whole install and nothing is written.
|
||||
//
|
||||
// - an absolute path, POSIX (`/etc/…`) or Windows (`C:\…`, `\\server\…`)
|
||||
// - any `..` segment, anywhere
|
||||
// - anything that is not a regular file or a directory — so no symlinks, no
|
||||
// hardlinks, no devices, no FIFOs. A module bundle has no legitimate use for
|
||||
// any of them, and every one is a documented escape primitive
|
||||
// - more than one top-level entry
|
||||
// - more entries, or more unpacked bytes, than the caps below
|
||||
//
|
||||
// That third rule deserves its own note, because it is why the `tar` dependency
|
||||
// is pinned forward rather than merely present: **the majority of node-tar's
|
||||
// published advisories are hardlink or symlink path traversal**, several of them
|
||||
// through interpretation differences in PAX and GNU long-name headers rather
|
||||
// than through anything the calling code did wrong. Refusing those entry types
|
||||
// outright means this extractor is not relying on the library to get their
|
||||
// containment right. The remaining advisories are parser denial-of-service, and
|
||||
// those are what the caps and the two-pass shape address.
|
||||
//
|
||||
// Two passes, deliberately: `inspect()` reads the archive and decides, and only
|
||||
// an archive that survived that is handed to `extract()`. A `filter` callback
|
||||
// during extraction cannot reject — by the time it is asked about entry 400, the
|
||||
// first 399 are already on disk.
|
||||
|
||||
const fs = require('fs')
|
||||
const fsp = require('fs/promises')
|
||||
const path = require('path')
|
||||
const tar = require('tar')
|
||||
|
||||
// Caps. Generous against a real bundle (module-uo's is a few megabytes, a few
|
||||
// hundred files) and small enough that a decompression bomb is refused rather
|
||||
// than paged in. Both are checked DURING the inspect pass, not after it, so a
|
||||
// hostile archive stops being read at the limit instead of at its end.
|
||||
const MAX_BYTES = 128 * 1024 * 1024
|
||||
const MAX_ENTRIES = 20000
|
||||
|
||||
// tar entry types worth naming. node-tar reports these as strings; anything not
|
||||
// in this set is refused, including the ones no one has thought of, which is the
|
||||
// point of an allowlist here rather than a list of the types known to be bad.
|
||||
const ALLOWED_TYPES = new Set(['File', 'Directory'])
|
||||
|
||||
class ArchiveError extends Error {
|
||||
constructor(message) {
|
||||
super(message)
|
||||
this.name = 'ArchiveError'
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Is this entry path safe to unpack anywhere?
|
||||
*
|
||||
* Returns a reason string, or null when the path is fine. Written as a reason
|
||||
* rather than a boolean because the operator pasting a URL needs to be told what
|
||||
* was wrong with what they were served, and "the archive is invalid" is not that.
|
||||
*/
|
||||
function pathProblem(entryPath) {
|
||||
const p = String(entryPath)
|
||||
|
||||
// NUL is not a path character. node-tar has had uncaught-exception advisories
|
||||
// for NUL bytes inside PAX records, so this is checked here rather than left
|
||||
// to the parser.
|
||||
if (p.includes('\0')) return 'an entry path contains a NUL byte'
|
||||
|
||||
// A backslash is a path separator on the platform this may be unpacked on, so
|
||||
// an entry that contains one is not the single path component it looks like.
|
||||
if (p.includes('\\')) return `an entry path contains a backslash: "${p}"`
|
||||
|
||||
if (p.startsWith('/')) return `an entry path is absolute: "${p}"`
|
||||
// Drive-relative and UNC. Both are the subject of their own node-tar
|
||||
// advisories, and neither is refused by a leading-slash check.
|
||||
if (/^[a-zA-Z]:/.test(p)) return `an entry path is drive-absolute: "${p}"`
|
||||
|
||||
const segments = p.split('/')
|
||||
if (segments.includes('..')) return `an entry path escapes upward: "${p}"`
|
||||
|
||||
return null
|
||||
}
|
||||
|
||||
/**
|
||||
* Read an archive without unpacking it, and decide whether it may be unpacked.
|
||||
*
|
||||
* @param {string} file absolute path to the .tar.gz
|
||||
* @returns {Promise<{root: string, entries: number, bytes: number}>} the single
|
||||
* top-level directory name, and what is inside it.
|
||||
* @throws {ArchiveError} on anything in this file's header list.
|
||||
*/
|
||||
async function inspect(file) {
|
||||
const roots = new Set()
|
||||
let entries = 0
|
||||
let bytes = 0
|
||||
// The first problem found, kept rather than thrown from inside the callback:
|
||||
// throwing out of `onentry` escapes through the parser's stream machinery and
|
||||
// arrives as an unhelpful wrapped error, when it arrives at all.
|
||||
let problem = null
|
||||
|
||||
const note = (message) => {
|
||||
if (!problem) problem = message
|
||||
}
|
||||
|
||||
await tar.t({
|
||||
file,
|
||||
onentry(entry) {
|
||||
if (problem) return
|
||||
|
||||
entries += 1
|
||||
if (entries > MAX_ENTRIES) {
|
||||
note(`the archive has more than ${MAX_ENTRIES} entries`)
|
||||
return
|
||||
}
|
||||
|
||||
if (!ALLOWED_TYPES.has(entry.type)) {
|
||||
// The message names the type because "symbolic link" is a much more
|
||||
// useful thing for an operator to read than "invalid entry".
|
||||
note(`the archive contains a ${entry.type} ("${entry.path}") — a module bundle may only contain files and directories`)
|
||||
return
|
||||
}
|
||||
|
||||
const bad = pathProblem(entry.path)
|
||||
if (bad) {
|
||||
note(bad)
|
||||
return
|
||||
}
|
||||
|
||||
// A negative or absurd size is a parser-confusion primitive in its own
|
||||
// right; clamping at zero keeps the running total honest.
|
||||
bytes += Math.max(0, Number(entry.size) || 0)
|
||||
if (bytes > MAX_BYTES) {
|
||||
note(`the archive unpacks to more than ${Math.round(MAX_BYTES / 1024 / 1024)} MB`)
|
||||
return
|
||||
}
|
||||
|
||||
const [root] = String(entry.path).split('/')
|
||||
if (root) roots.add(root)
|
||||
},
|
||||
})
|
||||
|
||||
if (problem) throw new ArchiveError(problem)
|
||||
if (entries === 0) throw new ArchiveError('the archive is empty')
|
||||
if (roots.size !== 1) {
|
||||
// A bundle is one directory. More than one top-level entry means `strip: 1`
|
||||
// below would silently merge or discard things, and a bundle that needs
|
||||
// interpreting is not a bundle.
|
||||
throw new ArchiveError(
|
||||
`the archive must contain exactly one top-level directory, found ${roots.size}` +
|
||||
(roots.size > 1 ? ` (${[...roots].slice(0, 4).join(', ')})` : ''),
|
||||
)
|
||||
}
|
||||
|
||||
return { root: [...roots][0], entries, bytes }
|
||||
}
|
||||
|
||||
/**
|
||||
* Unpack an inspected archive into `dest`, stripping its single top level.
|
||||
*
|
||||
* The top-level directory is stripped rather than kept, because its name belongs
|
||||
* to whoever published the bundle — module-uo's release workflow packs
|
||||
* `module-uo-<version>/`, not `uo/` — while the directory it lands in is core's
|
||||
* decision and has to be the module id the loader scans for. What is inside the
|
||||
* bundle is the module; what the wrapper is called is packaging.
|
||||
*
|
||||
* `dest` must not exist. Callers unpack to a temporary directory and move it
|
||||
* into place, so a failure halfway through leaves nothing for the next boot's
|
||||
* scan to find.
|
||||
*/
|
||||
async function extract(file, dest) {
|
||||
if (fs.existsSync(dest)) throw new ArchiveError(`extraction target already exists: ${dest}`)
|
||||
await fsp.mkdir(dest, { recursive: true })
|
||||
|
||||
await tar.x({
|
||||
file,
|
||||
cwd: dest,
|
||||
strip: 1,
|
||||
// Belt and braces on top of inspect(): the library's own refusal to write
|
||||
// outside cwd, and its refusal to overwrite through a link. Neither is
|
||||
// load-bearing here — inspect() has already rejected every entry that could
|
||||
// exercise them — but a second lock on a door that is already locked costs
|
||||
// nothing, and this is the door.
|
||||
preservePaths: false,
|
||||
// Reject rather than warn on anything the parser itself objects to.
|
||||
strict: true,
|
||||
})
|
||||
|
||||
return dest
|
||||
}
|
||||
|
||||
/**
|
||||
* Inspect, then extract. The only entry point install.js uses.
|
||||
*/
|
||||
async function unpack(file, dest) {
|
||||
const stats = await inspect(file)
|
||||
await extract(file, dest)
|
||||
return stats
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
ArchiveError,
|
||||
inspect,
|
||||
extract,
|
||||
unpack,
|
||||
pathProblem,
|
||||
MAX_BYTES,
|
||||
MAX_ENTRIES,
|
||||
ALLOWED_TYPES,
|
||||
}
|
||||
421
server/src/modules/install.js
Normal file
421
server/src/modules/install.js
Normal file
@@ -0,0 +1,421 @@
|
||||
// ── Installing and removing a module bundle ────────────────────────────────
|
||||
//
|
||||
// Phase 4, slice 1 of docs/website/MODULE_SYSTEM.md §2.7.2 — the consumer half
|
||||
// of a release the module's own CI already publishes. Nothing here touches the
|
||||
// database or the loader: this file moves bytes onto the volume and off it, and
|
||||
// the caller (router/v1/admin/modules.controller.js) writes down what happened.
|
||||
//
|
||||
// The shape of an install, and why it is this shape:
|
||||
//
|
||||
// 1. The operator pastes the URL of a release's install manifest — a small
|
||||
// JSON document naming the artifact, its sha256 and its size (decision 2).
|
||||
// There is no catalog, because a catalog would make core's release cadence
|
||||
// decide which modules can exist.
|
||||
// 2. Every URL fetched — the manifest, the artifact, and every redirect hop —
|
||||
// is checked against the admin-managed host allowlist (decision 6). That is
|
||||
// what keeps a pasted URL from also being an SSRF primitive.
|
||||
// 3. The artifact is streamed to a temporary file under a byte cap, hashed as
|
||||
// it arrives, and compared against the manifest's `sha256`. The hash is the
|
||||
// trust anchor; releases are unsigned and say so.
|
||||
// 4. modules/archive.js inspects the archive in full before a single byte is
|
||||
// unpacked, and only then unpacks it — into a TEMPORARY directory.
|
||||
// 5. The unpacked tree's own `module.json` must agree with the manifest about
|
||||
// what it is. A manifest that promises `uo` and delivers something else is
|
||||
// refused rather than installed under the name it was promised.
|
||||
// 6. Only then is anything moved into `modules/<id>/`, and the directory it
|
||||
// replaces is kept aside until the move has succeeded.
|
||||
//
|
||||
// **Nothing is written into the modules directory until every check has passed**,
|
||||
// and that is not belt-and-braces. Verified against node-tar 7.5.22 while writing
|
||||
// this: extracting an archive whose fourth member escapes upward throws — and
|
||||
// leaves the first three members on disk. A loader scans that directory at
|
||||
// require time on the next boot; a half-unpacked module is a module.
|
||||
//
|
||||
// One thing this file deliberately does NOT do: mount anything. Installing puts
|
||||
// a directory on the volume, and §1.12 means the volume is read at require time,
|
||||
// so the module appears when the process restarts. The admin screen offers that
|
||||
// restart (decision 1); it is not implied here.
|
||||
|
||||
const crypto = require('crypto')
|
||||
const fs = require('fs')
|
||||
const fsp = require('fs/promises')
|
||||
const path = require('path')
|
||||
const { Readable, Transform } = require('stream')
|
||||
const { pipeline } = require('stream/promises')
|
||||
|
||||
const archive = require('./archive')
|
||||
const loader = require('./loader')
|
||||
|
||||
const log = require('../utils/logger')('modules')
|
||||
|
||||
// An install manifest is a small JSON document. A megabyte of it is not a
|
||||
// manifest, and reading it into memory unbounded is the one place this file
|
||||
// would otherwise trust a remote length.
|
||||
const MAX_MANIFEST_BYTES = 256 * 1024
|
||||
// The artifact cap is the archive's own unpacked cap — a compressed bundle
|
||||
// larger than what it is allowed to unpack to has nothing to offer.
|
||||
const MAX_ARTIFACT_BYTES = archive.MAX_BYTES
|
||||
const FETCH_TIMEOUT_MS = 30000
|
||||
// Gitea serves a release asset through at least one redirect. Following them is
|
||||
// necessary; following them blindly is how an allowlist gets bypassed, so each
|
||||
// hop is re-checked and the chain is bounded.
|
||||
const MAX_REDIRECTS = 5
|
||||
|
||||
// Same id rule the loader enforces when scanning, restated rather than imported
|
||||
// so an install cannot put a directory on the volume that the loader would then
|
||||
// refuse to look at.
|
||||
const ID = /^[a-z][a-z0-9-]{1,31}$/
|
||||
const SHA256 = /^[0-9a-f]{64}$/
|
||||
|
||||
class InstallError extends Error {
|
||||
constructor(message, { status = 400 } = {}) {
|
||||
super(message)
|
||||
this.name = 'InstallError'
|
||||
// Carried so the controller can answer 400 for "your URL is wrong" and 502
|
||||
// for "the host you named misbehaved" without re-deriving it from the text.
|
||||
this.status = status
|
||||
}
|
||||
}
|
||||
|
||||
// ── The allowlist ──────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Parse the stored allowlist setting into hostnames.
|
||||
*
|
||||
* Comma or whitespace separated, case-insensitive, empty entries dropped. An
|
||||
* empty list means nothing may be installed from anywhere — a refusal, never a
|
||||
* wildcard. That direction matters: a setting an admin accidentally blanks
|
||||
* should stop installs, not permit every host on the internet.
|
||||
*/
|
||||
function parseHosts(value) {
|
||||
return String(value || '')
|
||||
.split(/[,\s]+/)
|
||||
.map((h) => h.trim().toLowerCase())
|
||||
.filter(Boolean)
|
||||
}
|
||||
|
||||
/**
|
||||
* Check one URL against the allowlist, and return it parsed.
|
||||
*
|
||||
* `https` only. A plaintext fetch of code this process is going to execute is
|
||||
* not something an operator should be able to opt into by typing a URL, and the
|
||||
* sha256 does not help — whoever can rewrite the artifact in flight can rewrite
|
||||
* the manifest that declares its hash.
|
||||
*/
|
||||
function checkUrl(raw, hosts) {
|
||||
let url
|
||||
try {
|
||||
url = new URL(String(raw))
|
||||
} catch {
|
||||
throw new InstallError(`"${raw}" is not a valid URL`)
|
||||
}
|
||||
if (url.protocol !== 'https:') {
|
||||
throw new InstallError(`only https URLs may be installed from (got "${url.protocol}")`)
|
||||
}
|
||||
if (!hosts.length) {
|
||||
throw new InstallError(
|
||||
'no module source hosts are allowed — set one in Admin → Modules before installing',
|
||||
)
|
||||
}
|
||||
if (!hosts.includes(url.hostname.toLowerCase())) {
|
||||
throw new InstallError(
|
||||
`"${url.hostname}" is not an allowed module source host (allowed: ${hosts.join(', ')})`,
|
||||
)
|
||||
}
|
||||
return url
|
||||
}
|
||||
|
||||
// ── Fetching ───────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* GET a URL, following redirects by hand so every hop is re-checked.
|
||||
*
|
||||
* `fetch`'s own redirect following would take the first hop off the allowlist
|
||||
* and the rest wherever it was pointed, which is exactly the hole the allowlist
|
||||
* exists to close.
|
||||
*
|
||||
* `fetchImpl` is the same injection seam `replayFragments({query})` and
|
||||
* `lifecycle.boot({model})` use, and it exists for the same reason: the rules
|
||||
* this file enforces — https only, allowlisted host, allowlisted REDIRECT host,
|
||||
* bounded body, matching hash — are all decisions about a response, and testing
|
||||
* them against a real TLS server would mean testing Node's certificate handling
|
||||
* instead. Production never passes it.
|
||||
*
|
||||
* @returns {Promise<Response>} a response whose body has not been read.
|
||||
*/
|
||||
async function get(rawUrl, hosts, fetchImpl = fetch) {
|
||||
let url = checkUrl(rawUrl, hosts)
|
||||
|
||||
for (let hop = 0; hop <= MAX_REDIRECTS; hop += 1) {
|
||||
let res
|
||||
try {
|
||||
res = await fetchImpl(url, {
|
||||
redirect: 'manual',
|
||||
signal: AbortSignal.timeout(FETCH_TIMEOUT_MS),
|
||||
headers: { accept: '*/*' },
|
||||
})
|
||||
} catch (err) {
|
||||
throw new InstallError(`could not reach ${url.hostname}: ${err.message}`, { status: 502 })
|
||||
}
|
||||
|
||||
if (res.status >= 300 && res.status < 400) {
|
||||
const location = res.headers.get('location')
|
||||
if (!location) throw new InstallError(`${url.hostname} redirected without a location`, { status: 502 })
|
||||
// Resolved against the current URL, then re-checked — a relative redirect
|
||||
// is normal and a cross-host one is the thing being guarded against.
|
||||
url = checkUrl(new URL(location, url).toString(), hosts)
|
||||
continue
|
||||
}
|
||||
|
||||
if (!res.ok) {
|
||||
throw new InstallError(`${url.href} returned ${res.status}`, { status: 502 })
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
throw new InstallError(`too many redirects (more than ${MAX_REDIRECTS})`, { status: 502 })
|
||||
}
|
||||
|
||||
/** Read a bounded response body as text. */
|
||||
async function readText(res, cap, what) {
|
||||
const declared = Number(res.headers.get('content-length'))
|
||||
if (Number.isFinite(declared) && declared > cap) {
|
||||
throw new InstallError(`${what} is larger than ${cap} bytes`)
|
||||
}
|
||||
const buf = Buffer.from(await res.arrayBuffer())
|
||||
// Checked again after reading: content-length is the server's claim, not a
|
||||
// limit it is obliged to honour.
|
||||
if (buf.length > cap) throw new InstallError(`${what} is larger than ${cap} bytes`)
|
||||
return buf.toString('utf8')
|
||||
}
|
||||
|
||||
/**
|
||||
* Stream a response body to a file, hashing as it goes and stopping at the cap.
|
||||
*
|
||||
* The hash is computed from the bytes that were written rather than by re-reading
|
||||
* the file, so there is no window in which the file could differ from what was
|
||||
* verified.
|
||||
*/
|
||||
async function download(res, dest, cap) {
|
||||
const hash = crypto.createHash('sha256')
|
||||
let bytes = 0
|
||||
|
||||
const body = Readable.fromWeb(res.body)
|
||||
const counter = new Transform({
|
||||
transform(chunk, _enc, cb) {
|
||||
bytes += chunk.length
|
||||
if (bytes > cap) {
|
||||
cb(new InstallError(`the artifact is larger than ${Math.round(cap / 1024 / 1024)} MB`))
|
||||
return
|
||||
}
|
||||
hash.update(chunk)
|
||||
cb(null, chunk)
|
||||
},
|
||||
})
|
||||
|
||||
await pipeline(body, counter, fs.createWriteStream(dest))
|
||||
return { sha256: hash.digest('hex'), bytes }
|
||||
}
|
||||
|
||||
// ── The install manifest ───────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Fetch and validate a release's install manifest.
|
||||
*
|
||||
* The shape is the one module-uo's release workflow already writes:
|
||||
* `{schema, id, name, version, coreApi, artifact, url, sha256, size}`. Only the
|
||||
* fields this side needs are required — a module publisher may carry more.
|
||||
*/
|
||||
async function fetchManifest(manifestUrl, hosts, fetchImpl = fetch) {
|
||||
const res = await get(manifestUrl, hosts, fetchImpl)
|
||||
const text = await readText(res, MAX_MANIFEST_BYTES, 'the install manifest')
|
||||
|
||||
let manifest
|
||||
try {
|
||||
manifest = JSON.parse(text)
|
||||
} catch (err) {
|
||||
throw new InstallError(`the install manifest is not valid JSON: ${err.message}`)
|
||||
}
|
||||
if (!manifest || typeof manifest !== 'object' || Array.isArray(manifest)) {
|
||||
throw new InstallError('the install manifest is not a JSON object')
|
||||
}
|
||||
|
||||
const { id, name, version, sha256 } = manifest
|
||||
if (!ID.test(String(id || ''))) {
|
||||
throw new InstallError(`the install manifest has an invalid module id: ${JSON.stringify(id)}`)
|
||||
}
|
||||
if (!name || !version) throw new InstallError('the install manifest is missing name or version')
|
||||
if (!SHA256.test(String(sha256 || '').toLowerCase())) {
|
||||
throw new InstallError('the install manifest has no valid sha256 for its artifact')
|
||||
}
|
||||
|
||||
// `url` is the absolute artifact URL the publisher wrote; `artifact` is its
|
||||
// filename. Prefer the URL, fall back to resolving the filename beside the
|
||||
// manifest — which is where a release's assets sit — so a manifest that
|
||||
// travelled without its absolute URL still installs.
|
||||
const artifactUrl = new URL(manifest.url || manifest.artifact || '', manifestUrl).toString()
|
||||
|
||||
return {
|
||||
id: String(id),
|
||||
name: String(name),
|
||||
version: String(version),
|
||||
sha256: String(sha256).toLowerCase(),
|
||||
size: Number(manifest.size) || null,
|
||||
artifactUrl,
|
||||
coreApi: manifest.coreApi ? String(manifest.coreApi) : null,
|
||||
}
|
||||
}
|
||||
|
||||
// ── Paths on the volume ────────────────────────────────────────────────────
|
||||
|
||||
/** Where a module with this id lives. Rejects an id that is not one. */
|
||||
function moduleDir(id) {
|
||||
if (!ID.test(String(id || ''))) throw new InstallError(`invalid module id: ${JSON.stringify(id)}`)
|
||||
return path.join(loader.dir(), String(id))
|
||||
}
|
||||
|
||||
/** Is there a directory for this module on the volume right now? */
|
||||
function isInstalled(id) {
|
||||
try {
|
||||
return fs.statSync(moduleDir(id)).isDirectory()
|
||||
} catch {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The absolute path of a module's `purge.sql`, or null.
|
||||
*
|
||||
* Read from the module's own `module.json` on disk rather than from the loader,
|
||||
* because purge has to work for a module that never loaded — a `startup_failed`
|
||||
* one is exactly when an operator wants its tables gone.
|
||||
*/
|
||||
function purgeFile(id) {
|
||||
try {
|
||||
const manifest = JSON.parse(fs.readFileSync(path.join(moduleDir(id), 'module.json'), 'utf8'))
|
||||
if (!manifest.purge) return null
|
||||
const file = path.resolve(moduleDir(id), manifest.purge)
|
||||
// The same containment check the loader applies to `client.entry`: a
|
||||
// manifest may not point core at a file outside the module.
|
||||
if (!file.startsWith(moduleDir(id) + path.sep)) return null
|
||||
return fs.existsSync(file) ? file : null
|
||||
} catch {
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
// ── Install ────────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Download, verify, unpack and install a module from an install-manifest URL.
|
||||
*
|
||||
* Never leaves a partial module on the volume: everything happens under a
|
||||
* scratch directory that is removed on any failure, and the move into place is
|
||||
* the last step.
|
||||
*
|
||||
* @param {object} args
|
||||
* @param {string} args.url the install manifest URL the admin pasted
|
||||
* @param {string[]} args.hosts the allowlist, already parsed
|
||||
* @returns {Promise<{id,name,version,sha256,source,bytes,replaced}>}
|
||||
*/
|
||||
async function install({ url, hosts, fetchImpl = fetch }) {
|
||||
const manifest = await fetchManifest(url, hosts, fetchImpl)
|
||||
const target = moduleDir(manifest.id)
|
||||
const scratch = await fsp.mkdtemp(path.join(loader.dir(), `.install-${manifest.id}-`))
|
||||
const tarball = path.join(scratch, 'bundle.tar.gz')
|
||||
const unpacked = path.join(scratch, 'unpacked')
|
||||
|
||||
try {
|
||||
const res = await get(manifest.artifactUrl, hosts, fetchImpl)
|
||||
const { sha256, bytes } = await download(res, tarball, MAX_ARTIFACT_BYTES)
|
||||
|
||||
if (sha256 !== manifest.sha256) {
|
||||
// The whole trust model in one comparison. Deliberately does not name the
|
||||
// computed hash in a way that reads like a value to copy into the
|
||||
// manifest — a mismatch means stop, not reconcile.
|
||||
throw new InstallError(
|
||||
`the downloaded artifact does not match the sha256 in the install manifest — refusing to install`,
|
||||
)
|
||||
}
|
||||
if (manifest.size && bytes !== manifest.size) {
|
||||
throw new InstallError(`the artifact is ${bytes} bytes but the manifest declares ${manifest.size}`)
|
||||
}
|
||||
|
||||
await archive.unpack(tarball, unpacked)
|
||||
|
||||
// What the bundle says it is, checked against what the manifest promised.
|
||||
let inner
|
||||
try {
|
||||
inner = JSON.parse(await fsp.readFile(path.join(unpacked, 'module.json'), 'utf8'))
|
||||
} catch {
|
||||
throw new InstallError('the bundle has no readable module.json at its root')
|
||||
}
|
||||
if (inner.id !== manifest.id) {
|
||||
throw new InstallError(
|
||||
`the bundle declares module id "${inner.id}" but the install manifest promised "${manifest.id}"`,
|
||||
)
|
||||
}
|
||||
if (inner.version !== manifest.version) {
|
||||
throw new InstallError(
|
||||
`the bundle declares version "${inner.version}" but the install manifest promised "${manifest.version}"`,
|
||||
)
|
||||
}
|
||||
|
||||
// The swap. The outgoing directory is moved aside rather than deleted first,
|
||||
// so a failed rename leaves the previous version recoverable instead of
|
||||
// leaving no module at all — the same reasoning the installer repo applies
|
||||
// to a ServUO tree.
|
||||
const replaced = fs.existsSync(target)
|
||||
const aside = `${target}.replaced-${Date.now()}`
|
||||
if (replaced) await fsp.rename(target, aside)
|
||||
try {
|
||||
await fsp.rename(unpacked, target)
|
||||
} catch (err) {
|
||||
if (replaced) await fsp.rename(aside, target).catch(() => {})
|
||||
throw err
|
||||
}
|
||||
if (replaced) await fsp.rm(aside, { recursive: true, force: true })
|
||||
|
||||
log.info(`installed module "${manifest.id}" v${manifest.version}`, {
|
||||
source: url,
|
||||
sha256: manifest.sha256,
|
||||
bytes,
|
||||
replaced,
|
||||
})
|
||||
|
||||
return { ...manifest, source: url, bytes, replaced }
|
||||
} finally {
|
||||
await fsp.rm(scratch, { recursive: true, force: true }).catch(() => {})
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove a module's directory from the volume.
|
||||
*
|
||||
* Returns whether there was one. Does not touch the database, does not run
|
||||
* purge.sql, and does not stop the running module — the controller sequences
|
||||
* all three, because the order matters and only it knows what the operator
|
||||
* asked for.
|
||||
*/
|
||||
async function removeDir(id) {
|
||||
const dir = moduleDir(id)
|
||||
if (!fs.existsSync(dir)) return false
|
||||
await fsp.rm(dir, { recursive: true, force: true })
|
||||
log.info(`removed module directory for "${id}"`)
|
||||
return true
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
InstallError,
|
||||
parseHosts,
|
||||
checkUrl,
|
||||
get,
|
||||
fetchManifest,
|
||||
install,
|
||||
removeDir,
|
||||
moduleDir,
|
||||
isInstalled,
|
||||
purgeFile,
|
||||
MAX_MANIFEST_BYTES,
|
||||
MAX_ARTIFACT_BYTES,
|
||||
}
|
||||
@@ -205,4 +205,67 @@ async function shutdown({ modules, budgetMs = SHUTDOWN_BUDGET_MS } = {}) {
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { boot, shutdown, SHUTDOWN_BUDGET_MS }
|
||||
/**
|
||||
* Stop ONE module and mark it disabled — the admin panel's Disable (§2.7.2
|
||||
* decision 3).
|
||||
*
|
||||
* Phase 2 built disable as a pure state flip: the record moved to `disabled` and
|
||||
* the dispatch guard started answering 404. That makes a module invisible, not
|
||||
* stopped. Everything a module does that is not a response to a request — the
|
||||
* sockets and timers its `onBoot` armed — carried on running, so an operator
|
||||
* disabling a module *because* it was misbehaving got nothing until the next
|
||||
* restart, which is the one thing the button was supposed to save them.
|
||||
*
|
||||
* So the hook runs first, and the state moves after it: while `onShutdown` is
|
||||
* running the module is still `started`, which is the only state in which its
|
||||
* own routes and the things it is tearing down are consistent with each other.
|
||||
* The hook gets the same budget the exit path gives it.
|
||||
*
|
||||
* A hook that throws does NOT stop the disable. The operator asked for this
|
||||
* module to stop answering; a module that could not close cleanly is a reason to
|
||||
* log loudly, not a reason to leave it serving. That is the opposite of the boot
|
||||
* path's rule, and deliberately: there, a failure means the module never became
|
||||
* safe to use.
|
||||
*
|
||||
* **Enable is not the mirror of this, and there is no `start(id)` beside it.**
|
||||
* There is no `onBoot` re-dispatch, and MODULE_API.md has never promised the
|
||||
* hooks are re-entrant — no module author has written `onBoot` to be safe to run
|
||||
* twice in one process. Re-enabling therefore moves the row and waits for a
|
||||
* restart, which the admin screen offers.
|
||||
*
|
||||
* @returns {Promise<{stopped: boolean, error: string|null}>} whether a hook ran.
|
||||
*/
|
||||
async function stop(id, { modules, model, budgetMs = SHUTDOWN_BUDGET_MS } = {}) {
|
||||
/* eslint-disable global-require */
|
||||
const loader = modules || require('./loader')
|
||||
const rows = model || require('../model/modules/modules.model')
|
||||
/* eslint-enable global-require */
|
||||
|
||||
let error = null
|
||||
let stopped = false
|
||||
|
||||
if (loader.isLoaded()) {
|
||||
const target = loader.stopHook(id)
|
||||
if (target) {
|
||||
try {
|
||||
await withBudget(target.hook, budgetMs)
|
||||
stopped = true
|
||||
log.info(`module "${id}" stopped by an operator`)
|
||||
} catch (err) {
|
||||
error = err.message
|
||||
log.warn(`module "${id}" onShutdown failed or timed out while being disabled — disabling anyway`, {
|
||||
error: err.message,
|
||||
})
|
||||
}
|
||||
}
|
||||
// Unconditional, and after the hook: this is what makes its routes and its
|
||||
// client chunk answer 404 (§4.5). A module that is not loaded in this
|
||||
// process has no record to move, and setState ignores an unknown id.
|
||||
loader.setState(id, 'disabled')
|
||||
}
|
||||
|
||||
await safe(`disabling module "${id}"`, () => rows.disable(id))
|
||||
return { stopped, error }
|
||||
}
|
||||
|
||||
module.exports = { boot, shutdown, stop, SHUTDOWN_BUDGET_MS }
|
||||
|
||||
@@ -796,6 +796,31 @@ function shutdownHooks() {
|
||||
.reverse()
|
||||
}
|
||||
|
||||
/**
|
||||
* One module's `onShutdown`, for stopping it on its own rather than at exit.
|
||||
*
|
||||
* Phase 4 (§2.7.2 decision 3) gave the admin panel's Disable a real meaning.
|
||||
* Until then, disabling flipped this record's state and the dispatch guard began
|
||||
* answering 404 — which made the module invisible without making it stop. A
|
||||
* module's `onBoot` is where it opens its sockets and arms its timers, and none
|
||||
* of that is reachable through a URL, so an operator disabling a misbehaving
|
||||
* module got no relief from it at all until the next restart.
|
||||
*
|
||||
* `started` only, the same rule shutdownHooks() applies and for the same reason:
|
||||
* a module whose `onBoot` threw has a half-built world its `onShutdown` was
|
||||
* never written to tear down. Returns null when there is nothing to run — which
|
||||
* covers "not started", "no hook", and "no such module", none of which is an
|
||||
* error the caller can act on differently.
|
||||
*
|
||||
* @returns {{id: string, hook: Function}|null}
|
||||
*/
|
||||
function stopHook(id) {
|
||||
assertLoaded('stopHook')
|
||||
const record = modules.get(id)
|
||||
if (!record || record.state !== 'started' || !record.hooks.onShutdown) return null
|
||||
return { id: record.id, hook: record.hooks.onShutdown }
|
||||
}
|
||||
|
||||
/**
|
||||
* Every schema fragment waiting to be replayed, in scan order.
|
||||
*
|
||||
@@ -896,6 +921,7 @@ module.exports = {
|
||||
fragments,
|
||||
bootable,
|
||||
shutdownHooks,
|
||||
stopHook,
|
||||
clientChunks,
|
||||
clientEntryUrls,
|
||||
specFragments,
|
||||
|
||||
@@ -81,4 +81,48 @@ async function replayFragments({ query, modules } = {}) {
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { replayFragments }
|
||||
/**
|
||||
* Run one module's `purge.sql` — the destructive twin of the replay above
|
||||
* (MODULE_SYSTEM.md §2.5, and §2.7.2 decision 5 for when it is offered).
|
||||
*
|
||||
* It lives beside replayFragments because they are the same operation pointed in
|
||||
* opposite directions: a file of statements the module ships, split by the same
|
||||
* splitter, run serially on the same pool. Keeping them together is what makes
|
||||
* it obvious that the fragment's leading-verb allowlist does NOT apply here —
|
||||
* `purge.sql` is the one file a module may put a DROP in, precisely because it
|
||||
* is the one file that never runs on a boot.
|
||||
*
|
||||
* Unlike the replay, this **throws**. A replay failure is one module failing to
|
||||
* start, which the site survives by 503ing that module; a purge failure is an
|
||||
* operator's explicit destructive request not having happened, and reporting
|
||||
* success for that would leave them believing data is gone when it is not.
|
||||
*
|
||||
* Statements run serially and are not wrapped in a transaction, for the reason
|
||||
* the replay's header already gives: MariaDB commits DDL implicitly, so there is
|
||||
* no rollback to have. A purge that fails halfway has dropped some tables — the
|
||||
* error names the statement that stopped it, and running it again is safe
|
||||
* because a purge script is required to be idempotent in the same way a fragment
|
||||
* is (`DROP TABLE IF EXISTS`).
|
||||
*
|
||||
* @param {string} file absolute path to the module's purge.sql
|
||||
* @param {object} [deps] injection seam for tests
|
||||
* @returns {Promise<number>} how many statements ran
|
||||
*/
|
||||
async function runPurge(file, { query } = {}) {
|
||||
// eslint-disable-next-line global-require
|
||||
const run = query || require('../utils/db').query
|
||||
|
||||
const statements = splitStatements(fs.readFileSync(file, 'utf8'))
|
||||
let ran = 0
|
||||
for (const statement of statements) {
|
||||
try {
|
||||
await run(statement)
|
||||
ran += 1
|
||||
} catch (err) {
|
||||
throw new Error(`purge failed at statement ${ran + 1} of ${statements.length}: ${err.message}`)
|
||||
}
|
||||
}
|
||||
return ran
|
||||
}
|
||||
|
||||
module.exports = { replayFragments, runPurge }
|
||||
|
||||
@@ -30,6 +30,7 @@ const pagesRouter = require('./pages.router')
|
||||
const emailRouter = require('./email.router')
|
||||
const discordBotRouter = require('./discordBot.router')
|
||||
const settingsRouter = require('./settings.router')
|
||||
const modulesRouter = require('./modules.router')
|
||||
const dashboardRouter = require('./dashboard.router')
|
||||
|
||||
const adminRouter = express.Router()
|
||||
@@ -73,6 +74,11 @@ adminRouter.use('/pages', pagesRouter)
|
||||
adminRouter.use('/email', emailRouter)
|
||||
adminRouter.use('/discord-bot', discordBotRouter)
|
||||
adminRouter.use('/settings', settingsRouter)
|
||||
// The Modules screen (Phase 4). Core's, not a module's — and it has to be
|
||||
// core's: it is how a module gets onto the volume in the first place. Mounted
|
||||
// here alongside the other configuration capabilities, and admin-only per route
|
||||
// rather than at this line, so the gate sits next to what it is guarding.
|
||||
adminRouter.use('/modules', modulesRouter)
|
||||
|
||||
// The two singletons that own no path segment of their own: GET /dashboard and
|
||||
// PUT /site-mode. Mounted at the group root, last, exactly where the residual
|
||||
|
||||
379
server/src/router/v1/admin/modules.controller.js
Normal file
379
server/src/router/v1/admin/modules.controller.js
Normal file
@@ -0,0 +1,379 @@
|
||||
// ── Admin: installed modules ───────────────────────────────────────────────
|
||||
//
|
||||
// Phase 4, slice 1 of docs/website/MODULE_SYSTEM.md §2.7.2. Admin-only, and more
|
||||
// so than anything else in this directory: installing a module puts JavaScript on
|
||||
// the volume that core will `require` into its own process on the next boot. That
|
||||
// is the feature — it is what "an operator never builds anything" means (§1.14) —
|
||||
// but it is worth being plain that this controller is remote code execution with
|
||||
// an audit trail, not a settings screen.
|
||||
//
|
||||
// What guards it, in the order an attacker would meet them:
|
||||
//
|
||||
// 1. `requireRole('admin')` on every route, on top of the group's staff gate.
|
||||
// 2. An `https`-only host allowlist, re-checked on every redirect hop, so a
|
||||
// pasted URL cannot be pointed at the compose network or a metadata service
|
||||
// (install.js).
|
||||
// 3. The sha256 the release published, compared against the bytes that arrived.
|
||||
// 4. A full inspection of the archive before a byte of it is unpacked, and an
|
||||
// unpack into a scratch directory that is only moved into place once the
|
||||
// bundle has agreed with the manifest about what it is (archive.js).
|
||||
// 5. Every action here writes to core's one audit log.
|
||||
//
|
||||
// The one thing this file cannot do is mount anything. §1.12 makes the volume the
|
||||
// mounting source of truth, read once at require time, so install and uninstall
|
||||
// take effect on the next boot — which is why `restart` is a route here rather
|
||||
// than a sentence in a tooltip (decision 1).
|
||||
|
||||
const modules = require('../../../model/modules/modules.model')
|
||||
const activity = require('../../../model/activity/activity.model')
|
||||
const settings = require('../../../model/settings/settings.model')
|
||||
const loader = require('../../../modules/loader')
|
||||
const lifecycle = require('../../../modules/lifecycle')
|
||||
const install = require('../../../modules/install')
|
||||
// A namespace import, like every other require in this file, and not
|
||||
// `const { runPurge } = …`: destructuring at require time captures the function
|
||||
// rather than the module, which makes it the one dependency here that cannot be
|
||||
// substituted. That matters because the two tests worth having about purge are
|
||||
// about the ORDER it runs in relative to the directory being removed.
|
||||
const schema = require('../../../modules/schema')
|
||||
|
||||
const log = require('../../../utils/logger')('admin-modules')
|
||||
|
||||
// The allowlist setting. Seeded from MODULE_SOURCE_HOSTS on first boot and
|
||||
// admin-managed from then on (decision 6) — db/seed.js writes it once and never
|
||||
// overwrites it, so changing the variable later does not silently reach in and
|
||||
// undo an operator's choice.
|
||||
const HOSTS_KEY = 'module_source_hosts'
|
||||
|
||||
// A hostname, not a URL: no scheme, no path, no port, no wildcard. Deliberately
|
||||
// strict — every character allowed here is a character that can appear in the
|
||||
// host of a URL this server will fetch and execute the contents of.
|
||||
const HOSTNAME = /^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)*$/
|
||||
|
||||
async function allowedHosts() {
|
||||
return install.parseHosts(await settings.get(HOSTS_KEY))
|
||||
}
|
||||
|
||||
/**
|
||||
* One module, as the admin screen needs it.
|
||||
*
|
||||
* Three sources have to be reconciled, and which one answers which question is
|
||||
* the whole of §2.4:
|
||||
*
|
||||
* - the ROW says what the operator decided and what the last boot recorded;
|
||||
* - the LOADER says what is mounted and answering right now;
|
||||
* - the VOLUME says whether there is still a directory there at all.
|
||||
*
|
||||
* They can legitimately disagree, and the screen has to show that rather than
|
||||
* pick a winner. A row `enabled` with a loader state of `disabled` is a module
|
||||
* the operator has just switched back on and which is waiting for a restart —
|
||||
* exactly the case decision 3 creates, and it would be a lie to render it as
|
||||
* either "running" or "off".
|
||||
*/
|
||||
function present(row, live, onVolume) {
|
||||
return {
|
||||
id: row ? row.id : live.id,
|
||||
name: row ? row.name : live.name,
|
||||
version: row ? row.version : live.version,
|
||||
// What the database records.
|
||||
state: row ? row.state : null,
|
||||
failureStage: row ? row.failureStage : (live && live.stage) || null,
|
||||
failureReason: row ? row.failureReason : (live && live.reason) || null,
|
||||
source: row ? row.source : null,
|
||||
sha256: row ? row.sha256 : null,
|
||||
installedAt: row ? row.installedAt : null,
|
||||
startedAt: row ? row.startedAt : null,
|
||||
// What is actually mounted in this process, and what it is answering.
|
||||
liveState: live ? live.state : null,
|
||||
capabilities: live ? live.capabilities : [],
|
||||
// What is on the volume.
|
||||
onVolume,
|
||||
canPurge: onVolume && Boolean(install.purgeFile(row ? row.id : live.id)),
|
||||
}
|
||||
}
|
||||
|
||||
// GET /admin/modules — every module core knows about, from all three sources,
|
||||
// plus the source allowlist the install form needs.
|
||||
async function list(req, res) {
|
||||
try {
|
||||
const rows = await modules.list()
|
||||
// The loader throws rather than returning [] before load() has run (§7.6),
|
||||
// and this controller is reachable from a process where that is true —
|
||||
// `npm run seed` never gets here, but a test harness might.
|
||||
const live = loader.isLoaded() ? loader.list() : []
|
||||
const byId = new Map(live.map((m) => [m.id, m]))
|
||||
|
||||
const seen = new Set()
|
||||
const out = []
|
||||
for (const row of rows) {
|
||||
seen.add(row.id)
|
||||
out.push(present(row, byId.get(row.id) || null, install.isInstalled(row.id)))
|
||||
}
|
||||
// A directory on the volume that has no row yet — a hand-placed install
|
||||
// before its first boot. It has to be listed, or the screen would show
|
||||
// nothing for a module whose routes are already being served.
|
||||
for (const m of live) {
|
||||
if (!seen.has(m.id)) out.push(present(null, m, true))
|
||||
}
|
||||
|
||||
return res.json({ modules: out, sourceHosts: await allowedHosts() })
|
||||
} catch (err) {
|
||||
log.error('list modules', err)
|
||||
return res.status(500).json({ message: 'Internal Server Error' })
|
||||
}
|
||||
}
|
||||
|
||||
// POST /admin/modules — install (or upgrade) from an install-manifest URL.
|
||||
async function create(req, res) {
|
||||
const url = String(req.body.url || '').trim()
|
||||
try {
|
||||
const hosts = await allowedHosts()
|
||||
const result = await install.install({ url, hosts })
|
||||
|
||||
// Provenance is written here and nowhere else: the boot reconcile records a
|
||||
// module with NULL source/sha256 and leaves what it is not given, precisely
|
||||
// so that a refresh cannot overwrite what an install knew (lifecycle.js).
|
||||
const row = await modules.recordInstalled({
|
||||
id: result.id,
|
||||
name: result.name,
|
||||
version: result.version,
|
||||
source: result.source,
|
||||
sha256: result.sha256,
|
||||
})
|
||||
|
||||
await activity.log({
|
||||
req,
|
||||
userId: req.user.id,
|
||||
action: 'module.install',
|
||||
detail: { id: result.id, version: result.version, source: url, sha256: result.sha256, replaced: result.replaced },
|
||||
})
|
||||
log.warn('module installed — it will mount on the next restart', {
|
||||
id: result.id,
|
||||
version: result.version,
|
||||
by: req.user.username,
|
||||
})
|
||||
|
||||
return res.status(201).json({ module: row, restartRequired: true, replaced: result.replaced })
|
||||
} catch (err) {
|
||||
if (err.name === 'InstallError' || err.name === 'ArchiveError') {
|
||||
// The operator pasted a URL and something about what came back was wrong.
|
||||
// The message is the useful part and is written to be read by them.
|
||||
log.warn('module install refused', { url, reason: err.message })
|
||||
return res.status(err.status || 400).json({ message: err.message })
|
||||
}
|
||||
log.error('install module', err)
|
||||
return res.status(500).json({ message: 'Internal Server Error' })
|
||||
}
|
||||
}
|
||||
|
||||
// POST /admin/modules/:id/enable — switch a module back on, for the next boot.
|
||||
//
|
||||
// Deliberately does NOT touch the loader's record. Disable ran the module's
|
||||
// onShutdown (decision 3), and there is no onBoot re-dispatch to undo that: a
|
||||
// module whose sockets were closed and timers cleared cannot be made to serve
|
||||
// again by flipping a flag, and pretending otherwise would put it back on the
|
||||
// nav with a torn-down world behind it. The row moves; the restart starts it.
|
||||
async function enable(req, res) {
|
||||
const { id } = req.params
|
||||
try {
|
||||
const row = await modules.enable(id)
|
||||
if (!row) return res.status(404).json({ message: 'No such module.' })
|
||||
|
||||
await activity.log({ req, userId: req.user.id, action: 'module.enable', detail: { id } })
|
||||
return res.json({ module: row, restartRequired: true })
|
||||
} catch (err) {
|
||||
if (err.name === 'ModuleStateError') return res.status(409).json({ message: err.message })
|
||||
log.error('enable module', err)
|
||||
return res.status(500).json({ message: 'Internal Server Error' })
|
||||
}
|
||||
}
|
||||
|
||||
// POST /admin/modules/:id/disable — stop it now.
|
||||
//
|
||||
// The one action on this screen that takes effect without a restart, and the
|
||||
// reason it does is that it is the one an operator reaches for when something is
|
||||
// going wrong. Its routes answer 404 from the moment this returns, and its
|
||||
// onShutdown has already run.
|
||||
async function disable(req, res) {
|
||||
const { id } = req.params
|
||||
try {
|
||||
const current = await modules.get(id)
|
||||
if (!current) return res.status(404).json({ message: 'No such module.' })
|
||||
|
||||
const { stopped, error } = await lifecycle.stop(id)
|
||||
const row = await modules.get(id)
|
||||
|
||||
await activity.log({
|
||||
req,
|
||||
userId: req.user.id,
|
||||
action: 'module.disable',
|
||||
detail: { id, hookRan: stopped, hookError: error },
|
||||
})
|
||||
log.warn('module disabled by an operator', { id, hookRan: stopped, by: req.user.username })
|
||||
|
||||
// `shutdownError` is reported rather than swallowed: the module IS disabled
|
||||
// either way, and an operator whose module could not close cleanly should be
|
||||
// told so while they still have the logs to look at.
|
||||
return res.json({ module: row, stopped, shutdownError: error })
|
||||
} catch (err) {
|
||||
log.error('disable module', err)
|
||||
return res.status(500).json({ message: 'Internal Server Error' })
|
||||
}
|
||||
}
|
||||
|
||||
// DELETE /admin/modules/:id[?purge=true] — uninstall.
|
||||
//
|
||||
// Non-destructive by default (§2.5): the directory goes, the row stays
|
||||
// `disabled`, and the module's tables and data are left alone.
|
||||
//
|
||||
// The purge option is here rather than as a follow-up action because it cannot
|
||||
// be a follow-up action (decision 5): `purge.sql` is a file inside the directory
|
||||
// this is about to delete, so after an uninstall there is nothing left to purge
|
||||
// with. Ticking the box is the last moment the file exists.
|
||||
//
|
||||
// The order below is the whole of it, and each step depends on the one above:
|
||||
// purge while the SQL is still readable, stop while the code is still loaded,
|
||||
// then delete.
|
||||
async function remove(req, res) {
|
||||
const { id } = req.params
|
||||
const purge = req.query.purge === 'true' || req.query.purge === '1'
|
||||
try {
|
||||
const current = await modules.get(id)
|
||||
const onVolume = install.isInstalled(id)
|
||||
if (!current && !onVolume) return res.status(404).json({ message: 'No such module.' })
|
||||
|
||||
let purged = null
|
||||
if (purge) {
|
||||
const file = install.purgeFile(id)
|
||||
if (!file) {
|
||||
return res.status(400).json({
|
||||
message: 'This module ships no purge.sql, so its data cannot be deleted. Uninstall without purging instead.',
|
||||
})
|
||||
}
|
||||
purged = await schema.runPurge(file)
|
||||
}
|
||||
|
||||
// Stop it before its files vanish. A module whose directory is deleted out
|
||||
// from under a running onShutdown is being asked to tear down a world whose
|
||||
// code may already be half-unreadable — and its sockets would otherwise stay
|
||||
// open until the restart, holding a connection on behalf of a module that no
|
||||
// longer exists on disk.
|
||||
await lifecycle.stop(id)
|
||||
const removed = await install.removeDir(id)
|
||||
|
||||
// A purge leaves nothing: no directory, no tables, no data. Keeping a
|
||||
// `disabled` row for that is a tombstone with nothing to offer and a Purge
|
||||
// button that would fail. A plain uninstall keeps its row, which is what
|
||||
// makes the retained data visible and reinstallable.
|
||||
if (purge) await modules.remove(id)
|
||||
|
||||
await activity.log({
|
||||
req,
|
||||
userId: req.user.id,
|
||||
action: purge ? 'module.purge' : 'module.uninstall',
|
||||
detail: { id, purged, removed },
|
||||
})
|
||||
log.warn(`module ${purge ? 'uninstalled and purged' : 'uninstalled'}`, {
|
||||
id,
|
||||
statements: purged,
|
||||
by: req.user.username,
|
||||
})
|
||||
|
||||
return res.json({ id, removed, purged, restartRequired: true })
|
||||
} catch (err) {
|
||||
log.error('uninstall module', err)
|
||||
return res.status(500).json({ message: err.message || 'Internal Server Error' })
|
||||
}
|
||||
}
|
||||
|
||||
// POST /admin/modules/:id/purge — drop a still-installed module's data.
|
||||
//
|
||||
// Refuses unless the module is already disabled, and that guard is the point:
|
||||
// dropping the tables under a module that is still serving requests leaves it
|
||||
// answering out of a world that no longer exists. Disabling first is one click
|
||||
// and makes the destructive step happen against something that has stopped.
|
||||
async function purge(req, res) {
|
||||
const { id } = req.params
|
||||
try {
|
||||
const current = await modules.get(id)
|
||||
if (!current) return res.status(404).json({ message: 'No such module.' })
|
||||
if (current.state !== 'disabled') {
|
||||
return res.status(409).json({
|
||||
message: 'Disable this module before purging its data, so nothing is serving out of the tables being dropped.',
|
||||
})
|
||||
}
|
||||
|
||||
const file = install.purgeFile(id)
|
||||
if (!file) {
|
||||
return res.status(400).json({ message: 'This module ships no purge.sql, so its data cannot be deleted.' })
|
||||
}
|
||||
|
||||
const statements = await schema.runPurge(file)
|
||||
await activity.log({ req, userId: req.user.id, action: 'module.purge', detail: { id, statements } })
|
||||
log.warn('module data purged', { id, statements, by: req.user.username })
|
||||
|
||||
return res.json({ id, purged: statements })
|
||||
} catch (err) {
|
||||
log.error('purge module', err)
|
||||
return res.status(500).json({ message: err.message || 'Internal Server Error' })
|
||||
}
|
||||
}
|
||||
|
||||
// PUT /admin/modules/sources — the host allowlist.
|
||||
async function setSources(req, res) {
|
||||
const hosts = install.parseHosts(req.body.hosts)
|
||||
const bad = hosts.find((h) => !HOSTNAME.test(h))
|
||||
if (bad) return res.status(400).json({ message: `"${bad}" is not a valid hostname.` })
|
||||
|
||||
try {
|
||||
const before = await allowedHosts()
|
||||
await settings.set(HOSTS_KEY, hosts.join(','), req.user.id)
|
||||
await activity.log({
|
||||
req,
|
||||
userId: req.user.id,
|
||||
action: 'module.sources',
|
||||
detail: { before, after: hosts },
|
||||
})
|
||||
log.warn('module source allowlist changed', { before, after: hosts, by: req.user.username })
|
||||
return res.json({ sourceHosts: hosts })
|
||||
} catch (err) {
|
||||
log.error('set module sources', err)
|
||||
return res.status(500).json({ message: 'Internal Server Error' })
|
||||
}
|
||||
}
|
||||
|
||||
// POST /admin/modules/restart — restart the server process.
|
||||
//
|
||||
// Decision 1. Install, uninstall and re-enable all only take effect at boot
|
||||
// because §1.12 reads the volume at require time, and §2.4 promises recovery
|
||||
// "with no shell access to the box" — which a banner saying "please restart your
|
||||
// container" does not deliver.
|
||||
//
|
||||
// It raises SIGTERM against its own process rather than calling the shutdown
|
||||
// path directly. server.js already has a handler that stops the modules, the
|
||||
// workers and the listeners in the right order and closes the pool and the log
|
||||
// file before exiting 0; reaching that through the signal means there is exactly
|
||||
// one graceful-shutdown path and this route cannot drift from it.
|
||||
//
|
||||
// What brings the process BACK is the supervisor, not this. The shipped
|
||||
// docker-compose.yml declares `restart: unless-stopped` on `app`, which restarts
|
||||
// on a clean exit as well as a crash. A bare `npm start` does not come back, and
|
||||
// the screen says so before it asks.
|
||||
function restart(req, res) {
|
||||
log.warn('restart requested from the admin panel', { by: req.user.username })
|
||||
|
||||
// Logged and answered first. Once the signal is raised the response has no
|
||||
// listener left to flush through, so the operator would be told nothing.
|
||||
res.status(202).json({ restarting: true })
|
||||
|
||||
activity
|
||||
.log({ req, userId: req.user.id, action: 'module.restart', detail: {} })
|
||||
.catch((err) => log.error('failed to record the restart in the audit log', err))
|
||||
.finally(() => {
|
||||
// A beat, so the 202 is on the wire. `unref` so this timer is not itself
|
||||
// something keeping the process alive.
|
||||
setTimeout(() => process.kill(process.pid, 'SIGTERM'), 250).unref()
|
||||
})
|
||||
}
|
||||
|
||||
module.exports = { list, create, enable, disable, remove, purge, setSources, restart, HOSTS_KEY }
|
||||
143
server/src/router/v1/admin/modules.router.js
Normal file
143
server/src/router/v1/admin/modules.router.js
Normal file
@@ -0,0 +1,143 @@
|
||||
// Admin · Modules — install, enable, disable, uninstall, purge and restart.
|
||||
//
|
||||
// Mounted at /api/v1/admin/modules by admin/index.js, which has already applied
|
||||
// `noindex, isLoggedIn, staffOnly`. Every route here re-gates to `admin`: an
|
||||
// editor or moderator has no business installing code into the server process,
|
||||
// and the group gate alone would let them.
|
||||
//
|
||||
// Route order matters in one place. `/restart` and `/sources` are declared
|
||||
// BEFORE the `/:id/...` routes, because express matches in declaration order and
|
||||
// a module whose id was `restart` would otherwise shadow — or be shadowed by —
|
||||
// the literal path. The id pattern below makes that unreachable in practice; the
|
||||
// ordering makes it unreachable by construction.
|
||||
|
||||
const express = require('express')
|
||||
const { body, param, query } = require('express-validator')
|
||||
|
||||
const controller = require('./modules.controller')
|
||||
const { requireRole } = require('../../../utils/auth')
|
||||
const validate = require('../../../middleware/validate')
|
||||
|
||||
const modulesRouter = express.Router()
|
||||
const adminOnly = requireRole('admin')
|
||||
|
||||
// The loader's own id rule (MODULE_API.md §2.1). Applied at the edge so a
|
||||
// traversal-shaped id never reaches a path join, even though install.js checks
|
||||
// it again — this one produces a 400 with a readable message, that one is the
|
||||
// guarantee.
|
||||
const ID = /^[a-z][a-z0-9-]{1,31}$/
|
||||
|
||||
modulesRouter.get(
|
||||
'/',
|
||||
// #swagger.tags = ['Admin · Modules']
|
||||
// #swagger.summary = 'List installed modules, their live state, and the source allowlist'
|
||||
// #swagger.security = [{ "cookieAuth": [] }, { "bearerAuth": [] }]
|
||||
/* #swagger.responses[200] = { description: 'Modules and the install source allowlist', content: { "application/json": { schema: { type: "object", properties: { modules: { type: "array", items: { type: "object", additionalProperties: true } }, sourceHosts: { type: "array", items: { type: "string" } } } } } } } */
|
||||
adminOnly,
|
||||
controller.list,
|
||||
)
|
||||
|
||||
modulesRouter.post(
|
||||
'/',
|
||||
// #swagger.tags = ['Admin · Modules']
|
||||
// #swagger.summary = 'Install or upgrade a module from a release install-manifest URL'
|
||||
// #swagger.description = 'Downloads the artifact the manifest names, verifies its sha256, inspects the archive in full and unpacks it onto the modules volume. The module mounts on the next restart.'
|
||||
// #swagger.security = [{ "cookieAuth": [] }, { "bearerAuth": [] }]
|
||||
/* #swagger.requestBody = { required: true, content: { "application/json": { schema: { type: "object", required: ["url"], properties: { url: { type: "string", description: "https URL of the release install manifest, on an allowed host" } } } } } } */
|
||||
/* #swagger.responses[201] = { description: 'Installed — restart to mount it', content: { "application/json": { schema: { type: "object", additionalProperties: true } } } } */
|
||||
/* #swagger.responses[400] = { description: 'The URL, the manifest, the hash or the archive was refused', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
|
||||
/* #swagger.responses[502] = { description: 'The source host could not be reached or answered badly', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
|
||||
adminOnly,
|
||||
body('url').isString().trim().isLength({ min: 1, max: 2048 }),
|
||||
validate,
|
||||
controller.create,
|
||||
)
|
||||
|
||||
modulesRouter.put(
|
||||
'/sources',
|
||||
// #swagger.tags = ['Admin · Modules']
|
||||
// #swagger.summary = 'Replace the allowlist of hosts modules may be installed from'
|
||||
// #swagger.security = [{ "cookieAuth": [] }, { "bearerAuth": [] }]
|
||||
/* #swagger.requestBody = { required: true, content: { "application/json": { schema: { type: "object", required: ["hosts"], properties: { hosts: { type: "string", description: "Comma- or space-separated hostnames. An empty list forbids all installs." } } } } } } */
|
||||
/* #swagger.responses[200] = { description: 'The new allowlist', content: { "application/json": { schema: { type: "object", properties: { sourceHosts: { type: "array", items: { type: "string" } } } } } } } */
|
||||
/* #swagger.responses[400] = { description: 'One of the entries is not a hostname', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
|
||||
adminOnly,
|
||||
body('hosts').isString().isLength({ max: 2048 }),
|
||||
validate,
|
||||
controller.setSources,
|
||||
)
|
||||
|
||||
modulesRouter.post(
|
||||
'/restart',
|
||||
// #swagger.tags = ['Admin · Modules']
|
||||
// #swagger.summary = 'Restart the server process so module changes take effect'
|
||||
// #swagger.description = 'Runs the same graceful shutdown a SIGTERM does. The process is brought back by the supervisor, which the shipped docker-compose.yml provides; a bare `npm start` will not come back.'
|
||||
// #swagger.security = [{ "cookieAuth": [] }, { "bearerAuth": [] }]
|
||||
/* #swagger.responses[202] = { description: 'Shutting down', content: { "application/json": { schema: { type: "object", properties: { restarting: { type: "boolean" } } } } } } */
|
||||
adminOnly,
|
||||
controller.restart,
|
||||
)
|
||||
|
||||
modulesRouter.post(
|
||||
'/:id/enable',
|
||||
// #swagger.tags = ['Admin · Modules']
|
||||
// #swagger.summary = 'Enable a module (takes effect on the next restart)'
|
||||
// #swagger.security = [{ "cookieAuth": [] }, { "bearerAuth": [] }]
|
||||
// #swagger.parameters['id'] = { in: 'path', required: true, schema: { type: 'string' }, description: 'Module id.' }
|
||||
/* #swagger.responses[200] = { description: 'Enabled — restart to start it', content: { "application/json": { schema: { type: "object", additionalProperties: true } } } } */
|
||||
/* #swagger.responses[404] = { description: 'No such module', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
|
||||
adminOnly,
|
||||
param('id').matches(ID),
|
||||
validate,
|
||||
controller.enable,
|
||||
)
|
||||
|
||||
modulesRouter.post(
|
||||
'/:id/disable',
|
||||
// #swagger.tags = ['Admin · Modules']
|
||||
// #swagger.summary = 'Stop a module now — runs its onShutdown, then its routes answer 404'
|
||||
// #swagger.description = 'The only module action that takes effect without a restart. Re-enabling needs one, because there is no onBoot re-dispatch.'
|
||||
// #swagger.security = [{ "cookieAuth": [] }, { "bearerAuth": [] }]
|
||||
// #swagger.parameters['id'] = { in: 'path', required: true, schema: { type: 'string' }, description: 'Module id.' }
|
||||
/* #swagger.responses[200] = { description: 'Disabled', content: { "application/json": { schema: { type: "object", additionalProperties: true } } } } */
|
||||
/* #swagger.responses[404] = { description: 'No such module', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
|
||||
adminOnly,
|
||||
param('id').matches(ID),
|
||||
validate,
|
||||
controller.disable,
|
||||
)
|
||||
|
||||
modulesRouter.post(
|
||||
'/:id/purge',
|
||||
// #swagger.tags = ['Admin · Modules']
|
||||
// #swagger.summary = "Run a disabled module’s purge.sql, dropping its tables and data"
|
||||
// #swagger.security = [{ "cookieAuth": [] }, { "bearerAuth": [] }]
|
||||
// #swagger.parameters['id'] = { in: 'path', required: true, schema: { type: 'string' }, description: 'Module id.' }
|
||||
/* #swagger.responses[200] = { description: 'Purged', content: { "application/json": { schema: { type: "object", properties: { id: { type: "string" }, purged: { type: "integer" } } } } } } */
|
||||
/* #swagger.responses[400] = { description: 'The module ships no purge.sql', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
|
||||
/* #swagger.responses[409] = { description: 'The module must be disabled first', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
|
||||
adminOnly,
|
||||
param('id').matches(ID),
|
||||
validate,
|
||||
controller.purge,
|
||||
)
|
||||
|
||||
modulesRouter.delete(
|
||||
'/:id',
|
||||
// #swagger.tags = ['Admin · Modules']
|
||||
// #swagger.summary = 'Uninstall a module, optionally deleting its data too'
|
||||
// #swagger.description = "Removes the module directory and leaves its row disabled. With purge=true it also runs purge.sql first — which is the only moment it can, since purge.sql lives inside the directory being deleted."
|
||||
// #swagger.security = [{ "cookieAuth": [] }, { "bearerAuth": [] }]
|
||||
// #swagger.parameters['id'] = { in: 'path', required: true, schema: { type: 'string' }, description: 'Module id.' }
|
||||
// #swagger.parameters['purge'] = { in: 'query', required: false, schema: { type: 'boolean' }, description: "Also run the module’s purge.sql and drop its row. Destructive and irreversible." }
|
||||
/* #swagger.responses[200] = { description: 'Uninstalled — restart to unmount it', content: { "application/json": { schema: { type: "object", additionalProperties: true } } } } */
|
||||
/* #swagger.responses[400] = { description: 'Purge was asked for and the module ships no purge.sql', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
|
||||
/* #swagger.responses[404] = { description: 'No such module', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
|
||||
adminOnly,
|
||||
param('id').matches(ID),
|
||||
query('purge').optional().isIn(['true', 'false', '1', '0']),
|
||||
validate,
|
||||
controller.remove,
|
||||
)
|
||||
|
||||
module.exports = modulesRouter
|
||||
Reference in New Issue
Block a user