feat(modules): install, uninstall, purge and restart (phase 4, slice 1)
The consumer half of a release module-uo's CI has been publishing since
phase 3 closed. Before this, core had the installed_modules provenance
columns and no code that could ever fill them: nothing fetched, verified,
unpacked, removed or purged anything, and there was no admin route at all.
Adds modules/archive.js, modules/install.js, schema.runPurge(),
lifecycle.stop(), loader.stopHook(), and /api/v1/admin/modules with eight
routes. 797 server tests (+76), manifest 158 -> 166 + 2 internal, OpenAPI
gains 8 operations and loses nothing.
Reject, never sanitise
----------------------
The download is the easy part: an https-only allowlist re-checked on every
redirect hop, a declared sha256 compared against the bytes that arrived, and
a byte cap. Unpacking is where the archive chooses the filenames, and core
writes into a directory bind-mounted from the host, so an escape is not
confined to the container.
archive.js inspects the whole archive before a byte is unpacked and refuses
absolute and drive-absolute paths, `..` segments, NUL bytes, backslashes,
anything that is not a regular file or a directory, more than one top-level
entry, and anything over the entry or byte caps. Refusing symlinks and
hardlinks outright is what keeps this off the majority of node-tar's
published advisories rather than depending on the library to contain them.
That two-pass shape is load-bearing, and it was measured rather than assumed:
extracting an archive whose fourth member escapes upward throws under
node-tar 7.5.22 -- and leaves the first three members on disk. The loader
scans that directory at require time on the next boot, so a half-unpacked
module is a module. Everything therefore happens in a scratch directory that
is removed on any failure, and the move into place is the last step.
`tar` is pinned to ^7.5.22 rather than the ^6 that installs by default: 6.x
is flagged critical, and reading the advisory list is what the file's header
now says out loud -- almost all of it is hardlink or symlink traversal and
PAX header interpretation differentials, which is exactly this feature's
threat model.
Two things the plan had wrong
-----------------------------
The bundle's top-level directory is `module-uo-<version>`, not the module id
-- so "the top-level name must equal the id" was checked against nothing real.
The extractor strips that level instead, because its name belongs to whoever
published the bundle and the directory it lands in is core's. What is checked
instead is the unpacked module.json: a manifest promising `uo` and delivering
something else is refused rather than installed under the name it promised.
And purge cannot be a follow-up action (decision 5): purge.sql lives inside
the directory uninstall deletes. It is offered in the uninstall flow and as a
standalone action on a still-installed module, and the standalone one refuses
unless the module is already disabled -- dropping tables under something that
is still serving leaves it answering out of a world that no longer exists.
Disable now means stopped
-------------------------
lifecycle.stop() dispatches that one module's onShutdown before flipping the
guard, so a module an operator switches off actually releases its sockets and
closes its streams instead of merely becoming unreachable. The hook runs
first and the state moves after it, because while onShutdown runs the module
is still `started` and that is the only state in which its routes and the
world it is tearing down agree. A hook that throws does not stop the disable
-- the opposite of the boot path's rule, and deliberately.
Enable is not its mirror and there is no start(id) beside it. There is no
onBoot re-dispatch and the hooks were never promised re-entrant, so enable
moves the row and the restart route starts it. A test pins that enable does
not touch the loader, because "fixing" it is a one-line change that would put
a module with closed sockets back on the nav.
Restart raises SIGTERM against its own process rather than calling the
shutdown path directly, so server.js's handler stays the one graceful-shutdown
path and this route cannot drift from it.
The allowlist bootstraps from MODULE_SOURCE_HOSTS into a settings row and is
admin-managed after that (decision 6); seedDefault is INSERT IGNORE, so
changing the variable on an existing deployment is a no-op by design. An empty
list forbids every install rather than allowing every host -- the safe
direction for a value someone might blank by accident.
Verified against the real v0.3.0 release
----------------------------------------
Not a fixture: fetched the published install manifest over the real Gitea
host and its redirect chain, verified the sha256, inspected and unpacked the
252,517-byte artifact to 82 files, and then booted core against the result --
the module registered its five mounts, seven streams and eight capabilities
and resolved its client chunk, with no scratch directory left behind.
Two defects this slice's own tooling caught, both of which had already been
written down as classes:
- the controller destructured runPurge at require time, capturing the
function rather than the module, which made the one dependency whose
ORDER matters the one that could not be substituted;
- two swagger annotations carried an apostrophe inside a quoted string,
dropped silently by swagger-autogen before slice 5 taught it to fail loudly.
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
252
server/test/moduleArchive.test.js
Normal file
252
server/test/moduleArchive.test.js
Normal file
@@ -0,0 +1,252 @@
|
||||
// modules/archive.js — the hardened bundle extractor (MODULE_SYSTEM.md §2.7.2).
|
||||
//
|
||||
// Every rejection case is exercised against a REAL archive rather than a mocked
|
||||
// tar parser, because the thing being tested is what the parser reports for a
|
||||
// given sequence of bytes. A fake that returns `{type: 'SymbolicLink'}` proves
|
||||
// only that the `if` is spelled correctly.
|
||||
//
|
||||
// The hostile archives are written here as raw ustar headers instead of being
|
||||
// produced with `tar`, for two reasons that both bit during this slice:
|
||||
//
|
||||
// 1. `ln -s` needs a privilege Windows does not hand out by default, so a
|
||||
// symlink fixture built with the shell is a fixture that silently is not
|
||||
// one, and the test passes for the wrong reason on the machine most of this
|
||||
// work happens on.
|
||||
// 2. GNU tar will not emit `../escape` or `/etc/passwd` as a member name — it
|
||||
// strips them and tells you so. The archives worth defending against are
|
||||
// exactly the ones a cooperative archiver refuses to produce.
|
||||
//
|
||||
// A ustar header is 512 bytes of fixed-offset fields, so writing one is less
|
||||
// code than persuading a tool to misbehave.
|
||||
|
||||
const test = require('node:test')
|
||||
const assert = require('node:assert/strict')
|
||||
const fs = require('fs')
|
||||
const os = require('os')
|
||||
const path = require('path')
|
||||
const zlib = require('zlib')
|
||||
|
||||
const archive = require('../src/modules/archive')
|
||||
|
||||
// ── A minimal ustar writer ─────────────────────────────────────────────────
|
||||
|
||||
const BLOCK = 512
|
||||
|
||||
function octal(value, width) {
|
||||
// ustar numeric fields are NUL-terminated octal, right-aligned with zeros.
|
||||
return Number(value).toString(8).padStart(width - 1, '0') + '\0'
|
||||
}
|
||||
|
||||
/**
|
||||
* One 512-byte header plus its padded data.
|
||||
*
|
||||
* @param {object} entry
|
||||
* @param {string} entry.name member path
|
||||
* @param {string} [entry.type] '0' file · '5' dir · '2' symlink · '1' hardlink
|
||||
* · '3' char dev · '6' FIFO
|
||||
* @param {string} [entry.linkname] target, for the link types
|
||||
* @param {string} [entry.body] file contents
|
||||
* @param {number} [entry.size] declared size — defaults to the body's, and
|
||||
* may be set independently to build a header
|
||||
* that lies about its payload
|
||||
*/
|
||||
function member({ name, type = '0', linkname = '', body = '', size = null }) {
|
||||
const header = Buffer.alloc(BLOCK, 0)
|
||||
const data = Buffer.from(body, 'utf8')
|
||||
const declared = size === null ? data.length : size
|
||||
|
||||
header.write(name, 0, 100, 'utf8')
|
||||
header.write(octal(0o644, 8), 100, 8, 'ascii') // mode
|
||||
header.write(octal(0, 8), 108, 8, 'ascii') // uid
|
||||
header.write(octal(0, 8), 116, 8, 'ascii') // gid
|
||||
header.write(octal(declared, 12), 124, 12, 'ascii')
|
||||
header.write(octal(0, 12), 136, 12, 'ascii') // mtime
|
||||
header.write(' ', 148, 8, 'ascii') // checksum field is spaces while summing
|
||||
header.write(type, 156, 1, 'ascii')
|
||||
header.write(linkname, 157, 100, 'utf8')
|
||||
header.write('ustar\0', 257, 6, 'ascii')
|
||||
header.write('00', 263, 2, 'ascii')
|
||||
|
||||
let sum = 0
|
||||
for (const byte of header) sum += byte
|
||||
header.write(`${sum.toString(8).padStart(6, '0')}\0 `, 148, 8, 'ascii')
|
||||
|
||||
const padding = Buffer.alloc((BLOCK - (data.length % BLOCK)) % BLOCK, 0)
|
||||
return Buffer.concat([header, data, padding])
|
||||
}
|
||||
|
||||
/** Gzip a set of members into a .tar.gz on disk, and return its path. */
|
||||
function writeArchive(dir, filename, members) {
|
||||
const tarball = Buffer.concat([
|
||||
...members.map(member),
|
||||
Buffer.alloc(BLOCK * 2, 0), // two zero blocks end the archive
|
||||
])
|
||||
const file = path.join(dir, filename)
|
||||
fs.writeFileSync(file, zlib.gzipSync(tarball))
|
||||
return file
|
||||
}
|
||||
|
||||
/** A well-formed bundle: one top-level directory, ordinary files inside it. */
|
||||
function goodMembers(root = 'module-uo-1.0.0') {
|
||||
return [
|
||||
{ name: `${root}/`, type: '5' },
|
||||
{ name: `${root}/module.json`, body: '{"id":"uo","name":"UO","version":"1.0.0"}' },
|
||||
{ name: `${root}/server/`, type: '5' },
|
||||
{ name: `${root}/server/index.js`, body: 'module.exports = () => {}\n' },
|
||||
]
|
||||
}
|
||||
|
||||
// One scratch directory for the whole file, removed at the end.
|
||||
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'rg-archive-'))
|
||||
test.after(() => fs.rmSync(tmp, { recursive: true, force: true }))
|
||||
|
||||
/** Assert that inspecting `members` fails, and that the message says why. */
|
||||
async function rejects(name, members, matcher) {
|
||||
const file = writeArchive(tmp, `${name}.tar.gz`, members)
|
||||
await assert.rejects(
|
||||
() => archive.inspect(file),
|
||||
(err) => {
|
||||
assert.equal(err.name, 'ArchiveError', `expected an ArchiveError, got ${err.name}: ${err.message}`)
|
||||
assert.match(err.message, matcher)
|
||||
return true
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
// ── What a good bundle does ────────────────────────────────────────────────
|
||||
|
||||
test('inspect accepts a well-formed bundle and reports its single root', async () => {
|
||||
const file = writeArchive(tmp, 'good.tar.gz', goodMembers())
|
||||
const stats = await archive.inspect(file)
|
||||
|
||||
assert.equal(stats.root, 'module-uo-1.0.0')
|
||||
assert.equal(stats.entries, 4)
|
||||
assert.ok(stats.bytes > 0)
|
||||
})
|
||||
|
||||
test('extract strips the top level, so the bundle lands as the module id', async () => {
|
||||
// The wrapper directory is the publisher's naming (module-uo's release
|
||||
// workflow packs `module-uo-<version>/`); the directory it lands in is core's,
|
||||
// and has to be the id the loader scans for.
|
||||
const file = writeArchive(tmp, 'strip.tar.gz', goodMembers())
|
||||
const dest = path.join(tmp, 'unpacked-uo')
|
||||
|
||||
await archive.unpack(file, dest)
|
||||
|
||||
assert.ok(fs.existsSync(path.join(dest, 'module.json')), 'module.json should be at the root of the destination')
|
||||
assert.ok(fs.existsSync(path.join(dest, 'server', 'index.js')))
|
||||
assert.ok(!fs.existsSync(path.join(dest, 'module-uo-1.0.0')), 'the wrapper directory should not survive')
|
||||
})
|
||||
|
||||
test('extract refuses a destination that already exists', async () => {
|
||||
const file = writeArchive(tmp, 'exists.tar.gz', goodMembers())
|
||||
const dest = path.join(tmp, 'already-there')
|
||||
fs.mkdirSync(dest)
|
||||
|
||||
await assert.rejects(() => archive.extract(file, dest), /already exists/)
|
||||
})
|
||||
|
||||
// ── What a hostile bundle does ─────────────────────────────────────────────
|
||||
|
||||
test('an absolute member path is refused', async () => {
|
||||
await rejects('absolute', [
|
||||
{ name: 'mod/', type: '5' },
|
||||
{ name: '/etc/cron.d/pwned', body: '* * * * * root sh\n' },
|
||||
], /absolute/)
|
||||
})
|
||||
|
||||
test('a drive-absolute member path is refused', async () => {
|
||||
// Its own node-tar advisory, and invisible to a leading-slash check.
|
||||
await rejects('drive', [
|
||||
{ name: 'mod/', type: '5' },
|
||||
{ name: 'C:\\Windows\\Temp\\pwned', body: 'x' },
|
||||
], /backslash|drive-absolute/)
|
||||
})
|
||||
|
||||
test('an upward-escaping member path is refused', async () => {
|
||||
await rejects('escape', [
|
||||
{ name: 'mod/', type: '5' },
|
||||
{ name: 'mod/../../../etc/passwd', body: 'root::0:0\n' },
|
||||
], /escapes upward/)
|
||||
})
|
||||
|
||||
test('a symlink member is refused, and the message names the type', async () => {
|
||||
await rejects('symlink', [
|
||||
{ name: 'mod/', type: '5' },
|
||||
{ name: 'mod/passwd', type: '2', linkname: '/etc/passwd' },
|
||||
], /SymbolicLink/)
|
||||
})
|
||||
|
||||
test('a hardlink member is refused', async () => {
|
||||
// The single most-published node-tar escape primitive. Refusing the type
|
||||
// outright is what keeps this file from depending on the library getting
|
||||
// hardlink containment right.
|
||||
await rejects('hardlink', [
|
||||
{ name: 'mod/', type: '5' },
|
||||
{ name: 'mod/shadow', type: '1', linkname: '../../../etc/shadow' },
|
||||
], /Link/)
|
||||
})
|
||||
|
||||
test('a device node is refused', async () => {
|
||||
await rejects('device', [
|
||||
{ name: 'mod/', type: '5' },
|
||||
{ name: 'mod/zero', type: '3', linkname: '' },
|
||||
], /may only contain files and directories/)
|
||||
})
|
||||
|
||||
test('a FIFO is refused', async () => {
|
||||
await rejects('fifo', [
|
||||
{ name: 'mod/', type: '5' },
|
||||
{ name: 'mod/pipe', type: '6' },
|
||||
], /may only contain files and directories/)
|
||||
})
|
||||
|
||||
test('two top-level directories are refused', async () => {
|
||||
await rejects('two-roots', [
|
||||
{ name: 'mod-a/', type: '5' },
|
||||
{ name: 'mod-a/module.json', body: '{}' },
|
||||
{ name: 'mod-b/', type: '5' },
|
||||
{ name: 'mod-b/module.json', body: '{}' },
|
||||
], /exactly one top-level directory, found 2/)
|
||||
})
|
||||
|
||||
test('a bundle whose declared sizes exceed the cap is refused before it is read to the end', async () => {
|
||||
// The header lies: it declares a gigabyte and carries nothing. That is the
|
||||
// decompression-bomb shape, and the point is that inspect() decides on the
|
||||
// DECLARED size without ever materialising the payload.
|
||||
await rejects('bomb', [
|
||||
{ name: 'mod/', type: '5' },
|
||||
{ name: 'mod/big', size: archive.MAX_BYTES + 1, body: '' },
|
||||
], /unpacks to more than/)
|
||||
})
|
||||
|
||||
test('an empty archive is refused', async () => {
|
||||
await rejects('empty', [], /empty/)
|
||||
})
|
||||
|
||||
// ── The path check on its own ──────────────────────────────────────────────
|
||||
//
|
||||
// pathProblem is exported so the cases that are awkward to express as archive
|
||||
// bytes can still be asserted directly.
|
||||
|
||||
test('pathProblem accepts ordinary bundle paths', () => {
|
||||
for (const ok of ['mod/module.json', 'mod/server/router/x.js', 'mod/a.b-c_d/e.js']) {
|
||||
assert.equal(archive.pathProblem(ok), null, `${ok} should be accepted`)
|
||||
}
|
||||
})
|
||||
|
||||
test('pathProblem rejects the escape shapes', () => {
|
||||
assert.match(archive.pathProblem('/etc/passwd'), /absolute/)
|
||||
assert.match(archive.pathProblem('C:/Windows/x'), /drive-absolute/)
|
||||
assert.match(archive.pathProblem('mod/../../x'), /escapes upward/)
|
||||
assert.match(archive.pathProblem('..'), /escapes upward/)
|
||||
assert.match(archive.pathProblem('mod\\x'), /backslash/)
|
||||
assert.match(archive.pathProblem('mod/\0/x'), /NUL byte/)
|
||||
})
|
||||
|
||||
test('pathProblem does not reject a filename that merely contains two dots', () => {
|
||||
// `..` is a SEGMENT, not a substring — a file called `version..js` is fine,
|
||||
// and a check written with `includes('..')` would refuse it.
|
||||
assert.equal(archive.pathProblem('mod/version..js'), null)
|
||||
assert.equal(archive.pathProblem('mod/..hidden'), null)
|
||||
})
|
||||
Reference in New Issue
Block a user