50d719cf46d9a95a9c026b199ba5396045471703
9 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| b30e82cde2 |
feat(modules): install, uninstall, purge and restart (phase 4, slice 1)
The consumer half of a release module-uo's CI has been publishing since
phase 3 closed. Before this, core had the installed_modules provenance
columns and no code that could ever fill them: nothing fetched, verified,
unpacked, removed or purged anything, and there was no admin route at all.
Adds modules/archive.js, modules/install.js, schema.runPurge(),
lifecycle.stop(), loader.stopHook(), and /api/v1/admin/modules with eight
routes. 797 server tests (+76), manifest 158 -> 166 + 2 internal, OpenAPI
gains 8 operations and loses nothing.
Reject, never sanitise
----------------------
The download is the easy part: an https-only allowlist re-checked on every
redirect hop, a declared sha256 compared against the bytes that arrived, and
a byte cap. Unpacking is where the archive chooses the filenames, and core
writes into a directory bind-mounted from the host, so an escape is not
confined to the container.
archive.js inspects the whole archive before a byte is unpacked and refuses
absolute and drive-absolute paths, `..` segments, NUL bytes, backslashes,
anything that is not a regular file or a directory, more than one top-level
entry, and anything over the entry or byte caps. Refusing symlinks and
hardlinks outright is what keeps this off the majority of node-tar's
published advisories rather than depending on the library to contain them.
That two-pass shape is load-bearing, and it was measured rather than assumed:
extracting an archive whose fourth member escapes upward throws under
node-tar 7.5.22 -- and leaves the first three members on disk. The loader
scans that directory at require time on the next boot, so a half-unpacked
module is a module. Everything therefore happens in a scratch directory that
is removed on any failure, and the move into place is the last step.
`tar` is pinned to ^7.5.22 rather than the ^6 that installs by default: 6.x
is flagged critical, and reading the advisory list is what the file's header
now says out loud -- almost all of it is hardlink or symlink traversal and
PAX header interpretation differentials, which is exactly this feature's
threat model.
Two things the plan had wrong
-----------------------------
The bundle's top-level directory is `module-uo-<version>`, not the module id
-- so "the top-level name must equal the id" was checked against nothing real.
The extractor strips that level instead, because its name belongs to whoever
published the bundle and the directory it lands in is core's. What is checked
instead is the unpacked module.json: a manifest promising `uo` and delivering
something else is refused rather than installed under the name it promised.
And purge cannot be a follow-up action (decision 5): purge.sql lives inside
the directory uninstall deletes. It is offered in the uninstall flow and as a
standalone action on a still-installed module, and the standalone one refuses
unless the module is already disabled -- dropping tables under something that
is still serving leaves it answering out of a world that no longer exists.
Disable now means stopped
-------------------------
lifecycle.stop() dispatches that one module's onShutdown before flipping the
guard, so a module an operator switches off actually releases its sockets and
closes its streams instead of merely becoming unreachable. The hook runs
first and the state moves after it, because while onShutdown runs the module
is still `started` and that is the only state in which its routes and the
world it is tearing down agree. A hook that throws does not stop the disable
-- the opposite of the boot path's rule, and deliberately.
Enable is not its mirror and there is no start(id) beside it. There is no
onBoot re-dispatch and the hooks were never promised re-entrant, so enable
moves the row and the restart route starts it. A test pins that enable does
not touch the loader, because "fixing" it is a one-line change that would put
a module with closed sockets back on the nav.
Restart raises SIGTERM against its own process rather than calling the
shutdown path directly, so server.js's handler stays the one graceful-shutdown
path and this route cannot drift from it.
The allowlist bootstraps from MODULE_SOURCE_HOSTS into a settings row and is
admin-managed after that (decision 6); seedDefault is INSERT IGNORE, so
changing the variable on an existing deployment is a no-op by design. An empty
list forbids every install rather than allowing every host -- the safe
direction for a value someone might blank by accident.
Verified against the real v0.3.0 release
----------------------------------------
Not a fixture: fetched the published install manifest over the real Gitea
host and its redirect chain, verified the sha256, inspected and unpacked the
252,517-byte artifact to 82 files, and then booted core against the result --
the module registered its five mounts, seven streams and eight capabilities
and resolved its client chunk, with no scratch directory left behind.
Two defects this slice's own tooling caught, both of which had already been
written down as classes:
- the controller destructured runPurge at require time, capturing the
function rather than the module, which made the one dependency whose
ORDER matters the one that could not be substituted;
- two swagger annotations carried an apostrophe inside a quoted string,
dropped silently by swagger-autogen before slice 5 taught it to fail loudly.
Co-Authored-By: Claude <noreply@anthropic.com>
|
|||
| adff20be7b |
feat(modules): merge module OpenAPI fragments into /api/docs.json (phase 3, slice 5)
Core's half of the slice that closes phase 3. Two things: the request-time
fragment merge core has owed since phase 1, and the last of core's UO copy.
**The merge (MODULE_API.md §6.1a).** `swagger-output.json` is core's own routes
and cannot be anything else — it is generated on a developer's machine and
committed, so it must come out the same regardless of what they had checked out,
and a module arrives on a volume long after the image was built. Module routes
therefore reach the document at request time, from the `swagger-fragment.json`
each module ships: `swagger/docsSpec.js` merges the fragments of STARTED modules
over the committed spec, cached on a new loader state version and rebuilt when a
module's state moves.
Until now neither half existed. `swagger/mergeSpec.js` named the request-time
caller in its header and that caller was never written, so the 72 routes
module-uo serves were in no OpenAPI spec at all — core's standing rule ("never
ship a route that isn't in the spec") broken by the extraction rather than by a
route.
Core wins every key collision, `swagger-output.json` is never mutated (it is a
require()d JSON module — one in-place merge would be permanent AND cumulative),
and a fragment that is missing or unreadable costs that module its paths and
nothing else. The Swagger UI is now built per request for the same reason the
JSON is: bound once at require time it would show core's routes for the life of
the process while /api/docs.json showed the merged set.
**The last of core's UO copy** (slice 4 deferred it; §5.2's check reads code, not
prose, so none of this was caught):
- 31 UO schemas and 4 UO tags in `swagger/swagger.js`, describing routes core has
not served since slice 1 — 578 lines. They moved to module-uo, namespaced
`Uo…`, and arrive back through the merge on an instance that installs it.
- `info.description` said "a private Ultima Online shard".
- README.md's 48 UO mentions, including the architecture diagram and the whole
`## Shard integration (uo-link)` section, now `## Modules`.
- `TOWNCRIER_DURATION_SEC` and `UOLINK_*` in the two `.env.example`s: read by the
module, not by core, and documented in the module's README instead.
**Two dropped annotations, and the reason nobody knew.** swagger-autogen reports
an annotation it cannot parse and then prints Success in green, having skipped
it. `npm run swagger` now captures its diagnostics and fails — which immediately
found `POST /api/v1/admin/invites` and `POST /api/v1/auth/invite/:token/accept`
documented with an EMPTY request body, both since the day they were written.
Fixing the tag list also cleared five tags used by routes but never declared
(`Admin · Email`, `Admin · Invites`, `Admin · Moderation`, `Admin · Pages`,
`Auth · Me`) — the same defect class, in the other direction.
- 646 server tests (+9), 157 client tests unchanged
- routes.manifest.json unchanged (158 public + 2 internal); check:modules clean
- swagger-output.json: 128 paths, 69 schemas, 0 orphan tags, 0 orphan schemas
- verified against a real boot with module-uo installed: 197 merged paths
(128 core + 69 module), all four module tags, 31 Uo schemas, no dangling $refs,
/api/docs renders the module's operations with zero console errors
Refs: docs/website/MODULE_API.md §2.8, §6.1a; MODULE_SYSTEM.md §2.7.1
Co-Authored-By: Claude <noreply@anthropic.com>
|
|||
| 5bdb6a7e10 |
fix(modules): guard the portal's nav icon, resolve MODULES_DIR absolutely
Both found by the §7.7 browser smoke, running the slice-3 pair together, and neither is visible to any test in either repo. `PlayerPortalLayout` rendered `<n.icon />` unguarded while `AdminLayout` guarded its equivalent. `icon` is optional in the nav contract, and every core row in that sidebar has always had one — so the difference cost nothing until a module registered a row without, and then it was not a missing glyph, it was React error #130 and a blank player portal. Guarded now, like its neighbour. `MODULES_DIR` is resolved absolute. `resolveClient` checks containment by comparing an absolute `path.resolve(dir, entry)` against the module directory, so a RELATIVE `MODULES_DIR` — which is what §7.7's own recipe produces when run from `server/` — failed every module with "client.entry escapes the module directory". A perfectly-placed entry, and a message pointing at the module. Co-Authored-By: Claude <noreply@anthropic.com> |
|||
| f50541f374 |
feat(modules): ctx additions and the post-hook registry (API 1.1.0)
Everything the extraction needed from core that ctx did not already offer. Additions only, so minor. ctx.activity.log, because an admin action a module performs has to land in core's one audit log or the trail has a hole exactly where a module operates the game -- a module keeping its own log would be a second place to look, which in practice means a place nobody looks. Write-only; reading the log is the admin panel's job and it spans every actor. ctx.users.getById, one function for one caller: the admin.users.detail slot router needs the user its prefix names. ctx.site.baseUrl, because a module has to build absolute links and §2.7 forbids it reading core's APP_BASE_URL -- a getter, not a captured string, so it cannot go stale against the env. ctx.middleware.rateLimit is core's makeLimiter, plus accountChangeLimiter handed over whole. The split is deliberate: a module states its own window and cap because it knows what its endpoints cost, and takes the plumbing from core so there is one express-rate-limit in the process and one place a breach is logged. accountChangeLimiter is shared policy -- core's /auth/me and /player/account sit behind the same counter -- so a module's account-change route has to land IN it rather than beside it. marketLimiter was UO policy living in core's file and leaves with the route it guards. registerPostHook is the fourth registry, and the last thing binding core to the module. Core's post controller called newsGump.syncPost directly: core's CMS naming a UO file. It now publishes what it already knows and a subscriber decides what to do with it. Not folded into registerAnnounceLeg, which fires on the same transition, because a leg is a one-shot DELIVERY with retry and classification while a post hook maintains idempotent STATE, runs on delete as well as save, and refreshes silently on an edit. Also fixes a real loader defect the extraction exposed: schema table names were matched against the RAW file, so a fragment whose header says "every CREATE TABLE carries IF NOT EXISTS" was rejected for a prefix violation on a table called `carries`. module-uo's fragment hit exactly that. Both scans now read split statements, which strip comments -- the same class of bug as a boundary check failing on its own documentation. Co-Authored-By: Claude <noreply@anthropic.com> |
|||
| e0927bc255 |
feat(modules): the client registry, window.__rg and the chunk's script injection
Phase 2, PR 7 of docs/website/MODULE_SYSTEM.md 2.7 — the client half's
delivery. A module's prebuilt chunk is served, injected, handed core's React
and its UI kit, and its routes are rendered by App.jsx. The registry is empty
on a bare core, so nothing an operator can see changes.
Client:
- modules/registry.js — registerRoutes/registerNav/registerFeatureProvider,
with the URL namespace written by core, never by the module
- modules/shared.js — window.__rg: React, react-dom/client, react-router-dom,
react/jsx-runtime, the registry, the seven-member UI kit and the request
primitive, frozen
- App.jsx reads routesFor for all three areas; nav consumption is PR 8
- main.jsx publishes the global, then mounts on DOMContentLoaded
Server:
- the loader validates client.entry and publishes clientChunks() and
clientEntryUrls(); an entry in the module root is rejected, because the
directory it sits in is what gets served
- app.js mounts each chunk at /modules/<id>/ behind the module's state guard
with no-cache; anything else under /modules is a 404, not the SPA shell
- htmlShell injects the tag before </body>, so core's bundle runs first
wherever a bundler puts it
Found by loading a real chunk in a browser, and fixed here: core mounted before
any module chunk had evaluated, because document.readyState during a deferred
script is 'interactive', not 'loading'. Every test passed against that build.
The smoke is written down in MODULE_API.md 7.7.
933 server tests (+23), 123 client tests (+14). routes.manifest.json unchanged
at 230 routes; the OpenAPI spec regenerates byte-identical.
Co-Authored-By: Claude <noreply@anthropic.com>
|
|||
| 21196466ed |
feat(modules): boot/shutdown hook dispatch and the installed_modules reconcile
Phase 2, PR 5 of docs/website/MODULE_SYSTEM.md 2.7. api.onBoot/api.onShutdown stop throwing, server.js gains one call on each side, and the 2.4 state machine finally runs against real outcomes -- which is what makes 4.5's `disabled` 404 leg reachable for the first time. Dispatch and reconcile live in src/modules/lifecycle.js rather than in the loader, for the reason the schema replay does: routeManifest.js and swagger.js both require app.js against a dead pool, so the loader may not reach the database. The two halves meet at exactly one function, loader.setState(), so the in-memory record the dispatch guard reads and the row the admin panel reads are moved together and cannot disagree. Four decisions, all recorded in MODULE_API.md 2.5 and 4.4: - The loader classifies its failures by 4.3 step, so failure_stage says where a module broke instead of being a column nothing ever filled. The four steps readManifest covers in one pass label themselves; the rest are inferred from how far load() had got, and an unlabelled throw is recorded against the step that was running rather than guessed at. - A row whose directory is gone is marked startup_failed rather than left claiming `enabled` -- the boot reset has just moved it there, and a row claiming to be enabled for a module that is not on the volume is the one state that is simply untrue. An uninstall leaves `disabled`, which the reset never touches, so this catches only a hand-deleted directory. - Core's eight UO boot call sites stay in server.js until Phase 3. Unlike a registered announce leg, a boot call site already has somewhere to live, so moving it now would be extraction done early in a phase whose exit criterion is that nothing changes. - onBoot gets no timeout. Shutdown races a SIGKILL and boot does not, and a slow onBoot delaying the listener is the contract's promise to a module that must warm up before it serves. The operator's switch wins over everything: a disabled module is guarded, not booted, and does not have its failure re-recorded, or an outcome would silently switch it back on next boot. Every database write in the reconcile is individually caught -- a row that will not update is worse reporting, never a failed boot. 900 tests pass (17 new). routes.manifest.json is unchanged at 229 routes and the OpenAPI spec regenerates byte-identical. Co-Authored-By: Claude <noreply@anthropic.com> |
|||
| 6195c76d61 |
feat(modules): the three de-entanglement registries, with core as the registrant
Phase 2 PR 4 of docs/website/MODULE_SYSTEM.md §2.7. Adds server/src/modules/registries.js and moves core's own notification streams, announce leg and users-detail routes behind it, so the three seams §1.8 and §1.9 named are exercised on every boot before any module depends on them. Registering is validate-then-commit per registrant: the loader stages what a module claims and the second pass commits it, so a module that throws halfway through register() — or fails checkDeclared after it — leaves nothing behind. That is the registry-side twin of PR 2's second-pass mount rule. Four decisions, all the recommended option: - announce legs became a child table. `announce_job_legs` replaces the towncrier_*/discord_* column groups, so the leg set is data: core registers `discord`, module-uo will register `towncrier`, and a module cannot ALTER a core table to add its own. Backfill is guarded on information_schema (a SELECT of a dropped column is a parse error, not a runtime one) and the columns go with DROP COLUMN IF EXISTS. Verified against the live dev DB: three legacy jobs migrated faithfully, three replays, no duplicates. - `mapEvent` dropped from registerNotificationStreams. §1.8 already inverts the push path so a module owns fromShardEvent and calls core's publish() with a stream id it resolved; a second mapping mechanism was a leftover. The public safety filter, the kinds it reads and the streams it protects now live in one file and move together. - core registers through the same staging area a module uses, via an explicit registries.registerCore() in app.js before modules.load(). - core's six /admin/users/:id/shard/* paths now go through the `admin.users.detail` slot, and getUser moved back to admin.controller.js. Found on the way, and the reason two build tools changed: - scripts/routeManifest.js could not decode a parameterised mount. Its unwinder expected `(?:([^\/]+?))`; express 4.22 emits `(?:\/([^/]+?))` with the separator inside the group. The branch had never run. It threw rather than guessing, which is what it is for. - swagger-autogen cannot follow a route into an extension slot — the slot's router is created by declareSlot() and filled later, so there is no literal mount for a static parse. Regenerating deleted 407 lines and printed `Swagger-autogen: Success`, the spike's exact failure (MODULE_API.md §7.4). swagger/slotSpecs.js generates a fragment per filled slot and re-roots it at the prefix the router actually hangs at in the live app — read from the express stack via routeManifest's own mountPath, so the manifest and the spec cannot disagree. swagger/mergeSpec.js is the merge helper core owes for module fragments anyway (§6.1a), proved here against core's own slot first. 884 tests pass (856 before). routes.manifest.json is unchanged at 229 routes. The OpenAPI spec diff is two lines of intent: the retry endpoint's summary, and its `leg` no longer being a fixed enum. Co-Authored-By: Claude <noreply@anthropic.com> |
|||
| 2892d01b24 |
feat(modules): replay module schema fragments after core's
Phase 2, PR 3 of docs/website/MODULE_SYSTEM.md 2.7. ensureSchema() now replays every installed module's schema fragment immediately after core's schema.sql, per MODULE_API.md 2.6. The work splits across two files on the line of whether a database is needed to know the answer. loader.js VALIDATES a fragment at load time, before anything is mounted, because every rule 2.6 states about the SQL is knowable by reading it; a module that breaks one never mounts (4.4, left column). modules/schema.js EXECUTES it, so the only failures there are the ones the database alone could report, and those are post-mount and answer 503 (4.4, right column). Validation is a leading-verb allowlist -- CREATE, ALTER, INSERT, UPDATE, the four core's own schema.sql uses -- rather than the DROP denylist 2.6 words it as. A fragment is replayed on every boot, so TRUNCATE and DELETE would empty a table at each restart and RENAME would fail at the second one; a denylist only ever bans what somebody thought of. A CREATE TABLE missing IF NOT EXISTS is rejected for the same reason: it works once and fails every boot after, which presents to an operator as a module that broke on restart. The splitter moves to utils/sqlStatements.js so core's schema and a fragment are split by literally the same code, which is what 2.6 promises. It is its own file rather than an export of utils/db.js because the loader validates fragments at require time and must not drag the mariadb pool into app.js's require chain. The replay sits outside ensureSchema's wait-for-the-database retry loop: a fragment that throws is one module's failure, not a signal the database is coming up, and retrying core's whole schema nine more times over one module's bad SQL would turn a 503'd module into a two-minute boot. Found while wiring it: db/seed.js calls ensureSchema() standalone for `npm run seed`, without ever requiring app.js, so the loader has not scanned and fragments()'s 7.6 throw would have broken seeding outright. The replay asks isLoaded() and logs the skip rather than swallowing it -- a booting server quietly getting no module tables is the thing 7.6 exists to prevent. Verification: - 856 server tests pass, 14 new. moduleSchema.test.js injects the query fn, so the exact statements and their order are asserted with the pool at a dead port like every other suite. - routes.manifest.json and routes.guards.json diffs are zero lines, 229 routes -- the phase 2 exit criterion. swagger-output.json regenerates byte-identical. - Run for real against the local MariaDB with two fixture modules: a good fragment created its table, applied its ALTER and seeded its row; a fragment whose SQL passes validation but the server rejects (`id NOTATYPE`) marked only that module startup_failed, its route answering 503 while the other answered 200; a second ensureSchema on the same database was a clean no-op. Co-Authored-By: Claude <noreply@anthropic.com> |
|||
| ec1ca7e794 |
feat(modules): the filesystem module loader
Phase 2 PR 2 of docs/website/MODULE_SYSTEM.md 2.7. Adds
server/src/modules/{loader,semver,version}.js: the synchronous scan of
MODULES_DIR, manifest validation, prefix and table-name collision
rejection, per-module try/catch and the mount into the three tier
routers behind the MODULE_API.md 4.5 dispatch guard.
Two decisions the contract left open, both now written up there:
- The load trigger is one explicit modules.load(tierRouters) call in
app.js, not a lazy scan (API 7.6). Accessors throw until it has run,
because "no modules installed" is a real answer a caller must not be
handed by accident.
- Whether core owns a prefix is asked of the live tier routers via
express's own layer.match(), skipping root-mounted layers, rather than
a hardcoded table -- the spike's was already stale when written
(API 4.3).
Mounting is a second pass after every module is validated. Doing it
inside the scan loop makes the first module's layers indistinguishable
from core's, so the second module claiming a taken prefix is told it
collided with core and the module-versus-module check is unreachable.
registerExtension/NotificationStreams/AnnounceLeg and onBoot/onShutdown
throw "not available until phase 2 PR 4/5" rather than no-op; an
accepting stub would let a module believe it had registered something.
No schema replay, no boot dispatch, no installed_modules reconcile --
those are PRs 3 and 5, and until PR 5 a record's state is in memory only.
No module ships on the volume, so nothing an operator or client can see
changes: 842 tests pass, routes.manifest.json is unchanged at 229 routes
and swagger-output.json regenerates byte-identical.
Co-Authored-By: Claude <noreply@anthropic.com>
|
