From 102ea6db193bd8dff24d6f85a9f432e764eeeb4c Mon Sep 17 00:00:00 2001 From: wtclaude Date: Wed, 12 Aug 2026 07:57:21 -0500 Subject: [PATCH] docs(website): record slice 3, and the boot-order defect it created MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MODULE_SYSTEM.md §2.7.2 gains the slice 3 write-up: why the id and version are written out in MODULES (the offline no-op has to be a file read, not a fetch), the four decisions, and the table of what the smoke proved against the real v0.3.0 release on a fresh database. Two sections are corrected rather than extended. §2.5's "declarative module set" promise did not anticipate that the two install surfaces need a rule about who wins — the declaration owns the volume, the row owns whether a module runs — and decision 4's "resolution runs before the server starts" is true of the SCAN, not of the process: it runs inside start(), which is what buys it the database. MODULE_API.md §2.6 no longer says the fragments are replayed by ensureSchema(). They are, for every caller except the server, which scans the volume later than it ensures the schema and so replays them itself. That was a live defect for an afternoon: it announced itself only as the "no module scan in this process" skip line, which is correct output for `npm run seed` and means the opposite in a booting server, and on a database that already had the tables the module started perfectly. Nothing in the contract moves, so MODULE_API_VERSION is unchanged. BACKEND_DESIGN.md: the boot sequence in the tree, and MODULES beside the module_source_hosts setting it installs through. Co-Authored-By: Claude --- website/BACKEND_DESIGN.md | 10 +++- website/MODULE_API.md | 20 +++++-- website/MODULE_SYSTEM.md | 111 +++++++++++++++++++++++++++++++++++++- 3 files changed, 136 insertions(+), 5 deletions(-) diff --git a/website/BACKEND_DESIGN.md b/website/BACKEND_DESIGN.md index 7e59a7d..f9d341f 100644 --- a/website/BACKEND_DESIGN.md +++ b/website/BACKEND_DESIGN.md @@ -55,7 +55,8 @@ server/ schema.sql (+) DDL, also auto-run by the MariaDB container seed.js (+) seed wiki pages, default settings, first admin src/ - server.js bootstrap: ensure schema, then listen on 0.0.0.0 + server.js bootstrap: core schema, seed, resolve MODULES, require app, module + schema fragments, module onBoot, then listen on 0.0.0.0 app.js express app + middleware wiring router/ api.router.js mounts /v1 @@ -285,6 +286,13 @@ fresh install and changing the variable later cannot reach back in and overwrite an operator chose. Installs are `https`-only, every redirect hop is re-checked against this list, and an empty value forbids every install rather than allowing every host. +The **`MODULES`** environment variable (MODULE_SYSTEM.md §2.7.2 decision 4) installs through the same +allowlist and the same verification, without a request: each `@=` entry is +resolved onto the modules volume during boot, between `seedDefaults()` and the `require` of `app.js` +that scans it. It is not a settings row and is not editable from the panel — a deployment declares +what it runs, the panel shows that it did, and neither owns the other: the variable decides what is +on the volume and `installed_modules.state` decides whether a module answers. + **Keys a MODULE seeds into this table.** `settings` is core's, but a module's schema fragment may `INSERT IGNORE` its own rows into it, and module-uo seeds two: `game_account_signup` (default `disabled`) and the one-shot migration marker diff --git a/website/MODULE_API.md b/website/MODULE_API.md index 95451bd..a5d0885 100644 --- a/website/MODULE_API.md +++ b/website/MODULE_API.md @@ -296,8 +296,10 @@ naming a UO file, and the last thing binding core to the module. ### 2.5 Lifecycle ``` +(core schema + seed) → MODULES resolved onto the volume ← §2.7.2 decision 4 + ↓ require(module) → register(ctx, api) → [routes mounted, app.js require returns] - ↓ (server.js, after ensureSchema + seed) + ↓ (server.js: schema fragments, then) onBoot(ctx) → started ↓ (SIGINT/SIGTERM) onShutdown() @@ -365,8 +367,8 @@ the dispatch guard reads and the row the admin panel reads are moved together an ### 2.6 Schema fragments -`schema` is an idempotent `.sql` file replayed by the same `ensureSchema()` that replays core's, -immediately after it, statement by statement, split the same way. It is subject to the same rules +`schema` is an idempotent `.sql` file replayed immediately after core's own `schema.sql`, statement +by statement, split the same way. It is subject to the same rules core's file already follows: `CREATE TABLE IF NOT EXISTS`, `ALTER TABLE … ADD COLUMN IF NOT EXISTS`, no `--` inside a string literal, no `DROP`. @@ -404,6 +406,18 @@ break seeding outright. The replay asks `isLoaded()` and logs the skip. That is use of that predicate: everywhere else, reading the module list before `load()` still throws, because a booting server quietly getting no module tables is precisely what §7.6 exists to prevent. +**And the server replays them itself, because it now scans the volume LATER than it ensures the +schema.** `ensureSchema()` carries the replay for every ordinary caller, but `server.js` passes +`replayModules: false` and calls `replayFragments()` of its own accord after requiring `app.js`. The +reason is `MODULES` (`MODULE_SYSTEM.md` §2.7.2 decision 4): resolving a declared module set has to +happen before the scan, it needs the host-allowlist setting to do it, and that setting does not exist +until the schema and the seed have run — so core's schema now precedes the scan, and a replay wired +to core's schema would run when there was nothing yet to replay. Nothing about the contract moves: +fragments still run after core's tables exist and before any `onBoot`. **This was a live defect for +the length of one afternoon**, and the shape of it is worth keeping: it announced itself only as the +skip line above, appearing in a *booting server's* log where it means the opposite of what it means +in `npm run seed`, and on a database whose tables already existed the module started perfectly. + **Table names are namespaced and collision-checked.** New tables must be prefixed `_`. The loader extracts every `CREATE TABLE IF NOT EXISTS ` from the fragment and rejects the module if a name collides with a core table or with another module's — a wrong `DROP`-free fragment can diff --git a/website/MODULE_SYSTEM.md b/website/MODULE_SYSTEM.md index b851e63..e1cd671 100644 --- a/website/MODULE_SYSTEM.md +++ b/website/MODULE_SYSTEM.md @@ -473,6 +473,15 @@ first.)* resolved at container start from the mounted volume, so a compose-managed host is not driven by clicking. Both paths write the same `installed_modules` row and neither requires a build step. +*(Built in Phase 4 slice 3. The variable is `MODULES`, each entry `@=`; resolution runs inside the server process, before the volume is scanned, and a module already +unpacked at the declared version is a no-op that makes no network call at all. "Both paths write the +same row" held exactly — including its provenance columns, which is why resolution runs where it can +reach the database. What §2.5 did not anticipate is that the two surfaces need a rule about who wins: +the declaration owns what is **on the volume**, the row owns whether a module **runs**. Uninstalling +a declared module from the panel therefore returns its files at the next start and leaves it +disabled.)* + ### 2.6 How the client half loads This is the piece §1.14 constrains hardest. Three requirements had to hold at once: the operator @@ -1470,6 +1479,13 @@ restart with the network down brings the site up exactly as it was. Only a missi wrong-version module reaches out, and it reaches out through the same fetch-verify-unpack path the admin panel uses. +*(Built in slice 3, and one phrase in this paragraph turned out to be the wrong shape. "Resolution +runs before the server starts" is true of the SCAN, not of the process: it runs inside `start()`, +before `app.js` is required, because that is what buys it the database — the host allowlist is a +settings row, and provenance has to be written where the admin route writes it. A pre-flight script +would have had neither. Also settled there: the entry syntax is `@=`, +with the version written out precisely so the offline no-op is a file read rather than a fetch.)* + **5. Purge is offered inside the uninstall flow, because it cannot be offered after it.** Found while writing slice 1 against §2.5, which promised a `disabled` row an operator could purge later. `purge.sql` is a file *inside the module directory*, declared by `module.json` — and uninstall @@ -1499,7 +1515,7 @@ core-only, so the two-repo merge dance does not apply — `module-uo` is touched | 0 | **SonarQube for `module-uo`** — `sonar-project.properties` (project key `Module-uo`) plus the scan workflow, mirroring `website`'s. The module is 40 server + 35 client files of extracted code that has never been scanned. | `Module-uo`, `docs` | | 1 | **The install service and the admin API** — fetch, verify, hardened unpack, remove, purge; `/api/v1/admin/modules` with list, install, enable, disable, uninstall, purge and restart; the per-module `onShutdown` dispatch decision 3 requires; the host allowlist. | `website`, `docs` | | 2 | **The Modules screen** — `ModulesAdmin.jsx`, its nav row, its API bindings, and the §7.7 browser smoke against a real install of `module-uo` from a real release URL. | `website`, `docs` | -| 3 | **The declarative Docker path** — container-start resolution of decision 4's variable, the compose and image changes it needs, and the operator documentation. | `website`, `docs` | +| 3 | **The declarative Docker path** — container-start resolution of decision 4's variable, the compose and image changes it needs, and the operator documentation. The variable is `MODULES`; resolution runs in the server process, before the volume is scanned. | `website`, `docs` | | 4 | **Close the phase** — `BACKEND_DESIGN.md` §3, the `api-route-inventory.json` mirror, `docs/modules/uo/`'s install instructions, and the acceptance table. | `docs`, `website` | ##### The unpack is the dangerous part, and it is not the download @@ -1675,6 +1691,99 @@ uo-link WebSocket closed, its routes answered 404 and it left `/api/v1/public/mo produced the decision-3 state with the banner. That is acceptance criteria 1 and most of 2, through the screen, with no shell. +#### Slice 3 — the declarative Docker path (website#144, 2026-08-12) + +`server/src/modules/declared.js`, the boot-order change it forced in `server.js`, a fourth source on +the admin screen, and the operator documentation in three files. 741 server tests (+18) and 187 +client (+5); manifest unchanged at 166 public + 2 internal, OpenAPI byte-identical. + +An operator declares the set their deployment runs, in the environment, and the container arrives at +it by itself: + +``` +MODULES=uo@0.3.0=https://gitea.whitlocktech.com/RunicGateway/Module-uo/releases/download/v0.3.0/module-uo-0.3.0.json +``` + +##### Why the id and the version are written out + +The alternative — declaring only the URLs and reading the id and version out of each manifest — is +shorter and cannot answer the question this feature exists to answer offline. **The no-op case must +need no network**, because that is what "a restart with the network down brings the site up exactly +as it was" means, and deciding that a module is already at the declared version by fetching its +manifest is not that. With the version in the variable the check is a `module.json` read on the +volume, and it holds no matter what state the database is in. Only a missing or different version +reaches out, through `modules/install.js` — the same allowlist, the same sha256, the same +inspect-then-extract — and `install()` grew an `expect: {id, version}` so a URL that resolves to +another module or another version is refused **while it is still only a manifest**, before its +artifact is downloaded rather than after it is unpacked. + +##### Four decisions, all as recommended + +**Resolution runs in the server process, not in a script before it.** It sits in `start()` between +the seed and the require of `app.js`, which buys the database: the host allowlist it installs under +is the same admin-managed settings row the panel uses, and a module it installs gets its `source` and +`sha256` written exactly as the admin route writes them. A compose-installed module and a +panel-installed one are then the same thing on the screen rather than two features that look alike. +It also means a bare `npm start` with `MODULES` set behaves identically; Docker is the reason it +exists, not a special case inside it. + +**A failure is loud and not fatal.** An unreachable release host leaves the site serving without that +module, or on the version already unpacked — never in a restart loop, which is what refusing to boot +would mean on a compose host with `restart: unless-stopped`. Core is built to serve with a module +absent (§1.6); a module publisher's outage should not be able to take a shard's website down. + +**The declaration owns the volume; the row owns whether a module runs.** An admin who uninstalls a +declared module gets the files back at the next start and the module stays `disabled` until they +enable it, because `upsert` never touches `state`. No tombstone, no precedence rule, and nothing for +the two to fight about — they are answering different questions. The screen says so, which matters +more than it sounds: files reappearing unexplained is an afternoon of debugging. + +**And so the admin screen has a fourth source of truth.** §2.4's row / loader / volume become row / +loader / volume / declaration, and it is the only one of the four no button on that screen can +change. A declared module that failed to resolve has *none* of the other three — no row, no +directory, nothing mounted — so without listing it the screen would be identical to one where nobody +had asked for it. The declaration also gets its own line rather than being folded into the status +label, because a module can be running perfectly while its declared upgrade is failing and a single +label would have to discard one of those two facts. + +##### The defect this slice created, and the log line that gave it away + +Deferring the `require('./app')` is what makes any of this possible — the loader scans and mounts at +require time (§1.12), so that require is the last moment a module can be put on the volume and still +be part of the process. It also moved core's schema *before* the scan, and the module schema-fragment +replay was wired to core's schema. **Every installed module silently got no tables.** + +It survived the entire test suite, because every suite here stubs either the loader or the pool. It +survived the first browser smoke too: this machine's development database already had the 27 `shard_*` +tables from earlier phases, so the module started perfectly. The only trace was one line — +`no module scan in this process — skipping schema fragment replay` — which is *correct* output for +`npm run seed` (API §2.6) and means the opposite in a booting server. It was caught by booting +against a brand-new empty database, which is the only place the defect is visible at all. + +`ensureSchema()` now takes `replayModules: false` for the one caller that intends to scan later, and +`server.js` replays the fragments itself immediately after the require. There is a `bootOrder` test +that reads `server.js` and asserts the five steps are in the one order they can be in — a structural +test, which proves only that nobody has moved a line, and is worth having precisely because that is +the failure mode: nothing else in the repo can see this one. + +##### What the smoke proved, and what it could not + +Against the real published `module-uo` v0.3.0, on a fresh database, with the real Gitea release: + +| Claim | Result | +| --- | --- | +| A declared module installs and mounts in the **same boot** | 252,517 bytes fetched, sha256 verified, unpacked, 37 schema statements, `started`, `/api/v1/public/shard/status` → 200 | +| A restart at the declared version touches no network | `already at the declared version 0.3.0`, zero requests, module up | +| Two unresolvable declarations do not stop the site | Both 404s logged as errors; site listening; `uo` still serving 0.3.0 | +| Uninstall then restart returns the files and not the module | Directory back with provenance recorded, row still `disabled`, `/api/v1/public/modules` empty, shard routes 404 | + +The admin screen's new line was verified through the API rather than the browser — the extension's +viewport came back 0×0 and screenshots failed at the CDP level for the whole session. The live +`GET /admin/modules` carries `declared`, `declaredVersion` and `declaredError` in the two shapes that +matter (a running module whose declared upgrade failed, and a declared module with nothing on the +volume); the rendering above that is one derived line, covered by the client suite. It is the one +thing in this slice not proved in a browser. + ### 2.8 SPA URL namespacing — a deliberate break **Decision: module pages are namespaced, and old paths are not redirected.** The site is not public