diff --git a/website/BACKEND_DESIGN.md b/website/BACKEND_DESIGN.md index 7e59a7d..f9d341f 100644 --- a/website/BACKEND_DESIGN.md +++ b/website/BACKEND_DESIGN.md @@ -55,7 +55,8 @@ server/ schema.sql (+) DDL, also auto-run by the MariaDB container seed.js (+) seed wiki pages, default settings, first admin src/ - server.js bootstrap: ensure schema, then listen on 0.0.0.0 + server.js bootstrap: core schema, seed, resolve MODULES, require app, module + schema fragments, module onBoot, then listen on 0.0.0.0 app.js express app + middleware wiring router/ api.router.js mounts /v1 @@ -285,6 +286,13 @@ fresh install and changing the variable later cannot reach back in and overwrite an operator chose. Installs are `https`-only, every redirect hop is re-checked against this list, and an empty value forbids every install rather than allowing every host. +The **`MODULES`** environment variable (MODULE_SYSTEM.md §2.7.2 decision 4) installs through the same +allowlist and the same verification, without a request: each `@=` entry is +resolved onto the modules volume during boot, between `seedDefaults()` and the `require` of `app.js` +that scans it. It is not a settings row and is not editable from the panel — a deployment declares +what it runs, the panel shows that it did, and neither owns the other: the variable decides what is +on the volume and `installed_modules.state` decides whether a module answers. + **Keys a MODULE seeds into this table.** `settings` is core's, but a module's schema fragment may `INSERT IGNORE` its own rows into it, and module-uo seeds two: `game_account_signup` (default `disabled`) and the one-shot migration marker diff --git a/website/MODULE_API.md b/website/MODULE_API.md index 95451bd..a5d0885 100644 --- a/website/MODULE_API.md +++ b/website/MODULE_API.md @@ -296,8 +296,10 @@ naming a UO file, and the last thing binding core to the module. ### 2.5 Lifecycle ``` +(core schema + seed) → MODULES resolved onto the volume ← §2.7.2 decision 4 + ↓ require(module) → register(ctx, api) → [routes mounted, app.js require returns] - ↓ (server.js, after ensureSchema + seed) + ↓ (server.js: schema fragments, then) onBoot(ctx) → started ↓ (SIGINT/SIGTERM) onShutdown() @@ -365,8 +367,8 @@ the dispatch guard reads and the row the admin panel reads are moved together an ### 2.6 Schema fragments -`schema` is an idempotent `.sql` file replayed by the same `ensureSchema()` that replays core's, -immediately after it, statement by statement, split the same way. It is subject to the same rules +`schema` is an idempotent `.sql` file replayed immediately after core's own `schema.sql`, statement +by statement, split the same way. It is subject to the same rules core's file already follows: `CREATE TABLE IF NOT EXISTS`, `ALTER TABLE … ADD COLUMN IF NOT EXISTS`, no `--` inside a string literal, no `DROP`. @@ -404,6 +406,18 @@ break seeding outright. The replay asks `isLoaded()` and logs the skip. That is use of that predicate: everywhere else, reading the module list before `load()` still throws, because a booting server quietly getting no module tables is precisely what §7.6 exists to prevent. +**And the server replays them itself, because it now scans the volume LATER than it ensures the +schema.** `ensureSchema()` carries the replay for every ordinary caller, but `server.js` passes +`replayModules: false` and calls `replayFragments()` of its own accord after requiring `app.js`. The +reason is `MODULES` (`MODULE_SYSTEM.md` §2.7.2 decision 4): resolving a declared module set has to +happen before the scan, it needs the host-allowlist setting to do it, and that setting does not exist +until the schema and the seed have run — so core's schema now precedes the scan, and a replay wired +to core's schema would run when there was nothing yet to replay. Nothing about the contract moves: +fragments still run after core's tables exist and before any `onBoot`. **This was a live defect for +the length of one afternoon**, and the shape of it is worth keeping: it announced itself only as the +skip line above, appearing in a *booting server's* log where it means the opposite of what it means +in `npm run seed`, and on a database whose tables already existed the module started perfectly. + **Table names are namespaced and collision-checked.** New tables must be prefixed `_`. The loader extracts every `CREATE TABLE IF NOT EXISTS ` from the fragment and rejects the module if a name collides with a core table or with another module's — a wrong `DROP`-free fragment can diff --git a/website/MODULE_SYSTEM.md b/website/MODULE_SYSTEM.md index b851e63..e1cd671 100644 --- a/website/MODULE_SYSTEM.md +++ b/website/MODULE_SYSTEM.md @@ -473,6 +473,15 @@ first.)* resolved at container start from the mounted volume, so a compose-managed host is not driven by clicking. Both paths write the same `installed_modules` row and neither requires a build step. +*(Built in Phase 4 slice 3. The variable is `MODULES`, each entry `@=`; resolution runs inside the server process, before the volume is scanned, and a module already +unpacked at the declared version is a no-op that makes no network call at all. "Both paths write the +same row" held exactly — including its provenance columns, which is why resolution runs where it can +reach the database. What §2.5 did not anticipate is that the two surfaces need a rule about who wins: +the declaration owns what is **on the volume**, the row owns whether a module **runs**. Uninstalling +a declared module from the panel therefore returns its files at the next start and leaves it +disabled.)* + ### 2.6 How the client half loads This is the piece §1.14 constrains hardest. Three requirements had to hold at once: the operator @@ -1470,6 +1479,13 @@ restart with the network down brings the site up exactly as it was. Only a missi wrong-version module reaches out, and it reaches out through the same fetch-verify-unpack path the admin panel uses. +*(Built in slice 3, and one phrase in this paragraph turned out to be the wrong shape. "Resolution +runs before the server starts" is true of the SCAN, not of the process: it runs inside `start()`, +before `app.js` is required, because that is what buys it the database — the host allowlist is a +settings row, and provenance has to be written where the admin route writes it. A pre-flight script +would have had neither. Also settled there: the entry syntax is `@=`, +with the version written out precisely so the offline no-op is a file read rather than a fetch.)* + **5. Purge is offered inside the uninstall flow, because it cannot be offered after it.** Found while writing slice 1 against §2.5, which promised a `disabled` row an operator could purge later. `purge.sql` is a file *inside the module directory*, declared by `module.json` — and uninstall @@ -1499,7 +1515,7 @@ core-only, so the two-repo merge dance does not apply — `module-uo` is touched | 0 | **SonarQube for `module-uo`** — `sonar-project.properties` (project key `Module-uo`) plus the scan workflow, mirroring `website`'s. The module is 40 server + 35 client files of extracted code that has never been scanned. | `Module-uo`, `docs` | | 1 | **The install service and the admin API** — fetch, verify, hardened unpack, remove, purge; `/api/v1/admin/modules` with list, install, enable, disable, uninstall, purge and restart; the per-module `onShutdown` dispatch decision 3 requires; the host allowlist. | `website`, `docs` | | 2 | **The Modules screen** — `ModulesAdmin.jsx`, its nav row, its API bindings, and the §7.7 browser smoke against a real install of `module-uo` from a real release URL. | `website`, `docs` | -| 3 | **The declarative Docker path** — container-start resolution of decision 4's variable, the compose and image changes it needs, and the operator documentation. | `website`, `docs` | +| 3 | **The declarative Docker path** — container-start resolution of decision 4's variable, the compose and image changes it needs, and the operator documentation. The variable is `MODULES`; resolution runs in the server process, before the volume is scanned. | `website`, `docs` | | 4 | **Close the phase** — `BACKEND_DESIGN.md` §3, the `api-route-inventory.json` mirror, `docs/modules/uo/`'s install instructions, and the acceptance table. | `docs`, `website` | ##### The unpack is the dangerous part, and it is not the download @@ -1675,6 +1691,99 @@ uo-link WebSocket closed, its routes answered 404 and it left `/api/v1/public/mo produced the decision-3 state with the banner. That is acceptance criteria 1 and most of 2, through the screen, with no shell. +#### Slice 3 — the declarative Docker path (website#144, 2026-08-12) + +`server/src/modules/declared.js`, the boot-order change it forced in `server.js`, a fourth source on +the admin screen, and the operator documentation in three files. 741 server tests (+18) and 187 +client (+5); manifest unchanged at 166 public + 2 internal, OpenAPI byte-identical. + +An operator declares the set their deployment runs, in the environment, and the container arrives at +it by itself: + +``` +MODULES=uo@0.3.0=https://gitea.whitlocktech.com/RunicGateway/Module-uo/releases/download/v0.3.0/module-uo-0.3.0.json +``` + +##### Why the id and the version are written out + +The alternative — declaring only the URLs and reading the id and version out of each manifest — is +shorter and cannot answer the question this feature exists to answer offline. **The no-op case must +need no network**, because that is what "a restart with the network down brings the site up exactly +as it was" means, and deciding that a module is already at the declared version by fetching its +manifest is not that. With the version in the variable the check is a `module.json` read on the +volume, and it holds no matter what state the database is in. Only a missing or different version +reaches out, through `modules/install.js` — the same allowlist, the same sha256, the same +inspect-then-extract — and `install()` grew an `expect: {id, version}` so a URL that resolves to +another module or another version is refused **while it is still only a manifest**, before its +artifact is downloaded rather than after it is unpacked. + +##### Four decisions, all as recommended + +**Resolution runs in the server process, not in a script before it.** It sits in `start()` between +the seed and the require of `app.js`, which buys the database: the host allowlist it installs under +is the same admin-managed settings row the panel uses, and a module it installs gets its `source` and +`sha256` written exactly as the admin route writes them. A compose-installed module and a +panel-installed one are then the same thing on the screen rather than two features that look alike. +It also means a bare `npm start` with `MODULES` set behaves identically; Docker is the reason it +exists, not a special case inside it. + +**A failure is loud and not fatal.** An unreachable release host leaves the site serving without that +module, or on the version already unpacked — never in a restart loop, which is what refusing to boot +would mean on a compose host with `restart: unless-stopped`. Core is built to serve with a module +absent (§1.6); a module publisher's outage should not be able to take a shard's website down. + +**The declaration owns the volume; the row owns whether a module runs.** An admin who uninstalls a +declared module gets the files back at the next start and the module stays `disabled` until they +enable it, because `upsert` never touches `state`. No tombstone, no precedence rule, and nothing for +the two to fight about — they are answering different questions. The screen says so, which matters +more than it sounds: files reappearing unexplained is an afternoon of debugging. + +**And so the admin screen has a fourth source of truth.** §2.4's row / loader / volume become row / +loader / volume / declaration, and it is the only one of the four no button on that screen can +change. A declared module that failed to resolve has *none* of the other three — no row, no +directory, nothing mounted — so without listing it the screen would be identical to one where nobody +had asked for it. The declaration also gets its own line rather than being folded into the status +label, because a module can be running perfectly while its declared upgrade is failing and a single +label would have to discard one of those two facts. + +##### The defect this slice created, and the log line that gave it away + +Deferring the `require('./app')` is what makes any of this possible — the loader scans and mounts at +require time (§1.12), so that require is the last moment a module can be put on the volume and still +be part of the process. It also moved core's schema *before* the scan, and the module schema-fragment +replay was wired to core's schema. **Every installed module silently got no tables.** + +It survived the entire test suite, because every suite here stubs either the loader or the pool. It +survived the first browser smoke too: this machine's development database already had the 27 `shard_*` +tables from earlier phases, so the module started perfectly. The only trace was one line — +`no module scan in this process — skipping schema fragment replay` — which is *correct* output for +`npm run seed` (API §2.6) and means the opposite in a booting server. It was caught by booting +against a brand-new empty database, which is the only place the defect is visible at all. + +`ensureSchema()` now takes `replayModules: false` for the one caller that intends to scan later, and +`server.js` replays the fragments itself immediately after the require. There is a `bootOrder` test +that reads `server.js` and asserts the five steps are in the one order they can be in — a structural +test, which proves only that nobody has moved a line, and is worth having precisely because that is +the failure mode: nothing else in the repo can see this one. + +##### What the smoke proved, and what it could not + +Against the real published `module-uo` v0.3.0, on a fresh database, with the real Gitea release: + +| Claim | Result | +| --- | --- | +| A declared module installs and mounts in the **same boot** | 252,517 bytes fetched, sha256 verified, unpacked, 37 schema statements, `started`, `/api/v1/public/shard/status` → 200 | +| A restart at the declared version touches no network | `already at the declared version 0.3.0`, zero requests, module up | +| Two unresolvable declarations do not stop the site | Both 404s logged as errors; site listening; `uo` still serving 0.3.0 | +| Uninstall then restart returns the files and not the module | Directory back with provenance recorded, row still `disabled`, `/api/v1/public/modules` empty, shard routes 404 | + +The admin screen's new line was verified through the API rather than the browser — the extension's +viewport came back 0×0 and screenshots failed at the CDP level for the whole session. The live +`GET /admin/modules` carries `declared`, `declaredVersion` and `declaredError` in the two shapes that +matter (a running module whose declared upgrade failed, and a declared module with nothing on the +volume); the rendering above that is one derived line, covered by the client suite. It is the one +thing in this slice not proved in a browser. + ### 2.8 SPA URL namespacing — a deliberate break **Decision: module pages are namespaced, and old paths are not redirected.** The site is not public