feat(modules): replay module schema fragments after core's (phase 2, PR 3) #130

wtclaude · 2026-08-10T21:59:14Z

wtclaude commented

2026-08-10 21:59:14 +00:00

Phase 2 PR 3 of docs/website/MODULE_SYSTEM.md §2.7. Docs half: RunicGateway/docs#128.

ensureSchema() now replays every installed module's schema fragment immediately after core's schema.sql, per MODULE_API.md §2.6.

The decision that shaped it

The work splits across two files on the line of whether a database is needed to know the answer.

`modules/loader.js` — validates	`modules/schema.js` — executes
At load time, no DB, before anything is mounted	On the boot path, after core's schema
Every rule §2.6 states about the SQL is knowable by reading it	Only the failures the server alone can report — unknown column type, bad FK
Module never mounts (§4.4, left column)	Routes stay mounted, answer 503 (§4.4, right column)

The alternative — check it all at replay time — would let a module with a plainly bad fragment mount, then 503 with whatever its fragment managed to execute already behind it.

Three smaller ones

A leading-verb allowlist, not a DROP denylist: CREATE, ALTER, INSERT, UPDATE, the four core's own schema.sql uses. §2.6 words the rule as "no DROP", but this file is replayed on every boot — TRUNCATE and DELETE would empty a table at every restart, RENAME would fail at the second one, and a denylist only ever bans what somebody thought of. The code says plainly that it is a leading-verb check and that ALTER TABLE x DROP COLUMN y passes it. A CREATE TABLE missing IF NOT EXISTS is rejected on the same grounds: it works once and fails every boot after, which reaches an operator as a module that broke on restart.
The splitter moves to utils/sqlStatements.js so core's schema and a fragment are split by literally the same code — §2.6 promises that, and two copies would hold it only until one was edited. Its own file rather than an export of utils/db.js because the loader validates at require time and must not drag the mariadb pool into app.js's require chain.
The replay is outside the wait-for-the-database retry loop. A fragment that throws is one module's failure, not a signal the DB is still coming up; retrying core's whole schema nine more times over one module's bad SQL would turn a 503'd module into a two-minute boot.

One thing found while wiring it

db/seed.js calls ensureSchema() standalone for npm run seed, without ever requiring app.js — so the loader has not scanned, and fragments()'s §7.6 throw would have broken seeding outright. The replay asks isLoaded() and logs the skip rather than swallowing it; a booting server quietly getting no module tables is the thing §7.6 exists to prevent, so the predicate is documented as having exactly this one sanctioned use.

Verification

856 server tests pass, 14 new. moduleSchema.test.js injects the query fn, so the exact statements and their order are asserted with the pool at a dead port like every other suite here.
routes.manifest.json and routes.guards.json diffs are zero lines, 229 routes — the Phase 2 exit criterion. swagger-output.json regenerates byte-identical.
Run for real against the local MariaDB with two fixture modules, which is the part unit tests cannot reach:
- the good fragment created its table, applied its ALTER … ADD COLUMN IF NOT EXISTS and seeded its row (schema ensured for module "demo" {statements: 3});
- a fragment that passes validation but the server rejects (id NOTATYPE) marked only that module startup_failed, carrying MariaDB's own message as the reason;
- GET /api/v1/public/demo → 200, GET /api/v1/public/brokendb → 503, in the same process;
- a second ensureSchema() on the same database was a clean no-op. Fixture tables dropped afterwards.
No module ships on the volume, so nothing an operator or a client can see changes.

AI-assisted: written with Claude Code (Opus 5), reviewed before opening.

Phase 2 PR 3 of [`docs/website/MODULE_SYSTEM.md` §2.7](https://gitea.whitlocktech.com/RunicGateway/docs/src/branch/main/website/MODULE_SYSTEM.md). Docs half: RunicGateway/docs#128. `ensureSchema()` now replays every installed module's schema fragment immediately after core's `schema.sql`, per `MODULE_API.md` §2.6. ## The decision that shaped it The work splits across two files on the line of **whether a database is needed to know the answer**. | `modules/loader.js` — **validates** | `modules/schema.js` — **executes** | | --- | --- | | At load time, no DB, before anything is mounted | On the boot path, after core's schema | | Every rule §2.6 states about the SQL is knowable by reading it | Only the failures the server alone can report — unknown column type, bad FK | | Module never mounts (§4.4, left column) | Routes stay mounted, answer **503** (§4.4, right column) | The alternative — check it all at replay time — would let a module with a plainly bad fragment mount, then 503 with whatever its fragment managed to execute already behind it. ## Three smaller ones - **A leading-verb allowlist, not a `DROP` denylist**: `CREATE`, `ALTER`, `INSERT`, `UPDATE`, the four core's own `schema.sql` uses. §2.6 words the rule as "no `DROP`", but this file is **replayed on every boot** — `TRUNCATE` and `DELETE` would empty a table at every restart, `RENAME` would fail at the second one, and a denylist only ever bans what somebody thought of. The code says plainly that it is a leading-verb check and that `ALTER TABLE x DROP COLUMN y` passes it. A `CREATE TABLE` missing `IF NOT EXISTS` is rejected on the same grounds: it works once and fails every boot after, which reaches an operator as a module that broke on restart. - **The splitter moves to `utils/sqlStatements.js`** so core's schema and a fragment are split by literally the same code — §2.6 promises that, and two copies would hold it only until one was edited. Its own file rather than an export of `utils/db.js` because the loader validates at require time and must not drag the mariadb pool into `app.js`'s require chain. - **The replay is outside the wait-for-the-database retry loop.** A fragment that throws is one module's failure, not a signal the DB is still coming up; retrying core's whole schema nine more times over one module's bad SQL would turn a 503'd module into a two-minute boot. ## One thing found while wiring it `db/seed.js` calls `ensureSchema()` standalone for `npm run seed`, **without ever requiring `app.js`** — so the loader has not scanned, and `fragments()`'s §7.6 throw would have broken seeding outright. The replay asks `isLoaded()` and logs the skip rather than swallowing it; a booting server quietly getting no module tables is the thing §7.6 exists to prevent, so the predicate is documented as having exactly this one sanctioned use. ## Verification - **856 server tests pass**, 14 new. `moduleSchema.test.js` injects the `query` fn, so the exact statements and their order are asserted with the pool at a dead port like every other suite here. - **`routes.manifest.json` and `routes.guards.json` diffs are zero lines**, 229 routes — the Phase 2 exit criterion. `swagger-output.json` regenerates byte-identical. - **Run for real against the local MariaDB** with two fixture modules, which is the part unit tests cannot reach: - the good fragment created its table, applied its `ALTER … ADD COLUMN IF NOT EXISTS` and seeded its row (`schema ensured for module "demo" {statements: 3}`); - a fragment that passes validation but the *server* rejects (`id NOTATYPE`) marked only that module `startup_failed`, carrying MariaDB's own message as the reason; - `GET /api/v1/public/demo` → **200**, `GET /api/v1/public/brokendb` → **503**, in the same process; - a second `ensureSchema()` on the same database was a clean no-op. Fixture tables dropped afterwards. - No module ships on the volume, so nothing an operator or a client can see changes. --- - [x] AI-assisted: written with Claude Code (Opus 5), reviewed before opening.

wtclaude added 1 commit 2026-08-10 21:59:16 +00:00

feat(modules): replay module schema fragments after core's

PR Checks / bot-install (pull_request) Successful in 22s

Details

PR Checks / client-build (pull_request) Successful in 28s

Details

PR Checks / server-tests (pull_request) Successful in 10m9s

Details

2892d01b24

Phase 2, PR 3 of docs/website/MODULE_SYSTEM.md 2.7. ensureSchema() now replays
every installed module's schema fragment immediately after core's schema.sql,
per MODULE_API.md 2.6.

The work splits across two files on the line of whether a database is needed to
know the answer. loader.js VALIDATES a fragment at load time, before anything is
mounted, because every rule 2.6 states about the SQL is knowable by reading it;
a module that breaks one never mounts (4.4, left column). modules/schema.js
EXECUTES it, so the only failures there are the ones the database alone could
report, and those are post-mount and answer 503 (4.4, right column).

Validation is a leading-verb allowlist -- CREATE, ALTER, INSERT, UPDATE, the
four core's own schema.sql uses -- rather than the DROP denylist 2.6 words it
as. A fragment is replayed on every boot, so TRUNCATE and DELETE would empty a
table at each restart and RENAME would fail at the second one; a denylist only
ever bans what somebody thought of. A CREATE TABLE missing IF NOT EXISTS is
rejected for the same reason: it works once and fails every boot after, which
presents to an operator as a module that broke on restart.

The splitter moves to utils/sqlStatements.js so core's schema and a fragment are
split by literally the same code, which is what 2.6 promises. It is its own file
rather than an export of utils/db.js because the loader validates fragments at
require time and must not drag the mariadb pool into app.js's require chain.

The replay sits outside ensureSchema's wait-for-the-database retry loop: a
fragment that throws is one module's failure, not a signal the database is
coming up, and retrying core's whole schema nine more times over one module's
bad SQL would turn a 503'd module into a two-minute boot.

Found while wiring it: db/seed.js calls ensureSchema() standalone for
`npm run seed`, without ever requiring app.js, so the loader has not scanned and
fragments()'s 7.6 throw would have broken seeding outright. The replay asks
isLoaded() and logs the skip rather than swallowing it -- a booting server
quietly getting no module tables is the thing 7.6 exists to prevent.

Verification:

- 856 server tests pass, 14 new. moduleSchema.test.js injects the query fn, so
  the exact statements and their order are asserted with the pool at a dead port
  like every other suite.
- routes.manifest.json and routes.guards.json diffs are zero lines, 229 routes
  -- the phase 2 exit criterion. swagger-output.json regenerates byte-identical.
- Run for real against the local MariaDB with two fixture modules: a good
  fragment created its table, applied its ALTER and seeded its row; a fragment
  whose SQL passes validation but the server rejects (`id NOTATYPE`) marked only
  that module startup_failed, its route answering 503 while the other answered
  200; a second ensureSchema on the same database was a clean no-op.

Co-Authored-By: Claude <noreply@anthropic.com>

whitlocktech merged commit bd749d4f1f into edge

2026-08-10 22:06:46 +00:00

whitlocktech deleted branch feat/modules-schema

2026-08-10 22:06:47 +00:00

whitlocktech referenced this issue from a commit

2026-08-10 22:06:48 +00:00

Merge pull request 'feat(modules): replay module schema fragments after core's (phase 2, PR 3)' (#130) from feat/modules-schema into edge

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: RunicGateway/website#130