RunicGateway/docs
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(website): settle the three de-entanglement registries and what they cost

Browse Source 
Phase 2 PR 4 of MODULE_SYSTEM.md §2.7. Records what §1.8's three entangled files
and §1.9's extension slot actually became, and four decisions taken with them.

MODULE_API.md §2.4:
- registerNotificationStreams takes the catalog ALONE. `mapEvent` was a leftover
  from before §1.8's push inversion was settled — a module owns fromShardEvent
  and calls publish() with an id it resolved, so core never needs a second route
  to the same place. It follows that the public-safety filter is module-internal,
  which is the right home: the kinds, the streams and the filter become one file
  that moves together.
- the entry shape is two booleans, not a single `scope` — that object is the body
  of GET /auth/me/notifications/streams and a shipped Android client reads both.
- registerAnnounceLeg gains `label`, so a module's leg renders in the admin panel
  with no client change; and legs became `announce_job_legs` ROWS, because a
  module cannot ALTER a core table and a registered leg had nowhere to live.
- every call STAGES; nothing commits until the module as a whole is known good.

New §6.5 — the eight grandfathered names (seven stream ids, one leg id), allowed
to `uo` alone by explicit allowlist, the same shape as the loader's legacy table
prefixes. Grandfathering rather than dropping the rule is what keeps it real for
every module written after this one.

New §6.6 — an extension slot is invisible to static analysis, so core needs §6.1's
fragment merge for its OWN slot fills, a phase earlier than the plan expected.
Moving the six users-detail routes behind the slot deleted 407 lines from
swagger-output.json while printing `Success`. Which slots and where each hangs are
both derived — from the registry and from the live express stack — because a
written-down copy drifts.

MODULE_SYSTEM.md §1.8 / §1.9 record the outcome per file, and §2.7 the progress.
BACKEND_DESIGN.md and website-README.md follow the moved files; the PUBLIC_KINDS
reference at §"visibility" named the wrong file and is corrected to
utils/shardBroadcast.js.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
claude
2026-08-10 17:51:02 -05:00
parent bcd3a750e7
commit 58b435be70
4 changed files with 162 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
website/BACKEND_DESIGN.md
View File

@@ -365,7 +365,7 @@ A DB read never yields a usable reset link. See §4 `/auth/password/*`.
| col | type | notes |
|---|---|---|
| user_id | INT NOT NULL FK→users(id) ON DELETE CASCADE | |
| stream_id | VARCHAR(64) NOT NULL | an id from the catalog (`config/notificationStreams.js`), validated on write |
| stream_id | VARCHAR(64) NOT NULL | an id from the catalog (`modules/registries.js` — core's plus every installed module's), validated on write |
| created_at | DATETIME | |
`PRIMARY KEY(user_id, stream_id)`. Subscriptions are per-user (applied to every device); a PUT
@@ -793,14 +793,18 @@ registers device endpoints (`/auth/me/devices`); nothing is pushed unless subscr
self-hosted **ntfy** endpoint (`utils/pushDispatch`); the app wakes and pulls the real, ownership-
checked content over the authenticated API. Two producers fan out through the one publisher: the shard
ingest dispatcher (`utils/shardIngest`, beside the SSE broadcast) for shard-derived streams, and the
create/publish-post path for `news.post`. The stream catalog + event→stream mapping is
`config/notificationStreams.js`. Security invariants:
create/publish-post path for `news.post`. The catalog is assembled at boot by
`modules/registries.js` from core's own streams (`config/coreStreams.js` — just `news.post`) plus
each installed module's; the shard streams and their event→stream mapping are
`config/shardStreams.js`, which belongs to module-uo and moves out with it
(MODULE_SYSTEM.md §1.8). Security invariants:
- **Same public/admin split as the SSE feed.** Public streams are drawn *only* from the SSE
`PUBLIC_KINDS` allowlist; a sensitive kind (audit/cheat/IP/login-attempt) can never produce a public
push.
- **Personal streams are owner-keyed.** `vendor.sale` / `house.idoc` / `account.login` are delivered
only to the *owning* user's devices, resolved via `shardLinks` (the same ownership check as
`/player/shard/*`).
`/player/shard/*`) in `utils/shardPush.js`. `utils/pushDispatch.js` itself only publishes to a
stream id someone else resolved — it has no idea what a shard event is.
- **SSRF guard.** A device `endpoint` is a client-supplied URL the server POSTs to, so registration and
every publish validate it is HTTPS, non-private/loopback, and (when configured) on the shard's ntfy
allow-set (`NTFY_BASE_URL` / `NTFY_ALLOWED_ORIGINS`).
@@ -1130,7 +1134,7 @@ it picks, it fails open on one side.)
| Nav | `GET /public/shard/features` returns only what the caller may reach, so the SPA never renders a link that would 403. Presentation only. |
Config reads are cached ~5s, so admin changes take effect within seconds **including on already-open
streams**. `PUBLIC_KINDS` still exists and is still exported (`notificationStreams.js`) but is now
streams**. `PUBLIC_KINDS` still exists and is still exported (`utils/shardBroadcast.js`) but is now
**derived** from the kind map rather than hand-maintained, so the two cannot drift.
**`PUBLIC_KINDS` is a module-load constant and must not be used to answer "may this caller read this

104
website/MODULE_API.md
View File

@@ -157,12 +157,21 @@ validated at once rather than at first use.
```js
api.registerRoutes({ public: {...}, admin: {...}, player: {...} })
api.registerExtension(slot, router)
api.registerNotificationStreams({ streams, mapEvent })
api.registerAnnounceLeg({ leg, dispatch, classify })
api.registerNotificationStreams(streams)
api.registerAnnounceLeg({ leg, label, dispatch, classify })
api.onBoot(async (ctx) => {})
api.onShutdown(async () => {})
```
**Every call STAGES; nothing is committed until the module as a whole is known good.** A claim's
shape is checked at the call, so a malformed one throws with the registrant's own stack; whether the
name is *taken* can only be answered once the batch is complete, and is checked when the loader
commits it in its second pass. The consequence is the one that matters: a module that registers two
streams and then throws — or fails `checkDeclared` after `register()` returns — has left nothing
behind. A half-registered catalog would be worse than a missing one, because it is a subscribable
stream nothing will ever publish to. This is the registry-side twin of §4.3's second-pass mount rule,
and both exist for the same reason.
**`registerRoutes(mounts)`** — one `express.Router()` per prefix per tier:
```js
@@ -194,15 +203,42 @@ The router receives `req.params.id` from the parent (`mergeParams: true`). Two m
same slot is a collision and is rejected; core's own routes on the resource always win a path
conflict.
**`registerNotificationStreams({ streams, mapEvent })`** — §1.8's push catalog.
`streams` is an array of `{ id, label, description, scope }` appended to core's catalog (ids are
namespaced `<moduleId>.<name>` and rejected otherwise); `mapEvent(event) => streamId | null` is
called by core's dispatcher for events the module's own code publishes.
**`registerNotificationStreams(streams)`** — §1.8's push catalog.
An array of `{ id, label, description, personal, requiresLinkedAccount }` appended to core's catalog.
Ids are namespaced `<moduleId>.<name>` and rejected otherwise, save for the seven grandfathered ones
in §6.4.
**`registerAnnounceLeg({ leg, dispatch, classify })`** — §1.8's news dispatcher.
`leg` is a namespaced id, `dispatch(post) => Promise<void>` delivers, `classify(post) => boolean`
decides whether this leg wants the post. A leg that throws is retried by core's existing per-leg
retry and never blocks another leg.
Two amendments this signature carries, both settled 2026-08-10 with PR 4:
- **`mapEvent` is gone.** The earlier signature took `{ streams, mapEvent }`, with core's dispatcher
calling `mapEvent(event) => streamId`. That was a leftover from before §1.8's push inversion was
settled: the module owns `fromShardEvent` outright and calls `ctx.push.publish(streamId, …)` with
an id it has already resolved, so core never needs a second way to get there. What core wants from
a module here is the catalog — for the subscribe endpoint, for validating a subscription write, and
for the personal/linked-account gate. It follows that the public-safety filter (a sensitive event
kind can never produce a *public* push) is module-internal; that is the right home, because the
kinds, the streams and the filter are then one file that moves together, rather than a rule in core
about data only the module defines.
- **Two booleans, not one `scope`.** The entry shape above is the response body of
`GET /auth/me/notifications/streams`, which a shipped Android client already reads
(`NotificationsDto.kt`). `scope` was never the wire shape.
**`registerAnnounceLeg({ leg, label, dispatch, classify })`** — §1.8's news dispatcher.
`leg` is a namespaced id, `label` is what the admin panel shows, `dispatch(post) => Promise<result>`
delivers, and `classify(result) => { outcome, error }` maps the client's result to
`done` / `retry` / `terminal`. A leg that throws is caught, classified as a retry, and never blocks
another leg.
`label` is an addition: the panel used to hold a client-side `{ towncrier, discord }` label table,
which would have left a module's leg rendering as a bare id. It comes from the registration so a
module needs no client change.
**Legs are rows, not columns.** `announce_jobs` carried a `towncrier_*` and a `discord_*` column
group until PR 4; a module cannot `ALTER` a core table, so a registered leg had nowhere to live. The
per-leg state moved to `announce_job_legs (job_id, leg, status, attempts, last_error,
next_attempt_at)` and `leg` is a stored value. The parent `status` rollup is over *all* the job's
legs — done when every leg delivered, failed when every leg gave up, partial in between; and `done`
when a job has no legs at all, since nothing is left to deliver.
**`onBoot(fn)` / `onShutdown(fn)`** — §2.5.
@@ -665,6 +701,54 @@ files. Two consequences:
Neither changes a decision; both are corrected here rather than left to be tripped over when the
extraction is counted against the plan.
### 6.5 Grandfathered names, and why the prefix rules survive them
§2.4 requires a module's stream ids and announce legs to carry its module id. Eight names predate the
module system and cannot take it:
| Kind | Names | Why they cannot be renamed |
| --- | --- | --- |
| Streams | `server.status`, `idoc.warning`, `champ.start`, `governor.election`, `vendor.sale`, `house.idoc`, `account.login` | stored in `notification_subs` rows; read by a shipped Android client |
| Announce leg | `towncrier` | a stored value in `announce_job_legs.leg` and the body of the retry endpoint |
They are allowed to **`uo` alone**, by an explicit per-module allowlist — the same shape and the same
reasoning as the loader's `LEGACY_TABLE_PREFIXES` for module-uo's 27 tables. Grandfathering by
allowlist rather than dropping the rule is what keeps the rule real for every module written after
this one; the alternative leaves the first name collision to be discovered by a module silently
adopting someone else's stream.
### 6.6 An extension slot is invisible to static analysis — core needs the merge too
§6.1 settled the fragment merge for *modules*. PR 4 found that core needs the identical machinery for
its **own** slot fills, one phase earlier than the plan expected.
A slot's router is created by `registries.declareSlot()` and filled later, so there is no literal
`use(require(...))` for swagger-autogen to follow. Moving the six `/admin/users/:id/shard/*` routes
behind `admin.users.detail` therefore deleted 407 lines from `swagger-output.json` — with
`Swagger-autogen: Success` and no warning. Same failure as §7.4, different cause, and it would have
shipped six undocumented core routes against CLAUDE.md's standing rule.
`npm run swagger` now has a second step (`swagger/slotSpecs.js`): for each **filled** slot, generate a
fragment by pointing swagger-autogen at that router's own file, re-root its paths at the prefix the
router actually hangs at, and merge. Two things are derived rather than written down, because a
written-down copy drifts:
- **which** slots — from `registries.filledSlots()`;
- **where** each hangs — by finding the slot's own router object in the live express stack, decoding
the mount prefixes above it with `scripts/routeManifest.js`'s own `mountPath`, so the manifest and
the spec can never disagree about what a mount decodes to.
An empty fragment is a hard build failure, because an empty fragment is exactly what the silent drop
looks like. The merge itself is `swagger/mergeSpec.js` — the ~40-line helper §6.1a already owed core
for module fragments, written here and proved against core's own slot before a module depends on it.
This is **build**-time and lands in the committed spec, because slot routes are core's; a module's
fragment is still merged at **request** time into `/api/docs.json` (§6.1a), and `swagger-output.json`
stays reproducible on any machine regardless of what is installed.
`registerExtension` therefore takes a third, **core-only** argument: the file its router is generated
from. A module needs no equivalent — it ships a prebuilt `swagger-fragment.json`, because core never
has its sources to analyse.
---
## Part 7 — What the spike proved

62
website/MODULE_SYSTEM.md
View File

@@ -153,16 +153,36 @@ Everything else is a folder move. These are not:
1. **`src/config/notificationStreams.js`** — the push stream catalog. `mapShardEvent()` and most of
`STREAMS` are shard-derived, and it imports `PUBLIC_KINDS` from `utils/shardBroadcast`. Push
*infrastructure* is core; this *catalog* is module content.
→ `registerNotificationStreams({ streams, mapEvent })`.
→ `registerNotificationStreams(streams)`.
2. **`src/utils/pushDispatch.js`** — core infrastructure, but `fromShardEvent()` (line 112) requires
the `shardLinks` model (line 21) and `mapShardEvent` (line 23).
→ invert: `publish()` stays core, `fromShardEvent` moves into the module and calls it.
3. **`src/utils/announceWorker.js`** — the news dispatcher, with two delivery legs: Discord (core)
and town crier (module, via `uoLinkClient.postTownCrier`, line 36).
→ `registerAnnounceLeg({ leg, dispatch, classify })`.
→ `registerAnnounceLeg({ leg, label, dispatch, classify })`.
`src/utils/newsGump.js` is module-side (news → in-game gump) and moves whole.
**Done in Phase 2 PR 4**, with core still the only registrant — the registries are
`src/modules/registries.js` and core goes through them by the same door a module will
(`registerCore()`, called explicitly from `app.js` before `modules.load()`). What each of the three
became:
1. Split in two. `config/coreStreams.js` is core's one stream (`news.post`, produced by the website's
own posts path); `config/shardStreams.js` is the other seven plus `mapShardEvent` and the
public-safety filter, and moves to module-uo whole. `registerNotificationStreams` lost its
`mapEvent` half — see [`MODULE_API.md`](MODULE_API.md) §2.4 for why that was a leftover, and what
follows for the public/personal split.
2. Inverted. `pushDispatch.js` is `publish` + `isAllowedEndpoint` and nothing else;
`utils/shardPush.js` holds `fromShardEvent` and is what `shardIngest` now calls.
3. Legs became registrations, and per-leg **rows**. The `towncrier_*` / `discord_*` column groups on
`announce_jobs` could never have held a module's leg — a module cannot `ALTER` a core table — so
they became `announce_job_legs`, backfilled and dropped in the same idempotent replay. The worker
no longer contains the word "towncrier": it iterates whatever is registered.
The residue in core is a one-time backfill block in `schema.sql`, deletable once every deployment has
booted it, and the two lines of `registerCore()` that Phase 3 turns into module-uo's `register()`.
### 1.9 A fourth mount shape: module routes under a core resource
`router/v1/admin/users.router.js` mounts `usersShard.controller.js` at six UO sub-paths of a **core**
@@ -175,6 +195,17 @@ narrow **extension slot** on `/admin/users/:id` that the module mounts into, so
what "shard" means and all six URLs are preserved. Only core may declare an extension slot; a module
may not invent one.
**Both done in Phase 2 PR 4**, with core filling its own slot: the six paths are
`router/v1/admin/usersShard.router.js`, registered into `admin.users.detail` by `registerCore()`, and
Phase 3 changes the registrant rather than the routes. The slot's router is created at declare time
and filled later, because `users.router.js` is required while `app.js` is still being built. It is
mounted **last** on the resource, so core wins any path conflict by first-match.
One consequence was not foreseen and is worth the warning: **a slot is invisible to static analysis.**
There is no literal mount for `swagger-autogen` to follow, so the move silently deleted all six paths
from `swagger-output.json` while printing `Success`. The OpenAPI build now merges a generated
fragment per filled slot — [`MODULE_API.md`](MODULE_API.md) §6.6.
### 1.10 The Discord bot has no UO logic
The draft listed the bot's "UO-specific event/moderation logic" as an extraction candidate. Grepping
@@ -453,7 +484,7 @@ too (API §7.2).
Exit criterion: `routes.manifest.json` diff is zero lines and every existing test passes. If Phase 2
changes one URL, it is wrong.
**Progress: PRs 1-3 done.**
**Progress: PRs 1-4 done.**
- **PR 1** — `installed_modules` and the state machine, with the stored shape and the boot rules
settled in §2.4 above.
@@ -476,8 +507,31 @@ changes one URL, it is wrong.
as, because the file is replayed on **every boot**. Found while wiring it: `npm run seed` calls
`ensureSchema()` without ever requiring `app.js`, so the replay has to tolerate an unscanned loader.
- **PR 4** — the three de-entanglement registries, `src/modules/registries.js`. Core's own streams,
its Discord announce leg and its users-detail routes all go through them, so the seams are
exercised on every boot before a module depends on them; §1.8 and §1.9 above record what each
became. Four decisions landed with it, all recorded in [`MODULE_API.md`](MODULE_API.md): announce
legs became a **child table** rather than waiting for Phase 3 (§2.4 — a module cannot alter a core
table, so a registered leg had nowhere to live); **`mapEvent` dropped** from the stream registry
(§2.4 — a leftover from before the push inversion was settled); **core registers through the same
staging area a module uses**; and core's six shard sub-paths **moved behind the slot now** rather
than in Phase 3.
Registering is **validate-then-commit**: the loader stages a module's claims and the second pass
commits them, so a module that throws halfway through `register()` — or fails a later validation
step — leaves nothing behind. That is the registry-side twin of PR 2's second-pass mount rule.
Two build tools needed teaching, both because a mechanism this PR introduced is one they had never
seen. `scripts/routeManifest.js` could not decode a **parameterised mount**: its unwinder expected
a group shape express does not emit, and the branch had never run. It threw rather than guessing,
which is exactly what it is for. And `swagger-autogen` could not follow a route into an extension
**slot**, deleting 407 lines while reporting success; the fix is the fragment merge core owed
anyway ([`MODULE_API.md`](MODULE_API.md) §6.6).
There is still no module on the volume and no boot wiring, so this changes nothing an operator or a
client can see: 856 tests pass and `routes.manifest.json` is unchanged at 229 routes.
client can see: **884 tests pass** and `routes.manifest.json` is unchanged at 229 routes. The two
lines of OpenAPI that do move are the retry endpoint's summary and its `leg`, which is no longer a
fixed enum because the leg set is whatever has been registered.
**Phase 3 — Extract `module-uo`.** Moves out of `website/`: the 8 model directories and their 25
tables; the nine UO `utils/` files plus `newsGump.js`; the 13 router/controller files;

2
website/website-README.md
View File

@@ -482,7 +482,7 @@ Copy `.env.example` (Compose) or `server/.env.example` (local) and fill in. **`.
| `CLIENT_ORIGIN` | `http://localhost:5173` | enables CORS in dev only |
| `LOG_LEVEL` / `FILE_LOG_LEVEL` | `info` / `debug` | console / file verbosity |
| `LOG_TO_FILE` / `LOG_DIR` / `LOG_FILE` | `true` / `<server>/logs` / `app.log` | log file (bind-mounted to `./logs` in Docker) |
| `ANNOUNCE_POLL_MS` | `15000` | how often the news-announcement dispatcher sweeps `announce_jobs` for due/retry legs (town crier + Discord) |
| `ANNOUNCE_POLL_MS` | `15000` | how often the news-announcement dispatcher sweeps `announce_job_legs` for due/retry legs (whichever are registered — Discord is core's, the town crier is module-uo's) |
| `TOWNCRIER_DURATION_SEC` | `3600` | how long a news post's in-game town-crier message stays up (≤ `86400`) |
---