RunicGateway/docs
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(website): the spawn atlas API, the admin panel, and the delay-unit trap #68

Merged
whitlocktech merged 3 commits from docs/spawn-atlas-api into edge 2026-07-29 00:53:56 +00:00
Conversation 1 Commits 3 Files Changed 4 +426 -132
4 changed files with 426 additions and 132 deletions
Download Patch File Download Diff File Expand all files Collapse all files

124
link/v3.md
View File

@@ -13,7 +13,7 @@ Each part is marked off here as it lands on `edge`. §9 carries the same state p
|---|---|---|---|
| 1 | **A** — visibility framework + actor-leak fix (§3) | ✅ **Done** | website [#109](https://gitea.whitlocktech.com/RunicGateway/website/pulls/109) + [#110](https://gitea.whitlocktech.com/RunicGateway/website/pulls/110), docs [#64](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/64) + [#65](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/65) |
| 2 | **B/1** — `world.ruleset` (§5) | ✅ **Done** | servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66) |
| 3 | **C** — spawn atlas (§6) | 🟡 **Data pipeline done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables); API/client PR next |
| 3 | **C** — spawn atlas (§6) | ✅ **Done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables) + [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113) (API + pages + admin panel), docs [#67](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/67) + [#68](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/68) |
| 4 | **B/2** — `points.board` (§7) | ⬜ Not started | — |
| 5 | **B/3** — `vendor.listing` (§8) | ⬜ Not started | — |
| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | ⬜ Not started | — |
@@ -329,28 +329,33 @@ frame during verification.
**No plugin, no sidecar, no `Bridge.cfg` knob, no new kinds.** Not part of the v3 wire change.
> **Status:** data pipeline landed on `edge` — website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112)
> (parsers, build/import CLI, tables, artifact). API + client pages are the second website PR.
> **Status:** complete on `edge` — website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112)
> (parsers, import CLI, tables) and [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113)
> (the six public routes, the five admin ones, `/site/atlas` + `/site/atlas/:slug`, and the
> Admin → Spawn Atlas panel).
> Part C ships as **two** website PRs, not one: the parsing half is where the correctness risk
> lives, and burying it under routes and React would have meant reviewing it in a 10k-line diff.
> Full operator documentation: [`docs/website/SPAWN_ATLAS.md`](../website/SPAWN_ATLAS.md).
>
> §6.1 below records where the shipped implementation differs from this design. The differences are
> all things the real ServUO data forced, not changes of mind.
> **§6 below is the original design and is partly superseded.** §6.1 records two decisions that were
> rejected in review and replaced (the committed artifact, and the fixed facet list); §6.2 records
> the corrections the real ServUO data forced. Read both before trusting §6.
**Decision: committed generated artifact + idempotent DB import**, split in two because the build
needs the ServUO tree (which the website container does not have) and the import does not. Not
runtime import (10.5 MB of XML per boot), not a browser-served blob.
**Decision (revised at implementation time): the shard's ServUO tree is the single source of truth,
re-derived on every server boot.** The original plan here was a committed generated artifact plus an
idempotent import. That was rejected in review for two reasons, recorded in §6.1: a snapshot in the
repo goes stale as a shard's maps change, and the design leaned on a fixed facet list that no shard
is obliged to keep. Still not a browser-served blob; still parsed server-side only.
New in `website/server/`:
- `src/utils/spawnAtlasParse.js` — **pure functions, no fs**, so they are unit-testable in CI without
a ServUO tree: `parseObjects2()`, `parsePoints()`, `parseRegions()`, `parseLocations()`,
`resolveRegion()`.
- `scripts/buildSpawnAtlas.js` (`--servuo <path> --out db/data/`) and `scripts/importSpawnAtlas.js`
(TRUNCATE + batched INSERT in one transaction); `package.json` scripts `atlas:build`, `atlas:import`.
- `db/data/spawnAtlas.<facet>.json` ×13 + `spawnAtlas.index.json` (creatures, champions, regions,
landmarks, meta with per-source-file hashes).
- ~~`scripts/buildSpawnAtlas.js` and a committed `db/data/spawnAtlas.*.json` artifact~~ — dropped,
see §6.1 R1. Replaced by `src/utils/spawnAtlasSource.js` (the only thing that reads a ServUO tree,
shared by the boot path and the CLI) and a `scripts/importSpawnAtlas.js` that is a thin CLI over
the model. `package.json` gains `atlas:import` only.
- `src/model/shardAtlas/{shardAtlas.db.js,shardAtlas.model.js}` following the `shardState` split.
- `src/router/v1/public/atlas.{router,controller}.js`; `test/spawnAtlas.parse.test.js`.
@@ -381,18 +386,48 @@ stays CLI-only.**
Client: `routes/public/Atlas.jsx` (`/site/atlas`) and `AtlasCreature.jsx` (`/site/atlas/:slug`).
**Payload risk** — a monolithic artifact would be 2–3 MB of committed JSON. Shard per facet and drop
every `<Points>` field the site cannot use (`UniqueId`, all trigger/refractory/proximity/sequential
fields, sound ids), keeping Name/Map/X/Y/W/H/Range/MaxCount/MinDelay/MaxDelay/TOD*/types — well under
1 MB. The artifact never reaches the browser; the browser sees only paginated API responses.
**Payload risk** — *superseded by §6.1 R1; nothing is committed.* The field selection it describes
still applies at parse time: every `<Points>` field the site cannot use (`UniqueId`, all
trigger/refractory/proximity/sequential fields, sound ids) is dropped, keeping
Name/Map/X/Y/W/H/Range/MaxCount/MinDelay/MaxDelay/TOD*/types. Parsed data never reaches the browser;
the browser sees only paginated API responses.
**Operator re-run story** — spawns changed → `npm run atlas:build -- --servuo <path>` on a machine
with the tree → commit the regenerated `db/data/spawnAtlas.*.json` → deploy → `npm run atlas:import`
(or `POST /admin/shard/atlas/import`). `shard_atlas_meta.source` holds per-file hashes, so
`GET /admin/shard/atlas/status` reports when the DB is behind the artifact. Full detail in
`docs/website/SPAWN_ATLAS.md`.
**Operator re-run story** — *revised by §6.1 R1.* Spawns changed → restart, or
`npm run atlas:import` / `POST /admin/shard/atlas/import` to apply without one. `shard_atlas_meta`
holds a sha256 per source file, so the server can tell on boot whether anything changed, and
`GET /admin/shard/atlas/status` reports drift. If the change would remove a facet it is staged for
approval rather than applied (§6.1 R3). Full detail in `docs/website/SPAWN_ATLAS.md`.
### 6.1 What the build against real data changed
### 6.1 What implementation changed
Two design decisions in §6 were rejected in review and replaced; the rest are corrections the real
ServUO data forced. Kept as a diff rather than edited in place, because each is a trap the next
person would otherwise re-enter.
**R1. The committed artifact is gone — the tree is re-parsed on every boot.** §6 proposed building a
generated artifact, committing it, and importing it. Two problems. A shard's maps change over its
life, so a snapshot in the repo silently drifts from the world players actually see; and the build/
import split existed only to work around the website container not having a tree, which is a
deployment question (mount it) rather than a reason to freeze data. The server now hashes the source
files on boot and re-derives the atlas when they differ. `scripts/buildSpawnAtlas.js`, the 1.41 MB
artifact, and the whole encode/decode seam it needed are deleted.
**R2. Nothing may name a facet.** The first implementation carried a lookup table of the six stock
UO facets to reconcile the spelling drift between sources. A shard may add facets, replace them
outright, or rename them when its maps are updated, and a built-in list mishandles all three
silently. Reconciliation is now by *matching* against the facet set discovered from the shard's own
spawn and region data — exact key, then prefix in either direction — with an unmatched name keeping
its own rather than being forced into a wrong bucket.
**R3. Two contracts on the boot path.** It never blocks startup: no path, an unreadable mount, a
malformed file or a database error is caught and logged, and the site comes up serving whatever
atlas it had. And a refresh that would REMOVE a facet is never applied automatically — facet loss
is indistinguishable at boot from a half-copied or mid-update tree, so it is staged in
`shard_atlas_pending` for an admin to approve or reject. Only the decision is stored (source hashes
+ the facet diff, a few KB); approving re-parses, so what lands matches the tree at approval time.
A rejection is remembered against those hashes so it does not re-prompt every restart.
### 6.2 What the build against real data changed
Six corrections to the design above, from running it against stock ServUO 57.4. Kept as a diff
rather than edited in place, because each one is a trap the next person would otherwise re-enter.
@@ -419,12 +454,11 @@ they invent creatures that do not exist *and* split real ones in two, since `Fai
leaves **800** real creatures. (The design's "~1,500 creature rows" estimate was high; 800 only
reinforces the plain-`INDEX`-not-`FULLTEXT` call.)
**5. The artifact is 1.41 MB, not "well under 1 MB".** Dropping the unused `<Points>` fields as the
design directed still left 4.40 MB. Three further encodings — `facet` dropped per record,
default-valued fields omitted rather than written as `0`, and `types` as `[name, max]` tuples
(~24,000 entries × 15 bytes of repeated key names) — brought it to 1.41 MB. Getting under 1 MB would
mean dropping the spawner `name`, which is the only human handle on a specific spawner and worth
keeping. `encodePoint()` and `readPoint()` are exact inverses and are round-tripped in tests.
**5. The artifact would have been 1.41 MB, not "well under 1 MB" — and is now moot.** Dropping the
unused `<Points>` fields as the design directed still left 4.40 MB; three further encodings brought
it to 1.41 MB, and getting under 1 MB would have meant dropping the spawner `name`. The size budget
in §6 was simply optimistic for 6,455 points. Superseded by §6.1 R1: there is no artifact, so there
is no payload to budget and no encode/decode seam to keep in sync.
**6. `DELETE`, not `TRUNCATE`.** The design said "TRUNCATE + batched INSERT in one transaction",
which does not hold: `TRUNCATE` is DDL in MariaDB and implicitly commits, so a mid-import failure
@@ -439,6 +473,38 @@ region, 1,690 by landmark, 1,086 Wilderness).
**One thing the design got exactly right:** the point-in-rect transform really is the reason to
build this. "Where does a lizardman spawn?" answers *Shrines, Isamu-Jima, Yew* across three facets.
### 6.3 What the API/client half added
The second website PR built the six public routes, the five admin ones, `/site/atlas` +
`/site/atlas/:slug`, and the Admin → Spawn Atlas panel. Three things it changed or established:
**1. Respawn delays were being read in the wrong unit — sometimes.** XmlSpawner writes
`MinDelay`/`MaxDelay` in minutes and switches to seconds only when a delay does not divide into
whole minutes, flagging that per record with `DelayInSec`
(`XmlSpawner2.cs:7462-7480`, read back at `:6345-6358`). So a `5` means five *minutes* on one
spawner and five *seconds* on the next, both plausible, and the pipeline stored the raw number.
170 of 6,455 stock spawners are second-flagged — few enough to look like noise on a page and be
believed. The parser now normalises to **seconds**, and the API and UI carry seconds throughout.
*This is the class of bug §6.2 is a list of: the atlas still builds, it is just quietly wrong.*
**2. The hash gate needed a parser version, and this generalises.** Fixing the parse exposed that
"has the tree changed?" is the wrong question on its own — an install whose maps never change would
have kept serving the old readings forever, because the only thing compared was the tree.
`spawnAtlasSource.PARSER_VERSION` is stored in `shard_atlas_meta` beside the source hashes, and a
mismatch counts as drift. Any future parse correction lands on the next boot without an operator
having to know it happened. **Bump it whenever the parser derives different data from identical
files.**
**3. `points` is a count; `spawners` is the list.** The first cut of the detail route spread the
creature row and then set `points` to the array of spawn points — the same key meaning a number on
the search route and an array on the detail route. Renamed before it shipped, and worth recording
because the two names are one letter apart in meaning and it reads as correct.
**On projection.** The `atlas` feature declares no sensitive fields, so `projectFeature` is a no-op
on every one of these routes today. Every handler calls it anyway, per §3.6.1's rule — the point of
the rule is that the *first* field that needs gating is covered by construction rather than by a
retrofit nobody remembers to do.
---
## 7. Part B/2 — `points.board`
@@ -633,7 +699,7 @@ inherently up to one full cycle old, and the UI must say so.
|---|---|---|---|---|
| 1 | **A** — visibility framework + actor-leak fix | website, docs | none | ✅ Done |
| 2 | **B/1** — `world.ruleset` (§5) | all four | new kind | ✅ Done |
| 3 | **C** — spawn atlas (§6) | website, docs | none | 🟡 Pipeline done, API/client next |
| 3 | **C** — spawn atlas (§6) | website, docs | none | ✅ Done |
| 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | ⬜ |
| 5 | **B/3** — `vendor.listing` (§8) | all four | new kinds | ⬜ |
| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | ⬜ |

53
website/BACKEND_DESIGN.md
View File

@@ -101,6 +101,10 @@ server/
deliberately not site-mode gated
shard.router.js (14) /public/shard/* incl. the anonymous
SSE stream; never site-mode gated
atlas.router.js (6) /public/atlas/* — the spawn atlas.
NOT under /shard: nothing here
touches the sidecar, and unlike
/shard it IS site-mode gated
site.router.js (4) /settings /status /version /contact —
the group-root singletons; declares no
router-level middleware
@@ -391,9 +395,15 @@ discarded. See §6.5.
### shard_spawn_* / shard_regions / shard_landmarks / shard_champion_spawns / shard_atlas_meta — the spawn atlas (Protocol 3.0)
Static shard **content**, not live shard state. Nothing here comes from the sidecar: the atlas is
built from a ServUO tree by a CLI script, committed as JSON under `server/db/data/`, and loaded with
`npm run atlas:import`. These tables stay populated whether the shard is up or not. Full operator
detail in [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md); the design is `docs/link/v3.md` §6.
derived from the shard's own ServUO tree, re-read on **every server boot** and hash-gated so an
unchanged tree costs one read pass and no write. Nothing is precomputed and committed — a shard's
maps change over its life, and a snapshot in the repo would silently drift from the world players
actually see. These tables stay populated whether the shard is up or not. Full operator detail in
[`SPAWN_ATLAS.md`](SPAWN_ATLAS.md); the design is `docs/link/v3.md` §6.
**No facet name appears anywhere in the code.** A shard may add facets, replace them, or rename them
when its maps are updated; the facet set is discovered from the tree, and the loose spellings in
`Data/Locations` are matched against it rather than looked up in a table.
| Table | Key columns |
|---|---|
@@ -403,13 +413,32 @@ detail in [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md); the design is `docs/link/v3.md` §
| `shard_regions` | `facet`, `name`, `type`, `priority`, `parent`, `rects` JSON |
| `shard_landmarks` | `facet`, `name`, `grp`, `x`, `y`, `z` |
| `shard_champion_spawns` | `slug` PK, `name`, `grp`, `type`, `random_type`, `facet`, `x`, `y`, `z`, `radius`, `label` |
| `shard_atlas_meta` | Singleton (`id = 1`), `payload` JSON, `imported_at` |
| `shard_atlas_meta` | Singleton (`id = 1`), `payload` JSON (counts, a sha256 per source file, `parserVersion`), `imported_at` |
| `shard_atlas_pending` | Singleton (`id = 1`), `status` (`pending`/`rejected`), `payload` JSON, `detected_at` |
All of them are **import-owned**: `atlas:import` empties and reloads every one inside a single
transaction, so a failed import leaves the previous atlas intact rather than a half-loaded world.
The first seven are **import-owned**: a refresh empties and reloads every one inside a single
transaction, so a failed reload leaves the previous atlas intact rather than a half-loaded world.
Nothing else writes to them, and nothing holds a foreign key to them — no FKs at all, consistent with
every other `shard_*` table.
**`shard_atlas_pending` is the security-relevant one.** A refresh that would REMOVE a facet is never
applied automatically: facet loss is indistinguishable at boot from a half-copied or mid-update tree,
so it is staged here for an admin to approve or reject, and **startup is never blocked by it**. Only
the decision is stored — source hashes plus the facet diff, a few KB — and approving re-parses the
tree, so a multi-megabyte blob never lands in the database and what gets applied matches the tree at
approval time. A rejection is remembered against those exact hashes so a declined refresh does not
re-prompt on every restart. Everything else (new facets, renamed regions, changed spawns) applies
immediately, since none of it can destroy data an operator would miss.
The boot refresh is **best-effort by contract**: no configured path, an unreadable mount, a malformed
file or a database error is caught and logged, and the site comes up serving whatever atlas it had.
The tree path comes from the `spawn_atlas_servuo_path` setting, falling back to `SERVUO_PATH`.
**A refresh re-derives when the tree changed OR the parser did.** `spawnAtlasSource.PARSER_VERSION`
is stored in `shard_atlas_meta` beside the source hashes and bumped whenever the parser produces
different data from identical files. Hashing the tree alone would strand an install whose maps never
change on whatever an older build derived — a corrected parse would ship and never reach the data.
Four column choices worth stating, because each one is a trap:
- **`spawn_range`, not `range`**, and **`grp`, not `group`** — both are reserved words.
@@ -443,7 +472,7 @@ are authoritative, and they answer different questions:
| Artifact | Source of truth for | Generated by |
|---|---|---|
| `server/routes.manifest.json` — mirrored as [api-route-inventory.json](./api-route-inventory.json) | **What URLs exist.** 200 public routes + 2 on the internal listener, sorted, method + path only. | `npm run routes:manifest`, by walking the live Express stack |
| `server/routes.manifest.json` — mirrored as [api-route-inventory.json](./api-route-inventory.json) | **What URLs exist.** 215 public routes + 2 on the internal listener, sorted, method + path only. | `npm run routes:manifest`, by walking the live Express stack |
| `server/swagger/swagger-output.json` — served at `/api/docs` | **What each route means.** Parameters, bodies, response codes, security. | `npm run swagger`, from `#swagger.*` annotations |
The split is deliberate: Swagger is annotation-derived, so an unannotated route is invisible in it and
@@ -629,6 +658,12 @@ from the per-route **siteMode** middleware (§5), never from an auth gate.
| POST | `/contact` | (rate-limited) send mail via SMTP; if unconfigured, respond `{fallback:"mailto", email}` |
| GET | `/shard/ruleset` | the shard's own published ruleset (Protocol 3.0 `world.ruleset`): expansion, which optional systems are on, skill/stat caps, account and house limits, champion scroll rules, the save/restart schedule. Served from `shard_ruleset`, so it renders while the shard is down; live via `world.ruleset` on `/shard/stream`. Behind `requireFeature('ruleset')`. **`null`** means the shard has never published one — a real answer, distinct from a published ruleset. `caps.skill` / `caps.totalSkill` are in **tenths** (1000 = 100.0). |
| GET | `/shard/features` | the shard features **this caller** may reach plus the audience rung they resolved to (§6.5), so a client hides nav it can't follow. Reports only what the caller can see — the list itself never discloses a gated feature. Consumed by the SPA header and (pending) the Android nav. |
| GET | `/atlas/creatures?q=&facet=&limit=&offset=` | the bestiary, most numerous first, with an unpaginated `total`. Static content parsed from the shard's ServUO tree — **not** sidecar-backed, which is why the atlas sits outside `/shard`, and unlike `/shard/*` it **is** site-mode gated. Behind `requireFeature('atlas')`. `?facet=` is matched exactly and never validated against a list (no facet name exists in the code); the filter is an `EXISTS` over the points rather than a JSON path or `JSON_SEARCH` built from caller input, whose `%`/`_` wildcards would make `?facet=%` match everything. |
| GET | `/atlas/creatures/:slug` | one creature: `places` (the point-in-rect aggregate — "lizardman → Shrines, Isamu-Jima, Yew"), `spawners` (the bounded raw list, with `spawnersTruncated`), `alsoHere`. **`points` is a COUNT and `spawners` is the LIST** — named apart so one key never means a number on one route and an array on another. `minDelay`/`maxDelay` are in **seconds**, normalised at parse time from the source's per-record minutes-or-seconds. 404 = no such creature in this atlas. |
| GET | `/atlas/regions?facet=&q=` | named regions and the rectangles that placed each spawner |
| GET | `/atlas/landmarks?facet=&q=` | points of interest, labelled by `group` ("Covetous", not "Level 1") |
| GET | `/atlas/champions?facet=` | the **configured** altar roster. Not `/shard/champs`, which is the live board. |
| GET | `/atlas/meta` | facets, counts and when the atlas was parsed. Game-world facts only — the ServUO path, source hashes and any pending refresh are operator detail and live on the admin route. |
Public content GETs pass through the **siteMode** gate (§5).
@@ -672,6 +707,10 @@ file a route sits in — that is the property the route manifest freezes.
| GET | `/users/:id/trusted-devices` | list a user's active trusted devices (never tokens) |
| DELETE | `/users/:id/trusted-devices` · `…/:deviceId` | revoke all / one of a user's trusted devices (logs `admin.trusted_device.revoke[_all]`) |
| POST | `/users/:id/mfa/reset` | recover a locked-out user: disable TOTP + revoke all trusted devices + clear recovery codes (logs `admin.user.totp.reset`) |
| GET | `/shard/atlas` | spawn-atlas status (`adminOnly`): the ServUO path, whether the tree is readable, whether it has drifted from what is loaded, counts, facets, and any refresh staged for review. The public `/atlas/meta` reports the game world only; the filesystem detail is here. |
| POST | `/shard/atlas/import` | re-import without restarting; `{force}` ignores the hash gate. **An unreadable tree answers 200 with `status:"unavailable"`, not 500** — `refresh()` reports outcomes rather than throwing (the boot path must never be blocked by a bad tree) and that contract is preserved at the API. |
| POST | `/shard/atlas/approve` · `/shard/atlas/reject` | answer a refresh staged because it would REMOVE a facet. Approving **re-parses** the tree, so what lands matches it at approval time; rejecting is remembered against those source hashes so it does not re-prompt every restart. 404 when nothing is staged. |
| PUT | `/shard/atlas/path` | point the atlas at a different tree (persisted as `spawn_atlas_servuo_path`, which wins over `SERVUO_PATH`). Blank clears it. Deliberately **does not import** — moving the mount and reloading the world are separate decisions — and returns fresh status so the panel can offer the import next. |
Every admin write logs to `activity_log`.

321
website/SPAWN_ATLAS.md
View File

@@ -1,6 +1,6 @@
# Spawn atlas
**Status:** Data pipeline landed on `edge` (website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112)); API and client pages follow in a second PR.
**Status:** Complete on `edge` — data pipeline in website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112), API + pages in website [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113).
**Design:** [`docs/link/v3.md` §6](../link/v3.md) — Protocol 3.0 Part C.
The spawn atlas is a browsable catalogue of what the shard *contains*: which
@@ -8,6 +8,19 @@ creatures spawn, where, how many, and which champion altars are configured. It
answers "where do I find a lizardman?" with **"Shrines, Yew, Isamu-Jima"** rather
than with a list of raw coordinates.
## Two things that shape the whole design
**The shard's ServUO tree is the single source of truth.** Nothing is
precomputed and committed to the repository. A shard's maps change over its
lifetime — facets get added, replaced, or renamed — and a snapshot in the repo
would silently drift from the world players actually see. The atlas is therefore
re-derived from the tree **on every server boot**.
**Facets are not a fixed list.** Nothing in the codebase names Felucca, Trammel,
or any other stock facet. The facet set is whatever the shard's own files
declare, discovered at parse time. A shard running entirely custom maps gets
exactly the same treatment as a stock one, with no code change.
## What it is not
The atlas is **static shard content, not live shard state.**
@@ -24,44 +37,83 @@ The atlas is **static shard content, not live shard state.**
Routes live at `/api/v1/public/atlas`, deliberately **not** under `/shard`,
because `/shard/*` means sidecar-dependent.
## The two commands
## Configuring the tree
Building needs a ServUO tree. Importing does not. That split is the whole design:
the website container ships the committed artifact but has no ServUO tree, so it
can import but never build.
The website needs to be able to *read* the ServUO tree — same host, a bind mount,
or a shared volume. Two ways to point at it, the setting winning over the
environment:
```bash
# On a machine that has the ServUO tree (writes server/db/data/spawnAtlas.*.json)
cd website/server
npm run atlas:build -- --servuo /path/to/ServUO
| Source | Notes |
|---|---|
| `spawn_atlas_servuo_path` setting | Admin-editable; changes take effect on the next refresh without a redeploy |
| `SERVUO_PATH` env var | The deploy-time default, since the path usually describes a mount the deployment sets up |
# Anywhere, including the deployed container
npm run atlas:import
With neither set the atlas is simply skipped — the site runs normally without
one.
## The boot path
On every start the server hashes the source files and compares them against what
is loaded. Unchanged (the normal case on a restart) costs one read pass, ~120 ms,
and no database write. A real change costs a ~400 ms parse and a reload.
Two contracts govern it:
**1. It never blocks startup.** No configured path, an unreadable mount, a
malformed file, a database error — every one is caught and logged, and the site
comes up serving whatever atlas it already had.
**2. A facet disappearing is never applied automatically.** Losing a facet looks
exactly like a half-copied or mid-update tree, and boot cannot tell that apart
from a real map change. That refresh is *staged* for a human instead. Everything
else — new facets, renamed regions, changed spawns — applies immediately, since
none of it can destroy something an operator would miss.
```
boot
└─ path configured? no ──▶ skip
└─ tree readable? no ──▶ warn, carry on
└─ hashes changed? no ──▶ done (nothing parsed)
└─ parse
└─ a facet would be removed?
no ──▶ import
yes ──▶ stage for admin review; atlas unchanged
```
`atlas:build` flags:
### Approving or rejecting a staged refresh
| Flag | Default | Meaning |
|---|---|---|
| `--servuo` | *(required)* | ServUO server root — the directory holding `Spawns/`, `Data/`, `Config/` |
| `--out` | `server/db/data` | Where to write the artifact |
| `--landmark-radius` | `200` | Max tile distance for the landmark fallback |
Only the *decision* is stored, never the parsed world — a few KB of source hashes
plus the facet diff. Approving **re-parses** the tree, so what lands matches the
tree at approval time rather than at boot, and a multi-megabyte blob never sits
in the database.
`atlas:import` takes `--dir` (default `server/db/data`).
A rejection is remembered against those exact source hashes, so a declined
refresh does not re-prompt on every restart. Change the tree and the hashes
differ, which asks again.
## Operator re-run story
From **Admin → Spawn Atlas**, or from the CLI:
Spawns changed → rebuild → commit → deploy → import.
```bash
cd website/server
npm run atlas:import -- --status # what is loaded, and what is pending
npm run atlas:import -- --approve # apply the staged refresh
npm run atlas:import -- --reject # keep the current atlas, dismiss it
```
1. `npm run atlas:build -- --servuo <path>` on a machine with the tree.
2. Commit the regenerated `server/db/data/spawnAtlas.*.json`.
3. Deploy.
4. `npm run atlas:import`, or `POST /api/v1/admin/shard/atlas/import`.
## The CLI
`shard_atlas_meta` stores a sha256 per source file, so
`GET /api/v1/admin/shard/atlas/status` reports when the database is behind the
committed artifact. **Build stays CLI-only** — there is no admin button that
reads a ServUO tree.
The server refreshes itself on boot, so this is for applying a map change
*without* a restart, and for the approve/reject flow above.
```bash
npm run atlas:import # import if the tree differs
npm run atlas:import -- --servuo <path> # override the path for this run
npm run atlas:import -- --force # reimport even if unchanged
```
`--servuo` is a per-run override and deliberately does **not** persist — changing
where the atlas permanently reads from is an admin action, not a side effect of a
one-off import.
## Sources
@@ -72,11 +124,10 @@ reads a ServUO tree.
| `Data/Locations/*.xml` | 6 files | Landmarks (dungeon levels, town markers) |
| `Config/ChampionSpawns.xml` | 4.8 KB | Configured champion altars |
**There are 13 spawn files but only 6 facets.** `Eodon.xml`,
**A stock tree has 13 spawn files but only 6 facets.** `Eodon.xml`,
`GravewaterLake.xml`, `TreasuresOfKotl.xml` and the other named-area files hold
TerMur/Trammel points. The facet always comes from each record's own `<Map>`,
never from the file name, and the artifact shards by facet — Felucca, Trammel,
Ilshenar, Malas, Tokuno, TerMur.
never from the file name.
## How a coordinate becomes a place name
@@ -85,16 +136,17 @@ This is the transform the atlas exists for, in `resolveRegion()`:
1. The highest-`priority` named region whose rectangle contains the point. Ties
break toward the **smallest** rect, so a specific room wins over the
dungeon-wide rect enclosing it.
2. Otherwise the nearest landmark within `--landmark-radius` tiles, labelled by
its **group** ("Covetous"), not its individual marker ("Level 1").
2. Otherwise the nearest landmark within the landmark radius (200 tiles by
default), labelled by its **group** ("Covetous"), not its individual marker
("Level 1").
3. Otherwise `"Wilderness"`.
The radius cap in step 2 is what keeps step 3 reachable. Without it the nearest
landmark is always *some* landmark however far away, and open countryside gets
labelled with a dungeon on the far side of the map.
Against stock ServUO this resolves **83.2%** of points (5,369 of 6,455): 3,689 by
region, 1,690 by landmark, 1,086 Wilderness.
Against stock ServUO this resolves **83.2%** of points (5,369 of 6,455): 3,681 by
region, 1,688 by landmark, 1,086 Wilderness.
## Three quirks in the source data
@@ -102,11 +154,17 @@ Each of these is silent if unhandled — the atlas still builds, it is just wron
**Facet names disagree between sources.** `Data/Locations/*.xml` spells them
`Ter Mur` and `Tokuno Islands`, while `<Map>` and `<Facet name>` say `TerMur` and
`Tokuno`. Unreconciled, the landmark bucket for those two facets is keyed
differently from the points looking it up, so the fallback never fires and every
unregioned spawn in Ter Mur and Tokuno reads "Wilderness". All facet names are
canonicalised through `normalizeFacet()`; unknown facets pass through unchanged
so a custom shard facet still gets an atlas.
`Tokuno`. Unreconciled, the landmark bucket is keyed differently from the points
looking it up, so the fallback never fires and every unregioned spawn on those
facets reads "Wilderness".
This is reconciled **by matching, not by a lookup table** — there is no list of
facet names anywhere. `facetKey()` collapses spelling differences (lowercase,
alphanumerics only), and `resolveFacetName()` matches a loose spelling against
the canonical set discovered from the shard's own spawn and region data, by exact
key then by prefix in either direction. A name matching nothing keeps its own
name: forcing a wrong match would file a real custom facet's landmarks under the
wrong facet, which is worse than leaving it alone.
**Spawn type tokens carry XmlSpawner directives.** The `<Objects2>` type is not
always a bare class name:
@@ -125,38 +183,35 @@ real creatures.
file and `lizardman` in another. Slugging collapses them correctly, but the
display name is chosen deterministically — most common spelling wins, ties break
to the more capitalised form, then alphabetically — because otherwise it would
depend on file read order and produce a spurious artifact diff on every unrelated
rebuild.
depend on file read order and change on an unrelated restart.
## Artifact format
## Tables
`server/db/data/`, all committed:
All are **import-owned**: a refresh empties and reloads them in one transaction,
so a failed reload leaves the previous atlas intact rather than a half-loaded
world. Nothing else writes to them and nothing holds a foreign key to them — no
FKs at all, consistent with every other `shard_*` table. Full column listings in
[`BACKEND_DESIGN.md`](BACKEND_DESIGN.md).
| File | Contents |
|---|---|
| `spawnAtlas.meta.json` | Indented. Build time, counts, sha256 per source file |
| `spawnAtlas.index.json` | Compact. Facets, creatures, regions, landmarks, champions |
| `spawnAtlas.<facet>.json` × 6 | Compact. That facet's spawn points |
| Table | Rows (stock) | Notes |
|---|---|---|
| `shard_spawn_creatures` | 800 | `slug` PK; `total` = sum of each type's own max; nullable `art` |
| `shard_spawn_points` | 6,455 | `spawn_range`, since `range` is reserved in MariaDB |
| `shard_spawn_point_types` | 23,927 | The many-to-many; one spawner commonly carries six types |
| `shard_regions` | 387 | Flattened out of the nesting; `rects` JSON |
| `shard_landmarks` | 558 | `grp`, since `group` is reserved in SQL |
| `shard_champion_spawns` | 25 | Configured altars, not the live feed |
| `shard_atlas_meta` | 1 | Singleton; source hashes, for the change check |
| `shard_atlas_pending` | 0–1 | Singleton; a staged refresh awaiting admin review |
**1.41 MB total**, down from 4.40 MB. The design budgeted "well under 1 MB",
which turned out optimistic for 6,455 points; three encodings closed most of the
gap:
`shard_spawn_creatures.name` carries a plain `INDEX`, deliberately **not
`FULLTEXT`**: ~800 rows makes a `LIKE` scan free, and FULLTEXT's minimum token
length would break searches for names like "orc".
- `facet` is dropped per record — the shard file names it once at the top.
- Fields at their default are omitted rather than written as `0`. Most spawners
are a single point with no time-of-day gating, so `width`, `height`, `range`
and the three `tod*` fields are zero on the large majority of records.
- `types` are `[name, max]` tuples. There are ~24,000 type entries and
`{"type":"Orc","max":1}` spends 15 bytes apiece restating two key names that
never vary.
`label` is not stored at all — it is exactly `region || landmark || "Wilderness"`
and the importer recomputes it.
`buildSpawnAtlas.encodePoint()` and `importSpawnAtlas.readPoint()` are exact
inverses, round-tripped in `test/spawnAtlas.build.test.js`. **Change one, change
both.** The artifact never reaches the browser; the browser sees only paginated
API responses.
The reload uses `DELETE`, not `TRUNCATE` — `TRUNCATE` is DDL in MariaDB and would
implicitly commit, defeating the all-or-nothing guarantee. Point ids are assigned
explicitly rather than left to `AUTO_INCREMENT`, because the join rows need them
and `conn.batch()` reports no usable `insertId`.
## Artwork — operator-supplied, never shipped
@@ -175,41 +230,22 @@ An operator who wants art:
2. Drops the images under `server/uploads/atlas/`.
3. Copies `server/db/data/spawnAtlas.art.example.json` to `spawnAtlas.art.json`
and maps creature slugs to file names.
4. Re-runs `npm run atlas:import`.
4. Restarts, or runs `npm run atlas:import -- --force`.
Both `spawnAtlas.art.json` and `server/uploads/` are gitignored, so neither the
map nor the images can be committed by accident.
## Tables
## Code layout
All six are **import-owned**: `atlas:import` empties and reloads them in one
transaction. Nothing else writes to them and nothing holds a foreign key to them
— no FKs at all, consistent with every other `shard_*` table. Full column
listings in [`BACKEND_DESIGN.md`](BACKEND_DESIGN.md).
| File | Role |
|---|---|
| `src/utils/spawnAtlasParse.js` | **Pure and fs-free** parsers, so CI covers them with no ServUO tree. Zero dependencies. |
| `src/utils/spawnAtlasSource.js` | The only thing that reads a ServUO tree; shared by the boot path and the CLI |
| `src/model/shardAtlas/shardAtlas.db.js` | The one-transaction replace |
| `src/model/shardAtlas/shardAtlas.model.js` | The refresh decision, staging, approve/reject |
| `scripts/importSpawnAtlas.js` | Thin CLI over the model |
| Table | Rows (stock) | Notes |
|---|---|---|
| `shard_spawn_creatures` | 800 | `slug` PK; `total` = sum of each type's own max; nullable `art` |
| `shard_spawn_points` | 6,455 | `spawn_range`, since `range` is reserved in MariaDB |
| `shard_spawn_point_types` | 23,927 | The many-to-many; one spawner commonly carries six types |
| `shard_regions` | 387 | Flattened out of the nesting; `rects` JSON |
| `shard_landmarks` | 558 | `grp`, since `group` is reserved in SQL |
| `shard_champion_spawns` | 25 | Configured altars, not the live feed |
| `shard_atlas_meta` | 1 | Singleton (`id = 1`); source hashes for the drift check |
`shard_spawn_creatures.name` carries a plain `INDEX`, deliberately **not
`FULLTEXT`**: ~800 rows makes a `LIKE` scan free, and FULLTEXT's minimum token
length would break searches for names like "orc".
The importer uses `DELETE`, not `TRUNCATE` — `TRUNCATE` is DDL in MariaDB and
would implicitly commit, defeating the all-or-nothing reload. Point ids are
assigned explicitly rather than left to `AUTO_INCREMENT`, because the join rows
need to know them and `conn.batch()` reports no usable `insertId`.
## Parsing notes
`server/src/utils/spawnAtlasParse.js` is **pure and fs-free**, so CI covers it
with no ServUO tree. It adds **zero dependencies**.
Parsing notes:
- `Regions.xml`, `Locations/*.xml` and `ChampionSpawns.xml` genuinely nest, and
get a small hand-rolled **subset** tokenizer — elements, attributes,
@@ -223,3 +259,96 @@ with no ServUO tree. It adds **zero dependencies**.
- `<Objects2>` is `Type:MX=n:SB=…` segments joined by `:OBJ=`. Split on `:OBJ=`
*first* — a naive `split(':')` shreds it. A single Trammel point carries six
types.
- **Respawn delays are stored in two different units, per record.** XmlSpawner
writes `MinDelay`/`MaxDelay` in minutes, and switches to seconds only when a
spawner's delay does not divide into whole minutes — flagging that with
`DelayInSec` on the same record. A `5` therefore means five *minutes* on one
spawner and five *seconds* on the next, and both are plausible respawn times,
so a reader assuming either unit is silently wrong about the other. Stock
ServUO 57.4 has ~170 second-flagged spawners out of 6,455. The parser
normalises everything to **seconds**; the API and UI carry seconds throughout.
### The parser version
`spawnAtlasSource.js` exports `PARSER_VERSION`, stored in `shard_atlas_meta`
alongside the source hashes and bumped whenever the parser derives **different
data from identical files** — a fixed misreading, a new field, a changed unit.
A refresh re-derives when the tree changed **or** the parser did. Hashing the
tree alone would be a trap: an install whose maps never change would keep serving
whatever an older build derived, indefinitely, and a deploy that corrects the
parse would never reach the data. A version mismatch counts as drift, so the
correction lands on the next boot without an operator having to know it happened.
## The API
Everything is served from MariaDB. Nothing on this path touches the sidecar, so
the pages stay complete while the shard is down — which is why the routes sit at
`/api/v1/public/atlas` and **not** under `/public/shard`, where a prefix means
"sidecar-dependent". Unlike `/shard/*`, they *are* `siteMode`-gated, like
`/posts` and `/wiki`: a bestiary is site content and follows site content's rules.
Every route carries `requireFeature('atlas')` — **404** when an admin has
disabled the feature (its pages must not reveal that it exists) and **403** when
the caller sits below its configured audience. The default is `anonymous`, so the
gates are inert until an admin changes something. Responses are field-projected
like every other shard read; `atlas` declares no sensitive fields today, and the
projection call is there so the first one that does is covered by construction
rather than by a retrofit ([`v3.md` §3.6.1](../link/v3.md)).
| Route | Answers |
|---|---|
| `GET /atlas/creatures?q=&facet=&limit=&offset=` | The bestiary, most numerous first, paginated with an unpaginated `total` |
| `GET /atlas/creatures/:slug?facet=&points=` | One creature: `places`, `spawners`, `alsoHere` |
| `GET /atlas/regions?facet=&q=` | Named regions and their rectangles |
| `GET /atlas/landmarks?facet=&q=` | Points of interest, labelled by `group` |
| `GET /atlas/champions?facet=` | The configured altar roster |
| `GET /atlas/meta` | Facets, counts and when the atlas was parsed |
Two shapes worth knowing:
- **`places` is the aggregate the atlas exists for.** "Lizardman → Shrines,
Isamu-Jima, Yew", grouped in SQL rather than by summing 6,455 point rows in
Node. `spawners` is the raw list underneath it, bounded, with
`spawnersTruncated` saying when it was cut.
- **`points` is a COUNT, `spawners` is the LIST.** The two are named apart
deliberately: the same key meaning a number on the search route and an array on
the detail route is the kind of thing a client only discovers in production.
`GET /atlas/meta` reports the **game world only**. The ServUO path, the per-file
hashes and any pending refresh describe the operator's filesystem, and live on
the admin route instead.
A facet is never validated against a list — nothing in the codebase names one.
`?facet=` is length-bounded and matched exactly, so an unknown name returns an
empty result rather than an error. The filter is an `EXISTS` over the points and
deliberately not a JSON path or `JSON_SEARCH` built from caller input: that
function treats `%` and `_` as wildcards, which would make `?facet=%` match
everything.
## The admin panel
**Admin → Spawn Atlas** (`/admin/shard-atlas`, admin-only — it reads a path on
the server's filesystem and replaces every atlas table, which is closer to a
deploy action than to moderation).
| Route | Does |
|---|---|
| `GET /admin/shard/atlas` | Status: path, readable, drift, counts, facets, pending |
| `POST /admin/shard/atlas/import` | Import now; `{ force: true }` ignores the hash gate |
| `POST /admin/shard/atlas/approve` | Apply a staged refresh, facet loss and all |
| `POST /admin/shard/atlas/reject` | Keep the current atlas; remember the decision |
| `PUT /admin/shard/atlas/path` | Point the atlas at a different tree |
Three behaviours that are deliberate:
- **An unreadable tree is a 200, not a 500.** `refresh()` reports outcomes rather
than throwing, because the boot path must never be stopped by a bad tree, and
that contract is preserved at the API. The panel says *"The tree could not be
read: …"*; a 500 would say only that something broke.
- **Setting the path does not import.** Moving the mount and reloading the world
are separate decisions, and an operator fixing a typo should not have a
multi-thousand-row replace happen under them. The response carries fresh status
so the panel can offer the import as the next step.
- **Every action is written to the admin activity log** (`shard.atlas.import` /
`.approve` / `.reject` / `.path`).

60
website/api-route-inventory.json
View File

@@ -257,6 +257,26 @@
"method": "GET",
"path": "/api/v1/admin/shard/accounts"
},
{
"method": "GET",
"path": "/api/v1/admin/shard/atlas"
},
{
"method": "POST",
"path": "/api/v1/admin/shard/atlas/approve"
},
{
"method": "POST",
"path": "/api/v1/admin/shard/atlas/import"
},
{
"method": "PUT",
"path": "/api/v1/admin/shard/atlas/path"
},
{
"method": "POST",
"path": "/api/v1/admin/shard/atlas/reject"
},
{
"method": "GET",
"path": "/api/v1/admin/shard/audit"
@@ -313,6 +333,14 @@
"method": "GET",
"path": "/api/v1/admin/shard/vendors/:account"
},
{
"method": "GET",
"path": "/api/v1/admin/shard/visibility"
},
{
"method": "PUT",
"path": "/api/v1/admin/shard/visibility"
},
{
"method": "PUT",
"path": "/api/v1/admin/site-mode"
@@ -705,6 +733,30 @@
"method": "GET",
"path": "/api/v1/player/shard/vendors/:account"
},
{
"method": "GET",
"path": "/api/v1/public/atlas/champions"
},
{
"method": "GET",
"path": "/api/v1/public/atlas/creatures"
},
{
"method": "GET",
"path": "/api/v1/public/atlas/creatures/:slug"
},
{
"method": "GET",
"path": "/api/v1/public/atlas/landmarks"
},
{
"method": "GET",
"path": "/api/v1/public/atlas/meta"
},
{
"method": "GET",
"path": "/api/v1/public/atlas/regions"
},
{
"method": "POST",
"path": "/api/v1/public/contact"
@@ -737,6 +789,10 @@
"method": "GET",
"path": "/api/v1/public/shard/economy"
},
{
"method": "GET",
"path": "/api/v1/public/shard/features"
},
{
"method": "GET",
"path": "/api/v1/public/shard/feed"
@@ -769,6 +825,10 @@
"method": "GET",
"path": "/api/v1/public/shard/presence"
},
{
"method": "GET",
"path": "/api/v1/public/shard/ruleset"
},
{
"method": "GET",
"path": "/api/v1/public/shard/status"