diff --git a/link/v3.md b/link/v3.md index 79e9939..d23b6a7 100644 --- a/link/v3.md +++ b/link/v3.md @@ -13,7 +13,7 @@ Each part is marked off here as it lands on `edge`. §9 carries the same state p |---|---|---|---| | 1 | **A** — visibility framework + actor-leak fix (§3) | ✅ **Done** | website [#109](https://gitea.whitlocktech.com/RunicGateway/website/pulls/109) + [#110](https://gitea.whitlocktech.com/RunicGateway/website/pulls/110), docs [#64](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/64) + [#65](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/65) | | 2 | **B/1** — `world.ruleset` (§5) | ✅ **Done** | servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66) | -| 3 | **C** — spawn atlas (§6) | 🟡 **Data pipeline done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables); API/client PR next | +| 3 | **C** — spawn atlas (§6) | ✅ **Done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables) + [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113) (API + pages + admin panel), docs [#67](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/67) + [#68](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/68) | | 4 | **B/2** — `points.board` (§7) | ⬜ Not started | — | | 5 | **B/3** — `vendor.listing` (§8) | ⬜ Not started | — | | 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | ⬜ Not started | — | @@ -329,28 +329,33 @@ frame during verification. **No plugin, no sidecar, no `Bridge.cfg` knob, no new kinds.** Not part of the v3 wire change. -> **Status:** data pipeline landed on `edge` — website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) -> (parsers, build/import CLI, tables, artifact). API + client pages are the second website PR. +> **Status:** complete on `edge` — website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) +> (parsers, import CLI, tables) and [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113) +> (the six public routes, the five admin ones, `/site/atlas` + `/site/atlas/:slug`, and the +> Admin → Spawn Atlas panel). > Part C ships as **two** website PRs, not one: the parsing half is where the correctness risk > lives, and burying it under routes and React would have meant reviewing it in a 10k-line diff. > Full operator documentation: [`docs/website/SPAWN_ATLAS.md`](../website/SPAWN_ATLAS.md). > -> §6.1 below records where the shipped implementation differs from this design. The differences are -> all things the real ServUO data forced, not changes of mind. +> **§6 below is the original design and is partly superseded.** §6.1 records two decisions that were +> rejected in review and replaced (the committed artifact, and the fixed facet list); §6.2 records +> the corrections the real ServUO data forced. Read both before trusting §6. -**Decision: committed generated artifact + idempotent DB import**, split in two because the build -needs the ServUO tree (which the website container does not have) and the import does not. Not -runtime import (10.5 MB of XML per boot), not a browser-served blob. +**Decision (revised at implementation time): the shard's ServUO tree is the single source of truth, +re-derived on every server boot.** The original plan here was a committed generated artifact plus an +idempotent import. That was rejected in review for two reasons, recorded in §6.1: a snapshot in the +repo goes stale as a shard's maps change, and the design leaned on a fixed facet list that no shard +is obliged to keep. Still not a browser-served blob; still parsed server-side only. New in `website/server/`: - `src/utils/spawnAtlasParse.js` — **pure functions, no fs**, so they are unit-testable in CI without a ServUO tree: `parseObjects2()`, `parsePoints()`, `parseRegions()`, `parseLocations()`, `resolveRegion()`. -- `scripts/buildSpawnAtlas.js` (`--servuo --out db/data/`) and `scripts/importSpawnAtlas.js` - (TRUNCATE + batched INSERT in one transaction); `package.json` scripts `atlas:build`, `atlas:import`. -- `db/data/spawnAtlas..json` ×13 + `spawnAtlas.index.json` (creatures, champions, regions, - landmarks, meta with per-source-file hashes). +- ~~`scripts/buildSpawnAtlas.js` and a committed `db/data/spawnAtlas.*.json` artifact~~ — dropped, + see §6.1 R1. Replaced by `src/utils/spawnAtlasSource.js` (the only thing that reads a ServUO tree, + shared by the boot path and the CLI) and a `scripts/importSpawnAtlas.js` that is a thin CLI over + the model. `package.json` gains `atlas:import` only. - `src/model/shardAtlas/{shardAtlas.db.js,shardAtlas.model.js}` following the `shardState` split. - `src/router/v1/public/atlas.{router,controller}.js`; `test/spawnAtlas.parse.test.js`. @@ -381,18 +386,48 @@ stays CLI-only.** Client: `routes/public/Atlas.jsx` (`/site/atlas`) and `AtlasCreature.jsx` (`/site/atlas/:slug`). -**Payload risk** — a monolithic artifact would be 2–3 MB of committed JSON. Shard per facet and drop -every `` field the site cannot use (`UniqueId`, all trigger/refractory/proximity/sequential -fields, sound ids), keeping Name/Map/X/Y/W/H/Range/MaxCount/MinDelay/MaxDelay/TOD*/types — well under -1 MB. The artifact never reaches the browser; the browser sees only paginated API responses. +**Payload risk** — *superseded by §6.1 R1; nothing is committed.* The field selection it describes +still applies at parse time: every `` field the site cannot use (`UniqueId`, all +trigger/refractory/proximity/sequential fields, sound ids) is dropped, keeping +Name/Map/X/Y/W/H/Range/MaxCount/MinDelay/MaxDelay/TOD*/types. Parsed data never reaches the browser; +the browser sees only paginated API responses. -**Operator re-run story** — spawns changed → `npm run atlas:build -- --servuo ` on a machine -with the tree → commit the regenerated `db/data/spawnAtlas.*.json` → deploy → `npm run atlas:import` -(or `POST /admin/shard/atlas/import`). `shard_atlas_meta.source` holds per-file hashes, so -`GET /admin/shard/atlas/status` reports when the DB is behind the artifact. Full detail in -`docs/website/SPAWN_ATLAS.md`. +**Operator re-run story** — *revised by §6.1 R1.* Spawns changed → restart, or +`npm run atlas:import` / `POST /admin/shard/atlas/import` to apply without one. `shard_atlas_meta` +holds a sha256 per source file, so the server can tell on boot whether anything changed, and +`GET /admin/shard/atlas/status` reports drift. If the change would remove a facet it is staged for +approval rather than applied (§6.1 R3). Full detail in `docs/website/SPAWN_ATLAS.md`. -### 6.1 What the build against real data changed +### 6.1 What implementation changed + +Two design decisions in §6 were rejected in review and replaced; the rest are corrections the real +ServUO data forced. Kept as a diff rather than edited in place, because each is a trap the next +person would otherwise re-enter. + +**R1. The committed artifact is gone — the tree is re-parsed on every boot.** §6 proposed building a +generated artifact, committing it, and importing it. Two problems. A shard's maps change over its +life, so a snapshot in the repo silently drifts from the world players actually see; and the build/ +import split existed only to work around the website container not having a tree, which is a +deployment question (mount it) rather than a reason to freeze data. The server now hashes the source +files on boot and re-derives the atlas when they differ. `scripts/buildSpawnAtlas.js`, the 1.41 MB +artifact, and the whole encode/decode seam it needed are deleted. + +**R2. Nothing may name a facet.** The first implementation carried a lookup table of the six stock +UO facets to reconcile the spelling drift between sources. A shard may add facets, replace them +outright, or rename them when its maps are updated, and a built-in list mishandles all three +silently. Reconciliation is now by *matching* against the facet set discovered from the shard's own +spawn and region data — exact key, then prefix in either direction — with an unmatched name keeping +its own rather than being forced into a wrong bucket. + +**R3. Two contracts on the boot path.** It never blocks startup: no path, an unreadable mount, a +malformed file or a database error is caught and logged, and the site comes up serving whatever +atlas it had. And a refresh that would REMOVE a facet is never applied automatically — facet loss +is indistinguishable at boot from a half-copied or mid-update tree, so it is staged in +`shard_atlas_pending` for an admin to approve or reject. Only the decision is stored (source hashes ++ the facet diff, a few KB); approving re-parses, so what lands matches the tree at approval time. +A rejection is remembered against those hashes so it does not re-prompt every restart. + +### 6.2 What the build against real data changed Six corrections to the design above, from running it against stock ServUO 57.4. Kept as a diff rather than edited in place, because each one is a trap the next person would otherwise re-enter. @@ -419,12 +454,11 @@ they invent creatures that do not exist *and* split real ones in two, since `Fai leaves **800** real creatures. (The design's "~1,500 creature rows" estimate was high; 800 only reinforces the plain-`INDEX`-not-`FULLTEXT` call.) -**5. The artifact is 1.41 MB, not "well under 1 MB".** Dropping the unused `` fields as the -design directed still left 4.40 MB. Three further encodings — `facet` dropped per record, -default-valued fields omitted rather than written as `0`, and `types` as `[name, max]` tuples -(~24,000 entries × 15 bytes of repeated key names) — brought it to 1.41 MB. Getting under 1 MB would -mean dropping the spawner `name`, which is the only human handle on a specific spawner and worth -keeping. `encodePoint()` and `readPoint()` are exact inverses and are round-tripped in tests. +**5. The artifact would have been 1.41 MB, not "well under 1 MB" — and is now moot.** Dropping the +unused `` fields as the design directed still left 4.40 MB; three further encodings brought +it to 1.41 MB, and getting under 1 MB would have meant dropping the spawner `name`. The size budget +in §6 was simply optimistic for 6,455 points. Superseded by §6.1 R1: there is no artifact, so there +is no payload to budget and no encode/decode seam to keep in sync. **6. `DELETE`, not `TRUNCATE`.** The design said "TRUNCATE + batched INSERT in one transaction", which does not hold: `TRUNCATE` is DDL in MariaDB and implicitly commits, so a mid-import failure @@ -439,6 +473,38 @@ region, 1,690 by landmark, 1,086 Wilderness). **One thing the design got exactly right:** the point-in-rect transform really is the reason to build this. "Where does a lizardman spawn?" answers *Shrines, Isamu-Jima, Yew* across three facets. +### 6.3 What the API/client half added + +The second website PR built the six public routes, the five admin ones, `/site/atlas` + +`/site/atlas/:slug`, and the Admin → Spawn Atlas panel. Three things it changed or established: + +**1. Respawn delays were being read in the wrong unit — sometimes.** XmlSpawner writes +`MinDelay`/`MaxDelay` in minutes and switches to seconds only when a delay does not divide into +whole minutes, flagging that per record with `DelayInSec` +(`XmlSpawner2.cs:7462-7480`, read back at `:6345-6358`). So a `5` means five *minutes* on one +spawner and five *seconds* on the next, both plausible, and the pipeline stored the raw number. +170 of 6,455 stock spawners are second-flagged — few enough to look like noise on a page and be +believed. The parser now normalises to **seconds**, and the API and UI carry seconds throughout. +*This is the class of bug §6.2 is a list of: the atlas still builds, it is just quietly wrong.* + +**2. The hash gate needed a parser version, and this generalises.** Fixing the parse exposed that +"has the tree changed?" is the wrong question on its own — an install whose maps never change would +have kept serving the old readings forever, because the only thing compared was the tree. +`spawnAtlasSource.PARSER_VERSION` is stored in `shard_atlas_meta` beside the source hashes, and a +mismatch counts as drift. Any future parse correction lands on the next boot without an operator +having to know it happened. **Bump it whenever the parser derives different data from identical +files.** + +**3. `points` is a count; `spawners` is the list.** The first cut of the detail route spread the +creature row and then set `points` to the array of spawn points — the same key meaning a number on +the search route and an array on the detail route. Renamed before it shipped, and worth recording +because the two names are one letter apart in meaning and it reads as correct. + +**On projection.** The `atlas` feature declares no sensitive fields, so `projectFeature` is a no-op +on every one of these routes today. Every handler calls it anyway, per §3.6.1's rule — the point of +the rule is that the *first* field that needs gating is covered by construction rather than by a +retrofit nobody remembers to do. + --- ## 7. Part B/2 — `points.board` @@ -633,7 +699,7 @@ inherently up to one full cycle old, and the UI must say so. |---|---|---|---|---| | 1 | **A** — visibility framework + actor-leak fix | website, docs | none | ✅ Done | | 2 | **B/1** — `world.ruleset` (§5) | all four | new kind | ✅ Done | -| 3 | **C** — spawn atlas (§6) | website, docs | none | 🟡 Pipeline done, API/client next | +| 3 | **C** — spawn atlas (§6) | website, docs | none | ✅ Done | | 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | ⬜ | | 5 | **B/3** — `vendor.listing` (§8) | all four | new kinds | ⬜ | | 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | ⬜ | diff --git a/website/BACKEND_DESIGN.md b/website/BACKEND_DESIGN.md index cdffd8c..fd26513 100644 --- a/website/BACKEND_DESIGN.md +++ b/website/BACKEND_DESIGN.md @@ -101,6 +101,10 @@ server/ deliberately not site-mode gated shard.router.js (14) /public/shard/* incl. the anonymous SSE stream; never site-mode gated + atlas.router.js (6) /public/atlas/* — the spawn atlas. + NOT under /shard: nothing here + touches the sidecar, and unlike + /shard it IS site-mode gated site.router.js (4) /settings /status /version /contact — the group-root singletons; declares no router-level middleware @@ -391,9 +395,15 @@ discarded. See §6.5. ### shard_spawn_* / shard_regions / shard_landmarks / shard_champion_spawns / shard_atlas_meta — the spawn atlas (Protocol 3.0) Static shard **content**, not live shard state. Nothing here comes from the sidecar: the atlas is -built from a ServUO tree by a CLI script, committed as JSON under `server/db/data/`, and loaded with -`npm run atlas:import`. These tables stay populated whether the shard is up or not. Full operator -detail in [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md); the design is `docs/link/v3.md` §6. +derived from the shard's own ServUO tree, re-read on **every server boot** and hash-gated so an +unchanged tree costs one read pass and no write. Nothing is precomputed and committed — a shard's +maps change over its life, and a snapshot in the repo would silently drift from the world players +actually see. These tables stay populated whether the shard is up or not. Full operator detail in +[`SPAWN_ATLAS.md`](SPAWN_ATLAS.md); the design is `docs/link/v3.md` §6. + +**No facet name appears anywhere in the code.** A shard may add facets, replace them, or rename them +when its maps are updated; the facet set is discovered from the tree, and the loose spellings in +`Data/Locations` are matched against it rather than looked up in a table. | Table | Key columns | |---|---| @@ -403,13 +413,32 @@ detail in [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md); the design is `docs/link/v3.md` § | `shard_regions` | `facet`, `name`, `type`, `priority`, `parent`, `rects` JSON | | `shard_landmarks` | `facet`, `name`, `grp`, `x`, `y`, `z` | | `shard_champion_spawns` | `slug` PK, `name`, `grp`, `type`, `random_type`, `facet`, `x`, `y`, `z`, `radius`, `label` | -| `shard_atlas_meta` | Singleton (`id = 1`), `payload` JSON, `imported_at` | +| `shard_atlas_meta` | Singleton (`id = 1`), `payload` JSON (counts, a sha256 per source file, `parserVersion`), `imported_at` | +| `shard_atlas_pending` | Singleton (`id = 1`), `status` (`pending`/`rejected`), `payload` JSON, `detected_at` | -All of them are **import-owned**: `atlas:import` empties and reloads every one inside a single -transaction, so a failed import leaves the previous atlas intact rather than a half-loaded world. +The first seven are **import-owned**: a refresh empties and reloads every one inside a single +transaction, so a failed reload leaves the previous atlas intact rather than a half-loaded world. Nothing else writes to them, and nothing holds a foreign key to them — no FKs at all, consistent with every other `shard_*` table. +**`shard_atlas_pending` is the security-relevant one.** A refresh that would REMOVE a facet is never +applied automatically: facet loss is indistinguishable at boot from a half-copied or mid-update tree, +so it is staged here for an admin to approve or reject, and **startup is never blocked by it**. Only +the decision is stored — source hashes plus the facet diff, a few KB — and approving re-parses the +tree, so a multi-megabyte blob never lands in the database and what gets applied matches the tree at +approval time. A rejection is remembered against those exact hashes so a declined refresh does not +re-prompt on every restart. Everything else (new facets, renamed regions, changed spawns) applies +immediately, since none of it can destroy data an operator would miss. + +The boot refresh is **best-effort by contract**: no configured path, an unreadable mount, a malformed +file or a database error is caught and logged, and the site comes up serving whatever atlas it had. +The tree path comes from the `spawn_atlas_servuo_path` setting, falling back to `SERVUO_PATH`. + +**A refresh re-derives when the tree changed OR the parser did.** `spawnAtlasSource.PARSER_VERSION` +is stored in `shard_atlas_meta` beside the source hashes and bumped whenever the parser produces +different data from identical files. Hashing the tree alone would strand an install whose maps never +change on whatever an older build derived — a corrected parse would ship and never reach the data. + Four column choices worth stating, because each one is a trap: - **`spawn_range`, not `range`**, and **`grp`, not `group`** — both are reserved words. @@ -443,7 +472,7 @@ are authoritative, and they answer different questions: | Artifact | Source of truth for | Generated by | |---|---|---| -| `server/routes.manifest.json` — mirrored as [api-route-inventory.json](./api-route-inventory.json) | **What URLs exist.** 200 public routes + 2 on the internal listener, sorted, method + path only. | `npm run routes:manifest`, by walking the live Express stack | +| `server/routes.manifest.json` — mirrored as [api-route-inventory.json](./api-route-inventory.json) | **What URLs exist.** 215 public routes + 2 on the internal listener, sorted, method + path only. | `npm run routes:manifest`, by walking the live Express stack | | `server/swagger/swagger-output.json` — served at `/api/docs` | **What each route means.** Parameters, bodies, response codes, security. | `npm run swagger`, from `#swagger.*` annotations | The split is deliberate: Swagger is annotation-derived, so an unannotated route is invisible in it and @@ -629,6 +658,12 @@ from the per-route **siteMode** middleware (§5), never from an auth gate. | POST | `/contact` | (rate-limited) send mail via SMTP; if unconfigured, respond `{fallback:"mailto", email}` | | GET | `/shard/ruleset` | the shard's own published ruleset (Protocol 3.0 `world.ruleset`): expansion, which optional systems are on, skill/stat caps, account and house limits, champion scroll rules, the save/restart schedule. Served from `shard_ruleset`, so it renders while the shard is down; live via `world.ruleset` on `/shard/stream`. Behind `requireFeature('ruleset')`. **`null`** means the shard has never published one — a real answer, distinct from a published ruleset. `caps.skill` / `caps.totalSkill` are in **tenths** (1000 = 100.0). | | GET | `/shard/features` | the shard features **this caller** may reach plus the audience rung they resolved to (§6.5), so a client hides nav it can't follow. Reports only what the caller can see — the list itself never discloses a gated feature. Consumed by the SPA header and (pending) the Android nav. | +| GET | `/atlas/creatures?q=&facet=&limit=&offset=` | the bestiary, most numerous first, with an unpaginated `total`. Static content parsed from the shard's ServUO tree — **not** sidecar-backed, which is why the atlas sits outside `/shard`, and unlike `/shard/*` it **is** site-mode gated. Behind `requireFeature('atlas')`. `?facet=` is matched exactly and never validated against a list (no facet name exists in the code); the filter is an `EXISTS` over the points rather than a JSON path or `JSON_SEARCH` built from caller input, whose `%`/`_` wildcards would make `?facet=%` match everything. | +| GET | `/atlas/creatures/:slug` | one creature: `places` (the point-in-rect aggregate — "lizardman → Shrines, Isamu-Jima, Yew"), `spawners` (the bounded raw list, with `spawnersTruncated`), `alsoHere`. **`points` is a COUNT and `spawners` is the LIST** — named apart so one key never means a number on one route and an array on another. `minDelay`/`maxDelay` are in **seconds**, normalised at parse time from the source's per-record minutes-or-seconds. 404 = no such creature in this atlas. | +| GET | `/atlas/regions?facet=&q=` | named regions and the rectangles that placed each spawner | +| GET | `/atlas/landmarks?facet=&q=` | points of interest, labelled by `group` ("Covetous", not "Level 1") | +| GET | `/atlas/champions?facet=` | the **configured** altar roster. Not `/shard/champs`, which is the live board. | +| GET | `/atlas/meta` | facets, counts and when the atlas was parsed. Game-world facts only — the ServUO path, source hashes and any pending refresh are operator detail and live on the admin route. | Public content GETs pass through the **siteMode** gate (§5). @@ -672,6 +707,10 @@ file a route sits in — that is the property the route manifest freezes. | GET | `/users/:id/trusted-devices` | list a user's active trusted devices (never tokens) | | DELETE | `/users/:id/trusted-devices` · `…/:deviceId` | revoke all / one of a user's trusted devices (logs `admin.trusted_device.revoke[_all]`) | | POST | `/users/:id/mfa/reset` | recover a locked-out user: disable TOTP + revoke all trusted devices + clear recovery codes (logs `admin.user.totp.reset`) | +| GET | `/shard/atlas` | spawn-atlas status (`adminOnly`): the ServUO path, whether the tree is readable, whether it has drifted from what is loaded, counts, facets, and any refresh staged for review. The public `/atlas/meta` reports the game world only; the filesystem detail is here. | +| POST | `/shard/atlas/import` | re-import without restarting; `{force}` ignores the hash gate. **An unreadable tree answers 200 with `status:"unavailable"`, not 500** — `refresh()` reports outcomes rather than throwing (the boot path must never be blocked by a bad tree) and that contract is preserved at the API. | +| POST | `/shard/atlas/approve` · `/shard/atlas/reject` | answer a refresh staged because it would REMOVE a facet. Approving **re-parses** the tree, so what lands matches it at approval time; rejecting is remembered against those source hashes so it does not re-prompt every restart. 404 when nothing is staged. | +| PUT | `/shard/atlas/path` | point the atlas at a different tree (persisted as `spawn_atlas_servuo_path`, which wins over `SERVUO_PATH`). Blank clears it. Deliberately **does not import** — moving the mount and reloading the world are separate decisions — and returns fresh status so the panel can offer the import next. | Every admin write logs to `activity_log`. diff --git a/website/SPAWN_ATLAS.md b/website/SPAWN_ATLAS.md index 57e735e..f74d251 100644 --- a/website/SPAWN_ATLAS.md +++ b/website/SPAWN_ATLAS.md @@ -1,6 +1,6 @@ # Spawn atlas -**Status:** Data pipeline landed on `edge` (website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112)); API and client pages follow in a second PR. +**Status:** Complete on `edge` — data pipeline in website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112), API + pages in website [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113). **Design:** [`docs/link/v3.md` §6](../link/v3.md) — Protocol 3.0 Part C. The spawn atlas is a browsable catalogue of what the shard *contains*: which @@ -8,6 +8,19 @@ creatures spawn, where, how many, and which champion altars are configured. It answers "where do I find a lizardman?" with **"Shrines, Yew, Isamu-Jima"** rather than with a list of raw coordinates. +## Two things that shape the whole design + +**The shard's ServUO tree is the single source of truth.** Nothing is +precomputed and committed to the repository. A shard's maps change over its +lifetime — facets get added, replaced, or renamed — and a snapshot in the repo +would silently drift from the world players actually see. The atlas is therefore +re-derived from the tree **on every server boot**. + +**Facets are not a fixed list.** Nothing in the codebase names Felucca, Trammel, +or any other stock facet. The facet set is whatever the shard's own files +declare, discovered at parse time. A shard running entirely custom maps gets +exactly the same treatment as a stock one, with no code change. + ## What it is not The atlas is **static shard content, not live shard state.** @@ -24,44 +37,83 @@ The atlas is **static shard content, not live shard state.** Routes live at `/api/v1/public/atlas`, deliberately **not** under `/shard`, because `/shard/*` means sidecar-dependent. -## The two commands +## Configuring the tree -Building needs a ServUO tree. Importing does not. That split is the whole design: -the website container ships the committed artifact but has no ServUO tree, so it -can import but never build. +The website needs to be able to *read* the ServUO tree — same host, a bind mount, +or a shared volume. Two ways to point at it, the setting winning over the +environment: -```bash -# On a machine that has the ServUO tree (writes server/db/data/spawnAtlas.*.json) -cd website/server -npm run atlas:build -- --servuo /path/to/ServUO +| Source | Notes | +|---|---| +| `spawn_atlas_servuo_path` setting | Admin-editable; changes take effect on the next refresh without a redeploy | +| `SERVUO_PATH` env var | The deploy-time default, since the path usually describes a mount the deployment sets up | -# Anywhere, including the deployed container -npm run atlas:import +With neither set the atlas is simply skipped — the site runs normally without +one. + +## The boot path + +On every start the server hashes the source files and compares them against what +is loaded. Unchanged (the normal case on a restart) costs one read pass, ~120 ms, +and no database write. A real change costs a ~400 ms parse and a reload. + +Two contracts govern it: + +**1. It never blocks startup.** No configured path, an unreadable mount, a +malformed file, a database error — every one is caught and logged, and the site +comes up serving whatever atlas it already had. + +**2. A facet disappearing is never applied automatically.** Losing a facet looks +exactly like a half-copied or mid-update tree, and boot cannot tell that apart +from a real map change. That refresh is *staged* for a human instead. Everything +else — new facets, renamed regions, changed spawns — applies immediately, since +none of it can destroy something an operator would miss. + +``` +boot + └─ path configured? no ──▶ skip + └─ tree readable? no ──▶ warn, carry on + └─ hashes changed? no ──▶ done (nothing parsed) + └─ parse + └─ a facet would be removed? + no ──▶ import + yes ──▶ stage for admin review; atlas unchanged ``` -`atlas:build` flags: +### Approving or rejecting a staged refresh -| Flag | Default | Meaning | -|---|---|---| -| `--servuo` | *(required)* | ServUO server root — the directory holding `Spawns/`, `Data/`, `Config/` | -| `--out` | `server/db/data` | Where to write the artifact | -| `--landmark-radius` | `200` | Max tile distance for the landmark fallback | +Only the *decision* is stored, never the parsed world — a few KB of source hashes +plus the facet diff. Approving **re-parses** the tree, so what lands matches the +tree at approval time rather than at boot, and a multi-megabyte blob never sits +in the database. -`atlas:import` takes `--dir` (default `server/db/data`). +A rejection is remembered against those exact source hashes, so a declined +refresh does not re-prompt on every restart. Change the tree and the hashes +differ, which asks again. -## Operator re-run story +From **Admin → Spawn Atlas**, or from the CLI: -Spawns changed → rebuild → commit → deploy → import. +```bash +cd website/server +npm run atlas:import -- --status # what is loaded, and what is pending +npm run atlas:import -- --approve # apply the staged refresh +npm run atlas:import -- --reject # keep the current atlas, dismiss it +``` -1. `npm run atlas:build -- --servuo ` on a machine with the tree. -2. Commit the regenerated `server/db/data/spawnAtlas.*.json`. -3. Deploy. -4. `npm run atlas:import`, or `POST /api/v1/admin/shard/atlas/import`. +## The CLI -`shard_atlas_meta` stores a sha256 per source file, so -`GET /api/v1/admin/shard/atlas/status` reports when the database is behind the -committed artifact. **Build stays CLI-only** — there is no admin button that -reads a ServUO tree. +The server refreshes itself on boot, so this is for applying a map change +*without* a restart, and for the approve/reject flow above. + +```bash +npm run atlas:import # import if the tree differs +npm run atlas:import -- --servuo # override the path for this run +npm run atlas:import -- --force # reimport even if unchanged +``` + +`--servuo` is a per-run override and deliberately does **not** persist — changing +where the atlas permanently reads from is an admin action, not a side effect of a +one-off import. ## Sources @@ -72,11 +124,10 @@ reads a ServUO tree. | `Data/Locations/*.xml` | 6 files | Landmarks (dungeon levels, town markers) | | `Config/ChampionSpawns.xml` | 4.8 KB | Configured champion altars | -**There are 13 spawn files but only 6 facets.** `Eodon.xml`, +**A stock tree has 13 spawn files but only 6 facets.** `Eodon.xml`, `GravewaterLake.xml`, `TreasuresOfKotl.xml` and the other named-area files hold TerMur/Trammel points. The facet always comes from each record's own ``, -never from the file name, and the artifact shards by facet — Felucca, Trammel, -Ilshenar, Malas, Tokuno, TerMur. +never from the file name. ## How a coordinate becomes a place name @@ -85,16 +136,17 @@ This is the transform the atlas exists for, in `resolveRegion()`: 1. The highest-`priority` named region whose rectangle contains the point. Ties break toward the **smallest** rect, so a specific room wins over the dungeon-wide rect enclosing it. -2. Otherwise the nearest landmark within `--landmark-radius` tiles, labelled by - its **group** ("Covetous"), not its individual marker ("Level 1"). +2. Otherwise the nearest landmark within the landmark radius (200 tiles by + default), labelled by its **group** ("Covetous"), not its individual marker + ("Level 1"). 3. Otherwise `"Wilderness"`. The radius cap in step 2 is what keeps step 3 reachable. Without it the nearest landmark is always *some* landmark however far away, and open countryside gets labelled with a dungeon on the far side of the map. -Against stock ServUO this resolves **83.2%** of points (5,369 of 6,455): 3,689 by -region, 1,690 by landmark, 1,086 Wilderness. +Against stock ServUO this resolves **83.2%** of points (5,369 of 6,455): 3,681 by +region, 1,688 by landmark, 1,086 Wilderness. ## Three quirks in the source data @@ -102,11 +154,17 @@ Each of these is silent if unhandled — the atlas still builds, it is just wron **Facet names disagree between sources.** `Data/Locations/*.xml` spells them `Ter Mur` and `Tokuno Islands`, while `` and `` say `TerMur` and -`Tokuno`. Unreconciled, the landmark bucket for those two facets is keyed -differently from the points looking it up, so the fallback never fires and every -unregioned spawn in Ter Mur and Tokuno reads "Wilderness". All facet names are -canonicalised through `normalizeFacet()`; unknown facets pass through unchanged -so a custom shard facet still gets an atlas. +`Tokuno`. Unreconciled, the landmark bucket is keyed differently from the points +looking it up, so the fallback never fires and every unregioned spawn on those +facets reads "Wilderness". + +This is reconciled **by matching, not by a lookup table** — there is no list of +facet names anywhere. `facetKey()` collapses spelling differences (lowercase, +alphanumerics only), and `resolveFacetName()` matches a loose spelling against +the canonical set discovered from the shard's own spawn and region data, by exact +key then by prefix in either direction. A name matching nothing keeps its own +name: forcing a wrong match would file a real custom facet's landmarks under the +wrong facet, which is worse than leaving it alone. **Spawn type tokens carry XmlSpawner directives.** The `` type is not always a bare class name: @@ -125,38 +183,35 @@ real creatures. file and `lizardman` in another. Slugging collapses them correctly, but the display name is chosen deterministically — most common spelling wins, ties break to the more capitalised form, then alphabetically — because otherwise it would -depend on file read order and produce a spurious artifact diff on every unrelated -rebuild. +depend on file read order and change on an unrelated restart. -## Artifact format +## Tables -`server/db/data/`, all committed: +All are **import-owned**: a refresh empties and reloads them in one transaction, +so a failed reload leaves the previous atlas intact rather than a half-loaded +world. Nothing else writes to them and nothing holds a foreign key to them — no +FKs at all, consistent with every other `shard_*` table. Full column listings in +[`BACKEND_DESIGN.md`](BACKEND_DESIGN.md). -| File | Contents | -|---|---| -| `spawnAtlas.meta.json` | Indented. Build time, counts, sha256 per source file | -| `spawnAtlas.index.json` | Compact. Facets, creatures, regions, landmarks, champions | -| `spawnAtlas..json` × 6 | Compact. That facet's spawn points | +| Table | Rows (stock) | Notes | +|---|---|---| +| `shard_spawn_creatures` | 800 | `slug` PK; `total` = sum of each type's own max; nullable `art` | +| `shard_spawn_points` | 6,455 | `spawn_range`, since `range` is reserved in MariaDB | +| `shard_spawn_point_types` | 23,927 | The many-to-many; one spawner commonly carries six types | +| `shard_regions` | 387 | Flattened out of the nesting; `rects` JSON | +| `shard_landmarks` | 558 | `grp`, since `group` is reserved in SQL | +| `shard_champion_spawns` | 25 | Configured altars, not the live feed | +| `shard_atlas_meta` | 1 | Singleton; source hashes, for the change check | +| `shard_atlas_pending` | 0–1 | Singleton; a staged refresh awaiting admin review | -**1.41 MB total**, down from 4.40 MB. The design budgeted "well under 1 MB", -which turned out optimistic for 6,455 points; three encodings closed most of the -gap: +`shard_spawn_creatures.name` carries a plain `INDEX`, deliberately **not +`FULLTEXT`**: ~800 rows makes a `LIKE` scan free, and FULLTEXT's minimum token +length would break searches for names like "orc". -- `facet` is dropped per record — the shard file names it once at the top. -- Fields at their default are omitted rather than written as `0`. Most spawners - are a single point with no time-of-day gating, so `width`, `height`, `range` - and the three `tod*` fields are zero on the large majority of records. -- `types` are `[name, max]` tuples. There are ~24,000 type entries and - `{"type":"Orc","max":1}` spends 15 bytes apiece restating two key names that - never vary. - -`label` is not stored at all — it is exactly `region || landmark || "Wilderness"` -and the importer recomputes it. - -`buildSpawnAtlas.encodePoint()` and `importSpawnAtlas.readPoint()` are exact -inverses, round-tripped in `test/spawnAtlas.build.test.js`. **Change one, change -both.** The artifact never reaches the browser; the browser sees only paginated -API responses. +The reload uses `DELETE`, not `TRUNCATE` — `TRUNCATE` is DDL in MariaDB and would +implicitly commit, defeating the all-or-nothing guarantee. Point ids are assigned +explicitly rather than left to `AUTO_INCREMENT`, because the join rows need them +and `conn.batch()` reports no usable `insertId`. ## Artwork — operator-supplied, never shipped @@ -175,41 +230,22 @@ An operator who wants art: 2. Drops the images under `server/uploads/atlas/`. 3. Copies `server/db/data/spawnAtlas.art.example.json` to `spawnAtlas.art.json` and maps creature slugs to file names. -4. Re-runs `npm run atlas:import`. +4. Restarts, or runs `npm run atlas:import -- --force`. Both `spawnAtlas.art.json` and `server/uploads/` are gitignored, so neither the map nor the images can be committed by accident. -## Tables +## Code layout -All six are **import-owned**: `atlas:import` empties and reloads them in one -transaction. Nothing else writes to them and nothing holds a foreign key to them -— no FKs at all, consistent with every other `shard_*` table. Full column -listings in [`BACKEND_DESIGN.md`](BACKEND_DESIGN.md). +| File | Role | +|---|---| +| `src/utils/spawnAtlasParse.js` | **Pure and fs-free** parsers, so CI covers them with no ServUO tree. Zero dependencies. | +| `src/utils/spawnAtlasSource.js` | The only thing that reads a ServUO tree; shared by the boot path and the CLI | +| `src/model/shardAtlas/shardAtlas.db.js` | The one-transaction replace | +| `src/model/shardAtlas/shardAtlas.model.js` | The refresh decision, staging, approve/reject | +| `scripts/importSpawnAtlas.js` | Thin CLI over the model | -| Table | Rows (stock) | Notes | -|---|---|---| -| `shard_spawn_creatures` | 800 | `slug` PK; `total` = sum of each type's own max; nullable `art` | -| `shard_spawn_points` | 6,455 | `spawn_range`, since `range` is reserved in MariaDB | -| `shard_spawn_point_types` | 23,927 | The many-to-many; one spawner commonly carries six types | -| `shard_regions` | 387 | Flattened out of the nesting; `rects` JSON | -| `shard_landmarks` | 558 | `grp`, since `group` is reserved in SQL | -| `shard_champion_spawns` | 25 | Configured altars, not the live feed | -| `shard_atlas_meta` | 1 | Singleton (`id = 1`); source hashes for the drift check | - -`shard_spawn_creatures.name` carries a plain `INDEX`, deliberately **not -`FULLTEXT`**: ~800 rows makes a `LIKE` scan free, and FULLTEXT's minimum token -length would break searches for names like "orc". - -The importer uses `DELETE`, not `TRUNCATE` — `TRUNCATE` is DDL in MariaDB and -would implicitly commit, defeating the all-or-nothing reload. Point ids are -assigned explicitly rather than left to `AUTO_INCREMENT`, because the join rows -need to know them and `conn.batch()` reports no usable `insertId`. - -## Parsing notes - -`server/src/utils/spawnAtlasParse.js` is **pure and fs-free**, so CI covers it -with no ServUO tree. It adds **zero dependencies**. +Parsing notes: - `Regions.xml`, `Locations/*.xml` and `ChampionSpawns.xml` genuinely nest, and get a small hand-rolled **subset** tokenizer — elements, attributes, @@ -223,3 +259,96 @@ with no ServUO tree. It adds **zero dependencies**. - `` is `Type:MX=n:SB=…` segments joined by `:OBJ=`. Split on `:OBJ=` *first* — a naive `split(':')` shreds it. A single Trammel point carries six types. +- **Respawn delays are stored in two different units, per record.** XmlSpawner + writes `MinDelay`/`MaxDelay` in minutes, and switches to seconds only when a + spawner's delay does not divide into whole minutes — flagging that with + `DelayInSec` on the same record. A `5` therefore means five *minutes* on one + spawner and five *seconds* on the next, and both are plausible respawn times, + so a reader assuming either unit is silently wrong about the other. Stock + ServUO 57.4 has ~170 second-flagged spawners out of 6,455. The parser + normalises everything to **seconds**; the API and UI carry seconds throughout. + +### The parser version + +`spawnAtlasSource.js` exports `PARSER_VERSION`, stored in `shard_atlas_meta` +alongside the source hashes and bumped whenever the parser derives **different +data from identical files** — a fixed misreading, a new field, a changed unit. + +A refresh re-derives when the tree changed **or** the parser did. Hashing the +tree alone would be a trap: an install whose maps never change would keep serving +whatever an older build derived, indefinitely, and a deploy that corrects the +parse would never reach the data. A version mismatch counts as drift, so the +correction lands on the next boot without an operator having to know it happened. + +## The API + +Everything is served from MariaDB. Nothing on this path touches the sidecar, so +the pages stay complete while the shard is down — which is why the routes sit at +`/api/v1/public/atlas` and **not** under `/public/shard`, where a prefix means +"sidecar-dependent". Unlike `/shard/*`, they *are* `siteMode`-gated, like +`/posts` and `/wiki`: a bestiary is site content and follows site content's rules. + +Every route carries `requireFeature('atlas')` — **404** when an admin has +disabled the feature (its pages must not reveal that it exists) and **403** when +the caller sits below its configured audience. The default is `anonymous`, so the +gates are inert until an admin changes something. Responses are field-projected +like every other shard read; `atlas` declares no sensitive fields today, and the +projection call is there so the first one that does is covered by construction +rather than by a retrofit ([`v3.md` §3.6.1](../link/v3.md)). + +| Route | Answers | +|---|---| +| `GET /atlas/creatures?q=&facet=&limit=&offset=` | The bestiary, most numerous first, paginated with an unpaginated `total` | +| `GET /atlas/creatures/:slug?facet=&points=` | One creature: `places`, `spawners`, `alsoHere` | +| `GET /atlas/regions?facet=&q=` | Named regions and their rectangles | +| `GET /atlas/landmarks?facet=&q=` | Points of interest, labelled by `group` | +| `GET /atlas/champions?facet=` | The configured altar roster | +| `GET /atlas/meta` | Facets, counts and when the atlas was parsed | + +Two shapes worth knowing: + +- **`places` is the aggregate the atlas exists for.** "Lizardman → Shrines, + Isamu-Jima, Yew", grouped in SQL rather than by summing 6,455 point rows in + Node. `spawners` is the raw list underneath it, bounded, with + `spawnersTruncated` saying when it was cut. +- **`points` is a COUNT, `spawners` is the LIST.** The two are named apart + deliberately: the same key meaning a number on the search route and an array on + the detail route is the kind of thing a client only discovers in production. + +`GET /atlas/meta` reports the **game world only**. The ServUO path, the per-file +hashes and any pending refresh describe the operator's filesystem, and live on +the admin route instead. + +A facet is never validated against a list — nothing in the codebase names one. +`?facet=` is length-bounded and matched exactly, so an unknown name returns an +empty result rather than an error. The filter is an `EXISTS` over the points and +deliberately not a JSON path or `JSON_SEARCH` built from caller input: that +function treats `%` and `_` as wildcards, which would make `?facet=%` match +everything. + +## The admin panel + +**Admin → Spawn Atlas** (`/admin/shard-atlas`, admin-only — it reads a path on +the server's filesystem and replaces every atlas table, which is closer to a +deploy action than to moderation). + +| Route | Does | +|---|---| +| `GET /admin/shard/atlas` | Status: path, readable, drift, counts, facets, pending | +| `POST /admin/shard/atlas/import` | Import now; `{ force: true }` ignores the hash gate | +| `POST /admin/shard/atlas/approve` | Apply a staged refresh, facet loss and all | +| `POST /admin/shard/atlas/reject` | Keep the current atlas; remember the decision | +| `PUT /admin/shard/atlas/path` | Point the atlas at a different tree | + +Three behaviours that are deliberate: + +- **An unreadable tree is a 200, not a 500.** `refresh()` reports outcomes rather + than throwing, because the boot path must never be stopped by a bad tree, and + that contract is preserved at the API. The panel says *"The tree could not be + read: …"*; a 500 would say only that something broke. +- **Setting the path does not import.** Moving the mount and reloading the world + are separate decisions, and an operator fixing a typo should not have a + multi-thousand-row replace happen under them. The response carries fresh status + so the panel can offer the import as the next step. +- **Every action is written to the admin activity log** (`shard.atlas.import` / + `.approve` / `.reject` / `.path`). diff --git a/website/api-route-inventory.json b/website/api-route-inventory.json index 63a0207..f42fcb4 100644 --- a/website/api-route-inventory.json +++ b/website/api-route-inventory.json @@ -257,6 +257,26 @@ "method": "GET", "path": "/api/v1/admin/shard/accounts" }, + { + "method": "GET", + "path": "/api/v1/admin/shard/atlas" + }, + { + "method": "POST", + "path": "/api/v1/admin/shard/atlas/approve" + }, + { + "method": "POST", + "path": "/api/v1/admin/shard/atlas/import" + }, + { + "method": "PUT", + "path": "/api/v1/admin/shard/atlas/path" + }, + { + "method": "POST", + "path": "/api/v1/admin/shard/atlas/reject" + }, { "method": "GET", "path": "/api/v1/admin/shard/audit" @@ -313,6 +333,14 @@ "method": "GET", "path": "/api/v1/admin/shard/vendors/:account" }, + { + "method": "GET", + "path": "/api/v1/admin/shard/visibility" + }, + { + "method": "PUT", + "path": "/api/v1/admin/shard/visibility" + }, { "method": "PUT", "path": "/api/v1/admin/site-mode" @@ -705,6 +733,30 @@ "method": "GET", "path": "/api/v1/player/shard/vendors/:account" }, + { + "method": "GET", + "path": "/api/v1/public/atlas/champions" + }, + { + "method": "GET", + "path": "/api/v1/public/atlas/creatures" + }, + { + "method": "GET", + "path": "/api/v1/public/atlas/creatures/:slug" + }, + { + "method": "GET", + "path": "/api/v1/public/atlas/landmarks" + }, + { + "method": "GET", + "path": "/api/v1/public/atlas/meta" + }, + { + "method": "GET", + "path": "/api/v1/public/atlas/regions" + }, { "method": "POST", "path": "/api/v1/public/contact" @@ -737,6 +789,10 @@ "method": "GET", "path": "/api/v1/public/shard/economy" }, + { + "method": "GET", + "path": "/api/v1/public/shard/features" + }, { "method": "GET", "path": "/api/v1/public/shard/feed" @@ -769,6 +825,10 @@ "method": "GET", "path": "/api/v1/public/shard/presence" }, + { + "method": "GET", + "path": "/api/v1/public/shard/ruleset" + }, { "method": "GET", "path": "/api/v1/public/shard/status"