Merge pull request 'docs(website): record the spawn atlas pipeline and what real data changed' (#67) from docs/spawn-atlas into edge

Reviewed-on: #67
Reviewed-by: Colby Whitlock <whitlocktech@gmail.com>

link/v3.md

@@ -13,7 +13,7 @@ Each part is marked off here as it lands on `edge`. §9 carries the same state p
 |---|---|---|---|
 | 1 | **A** — visibility framework + actor-leak fix (§3) | ✅ **Done** | website [#109](https://gitea.whitlocktech.com/RunicGateway/website/pulls/109) + [#110](https://gitea.whitlocktech.com/RunicGateway/website/pulls/110), docs [#64](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/64) + [#65](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/65) |
 | 2 | **B/1** — `world.ruleset` (§5) | ✅ **Done** | servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66) |
-| 3 | **C** — spawn atlas (§6) | ⬜ **Next** | — |
+| 3 | **C** — spawn atlas (§6) | 🟡 **Data pipeline done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables); API/client PR next |
 | 4 | **B/2** — `points.board` (§7) | ⬜ Not started | — |
 | 5 | **B/3** — `vendor.listing` (§8) | ⬜ Not started | — |
 | 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | ⬜ Not started | — |
@@ -329,6 +329,15 @@ frame during verification.
 
 **No plugin, no sidecar, no `Bridge.cfg` knob, no new kinds.** Not part of the v3 wire change.
 
+> **Status:** data pipeline landed on `edge` — website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112)
+> (parsers, build/import CLI, tables, artifact). API + client pages are the second website PR.
+> Part C ships as **two** website PRs, not one: the parsing half is where the correctness risk
+> lives, and burying it under routes and React would have meant reviewing it in a 10k-line diff.
+> Full operator documentation: [`docs/website/SPAWN_ATLAS.md`](../website/SPAWN_ATLAS.md).
+>
+> §6.1 below records where the shipped implementation differs from this design. The differences are
+> all things the real ServUO data forced, not changes of mind.
+
 **Decision: committed generated artifact + idempotent DB import**, split in two because the build
 needs the ServUO tree (which the website container does not have) and the import does not. Not
 runtime import (10.5 MB of XML per boot), not a browser-served blob.
@@ -383,6 +392,53 @@ with the tree → commit the regenerated `db/data/spawnAtlas.*.json` → deploy
 `GET /admin/shard/atlas/status` reports when the DB is behind the artifact. Full detail in
 `docs/website/SPAWN_ATLAS.md`.
 
+### 6.1 What the build against real data changed
+
+Six corrections to the design above, from running it against stock ServUO 57.4. Kept as a diff
+rather than edited in place, because each one is a trap the next person would otherwise re-enter.
+
+**1. Six facets, not thirteen.** The design said `spawnAtlas.<facet>.json ×13`, assuming one facet
+per spawn file. There are 13 files but only **6** facets — `Eodon.xml`, `GravewaterLake.xml`,
+`TreasuresOfKotl.xml` and the other named-area files carry TerMur/Trammel points. The facet comes
+from each record's own `<Map>`, never the file name, and the artifact shards 6 ways.
+
+**2. The XML dependency call: hand-rolled, zero deps.** §6 left `fast-xml-parser` vs a ~120-line
+tokenizer open. Resolved as the tokenizer — a deliberate *subset* parser covering only what these
+files use. The server keeps zero XML dependencies at any tier.
+
+**3. Facet names disagree between sources — a silent failure.** `Data/Locations/*.xml` spells them
+`Ter Mur` and `Tokuno Islands`; `<Map>` and `<Facet name>` say `TerMur` and `Tokuno`. Unreconciled,
+the landmark bucket is keyed differently from the points looking it up, so the fallback never fires
+and **every unregioned spawn in Ter Mur and Tokuno reads "Wilderness"** — a plausible-looking atlas
+that is quietly wrong for two facets. All facet names now pass through `normalizeFacet()`.
+
+**4. Spawn type tokens carry XmlSpawner directives.** `<Objects2>` types are not always bare class
+names: `Fairy,{RND,4,8}`, `alchemist/z/-50`, `Agralem/Name/Agralem`, `greatape,true`. Taken literally
+they invent creatures that do not exist *and* split real ones in two, since `Fairy` and
+`Fairy,{RND,4,8}` slug apart. 71 of 845 entries were affected; stripping at the first `/` or `,`
+leaves **800** real creatures. (The design's "~1,500 creature rows" estimate was high; 800 only
+reinforces the plain-`INDEX`-not-`FULLTEXT` call.)
+
+**5. The artifact is 1.41 MB, not "well under 1 MB".** Dropping the unused `<Points>` fields as the
+design directed still left 4.40 MB. Three further encodings — `facet` dropped per record,
+default-valued fields omitted rather than written as `0`, and `types` as `[name, max]` tuples
+(~24,000 entries × 15 bytes of repeated key names) — brought it to 1.41 MB. Getting under 1 MB would
+mean dropping the spawner `name`, which is the only human handle on a specific spawner and worth
+keeping. `encodePoint()` and `readPoint()` are exact inverses and are round-tripped in tests.
+
+**6. `DELETE`, not `TRUNCATE`.** The design said "TRUNCATE + batched INSERT in one transaction",
+which does not hold: `TRUNCATE` is DDL in MariaDB and implicitly commits, so a mid-import failure
+would leave the atlas half-loaded. `DELETE` is transactional, and at ~7k rows the cost is
+irrelevant. Point ids are also assigned explicitly rather than by `AUTO_INCREMENT`, because the
+join rows need them and `conn.batch()` reports no usable `insertId`.
+
+**Measured result:** 6,455 points, 800 creatures, 23,927 point/type rows, 387 regions, 558
+landmarks, 25 champion altars. The placement transform resolves **83.2%** of points (3,689 by
+region, 1,690 by landmark, 1,086 Wilderness).
+
+**One thing the design got exactly right:** the point-in-rect transform really is the reason to
+build this. "Where does a lizardman spawn?" answers *Shrines, Isamu-Jima, Yew* across three facets.
+
 ---
 
 ## 7. Part B/2 — `points.board`
@@ -577,7 +633,7 @@ inherently up to one full cycle old, and the UI must say so.
 |---|---|---|---|---|
 | 1 | **A** — visibility framework + actor-leak fix | website, docs | none | ✅ Done |
 | 2 | **B/1** — `world.ruleset` (§5) | all four | new kind | ✅ Done |
-| 3 | **C** — spawn atlas (§6) | website, docs | none | ⬜ **Next** |
+| 3 | **C** — spawn atlas (§6) | website, docs | none | 🟡 Pipeline done, API/client next |
 | 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | ⬜ |
 | 5 | **B/3** — `vendor.listing` (§8) | all four | new kinds | ⬜ |
 | 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | ⬜ |

website/BACKEND_DESIGN.md

@@ -388,6 +388,47 @@ feature is ignored (a stale row must not resurrect a removed feature), an invali
 the default rather than failing open, and a rule touching a locked field (`acct` / `webId`) is
 discarded. See §6.5.
 
+### shard_spawn_* / shard_regions / shard_landmarks / shard_champion_spawns / shard_atlas_meta — the spawn atlas (Protocol 3.0)
+
+Static shard **content**, not live shard state. Nothing here comes from the sidecar: the atlas is
+built from a ServUO tree by a CLI script, committed as JSON under `server/db/data/`, and loaded with
+`npm run atlas:import`. These tables stay populated whether the shard is up or not. Full operator
+detail in [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md); the design is `docs/link/v3.md` §6.
+
+| Table | Key columns |
+|---|---|
+| `shard_spawn_creatures` | `slug` PK, `name`, `total`, `points`, `facets` JSON, `art` NULL |
+| `shard_spawn_points` | `id` PK, `facet`, `name`, `x`, `y`, `width`, `height`, `spawn_range`, `max_count`, `min_delay`, `max_delay`, `tod_start/end/mode`, `region`, `landmark`, `label` |
+| `shard_spawn_point_types` | `(point_id, slug)` PK, `max_count` |
+| `shard_regions` | `facet`, `name`, `type`, `priority`, `parent`, `rects` JSON |
+| `shard_landmarks` | `facet`, `name`, `grp`, `x`, `y`, `z` |
+| `shard_champion_spawns` | `slug` PK, `name`, `grp`, `type`, `random_type`, `facet`, `x`, `y`, `z`, `radius`, `label` |
+| `shard_atlas_meta` | Singleton (`id = 1`), `payload` JSON, `imported_at` |
+
+All of them are **import-owned**: `atlas:import` empties and reloads every one inside a single
+transaction, so a failed import leaves the previous atlas intact rather than a half-loaded world.
+Nothing else writes to them, and nothing holds a foreign key to them — no FKs at all, consistent with
+every other `shard_*` table.
+
+Four column choices worth stating, because each one is a trap:
+
+- **`spawn_range`, not `range`**, and **`grp`, not `group`** — both are reserved words.
+- **`DELETE`, not `TRUNCATE`.** `TRUNCATE` is DDL in MariaDB and implicitly commits, which would
+  defeat the all-or-nothing reload. At ~7k rows the difference does not matter.
+- **Point ids are assigned explicitly**, not left to `AUTO_INCREMENT`: the `shard_spawn_point_types`
+  rows need to know them, and `conn.batch()` reports no usable `insertId` for a multi-row insert.
+- **Plain `INDEX` on `name`, deliberately not `FULLTEXT`.** ~800 creature rows makes a `LIKE` scan
+  free, and FULLTEXT's minimum token length would break searches for names like "orc".
+
+`shard_champion_spawns` is the *configured* altar roster ("there is an Unholy Terror altar in
+Deceit"). The live `champ.update` feed in `shard_champs` is the separate answer to "it is on level 3
+right now". Both exist; they are not the same data.
+
+**`shard_spawn_creatures.art` is always NULL on a fresh import.** The project ships no creature
+artwork: sprites live in the operator's own client `.mul`/`.uop` files and are theirs, not ours to
+redistribute. An operator supplies art via a gitignored map plus images under the (already
+gitignored) `server/uploads/atlas/`. Text-only is the normal, supported state.
+
 ---
 
 ## 4. API contract

website/SPAWN_ATLAS.md

@@ -0,0 +1,225 @@
+# Spawn atlas
+
+**Status:** Data pipeline landed on `edge` (website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112)); API and client pages follow in a second PR.
+**Design:** [`docs/link/v3.md` §6](../link/v3.md) — Protocol 3.0 Part C.
+
+The spawn atlas is a browsable catalogue of what the shard *contains*: which
+creatures spawn, where, how many, and which champion altars are configured. It
+answers "where do I find a lizardman?" with **"Shrines, Yew, Isamu-Jima"** rather
+than with a list of raw coordinates.
+
+## What it is not
+
+The atlas is **static shard content, not live shard state.**
+
+- It does **not** come from the sidecar. Nothing here touches the bridge, and
+  there is no event kind, no wire change and no `PROTOCOL_VERSION` bump for it.
+  Part C is website-only.
+- It stays fully populated while the shard is down.
+- Its champion table (`shard_champion_spawns`) is the *configured roster* —
+  "there is an Unholy Terror altar in Deceit". The live `champ.update` feed in
+  `shard_champs` is the separate, sidecar-fed answer to "it is on level 3 right
+  now". Both exist; do not conflate them.
+
+Routes live at `/api/v1/public/atlas`, deliberately **not** under `/shard`,
+because `/shard/*` means sidecar-dependent.
+
+## The two commands
+
+Building needs a ServUO tree. Importing does not. That split is the whole design:
+the website container ships the committed artifact but has no ServUO tree, so it
+can import but never build.
+
+```bash
+# On a machine that has the ServUO tree (writes server/db/data/spawnAtlas.*.json)
+cd website/server
+npm run atlas:build -- --servuo /path/to/ServUO
+
+# Anywhere, including the deployed container
+npm run atlas:import
+```
+
+`atlas:build` flags:
+
+| Flag | Default | Meaning |
+|---|---|---|
+| `--servuo` | *(required)* | ServUO server root — the directory holding `Spawns/`, `Data/`, `Config/` |
+| `--out` | `server/db/data` | Where to write the artifact |
+| `--landmark-radius` | `200` | Max tile distance for the landmark fallback |
+
+`atlas:import` takes `--dir` (default `server/db/data`).
+
+## Operator re-run story
+
+Spawns changed → rebuild → commit → deploy → import.
+
+1. `npm run atlas:build -- --servuo <path>` on a machine with the tree.
+2. Commit the regenerated `server/db/data/spawnAtlas.*.json`.
+3. Deploy.
+4. `npm run atlas:import`, or `POST /api/v1/admin/shard/atlas/import`.
+
+`shard_atlas_meta` stores a sha256 per source file, so
+`GET /api/v1/admin/shard/atlas/status` reports when the database is behind the
+committed artifact. **Build stays CLI-only** — there is no admin button that
+reads a ServUO tree.
+
+## Sources
+
+| File | Count (stock ServUO 57.4) | Used for |
+|---|---|---|
+| `Spawns/*.xml` | 13 files, ~10.5 MB | Every spawner: location, size, delays, time-of-day, creature types |
+| `Data/Regions.xml` | 129 KB, nested | Named regions and their rectangles |
+| `Data/Locations/*.xml` | 6 files | Landmarks (dungeon levels, town markers) |
+| `Config/ChampionSpawns.xml` | 4.8 KB | Configured champion altars |
+
+**There are 13 spawn files but only 6 facets.** `Eodon.xml`,
+`GravewaterLake.xml`, `TreasuresOfKotl.xml` and the other named-area files hold
+TerMur/Trammel points. The facet always comes from each record's own `<Map>`,
+never from the file name, and the artifact shards by facet — Felucca, Trammel,
+Ilshenar, Malas, Tokuno, TerMur.
+
+## How a coordinate becomes a place name
+
+This is the transform the atlas exists for, in `resolveRegion()`:
+
+1. The highest-`priority` named region whose rectangle contains the point. Ties
+   break toward the **smallest** rect, so a specific room wins over the
+   dungeon-wide rect enclosing it.
+2. Otherwise the nearest landmark within `--landmark-radius` tiles, labelled by
+   its **group** ("Covetous"), not its individual marker ("Level 1").
+3. Otherwise `"Wilderness"`.
+
+The radius cap in step 2 is what keeps step 3 reachable. Without it the nearest
+landmark is always *some* landmark however far away, and open countryside gets
+labelled with a dungeon on the far side of the map.
+
+Against stock ServUO this resolves **83.2%** of points (5,369 of 6,455): 3,689 by
+region, 1,690 by landmark, 1,086 Wilderness.
+
+## Three quirks in the source data
+
+Each of these is silent if unhandled — the atlas still builds, it is just wrong.
+
+**Facet names disagree between sources.** `Data/Locations/*.xml` spells them
+`Ter Mur` and `Tokuno Islands`, while `<Map>` and `<Facet name>` say `TerMur` and
+`Tokuno`. Unreconciled, the landmark bucket for those two facets is keyed
+differently from the points looking it up, so the fallback never fires and every
+unregioned spawn in Ter Mur and Tokuno reads "Wilderness". All facet names are
+canonicalised through `normalizeFacet()`; unknown facets pass through unchanged
+so a custom shard facet still gets an atlas.
+
+**Spawn type tokens carry XmlSpawner directives.** The `<Objects2>` type is not
+always a bare class name:
+
+```
+Fairy,{RND,4,8}      alchemist/z/-50      Agralem/Name/Agralem
+GargishRouser,1      greatape,true        GargishRefugee/hue/34532
+```
+
+Taken literally these invent creatures that do not exist *and* split real ones in
+two, because `Fairy` and `Fairy,{RND,4,8}` slug apart into separate entries. 71 of
+845 were affected. Everything from the first `/` or `,` is stripped, leaving 800
+real creatures.
+
+**Case is inconsistent across files.** The same creature is `Lizardman` in one
+file and `lizardman` in another. Slugging collapses them correctly, but the
+display name is chosen deterministically — most common spelling wins, ties break
+to the more capitalised form, then alphabetically — because otherwise it would
+depend on file read order and produce a spurious artifact diff on every unrelated
+rebuild.
+
+## Artifact format
+
+`server/db/data/`, all committed:
+
+| File | Contents |
+|---|---|
+| `spawnAtlas.meta.json` | Indented. Build time, counts, sha256 per source file |
+| `spawnAtlas.index.json` | Compact. Facets, creatures, regions, landmarks, champions |
+| `spawnAtlas.<facet>.json` × 6 | Compact. That facet's spawn points |
+
+**1.41 MB total**, down from 4.40 MB. The design budgeted "well under 1 MB",
+which turned out optimistic for 6,455 points; three encodings closed most of the
+gap:
+
+- `facet` is dropped per record — the shard file names it once at the top.
+- Fields at their default are omitted rather than written as `0`. Most spawners
+  are a single point with no time-of-day gating, so `width`, `height`, `range`
+  and the three `tod*` fields are zero on the large majority of records.
+- `types` are `[name, max]` tuples. There are ~24,000 type entries and
+  `{"type":"Orc","max":1}` spends 15 bytes apiece restating two key names that
+  never vary.
+
+`label` is not stored at all — it is exactly `region || landmark || "Wilderness"`
+and the importer recomputes it.
+
+`buildSpawnAtlas.encodePoint()` and `importSpawnAtlas.readPoint()` are exact
+inverses, round-tripped in `test/spawnAtlas.build.test.js`. **Change one, change
+both.** The artifact never reaches the browser; the browser sees only paginated
+API responses.
+
+## Artwork — operator-supplied, never shipped
+
+**This project ships no creature art and no extraction tooling, and never will.**
+UO sprites live in the operator's own client `.mul`/`.uop` files. They are the
+operator's, not ours to redistribute.
+
+The atlas is fully functional as text. `shard_spawn_creatures.art` is nullable
+and is NULL on every fresh import; pages render without images, which is the
+normal and supported state, not a degraded one.
+
+An operator who wants art:
+
+1. Extracts it from **their own** client files (UOFiddler, ClassicUO tooling, or
+   any art extractor).
+2. Drops the images under `server/uploads/atlas/`.
+3. Copies `server/db/data/spawnAtlas.art.example.json` to `spawnAtlas.art.json`
+   and maps creature slugs to file names.
+4. Re-runs `npm run atlas:import`.
+
+Both `spawnAtlas.art.json` and `server/uploads/` are gitignored, so neither the
+map nor the images can be committed by accident.
+
+## Tables
+
+All six are **import-owned**: `atlas:import` empties and reloads them in one
+transaction. Nothing else writes to them and nothing holds a foreign key to them
+— no FKs at all, consistent with every other `shard_*` table. Full column
+listings in [`BACKEND_DESIGN.md`](BACKEND_DESIGN.md).
+
+| Table | Rows (stock) | Notes |
+|---|---|---|
+| `shard_spawn_creatures` | 800 | `slug` PK; `total` = sum of each type's own max; nullable `art` |
+| `shard_spawn_points` | 6,455 | `spawn_range`, since `range` is reserved in MariaDB |
+| `shard_spawn_point_types` | 23,927 | The many-to-many; one spawner commonly carries six types |
+| `shard_regions` | 387 | Flattened out of the nesting; `rects` JSON |
+| `shard_landmarks` | 558 | `grp`, since `group` is reserved in SQL |
+| `shard_champion_spawns` | 25 | Configured altars, not the live feed |
+| `shard_atlas_meta` | 1 | Singleton (`id = 1`); source hashes for the drift check |
+
+`shard_spawn_creatures.name` carries a plain `INDEX`, deliberately **not
+`FULLTEXT`**: ~800 rows makes a `LIKE` scan free, and FULLTEXT's minimum token
+length would break searches for names like "orc".
+
+The importer uses `DELETE`, not `TRUNCATE` — `TRUNCATE` is DDL in MariaDB and
+would implicitly commit, defeating the all-or-nothing reload. Point ids are
+assigned explicitly rather than left to `AUTO_INCREMENT`, because the join rows
+need to know them and `conn.batch()` reports no usable `insertId`.
+
+## Parsing notes
+
+`server/src/utils/spawnAtlasParse.js` is **pure and fs-free**, so CI covers it
+with no ServUO tree. It adds **zero dependencies**.
+
+- `Regions.xml`, `Locations/*.xml` and `ChampionSpawns.xml` genuinely nest, and
+  get a small hand-rolled **subset** tokenizer — elements, attributes,
+  self-closing tags, comments, the XML declaration, CDATA, and the five
+  predefined entities plus numeric refs. It is not a general-purpose XML parser
+  and must not be reused as one.
+- The ~10.5 MB of `Spawns/*.xml` never touches that tokenizer. Those records are
+  flat, so they get a streaming regex sweep instead; a DOM would allocate a node
+  per element across ~40 fields on every record to keep 14 of them. **Do not put
+  the Points files through a DOM parser.**
+- `<Objects2>` is `Type:MX=n:SB=…` segments joined by `:OBJ=`. Split on `:OBJ=`
+  *first* — a naive `split(':')` shreds it. A single Trammel point carries six
+  types.