wtclaude
3fb3f63f25
Protocol 3.0 order 3 (Part C), docs half of website #112. Part C is website-only — no plugin, no sidecar, no new kinds, no wire change. ## New: website/SPAWN_ATLAS.md The operator-facing reference: the build/import split and why it exists (build needs a ServUO tree, import does not, and the container has the artifact but not the tree), the re-run story, the artifact format, the placement transform, and the three quirks in the source data that are silent when unhandled. Also documents the artwork policy explicitly: **the project ships no creature art and no extraction tooling.** Sprites live in the operator's own client .mul/.uop files and are theirs, not ours to redistribute. `art` is nullable and NULL on every fresh import; an operator who wants art extracts it themselves into the gitignored uploads/atlas/ and maps slugs in a gitignored art map. Text-only is the normal, supported state — not a degraded one. ## New: v3.md §6.1 — what the build against real data changed Six corrections, kept as a diff rather than edited into §6 in place, because each is a trap the next person would otherwise re-enter: 1. **Six facets, not thirteen.** Eodon.xml and the other named-area files carry TerMur/Trammel points; the facet comes from each record's `<Map>`. 2. **The XML dependency call resolved: hand-rolled, zero deps.** §6 left fast-xml-parser vs a tokenizer open. 3. **Facet names disagree between sources** — Locations says `Ter Mur`, `<Map>` says `TerMur`. Unreconciled the landmark fallback never fires there and every unregioned Ter Mur/Tokuno spawn silently reads "Wilderness". 4. **Spawn type tokens carry XmlSpawner directives** (`Fairy,{RND,4,8}`, `alchemist/z/-50`). Taken literally they invent creatures that do not exist and split real ones in two. 71 of 845 affected; 800 remain after stripping. 5. **The artifact is 1.41 MB, not "well under 1 MB"** — down from 4.40 MB via three encodings. Getting under 1 MB would mean dropping the spawner name. 6. **DELETE, not TRUNCATE** — TRUNCATE is DDL in MariaDB and implicitly commits, which would defeat the all-or-nothing reload the design asked for. §6 also now records that Part C ships as two website PRs: the parsing half is where the correctness risk lives and should not be reviewed inside a 10k-line diff alongside routes and React. ## BACKEND_DESIGN.md The seven atlas tables, the import-owned contract, the four column choices that are traps (`spawn_range`/`grp` reserved words, DELETE vs TRUNCATE, explicit point ids, plain INDEX not FULLTEXT), and the distinction between the configured champion roster and the live champ.update feed. PROJECT_TREE.md is left alone — it is auto-generated by the sync-project-tree workflow. --- - [x] AI-assisted: written with **Claude Code** (Claude Opus 5), reviewed before opening. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U7CBg11prhLimL9iHSX1bP
226 lines
10 KiB
Markdown
226 lines
10 KiB
Markdown
# Spawn atlas
|
||
|
||
**Status:** Data pipeline landed on `edge` (website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112)); API and client pages follow in a second PR.
|
||
**Design:** [`docs/link/v3.md` §6](../link/v3.md) — Protocol 3.0 Part C.
|
||
|
||
The spawn atlas is a browsable catalogue of what the shard *contains*: which
|
||
creatures spawn, where, how many, and which champion altars are configured. It
|
||
answers "where do I find a lizardman?" with **"Shrines, Yew, Isamu-Jima"** rather
|
||
than with a list of raw coordinates.
|
||
|
||
## What it is not
|
||
|
||
The atlas is **static shard content, not live shard state.**
|
||
|
||
- It does **not** come from the sidecar. Nothing here touches the bridge, and
|
||
there is no event kind, no wire change and no `PROTOCOL_VERSION` bump for it.
|
||
Part C is website-only.
|
||
- It stays fully populated while the shard is down.
|
||
- Its champion table (`shard_champion_spawns`) is the *configured roster* —
|
||
"there is an Unholy Terror altar in Deceit". The live `champ.update` feed in
|
||
`shard_champs` is the separate, sidecar-fed answer to "it is on level 3 right
|
||
now". Both exist; do not conflate them.
|
||
|
||
Routes live at `/api/v1/public/atlas`, deliberately **not** under `/shard`,
|
||
because `/shard/*` means sidecar-dependent.
|
||
|
||
## The two commands
|
||
|
||
Building needs a ServUO tree. Importing does not. That split is the whole design:
|
||
the website container ships the committed artifact but has no ServUO tree, so it
|
||
can import but never build.
|
||
|
||
```bash
|
||
# On a machine that has the ServUO tree (writes server/db/data/spawnAtlas.*.json)
|
||
cd website/server
|
||
npm run atlas:build -- --servuo /path/to/ServUO
|
||
|
||
# Anywhere, including the deployed container
|
||
npm run atlas:import
|
||
```
|
||
|
||
`atlas:build` flags:
|
||
|
||
| Flag | Default | Meaning |
|
||
|---|---|---|
|
||
| `--servuo` | *(required)* | ServUO server root — the directory holding `Spawns/`, `Data/`, `Config/` |
|
||
| `--out` | `server/db/data` | Where to write the artifact |
|
||
| `--landmark-radius` | `200` | Max tile distance for the landmark fallback |
|
||
|
||
`atlas:import` takes `--dir` (default `server/db/data`).
|
||
|
||
## Operator re-run story
|
||
|
||
Spawns changed → rebuild → commit → deploy → import.
|
||
|
||
1. `npm run atlas:build -- --servuo <path>` on a machine with the tree.
|
||
2. Commit the regenerated `server/db/data/spawnAtlas.*.json`.
|
||
3. Deploy.
|
||
4. `npm run atlas:import`, or `POST /api/v1/admin/shard/atlas/import`.
|
||
|
||
`shard_atlas_meta` stores a sha256 per source file, so
|
||
`GET /api/v1/admin/shard/atlas/status` reports when the database is behind the
|
||
committed artifact. **Build stays CLI-only** — there is no admin button that
|
||
reads a ServUO tree.
|
||
|
||
## Sources
|
||
|
||
| File | Count (stock ServUO 57.4) | Used for |
|
||
|---|---|---|
|
||
| `Spawns/*.xml` | 13 files, ~10.5 MB | Every spawner: location, size, delays, time-of-day, creature types |
|
||
| `Data/Regions.xml` | 129 KB, nested | Named regions and their rectangles |
|
||
| `Data/Locations/*.xml` | 6 files | Landmarks (dungeon levels, town markers) |
|
||
| `Config/ChampionSpawns.xml` | 4.8 KB | Configured champion altars |
|
||
|
||
**There are 13 spawn files but only 6 facets.** `Eodon.xml`,
|
||
`GravewaterLake.xml`, `TreasuresOfKotl.xml` and the other named-area files hold
|
||
TerMur/Trammel points. The facet always comes from each record's own `<Map>`,
|
||
never from the file name, and the artifact shards by facet — Felucca, Trammel,
|
||
Ilshenar, Malas, Tokuno, TerMur.
|
||
|
||
## How a coordinate becomes a place name
|
||
|
||
This is the transform the atlas exists for, in `resolveRegion()`:
|
||
|
||
1. The highest-`priority` named region whose rectangle contains the point. Ties
|
||
break toward the **smallest** rect, so a specific room wins over the
|
||
dungeon-wide rect enclosing it.
|
||
2. Otherwise the nearest landmark within `--landmark-radius` tiles, labelled by
|
||
its **group** ("Covetous"), not its individual marker ("Level 1").
|
||
3. Otherwise `"Wilderness"`.
|
||
|
||
The radius cap in step 2 is what keeps step 3 reachable. Without it the nearest
|
||
landmark is always *some* landmark however far away, and open countryside gets
|
||
labelled with a dungeon on the far side of the map.
|
||
|
||
Against stock ServUO this resolves **83.2%** of points (5,369 of 6,455): 3,689 by
|
||
region, 1,690 by landmark, 1,086 Wilderness.
|
||
|
||
## Three quirks in the source data
|
||
|
||
Each of these is silent if unhandled — the atlas still builds, it is just wrong.
|
||
|
||
**Facet names disagree between sources.** `Data/Locations/*.xml` spells them
|
||
`Ter Mur` and `Tokuno Islands`, while `<Map>` and `<Facet name>` say `TerMur` and
|
||
`Tokuno`. Unreconciled, the landmark bucket for those two facets is keyed
|
||
differently from the points looking it up, so the fallback never fires and every
|
||
unregioned spawn in Ter Mur and Tokuno reads "Wilderness". All facet names are
|
||
canonicalised through `normalizeFacet()`; unknown facets pass through unchanged
|
||
so a custom shard facet still gets an atlas.
|
||
|
||
**Spawn type tokens carry XmlSpawner directives.** The `<Objects2>` type is not
|
||
always a bare class name:
|
||
|
||
```
|
||
Fairy,{RND,4,8} alchemist/z/-50 Agralem/Name/Agralem
|
||
GargishRouser,1 greatape,true GargishRefugee/hue/34532
|
||
```
|
||
|
||
Taken literally these invent creatures that do not exist *and* split real ones in
|
||
two, because `Fairy` and `Fairy,{RND,4,8}` slug apart into separate entries. 71 of
|
||
845 were affected. Everything from the first `/` or `,` is stripped, leaving 800
|
||
real creatures.
|
||
|
||
**Case is inconsistent across files.** The same creature is `Lizardman` in one
|
||
file and `lizardman` in another. Slugging collapses them correctly, but the
|
||
display name is chosen deterministically — most common spelling wins, ties break
|
||
to the more capitalised form, then alphabetically — because otherwise it would
|
||
depend on file read order and produce a spurious artifact diff on every unrelated
|
||
rebuild.
|
||
|
||
## Artifact format
|
||
|
||
`server/db/data/`, all committed:
|
||
|
||
| File | Contents |
|
||
|---|---|
|
||
| `spawnAtlas.meta.json` | Indented. Build time, counts, sha256 per source file |
|
||
| `spawnAtlas.index.json` | Compact. Facets, creatures, regions, landmarks, champions |
|
||
| `spawnAtlas.<facet>.json` × 6 | Compact. That facet's spawn points |
|
||
|
||
**1.41 MB total**, down from 4.40 MB. The design budgeted "well under 1 MB",
|
||
which turned out optimistic for 6,455 points; three encodings closed most of the
|
||
gap:
|
||
|
||
- `facet` is dropped per record — the shard file names it once at the top.
|
||
- Fields at their default are omitted rather than written as `0`. Most spawners
|
||
are a single point with no time-of-day gating, so `width`, `height`, `range`
|
||
and the three `tod*` fields are zero on the large majority of records.
|
||
- `types` are `[name, max]` tuples. There are ~24,000 type entries and
|
||
`{"type":"Orc","max":1}` spends 15 bytes apiece restating two key names that
|
||
never vary.
|
||
|
||
`label` is not stored at all — it is exactly `region || landmark || "Wilderness"`
|
||
and the importer recomputes it.
|
||
|
||
`buildSpawnAtlas.encodePoint()` and `importSpawnAtlas.readPoint()` are exact
|
||
inverses, round-tripped in `test/spawnAtlas.build.test.js`. **Change one, change
|
||
both.** The artifact never reaches the browser; the browser sees only paginated
|
||
API responses.
|
||
|
||
## Artwork — operator-supplied, never shipped
|
||
|
||
**This project ships no creature art and no extraction tooling, and never will.**
|
||
UO sprites live in the operator's own client `.mul`/`.uop` files. They are the
|
||
operator's, not ours to redistribute.
|
||
|
||
The atlas is fully functional as text. `shard_spawn_creatures.art` is nullable
|
||
and is NULL on every fresh import; pages render without images, which is the
|
||
normal and supported state, not a degraded one.
|
||
|
||
An operator who wants art:
|
||
|
||
1. Extracts it from **their own** client files (UOFiddler, ClassicUO tooling, or
|
||
any art extractor).
|
||
2. Drops the images under `server/uploads/atlas/`.
|
||
3. Copies `server/db/data/spawnAtlas.art.example.json` to `spawnAtlas.art.json`
|
||
and maps creature slugs to file names.
|
||
4. Re-runs `npm run atlas:import`.
|
||
|
||
Both `spawnAtlas.art.json` and `server/uploads/` are gitignored, so neither the
|
||
map nor the images can be committed by accident.
|
||
|
||
## Tables
|
||
|
||
All six are **import-owned**: `atlas:import` empties and reloads them in one
|
||
transaction. Nothing else writes to them and nothing holds a foreign key to them
|
||
— no FKs at all, consistent with every other `shard_*` table. Full column
|
||
listings in [`BACKEND_DESIGN.md`](BACKEND_DESIGN.md).
|
||
|
||
| Table | Rows (stock) | Notes |
|
||
|---|---|---|
|
||
| `shard_spawn_creatures` | 800 | `slug` PK; `total` = sum of each type's own max; nullable `art` |
|
||
| `shard_spawn_points` | 6,455 | `spawn_range`, since `range` is reserved in MariaDB |
|
||
| `shard_spawn_point_types` | 23,927 | The many-to-many; one spawner commonly carries six types |
|
||
| `shard_regions` | 387 | Flattened out of the nesting; `rects` JSON |
|
||
| `shard_landmarks` | 558 | `grp`, since `group` is reserved in SQL |
|
||
| `shard_champion_spawns` | 25 | Configured altars, not the live feed |
|
||
| `shard_atlas_meta` | 1 | Singleton (`id = 1`); source hashes for the drift check |
|
||
|
||
`shard_spawn_creatures.name` carries a plain `INDEX`, deliberately **not
|
||
`FULLTEXT`**: ~800 rows makes a `LIKE` scan free, and FULLTEXT's minimum token
|
||
length would break searches for names like "orc".
|
||
|
||
The importer uses `DELETE`, not `TRUNCATE` — `TRUNCATE` is DDL in MariaDB and
|
||
would implicitly commit, defeating the all-or-nothing reload. Point ids are
|
||
assigned explicitly rather than left to `AUTO_INCREMENT`, because the join rows
|
||
need to know them and `conn.batch()` reports no usable `insertId`.
|
||
|
||
## Parsing notes
|
||
|
||
`server/src/utils/spawnAtlasParse.js` is **pure and fs-free**, so CI covers it
|
||
with no ServUO tree. It adds **zero dependencies**.
|
||
|
||
- `Regions.xml`, `Locations/*.xml` and `ChampionSpawns.xml` genuinely nest, and
|
||
get a small hand-rolled **subset** tokenizer — elements, attributes,
|
||
self-closing tags, comments, the XML declaration, CDATA, and the five
|
||
predefined entities plus numeric refs. It is not a general-purpose XML parser
|
||
and must not be reused as one.
|
||
- The ~10.5 MB of `Spawns/*.xml` never touches that tokenizer. Those records are
|
||
flat, so they get a streaming regex sweep instead; a DOM would allocate a node
|
||
per element across ~40 fields on every record to keep 14 of them. **Do not put
|
||
the Points files through a DOM parser.**
|
||
- `<Objects2>` is `Type:MX=n:SB=…` segments joined by `:OBJ=`. Split on `:OBJ=`
|
||
*first* — a naive `split(':')` shreds it. A single Trammel point carries six
|
||
types.
|