RunicGateway/docs
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(website): record the spawn atlas pipeline and what real data changed

Browse Source 
Protocol 3.0 order 3 (Part C), docs half of website #112. Part C is website-only
— no plugin, no sidecar, no new kinds, no wire change.

## New: website/SPAWN_ATLAS.md

The operator-facing reference: the build/import split and why it exists (build
needs a ServUO tree, import does not, and the container has the artifact but not
the tree), the re-run story, the artifact format, the placement transform, and
the three quirks in the source data that are silent when unhandled.

Also documents the artwork policy explicitly: **the project ships no creature
art and no extraction tooling.** Sprites live in the operator's own client
.mul/.uop files and are theirs, not ours to redistribute. `art` is nullable and
NULL on every fresh import; an operator who wants art extracts it themselves into
the gitignored uploads/atlas/ and maps slugs in a gitignored art map. Text-only
is the normal, supported state — not a degraded one.

## New: v3.md §6.1 — what the build against real data changed

Six corrections, kept as a diff rather than edited into §6 in place, because
each is a trap the next person would otherwise re-enter:

1. **Six facets, not thirteen.** Eodon.xml and the other named-area files carry
   TerMur/Trammel points; the facet comes from each record's `<Map>`.
2. **The XML dependency call resolved: hand-rolled, zero deps.** §6 left
   fast-xml-parser vs a tokenizer open.
3. **Facet names disagree between sources** — Locations says `Ter Mur`, `<Map>`
   says `TerMur`. Unreconciled the landmark fallback never fires there and every
   unregioned Ter Mur/Tokuno spawn silently reads "Wilderness".
4. **Spawn type tokens carry XmlSpawner directives** (`Fairy,{RND,4,8}`,
   `alchemist/z/-50`). Taken literally they invent creatures that do not exist
   and split real ones in two. 71 of 845 affected; 800 remain after stripping.
5. **The artifact is 1.41 MB, not "well under 1 MB"** — down from 4.40 MB via
   three encodings. Getting under 1 MB would mean dropping the spawner name.
6. **DELETE, not TRUNCATE** — TRUNCATE is DDL in MariaDB and implicitly commits,
   which would defeat the all-or-nothing reload the design asked for.

§6 also now records that Part C ships as two website PRs: the parsing half is
where the correctness risk lives and should not be reviewed inside a 10k-line
diff alongside routes and React.

## BACKEND_DESIGN.md

The seven atlas tables, the import-owned contract, the four column choices that
are traps (`spawn_range`/`grp` reserved words, DELETE vs TRUNCATE, explicit point
ids, plain INDEX not FULLTEXT), and the distinction between the configured
champion roster and the live champ.update feed.

PROJECT_TREE.md is left alone — it is auto-generated by the sync-project-tree
workflow.

---

- [x] AI-assisted: written with **Claude Code** (Claude Opus 5), reviewed before opening.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U7CBg11prhLimL9iHSX1bP
This commit is contained in:
claude
2026-07-28 16:13:30 -05:00
parent e9ecdc0ecb
commit 3fb3f63f25
3 changed files with 324 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

41
website/BACKEND_DESIGN.md
View File

@@ -388,6 +388,47 @@ feature is ignored (a stale row must not resurrect a removed feature), an invali
the default rather than failing open, and a rule touching a locked field (`acct` / `webId`) is
discarded. See §6.5.
### shard_spawn_* / shard_regions / shard_landmarks / shard_champion_spawns / shard_atlas_meta — the spawn atlas (Protocol 3.0)
Static shard **content**, not live shard state. Nothing here comes from the sidecar: the atlas is
built from a ServUO tree by a CLI script, committed as JSON under `server/db/data/`, and loaded with
`npm run atlas:import`. These tables stay populated whether the shard is up or not. Full operator
detail in [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md); the design is `docs/link/v3.md` §6.
| Table | Key columns |
|---|---|
| `shard_spawn_creatures` | `slug` PK, `name`, `total`, `points`, `facets` JSON, `art` NULL |
| `shard_spawn_points` | `id` PK, `facet`, `name`, `x`, `y`, `width`, `height`, `spawn_range`, `max_count`, `min_delay`, `max_delay`, `tod_start/end/mode`, `region`, `landmark`, `label` |
| `shard_spawn_point_types` | `(point_id, slug)` PK, `max_count` |
| `shard_regions` | `facet`, `name`, `type`, `priority`, `parent`, `rects` JSON |
| `shard_landmarks` | `facet`, `name`, `grp`, `x`, `y`, `z` |
| `shard_champion_spawns` | `slug` PK, `name`, `grp`, `type`, `random_type`, `facet`, `x`, `y`, `z`, `radius`, `label` |
| `shard_atlas_meta` | Singleton (`id = 1`), `payload` JSON, `imported_at` |
All of them are **import-owned**: `atlas:import` empties and reloads every one inside a single
transaction, so a failed import leaves the previous atlas intact rather than a half-loaded world.
Nothing else writes to them, and nothing holds a foreign key to them — no FKs at all, consistent with
every other `shard_*` table.
Four column choices worth stating, because each one is a trap:
- **`spawn_range`, not `range`**, and **`grp`, not `group`** — both are reserved words.
- **`DELETE`, not `TRUNCATE`.** `TRUNCATE` is DDL in MariaDB and implicitly commits, which would
defeat the all-or-nothing reload. At ~7k rows the difference does not matter.
- **Point ids are assigned explicitly**, not left to `AUTO_INCREMENT`: the `shard_spawn_point_types`
rows need to know them, and `conn.batch()` reports no usable `insertId` for a multi-row insert.
- **Plain `INDEX` on `name`, deliberately not `FULLTEXT`.** ~800 creature rows makes a `LIKE` scan
free, and FULLTEXT's minimum token length would break searches for names like "orc".
`shard_champion_spawns` is the *configured* altar roster ("there is an Unholy Terror altar in
Deceit"). The live `champ.update` feed in `shard_champs` is the separate answer to "it is on level 3
right now". Both exist; they are not the same data.
**`shard_spawn_creatures.art` is always NULL on a fresh import.** The project ships no creature
artwork: sprites live in the operator's own client `.mul`/`.uop` files and are theirs, not ours to
redistribute. An operator supplies art via a gitignored map plus images under the (already
gitignored) `server/uploads/atlas/`. Text-only is the normal, supported state.
---
## 4. API contract

225
website/SPAWN_ATLAS.md Normal file
View File

@@ -0,0 +1,225 @@
# Spawn atlas
**Status:** Data pipeline landed on `edge` (website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112)); API and client pages follow in a second PR.
**Design:** [`docs/link/v3.md` §6](../link/v3.md) — Protocol 3.0 Part C.
The spawn atlas is a browsable catalogue of what the shard *contains*: which
creatures spawn, where, how many, and which champion altars are configured. It
answers "where do I find a lizardman?" with **"Shrines, Yew, Isamu-Jima"** rather
than with a list of raw coordinates.
## What it is not
The atlas is **static shard content, not live shard state.**
- It does **not** come from the sidecar. Nothing here touches the bridge, and
there is no event kind, no wire change and no `PROTOCOL_VERSION` bump for it.
Part C is website-only.
- It stays fully populated while the shard is down.
- Its champion table (`shard_champion_spawns`) is the *configured roster* —
"there is an Unholy Terror altar in Deceit". The live `champ.update` feed in
`shard_champs` is the separate, sidecar-fed answer to "it is on level 3 right
now". Both exist; do not conflate them.
Routes live at `/api/v1/public/atlas`, deliberately **not** under `/shard`,
because `/shard/*` means sidecar-dependent.
## The two commands
Building needs a ServUO tree. Importing does not. That split is the whole design:
the website container ships the committed artifact but has no ServUO tree, so it
can import but never build.
```bash
# On a machine that has the ServUO tree (writes server/db/data/spawnAtlas.*.json)
cd website/server
npm run atlas:build -- --servuo /path/to/ServUO
# Anywhere, including the deployed container
npm run atlas:import
```
`atlas:build` flags:
| Flag | Default | Meaning |
|---|---|---|
| `--servuo` | *(required)* | ServUO server root — the directory holding `Spawns/`, `Data/`, `Config/` |
| `--out` | `server/db/data` | Where to write the artifact |
| `--landmark-radius` | `200` | Max tile distance for the landmark fallback |
`atlas:import` takes `--dir` (default `server/db/data`).
## Operator re-run story
Spawns changed → rebuild → commit → deploy → import.
1. `npm run atlas:build -- --servuo <path>` on a machine with the tree.
2. Commit the regenerated `server/db/data/spawnAtlas.*.json`.
3. Deploy.
4. `npm run atlas:import`, or `POST /api/v1/admin/shard/atlas/import`.
`shard_atlas_meta` stores a sha256 per source file, so
`GET /api/v1/admin/shard/atlas/status` reports when the database is behind the
committed artifact. **Build stays CLI-only** — there is no admin button that
reads a ServUO tree.
## Sources
| File | Count (stock ServUO 57.4) | Used for |
|---|---|---|
| `Spawns/*.xml` | 13 files, ~10.5 MB | Every spawner: location, size, delays, time-of-day, creature types |
| `Data/Regions.xml` | 129 KB, nested | Named regions and their rectangles |
| `Data/Locations/*.xml` | 6 files | Landmarks (dungeon levels, town markers) |
| `Config/ChampionSpawns.xml` | 4.8 KB | Configured champion altars |
**There are 13 spawn files but only 6 facets.** `Eodon.xml`,
`GravewaterLake.xml`, `TreasuresOfKotl.xml` and the other named-area files hold
TerMur/Trammel points. The facet always comes from each record's own `<Map>`,
never from the file name, and the artifact shards by facet — Felucca, Trammel,
Ilshenar, Malas, Tokuno, TerMur.
## How a coordinate becomes a place name
This is the transform the atlas exists for, in `resolveRegion()`:
1. The highest-`priority` named region whose rectangle contains the point. Ties
break toward the **smallest** rect, so a specific room wins over the
dungeon-wide rect enclosing it.
2. Otherwise the nearest landmark within `--landmark-radius` tiles, labelled by
its **group** ("Covetous"), not its individual marker ("Level 1").
3. Otherwise `"Wilderness"`.
The radius cap in step 2 is what keeps step 3 reachable. Without it the nearest
landmark is always *some* landmark however far away, and open countryside gets
labelled with a dungeon on the far side of the map.
Against stock ServUO this resolves **83.2%** of points (5,369 of 6,455): 3,689 by
region, 1,690 by landmark, 1,086 Wilderness.
## Three quirks in the source data
Each of these is silent if unhandled — the atlas still builds, it is just wrong.
**Facet names disagree between sources.** `Data/Locations/*.xml` spells them
`Ter Mur` and `Tokuno Islands`, while `<Map>` and `<Facet name>` say `TerMur` and
`Tokuno`. Unreconciled, the landmark bucket for those two facets is keyed
differently from the points looking it up, so the fallback never fires and every
unregioned spawn in Ter Mur and Tokuno reads "Wilderness". All facet names are
canonicalised through `normalizeFacet()`; unknown facets pass through unchanged
so a custom shard facet still gets an atlas.
**Spawn type tokens carry XmlSpawner directives.** The `<Objects2>` type is not
always a bare class name:
```
Fairy,{RND,4,8} alchemist/z/-50 Agralem/Name/Agralem
GargishRouser,1 greatape,true GargishRefugee/hue/34532
```
Taken literally these invent creatures that do not exist *and* split real ones in
two, because `Fairy` and `Fairy,{RND,4,8}` slug apart into separate entries. 71 of
845 were affected. Everything from the first `/` or `,` is stripped, leaving 800
real creatures.
**Case is inconsistent across files.** The same creature is `Lizardman` in one
file and `lizardman` in another. Slugging collapses them correctly, but the
display name is chosen deterministically — most common spelling wins, ties break
to the more capitalised form, then alphabetically — because otherwise it would
depend on file read order and produce a spurious artifact diff on every unrelated
rebuild.
## Artifact format
`server/db/data/`, all committed:
| File | Contents |
|---|---|
| `spawnAtlas.meta.json` | Indented. Build time, counts, sha256 per source file |
| `spawnAtlas.index.json` | Compact. Facets, creatures, regions, landmarks, champions |
| `spawnAtlas.<facet>.json` × 6 | Compact. That facet's spawn points |
**1.41 MB total**, down from 4.40 MB. The design budgeted "well under 1 MB",
which turned out optimistic for 6,455 points; three encodings closed most of the
gap:
- `facet` is dropped per record — the shard file names it once at the top.
- Fields at their default are omitted rather than written as `0`. Most spawners
are a single point with no time-of-day gating, so `width`, `height`, `range`
and the three `tod*` fields are zero on the large majority of records.
- `types` are `[name, max]` tuples. There are ~24,000 type entries and
`{"type":"Orc","max":1}` spends 15 bytes apiece restating two key names that
never vary.
`label` is not stored at all — it is exactly `region || landmark || "Wilderness"`
and the importer recomputes it.
`buildSpawnAtlas.encodePoint()` and `importSpawnAtlas.readPoint()` are exact
inverses, round-tripped in `test/spawnAtlas.build.test.js`. **Change one, change
both.** The artifact never reaches the browser; the browser sees only paginated
API responses.
## Artwork — operator-supplied, never shipped
**This project ships no creature art and no extraction tooling, and never will.**
UO sprites live in the operator's own client `.mul`/`.uop` files. They are the
operator's, not ours to redistribute.
The atlas is fully functional as text. `shard_spawn_creatures.art` is nullable
and is NULL on every fresh import; pages render without images, which is the
normal and supported state, not a degraded one.
An operator who wants art:
1. Extracts it from **their own** client files (UOFiddler, ClassicUO tooling, or
any art extractor).
2. Drops the images under `server/uploads/atlas/`.
3. Copies `server/db/data/spawnAtlas.art.example.json` to `spawnAtlas.art.json`
and maps creature slugs to file names.
4. Re-runs `npm run atlas:import`.
Both `spawnAtlas.art.json` and `server/uploads/` are gitignored, so neither the
map nor the images can be committed by accident.
## Tables
All six are **import-owned**: `atlas:import` empties and reloads them in one
transaction. Nothing else writes to them and nothing holds a foreign key to them
— no FKs at all, consistent with every other `shard_*` table. Full column
listings in [`BACKEND_DESIGN.md`](BACKEND_DESIGN.md).
| Table | Rows (stock) | Notes |
|---|---|---|
| `shard_spawn_creatures` | 800 | `slug` PK; `total` = sum of each type's own max; nullable `art` |
| `shard_spawn_points` | 6,455 | `spawn_range`, since `range` is reserved in MariaDB |
| `shard_spawn_point_types` | 23,927 | The many-to-many; one spawner commonly carries six types |
| `shard_regions` | 387 | Flattened out of the nesting; `rects` JSON |
| `shard_landmarks` | 558 | `grp`, since `group` is reserved in SQL |
| `shard_champion_spawns` | 25 | Configured altars, not the live feed |
| `shard_atlas_meta` | 1 | Singleton (`id = 1`); source hashes for the drift check |
`shard_spawn_creatures.name` carries a plain `INDEX`, deliberately **not
`FULLTEXT`**: ~800 rows makes a `LIKE` scan free, and FULLTEXT's minimum token
length would break searches for names like "orc".
The importer uses `DELETE`, not `TRUNCATE` — `TRUNCATE` is DDL in MariaDB and
would implicitly commit, defeating the all-or-nothing reload. Point ids are
assigned explicitly rather than left to `AUTO_INCREMENT`, because the join rows
need to know them and `conn.batch()` reports no usable `insertId`.
## Parsing notes
`server/src/utils/spawnAtlasParse.js` is **pure and fs-free**, so CI covers it
with no ServUO tree. It adds **zero dependencies**.
- `Regions.xml`, `Locations/*.xml` and `ChampionSpawns.xml` genuinely nest, and
get a small hand-rolled **subset** tokenizer — elements, attributes,
self-closing tags, comments, the XML declaration, CDATA, and the five
predefined entities plus numeric refs. It is not a general-purpose XML parser
and must not be reused as one.
- The ~10.5 MB of `Spawns/*.xml` never touches that tokenizer. Those records are
flat, so they get a streaming regex sweep instead; a DOM would allocate a node
per element across ~40 fields on every record to keep 14 of them. **Do not put
the Points files through a DOM parser.**
- `<Objects2>` is `Type:MX=n:SB=…` segments joined by `:OBJ=`. Split on `:OBJ=`
*first* — a naive `split(':')` shreds it. A single Trammel point carries six
types.