docs/website/SPAWN_ATLAS.md

# Spawn atlas

**Status:** Complete on `edge` — data pipeline in website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112), API + pages in website [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113).
**Design:** [`docs/link/v3.md` §6](../link/v3.md) — Protocol 3.0 Part C.

The spawn atlas is a browsable catalogue of what the shard *contains*: which
creatures spawn, where, how many, and which champion altars are configured. It
answers "where do I find a lizardman?" with **"Shrines, Yew, Isamu-Jima"** rather
than with a list of raw coordinates.

## Two things that shape the whole design

**The shard's ServUO tree is the single source of truth.** Nothing is
precomputed and committed to the repository. A shard's maps change over its
lifetime — facets get added, replaced, or renamed — and a snapshot in the repo
would silently drift from the world players actually see. The atlas is therefore
re-derived from the tree **on every server boot**.

**Facets are not a fixed list.** Nothing in the codebase names Felucca, Trammel,
or any other stock facet. The facet set is whatever the shard's own files
declare, discovered at parse time. A shard running entirely custom maps gets
exactly the same treatment as a stock one, with no code change.

## What it is not

The atlas is **static shard content, not live shard state.**

- It does **not** come from the sidecar. Nothing here touches the bridge, and
  there is no event kind, no wire change and no `PROTOCOL_VERSION` bump for it.
  Part C is website-only.
- It stays fully populated while the shard is down.
- Its champion table (`shard_champion_spawns`) is the *configured roster* —
  "there is an Unholy Terror altar in Deceit". The live `champ.update` feed in
  `shard_champs` is the separate, sidecar-fed answer to "it is on level 3 right
  now". Both exist; do not conflate them.

Routes live at `/api/v1/public/atlas`, deliberately **not** under `/shard`,
because `/shard/*` means sidecar-dependent.

## Configuring the tree

The website needs to be able to *read* the ServUO tree — same host, a bind mount,
or a shared volume. Two ways to point at it, the setting winning over the
environment:

| Source | Notes |
|---|---|
| `spawn_atlas_servuo_path` setting | Admin-editable; changes take effect on the next refresh without a redeploy |
| `SERVUO_PATH` env var | The deploy-time default, since the path usually describes a mount the deployment sets up |

With neither set the atlas is simply skipped — the site runs normally without
one.

## The boot path

On every start the server hashes the source files and compares them against what
is loaded. Unchanged (the normal case on a restart) costs one read pass, ~120 ms,
and no database write. A real change costs a ~400 ms parse and a reload.

Two contracts govern it:

**1. It never blocks startup.** No configured path, an unreadable mount, a
malformed file, a database error — every one is caught and logged, and the site
comes up serving whatever atlas it already had.

**2. A facet disappearing is never applied automatically.** Losing a facet looks
exactly like a half-copied or mid-update tree, and boot cannot tell that apart
from a real map change. That refresh is *staged* for a human instead. Everything
else — new facets, renamed regions, changed spawns — applies immediately, since
none of it can destroy something an operator would miss.

```
boot
 └─ path configured?            no ──▶ skip
     └─ tree readable?          no ──▶ warn, carry on
         └─ hashes changed?     no ──▶ done (nothing parsed)
             └─ parse
                 └─ a facet would be removed?
                      no ──▶ import
                     yes ──▶ stage for admin review; atlas unchanged
```

### Approving or rejecting a staged refresh

Only the *decision* is stored, never the parsed world — a few KB of source hashes
plus the facet diff. Approving **re-parses** the tree, so what lands matches the
tree at approval time rather than at boot, and a multi-megabyte blob never sits
in the database.

A rejection is remembered against those exact source hashes, so a declined
refresh does not re-prompt on every restart. Change the tree and the hashes
differ, which asks again.

From **Admin → Spawn Atlas**, or from the CLI:

```bash
cd website/server
npm run atlas:import -- --status     # what is loaded, and what is pending
npm run atlas:import -- --approve    # apply the staged refresh
npm run atlas:import -- --reject     # keep the current atlas, dismiss it
```

## The CLI

The server refreshes itself on boot, so this is for applying a map change
*without* a restart, and for the approve/reject flow above.

```bash
npm run atlas:import                       # import if the tree differs
npm run atlas:import -- --servuo <path>    # override the path for this run
npm run atlas:import -- --force            # reimport even if unchanged
```

`--servuo` is a per-run override and deliberately does **not** persist — changing
where the atlas permanently reads from is an admin action, not a side effect of a
one-off import.

## Sources

| File | Count (stock ServUO 57.4) | Used for |
|---|---|---|
| `Spawns/*.xml` | 13 files, ~10.5 MB | Every spawner: location, size, delays, time-of-day, creature types |
| `Data/Regions.xml` | 129 KB, nested | Named regions and their rectangles |
| `Data/Locations/*.xml` | 6 files | Landmarks (dungeon levels, town markers) |
| `Config/ChampionSpawns.xml` | 4.8 KB | Configured champion altars |

**A stock tree has 13 spawn files but only 6 facets.** `Eodon.xml`,
`GravewaterLake.xml`, `TreasuresOfKotl.xml` and the other named-area files hold
TerMur/Trammel points. The facet always comes from each record's own `<Map>`,
never from the file name.

## How a coordinate becomes a place name

This is the transform the atlas exists for, in `resolveRegion()`:

1. The highest-`priority` named region whose rectangle contains the point. Ties
   break toward the **smallest** rect, so a specific room wins over the
   dungeon-wide rect enclosing it.
2. Otherwise the nearest landmark within the landmark radius (200 tiles by
   default), labelled by its **group** ("Covetous"), not its individual marker
   ("Level 1").
3. Otherwise `"Wilderness"`.

The radius cap in step 2 is what keeps step 3 reachable. Without it the nearest
landmark is always *some* landmark however far away, and open countryside gets
labelled with a dungeon on the far side of the map.

Against stock ServUO this resolves **83.2%** of points (5,369 of 6,455): 3,681 by
region, 1,688 by landmark, 1,086 Wilderness.

## Three quirks in the source data

Each of these is silent if unhandled — the atlas still builds, it is just wrong.

**Facet names disagree between sources.** `Data/Locations/*.xml` spells them
`Ter Mur` and `Tokuno Islands`, while `<Map>` and `<Facet name>` say `TerMur` and
`Tokuno`. Unreconciled, the landmark bucket is keyed differently from the points
looking it up, so the fallback never fires and every unregioned spawn on those
facets reads "Wilderness".

This is reconciled **by matching, not by a lookup table** — there is no list of
facet names anywhere. `facetKey()` collapses spelling differences (lowercase,
alphanumerics only), and `resolveFacetName()` matches a loose spelling against
the canonical set discovered from the shard's own spawn and region data, by exact
key then by prefix in either direction. A name matching nothing keeps its own
name: forcing a wrong match would file a real custom facet's landmarks under the
wrong facet, which is worse than leaving it alone.

**Spawn type tokens carry XmlSpawner directives.** The `<Objects2>` type is not
always a bare class name:

```
Fairy,{RND,4,8}      alchemist/z/-50      Agralem/Name/Agralem
GargishRouser,1      greatape,true        GargishRefugee/hue/34532
```

Taken literally these invent creatures that do not exist *and* split real ones in
two, because `Fairy` and `Fairy,{RND,4,8}` slug apart into separate entries. 71 of
845 were affected. Everything from the first `/` or `,` is stripped, leaving 800
real creatures.

**Case is inconsistent across files.** The same creature is `Lizardman` in one
file and `lizardman` in another. Slugging collapses them correctly, but the
display name is chosen deterministically — most common spelling wins, ties break
to the more capitalised form, then alphabetically — because otherwise it would
depend on file read order and change on an unrelated restart.

## Tables

All are **import-owned**: a refresh empties and reloads them in one transaction,
so a failed reload leaves the previous atlas intact rather than a half-loaded
world. Nothing else writes to them and nothing holds a foreign key to them — no
FKs at all, consistent with every other `shard_*` table. Full column listings in
[`../modules/uo/SCHEMA.md`](../modules/uo/SCHEMA.md) — these are module-uo's tables, not core's.

| Table | Rows (stock) | Notes |
|---|---|---|
| `shard_spawn_creatures` | 800 | `slug` PK; `total` = sum of each type's own max; nullable `art` |
| `shard_spawn_points` | 6,455 | `spawn_range`, since `range` is reserved in MariaDB |
| `shard_spawn_point_types` | 23,927 | The many-to-many; one spawner commonly carries six types |
| `shard_regions` | 387 | Flattened out of the nesting; `rects` JSON |
| `shard_landmarks` | 558 | `grp`, since `group` is reserved in SQL |
| `shard_champion_spawns` | 25 | Configured altars, not the live feed |
| `shard_atlas_meta` | 1 | Singleton; source hashes, for the change check |
| `shard_atlas_pending` | 0–1 | Singleton; a staged refresh awaiting admin review |

`shard_spawn_creatures.name` carries a plain `INDEX`, deliberately **not
`FULLTEXT`**: ~800 rows makes a `LIKE` scan free, and FULLTEXT's minimum token
length would break searches for names like "orc".

The reload uses `DELETE`, not `TRUNCATE` — `TRUNCATE` is DDL in MariaDB and would
implicitly commit, defeating the all-or-nothing guarantee. Point ids are assigned
explicitly rather than left to `AUTO_INCREMENT`, because the join rows need them
and `conn.batch()` reports no usable `insertId`.

## Artwork — the shard extracts it now (Protocol 8)

**This project ships no creature art and no extraction tooling, and never will.**
UO sprites live in the operator's own client `.mul`/`.uop` files. They are the
operator's, not ours to redistribute.

What changed in protocol 8 is not that rule — it is who does the extracting. The
shard already has those files (a ServUO server cannot boot without a UO client),
so as of [`v8.md`](../link/v8.md) phase 3 it decodes them itself and hands the
pictures to the website over the bridge. Nobody installs UOFiddler and nobody
copies images to a web host.

**Admin → Shard → Import.** The import walks the shard's asset manifest, fetches
only the sprites whose hash changed, writes them under `uploads/atlas/`, asks the
shard for a body id per creature (§8 — the shard constructs the creature and
reads `Body.BodyID`, which is the only thing that is right for a shard's own
custom creatures) and points each `shard_spawn_creatures.art` at its picture.
Boot never calls the shard for this: the files change when an operator patches
their client, which is an event they know about and the site does not.

On this machine's stock client that is **1,022 creature portraits**, about a
megabyte in total — 787 out of the legacy `anim*.mul` files and 235 more out of
`AnimationFrame*.uop`, which ServUO's own decoder never opens
([`../link/v8.md`](../link/v8.md) §4.9).

**NULL stays a first-class state, and always will be.** An install with no shard
link has never imported one; a Linux shard host without `libgdiplus` cannot
render a sprite at all (a named `NO_IMAGING` status, not an error); and about
half the addressable body range has no art in any client file. Pages render
without images, which is normal and supported, not degraded.

### The operator's own artwork still wins

An operator who has drawn their own portraits keeps them. The map is unchanged:

1. Drop the images under `server/uploads/atlas/`.
2. Copy `server/db/data/spawnAtlas.art.example.json` to `spawnAtlas.art.json` and
   map creature slugs to file names.
3. Restart, or run the import.

`spawnAtlas.art.json` is applied **over** anything imported, per slug, so a sprite
rip never replaces a hand-drawn portrait on the next Update. Both it and
`server/uploads/` are gitignored, so neither the map nor the images can be
committed by accident.

### Why the imported art is not stored on the creature row

`shard_spawn_creatures` is emptied and refilled by every atlas refresh, and a
refresh happens on every boot. So the body ids and the imported files live in
`shard_creature_bodies` and `shard_assets`, outside that blast radius, and the
atlas import re-derives `art` from them on the way past. Storing it on the row
would mean an ordinary re-parse of the ServUO tree silently deleting every
portrait — with the next asset Update finding the client files unchanged,
reporting "nothing to do", and never putting them back.

## Code layout

| File | Role |
|---|---|
| `src/utils/spawnAtlasParse.js` | **Pure and fs-free** parsers, so CI covers them with no ServUO tree. Zero dependencies. |
| `src/utils/spawnAtlasSource.js` | The only thing that reads a ServUO tree; shared by the boot path and the CLI |
| `src/model/shardAtlas/shardAtlas.db.js` | The one-transaction replace |
| `src/model/shardAtlas/shardAtlas.model.js` | The refresh decision, staging, approve/reject |
| `scripts/importSpawnAtlas.js` | Thin CLI over the model |

Parsing notes:

- `Regions.xml`, `Locations/*.xml` and `ChampionSpawns.xml` genuinely nest, and
  get a small hand-rolled **subset** tokenizer — elements, attributes,
  self-closing tags, comments, the XML declaration, CDATA, and the five
  predefined entities plus numeric refs. It is not a general-purpose XML parser
  and must not be reused as one.
- The ~10.5 MB of `Spawns/*.xml` never touches that tokenizer. Those records are
  flat, so they get a streaming regex sweep instead; a DOM would allocate a node
  per element across ~40 fields on every record to keep 14 of them. **Do not put
  the Points files through a DOM parser.**
- `<Objects2>` is `Type:MX=n:SB=…` segments joined by `:OBJ=`. Split on `:OBJ=`
  *first* — a naive `split(':')` shreds it. A single Trammel point carries six
  types.
- **Respawn delays are stored in two different units, per record.** XmlSpawner
  writes `MinDelay`/`MaxDelay` in minutes, and switches to seconds only when a
  spawner's delay does not divide into whole minutes — flagging that with
  `DelayInSec` on the same record. A `5` therefore means five *minutes* on one
  spawner and five *seconds* on the next, and both are plausible respawn times,
  so a reader assuming either unit is silently wrong about the other. Stock
  ServUO 57.4 has ~170 second-flagged spawners out of 6,455. The parser
  normalises everything to **seconds**; the API and UI carry seconds throughout.

### The parser version

`spawnAtlasSource.js` exports `PARSER_VERSION`, stored in `shard_atlas_meta`
alongside the source hashes and bumped whenever the parser derives **different
data from identical files** — a fixed misreading, a new field, a changed unit.

A refresh re-derives when the tree changed **or** the parser did. Hashing the
tree alone would be a trap: an install whose maps never change would keep serving
whatever an older build derived, indefinitely, and a deploy that corrects the
parse would never reach the data. A version mismatch counts as drift, so the
correction lands on the next boot without an operator having to know it happened.

## The API

Everything is served from MariaDB. Nothing on this path touches the sidecar, so
the pages stay complete while the shard is down — which is why the routes sit at
`/api/v1/public/atlas` and **not** under `/public/shard`, where a prefix means
"sidecar-dependent". Unlike `/shard/*`, they *are* `siteMode`-gated, like
`/posts` and `/wiki`: a bestiary is site content and follows site content's rules.

Every route carries `requireFeature('atlas')` — **404** when an admin has
disabled the feature (its pages must not reveal that it exists) and **403** when
the caller sits below its configured audience. The default is `anonymous`, so the
gates are inert until an admin changes something. Responses are field-projected
like every other shard read; `atlas` declares no sensitive fields today, and the
projection call is there so the first one that does is covered by construction
rather than by a retrofit ([`v3.md` §3.6.1](../link/v3.md)).

| Route | Answers |
|---|---|
| `GET /atlas/creatures?q=&facet=&limit=&offset=` | The bestiary, most numerous first, paginated with an unpaginated `total` |
| `GET /atlas/creatures/:slug?facet=&points=` | One creature: `places`, `spawners`, `alsoHere` |
| `GET /atlas/regions?facet=&q=` | Named regions and their rectangles |
| `GET /atlas/landmarks?facet=&q=` | Points of interest, labelled by `group` |
| `GET /atlas/champions?facet=` | The configured altar roster |
| `GET /atlas/meta` | Facets, counts and when the atlas was parsed |

Two shapes worth knowing:

- **`places` is the aggregate the atlas exists for.** "Lizardman → Shrines,
  Isamu-Jima, Yew", grouped in SQL rather than by summing 6,455 point rows in
  Node. `spawners` is the raw list underneath it, bounded, with
  `spawnersTruncated` saying when it was cut.
- **`points` is a COUNT, `spawners` is the LIST.** The two are named apart
  deliberately: the same key meaning a number on the search route and an array on
  the detail route is the kind of thing a client only discovers in production.

`GET /atlas/meta` reports the **game world only**. The ServUO path, the per-file
hashes and any pending refresh describe the operator's filesystem, and live on
the admin route instead.

A facet is never validated against a list — nothing in the codebase names one.
`?facet=` is length-bounded and matched exactly, so an unknown name returns an
empty result rather than an error. The filter is an `EXISTS` over the points and
deliberately not a JSON path or `JSON_SEARCH` built from caller input: that
function treats `%` and `_` as wildcards, which would make `?facet=%` match
everything.

## The admin panel

**Admin → Spawn Atlas** (`/admin/shard-atlas`, admin-only — it reads a path on
the server's filesystem and replaces every atlas table, which is closer to a
deploy action than to moderation).

| Route | Does |
|---|---|
| `GET /admin/shard/atlas` | Status: path, readable, drift, counts, facets, pending |
| `POST /admin/shard/atlas/import` | Import now; `{ force: true }` ignores the hash gate |
| `POST /admin/shard/atlas/approve` | Apply a staged refresh, facet loss and all |
| `POST /admin/shard/atlas/reject` | Keep the current atlas; remember the decision |
| `PUT /admin/shard/atlas/path` | Point the atlas at a different tree |

Three behaviours that are deliberate:

- **An unreadable tree is a 200, not a 500.** `refresh()` reports outcomes rather
  than throwing, because the boot path must never be stopped by a bad tree, and
  that contract is preserved at the API. The panel says *"The tree could not be
  read: …"*; a 500 would say only that something broke.
- **Setting the path does not import.** Moving the mount and reloading the world
  are separate decisions, and an operator fixing a typo should not have a
  multi-thousand-row replace happen under them. The response carries fresh status
  so the panel can offer the import as the next step.
- **Every action is written to the admin activity log** (`shard.atlas.import` /
  `.approve` / `.reject` / `.path`).