docs/website/SPAWN_ATLAS.md

# Spawn atlas

**Status:** Complete on `edge` — data pipeline in website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112), API + pages in website [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113).
The source files stopped needing a shared filesystem in Protocol 8 phase 7.
**Design:** [`docs/link/v3.md` §6](../link/v3.md) — Protocol 3.0 Part C;
[`docs/link/v8.md` §10](../link/v8.md) — the sources over the bridge.

The spawn atlas is a browsable catalogue of what the shard *contains*: which
creatures spawn, where, how many, and which champion altars are configured. It
answers "where do I find a lizardman?" with **"Shrines, Yew, Isamu-Jima"** rather
than with a list of raw coordinates.

## Two things that shape the whole design

**The shard's ServUO tree is the single source of truth.** Nothing is
precomputed and committed to the repository. A shard's maps change over its
lifetime — facets get added, replaced, or renamed — and a snapshot in the repo
would silently drift from the world players actually see. The atlas is therefore
re-derived from the tree **on every server boot**.

**Facets are not a fixed list.** Nothing in the codebase names Felucca, Trammel,
or any other stock facet. The facet set is whatever the shard's own files
declare, discovered at parse time. A shard running entirely custom maps gets
exactly the same treatment as a stock one, with no code change.

## What it is not

The atlas is **static shard content, not live shard state.**

- It stays fully populated while the shard is down. Nothing here is an event,
  nothing subscribes, and no live feed feeds it.
- Its source files now *travel* over the sidecar (Protocol 8 phase 7, below), but
  only when an admin asks — on the request/reply path, never the event path.
  Until Protocol 8 this section said the atlas never touched the bridge at all,
  which was true and was bought at a price: the website had to be able to read
  the shard's filesystem.
- Its champion table (`shard_champion_spawns`) is the *configured roster* —
  "there is an Unholy Terror altar in Deceit". The live `champ.update` feed in
  `shard_champs` is the separate, sidecar-fed answer to "it is on level 3 right
  now". Both exist; do not conflate them.

Routes live at `/api/v1/public/atlas`, deliberately **not** under `/shard`,
because `/shard/*` means sidecar-dependent.

## Where the source files come from

Two ends, and **the shard wins whenever uo-link is configured and enabled**:

| Source | When it is used |
|---|---|
| **The shard, over uo-link** (Protocol 8 phase 7) | Whenever a shard is linked and enabled. Nothing to configure — the sidecar connection the site already has is the whole setup |
| A local ServUO tree | When there is no shard link: development, and same-host installs. Also a one-off `--servuo <path>`, which is an instruction and overrules the bridge |

With neither the atlas is simply skipped — the site runs normally without one.

**Why this changed.** Reading a ServUO tree required the website to have
filesystem access to the shard — "same host, a bind mount, or a shared volume" —
and that was the one place the platform's own rule (only the sidecar bridges the
shard) was broken, by the component that faces the internet. The shard now serves
the same five labelled groups over the same request/reply path as every other
shard read, and the parsers did not move: `spawnAtlasParse.js` is still pure,
still fs-free, and still covered by CI without a ServUO tree anywhere near it.

The local path remains, and remains configurable two ways, the setting winning
over the environment:

| Setting | Notes |
|---|---|
| `spawn_atlas_servuo_path` | Admin-editable; changes take effect on the next refresh without a redeploy |
| `SERVUO_PATH` env var | The deploy-time default, since the path usually describes a mount the deployment sets up |

### What crosses the wire, and what it costs

A stock 57.4 tree is **141 files and 11.34 MB**, and a spawn file is the awkward
part: `Spawns/trammel.xml` alone is 4.03 MB against the sidecar's **1 MiB inbound
line cap**. So a file crosses as **chunks of 512 KiB, each gzipped** —
`tree/Spawns/trammel.xml/c0` and so on, which is the same key-depth scheme the
asset families use.

Measured end to end against a live shard and sidecar:

| | |
|---|---|
| Files / bytes | 141 / 11,895,427 |
| Chunks / pages | 158 / 3 |
| On the wire | **1.33 MB** (the tree gzips ~12.5x) |
| Full import | **~0.5 s** |
| "Has anything changed?" | one manifest call, ~32 KB, **~70 ms** — no file bytes at all |

The shard serves this under its own switch, **`Bridge.TreeEnabled`**, separate
from `Bridge.AssetsEnabled`. The asset switch is an operator consenting to the
website reading their *UO client*; this one is about the shard's *own
configuration*, which they wrote. An operator can decline the first and still
publish a spawn atlas.

## The boot path

**On the bridge, boot imports nothing.** A local tree hashes in ~120 ms and
skips; asking the shard would put a sidecar round trip in the boot sequence to
answer a question whose answer is "no" on every restart that did not follow a map
edit. Editing spawn files is an operator action, so importing is one too:
**Admin → Spawn Atlas → Import now**, or the CLI. Whatever atlas is loaded keeps
serving until then. (Identical reasoning, and the same decision, as the cliloc
table — see [`CLILOCS.md`](CLILOCS.md).)

From a local tree it behaves as it always has: on every start the server hashes
the source files and compares them against what is loaded. Unchanged (the normal
case on a restart) costs one read pass, ~120 ms, and no database write. A real
change costs a ~400 ms parse and a reload.

Two contracts govern it:

**1. It never blocks startup.** No configured path, an unreadable mount, a
malformed file, a database error — every one is caught and logged, and the site
comes up serving whatever atlas it already had.

**2. A facet disappearing is never applied automatically.** Losing a facet looks
exactly like a half-copied or mid-update tree, and boot cannot tell that apart
from a real map change. That refresh is *staged* for a human instead. Everything
else — new facets, renamed regions, changed spawns — applies immediately, since
none of it can destroy something an operator would miss.

```
boot
 └─ path configured?            no ──▶ skip
     └─ tree readable?          no ──▶ warn, carry on
         └─ hashes changed?     no ──▶ done (nothing parsed)
             └─ parse
                 └─ a facet would be removed?
                      no ──▶ import
                     yes ──▶ stage for admin review; atlas unchanged
```

### Approving or rejecting a staged refresh

Only the *decision* is stored, never the parsed world — a few KB of source hashes
plus the facet diff. Approving **re-parses** the tree, so what lands matches the
tree at approval time rather than at boot, and a multi-megabyte blob never sits
in the database.

A rejection is remembered against those exact source hashes, so a declined
refresh does not re-prompt on every restart. Change the tree and the hashes
differ, which asks again.

From **Admin → Spawn Atlas**, or from the CLI:

```bash
cd website/server
npm run atlas:import -- --status     # what is loaded, and what is pending
npm run atlas:import -- --approve    # apply the staged refresh
npm run atlas:import -- --reject     # keep the current atlas, dismiss it
```

## The CLI

On a linked shard this and the admin panel are the *only* ways an import happens.
From a local tree the server also refreshes itself on boot, so there it is for
applying a map change *without* a restart, and for the approve/reject flow above.

```bash
npm run atlas:import                       # import if the source differs
npm run atlas:import -- --servuo <path>    # read this local tree for this run
npm run atlas:import -- --force            # reimport even if unchanged
```

`--servuo` is a per-run override and deliberately does **not** persist — changing
where the atlas permanently reads from is an admin action, not a side effect of a
one-off import.

## Sources

These five groups are the whole of it, at both ends: they are what the
filesystem reader walks and they are the only labels the shard will serve.

| File | Count (stock ServUO 57.4) | Used for |
|---|---|---|
| `Spawns/*.xml` | 13 files, 10.4 MB | Every spawner: location, size, delays, time-of-day, creature types |
| `Data/Regions.xml` | 129 KB, nested | Named regions and their rectangles |
| `Data/Locations/*.xml` | 6 files, 37 KB | Landmarks (dungeon levels, town markers) |
| `Config/ChampionSpawns.xml` | 4.8 KB | Configured champion altars |
| `Data/Decoration/**/*.cfg` | 120 files, 1.28 MB, nested two deep | The decoration vocabulary world authoring offers |

A fetch for anything outside them is refused by name, and **the shard never joins
a path that arrived on the wire** — a request names a label the shard itself
enumerated, or it is answered `absent`. Two of those 120 decoration files are
**zero bytes** on a stock tree, which is worth knowing because it broke the first
implementation: .NET's `GZipStream` emits nothing at all for zero bytes of input,
which is not a valid gzip stream.

**A stock tree has 13 spawn files but only 6 facets.** `Eodon.xml`,
`GravewaterLake.xml`, `TreasuresOfKotl.xml` and the other named-area files hold
TerMur/Trammel points. The facet always comes from each record's own `<Map>`,
never from the file name.

## How a coordinate becomes a place name

This is the transform the atlas exists for, in `resolveRegion()`:

1. The highest-`priority` named region whose rectangle contains the point. Ties
   break toward the **smallest** rect, so a specific room wins over the
   dungeon-wide rect enclosing it.
2. Otherwise the nearest landmark within the landmark radius (200 tiles by
   default), labelled by its **group** ("Covetous"), not its individual marker
   ("Level 1").
3. Otherwise `"Wilderness"`.

The radius cap in step 2 is what keeps step 3 reachable. Without it the nearest
landmark is always *some* landmark however far away, and open countryside gets
labelled with a dungeon on the far side of the map.

Against stock ServUO this resolves **83.2%** of points (5,369 of 6,455): 3,681 by
region, 1,688 by landmark, 1,086 Wilderness.

## Three quirks in the source data

Each of these is silent if unhandled — the atlas still builds, it is just wrong.

**Facet names disagree between sources.** `Data/Locations/*.xml` spells them
`Ter Mur` and `Tokuno Islands`, while `<Map>` and `<Facet name>` say `TerMur` and
`Tokuno`. Unreconciled, the landmark bucket is keyed differently from the points
looking it up, so the fallback never fires and every unregioned spawn on those
facets reads "Wilderness".

This is reconciled **by matching, not by a lookup table** — there is no list of
facet names anywhere. `facetKey()` collapses spelling differences (lowercase,
alphanumerics only), and `resolveFacetName()` matches a loose spelling against
the canonical set discovered from the shard's own spawn and region data, by exact
key then by prefix in either direction. A name matching nothing keeps its own
name: forcing a wrong match would file a real custom facet's landmarks under the
wrong facet, which is worse than leaving it alone.

**Spawn type tokens carry XmlSpawner directives.** The `<Objects2>` type is not
always a bare class name:

```
Fairy,{RND,4,8}      alchemist/z/-50      Agralem/Name/Agralem
GargishRouser,1      greatape,true        GargishRefugee/hue/34532
```

Taken literally these invent creatures that do not exist *and* split real ones in
two, because `Fairy` and `Fairy,{RND,4,8}` slug apart into separate entries. 71 of
845 were affected. Everything from the first `/` or `,` is stripped, leaving 800
real creatures.

**Case is inconsistent across files.** The same creature is `Lizardman` in one
file and `lizardman` in another. Slugging collapses them correctly, but the
display name is chosen deterministically — most common spelling wins, ties break
to the more capitalised form, then alphabetically — because otherwise it would
depend on file read order and change on an unrelated restart.

## Tables

All are **import-owned**: a refresh empties and reloads them in one transaction,
so a failed reload leaves the previous atlas intact rather than a half-loaded
world. Nothing else writes to them and nothing holds a foreign key to them — no
FKs at all, consistent with every other `shard_*` table. Full column listings in
[`../modules/uo/SCHEMA.md`](../modules/uo/SCHEMA.md) — these are module-uo's tables, not core's.

| Table | Rows (stock) | Notes |
|---|---|---|
| `shard_spawn_creatures` | 800 | `slug` PK; `total` = sum of each type's own max; nullable `art` |
| `shard_spawn_points` | 6,455 | `spawn_range`, since `range` is reserved in MariaDB |
| `shard_spawn_point_types` | 23,927 | The many-to-many; one spawner commonly carries six types |
| `shard_regions` | 387 | Flattened out of the nesting; `rects` JSON |
| `shard_landmarks` | 558 | `grp`, since `group` is reserved in SQL |
| `shard_champion_spawns` | 25 | Configured altars, not the live feed |
| `shard_atlas_meta` | 1 | Singleton; source hashes, for the change check |
| `shard_atlas_pending` | 0–1 | Singleton; a staged refresh awaiting admin review |

`shard_spawn_creatures.name` carries a plain `INDEX`, deliberately **not
`FULLTEXT`**: ~800 rows makes a `LIKE` scan free, and FULLTEXT's minimum token
length would break searches for names like "orc".

The reload uses `DELETE`, not `TRUNCATE` — `TRUNCATE` is DDL in MariaDB and would
implicitly commit, defeating the all-or-nothing guarantee. Point ids are assigned
explicitly rather than left to `AUTO_INCREMENT`, because the join rows need them
and `conn.batch()` reports no usable `insertId`.

## Artwork — the shard extracts it now (Protocol 8)

**This project ships no creature art and no extraction tooling, and never will.**
UO sprites live in the operator's own client `.mul`/`.uop` files. They are the
operator's, not ours to redistribute.

What changed in protocol 8 is not that rule — it is who does the extracting. The
shard already has those files (a ServUO server cannot boot without a UO client),
so as of [`v8.md`](../link/v8.md) phase 3 it decodes them itself and hands the
pictures to the website over the bridge. Nobody installs UOFiddler and nobody
copies images to a web host.

**Admin → Client Files → Update.** The import walks the shard's asset manifest, fetches
only the sprites whose hash changed, writes them under `uploads/atlas/`, asks the
shard for a body id per creature (§8 — the shard constructs the creature and
reads `Body.BodyID`, which is the only thing that is right for a shard's own
custom creatures) and points each `shard_spawn_creatures.art` at its picture.
Boot never calls the shard for this: the files change when an operator patches
their client, which is an event they know about and the site does not.

On this machine's stock client that is **1,095 creature portraits**, about a
megabyte in total — 787 out of the legacy `anim*.mul` files, 235 more out of
`AnimationFrame*.uop`, which ServUO's own decoder never opens
([`../link/v8.md`](../link/v8.md) §4.9), and 73 more that have no art at the
walk's first action and real art at a later one, which the import now falls back
to (§11.2). Body 820 is one of them, and it is a horse.

**NULL stays a first-class state, and always will be.** An install with no shard
link has never imported one; a Linux shard host without `libgdiplus` cannot
render a sprite at all (a named `NO_IMAGING` status, not an error); and about
half the addressable body range has no art in any client file. Pages render
without images, which is normal and supported, not degraded.

### The operator's own artwork still wins

An operator who has drawn their own portraits keeps them. The map is unchanged:

1. Drop the images under `server/uploads/atlas/`.
2. Copy `server/db/data/spawnAtlas.art.example.json` to `spawnAtlas.art.json` and
   map creature slugs to file names.
3. Restart, or run the import.

`spawnAtlas.art.json` is applied **over** anything imported, per slug, so a sprite
rip never replaces a hand-drawn portrait on the next Update. Both it and
`server/uploads/` are gitignored, so neither the map nor the images can be
committed by accident.

### Why the imported art is not stored on the creature row

`shard_spawn_creatures` is emptied and refilled by every atlas refresh, and a
refresh happens on every boot. So the body ids and the imported files live in
`shard_creature_bodies` and `shard_assets`, outside that blast radius, and the
atlas import re-derives `art` from them on the way past. Storing it on the row
would mean an ordinary re-parse of the ServUO tree silently deleting every
portrait — with the next asset Update finding the client files unchanged,
reporting "nothing to do", and never putting them back.

## Code layout

| File | Role |
|---|---|
| `src/utils/spawnAtlasParse.js` | **Pure and fs-free** parsers, so CI covers them with no ServUO tree. Zero dependencies. |
| `src/utils/spawnAtlasSource.js` | The only thing that reads a ServUO tree; shared by the boot path and the CLI |
| `src/model/shardAtlas/shardAtlas.db.js` | The one-transaction replace |
| `src/model/shardAtlas/shardAtlas.model.js` | The refresh decision, staging, approve/reject |
| `scripts/importSpawnAtlas.js` | Thin CLI over the model |

Parsing notes:

- `Regions.xml`, `Locations/*.xml` and `ChampionSpawns.xml` genuinely nest, and
  get a small hand-rolled **subset** tokenizer — elements, attributes,
  self-closing tags, comments, the XML declaration, CDATA, and the five
  predefined entities plus numeric refs. It is not a general-purpose XML parser
  and must not be reused as one.
- The ~10.5 MB of `Spawns/*.xml` never touches that tokenizer. Those records are
  flat, so they get a streaming regex sweep instead; a DOM would allocate a node
  per element across ~40 fields on every record to keep 14 of them. **Do not put
  the Points files through a DOM parser.**
- `<Objects2>` is `Type:MX=n:SB=…` segments joined by `:OBJ=`. Split on `:OBJ=`
  *first* — a naive `split(':')` shreds it. A single Trammel point carries six
  types.
- **Respawn delays are stored in two different units, per record.** XmlSpawner
  writes `MinDelay`/`MaxDelay` in minutes, and switches to seconds only when a
  spawner's delay does not divide into whole minutes — flagging that with
  `DelayInSec` on the same record. A `5` therefore means five *minutes* on one
  spawner and five *seconds* on the next, and both are plausible respawn times,
  so a reader assuming either unit is silently wrong about the other. Stock
  ServUO 57.4 has ~170 second-flagged spawners out of 6,455. The parser
  normalises everything to **seconds**; the API and UI carry seconds throughout.

### The parser version

`spawnAtlasSource.js` exports `PARSER_VERSION`, stored in `shard_atlas_meta`
alongside the source hashes and bumped whenever the parser derives **different
data from identical files** — a fixed misreading, a new field, a changed unit.

A refresh re-derives when the tree changed **or** the parser did. Hashing the
tree alone would be a trap: an install whose maps never change would keep serving
whatever an older build derived, indefinitely, and a deploy that corrects the
parse would never reach the data. A version mismatch counts as drift, so the
correction lands on the next boot without an operator having to know it happened.

It is **6** as of Protocol 8 phase 7: the source files are now parsed in one
canonical label order whichever end read them. That matters because the parse is
order-sensitive in one place — the decoration index keeps the first item id it
sees for a type, and the two readers sorted a nested directory differently, so the
same tree could yield a different preview graphic depending on how it arrived.
Identical files, a different answer for a handful of types: precisely what this
number exists to push through the hash gate.

That change was written as **5** while 5 was being released from `main` meaning
something else — a spawn point keeping its `UniqueId` (v1.2.2) — so the Asset
Bridge cutover renumbered it to 6. The renumber is the mechanism working rather
than bookkeeping: an install that imported under v1.2.2 already stores 5, so a
build declaring 5 for a *different* derivation would have been called current and
the correction would have reached nobody already running. Two branches bumping
the same counter for different reasons is the one way this gate can be defeated,
and a merge is where it has to be caught.

The **source fingerprint is taken over raw bytes** at both ends for the same
reason. Hashing the decoded text would hash a UTF-8 *re-encoding* of the file —
identical for valid UTF-8, and different for a file that is not, because an
undecodable byte becomes U+FFFD and never comes back. One Latin-1 character in a
creature name would then fingerprint differently depending on which end read it,
and the drift gate would report a change on every import, forever, with the tree
untouched.

## The API

Everything is served from MariaDB. Nothing on the *read* path touches the sidecar
(the source files reach the database at import time and stay there), so the pages
stay complete while the shard is down — which is why the routes sit at
`/api/v1/public/atlas` and **not** under `/public/shard`, where a prefix means
"sidecar-dependent". Unlike `/shard/*`, they *are* `siteMode`-gated, like
`/posts` and `/wiki`: a bestiary is site content and follows site content's rules.

Every route carries `requireFeature('atlas')` — **404** when an admin has
disabled the feature (its pages must not reveal that it exists) and **403** when
the caller sits below its configured audience. The default is `anonymous`, so the
gates are inert until an admin changes something. Responses are field-projected
like every other shard read; `atlas` declares no sensitive fields today, and the
projection call is there so the first one that does is covered by construction
rather than by a retrofit ([`v3.md` §3.6.1](../link/v3.md)).

| Route | Answers |
|---|---|
| `GET /atlas/creatures?q=&facet=&limit=&offset=` | The bestiary, most numerous first, paginated with an unpaginated `total` |
| `GET /atlas/creatures/:slug?facet=&points=` | One creature: `places`, `spawners`, `alsoHere` |
| `GET /atlas/regions?facet=&q=` | Named regions and their rectangles |
| `GET /atlas/landmarks?facet=&q=` | Points of interest, labelled by `group` |
| `GET /atlas/champions?facet=` | The configured altar roster |
| `GET /atlas/meta` | Facets, counts and when the atlas was parsed |

Two shapes worth knowing:

- **`places` is the aggregate the atlas exists for.** "Lizardman → Shrines,
  Isamu-Jima, Yew", grouped in SQL rather than by summing 6,455 point rows in
  Node. `spawners` is the raw list underneath it, bounded, with
  `spawnersTruncated` saying when it was cut.
- **`points` is a COUNT, `spawners` is the LIST.** The two are named apart
  deliberately: the same key meaning a number on the search route and an array on
  the detail route is the kind of thing a client only discovers in production.

`GET /atlas/meta` reports the **game world only**. The ServUO path, the per-file
hashes and any pending refresh describe the operator's filesystem, and live on
the admin route instead.

A facet is never validated against a list — nothing in the codebase names one.
`?facet=` is length-bounded and matched exactly, so an unknown name returns an
empty result rather than an error. The filter is an `EXISTS` over the points and
deliberately not a JSON path or `JSON_SEARCH` built from caller input: that
function treats `%` and `_` as wildcards, which would make `?facet=%` match
everything.

## The admin panel

**Admin → Spawn Atlas** (`/admin/shard-atlas`, admin-only — it reads a path on
the server's filesystem and replaces every atlas table, which is closer to a
deploy action than to moderation).

| Route | Does |
|---|---|
| `GET /admin/shard/atlas` | Status: path, readable, drift, counts, facets, pending |
| `POST /admin/shard/atlas/import` | Import now; `{ force: true }` ignores the hash gate |
| `POST /admin/shard/atlas/approve` | Apply a staged refresh, facet loss and all |
| `POST /admin/shard/atlas/reject` | Keep the current atlas; remember the decision |
| `PUT /admin/shard/atlas/path` | Point the atlas at a different tree |

Three behaviours that are deliberate:

- **An unreadable tree is a 200, not a 500.** `refresh()` reports outcomes rather
  than throwing, because the boot path must never be stopped by a bad tree, and
  that contract is preserved at the API. The panel says *"The tree could not be
  read: …"*; a 500 would say only that something broke.
- **Setting the path does not import.** Moving the mount and reloading the world
  are separate decisions, and an operator fixing a typo should not have a
  multi-thousand-row replace happen under them. The response carries fresh status
  so the panel can offer the import as the next step.
- **Every action is written to the admin activity log** (`shard.atlas.import` /
  `.approve` / `.reject` / `.path`).