docs/website/SPAWN_ATLAS.md

Spawn atlas

Status: Complete on edge — data pipeline in website #112, API + pages in website #113. The source files stopped needing a shared filesystem in Protocol 8 phase 7. Design: docs/link/v3.md §6 — Protocol 3.0 Part C; docs/link/v8.md §10 — the sources over the bridge.

The spawn atlas is a browsable catalogue of what the shard contains: which creatures spawn, where, how many, and which champion altars are configured. It answers "where do I find a lizardman?" with "Shrines, Yew, Isamu-Jima" rather than with a list of raw coordinates.

Two things that shape the whole design

The shard's ServUO tree is the single source of truth. Nothing is precomputed and committed to the repository. A shard's maps change over its lifetime — facets get added, replaced, or renamed — and a snapshot in the repo would silently drift from the world players actually see. The atlas is therefore re-derived from the tree on every server boot.

Facets are not a fixed list. Nothing in the codebase names Felucca, Trammel, or any other stock facet. The facet set is whatever the shard's own files declare, discovered at parse time. A shard running entirely custom maps gets exactly the same treatment as a stock one, with no code change.

What it is not

The atlas is static shard content, not live shard state.

• It stays fully populated while the shard is down. Nothing here is an event, nothing subscribes, and no live feed feeds it.
• Its source files now travel over the sidecar (Protocol 8 phase 7, below), but only when an admin asks — on the request/reply path, never the event path. Until Protocol 8 this section said the atlas never touched the bridge at all, which was true and was bought at a price: the website had to be able to read the shard's filesystem.
• Its champion table (shard_champion_spawns) is the configured roster — "there is an Unholy Terror altar in Deceit". The live champ.update feed in shard_champs is the separate, sidecar-fed answer to "it is on level 3 right now". Both exist; do not conflate them.

Routes live at /api/v1/public/atlas, deliberately not under /shard, because /shard/* means sidecar-dependent.

Where the source files come from

Two ends, and the shard wins whenever uo-link is configured and enabled:

Source	When it is used
The shard, over uo-link (Protocol 8 phase 7)	Whenever a shard is linked and enabled. Nothing to configure — the sidecar connection the site already has is the whole setup
A local ServUO tree	When there is no shard link: development, and same-host installs. Also a one-off --servuo <path>, which is an instruction and overrules the bridge

With neither the atlas is simply skipped — the site runs normally without one.

Why this changed. Reading a ServUO tree required the website to have filesystem access to the shard — "same host, a bind mount, or a shared volume" — and that was the one place the platform's own rule (only the sidecar bridges the shard) was broken, by the component that faces the internet. The shard now serves the same five labelled groups over the same request/reply path as every other shard read, and the parsers did not move: spawnAtlasParse.js is still pure, still fs-free, and still covered by CI without a ServUO tree anywhere near it.

The local path remains, and remains configurable two ways, the setting winning over the environment:

Setting	Notes
spawn_atlas_servuo_path	Admin-editable; changes take effect on the next refresh without a redeploy
SERVUO_PATH env var	The deploy-time default, since the path usually describes a mount the deployment sets up

What crosses the wire, and what it costs

A stock 57.4 tree is 141 files and 11.34 MB, and a spawn file is the awkward part: Spawns/trammel.xml alone is 4.03 MB against the sidecar's 1 MiB inbound line cap. So a file crosses as chunks of 512 KiB, each gzipped — tree/Spawns/trammel.xml/c0 and so on, which is the same key-depth scheme the asset families use.

Measured end to end against a live shard and sidecar:

Files / bytes	141 / 11,895,427
Chunks / pages	158 / 3
On the wire	1.33 MB (the tree gzips ~12.5x)
Full import	~0.5 s
"Has anything changed?"	one manifest call, ~32 KB, ~70 ms — no file bytes at all

The shard serves this under its own switch, Bridge.TreeEnabled, separate from Bridge.AssetsEnabled. The asset switch is an operator consenting to the website reading their UO client; this one is about the shard's own configuration, which they wrote. An operator can decline the first and still publish a spawn atlas.

The boot path

On the bridge, boot imports nothing. A local tree hashes in ~120 ms and skips; asking the shard would put a sidecar round trip in the boot sequence to answer a question whose answer is "no" on every restart that did not follow a map edit. Editing spawn files is an operator action, so importing is one too: Admin → Spawn Atlas → Import now, or the CLI. Whatever atlas is loaded keeps serving until then. (Identical reasoning, and the same decision, as the cliloc table — see CLILOCS.md.)

From a local tree it behaves as it always has: on every start the server hashes the source files and compares them against what is loaded. Unchanged (the normal case on a restart) costs one read pass, ~120 ms, and no database write. A real change costs a ~400 ms parse and a reload.

Two contracts govern it:

1. It never blocks startup. No configured path, an unreadable mount, a malformed file, a database error — every one is caught and logged, and the site comes up serving whatever atlas it already had.

2. A facet disappearing is never applied automatically. Losing a facet looks exactly like a half-copied or mid-update tree, and boot cannot tell that apart from a real map change. That refresh is staged for a human instead. Everything else — new facets, renamed regions, changed spawns — applies immediately, since none of it can destroy something an operator would miss.

boot
 └─ path configured?            no ──▶ skip
     └─ tree readable?          no ──▶ warn, carry on
         └─ hashes changed?     no ──▶ done (nothing parsed)
             └─ parse
                 └─ a facet would be removed?
                      no ──▶ import
                     yes ──▶ stage for admin review; atlas unchanged

Approving or rejecting a staged refresh

Only the decision is stored, never the parsed world — a few KB of source hashes plus the facet diff. Approving re-parses the tree, so what lands matches the tree at approval time rather than at boot, and a multi-megabyte blob never sits in the database.

A rejection is remembered against those exact source hashes, so a declined refresh does not re-prompt on every restart. Change the tree and the hashes differ, which asks again.

From Admin → Spawn Atlas, or from the CLI:

cd website/server
npm run atlas:import -- --status     # what is loaded, and what is pending
npm run atlas:import -- --approve    # apply the staged refresh
npm run atlas:import -- --reject     # keep the current atlas, dismiss it

The CLI

On a linked shard this and the admin panel are the only ways an import happens. From a local tree the server also refreshes itself on boot, so there it is for applying a map change without a restart, and for the approve/reject flow above.

npm run atlas:import                       # import if the source differs
npm run atlas:import -- --servuo <path>    # read this local tree for this run
npm run atlas:import -- --force            # reimport even if unchanged

--servuo is a per-run override and deliberately does not persist — changing where the atlas permanently reads from is an admin action, not a side effect of a one-off import.

Sources

These five groups are the whole of it, at both ends: they are what the filesystem reader walks and they are the only labels the shard will serve.

File	Count (stock ServUO 57.4)	Used for
Spawns/*.xml	13 files, 10.4 MB	Every spawner: location, size, delays, time-of-day, creature types
Data/Regions.xml	129 KB, nested	Named regions and their rectangles
Data/Locations/*.xml	6 files, 37 KB	Landmarks (dungeon levels, town markers)
Config/ChampionSpawns.xml	4.8 KB	Configured champion altars
Data/Decoration/**/*.cfg	120 files, 1.28 MB, nested two deep	The decoration vocabulary world authoring offers

A fetch for anything outside them is refused by name, and the shard never joins a path that arrived on the wire — a request names a label the shard itself enumerated, or it is answered absent. Two of those 120 decoration files are zero bytes on a stock tree, which is worth knowing because it broke the first implementation: .NET's GZipStream emits nothing at all for zero bytes of input, which is not a valid gzip stream.

A stock tree has 13 spawn files but only 6 facets. Eodon.xml, GravewaterLake.xml, TreasuresOfKotl.xml and the other named-area files hold TerMur/Trammel points. The facet always comes from each record's own <Map>, never from the file name.

How a coordinate becomes a place name

This is the transform the atlas exists for, in resolveRegion():

1. The highest-priority named region whose rectangle contains the point. Ties break toward the smallest rect, so a specific room wins over the dungeon-wide rect enclosing it.
2. Otherwise the nearest landmark within the landmark radius (200 tiles by default), labelled by its group ("Covetous"), not its individual marker ("Level 1").
3. Otherwise "Wilderness".

The radius cap in step 2 is what keeps step 3 reachable. Without it the nearest landmark is always some landmark however far away, and open countryside gets labelled with a dungeon on the far side of the map.

Against stock ServUO this resolves 83.2% of points (5,369 of 6,455): 3,681 by region, 1,688 by landmark, 1,086 Wilderness.

Three quirks in the source data

Each of these is silent if unhandled — the atlas still builds, it is just wrong.

Facet names disagree between sources. Data/Locations/*.xml spells them Ter Mur and Tokuno Islands, while <Map> and <Facet name> say TerMur and Tokuno. Unreconciled, the landmark bucket is keyed differently from the points looking it up, so the fallback never fires and every unregioned spawn on those facets reads "Wilderness".

This is reconciled by matching, not by a lookup table — there is no list of facet names anywhere. facetKey() collapses spelling differences (lowercase, alphanumerics only), and resolveFacetName() matches a loose spelling against the canonical set discovered from the shard's own spawn and region data, by exact key then by prefix in either direction. A name matching nothing keeps its own name: forcing a wrong match would file a real custom facet's landmarks under the wrong facet, which is worse than leaving it alone.

Spawn type tokens carry XmlSpawner directives. The <Objects2> type is not always a bare class name:

Fairy,{RND,4,8}      alchemist/z/-50      Agralem/Name/Agralem
GargishRouser,1      greatape,true        GargishRefugee/hue/34532

Taken literally these invent creatures that do not exist and split real ones in two, because Fairy and Fairy,{RND,4,8} slug apart into separate entries. 71 of 845 were affected. Everything from the first / or , is stripped, leaving 800 real creatures.

Case is inconsistent across files. The same creature is Lizardman in one file and lizardman in another. Slugging collapses them correctly, but the display name is chosen deterministically — most common spelling wins, ties break to the more capitalised form, then alphabetically — because otherwise it would depend on file read order and change on an unrelated restart.

Tables

All are import-owned: a refresh empties and reloads them in one transaction, so a failed reload leaves the previous atlas intact rather than a half-loaded world. Nothing else writes to them and nothing holds a foreign key to them — no FKs at all, consistent with every other shard_* table. Full column listings in ../modules/uo/SCHEMA.md — these are module-uo's tables, not core's.

Table	Rows (stock)	Notes
shard_spawn_creatures	800	slug PK; total = sum of each type's own max; nullable art
shard_spawn_points	6,455	spawn_range, since range is reserved in MariaDB
shard_spawn_point_types	23,927	The many-to-many; one spawner commonly carries six types
shard_regions	387	Flattened out of the nesting; rects JSON
shard_landmarks	558	grp, since group is reserved in SQL
shard_champion_spawns	25	Configured altars, not the live feed
shard_atlas_meta	1	Singleton; source hashes, for the change check
shard_atlas_pending	0–1	Singleton; a staged refresh awaiting admin review

shard_spawn_creatures.name carries a plain INDEX, deliberately not FULLTEXT: ~800 rows makes a LIKE scan free, and FULLTEXT's minimum token length would break searches for names like "orc".

The reload uses DELETE, not TRUNCATE — TRUNCATE is DDL in MariaDB and would implicitly commit, defeating the all-or-nothing guarantee. Point ids are assigned explicitly rather than left to AUTO_INCREMENT, because the join rows need them and conn.batch() reports no usable insertId.

Artwork — the shard extracts it now (Protocol 8)

This project ships no creature art and no extraction tooling, and never will. UO sprites live in the operator's own client .mul/.uop files. They are the operator's, not ours to redistribute.

What changed in protocol 8 is not that rule — it is who does the extracting. The shard already has those files (a ServUO server cannot boot without a UO client), so as of v8.md phase 3 it decodes them itself and hands the pictures to the website over the bridge. Nobody installs UOFiddler and nobody copies images to a web host.

Admin → Client Files → Update. The import walks the shard's asset manifest, fetches only the sprites whose hash changed, writes them under uploads/atlas/, asks the shard for a body id per creature (§8 — the shard constructs the creature and reads Body.BodyID, which is the only thing that is right for a shard's own custom creatures) and points each shard_spawn_creatures.art at its picture. Boot never calls the shard for this: the files change when an operator patches their client, which is an event they know about and the site does not.

On this machine's stock client that is 1,095 creature portraits, about a megabyte in total — 787 out of the legacy anim*.mul files, 235 more out of AnimationFrame*.uop, which ServUO's own decoder never opens (../link/v8.md §4.9), and 73 more that have no art at the walk's first action and real art at a later one, which the import now falls back to (§11.2). Body 820 is one of them, and it is a horse.

NULL stays a first-class state, and always will be. An install with no shard link has never imported one; a Linux shard host without libgdiplus cannot render a sprite at all (a named NO_IMAGING status, not an error); and about half the addressable body range has no art in any client file. Pages render without images, which is normal and supported, not degraded.

The operator's own artwork still wins

An operator who has drawn their own portraits keeps them. The map is unchanged:

1. Drop the images under server/uploads/atlas/.
2. Copy server/db/data/spawnAtlas.art.example.json to spawnAtlas.art.json and map creature slugs to file names.
3. Restart, or run the import.

spawnAtlas.art.json is applied over anything imported, per slug, so a sprite rip never replaces a hand-drawn portrait on the next Update. Both it and server/uploads/ are gitignored, so neither the map nor the images can be committed by accident.

Why the imported art is not stored on the creature row

shard_spawn_creatures is emptied and refilled by every atlas refresh, and a refresh happens on every boot. So the body ids and the imported files live in shard_creature_bodies and shard_assets, outside that blast radius, and the atlas import re-derives art from them on the way past. Storing it on the row would mean an ordinary re-parse of the ServUO tree silently deleting every portrait — with the next asset Update finding the client files unchanged, reporting "nothing to do", and never putting them back.

Code layout

File	Role
src/utils/spawnAtlasParse.js	Pure and fs-free parsers, so CI covers them with no ServUO tree. Zero dependencies.
src/utils/spawnAtlasSource.js	The only thing that reads a ServUO tree; shared by the boot path and the CLI
src/model/shardAtlas/shardAtlas.db.js	The one-transaction replace
src/model/shardAtlas/shardAtlas.model.js	The refresh decision, staging, approve/reject
scripts/importSpawnAtlas.js	Thin CLI over the model

Parsing notes:

• Regions.xml, Locations/*.xml and ChampionSpawns.xml genuinely nest, and get a small hand-rolled subset tokenizer — elements, attributes, self-closing tags, comments, the XML declaration, CDATA, and the five predefined entities plus numeric refs. It is not a general-purpose XML parser and must not be reused as one.
• The ~10.5 MB of Spawns/*.xml never touches that tokenizer. Those records are flat, so they get a streaming regex sweep instead; a DOM would allocate a node per element across ~40 fields on every record to keep 14 of them. Do not put the Points files through a DOM parser.
• <Objects2> is Type:MX=n:SB=… segments joined by :OBJ=. Split on :OBJ= first — a naive split(':') shreds it. A single Trammel point carries six types.
• Respawn delays are stored in two different units, per record. XmlSpawner writes MinDelay/MaxDelay in minutes, and switches to seconds only when a spawner's delay does not divide into whole minutes — flagging that with DelayInSec on the same record. A 5 therefore means five minutes on one spawner and five seconds on the next, and both are plausible respawn times, so a reader assuming either unit is silently wrong about the other. Stock ServUO 57.4 has ~170 second-flagged spawners out of 6,455. The parser normalises everything to seconds; the API and UI carry seconds throughout.

The parser version

spawnAtlasSource.js exports PARSER_VERSION, stored in shard_atlas_meta alongside the source hashes and bumped whenever the parser derives different data from identical files — a fixed misreading, a new field, a changed unit.

A refresh re-derives when the tree changed or the parser did. Hashing the tree alone would be a trap: an install whose maps never change would keep serving whatever an older build derived, indefinitely, and a deploy that corrects the parse would never reach the data. A version mismatch counts as drift, so the correction lands on the next boot without an operator having to know it happened.

It is 6 as of Protocol 8 phase 7: the source files are now parsed in one canonical label order whichever end read them. That matters because the parse is order-sensitive in one place — the decoration index keeps the first item id it sees for a type, and the two readers sorted a nested directory differently, so the same tree could yield a different preview graphic depending on how it arrived. Identical files, a different answer for a handful of types: precisely what this number exists to push through the hash gate.

That change was written as 5 while 5 was being released from main meaning something else — a spawn point keeping its UniqueId (v1.2.2) — so the Asset Bridge cutover renumbered it to 6. The renumber is the mechanism working rather than bookkeeping: an install that imported under v1.2.2 already stores 5, so a build declaring 5 for a different derivation would have been called current and the correction would have reached nobody already running. Two branches bumping the same counter for different reasons is the one way this gate can be defeated, and a merge is where it has to be caught.

The source fingerprint is taken over raw bytes at both ends for the same reason. Hashing the decoded text would hash a UTF-8 re-encoding of the file — identical for valid UTF-8, and different for a file that is not, because an undecodable byte becomes U+FFFD and never comes back. One Latin-1 character in a creature name would then fingerprint differently depending on which end read it, and the drift gate would report a change on every import, forever, with the tree untouched.

The API

Everything is served from MariaDB. Nothing on the read path touches the sidecar (the source files reach the database at import time and stay there), so the pages stay complete while the shard is down — which is why the routes sit at /api/v1/public/atlas and not under /public/shard, where a prefix means "sidecar-dependent". Unlike /shard/*, they are siteMode-gated, like /posts and /wiki: a bestiary is site content and follows site content's rules.

Every route carries requireFeature('atlas') — 404 when an admin has disabled the feature (its pages must not reveal that it exists) and 403 when the caller sits below its configured audience. The default is anonymous, so the gates are inert until an admin changes something. Responses are field-projected like every other shard read; atlas declares no sensitive fields today, and the projection call is there so the first one that does is covered by construction rather than by a retrofit (v3.md §3.6.1).

Route	Answers
GET /atlas/creatures?q=&facet=&limit=&offset=	The bestiary, most numerous first, paginated with an unpaginated total
GET /atlas/creatures/:slug?facet=&points=	One creature: places, spawners, alsoHere
GET /atlas/regions?facet=&q=	Named regions and their rectangles
GET /atlas/landmarks?facet=&q=	Points of interest, labelled by group
GET /atlas/champions?facet=	The configured altar roster
GET /atlas/meta	Facets, counts and when the atlas was parsed

Two shapes worth knowing:

• places is the aggregate the atlas exists for. "Lizardman → Shrines, Isamu-Jima, Yew", grouped in SQL rather than by summing 6,455 point rows in Node. spawners is the raw list underneath it, bounded, with spawnersTruncated saying when it was cut.
• points is a COUNT, spawners is the LIST. The two are named apart deliberately: the same key meaning a number on the search route and an array on the detail route is the kind of thing a client only discovers in production.

GET /atlas/meta reports the game world only. The ServUO path, the per-file hashes and any pending refresh describe the operator's filesystem, and live on the admin route instead.

A facet is never validated against a list — nothing in the codebase names one. ?facet= is length-bounded and matched exactly, so an unknown name returns an empty result rather than an error. The filter is an EXISTS over the points and deliberately not a JSON path or JSON_SEARCH built from caller input: that function treats % and _ as wildcards, which would make ?facet=% match everything.

The admin panel

Admin → Spawn Atlas (/admin/shard-atlas, admin-only — it reads a path on the server's filesystem and replaces every atlas table, which is closer to a deploy action than to moderation).

Route	Does
GET /admin/shard/atlas	Status: path, readable, drift, counts, facets, pending
POST /admin/shard/atlas/import	Import now; { force: true } ignores the hash gate
POST /admin/shard/atlas/approve	Apply a staged refresh, facet loss and all
POST /admin/shard/atlas/reject	Keep the current atlas; remember the decision
PUT /admin/shard/atlas/path	Point the atlas at a different tree

Three behaviours that are deliberate:

• An unreadable tree is a 200, not a 500. refresh() reports outcomes rather than throwing, because the boot path must never be stopped by a bad tree, and that contract is preserved at the API. The panel says "The tree could not be read: …"; a 500 would say only that something broke.
• Setting the path does not import. Moving the mount and reloading the world are separate decisions, and an operator fixing a typo should not have a multi-thousand-row replace happen under them. The response carries fresh status so the panel can offer the import as the next step.
• Every action is written to the admin activity log (shard.atlas.import / .approve / .reject / .path).