wtclaude
1a7481e9f4
Phase 3 is built and walked on a live shard. What the walk measured, and the
two places the design of record needed correcting:
§8.1, new: the catalogue is 787 exactly as §4.8 predicted, and the whole scan
of bodies 1-2047 takes 734 ms cold -- so the wall-clock paging §11 designed
never fires on this client. Every §4.8/§5.2 prediction held when the bytes were
rendered and LOOKED at: 320, 607, 608 and 666 come back absent rather than as
another creature's picture, and the direction split is 783 at index 1 against 4
at index 0 -- four player bodies, not six.
44 of the 787 hashes are shared by two or three bodies, which is the exact
signature of the wrong-picture bug, so it was chased rather than assumed. It is
the client's own Body.def aliasing (83 {1}, 84 {1}, 106 {12, 59}), and the check
that settles it is at the source: Translate(ref body, ref hue) rewrites `body`
only when bit 31 is set, unlike the one-argument overload -- and ResolveAnimation
calls that same two-argument overload, so validator and decoder resolve the
identical record.
§12.1, new: **§12 is right about the outcome and wrong about the mechanism.**
`shard_spawn_creatures` is emptied and refilled by every atlas refresh, and a
refresh runs on every boot -- so an imported filename written to that row is
destroyed by an ordinary re-parse of the ServUO tree, and the next Update finds
the client files unchanged and never restores it. Three tables outside that
blast radius, and the atlas import re-derives `art` on the way past.
§14: **§16 listed phase 3 as servuo-plugins + module-uo and that was wrong.**
web.rs routes every command explicitly, so `link` is in the phase. Corrected in
both places.
UOFIDDLER.md is DELETED, two phases earlier than §9.1 predicted -- creature art
was the only thing still on it. SPAWN_ATLAS.md §Artwork is rewritten around the
import, keeping the operator's own map as the thing that wins; the module's
SCHEMA.md gains the three tables and API.md the two admin routes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
387 lines
19 KiB
Markdown
387 lines
19 KiB
Markdown
# Spawn atlas
|
||
|
||
**Status:** Complete on `edge` — data pipeline in website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112), API + pages in website [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113).
|
||
**Design:** [`docs/link/v3.md` §6](../link/v3.md) — Protocol 3.0 Part C.
|
||
|
||
The spawn atlas is a browsable catalogue of what the shard *contains*: which
|
||
creatures spawn, where, how many, and which champion altars are configured. It
|
||
answers "where do I find a lizardman?" with **"Shrines, Yew, Isamu-Jima"** rather
|
||
than with a list of raw coordinates.
|
||
|
||
## Two things that shape the whole design
|
||
|
||
**The shard's ServUO tree is the single source of truth.** Nothing is
|
||
precomputed and committed to the repository. A shard's maps change over its
|
||
lifetime — facets get added, replaced, or renamed — and a snapshot in the repo
|
||
would silently drift from the world players actually see. The atlas is therefore
|
||
re-derived from the tree **on every server boot**.
|
||
|
||
**Facets are not a fixed list.** Nothing in the codebase names Felucca, Trammel,
|
||
or any other stock facet. The facet set is whatever the shard's own files
|
||
declare, discovered at parse time. A shard running entirely custom maps gets
|
||
exactly the same treatment as a stock one, with no code change.
|
||
|
||
## What it is not
|
||
|
||
The atlas is **static shard content, not live shard state.**
|
||
|
||
- It does **not** come from the sidecar. Nothing here touches the bridge, and
|
||
there is no event kind, no wire change and no `PROTOCOL_VERSION` bump for it.
|
||
Part C is website-only.
|
||
- It stays fully populated while the shard is down.
|
||
- Its champion table (`shard_champion_spawns`) is the *configured roster* —
|
||
"there is an Unholy Terror altar in Deceit". The live `champ.update` feed in
|
||
`shard_champs` is the separate, sidecar-fed answer to "it is on level 3 right
|
||
now". Both exist; do not conflate them.
|
||
|
||
Routes live at `/api/v1/public/atlas`, deliberately **not** under `/shard`,
|
||
because `/shard/*` means sidecar-dependent.
|
||
|
||
## Configuring the tree
|
||
|
||
The website needs to be able to *read* the ServUO tree — same host, a bind mount,
|
||
or a shared volume. Two ways to point at it, the setting winning over the
|
||
environment:
|
||
|
||
| Source | Notes |
|
||
|---|---|
|
||
| `spawn_atlas_servuo_path` setting | Admin-editable; changes take effect on the next refresh without a redeploy |
|
||
| `SERVUO_PATH` env var | The deploy-time default, since the path usually describes a mount the deployment sets up |
|
||
|
||
With neither set the atlas is simply skipped — the site runs normally without
|
||
one.
|
||
|
||
## The boot path
|
||
|
||
On every start the server hashes the source files and compares them against what
|
||
is loaded. Unchanged (the normal case on a restart) costs one read pass, ~120 ms,
|
||
and no database write. A real change costs a ~400 ms parse and a reload.
|
||
|
||
Two contracts govern it:
|
||
|
||
**1. It never blocks startup.** No configured path, an unreadable mount, a
|
||
malformed file, a database error — every one is caught and logged, and the site
|
||
comes up serving whatever atlas it already had.
|
||
|
||
**2. A facet disappearing is never applied automatically.** Losing a facet looks
|
||
exactly like a half-copied or mid-update tree, and boot cannot tell that apart
|
||
from a real map change. That refresh is *staged* for a human instead. Everything
|
||
else — new facets, renamed regions, changed spawns — applies immediately, since
|
||
none of it can destroy something an operator would miss.
|
||
|
||
```
|
||
boot
|
||
└─ path configured? no ──▶ skip
|
||
└─ tree readable? no ──▶ warn, carry on
|
||
└─ hashes changed? no ──▶ done (nothing parsed)
|
||
└─ parse
|
||
└─ a facet would be removed?
|
||
no ──▶ import
|
||
yes ──▶ stage for admin review; atlas unchanged
|
||
```
|
||
|
||
### Approving or rejecting a staged refresh
|
||
|
||
Only the *decision* is stored, never the parsed world — a few KB of source hashes
|
||
plus the facet diff. Approving **re-parses** the tree, so what lands matches the
|
||
tree at approval time rather than at boot, and a multi-megabyte blob never sits
|
||
in the database.
|
||
|
||
A rejection is remembered against those exact source hashes, so a declined
|
||
refresh does not re-prompt on every restart. Change the tree and the hashes
|
||
differ, which asks again.
|
||
|
||
From **Admin → Spawn Atlas**, or from the CLI:
|
||
|
||
```bash
|
||
cd website/server
|
||
npm run atlas:import -- --status # what is loaded, and what is pending
|
||
npm run atlas:import -- --approve # apply the staged refresh
|
||
npm run atlas:import -- --reject # keep the current atlas, dismiss it
|
||
```
|
||
|
||
## The CLI
|
||
|
||
The server refreshes itself on boot, so this is for applying a map change
|
||
*without* a restart, and for the approve/reject flow above.
|
||
|
||
```bash
|
||
npm run atlas:import # import if the tree differs
|
||
npm run atlas:import -- --servuo <path> # override the path for this run
|
||
npm run atlas:import -- --force # reimport even if unchanged
|
||
```
|
||
|
||
`--servuo` is a per-run override and deliberately does **not** persist — changing
|
||
where the atlas permanently reads from is an admin action, not a side effect of a
|
||
one-off import.
|
||
|
||
## Sources
|
||
|
||
| File | Count (stock ServUO 57.4) | Used for |
|
||
|---|---|---|
|
||
| `Spawns/*.xml` | 13 files, ~10.5 MB | Every spawner: location, size, delays, time-of-day, creature types |
|
||
| `Data/Regions.xml` | 129 KB, nested | Named regions and their rectangles |
|
||
| `Data/Locations/*.xml` | 6 files | Landmarks (dungeon levels, town markers) |
|
||
| `Config/ChampionSpawns.xml` | 4.8 KB | Configured champion altars |
|
||
|
||
**A stock tree has 13 spawn files but only 6 facets.** `Eodon.xml`,
|
||
`GravewaterLake.xml`, `TreasuresOfKotl.xml` and the other named-area files hold
|
||
TerMur/Trammel points. The facet always comes from each record's own `<Map>`,
|
||
never from the file name.
|
||
|
||
## How a coordinate becomes a place name
|
||
|
||
This is the transform the atlas exists for, in `resolveRegion()`:
|
||
|
||
1. The highest-`priority` named region whose rectangle contains the point. Ties
|
||
break toward the **smallest** rect, so a specific room wins over the
|
||
dungeon-wide rect enclosing it.
|
||
2. Otherwise the nearest landmark within the landmark radius (200 tiles by
|
||
default), labelled by its **group** ("Covetous"), not its individual marker
|
||
("Level 1").
|
||
3. Otherwise `"Wilderness"`.
|
||
|
||
The radius cap in step 2 is what keeps step 3 reachable. Without it the nearest
|
||
landmark is always *some* landmark however far away, and open countryside gets
|
||
labelled with a dungeon on the far side of the map.
|
||
|
||
Against stock ServUO this resolves **83.2%** of points (5,369 of 6,455): 3,681 by
|
||
region, 1,688 by landmark, 1,086 Wilderness.
|
||
|
||
## Three quirks in the source data
|
||
|
||
Each of these is silent if unhandled — the atlas still builds, it is just wrong.
|
||
|
||
**Facet names disagree between sources.** `Data/Locations/*.xml` spells them
|
||
`Ter Mur` and `Tokuno Islands`, while `<Map>` and `<Facet name>` say `TerMur` and
|
||
`Tokuno`. Unreconciled, the landmark bucket is keyed differently from the points
|
||
looking it up, so the fallback never fires and every unregioned spawn on those
|
||
facets reads "Wilderness".
|
||
|
||
This is reconciled **by matching, not by a lookup table** — there is no list of
|
||
facet names anywhere. `facetKey()` collapses spelling differences (lowercase,
|
||
alphanumerics only), and `resolveFacetName()` matches a loose spelling against
|
||
the canonical set discovered from the shard's own spawn and region data, by exact
|
||
key then by prefix in either direction. A name matching nothing keeps its own
|
||
name: forcing a wrong match would file a real custom facet's landmarks under the
|
||
wrong facet, which is worse than leaving it alone.
|
||
|
||
**Spawn type tokens carry XmlSpawner directives.** The `<Objects2>` type is not
|
||
always a bare class name:
|
||
|
||
```
|
||
Fairy,{RND,4,8} alchemist/z/-50 Agralem/Name/Agralem
|
||
GargishRouser,1 greatape,true GargishRefugee/hue/34532
|
||
```
|
||
|
||
Taken literally these invent creatures that do not exist *and* split real ones in
|
||
two, because `Fairy` and `Fairy,{RND,4,8}` slug apart into separate entries. 71 of
|
||
845 were affected. Everything from the first `/` or `,` is stripped, leaving 800
|
||
real creatures.
|
||
|
||
**Case is inconsistent across files.** The same creature is `Lizardman` in one
|
||
file and `lizardman` in another. Slugging collapses them correctly, but the
|
||
display name is chosen deterministically — most common spelling wins, ties break
|
||
to the more capitalised form, then alphabetically — because otherwise it would
|
||
depend on file read order and change on an unrelated restart.
|
||
|
||
## Tables
|
||
|
||
All are **import-owned**: a refresh empties and reloads them in one transaction,
|
||
so a failed reload leaves the previous atlas intact rather than a half-loaded
|
||
world. Nothing else writes to them and nothing holds a foreign key to them — no
|
||
FKs at all, consistent with every other `shard_*` table. Full column listings in
|
||
[`../modules/uo/SCHEMA.md`](../modules/uo/SCHEMA.md) — these are module-uo's tables, not core's.
|
||
|
||
| Table | Rows (stock) | Notes |
|
||
|---|---|---|
|
||
| `shard_spawn_creatures` | 800 | `slug` PK; `total` = sum of each type's own max; nullable `art` |
|
||
| `shard_spawn_points` | 6,455 | `spawn_range`, since `range` is reserved in MariaDB |
|
||
| `shard_spawn_point_types` | 23,927 | The many-to-many; one spawner commonly carries six types |
|
||
| `shard_regions` | 387 | Flattened out of the nesting; `rects` JSON |
|
||
| `shard_landmarks` | 558 | `grp`, since `group` is reserved in SQL |
|
||
| `shard_champion_spawns` | 25 | Configured altars, not the live feed |
|
||
| `shard_atlas_meta` | 1 | Singleton; source hashes, for the change check |
|
||
| `shard_atlas_pending` | 0–1 | Singleton; a staged refresh awaiting admin review |
|
||
|
||
`shard_spawn_creatures.name` carries a plain `INDEX`, deliberately **not
|
||
`FULLTEXT`**: ~800 rows makes a `LIKE` scan free, and FULLTEXT's minimum token
|
||
length would break searches for names like "orc".
|
||
|
||
The reload uses `DELETE`, not `TRUNCATE` — `TRUNCATE` is DDL in MariaDB and would
|
||
implicitly commit, defeating the all-or-nothing guarantee. Point ids are assigned
|
||
explicitly rather than left to `AUTO_INCREMENT`, because the join rows need them
|
||
and `conn.batch()` reports no usable `insertId`.
|
||
|
||
## Artwork — the shard extracts it now (Protocol 8)
|
||
|
||
**This project ships no creature art and no extraction tooling, and never will.**
|
||
UO sprites live in the operator's own client `.mul`/`.uop` files. They are the
|
||
operator's, not ours to redistribute.
|
||
|
||
What changed in protocol 8 is not that rule — it is who does the extracting. The
|
||
shard already has those files (a ServUO server cannot boot without a UO client),
|
||
so as of [`v8.md`](../link/v8.md) phase 3 it decodes them itself and hands the
|
||
pictures to the website over the bridge. Nobody installs UOFiddler and nobody
|
||
copies images to a web host.
|
||
|
||
**Admin → Shard → Import.** The import walks the shard's asset manifest, fetches
|
||
only the sprites whose hash changed, writes them under `uploads/atlas/`, asks the
|
||
shard for a body id per creature (§8 — the shard constructs the creature and
|
||
reads `Body.BodyID`, which is the only thing that is right for a shard's own
|
||
custom creatures) and points each `shard_spawn_creatures.art` at its picture.
|
||
Boot never calls the shard for this: the files change when an operator patches
|
||
their client, which is an event they know about and the site does not.
|
||
|
||
On this machine's stock client that is **787 creature portraits**, about a
|
||
megabyte in total.
|
||
|
||
**NULL stays a first-class state, and always will be.** An install with no shard
|
||
link has never imported one; a Linux shard host without `libgdiplus` cannot
|
||
render a sprite at all (a named `NO_IMAGING` status, not an error); and even
|
||
after a complete import, two thirds of the playable ghost and gargoyle bodies
|
||
have no art in the client. Pages render without images, which is normal and
|
||
supported, not degraded.
|
||
|
||
### The operator's own artwork still wins
|
||
|
||
An operator who has drawn their own portraits keeps them. The map is unchanged:
|
||
|
||
1. Drop the images under `server/uploads/atlas/`.
|
||
2. Copy `server/db/data/spawnAtlas.art.example.json` to `spawnAtlas.art.json` and
|
||
map creature slugs to file names.
|
||
3. Restart, or run the import.
|
||
|
||
`spawnAtlas.art.json` is applied **over** anything imported, per slug, so a sprite
|
||
rip never replaces a hand-drawn portrait on the next Update. Both it and
|
||
`server/uploads/` are gitignored, so neither the map nor the images can be
|
||
committed by accident.
|
||
|
||
### Why the imported art is not stored on the creature row
|
||
|
||
`shard_spawn_creatures` is emptied and refilled by every atlas refresh, and a
|
||
refresh happens on every boot. So the body ids and the imported files live in
|
||
`shard_creature_bodies` and `shard_assets`, outside that blast radius, and the
|
||
atlas import re-derives `art` from them on the way past. Storing it on the row
|
||
would mean an ordinary re-parse of the ServUO tree silently deleting every
|
||
portrait — with the next asset Update finding the client files unchanged,
|
||
reporting "nothing to do", and never putting them back.
|
||
|
||
## Code layout
|
||
|
||
| File | Role |
|
||
|---|---|
|
||
| `src/utils/spawnAtlasParse.js` | **Pure and fs-free** parsers, so CI covers them with no ServUO tree. Zero dependencies. |
|
||
| `src/utils/spawnAtlasSource.js` | The only thing that reads a ServUO tree; shared by the boot path and the CLI |
|
||
| `src/model/shardAtlas/shardAtlas.db.js` | The one-transaction replace |
|
||
| `src/model/shardAtlas/shardAtlas.model.js` | The refresh decision, staging, approve/reject |
|
||
| `scripts/importSpawnAtlas.js` | Thin CLI over the model |
|
||
|
||
Parsing notes:
|
||
|
||
- `Regions.xml`, `Locations/*.xml` and `ChampionSpawns.xml` genuinely nest, and
|
||
get a small hand-rolled **subset** tokenizer — elements, attributes,
|
||
self-closing tags, comments, the XML declaration, CDATA, and the five
|
||
predefined entities plus numeric refs. It is not a general-purpose XML parser
|
||
and must not be reused as one.
|
||
- The ~10.5 MB of `Spawns/*.xml` never touches that tokenizer. Those records are
|
||
flat, so they get a streaming regex sweep instead; a DOM would allocate a node
|
||
per element across ~40 fields on every record to keep 14 of them. **Do not put
|
||
the Points files through a DOM parser.**
|
||
- `<Objects2>` is `Type:MX=n:SB=…` segments joined by `:OBJ=`. Split on `:OBJ=`
|
||
*first* — a naive `split(':')` shreds it. A single Trammel point carries six
|
||
types.
|
||
- **Respawn delays are stored in two different units, per record.** XmlSpawner
|
||
writes `MinDelay`/`MaxDelay` in minutes, and switches to seconds only when a
|
||
spawner's delay does not divide into whole minutes — flagging that with
|
||
`DelayInSec` on the same record. A `5` therefore means five *minutes* on one
|
||
spawner and five *seconds* on the next, and both are plausible respawn times,
|
||
so a reader assuming either unit is silently wrong about the other. Stock
|
||
ServUO 57.4 has ~170 second-flagged spawners out of 6,455. The parser
|
||
normalises everything to **seconds**; the API and UI carry seconds throughout.
|
||
|
||
### The parser version
|
||
|
||
`spawnAtlasSource.js` exports `PARSER_VERSION`, stored in `shard_atlas_meta`
|
||
alongside the source hashes and bumped whenever the parser derives **different
|
||
data from identical files** — a fixed misreading, a new field, a changed unit.
|
||
|
||
A refresh re-derives when the tree changed **or** the parser did. Hashing the
|
||
tree alone would be a trap: an install whose maps never change would keep serving
|
||
whatever an older build derived, indefinitely, and a deploy that corrects the
|
||
parse would never reach the data. A version mismatch counts as drift, so the
|
||
correction lands on the next boot without an operator having to know it happened.
|
||
|
||
## The API
|
||
|
||
Everything is served from MariaDB. Nothing on this path touches the sidecar, so
|
||
the pages stay complete while the shard is down — which is why the routes sit at
|
||
`/api/v1/public/atlas` and **not** under `/public/shard`, where a prefix means
|
||
"sidecar-dependent". Unlike `/shard/*`, they *are* `siteMode`-gated, like
|
||
`/posts` and `/wiki`: a bestiary is site content and follows site content's rules.
|
||
|
||
Every route carries `requireFeature('atlas')` — **404** when an admin has
|
||
disabled the feature (its pages must not reveal that it exists) and **403** when
|
||
the caller sits below its configured audience. The default is `anonymous`, so the
|
||
gates are inert until an admin changes something. Responses are field-projected
|
||
like every other shard read; `atlas` declares no sensitive fields today, and the
|
||
projection call is there so the first one that does is covered by construction
|
||
rather than by a retrofit ([`v3.md` §3.6.1](../link/v3.md)).
|
||
|
||
| Route | Answers |
|
||
|---|---|
|
||
| `GET /atlas/creatures?q=&facet=&limit=&offset=` | The bestiary, most numerous first, paginated with an unpaginated `total` |
|
||
| `GET /atlas/creatures/:slug?facet=&points=` | One creature: `places`, `spawners`, `alsoHere` |
|
||
| `GET /atlas/regions?facet=&q=` | Named regions and their rectangles |
|
||
| `GET /atlas/landmarks?facet=&q=` | Points of interest, labelled by `group` |
|
||
| `GET /atlas/champions?facet=` | The configured altar roster |
|
||
| `GET /atlas/meta` | Facets, counts and when the atlas was parsed |
|
||
|
||
Two shapes worth knowing:
|
||
|
||
- **`places` is the aggregate the atlas exists for.** "Lizardman → Shrines,
|
||
Isamu-Jima, Yew", grouped in SQL rather than by summing 6,455 point rows in
|
||
Node. `spawners` is the raw list underneath it, bounded, with
|
||
`spawnersTruncated` saying when it was cut.
|
||
- **`points` is a COUNT, `spawners` is the LIST.** The two are named apart
|
||
deliberately: the same key meaning a number on the search route and an array on
|
||
the detail route is the kind of thing a client only discovers in production.
|
||
|
||
`GET /atlas/meta` reports the **game world only**. The ServUO path, the per-file
|
||
hashes and any pending refresh describe the operator's filesystem, and live on
|
||
the admin route instead.
|
||
|
||
A facet is never validated against a list — nothing in the codebase names one.
|
||
`?facet=` is length-bounded and matched exactly, so an unknown name returns an
|
||
empty result rather than an error. The filter is an `EXISTS` over the points and
|
||
deliberately not a JSON path or `JSON_SEARCH` built from caller input: that
|
||
function treats `%` and `_` as wildcards, which would make `?facet=%` match
|
||
everything.
|
||
|
||
## The admin panel
|
||
|
||
**Admin → Spawn Atlas** (`/admin/shard-atlas`, admin-only — it reads a path on
|
||
the server's filesystem and replaces every atlas table, which is closer to a
|
||
deploy action than to moderation).
|
||
|
||
| Route | Does |
|
||
|---|---|
|
||
| `GET /admin/shard/atlas` | Status: path, readable, drift, counts, facets, pending |
|
||
| `POST /admin/shard/atlas/import` | Import now; `{ force: true }` ignores the hash gate |
|
||
| `POST /admin/shard/atlas/approve` | Apply a staged refresh, facet loss and all |
|
||
| `POST /admin/shard/atlas/reject` | Keep the current atlas; remember the decision |
|
||
| `PUT /admin/shard/atlas/path` | Point the atlas at a different tree |
|
||
|
||
Three behaviours that are deliberate:
|
||
|
||
- **An unreadable tree is a 200, not a 500.** `refresh()` reports outcomes rather
|
||
than throwing, because the boot path must never be stopped by a bad tree, and
|
||
that contract is preserved at the API. The panel says *"The tree could not be
|
||
read: …"*; a 500 would say only that something broke.
|
||
- **Setting the path does not import.** Moving the mount and reloading the world
|
||
are separate decisions, and an operator fixing a typo should not have a
|
||
multi-thousand-row replace happen under them. The response carries fresh status
|
||
so the panel can offer the import as the next step.
|
||
- **Every action is written to the admin activity log** (`shard.atlas.import` /
|
||
`.approve` / `.reject` / `.path`).
|