feat(atlas): derive a spawn atlas from the shard tree on every boot #112

Merged

whitlocktech merged 3 commits from feat/spawn-atlas-parse into edge 2026-07-28 21:51:30 +00:00

Conversation 1 Commits 3 Files Changed 12 +2740

wtclaude commented 2026-07-28 21:08:06 +00:00

Member

Copy Link

Protocol 3.0 order 3 (Part C), first of two website PRs. This half is the data pipeline only — parsers, the boot refresh, and the tables. No routes and no client, so nothing is user-visible yet; the API, pages and the admin approve/reject UI follow in PR 2.

Part C is website-only: no plugin, no sidecar, no new event kinds, no wire change.

Rewritten after review. The first version of this PR built a committed JSON artifact from a ServUO tree and imported it. Two things were wrong with that, and both are fixed here.

The shard's tree is the single source of truth

A shard's maps change over its life, so a snapshot committed to the repo silently drifts from the world players actually see. The atlas is now re-derived from the tree on every server boot, hash-gated: an unchanged tree costs one read pass (~120 ms) and no database write, a real change costs a ~400 ms parse.

The 1.41 MB artifact, scripts/buildSpawnAtlas.js, and the whole encode/decode seam it needed (encodePoint/readPoint, tuple encoding, omitted defaults) are deleted. Nothing to keep in sync, nothing to go stale.

Path comes from the spawn_atlas_servuo_path admin setting, falling back to SERVUO_PATH. The setting wins, matching how the rest of the shard integration is admin-managed rather than env-configured.

Nothing names a facet

The first version carried a lookup table of the six stock UO facets to reconcile spelling drift between sources. That is wrong — a shard may add facets, replace them outright, or rename them when its maps are updated, and a built-in list mishandles all three silently.

The facet set is now discovered from the tree (spawn records and region definitions are the authority), and the loose spellings in Data/Locations are matched against it by key then prefix. An unmatched name keeps its own rather than being forced into a wrong bucket — filing a real custom facet's landmarks under the wrong facet would be worse than leaving it alone. The tests use Sosaria and Underdark specifically so a stock-facet assumption cannot creep back in.

Two contracts on the boot path

It never blocks startup. No configured path, an unreadable mount, a malformed file, a database error — every one is caught and logged, and the site comes up serving whatever atlas it already had.

A facet disappearing is never applied automatically. Losing a facet is the signature of a half-copied or mid-update tree as much as of a real map change, and boot cannot tell them apart. That refresh is staged in shard_atlas_pending for an admin to approve or reject; startup continues either way. Additions and every other change apply immediately, since none of them can destroy something an operator would miss.

Only the decision is stored — source hashes plus the facet diff, a few KB. Approving re-parses, so what lands matches the tree at approval time rather than at boot, and a multi-megabyte blob never sits in the database. A rejection is remembered against those exact hashes, so a declined refresh does not re-prompt on every restart; changing the tree asks again.

Parsing

src/utils/spawnAtlasParse.js is pure and fs-free so CI covers it with no ServUO tree. Zero new dependencies — Regions.xml genuinely nests, so it gets a small hand-rolled subset tokenizer rather than a new XML package. The 10.5 MB of Spawns/*.xml never touches it: those records are flat and get a streaming regex sweep.

The high-value transform is point-in-rect placement — highest region priority wins, ties break to the smaller rect, then a nearest-landmark fallback, else "Wilderness". That is what turns "lizardman at 5411,1234" into "Despise, Felucca", and it resolves 83.2% of points.

Three things the real data forced:

• A stock tree has 13 spawn files but only 6 facets. Eodon.xml and the other named-area files carry TerMur/Trammel points, so the facet comes from each record's own <Map>.
• Facet names disagree across sources. Data/Locations says Ter Mur and Tokuno Islands; <Map> says TerMur and Tokuno. Unreconciled this is silent — the landmark fallback never fires on those facets and every unregioned spawn there reads "Wilderness".
• Spawn type tokens carry XmlSpawner directives: Fairy,{RND,4,8}, alchemist/z/-50, Agralem/Name/Agralem. Taken literally these invent creatures that do not exist AND split real ones in two, since Fairy and Fairy,{RND,4,8} slug apart. 71 of 845 entries were affected; stripping leaves 800 clean ones.

No artwork, by design

The repo ships no creature art and no extraction tooling. Sprites live in the operator's own client .mul/.uop files and are theirs, not ours to redistribute. shard_spawn_creatures.art is nullable and NULL on every fresh import; an operator who wants art extracts it themselves into the gitignored server/uploads/atlas/ and maps slugs in a gitignored spawnAtlas.art.json. Text-only is the normal, supported state.

Verification

564 server tests pass, 85 new across spawnAtlas.parse.test.js (the :OBJ= split, directive stripping, nested-region priority inheritance, half-open rects, facet matching, tokenizer edge cases) and spawnAtlas.source.test.js (custom-facet builds, spelling reconciliation, hash gating, and every branch of the refresh decision — including that refreshOnBoot survives a database that throws on every call).

End-to-end against the local MariaDB and the ServUO tree at C:\Users\colby\Desktop\ServUO:

Check	Result
Full import	6,455 points, 800 creatures, 23,927 point/type rows, 387 regions, 558 landmarks, 25 altars
Placement	83.2% resolved — 3,681 region, 1,688 landmark, 1,086 Wilderness
"Where does a lizardman spawn?"	Shrines, Isamu-Jima, Yew across Felucca, Trammel, Tokuno
Unchanged tree	Atlas is already up to date — nothing parsed, nothing written
Boot, no path configured	Site up, atlas skipped
Boot, broken path	WARN spawn atlas source unavailable, site up
Boot, tree changed	Detected and imported, site up

Facet gate exercised against a real tree copy with malas.xml removed:

Step	Result
Refresh	Refresh NOT applied — it would remove 1 facet(s): Malas.
Atlas after	Untouched — all 293 Malas points still present
--reject, then re-run	Atlas is already up to date (refresh previously rejected)
--approve	Applied; 6,162 points, Malas gone, pending cleared

No routes changed, so the OpenAPI spec and route manifest are untouched.

Docs: RunicGateway/docs#67.

---

• AI-assisted: written with Claude Code (Claude Opus 5), reviewed before opening.

🤖 Generated with Claude Code

https://claude.ai/code/session_01U7CBg11prhLimL9iHSX1bP

Protocol 3.0 order 3 (Part C), first of two website PRs. This half is the data pipeline only — parsers, the boot refresh, and the tables. No routes and no client, so nothing is user-visible yet; the API, pages and the admin approve/reject UI follow in PR 2. Part C is website-only: no plugin, no sidecar, no new event kinds, no wire change. > **Rewritten after review.** The first version of this PR built a committed JSON artifact from a ServUO tree and imported it. Two things were wrong with that, and both are fixed here. ## The shard's tree is the single source of truth A shard's maps change over its life, so a snapshot committed to the repo silently drifts from the world players actually see. The atlas is now re-derived from the tree on **every server boot**, hash-gated: an unchanged tree costs one read pass (~120 ms) and no database write, a real change costs a ~400 ms parse. The 1.41 MB artifact, `scripts/buildSpawnAtlas.js`, and the whole encode/decode seam it needed (`encodePoint`/`readPoint`, tuple encoding, omitted defaults) are deleted. Nothing to keep in sync, nothing to go stale. Path comes from the `spawn_atlas_servuo_path` admin setting, falling back to `SERVUO_PATH`. The setting wins, matching how the rest of the shard integration is admin-managed rather than env-configured. ## Nothing names a facet The first version carried a lookup table of the six stock UO facets to reconcile spelling drift between sources. That is wrong — a shard may add facets, replace them outright, or rename them when its maps are updated, and a built-in list mishandles all three silently. The facet set is now discovered from the tree (spawn records and region definitions are the authority), and the loose spellings in `Data/Locations` are matched against it by key then prefix. An unmatched name keeps its own rather than being forced into a wrong bucket — filing a real custom facet's landmarks under the wrong facet would be worse than leaving it alone. The tests use `Sosaria` and `Underdark` specifically so a stock-facet assumption cannot creep back in. ## Two contracts on the boot path **It never blocks startup.** No configured path, an unreadable mount, a malformed file, a database error — every one is caught and logged, and the site comes up serving whatever atlas it already had. **A facet disappearing is never applied automatically.** Losing a facet is the signature of a half-copied or mid-update tree as much as of a real map change, and boot cannot tell them apart. That refresh is staged in `shard_atlas_pending` for an admin to approve or reject; startup continues either way. Additions and every other change apply immediately, since none of them can destroy something an operator would miss. Only the *decision* is stored — source hashes plus the facet diff, a few KB. Approving re-parses, so what lands matches the tree at approval time rather than at boot, and a multi-megabyte blob never sits in the database. A rejection is remembered against those exact hashes, so a declined refresh does not re-prompt on every restart; changing the tree asks again. ## Parsing `src/utils/spawnAtlasParse.js` is pure and fs-free so CI covers it with no ServUO tree. **Zero new dependencies** — `Regions.xml` genuinely nests, so it gets a small hand-rolled subset tokenizer rather than a new XML package. The 10.5 MB of `Spawns/*.xml` never touches it: those records are flat and get a streaming regex sweep. The high-value transform is point-in-rect placement — highest region priority wins, ties break to the smaller rect, then a nearest-landmark fallback, else "Wilderness". That is what turns "lizardman at 5411,1234" into "Despise, Felucca", and it resolves **83.2%** of points. Three things the real data forced: - **A stock tree has 13 spawn files but only 6 facets.** `Eodon.xml` and the other named-area files carry TerMur/Trammel points, so the facet comes from each record's own `<Map>`. - **Facet names disagree across sources.** `Data/Locations` says `Ter Mur` and `Tokuno Islands`; `<Map>` says `TerMur` and `Tokuno`. Unreconciled this is silent — the landmark fallback never fires on those facets and every unregioned spawn there reads "Wilderness". - **Spawn type tokens carry XmlSpawner directives**: `Fairy,{RND,4,8}`, `alchemist/z/-50`, `Agralem/Name/Agralem`. Taken literally these invent creatures that do not exist AND split real ones in two, since `Fairy` and `Fairy,{RND,4,8}` slug apart. 71 of 845 entries were affected; stripping leaves 800 clean ones. ## No artwork, by design The repo ships no creature art and no extraction tooling. Sprites live in the operator's own client `.mul`/`.uop` files and are theirs, not ours to redistribute. `shard_spawn_creatures.art` is nullable and NULL on every fresh import; an operator who wants art extracts it themselves into the gitignored `server/uploads/atlas/` and maps slugs in a gitignored `spawnAtlas.art.json`. Text-only is the normal, supported state. ## Verification **564 server tests pass**, 85 new across `spawnAtlas.parse.test.js` (the `:OBJ=` split, directive stripping, nested-region priority inheritance, half-open rects, facet matching, tokenizer edge cases) and `spawnAtlas.source.test.js` (custom-facet builds, spelling reconciliation, hash gating, and every branch of the refresh decision — including that `refreshOnBoot` survives a database that throws on every call). End-to-end against the local MariaDB and the ServUO tree at `C:\Users\colby\Desktop\ServUO`: | Check | Result | |---|---| | Full import | 6,455 points, 800 creatures, 23,927 point/type rows, 387 regions, 558 landmarks, 25 altars | | Placement | 83.2% resolved — 3,681 region, 1,688 landmark, 1,086 Wilderness | | "Where does a lizardman spawn?" | Shrines, Isamu-Jima, Yew across Felucca, Trammel, Tokuno | | Unchanged tree | `Atlas is already up to date` — nothing parsed, nothing written | | Boot, no path configured | Site up, atlas skipped | | Boot, broken path | `WARN spawn atlas source unavailable`, site up | | Boot, tree changed | Detected and imported, site up | Facet gate exercised against a real tree copy with `malas.xml` removed: | Step | Result | |---|---| | Refresh | `Refresh NOT applied — it would remove 1 facet(s): Malas.` | | Atlas after | Untouched — all 293 Malas points still present | | `--reject`, then re-run | `Atlas is already up to date (refresh previously rejected)` | | `--approve` | Applied; 6,162 points, Malas gone, pending cleared | No routes changed, so the OpenAPI spec and route manifest are untouched. Docs: RunicGateway/docs#67. --- - [x] AI-assisted: written with **Claude Code** (Claude Opus 5), reviewed before opening. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01U7CBg11prhLimL9iHSX1bP

wtclaude added 1 commit 2026-07-28 21:08:07 +00:00

feat(atlas): parse a ServUO tree into a committed spawn atlas artifact ... 353cce9f26

Protocol 3.0 order 3 (Part C), first of two website PRs. This half is the data
pipeline only — parsers, the build/import CLI, and the tables. No routes and no
client, so nothing is user-visible yet; the API and pages follow in PR 2.

Part C is website-only: no plugin, no sidecar, no new event kinds, no wire
change.

## Parsing

`src/utils/spawnAtlasParse.js` is pure and fs-free so CI covers it with no
ServUO tree. Zero new dependencies — `Regions.xml` genuinely nests, so it gets a
small hand-rolled subset tokenizer rather than a new XML package. The 10.5 MB of
`Spawns/*.xml` never touches it: those records are flat and get a streaming
regex sweep instead.

The high-value transform is point-in-rect placement — highest region priority
wins, ties break to the smaller rect, then a nearest-landmark fallback within
200 tiles, else "Wilderness". That is what turns "lizardman at 5411,1234" into
"Despise, Felucca", and it resolves 83.2% of points (5,369 of 6,455).

Three things the real data forced, none of which were in the design:

- **Only 6 facets, not 13.** `Eodon.xml`, `GravewaterLake.xml` and the other
  named-area files carry TerMur/Trammel points, so the facet comes from each
  record's own `<Map>` and the artifact shards 6 ways.
- **Facet names disagree across sources.** `Data/Locations/*.xml` spells them
  `Ter Mur` and `Tokuno Islands`; `<Map>` and `<Facet name>` say `TerMur` and
  `Tokuno`. Unreconciled this is silent — the landmark fallback simply never
  fires on those facets and every unregioned spawn there reads "Wilderness".
- **Spawn type tokens carry XmlSpawner directives**: `Fairy,{RND,4,8}`,
  `alchemist/z/-50`, `Agralem/Name/Agralem`. Taken literally these invent
  creatures that do not exist AND split real ones in two, since `Fairy` and
  `Fairy,{RND,4,8}` slug apart. 71 of 845 entries were affected; stripping at
  the first `/` or `,` leaves 800 clean ones.

## Artifact

`npm run atlas:build -- --servuo <path>` writes `db/data/spawnAtlas.*.json`:
6 facet shards + a compact index + a small indented `meta`. 1.41 MB committed,
down from 4.40 MB by dropping `facet` per record, omitting defaulted fields, and
tuple-encoding the ~24,000 type entries. `encodePoint()` and the importer's
`readPoint()` are exact inverses and are round-tripped in tests.

Display spelling is chosen deterministically (most common, ties to the
capitalised form) because the spawn files are inconsistent about case and the
name would otherwise depend on file read order — a spurious diff on every
unrelated rebuild.

## Import

`npm run atlas:import` needs no ServUO tree, which is the whole reason build and
import are separate: the container has the artifact but not the tree. It
reloads all six tables in one transaction (DELETE, not TRUNCATE, which is DDL
and would implicitly commit), so a failed import leaves the previous atlas
intact.

## No artwork, by design

The repo ships no creature art and no extraction tooling. Sprites live in the
operator's own client `.mul`/`.uop` files and are theirs, not ours to
redistribute. `shard_spawn_creatures.art` is nullable and NULL on every fresh
import; an operator who wants art extracts it themselves, drops it under
`server/uploads/atlas/` (already gitignored) and maps slugs in a gitignored
`spawnAtlas.art.json`. Text-only is the normal, fully supported state.

## Verification

- **544 server tests pass**, 57 new across `spawnAtlas.parse.test.js` (the
  `:OBJ=` split, directive stripping, nested-region priority inheritance,
  half-open rects, the facet reconciliation, tokenizer edge cases) and
  `spawnAtlas.build.test.js` (aggregation, deterministic naming, and the
  encode/decode round trip).
- Built and imported for real against the local MariaDB and the ServUO tree at
  `C:\Users\colby\Desktop\ServUO`: 6,455 points, 800 creatures, 23,927
  point/type rows, 387 regions, 558 landmarks, 25 champion altars.
- "Where does a lizardman spawn?" answers Shrines / Isamu-Jima / Yew across
  Felucca, Trammel and Tokuno.

No routes changed, so the OpenAPI spec and route manifest are untouched.

---

- [x] AI-assisted: written with **Claude Code** (Claude Opus 5), reviewed before opening.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U7CBg11prhLimL9iHSX1bP

wtclaude added 1 commit 2026-07-28 21:41:51 +00:00

refactor(atlas): derive the atlas from the shard's tree on every boot ... 2801ec8f4d

Replaces the committed-artifact design from the first commit. Two problems with
it, both raised in review:

**Facets are not a fixed list.** The first pass carried a hardcoded table of the
six stock UO facets to reconcile the spelling drift between sources. That is
wrong: a shard may add facets, replace them outright, or rename them when its
maps are updated, and a built-in list quietly mishandles all three. Nothing in
the atlas names a facet any more. The facet set is discovered from the tree —
spawn records and region definitions are the authority — and the loose spellings
in Data/Locations are matched against it by key and prefix. Custom facets get
identical treatment; the tests use `Sosaria` and `Underdark` precisely so a
stock-facet assumption cannot creep back in.

**A snapshot goes stale.** Maps change over a server's life, so a build-once
artifact silently drifts from the world players actually see. The tree is now
the single source of truth and the atlas is re-derived on every boot.

## What that changed

- **The committed artifact is gone** — 1.41 MB of generated JSON removed, along
  with `scripts/buildSpawnAtlas.js` and the whole encode/decode seam it needed
  (`encodePoint`/`readPoint`, the tuple encoding, the omitted-defaults scheme and
  their round-trip tests). Nothing to keep in sync, nothing to go stale.
- **NEW `src/utils/spawnAtlasSource.js`** — the only thing that touches a ServUO
  tree; shared by the boot path and the CLI. Parsers stay pure and fs-free.
- **NEW `src/model/shardAtlas/`** — `.db.js` (the one-transaction replace) and
  `.model.js` (the refresh decision).
- **`scripts/importSpawnAtlas.js`** is now a thin CLI over the model:
  `--servuo`, `--force`, `--approve`, `--reject`, `--status`. `atlas:build` is
  gone; `atlas:import` remains.
- Path comes from the `spawn_atlas_servuo_path` admin setting, falling back to
  `SERVUO_PATH`. The setting wins, matching how the rest of the shard
  integration is admin-managed rather than env-configured.

## Two contracts on the boot path

**It never blocks startup.** No path, an unreadable mount, a malformed file, a
database error — every one is caught and logged, and the site comes up serving
whatever atlas it already had. Verified by booting the real server with no path,
a broken path, and a good path.

**A facet disappearing is never applied automatically.** Losing a facet is the
signature of a half-copied or mid-update tree as much as of a real map change,
and boot cannot tell them apart. The refresh is staged in `shard_atlas_pending`
for an admin to approve or reject, and startup continues regardless. Additions
and every other change apply immediately, since none of them can destroy
something an operator would miss.

Only the decision is stored, not the parsed world: a few KB of source hashes and
the facet diff. Approving re-parses, so what gets applied matches the tree at
approval time rather than at boot. A rejection is remembered against those exact
hashes, so a declined refresh does not re-prompt on every restart — changing the
tree changes the hashes and asks again.

Hash-gated, so the common case (restart, maps unchanged) reads and hashes the
tree (~120 ms) and writes nothing. A real change costs a ~400 ms parse.

The admin approve/reject UI is part of the second PR, with the rest of the
routes and pages. Until then the CLI covers it.

## Verification

- **564 server tests pass**, 28 new in `spawnAtlas.source.test.js` covering the
  custom-facet build, the spelling reconciliation, hash gating, and every branch
  of the refresh decision — including that `refreshOnBoot` survives a database
  that throws on every call.
- End-to-end against the local MariaDB and the real ServUO tree: 6,455 points,
  800 creatures, 23,927 point/type rows, 387 regions, 558 landmarks, 25 altars,
  83.2% of points resolved to a place name.
- The facet gate exercised against a real tree copy with `malas.xml` removed:
  staged rather than applied, atlas untouched with all 293 Malas points intact,
  reject then stays quiet on re-run, approve applies and drops the facet.
- Booted the real server under all three source conditions; none blocked.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U7CBg11prhLimL9iHSX1bP

wtclaude changed title from feat(atlas): parse a ServUO tree into a committed spawn atlas artifact to feat(atlas): derive a spawn atlas from the shard tree on every boot 2026-07-28 21:45:24 +00:00

wtclaude added 1 commit 2026-07-28 21:47:53 +00:00

Merge remote-tracking branch 'origin/edge' into feat/spawn-atlas-parse a4ef9d676d

whitlocktech approved these changes 2026-07-28 21:51:21 +00:00

whitlocktech merged commit f3d084e046 into edge 2026-07-28 21:51:30 +00:00

whitlocktech deleted branch feat/spawn-atlas-parse 2026-07-28 21:51:31 +00:00

whitlocktech referenced this issue from a commit 2026-07-28 21:51:31 +00:00

Merge pull request 'feat(atlas): derive a spawn atlas from the shard tree on every boot' (#112) from feat/spawn-atlas-parse into edge

wtclaude referenced this issue from a commit 2026-07-29 00:51:35 +00:00

feat(atlas): serve the spawn atlas and give operators a panel for it

wtclaude referenced this pull request 2026-07-29 00:52:31 +00:00

feat(atlas): serve the spawn atlas and give operators a panel for it #113

wtclaude referenced this pull request 2026-07-29 23:07:57 +00:00

feat(shard)!: Protocol 3.0 cutover — visibility framework, spawn atlas, marketplace #118

Sign in to join this conversation.

Reviewers

No Reviewers

whitlocktech

Labels

Clear labels

bug

Something is not working

duplicate

This issue or pull request already exists

enhancement

New feature

help wanted

Need some help

invalid

Something is wrong

question

More information is needed

severity:critical

Security audit: critical severity

severity:high

Security audit: high severity

severity:low

Security audit: low severity

severity:medium

Security audit: medium severity

wontfix

This won't be fixed

bug

Something is not working

duplicate

This issue or pull request already exists

enhancement

New feature

help wanted

Need some help

invalid

Something is wrong

question

More information is needed

wontfix

This won't be fixed

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

ksfixitman whitlocktech wtclaude

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: RunicGateway/website#112

No description provided.