RunicGateway/docs
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(link): what a 4 MB spawn file does to a 1 MiB wire (Phase 7)

Browse Source 
10 said the shard would serve `tree/<label>` -> bytes. Measured against a stock
57.4 tree it cannot: Spawns/trammel.xml is 4.03 MB, the sidecar discards any
inbound line over 1 MiB, and that file as one base64 row is 5.4 MiB. It would be
dropped, time out, and be re-requested forever with no error in it anywhere --
and two files on a STOCK tree are in that state.

10.1 records the three carriages measured before anything was written, and why
the winner is not the smallest one: whole-file gzip is 1.21 MB against chunked
gzip's 1.26 MB and is bounded by nothing, so it works on every tree anyone would
test and fails on the first one nobody did. The chunk is the guarantee; the
compression is only the saving.

10.2: it is a `tree` family on assets.fetch, not 14's separate tree.* commands
-- phase 5's registry already owns the single slot, the envelope, the ceiling
and the mid-import guard, so reusing it left `link` with nothing to do for the
third phase running. Its CONSENT is its own, though: Bridge.TreeEnabled, because
declining to serve an EA-licensed client is not the same as declining to serve
the spawn files an operator wrote, and the atlas would have been the casualty.

10.3 records the two defects and which harness found which. An empty `catalog`
is not an absent one. And GZipStream writes nothing at all for zero bytes of
input, which stock ServUO's two empty decoration files walk straight into -- an
offline probe called that a success, because .NET's own decompressor reads an
empty stream as empty data and the declared length and hash both agree with it.
Only a live walk through a reader on another runtime disagreed.

10.4: one canonical read order, because the decoration index keeps the first
item id it sees and the two readers agreed by coincidence rather than by
construction. PARSER_VERSION 4 -> 5.

10.5 has the end-to-end numbers against a live shard. 17 gains the phase's three
decisions, including the one that departed from the recommendation: boot never
calls the shard, so an install on the bridge has no automatic refresh at all.

SPAWN_ATLAS.md is rewritten around the two sources and stops requiring a shared
filesystem. Protocol stays 8; EXTRACTOR_VERSION stays 3.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
This commit is contained in:
claude
2026-09-14 02:00:58 -05:00
parent 48caedd3e0
commit dcf7be4975
2 changed files with 274 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files

128
website/SPAWN_ATLAS.md
View File

@@ -1,7 +1,9 @@
# Spawn atlas
**Status:** Complete on `edge` — data pipeline in website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112), API + pages in website [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113).
**Design:** [`docs/link/v3.md` §6](../link/v3.md) — Protocol 3.0 Part C.
The source files stopped needing a shared filesystem in Protocol 8 phase 7.
**Design:** [`docs/link/v3.md` §6](../link/v3.md) — Protocol 3.0 Part C;
[`docs/link/v8.md` §10](../link/v8.md) — the sources over the bridge.
The spawn atlas is a browsable catalogue of what the shard *contains*: which
creatures spawn, where, how many, and which champion altars are configured. It
@@ -25,10 +27,13 @@ exactly the same treatment as a stock one, with no code change.
The atlas is **static shard content, not live shard state.**
- It does **not** come from the sidecar. Nothing here touches the bridge, and
there is no event kind, no wire change and no `PROTOCOL_VERSION` bump for it.
Part C is website-only.
- It stays fully populated while the shard is down.
- It stays fully populated while the shard is down. Nothing here is an event,
nothing subscribes, and no live feed feeds it.
- Its source files now *travel* over the sidecar (Protocol 8 phase 7, below), but
only when an admin asks — on the request/reply path, never the event path.
Until Protocol 8 this section said the atlas never touched the bridge at all,
which was true and was bought at a price: the website had to be able to read
the shard's filesystem.
- Its champion table (`shard_champion_spawns`) is the *configured roster* —
"there is an Unholy Terror altar in Deceit". The live `champ.update` feed in
`shard_champs` is the separate, sidecar-fed answer to "it is on level 3 right
@@ -37,25 +42,71 @@ The atlas is **static shard content, not live shard state.**
Routes live at `/api/v1/public/atlas`, deliberately **not** under `/shard`,
because `/shard/*` means sidecar-dependent.
## Configuring the tree
## Where the source files come from
The website needs to be able to *read* the ServUO tree — same host, a bind mount,
or a shared volume. Two ways to point at it, the setting winning over the
environment:
Two ends, and **the shard wins whenever uo-link is configured and enabled**:
| Source | Notes |
| Source | When it is used |
|---|---|
| `spawn_atlas_servuo_path` setting | Admin-editable; changes take effect on the next refresh without a redeploy |
| **The shard, over uo-link** (Protocol 8 phase 7) | Whenever a shard is linked and enabled. Nothing to configure — the sidecar connection the site already has is the whole setup |
| A local ServUO tree | When there is no shard link: development, and same-host installs. Also a one-off `--servuo <path>`, which is an instruction and overrules the bridge |
With neither the atlas is simply skipped — the site runs normally without one.
**Why this changed.** Reading a ServUO tree required the website to have
filesystem access to the shard — "same host, a bind mount, or a shared volume" —
and that was the one place the platform's own rule (only the sidecar bridges the
shard) was broken, by the component that faces the internet. The shard now serves
the same five labelled groups over the same request/reply path as every other
shard read, and the parsers did not move: `spawnAtlasParse.js` is still pure,
still fs-free, and still covered by CI without a ServUO tree anywhere near it.
The local path remains, and remains configurable two ways, the setting winning
over the environment:
| Setting | Notes |
|---|---|
| `spawn_atlas_servuo_path` | Admin-editable; changes take effect on the next refresh without a redeploy |
| `SERVUO_PATH` env var | The deploy-time default, since the path usually describes a mount the deployment sets up |
With neither set the atlas is simply skipped — the site runs normally without
one.
### What crosses the wire, and what it costs
A stock 57.4 tree is **141 files and 11.34 MB**, and a spawn file is the awkward
part: `Spawns/trammel.xml` alone is 4.03 MB against the sidecar's **1 MiB inbound
line cap**. So a file crosses as **chunks of 512 KiB, each gzipped** —
`tree/Spawns/trammel.xml/c0` and so on, which is the same key-depth scheme the
asset families use.
Measured end to end against a live shard and sidecar:
| | |
|---|---|
| Files / bytes | 141 / 11,895,427 |
| Chunks / pages | 158 / 3 |
| On the wire | **1.33 MB** (the tree gzips ~12.5x) |
| Full import | **~0.5 s** |
| "Has anything changed?" | one manifest call, ~32 KB, **~70 ms** — no file bytes at all |
The shard serves this under its own switch, **`Bridge.TreeEnabled`**, separate
from `Bridge.AssetsEnabled`. The asset switch is an operator consenting to the
website reading their *UO client*; this one is about the shard's *own
configuration*, which they wrote. An operator can decline the first and still
publish a spawn atlas.
## The boot path
On every start the server hashes the source files and compares them against what
is loaded. Unchanged (the normal case on a restart) costs one read pass, ~120 ms,
and no database write. A real change costs a ~400 ms parse and a reload.
**On the bridge, boot imports nothing.** A local tree hashes in ~120 ms and
skips; asking the shard would put a sidecar round trip in the boot sequence to
answer a question whose answer is "no" on every restart that did not follow a map
edit. Editing spawn files is an operator action, so importing is one too:
**Admin → Spawn Atlas → Import now**, or the CLI. Whatever atlas is loaded keeps
serving until then. (Identical reasoning, and the same decision, as the cliloc
table — see [`CLILOCS.md`](CLILOCS.md).)
From a local tree it behaves as it always has: on every start the server hashes
the source files and compares them against what is loaded. Unchanged (the normal
case on a restart) costs one read pass, ~120 ms, and no database write. A real
change costs a ~400 ms parse and a reload.
Two contracts govern it:
@@ -102,12 +153,13 @@ npm run atlas:import -- --reject # keep the current atlas, dismiss it
## The CLI
The server refreshes itself on boot, so this is for applying a map change
*without* a restart, and for the approve/reject flow above.
On a linked shard this and the admin panel are the *only* ways an import happens.
From a local tree the server also refreshes itself on boot, so there it is for
applying a map change *without* a restart, and for the approve/reject flow above.
```bash
npm run atlas:import # import if the tree differs
npm run atlas:import -- --servuo <path> # override the path for this run
npm run atlas:import # import if the source differs
npm run atlas:import -- --servuo <path> # read this local tree for this run
npm run atlas:import -- --force # reimport even if unchanged
```
@@ -117,12 +169,23 @@ one-off import.
## Sources
These five groups are the whole of it, at both ends: they are what the
filesystem reader walks and they are the only labels the shard will serve.
| File | Count (stock ServUO 57.4) | Used for |
|---|---|---|
| `Spawns/*.xml` | 13 files, ~10.5 MB | Every spawner: location, size, delays, time-of-day, creature types |
| `Spawns/*.xml` | 13 files, 10.4 MB | Every spawner: location, size, delays, time-of-day, creature types |
| `Data/Regions.xml` | 129 KB, nested | Named regions and their rectangles |
| `Data/Locations/*.xml` | 6 files | Landmarks (dungeon levels, town markers) |
| `Data/Locations/*.xml` | 6 files, 37 KB | Landmarks (dungeon levels, town markers) |
| `Config/ChampionSpawns.xml` | 4.8 KB | Configured champion altars |
| `Data/Decoration/**/*.cfg` | 120 files, 1.28 MB, nested two deep | The decoration vocabulary world authoring offers |
A fetch for anything outside them is refused by name, and **the shard never joins
a path that arrived on the wire** — a request names a label the shard itself
enumerated, or it is answered `absent`. Two of those 120 decoration files are
**zero bytes** on a stock tree, which is worth knowing because it broke the first
implementation: .NET's `GZipStream` emits nothing at all for zero bytes of input,
which is not a valid gzip stream.
**A stock tree has 13 spawn files but only 6 facets.** `Eodon.xml`,
`GravewaterLake.xml`, `TreasuresOfKotl.xml` and the other named-area files hold
@@ -315,10 +378,27 @@ whatever an older build derived, indefinitely, and a deploy that corrects the
parse would never reach the data. A version mismatch counts as drift, so the
correction lands on the next boot without an operator having to know it happened.
It is **5** as of Protocol 8 phase 7: the source files are now parsed in one
canonical label order whichever end read them. That matters because the parse is
order-sensitive in one place — the decoration index keeps the first item id it
sees for a type, and the two readers sorted a nested directory differently, so the
same tree could yield a different preview graphic depending on how it arrived.
Identical files, a different answer for a handful of types: precisely what this
number exists to push through the hash gate.
The **source fingerprint is taken over raw bytes** at both ends for the same
reason. Hashing the decoded text would hash a UTF-8 *re-encoding* of the file —
identical for valid UTF-8, and different for a file that is not, because an
undecodable byte becomes U+FFFD and never comes back. One Latin-1 character in a
creature name would then fingerprint differently depending on which end read it,
and the drift gate would report a change on every import, forever, with the tree
untouched.
## The API
Everything is served from MariaDB. Nothing on this path touches the sidecar, so
the pages stay complete while the shard is down — which is why the routes sit at
Everything is served from MariaDB. Nothing on the *read* path touches the sidecar
(the source files reach the database at import time and stay there), so the pages
stay complete while the shard is down — which is why the routes sit at
`/api/v1/public/atlas` and **not** under `/public/shard`, where a prefix means
"sidecar-dependent". Unlike `/shard/*`, they *are* `siteMode`-gated, like
`/posts` and `/wiki`: a bestiary is site content and follows site content's rules.