RunicGateway/docs
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(link): what a 4 MB spawn file does to a 1 MiB wire (Phase 7)

Browse Source 
10 said the shard would serve `tree/<label>` -> bytes. Measured against a stock
57.4 tree it cannot: Spawns/trammel.xml is 4.03 MB, the sidecar discards any
inbound line over 1 MiB, and that file as one base64 row is 5.4 MiB. It would be
dropped, time out, and be re-requested forever with no error in it anywhere --
and two files on a STOCK tree are in that state.

10.1 records the three carriages measured before anything was written, and why
the winner is not the smallest one: whole-file gzip is 1.21 MB against chunked
gzip's 1.26 MB and is bounded by nothing, so it works on every tree anyone would
test and fails on the first one nobody did. The chunk is the guarantee; the
compression is only the saving.

10.2: it is a `tree` family on assets.fetch, not 14's separate tree.* commands
-- phase 5's registry already owns the single slot, the envelope, the ceiling
and the mid-import guard, so reusing it left `link` with nothing to do for the
third phase running. Its CONSENT is its own, though: Bridge.TreeEnabled, because
declining to serve an EA-licensed client is not the same as declining to serve
the spawn files an operator wrote, and the atlas would have been the casualty.

10.3 records the two defects and which harness found which. An empty `catalog`
is not an absent one. And GZipStream writes nothing at all for zero bytes of
input, which stock ServUO's two empty decoration files walk straight into -- an
offline probe called that a success, because .NET's own decompressor reads an
empty stream as empty data and the declared length and hash both agree with it.
Only a live walk through a reader on another runtime disagreed.

10.4: one canonical read order, because the decoration index keeps the first
item id it sees and the two readers agreed by coincidence rather than by
construction. PARSER_VERSION 4 -> 5.

10.5 has the end-to-end numbers against a live shard. 17 gains the phase's three
decisions, including the one that departed from the recommendation: boot never
calls the shard, so an install on the bridge has no automatic refresh at all.

SPAWN_ATLAS.md is rewritten around the two sources and stops requiring a shared
filesystem. Protocol stays 8; EXTRACTOR_VERSION stays 3.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
This commit is contained in:
claude
2026-09-14 02:00:58 -05:00
parent 48caedd3e0
commit dcf7be4975
2 changed files with 274 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files

175
link/v8.md
View File

@@ -1047,7 +1047,7 @@ portable key:
| `Config/ChampionSpawns.xml` | 1 | | `Config/ChampionSpawns.xml` | 1 |
| `Data/Decoration/**` | tree | | `Data/Decoration/**` | tree |
So the shard serves `tree/<label>` → bytes over the same batched request/reply path, and So the shard serves these files over the same batched request/reply path, and
`spawnAtlasSource.js` gains a second backend behind its existing interface: **filesystem** (today, `spawnAtlasSource.js` gains a second backend behind its existing interface: **filesystem** (today,
kept for same-host installs and for development) or **sidecar** (new, and the default once kept for same-host installs and for development) or **sidecar** (new, and the default once
configured). configured).
@@ -1060,6 +1060,132 @@ sidecar a forwarder here too.
`SERVUO_PATH` and the `spawn_atlas_servuo_path` setting remain, and select the filesystem backend. `SERVUO_PATH` and the `spawn_atlas_servuo_path` setting remain, and select the filesystem backend.
### 10.1 What phase 7 measured: `tree/<label>` → bytes cannot work
The sentence above said the shard would serve `tree/<label>` → bytes. Measured against a stock 57.4
tree, **it cannot**. `Spawns/trammel.xml` is **4.03 MB**; the sidecar discards any inbound line over
**1 MiB** (`shard.rs` `MAX_INBOUND_LINE_BYTES`); that file as one base64 row is 5.4 MiB. The reply
would be dropped, the request would time out, the import would re-request it, forever — a failure
with no error anywhere in it. **Two files on a stock tree are in that state**, and a shard with
hand-built spawn tables has more.
| Label group | Files | Bytes |
|---|---|---|
| `Spawns/*.xml` | 13 | 10,445,607 |
| `Data/Decoration/**/*.cfg` | 120 | 1,278,425 |
| `Data/Regions.xml` | 1 | 129,008 |
| `Data/Locations/*.xml` | 6 | 37,549 |
| `Config/ChampionSpawns.xml` | 1 | 4,838 |
| **Total** | **141** | **11,895,427** |
Three carriages were measured before anything was written:
| | Wire | Pages | Undeliverable |
|---|---|---|---|
| Raw base64 | 15.1 MB | 31 | **2 files** |
| Whole-file gzip | 1.21 MB | 3 | none *on this tree* |
| **512 KiB chunks, each gzipped** | **1.26 MB** | **3** | **none, by construction** |
**The chunk is the bound and the compression is the saving**, and which is which is the whole
decision. Compression is what makes it cheap — the tree gzips 12.5x, so the atlas source arrives in
three pages instead of thirty-one. But nothing guarantees an operator's files compress at all, so
the ceiling has to hold when they do not: a 512 KiB chunk that refuses to compress is ~683 KiB of
base64, still inside the wire cap that §3.3's deliberate factor of two leaves room for. Whole-file
gzip works on every tree anyone would test and fails on the first one nobody did — the difference
between the second and third rows is entirely a difference in what is *guaranteed*.
Chunking costs 0.2 of the ratio (12.3x against 12.5x) and adds a depth segment to the key:
```
tree/Spawns/trammel.xml the manifest row — size, hash, chunk count
tree/Spawns/trammel.xml/c0 the first 512 KiB of it, gzipped
```
which is §5's scheme doing the same job it does for `body/400/a0/f0`, and needing no protocol change
to do it — the second time that has paid for itself (§11.2 was the first).
### 10.2 It is a family, not two new commands
§14 planned `tree.manifest` and `tree.fetch` with `/tree/*` REST beside them. Phase 5 had since
built a **family registry** on `assets.fetch` — "the command is the transport and the family is a
property of the key" — and §5 had already written `tree/Spawns/Trammel.xml` as a key. So phase 7
registered a `tree` family instead, and generalised `assets.manifest` the same way, which is that
lesson landing one level up.
What that bought: the single slot, the byte budget, the paging envelope, the key-count ceiling, the
`catalog` mid-import guard and the 425 backoff, none of them written twice — **and `link` needed
nothing at all**, for the third phase running, because `/assets/manifest?family=` and
`POST /assets/fetch` forward verbatim. `tree.manifest`, `tree.fetch` and `/tree/*` are **not built**.
**It has its own consent, though: `Bridge.TreeEnabled`.** Reusing the command is not the same as
reusing the switch. `Bridge.AssetsEnabled` is an operator agreeing that the website may read *their
UO client* — art and animations licensed from EA. This is the operator's *own configuration*, which
they wrote, and which the public bestiary is built from. One switch could not express both, and the
thing that would have silently disappeared for an operator who declined the first is their spawn
atlas. So the consent check moved from the front of `assets.fetch` into the family lookup, and
`assets.sources` now answers whenever *either* plane is on, reporting `families` filtered to what is
actually enabled — which is how a tree-only shard's website discovers there is anything to ask for.
### 10.3 The three things the walk checks, and the two defects they found
Every check is a way this ends in a tree that *looks* imported. XML is forgiving about what it
skips, so a mis-assembled spawn file parses cleanly and simply has fewer spawns in it.
- **Each chunk re-declares its own address** — label, index, offset — and carries the hash of its own
uncompressed bytes. The reader places chunks by declared index rather than arrival order.
- **The whole file is hashed after reassembly** against the manifest row, which is also the
fingerprint the drift gate stores.
- **The `catalog` must not move mid-walk**, or the import is refused rather than stitched out of two
trees.
Two defects were found, and *which harness found which* is the part worth keeping.
**An empty `catalog` is not an absent one.** The shard compared `expected != null`, so a caller
sending `""` — a serialisation of "I have no fingerprint to assert" — had every fetch refused, with
a sentence naming no catalog at all ("catalog is now 8159778b"). Found by an offline probe that
passed one by accident.
**`GZipStream` writes nothing for zero bytes of input.** The gzip header is emitted lazily on the
first write, so a stream opened and closed without one yields a zero-length buffer rather than the
20-byte empty member — which is not a valid gzip stream. **Stock ServUO 57.4 ships two empty
decoration files**, so this broke every import off an untouched tree. The offline probe reassembled
all 141 files and reported success, because .NET's own decompressor treats an empty stream as empty
data and the chunk's declared length (0) and hash (of nothing) both agreed. **Only the live walk,
through a reader on a different runtime, disagreed.** The overlay now answers a literal empty gzip
member; teaching the reader to accept an empty payload was rejected as putting a special case on the
wire, where every future reader would have to know it.
### 10.4 One canonical order, because the parse is order-sensitive
The atlas parse keeps the **first** item id it sees for a decoration type, and writes `meta.source`
in iteration order. The two readers agreed on a stock tree and agreed *by coincidence*: the
filesystem reader walks each decoration directory with `localeCompare` while the shard sorts whole
relative paths, and those diverge the moment a directory mixes cases.
So `buildFromFiles` sorts by label, ordinally, once, whatever order the files arrived in — and
`PARSER_VERSION` goes **4 → 5**, because that is identical input producing a different answer for a
handful of types, which is exactly what that number exists to push through the hash gate. The
**source fingerprint also moved to raw bytes** at both ends: hashing decoded text hashes a UTF-8
*re-encoding*, which is identical for valid UTF-8 and different for a file that is not, and one
Latin-1 character in a creature name would have made the drift gate report a change on every import
forever with the tree untouched.
This was found by the live walk too — the unit parity test used `deepEqual`, which ignores key
order. It now asserts serialised equality as well.
### 10.5 Measured end to end
Against a live ServUO shard, the real sidecar and module-uo's own reader:
| | |
|---|---|
| Manifest | 141 rows, one page, 32 KB, **65-92 ms** |
| Full read | 158 chunks, 3 pages, **1.33 MB on the wire**, **512 ms** |
| Verification | all 141 files byte-identical to the tree on disk |
| Atlas parity | **identical** — 6,455 points, 800 creatures, 387 regions, 558 landmarks, 25 champions, 309 decoration types, built from the bridge and from the disk |
| Drift re-check | 141 rows, **~70 ms**, no file bytes |
| Traversal | `tree/../../Scripts/...`, `tree/Config/Bridge.cfg`, `tree/Saves/Accounts/accounts.xml` → all `absent` |
--- ---
## 11. What is bulk and what is on demand ## 11. What is bulk and what is on demand
@@ -1369,13 +1495,13 @@ new pipe.
| `assets.fetch` | `assets.fetch.ok` | Content for an explicit key list, paged; base64 PNG per row | **phase 3** (`source` phase 4; `static`/`land` families phase 5) | | `assets.fetch` | `assets.fetch.ok` | Content for an explicit key list, paged; base64 PNG per row | **phase 3** (`source` phase 4; `static`/`land` families phase 5) |
| `assets.bodies` | `assets.bodies.ok` | Slug → body id (§8, Core thread) | **phase 3** | | `assets.bodies` | `assets.bodies.ok` | Slug → body id (§8, Core thread) | **phase 3** |
| `cliloc.table` | `cliloc.table.ok` | The decompressed table, paged (`?lang=`, `?cursor=`) | **phase 2** | | `cliloc.table` | `cliloc.table.ok` | The decompressed table, paged (`?lang=`, `?cursor=`) | **phase 2** |
| `tree.manifest` / `tree.fetch` | `.ok` | §10, the ServUO tree files | phase 7 | | ~~`tree.manifest` / `tree.fetch`~~ | — | §10's ServUO tree files — **not built**; they are a `tree` family on the two commands above (§10.2) | phase 7 |
Every one of them is refused outright when `Bridge.AssetsEnabled` is off, and every one of them Every one of them is refused outright when `Bridge.AssetsEnabled` is off, and every one of them
requires a `reqId` — a request without one is refused rather than answered, because an uncorrelated requires a `reqId` — a request without one is refused rather than answered, because an uncorrelated
reply is by definition an event, and §3.1 is the reason none of this may ever be one. reply is by definition an event, and §3.1 is the reason none of this may ever be one.
**Sidecar REST** mirrors those one for one under `/assets/*`, `/cliloc`, `/tree/*`, carrying **Sidecar REST** mirrors those one for one under `/assets/*` and `/cliloc`, carrying
`X-UOLink-Version: 8` and forwarding verbatim. `GET /assets/sources` exists as of phase 1 and `X-UOLink-Version: 8` and forwarding verbatim. `GET /assets/sources` exists as of phase 1 and
`GET /cliloc` as of phase 2; their responder maps `bridge.busy` to **425** (flow control, and the `GET /cliloc` as of phase 2; their responder maps `bridge.busy` to **425** (flow control, and the
ordinary answer mid-import rather than a rare one) and reads the refusal's `code` for the rest — ordinary answer mid-import rather than a rare one) and reads the refusal's `code` for the rest —
@@ -1409,6 +1535,21 @@ discovers the gap as a refusal *per key, per pass, forever*, with no picture eve
warning in the log every few minutes. With it, that is one reported state carrying a sentence naming warning in the log every few minutes. With it, that is one reported state carrying a sentence naming
the fix. the fix.
**Phase 7 added a family, a switch and no command** (§10.2). The ServUO tree is served as the
`tree` key family on `assets.fetch`, and `assets.manifest` became family-dispatched the way
`assets.fetch` did in phase 5 — so the tree lists its files through the same envelope the body
catalogue lists its bodies through. Additive, so **the protocol stays 8**, and `EXTRACTOR_VERSION`
stays **3**: this family derives nothing, it forwards an operator's own file unchanged, so the
number that tracks *our derivation* has nothing to say about it. `link` needed nothing for the
third phase running.
Two things about it are not shared with the asset families. It answers to **`Bridge.TreeEnabled`**
rather than `Bridge.AssetsEnabled`, because a UO client and a shard's own configuration are
different consents; the gate therefore moved out of the front of `assets.fetch` and into the family
lookup, and `assets.sources` now answers whenever either plane is enabled, with `families` filtered
to what is actually on. And its rows carry `gzip` rather than `png`, chunked — see §10.1 for the
measurement that forced that and §10.3 for the two defects the checks around it caught.
**Phase 6 added one field and no command.** Manifest and fetch rows carry `action` — which action **Phase 6 added one field and no command.** Manifest and fetch rows carry `action` — which action
of the body the thumbnail came from (§11.2). Additive, so **the protocol stays 8**; a consumer that of the body the thumbnail came from (§11.2). Additive, so **the protocol stays 8**; a consumer that
ignores it sees the catalogue it always saw plus 73 rows, and one that reads it can build the right ignores it sees the catalogue it always saw plus 73 rows, and one that reads it can build the right
@@ -1445,7 +1586,7 @@ disagree, so a split bump means the next bundle silently fails to compose.
|---|---| |---|---|
| `servuo-plugins/` | Extraction over ServUO's own `Ultima` (§4), the cliloc decompressor (§9), body resolution (§8), the request handlers, `overlay.toml` | | `servuo-plugins/` | Extraction over ServUO's own `Ultima` (§4), the cliloc decompressor (§9), body resolution (§8), the request handlers, `overlay.toml` |
| `link/` | Six command families forwarded, the REST surface, **the inbound line cap (§3.3)**, `PROTOCOL_VERSION`. **Nothing in phases 4 or 5** — `assets_call` forwards a request body verbatim and `respond_assets` returns the reply verbatim, so a new key family and a new reply field both pass through untouched | | `link/` | Six command families forwarded, the REST surface, **the inbound line cap (§3.3)**, `PROTOCOL_VERSION`. **Nothing in phases 4 or 5** — `assets_call` forwards a request body verbatim and `respond_assets` returns the reply verbatim, so a new key family and a new reply field both pass through untouched |
| `module-uo/` | Client calls, asset store, the atlas source backend (§10), cliloc ingest, admin surface | | `module-uo/` | Client calls, asset store, the atlas source backend (§10), cliloc ingest, admin surface. Phase 7 also bumped `PARSER_VERSION` 4 → 5 (§10.4) |
| `website/` | Almost none — `ctx.uploads` already suffices (§12). Phase 2 deleted `server/tools/cliloc-export/`, the converter this protocol retires | | `website/` | Almost none — `ctx.uploads` already suffices (§12). Phase 2 deleted `server/tools/cliloc-export/`, the converter this protocol retires |
| `docs/` | This file; rewrite `CLILOCS.md` §Converting and `SPAWN_ATLAS.md` §Artwork + §Configuring; **delete `UOFIDDLER.md`**; add the libgdiplus prerequisite to `SHARD_PREREQS.md` (§4.4) | | `docs/` | This file; rewrite `CLILOCS.md` §Converting and `SPAWN_ATLAS.md` §Artwork + §Configuring; **delete `UOFIDDLER.md`**; add the libgdiplus prerequisite to `SHARD_PREREQS.md` (§4.4) |
| `installer/` | A `doctor` check for libgdiplus on Linux hosts (§4.4). Bundle pairing already enforces §15 | | `installer/` | A `doctor` check for libgdiplus on Linux hosts (§4.4). Bundle pairing already enforces §15 |
@@ -1465,7 +1606,7 @@ disagree, so a split bump means the next bundle silently fails to compose.
| 4 | **DONE 2026-09-11.** The UOP animation decoder (§4.3, §4.9): `BridgeUop` + a PNG encoder that never touches `System.Drawing`, wired in beneath the legacy reader. **Two of the eight player bodies turned out to exist** (gargoyles 666/667); the other six are in no client file, and ghost ids left the player-body set (§5.2, §17.9). The same fallback added **233 other bodies**: the catalogue is **1,022 rows, 1,409 ms cold**, and all six player bodies have art for the first time. `EXTRACTOR_VERSION` 1 → 2 | servuo-plugins | | 4 | **DONE 2026-09-11.** The UOP animation decoder (§4.3, §4.9): `BridgeUop` + a PNG encoder that never touches `System.Drawing`, wired in beneath the legacy reader. **Two of the eight player bodies turned out to exist** (gargoyles 666/667); the other six are in no client file, and ghost ids left the player-body set (§5.2, §17.9). The same fallback added **233 other bodies**: the catalogue is **1,022 rows, 1,409 ms cold**, and all six player bodies have art for the first time. `EXTRACTOR_VERSION` 1 → 2 | servuo-plugins |
| 5 | **DONE 2026-09-11.** Item statics and land on demand (§11.1): the `static` and `land` families, hue applied on the shard from `tiledata.mul`, the byte-bounded art cache, `assets.fetch` made family-aware, `families` on `assets.sources`. Website side: the warm pass, per-row `catalog` staleness, and pictures on the marketplace and the character sheet. **39,189 statics and 4,244 land tiles served; the only refusals are the 9,963 + 12,140 empty index slots §4.5 predicted.** Two traps found — the library's bitmap cache poisons a hued sprite, and `PartialHue` decides the picture from a file only the shard has. Protocol stays 8; `EXTRACTOR_VERSION` stays 2 | servuo-plugins, module-uo | | 5 | **DONE 2026-09-11.** Item statics and land on demand (§11.1): the `static` and `land` families, hue applied on the shard from `tiledata.mul`, the byte-bounded art cache, `assets.fetch` made family-aware, `families` on `assets.sources`. Website side: the warm pass, per-row `catalog` staleness, and pictures on the marketplace and the character sheet. **39,189 statics and 4,244 land tiles served; the only refusals are the 9,963 + 12,140 empty index slots §4.5 predicted.** Two traps found — the library's bitmap cache poisons a hued sprite, and `PartialHue` decides the picture from a file only the shard has. Protocol stays 8; `EXTRACTOR_VERSION` stays 2 | servuo-plugins, module-uo |
| 6 | **DONE 2026-09-11, and not what this row said.** The measurement came first and changed the phase: a complete one-direction animation set is **174,453 frames / 281.5 MB**, not the ~119,000 estimated, and **the site displays still pictures** — so the deep keys and the bulk-fill switch were **not built** (§11.2, org lead 2026-09-11). What shipped is what the still-picture site was missing: the **73 bodies with no art at action 0 and real art deeper** (a horse at `body/820/a23`), the catalogue key carrying its action, the atlas join that reads it, and §4.10's per-body **action ceiling** — without which the fallback walk itself would serve **452 validated pictures of the next body**. Catalogue **1,022 → 1,095**; `EXTRACTOR_VERSION` 2 → 3; protocol stays 8 | servuo-plugins, module-uo | | 6 | **DONE 2026-09-11, and not what this row said.** The measurement came first and changed the phase: a complete one-direction animation set is **174,453 frames / 281.5 MB**, not the ~119,000 estimated, and **the site displays still pictures** — so the deep keys and the bulk-fill switch were **not built** (§11.2, org lead 2026-09-11). What shipped is what the still-picture site was missing: the **73 bodies with no art at action 0 and real art deeper** (a horse at `body/820/a23`), the catalogue key carrying its action, the atlas join that reads it, and §4.10's per-body **action ceiling** — without which the fallback walk itself would serve **452 validated pictures of the next body**. Catalogue **1,022 → 1,095**; `EXTRACTOR_VERSION` 2 → 3; protocol stays 8 | servuo-plugins, module-uo |
| 7 | The atlas over the sidecar (§10); shared-filesystem requirement retired | module-uo | | 7 | **DONE 2026-09-14.** The atlas over the sidecar (§10); shared-filesystem requirement retired. The measurement came first again and changed the shape: `tree/<label>` → bytes **cannot work** — `Spawns/trammel.xml` is 4.03 MB against a 1 MiB line cap — so a file crosses as **512 KiB chunks, each gzipped**, which is §5's depth scheme paying for itself a second time (§10.1). It is a `tree` **family** on `assets.fetch` rather than §14's separate commands, with `assets.manifest` generalised to match and its **own consent, `Bridge.TreeEnabled`** (§10.2) — so `link` needed nothing for the third phase running. **141 files / 11.9 MB / 158 chunks / 3 pages / 1.33 MB on the wire / 512 ms**, and the atlas built over the bridge is identical to the one built off the disk. Two defects, each found by a different harness: an empty `catalog` refusing every fetch, and `GZipStream` emitting **nothing** for the two empty files stock ServUO ships (§10.3). `PARSER_VERSION` 4 → 5 for one canonical read order (§10.4); protocol stays 8; `EXTRACTOR_VERSION` stays 3 | servuo-plugins, module-uo |
| 8 | Admin surface, Import/Update, approve/reject, activity log | module-uo | | 8 | Admin surface, Import/Update, approve/reject, activity log | module-uo |
| 9 | Docs pass across five repos; live walk on the real rig | docs | | 9 | Docs pass across five repos; live walk on the real rig | docs |
@@ -1617,3 +1758,27 @@ in the document.
would otherwise hit it: one action past a body's band is 643 validated pictures and 452 would otherwise hit it: one action past a body's band is 643 validated pictures and 452
byte-identical copies of the next body. byte-identical copies of the next body.
12. **§10: the atlas's three shapes — settled 2026-09-14, phase 7.** Put to the org lead after the
tree was measured and before a line was written, because the first measurement said the phase as
written could not work:
- **A file crosses as 512 KiB chunks, each gzipped.** Not raw base64 (15.1 MB, 31 pages, and two
files on a *stock* tree that can never arrive under a 1 MiB line cap), and not whole-file gzip
— which is smaller on this tree and bounded by nothing, so it works everywhere anyone would
test and fails on the first tree nobody did. The chunk is the guarantee; the compression is
only the saving. See §10.1.
- **It is a `tree` family on `assets.fetch`, with its own switch.** §14's separate `tree.*`
commands and `/tree/*` REST are not built: phase 5's family registry already owns the single
slot, the paging envelope, the key ceiling and the mid-import guard, and reusing it left
`link` with nothing to do for the third phase running. But the *consent* is its own —
`Bridge.TreeEnabled` — because an operator declining to serve their EA-licensed client is not
the same as declining to serve the spawn files they wrote, and the atlas would have been the
casualty. See §10.2.
- **Boot never calls the shard; importing is an admin action** (departing from the
recommendation). The same answer §17.7 gave the cliloc table, and the same reasoning, with one
thing accepted in exchange: an install whose atlas comes over the bridge has *no* automatic
refresh at all, so an edited spawn file stays invisible until someone presses Import. The
alternative — one manifest round trip per boot, ~70 ms and no file bytes — was on the table and
was declined in favour of consistency. The panel and the CLI say so in as many words, and the
skip is logged rather than silent.

128
website/SPAWN_ATLAS.md
View File

@@ -1,7 +1,9 @@
# Spawn atlas # Spawn atlas
**Status:** Complete on `edge` — data pipeline in website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112), API + pages in website [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113). **Status:** Complete on `edge` — data pipeline in website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112), API + pages in website [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113).
**Design:** [`docs/link/v3.md` §6](../link/v3.md) — Protocol 3.0 Part C. The source files stopped needing a shared filesystem in Protocol 8 phase 7.
**Design:** [`docs/link/v3.md` §6](../link/v3.md) — Protocol 3.0 Part C;
[`docs/link/v8.md` §10](../link/v8.md) — the sources over the bridge.
The spawn atlas is a browsable catalogue of what the shard *contains*: which The spawn atlas is a browsable catalogue of what the shard *contains*: which
creatures spawn, where, how many, and which champion altars are configured. It creatures spawn, where, how many, and which champion altars are configured. It
@@ -25,10 +27,13 @@ exactly the same treatment as a stock one, with no code change.
The atlas is **static shard content, not live shard state.** The atlas is **static shard content, not live shard state.**
- It does **not** come from the sidecar. Nothing here touches the bridge, and - It stays fully populated while the shard is down. Nothing here is an event,
there is no event kind, no wire change and no `PROTOCOL_VERSION` bump for it. nothing subscribes, and no live feed feeds it.
Part C is website-only. - Its source files now *travel* over the sidecar (Protocol 8 phase 7, below), but
- It stays fully populated while the shard is down. only when an admin asks — on the request/reply path, never the event path.
Until Protocol 8 this section said the atlas never touched the bridge at all,
which was true and was bought at a price: the website had to be able to read
the shard's filesystem.
- Its champion table (`shard_champion_spawns`) is the *configured roster* — - Its champion table (`shard_champion_spawns`) is the *configured roster* —
"there is an Unholy Terror altar in Deceit". The live `champ.update` feed in "there is an Unholy Terror altar in Deceit". The live `champ.update` feed in
`shard_champs` is the separate, sidecar-fed answer to "it is on level 3 right `shard_champs` is the separate, sidecar-fed answer to "it is on level 3 right
@@ -37,25 +42,71 @@ The atlas is **static shard content, not live shard state.**
Routes live at `/api/v1/public/atlas`, deliberately **not** under `/shard`, Routes live at `/api/v1/public/atlas`, deliberately **not** under `/shard`,
because `/shard/*` means sidecar-dependent. because `/shard/*` means sidecar-dependent.
## Configuring the tree ## Where the source files come from
The website needs to be able to *read* the ServUO tree — same host, a bind mount, Two ends, and **the shard wins whenever uo-link is configured and enabled**:
or a shared volume. Two ways to point at it, the setting winning over the
environment:
| Source | Notes | | Source | When it is used |
|---|---| |---|---|
| `spawn_atlas_servuo_path` setting | Admin-editable; changes take effect on the next refresh without a redeploy | | **The shard, over uo-link** (Protocol 8 phase 7) | Whenever a shard is linked and enabled. Nothing to configure — the sidecar connection the site already has is the whole setup |
| A local ServUO tree | When there is no shard link: development, and same-host installs. Also a one-off `--servuo <path>`, which is an instruction and overrules the bridge |
With neither the atlas is simply skipped — the site runs normally without one.
**Why this changed.** Reading a ServUO tree required the website to have
filesystem access to the shard — "same host, a bind mount, or a shared volume" —
and that was the one place the platform's own rule (only the sidecar bridges the
shard) was broken, by the component that faces the internet. The shard now serves
the same five labelled groups over the same request/reply path as every other
shard read, and the parsers did not move: `spawnAtlasParse.js` is still pure,
still fs-free, and still covered by CI without a ServUO tree anywhere near it.
The local path remains, and remains configurable two ways, the setting winning
over the environment:
| Setting | Notes |
|---|---|
| `spawn_atlas_servuo_path` | Admin-editable; changes take effect on the next refresh without a redeploy |
| `SERVUO_PATH` env var | The deploy-time default, since the path usually describes a mount the deployment sets up | | `SERVUO_PATH` env var | The deploy-time default, since the path usually describes a mount the deployment sets up |
With neither set the atlas is simply skipped — the site runs normally without ### What crosses the wire, and what it costs
one.
A stock 57.4 tree is **141 files and 11.34 MB**, and a spawn file is the awkward
part: `Spawns/trammel.xml` alone is 4.03 MB against the sidecar's **1 MiB inbound
line cap**. So a file crosses as **chunks of 512 KiB, each gzipped** —
`tree/Spawns/trammel.xml/c0` and so on, which is the same key-depth scheme the
asset families use.
Measured end to end against a live shard and sidecar:
| | |
|---|---|
| Files / bytes | 141 / 11,895,427 |
| Chunks / pages | 158 / 3 |
| On the wire | **1.33 MB** (the tree gzips ~12.5x) |
| Full import | **~0.5 s** |
| "Has anything changed?" | one manifest call, ~32 KB, **~70 ms** — no file bytes at all |
The shard serves this under its own switch, **`Bridge.TreeEnabled`**, separate
from `Bridge.AssetsEnabled`. The asset switch is an operator consenting to the
website reading their *UO client*; this one is about the shard's *own
configuration*, which they wrote. An operator can decline the first and still
publish a spawn atlas.
## The boot path ## The boot path
On every start the server hashes the source files and compares them against what **On the bridge, boot imports nothing.** A local tree hashes in ~120 ms and
is loaded. Unchanged (the normal case on a restart) costs one read pass, ~120 ms, skips; asking the shard would put a sidecar round trip in the boot sequence to
and no database write. A real change costs a ~400 ms parse and a reload. answer a question whose answer is "no" on every restart that did not follow a map
edit. Editing spawn files is an operator action, so importing is one too:
**Admin → Spawn Atlas → Import now**, or the CLI. Whatever atlas is loaded keeps
serving until then. (Identical reasoning, and the same decision, as the cliloc
table — see [`CLILOCS.md`](CLILOCS.md).)
From a local tree it behaves as it always has: on every start the server hashes
the source files and compares them against what is loaded. Unchanged (the normal
case on a restart) costs one read pass, ~120 ms, and no database write. A real
change costs a ~400 ms parse and a reload.
Two contracts govern it: Two contracts govern it:
@@ -102,12 +153,13 @@ npm run atlas:import -- --reject # keep the current atlas, dismiss it
## The CLI ## The CLI
The server refreshes itself on boot, so this is for applying a map change On a linked shard this and the admin panel are the *only* ways an import happens.
*without* a restart, and for the approve/reject flow above. From a local tree the server also refreshes itself on boot, so there it is for
applying a map change *without* a restart, and for the approve/reject flow above.
```bash ```bash
npm run atlas:import # import if the tree differs npm run atlas:import # import if the source differs
npm run atlas:import -- --servuo <path> # override the path for this run npm run atlas:import -- --servuo <path> # read this local tree for this run
npm run atlas:import -- --force # reimport even if unchanged npm run atlas:import -- --force # reimport even if unchanged
``` ```
@@ -117,12 +169,23 @@ one-off import.
## Sources ## Sources
These five groups are the whole of it, at both ends: they are what the
filesystem reader walks and they are the only labels the shard will serve.
| File | Count (stock ServUO 57.4) | Used for | | File | Count (stock ServUO 57.4) | Used for |
|---|---|---| |---|---|---|
| `Spawns/*.xml` | 13 files, ~10.5 MB | Every spawner: location, size, delays, time-of-day, creature types | | `Spawns/*.xml` | 13 files, 10.4 MB | Every spawner: location, size, delays, time-of-day, creature types |
| `Data/Regions.xml` | 129 KB, nested | Named regions and their rectangles | | `Data/Regions.xml` | 129 KB, nested | Named regions and their rectangles |
| `Data/Locations/*.xml` | 6 files | Landmarks (dungeon levels, town markers) | | `Data/Locations/*.xml` | 6 files, 37 KB | Landmarks (dungeon levels, town markers) |
| `Config/ChampionSpawns.xml` | 4.8 KB | Configured champion altars | | `Config/ChampionSpawns.xml` | 4.8 KB | Configured champion altars |
| `Data/Decoration/**/*.cfg` | 120 files, 1.28 MB, nested two deep | The decoration vocabulary world authoring offers |
A fetch for anything outside them is refused by name, and **the shard never joins
a path that arrived on the wire** — a request names a label the shard itself
enumerated, or it is answered `absent`. Two of those 120 decoration files are
**zero bytes** on a stock tree, which is worth knowing because it broke the first
implementation: .NET's `GZipStream` emits nothing at all for zero bytes of input,
which is not a valid gzip stream.
**A stock tree has 13 spawn files but only 6 facets.** `Eodon.xml`, **A stock tree has 13 spawn files but only 6 facets.** `Eodon.xml`,
`GravewaterLake.xml`, `TreasuresOfKotl.xml` and the other named-area files hold `GravewaterLake.xml`, `TreasuresOfKotl.xml` and the other named-area files hold
@@ -315,10 +378,27 @@ whatever an older build derived, indefinitely, and a deploy that corrects the
parse would never reach the data. A version mismatch counts as drift, so the parse would never reach the data. A version mismatch counts as drift, so the
correction lands on the next boot without an operator having to know it happened. correction lands on the next boot without an operator having to know it happened.
It is **5** as of Protocol 8 phase 7: the source files are now parsed in one
canonical label order whichever end read them. That matters because the parse is
order-sensitive in one place — the decoration index keeps the first item id it
sees for a type, and the two readers sorted a nested directory differently, so the
same tree could yield a different preview graphic depending on how it arrived.
Identical files, a different answer for a handful of types: precisely what this
number exists to push through the hash gate.
The **source fingerprint is taken over raw bytes** at both ends for the same
reason. Hashing the decoded text would hash a UTF-8 *re-encoding* of the file —
identical for valid UTF-8, and different for a file that is not, because an
undecodable byte becomes U+FFFD and never comes back. One Latin-1 character in a
creature name would then fingerprint differently depending on which end read it,
and the drift gate would report a change on every import, forever, with the tree
untouched.
## The API ## The API
Everything is served from MariaDB. Nothing on this path touches the sidecar, so Everything is served from MariaDB. Nothing on the *read* path touches the sidecar
the pages stay complete while the shard is down — which is why the routes sit at (the source files reach the database at import time and stay there), so the pages
stay complete while the shard is down — which is why the routes sit at
`/api/v1/public/atlas` and **not** under `/public/shard`, where a prefix means `/api/v1/public/atlas` and **not** under `/public/shard`, where a prefix means
"sidecar-dependent". Unlike `/shard/*`, they *are* `siteMode`-gated, like "sidecar-dependent". Unlike `/shard/*`, they *are* `siteMode`-gated, like
`/posts` and `/wiki`: a bestiary is site content and follows site content's rules. `/posts` and `/wiki`: a bestiary is site content and follows site content's rules.