RunicGateway/docs
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(link): what a 4 MB spawn file does to a 1 MiB wire (Phase 7)

Browse Source 
10 said the shard would serve `tree/<label>` -> bytes. Measured against a stock
57.4 tree it cannot: Spawns/trammel.xml is 4.03 MB, the sidecar discards any
inbound line over 1 MiB, and that file as one base64 row is 5.4 MiB. It would be
dropped, time out, and be re-requested forever with no error in it anywhere --
and two files on a STOCK tree are in that state.

10.1 records the three carriages measured before anything was written, and why
the winner is not the smallest one: whole-file gzip is 1.21 MB against chunked
gzip's 1.26 MB and is bounded by nothing, so it works on every tree anyone would
test and fails on the first one nobody did. The chunk is the guarantee; the
compression is only the saving.

10.2: it is a `tree` family on assets.fetch, not 14's separate tree.* commands
-- phase 5's registry already owns the single slot, the envelope, the ceiling
and the mid-import guard, so reusing it left `link` with nothing to do for the
third phase running. Its CONSENT is its own, though: Bridge.TreeEnabled, because
declining to serve an EA-licensed client is not the same as declining to serve
the spawn files an operator wrote, and the atlas would have been the casualty.

10.3 records the two defects and which harness found which. An empty `catalog`
is not an absent one. And GZipStream writes nothing at all for zero bytes of
input, which stock ServUO's two empty decoration files walk straight into -- an
offline probe called that a success, because .NET's own decompressor reads an
empty stream as empty data and the declared length and hash both agree with it.
Only a live walk through a reader on another runtime disagreed.

10.4: one canonical read order, because the decoration index keeps the first
item id it sees and the two readers agreed by coincidence rather than by
construction. PARSER_VERSION 4 -> 5.

10.5 has the end-to-end numbers against a live shard. 17 gains the phase's three
decisions, including the one that departed from the recommendation: boot never
calls the shard, so an install on the bridge has no automatic refresh at all.

SPAWN_ATLAS.md is rewritten around the two sources and stops requiring a shared
filesystem. Protocol stays 8; EXTRACTOR_VERSION stays 3.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
This commit is contained in:
claude
2026-09-14 02:00:58 -05:00
parent 48caedd3e0
commit dcf7be4975
2 changed files with 274 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files

175
link/v8.md
View File

@@ -1047,7 +1047,7 @@ portable key:
| `Config/ChampionSpawns.xml` | 1 |
| `Data/Decoration/**` | tree |
So the shard serves `tree/<label>` → bytes over the same batched request/reply path, and
So the shard serves these files over the same batched request/reply path, and
`spawnAtlasSource.js` gains a second backend behind its existing interface: **filesystem** (today,
kept for same-host installs and for development) or **sidecar** (new, and the default once
configured).
@@ -1060,6 +1060,132 @@ sidecar a forwarder here too.
`SERVUO_PATH` and the `spawn_atlas_servuo_path` setting remain, and select the filesystem backend.
### 10.1 What phase 7 measured: `tree/<label>` → bytes cannot work
The sentence above said the shard would serve `tree/<label>` → bytes. Measured against a stock 57.4
tree, **it cannot**. `Spawns/trammel.xml` is **4.03 MB**; the sidecar discards any inbound line over
**1 MiB** (`shard.rs` `MAX_INBOUND_LINE_BYTES`); that file as one base64 row is 5.4 MiB. The reply
would be dropped, the request would time out, the import would re-request it, forever — a failure
with no error anywhere in it. **Two files on a stock tree are in that state**, and a shard with
hand-built spawn tables has more.
| Label group | Files | Bytes |
|---|---|---|
| `Spawns/*.xml` | 13 | 10,445,607 |
| `Data/Decoration/**/*.cfg` | 120 | 1,278,425 |
| `Data/Regions.xml` | 1 | 129,008 |
| `Data/Locations/*.xml` | 6 | 37,549 |
| `Config/ChampionSpawns.xml` | 1 | 4,838 |
| **Total** | **141** | **11,895,427** |
Three carriages were measured before anything was written:
| | Wire | Pages | Undeliverable |
|---|---|---|---|
| Raw base64 | 15.1 MB | 31 | **2 files** |
| Whole-file gzip | 1.21 MB | 3 | none *on this tree* |
| **512 KiB chunks, each gzipped** | **1.26 MB** | **3** | **none, by construction** |
**The chunk is the bound and the compression is the saving**, and which is which is the whole
decision. Compression is what makes it cheap — the tree gzips 12.5x, so the atlas source arrives in
three pages instead of thirty-one. But nothing guarantees an operator's files compress at all, so
the ceiling has to hold when they do not: a 512 KiB chunk that refuses to compress is ~683 KiB of
base64, still inside the wire cap that §3.3's deliberate factor of two leaves room for. Whole-file
gzip works on every tree anyone would test and fails on the first one nobody did — the difference
between the second and third rows is entirely a difference in what is *guaranteed*.
Chunking costs 0.2 of the ratio (12.3x against 12.5x) and adds a depth segment to the key:
```
tree/Spawns/trammel.xml the manifest row — size, hash, chunk count
tree/Spawns/trammel.xml/c0 the first 512 KiB of it, gzipped
```
which is §5's scheme doing the same job it does for `body/400/a0/f0`, and needing no protocol change
to do it — the second time that has paid for itself (§11.2 was the first).
### 10.2 It is a family, not two new commands
§14 planned `tree.manifest` and `tree.fetch` with `/tree/*` REST beside them. Phase 5 had since
built a **family registry** on `assets.fetch` — "the command is the transport and the family is a
property of the key" — and §5 had already written `tree/Spawns/Trammel.xml` as a key. So phase 7
registered a `tree` family instead, and generalised `assets.manifest` the same way, which is that
lesson landing one level up.
What that bought: the single slot, the byte budget, the paging envelope, the key-count ceiling, the
`catalog` mid-import guard and the 425 backoff, none of them written twice — **and `link` needed
nothing at all**, for the third phase running, because `/assets/manifest?family=` and
`POST /assets/fetch` forward verbatim. `tree.manifest`, `tree.fetch` and `/tree/*` are **not built**.
**It has its own consent, though: `Bridge.TreeEnabled`.** Reusing the command is not the same as
reusing the switch. `Bridge.AssetsEnabled` is an operator agreeing that the website may read *their
UO client* — art and animations licensed from EA. This is the operator's *own configuration*, which
they wrote, and which the public bestiary is built from. One switch could not express both, and the
thing that would have silently disappeared for an operator who declined the first is their spawn
atlas. So the consent check moved from the front of `assets.fetch` into the family lookup, and
`assets.sources` now answers whenever *either* plane is on, reporting `families` filtered to what is
actually enabled — which is how a tree-only shard's website discovers there is anything to ask for.
### 10.3 The three things the walk checks, and the two defects they found
Every check is a way this ends in a tree that *looks* imported. XML is forgiving about what it
skips, so a mis-assembled spawn file parses cleanly and simply has fewer spawns in it.
- **Each chunk re-declares its own address** — label, index, offset — and carries the hash of its own
uncompressed bytes. The reader places chunks by declared index rather than arrival order.
- **The whole file is hashed after reassembly** against the manifest row, which is also the
fingerprint the drift gate stores.
- **The `catalog` must not move mid-walk**, or the import is refused rather than stitched out of two
trees.
Two defects were found, and *which harness found which* is the part worth keeping.
**An empty `catalog` is not an absent one.** The shard compared `expected != null`, so a caller
sending `""` — a serialisation of "I have no fingerprint to assert" — had every fetch refused, with
a sentence naming no catalog at all ("catalog is now 8159778b"). Found by an offline probe that
passed one by accident.
**`GZipStream` writes nothing for zero bytes of input.** The gzip header is emitted lazily on the
first write, so a stream opened and closed without one yields a zero-length buffer rather than the
20-byte empty member — which is not a valid gzip stream. **Stock ServUO 57.4 ships two empty
decoration files**, so this broke every import off an untouched tree. The offline probe reassembled
all 141 files and reported success, because .NET's own decompressor treats an empty stream as empty
data and the chunk's declared length (0) and hash (of nothing) both agreed. **Only the live walk,
through a reader on a different runtime, disagreed.** The overlay now answers a literal empty gzip
member; teaching the reader to accept an empty payload was rejected as putting a special case on the
wire, where every future reader would have to know it.
### 10.4 One canonical order, because the parse is order-sensitive
The atlas parse keeps the **first** item id it sees for a decoration type, and writes `meta.source`
in iteration order. The two readers agreed on a stock tree and agreed *by coincidence*: the
filesystem reader walks each decoration directory with `localeCompare` while the shard sorts whole
relative paths, and those diverge the moment a directory mixes cases.
So `buildFromFiles` sorts by label, ordinally, once, whatever order the files arrived in — and
`PARSER_VERSION` goes **4 → 5**, because that is identical input producing a different answer for a
handful of types, which is exactly what that number exists to push through the hash gate. The
**source fingerprint also moved to raw bytes** at both ends: hashing decoded text hashes a UTF-8
*re-encoding*, which is identical for valid UTF-8 and different for a file that is not, and one
Latin-1 character in a creature name would have made the drift gate report a change on every import
forever with the tree untouched.
This was found by the live walk too — the unit parity test used `deepEqual`, which ignores key
order. It now asserts serialised equality as well.
### 10.5 Measured end to end
Against a live ServUO shard, the real sidecar and module-uo's own reader:
| | |
|---|---|
| Manifest | 141 rows, one page, 32 KB, **65-92 ms** |
| Full read | 158 chunks, 3 pages, **1.33 MB on the wire**, **512 ms** |
| Verification | all 141 files byte-identical to the tree on disk |
| Atlas parity | **identical** — 6,455 points, 800 creatures, 387 regions, 558 landmarks, 25 champions, 309 decoration types, built from the bridge and from the disk |
| Drift re-check | 141 rows, **~70 ms**, no file bytes |
| Traversal | `tree/../../Scripts/...`, `tree/Config/Bridge.cfg`, `tree/Saves/Accounts/accounts.xml` → all `absent` |
---
## 11. What is bulk and what is on demand
@@ -1369,13 +1495,13 @@ new pipe.
| `assets.fetch` | `assets.fetch.ok` | Content for an explicit key list, paged; base64 PNG per row | **phase 3** (`source` phase 4; `static`/`land` families phase 5) |
| `assets.bodies` | `assets.bodies.ok` | Slug → body id (§8, Core thread) | **phase 3** |
| `cliloc.table` | `cliloc.table.ok` | The decompressed table, paged (`?lang=`, `?cursor=`) | **phase 2** |
| `tree.manifest` / `tree.fetch` | `.ok` | §10, the ServUO tree files | phase 7 |
| ~~`tree.manifest` / `tree.fetch`~~ | — | §10's ServUO tree files — **not built**; they are a `tree` family on the two commands above (§10.2) | phase 7 |
Every one of them is refused outright when `Bridge.AssetsEnabled` is off, and every one of them
requires a `reqId` — a request without one is refused rather than answered, because an uncorrelated
reply is by definition an event, and §3.1 is the reason none of this may ever be one.
**Sidecar REST** mirrors those one for one under `/assets/*`, `/cliloc`, `/tree/*`, carrying
**Sidecar REST** mirrors those one for one under `/assets/*` and `/cliloc`, carrying
`X-UOLink-Version: 8` and forwarding verbatim. `GET /assets/sources` exists as of phase 1 and
`GET /cliloc` as of phase 2; their responder maps `bridge.busy` to **425** (flow control, and the
ordinary answer mid-import rather than a rare one) and reads the refusal's `code` for the rest —
@@ -1409,6 +1535,21 @@ discovers the gap as a refusal *per key, per pass, forever*, with no picture eve
warning in the log every few minutes. With it, that is one reported state carrying a sentence naming
the fix.
**Phase 7 added a family, a switch and no command** (§10.2). The ServUO tree is served as the
`tree` key family on `assets.fetch`, and `assets.manifest` became family-dispatched the way
`assets.fetch` did in phase 5 — so the tree lists its files through the same envelope the body
catalogue lists its bodies through. Additive, so **the protocol stays 8**, and `EXTRACTOR_VERSION`
stays **3**: this family derives nothing, it forwards an operator's own file unchanged, so the
number that tracks *our derivation* has nothing to say about it. `link` needed nothing for the
third phase running.
Two things about it are not shared with the asset families. It answers to **`Bridge.TreeEnabled`**
rather than `Bridge.AssetsEnabled`, because a UO client and a shard's own configuration are
different consents; the gate therefore moved out of the front of `assets.fetch` and into the family
lookup, and `assets.sources` now answers whenever either plane is enabled, with `families` filtered
to what is actually on. And its rows carry `gzip` rather than `png`, chunked — see §10.1 for the
measurement that forced that and §10.3 for the two defects the checks around it caught.
**Phase 6 added one field and no command.** Manifest and fetch rows carry `action` — which action
of the body the thumbnail came from (§11.2). Additive, so **the protocol stays 8**; a consumer that
ignores it sees the catalogue it always saw plus 73 rows, and one that reads it can build the right
@@ -1445,7 +1586,7 @@ disagree, so a split bump means the next bundle silently fails to compose.
|---|---|
| `servuo-plugins/` | Extraction over ServUO's own `Ultima` (§4), the cliloc decompressor (§9), body resolution (§8), the request handlers, `overlay.toml` |
| `link/` | Six command families forwarded, the REST surface, **the inbound line cap (§3.3)**, `PROTOCOL_VERSION`. **Nothing in phases 4 or 5** — `assets_call` forwards a request body verbatim and `respond_assets` returns the reply verbatim, so a new key family and a new reply field both pass through untouched |
| `module-uo/` | Client calls, asset store, the atlas source backend (§10), cliloc ingest, admin surface |
| `module-uo/` | Client calls, asset store, the atlas source backend (§10), cliloc ingest, admin surface. Phase 7 also bumped `PARSER_VERSION` 4 → 5 (§10.4) |
| `website/` | Almost none — `ctx.uploads` already suffices (§12). Phase 2 deleted `server/tools/cliloc-export/`, the converter this protocol retires |
| `docs/` | This file; rewrite `CLILOCS.md` §Converting and `SPAWN_ATLAS.md` §Artwork + §Configuring; **delete `UOFIDDLER.md`**; add the libgdiplus prerequisite to `SHARD_PREREQS.md` (§4.4) |
| `installer/` | A `doctor` check for libgdiplus on Linux hosts (§4.4). Bundle pairing already enforces §15 |
@@ -1465,7 +1606,7 @@ disagree, so a split bump means the next bundle silently fails to compose.
| 4 | **DONE 2026-09-11.** The UOP animation decoder (§4.3, §4.9): `BridgeUop` + a PNG encoder that never touches `System.Drawing`, wired in beneath the legacy reader. **Two of the eight player bodies turned out to exist** (gargoyles 666/667); the other six are in no client file, and ghost ids left the player-body set (§5.2, §17.9). The same fallback added **233 other bodies**: the catalogue is **1,022 rows, 1,409 ms cold**, and all six player bodies have art for the first time. `EXTRACTOR_VERSION` 1 → 2 | servuo-plugins |
| 5 | **DONE 2026-09-11.** Item statics and land on demand (§11.1): the `static` and `land` families, hue applied on the shard from `tiledata.mul`, the byte-bounded art cache, `assets.fetch` made family-aware, `families` on `assets.sources`. Website side: the warm pass, per-row `catalog` staleness, and pictures on the marketplace and the character sheet. **39,189 statics and 4,244 land tiles served; the only refusals are the 9,963 + 12,140 empty index slots §4.5 predicted.** Two traps found — the library's bitmap cache poisons a hued sprite, and `PartialHue` decides the picture from a file only the shard has. Protocol stays 8; `EXTRACTOR_VERSION` stays 2 | servuo-plugins, module-uo |
| 6 | **DONE 2026-09-11, and not what this row said.** The measurement came first and changed the phase: a complete one-direction animation set is **174,453 frames / 281.5 MB**, not the ~119,000 estimated, and **the site displays still pictures** — so the deep keys and the bulk-fill switch were **not built** (§11.2, org lead 2026-09-11). What shipped is what the still-picture site was missing: the **73 bodies with no art at action 0 and real art deeper** (a horse at `body/820/a23`), the catalogue key carrying its action, the atlas join that reads it, and §4.10's per-body **action ceiling** — without which the fallback walk itself would serve **452 validated pictures of the next body**. Catalogue **1,022 → 1,095**; `EXTRACTOR_VERSION` 2 → 3; protocol stays 8 | servuo-plugins, module-uo |
| 7 | The atlas over the sidecar (§10); shared-filesystem requirement retired | module-uo |
| 7 | **DONE 2026-09-14.** The atlas over the sidecar (§10); shared-filesystem requirement retired. The measurement came first again and changed the shape: `tree/<label>` → bytes **cannot work** — `Spawns/trammel.xml` is 4.03 MB against a 1 MiB line cap — so a file crosses as **512 KiB chunks, each gzipped**, which is §5's depth scheme paying for itself a second time (§10.1). It is a `tree` **family** on `assets.fetch` rather than §14's separate commands, with `assets.manifest` generalised to match and its **own consent, `Bridge.TreeEnabled`** (§10.2) — so `link` needed nothing for the third phase running. **141 files / 11.9 MB / 158 chunks / 3 pages / 1.33 MB on the wire / 512 ms**, and the atlas built over the bridge is identical to the one built off the disk. Two defects, each found by a different harness: an empty `catalog` refusing every fetch, and `GZipStream` emitting **nothing** for the two empty files stock ServUO ships (§10.3). `PARSER_VERSION` 4 → 5 for one canonical read order (§10.4); protocol stays 8; `EXTRACTOR_VERSION` stays 3 | servuo-plugins, module-uo |
| 8 | Admin surface, Import/Update, approve/reject, activity log | module-uo |
| 9 | Docs pass across five repos; live walk on the real rig | docs |
@@ -1617,3 +1758,27 @@ in the document.
would otherwise hit it: one action past a body's band is 643 validated pictures and 452
byte-identical copies of the next body.
12. **§10: the atlas's three shapes — settled 2026-09-14, phase 7.** Put to the org lead after the
tree was measured and before a line was written, because the first measurement said the phase as
written could not work:
- **A file crosses as 512 KiB chunks, each gzipped.** Not raw base64 (15.1 MB, 31 pages, and two
files on a *stock* tree that can never arrive under a 1 MiB line cap), and not whole-file gzip
— which is smaller on this tree and bounded by nothing, so it works everywhere anyone would
test and fails on the first tree nobody did. The chunk is the guarantee; the compression is
only the saving. See §10.1.
- **It is a `tree` family on `assets.fetch`, with its own switch.** §14's separate `tree.*`
commands and `/tree/*` REST are not built: phase 5's family registry already owns the single
slot, the paging envelope, the key ceiling and the mid-import guard, and reusing it left
`link` with nothing to do for the third phase running. But the *consent* is its own —
`Bridge.TreeEnabled` — because an operator declining to serve their EA-licensed client is not
the same as declining to serve the spawn files they wrote, and the atlas would have been the
casualty. See §10.2.
- **Boot never calls the shard; importing is an admin action** (departing from the
recommendation). The same answer §17.7 gave the cliloc table, and the same reasoning, with one
thing accepted in exchange: an install whose atlas comes over the bridge has *no* automatic
refresh at all, so an edited spawn file stays invisible until someone presses Import. The
alternative — one manifest round trip per boot, ~70 ms and no file bytes — was on the table and
was declined in favour of consistency. The panel and the CLI say so in as many words, and the
skip is logged rather than silent.