Phase 9c is done, and with it the Asset Bridge. §16's row 9c records what the leg actually was, and §17.17 what it found. The site is the only repo whose checks read the platform over the API instead of describing it, so its starting state is a measurement rather than an opinion: - `checkFacts` red in **ten** places — the protocol in all three independent reads, the bundle triple, and the three releases the cutover cut. `moduleApi` and the capability list did not move, which is §17.16's finding restated by a different mechanism. - `checkReference` red in **nine nobody planned for** — every `Bridge.cfg` key phases 5, 6 and 7 added, none of them ever mentioned on the public reference page. No phase row in this plan could have caught that: the rows name the repos a change edits, and this is a repo that only describes them. Two corrections only prose could make: §17.15's "five repositories" had become a page on the site that named `website` among them (protocol 8 touched four, and core's absence is the interesting half), and the site said the bridge has two ways in when the asset plane is a third — a paged bulk read on the reply path, which must not ride the event stream for a structural reason rather than a throughput one. The defect the phase found is a version pinned in a rig: `seedDemo.mjs` pinned protocol **4** under a comment explaining a debt since paid. It reads `platform.json` now. Screenshots retaken against the real shard (D42 holds): 1,095 portraits in 2.1 s, 67,496 names in 1.4 s, 313 item pictures in 0.8 s. Site: RunicGateway/runicgateway.com#31. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
2035 lines
138 KiB
Markdown
2035 lines
138 KiB
Markdown
# Asset Bridge (Protocol 8) — client assets without UOFiddler
|
||
|
||
**Call it the Asset Bridge.** That is the name for this work everywhere — commits, PR titles,
|
||
branches (`feat/asset-bridge-p<n>`), and conversation. The protocol number is **8** and this file is
|
||
`docs/link/v8.md`.
|
||
|
||
**Status:** Design of record, **phases 0-4 complete** (§16). Approved in principle 2026-09-09
|
||
(architecture, asset scope, built-in cliloc decoder, atlas cleanup); refined 2026-09-10 (one
|
||
direction not five, ServUO's own decoders, the UOP reader, the libgdiplus requirement). The phase 0
|
||
spike ran on 2026-09-10 and its findings are §4.5-§4.7 -- §4's decision stands, and the response to
|
||
a malformed record is now measured rather than proposed. Phase 4 built the one decoder §4.3
|
||
reserved, and §4.9 is what it measured: **two** of the eight player bodies it was scoped to exist
|
||
at all, and the same fallback found **233** other bodies, taking the catalogue to **1,022**.
|
||
**Nothing in §17 is open.**
|
||
**Supersedes the manual half of:** `../website/UOFIDDLER.md` (deleted in phase 3, once creature
|
||
art stopped needing it), [`../website/CLILOCS.md`](../website/CLILOCS.md) §Converting,
|
||
[`../website/SPAWN_ATLAS.md`](../website/SPAWN_ATLAS.md) §Artwork and §Configuring the tree.
|
||
|
||
Two features on this platform read data that only exists inside a UO client, and today both reach
|
||
the site by hand: the operator installs UOFiddler, converts `Cliloc.enu` on their own desktop,
|
||
exports sprites one at a time from a GUI, hand-writes a slug→filename JSON map, and copies the
|
||
result to the server. A third — the spawn atlas — avoids UOFiddler but pays a different price: the
|
||
**website** must be able to read the shard's ServUO tree directly, over a bind mount or a shared
|
||
volume.
|
||
|
||
This protocol deletes all three arrangements. The shard already has everything, and the bridge
|
||
already goes to the website.
|
||
|
||
---
|
||
|
||
## 1. The premise, which turns out to be free
|
||
|
||
**A ServUO shard cannot boot without a UO client installation.** It reads maps, statics, tiledata
|
||
and multis out of `.mul`/`.uop` files, and `Config/DataPath.cfg` is where an operator declares
|
||
where those live — *required* on Linux, auto-detected from the registry on Windows. At runtime the
|
||
resolved directories sit in `Server.Core.DataDirectories`, a public static the plugin can read on
|
||
any shard, with no new configuration and nothing for an operator to set up.
|
||
|
||
So the files the operator has been converting on their desktop are already on the shard host, in a
|
||
directory the shard already knows the path of, in a process the bridge already runs inside.
|
||
|
||
Everything below follows from that.
|
||
|
||
### 1.1 What was measured, not assumed
|
||
|
||
Against this machine's ServUO 57.4 tree (`C:\Users\colby\Desktop\ServUO`) and client
|
||
(`D:\Games\Electronic Arts\Ultima Online Classic`, 3.5 GB), loading **ServUO's own
|
||
`Ultima.dll`** — the assembly `overlay/Scripts/Scripts.csproj:39` already carries a
|
||
`<ProjectReference>` to:
|
||
|
||
| Call | Result |
|
||
|---|---|
|
||
| `Art.GetStatic(0…16383)` | 16,384 decoded, 0 errors |
|
||
| `Art.GetStatic(16384…65535)` | 32,766 decoded, 1 empty, 16,385 clean out-of-range errors |
|
||
| `Art.GetLand(0…16383)` | 16,384 decoded, 0 errors |
|
||
| `Animations.GetAnimation(0…2047, 0, 1)` | **1,144** bodies with a decodable first frame, 904 empty, 0 errors — of which **357 are wrong pictures**; see §4.8 |
|
||
| `Hues.GetHue(33)` | loads |
|
||
| `Bitmap.Save(…, Png)` | 852-byte PNG from one creature frame |
|
||
| `Gumps.GetGump(2)` | **hard crash** — `AccessViolationException`, process exit `0xC0000005` |
|
||
| `new StringList("enu", "Cliloc.enu")` | throws — `Non-negative number required` |
|
||
|
||
Two of those rows are load-bearing and are dealt with in §4 and §9. The rest say the same thing:
|
||
**most of the extraction this protocol needs is already implemented, already compiled, and already
|
||
referenced by the plugin's own build.**
|
||
|
||
**Every "decoded" count in that table is an upper bound, not a measurement**, and phases 0 and 1
|
||
spent themselves establishing by how much. `Ultima` reports success for records that do not exist
|
||
(§4.5, §4.8), so the honest reading of the animation row is 787 bodies with art, 357 that return
|
||
the previously-decoded body's bitmap, and 904 that return nothing. The static and land rows are
|
||
overstated the same way. This is not a table to size anything from any more — §11 is.
|
||
|
||
Depth, for §11's sizing: body 400 (human male) has **35 actions × 5 directions = 1,050 frames**.
|
||
One body. §5.1 cuts that by exactly 5×.
|
||
|
||
---
|
||
|
||
## 2. Architecture: the shard extracts, the sidecar forwards, the website decides
|
||
|
||
```
|
||
UO client files (operator's own, on the shard host)
|
||
│ read by the plugin, off the Core thread
|
||
▼
|
||
ServUO shard (servuo-plugins/) ← decodes; resolves body ids; hashes
|
||
│ loopback JSON, request/reply, one batch outstanding at a time
|
||
▼
|
||
uo-link sidecar (link/) ← forwards bytes; decides nothing
|
||
│ REST, bearer-token auth, X-UOLink-Version: 8
|
||
▼
|
||
website (module-uo/) ← stores, names, gates, serves
|
||
```
|
||
|
||
This is deliberately the *only* arrangement that keeps
|
||
[the bridge's standing rules](PLAN.md) intact:
|
||
|
||
- **The sidecar stays a dumb forwarder.** It moves opaque assets and decides nothing about them —
|
||
no audience, no projection, no capability advertisement. Putting the decoders in Rust would have
|
||
meant the sidecar deciding what an asset *is*, on top of re-deriving in Rust what is already
|
||
compiled next door in C#.
|
||
- **Access control stays on the website**, which has the auth machinery and the admin forms.
|
||
- **The shard is still never network-reachable.** Nothing here opens a port; the plugin answers
|
||
requests on the connection it already dialled out on.
|
||
|
||
### 2.1 Why not the sidecar, and why not the operator's desktop
|
||
|
||
A Rust extractor in the sidecar would need ports of: the Mythic cliloc decompressor, `FileIndex`
|
||
(including UOP), the ARGB1555 run-length frame decoder, `Body.def`/`Bodyconv.def` translation,
|
||
`Hues.mul`, and a PNG encoder — weeks of work to re-derive what §1.1 shows already runs. It also
|
||
cannot do §8: resolving a creature slug to a body id requires being inside ServUO.
|
||
|
||
Automating on the operator's desktop (shipping the converter with the installer) removes UOFiddler
|
||
but keeps a manual step and still cannot do §8. It was considered and rejected.
|
||
|
||
---
|
||
|
||
## 3. The transport, and the three traps in it
|
||
|
||
### 3.1 Assets go over the request/reply path, never the event path
|
||
|
||
`link/sidecar/src/app.rs:122` persists **every** non-`pong` event into the SQLite store *and*
|
||
broadcasts it to every WebSocket subscriber. An asset stream on that path would grow the sidecar's
|
||
store without bound and fan megabytes out to every connected client, forever.
|
||
|
||
`rpc.rs`'s `try_route` consumes a correlated reply and `continue`s **before** either of those
|
||
happens. So an asset batch is a reply, not an event. This is not a new mechanism — it is the one
|
||
`char.request`, `account.roster` and `vendor.snapshot` already use.
|
||
|
||
### 3.2 One batch outstanding, always
|
||
|
||
`BridgeLink.Emit()` enqueues onto a **bounded drop-oldest** queue (`Bridge.QueueCap`, default
|
||
10,000). It counts **lines, not bytes** — a design that is correct for live events and dangerous
|
||
for bulk transfer, because 10,000 queued 200 KB replies is 2 GB of shard memory.
|
||
|
||
The rule that makes this safe is flow control, not a bigger queue: **the website requests batch
|
||
*n+1* only after batch *n* has arrived.** Queue depth stays at approximately one. A dropped or
|
||
lost reply simply times out and the batch is re-requested, which is safe because reading a client
|
||
file is idempotent and has no world side effects.
|
||
|
||
**Phase 1 made that a rule the shard enforces rather than one the website is trusted to follow.**
|
||
The asset plane has a **single slot**: a request arriving while one is in flight is answered
|
||
`bridge.busy` — which the sidecar already maps to 425 — and runs nothing. The bound belongs on the
|
||
side where the memory actually is; a documented convention would have held right up until the
|
||
first website bug, and its failure mode is the 2 GB above.
|
||
|
||
The alternative considered was serialising in the sidecar, so a second caller waits instead of
|
||
being refused. It was rejected because a waiter spends the website's own 12 s timeout doing
|
||
nothing, and because it leaves the shard itself unguarded against anything that is not that one
|
||
sidecar.
|
||
|
||
What this costs, and it is deliberate: **a status poll shares the slot with a batch**, so polling
|
||
during a long import is answered 425 until the batch lands. That is honest — this plane really
|
||
does do one thing at a time — and it is why the one genuinely long operation on it, hashing 343 MB
|
||
of client files, is explicitly *not* a job on this worker (§6).
|
||
|
||
### 3.3 The size ceilings are already fixed, and one of them is missing
|
||
|
||
| Limit | Value | Where |
|
||
|---|---|---|
|
||
| Sidecar waits for a shard reply | **10 s** | `rpc.rs` `REPLY_TIMEOUT` |
|
||
| Website waits for the sidecar | **12 s** | `module-uo/server/utils/uoLinkClient.js` `TIMEOUT_MS` |
|
||
| Sidecar → shard line | 1 MiB | `BridgeLink.cs:283` |
|
||
| **Shard → sidecar line** | **1 MiB** | `shard.rs` `MAX_INBOUND_LINE_BYTES` — **added in phase 1**; it was unbounded |
|
||
| **Batch budget** | **512 KiB encoded** | `Bridge.AssetBatchBytes` — **added in phase 1** |
|
||
|
||
The first two bound a batch: it must decode, encode, serialise and cross the wire inside ten
|
||
seconds. The last is a gap this protocol must close — an unbounded `read_line` facing a component
|
||
that is now deliberately sending large lines is a memory-exhaustion shape we would be inventing
|
||
ourselves. **Protocol 8 adds an explicit inbound line cap to the sidecar**, set above the largest
|
||
legal batch and rejecting rather than buffering past it.
|
||
|
||
Batches are therefore sized by bytes, not by count, with the emitter cutting a batch short when it
|
||
would exceed the cap. Base64 costs 33%; the budget must be stated in encoded bytes.
|
||
|
||
**The two numbers, settled in phase 1: a 512 KiB batch budget under a 1 MiB line cap.** The cap is
|
||
symmetric with the one the shard has always applied to its own inbound lines, so both directions
|
||
of this link read the same. The factor of two between them is load-bearing rather than cautious: a
|
||
page always admits its **first** item even when that item alone exceeds the budget, because the
|
||
alternative is an oversized item being skipped for the budget on every page forever and its family
|
||
never making progress. The headroom is what makes that overshoot land on the wire instead of being
|
||
rejected by the cap.
|
||
|
||
An over-long line is **discarded and the connection kept**, which is what `BridgeLink.cs` has
|
||
always done in the other direction. Tearing the link down would take the live event feed with it
|
||
over a single malformed frame, and the reply that was lost simply times out and is re-requested —
|
||
safe, because reading a client file is idempotent.
|
||
|
||
### 3.4 One paging envelope, defined once
|
||
|
||
Five of the families in §14 page: clilocs (phase 2), the body catalogue (3), statics and land (5),
|
||
deep animation keys (6) and the ServUO tree files (7). Left to themselves that is five chances to
|
||
invent five slightly different shapes, and the website would have to learn each one — so phase 1
|
||
defines the envelope before the first family needs it, and `assets.sources` is its first user even
|
||
though it has nothing to page.
|
||
|
||
```
|
||
"files": [ … ], // the array, named by the family
|
||
"more": true, // ask again, with this cursor
|
||
"cursor": "s:4104", // opaque to everyone but the shard; absent when more is false
|
||
"cut": "budget" // budget | end | limit — WHY the page stopped
|
||
```
|
||
|
||
`cut` is the field that is easy to leave out and expensive not to have. "This page is short" has
|
||
three different meanings — the source ran out (`end`), the byte budget was spent (`budget`), or
|
||
the family stopped at a limit of its own (`limit`) — and **only the first means the import is
|
||
finished.** A website that had to infer completion from an item count would resume from the wrong
|
||
place the first time a page happened to land exactly on a boundary.
|
||
|
||
The cursor is deliberately **opaque and shard-defined**. The shard is the only side that knows how
|
||
its own walk is ordered, and a cursor the website could parse is a cursor the website would
|
||
eventually construct.
|
||
|
||
The budget is counted in **UTF-8 bytes**, not characters and not items. Characters would undercount
|
||
every non-ASCII byte in a cliloc row, and the ceiling this has to live inside — §3.3's line cap —
|
||
is measured in bytes.
|
||
|
||
---
|
||
|
||
## 4. The decoders are ServUO's own — decided, and the crash is narrower than it looked
|
||
|
||
**We call ServUO's vendored `Ultima` (decided 2026-09-10).** No decoders are reimplemented.
|
||
`overlay/Scripts/Scripts.csproj:39` already references the project, so the art half of this protocol
|
||
costs plumbing rather than pixel code, and only §9's cliloc decompressor is written from scratch.
|
||
|
||
The reason that is safe, rather than merely cheap, is a distinction §1.1 did not draw at first.
|
||
|
||
### 4.1 The crash lives on one code path, and nothing we call uses it
|
||
|
||
`Gumps.GetGump(2)` does not fail — it **corrupts the process**: `AccessViolationException`, exit
|
||
`0xC0000005`. That is a corrupted-state exception, uncatchable by an ordinary `try/catch` on .NET
|
||
Framework 4.8, so in-process on a live shard it is a crash with players on it. That much is
|
||
alarming, and on its own it looked like an argument against using this library at all.
|
||
|
||
It is not, because of how the three decoders construct their `FileIndex`:
|
||
|
||
| Decoder | UOP file | `hasExtra` | Probed |
|
||
|---|---|---|---|
|
||
| `Art` | `artLegacyMUL.uop` | **false** | 49,150 statics + 16,384 land tiles, **0 faults** |
|
||
| `Animations` | *none — legacy `anim*.mul` only* | — | 1,144 bodies, **0 faults** |
|
||
| `Gumps` | `gumpartLegacyMUL.uop` | **true** | **faults on the second id** |
|
||
|
||
`FileIndex.cs`'s own comment says the extra-field handling exists *for* `gumpartlegacy.uop` — it is
|
||
the one UOP layout carrying an extra field, and `hasExtra: true` is the branch written to cope with
|
||
it. **Gumps is the only caller that sets it.** So the fault is not a general fragility in this
|
||
library's `unsafe` code; it is a bug on a branch that exactly one decoder reaches, and that decoder
|
||
is already out of scope (§11).
|
||
|
||
The rule this turns into is a safety rule, not a preference: **nothing in this protocol calls
|
||
`Ultima.Gumps`.** Adding gump art later means fixing or replacing that path first, deliberately,
|
||
not discovering it in production.
|
||
|
||
### 4.2 What the decision accepts
|
||
|
||
Three costs come with it, all known and none of them blocking:
|
||
|
||
1. **Six of the twelve stock player-character bodies have no art on this path** — both human ghosts
|
||
and every gargoyle body (§5.2). `Animations` never reads `AnimationFrame*.uop`. **This one is
|
||
not merely accepted: §4.3 adds a decoder for it**, because it is the player character and the
|
||
scope says player models.
|
||
2. **`System.Drawing` is a hard dependency, in the decode and not just the encode.** `Frame`
|
||
writes ARGB1555 straight through a `LockBits` pointer, so a Linux shard needs **libgdiplus** to
|
||
read a sprite at all. That is a stated prerequisite — §4.4.
|
||
3. **We inherit whatever `Ultima` a given ServUO vendors**, which can change under a shard upgrade.
|
||
`EXTRACTOR_VERSION` (§7) is the mitigation: it already counts as drift, so a shard whose library
|
||
changed re-derives on the next import.
|
||
|
||
The residual risk that remains is a patched or custom client tripping an out-of-bounds read on a
|
||
path we *do* call. §16's phase 0 is where that gets exercised rather than assumed.
|
||
|
||
### 4.3 One decoder we do write: UOP animation
|
||
|
||
**Built in phase 4. What it reaches is not what this section predicted, and §4.9 is the
|
||
measurement.** The section stands as the argument; the numbers below are the corrected ones.
|
||
|
||
The bodies the legacy path cannot resolve get a decoder rather than a caveat. It is deliberately
|
||
the **narrowest possible** addition: a reader for `AnimationFrame*.uop`, used *only* where the
|
||
vendored code has nothing. Everything it can already decode keeps going through it — which after
|
||
phase 4 is still 787 of the catalogue's 1,022 bodies.
|
||
|
||
This client ships `AnimationFrame1/2/3/4/6.uop` (107, 118, 253, 115 and 24 MB) plus
|
||
`AnimationSequence.uop`. ServUO's `FileIndex` already contains a UOP reader — but `Animations`
|
||
never constructs one, and the UOP animation *payload* is its own format rather than a repackaged
|
||
mul record, so wiring the existing `FileIndex` in is not sufficient. Two license-compatible
|
||
references exist to work from: ClassicUO's animation loader (GPL-3, and we are GPL-3-or-later) and
|
||
UOFiddler 4.22 (Beerware, already established in §9).
|
||
|
||
**What it was for, and what it actually delivers.** The phase was scoped to the eight player
|
||
bodies §4.8 left undecodable. **Two of the eight are in the client at all** — gargoyles 666 and
|
||
667, in `AnimationFrame3.uop`. The six ghost bodies are in no package, and §4.9 shows how that was
|
||
established rather than assumed. What the same fallback *does* deliver is **233 further bodies**
|
||
the catalogue had nothing for, so the phase's real effect is 787 → **1,022**.
|
||
|
||
**The trap this must not fall into, and it is why the phase exists at all.** `Bodyconv.def` maps
|
||
gargoyle 666 to `anim5`, and `BodyConverter.Convert` faithfully returns fileType 5 — where this
|
||
client's index runs out before the entry. Asking the *other* anim files for index 666 does not
|
||
fail. It returns 175 decodable action/direction combinations of **a giant spider**, because
|
||
something unrelated occupies that index in `anim2.mul`; fileTypes 3 and 4 return misaligned colour
|
||
fragments. Rendered and confirmed.
|
||
|
||
So the extractor takes `BodyConverter.Convert`'s answer and, if that yields nothing, reports
|
||
nothing. **It must never sweep file types looking for a hit** — that does not find missing art, it
|
||
silently puts a spider on the gargoyle page, and no error is raised anywhere. A "0 rows" outcome is
|
||
correct behaviour; a plausible wrong picture is the failure this protocol most needs to avoid,
|
||
because nothing downstream can detect it.
|
||
|
||
**Searching all five UOP packages for one body is not that, and the difference is structural.** A
|
||
legacy index is addressed **by position**: nothing in the file says which body a record belongs to,
|
||
so a wrong lookup returns a confident picture of something else. A UOP entry is addressed by the
|
||
**hash of a name that contains the body id** — `build/animationlegacyframe/000666/00.bin` — and the
|
||
payload then declares that id again in its own header, which the reader checks against the id it
|
||
asked for. A hit is proof of identity rather than a coincidence of position. Measured: no hash
|
||
appears in two packages.
|
||
|
||
### 4.4 Requirement: a Linux shard host needs libgdiplus
|
||
|
||
**Stated prerequisite, not a soft recommendation.** ServUO targets `net48`, so on Linux it runs under
|
||
Mono, and Mono's `System.Drawing` is a thin layer over **libgdiplus**. §4.2 put `System.Drawing` in
|
||
the decode path, so without that library a Linux shard cannot extract art at all — the cliloc table
|
||
(§9) and the atlas files (§10) are unaffected, since neither touches pixels.
|
||
|
||
**Windows shard hosts need nothing.** `System.Drawing` ships with .NET Framework. This is a
|
||
Linux-only prerequisite and most shards will never read this section.
|
||
|
||
| Host | Get it with |
|
||
|---|---|
|
||
| Debian / Ubuntu | `sudo apt-get install libgdiplus` — in Debian since bullseye (6.0.4) and bookworm/trixie (6.1), and in Ubuntu universe |
|
||
| Fedora / RHEL | `sudo dnf install libgdiplus` (EPEL or the Mono repository) |
|
||
| Docker | `RUN apt-get update && apt-get install -y libgdiplus` in the shard image |
|
||
| Alpine, or a distro with no package | Build from source — see the repository below. This is the awkward case and is worth avoiding by choosing a Debian-based image |
|
||
|
||
Upstream is <https://github.com/mono/libgdiplus>, with the Mono project's own page at
|
||
<https://www.mono-project.com/docs/gui/libgdiplus/>.
|
||
|
||
**One thing to know before depending on it: that repository was archived in March 2025 and is
|
||
read-only.** Distributions still package and patch it, so `apt-get install libgdiplus` is a normal,
|
||
supported thing to do today — but upstream is not maintained. It is the strongest long-term argument
|
||
for eventually moving extraction off `System.Drawing`, and phase 4's UOP reader (§4.3) is written
|
||
without it precisely so that door stays open — decode and encode both (§4.9). It does not change
|
||
the decision now, and §17.9 is the decision not to walk through that door yet: a host with no
|
||
libgdiplus still gets the flat refusal below rather than the ~238 bodies that no longer need it.
|
||
|
||
**How its absence must present.** Never a stack trace and never a 500. Missing libgdiplus is a
|
||
named, actionable outcome in the same family as the cliloc reader's `COMPRESSED`:
|
||
|
||
```
|
||
status: unavailable
|
||
code: NO_IMAGING
|
||
reason: This shard host cannot render images — Mono's System.Drawing needs libgdiplus.
|
||
Install it (apt-get install libgdiplus) and re-run the import. Cliloc and
|
||
atlas import are unaffected.
|
||
```
|
||
|
||
The installer's `doctor` checks for it and reports it alongside its other host checks, so an
|
||
operator learns about this while setting the shard up rather than from an empty bestiary weeks
|
||
later.
|
||
|
||
---
|
||
|
||
### 4.5 What phase 0 measured, and the rule it produced
|
||
|
||
Phase 0 ran §4's decoders from inside a live ServUO 57.4 against a client broken in 21 catalogued
|
||
ways ([`servuo-plugins/tools/scaffolding/README.md`](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/src/branch/main/tools/scaffolding/README.md)
|
||
carries the full results). §4's decision **stands** — nothing faulted on a path this protocol
|
||
calls, and §9's cliloc reader reproduced UOFiddler's 123,490-entry table byte for byte in 218 ms.
|
||
|
||
But the spike was looking for the wrong kind of failure, and found a worse one.
|
||
|
||
**A malformed record does not usually throw. It renders the previous asset.** `LoadStatic` and
|
||
`LoadLand` decode out of `m_StreamBuffer`, which is reused across calls, only ever grown, and
|
||
filled by a `stream.Read` whose return value is discarded. So a record that is short, absent or
|
||
out of bounds produces a real bitmap of whatever was decoded last — reported as success by every
|
||
count in the library and undetectable by anything downstream.
|
||
|
||
On the **stock, unmodified** client on this machine that is **22,102 ids**: 9,962 statics and
|
||
12,140 land tiles whose index entry reads `lookup 0, length 0`. `FileIndex.Seek` rejects
|
||
`lookup < 0` and `length < 0`, and zero is neither, so it treats an empty slot as a hit. A bulk
|
||
import that trusted the library would have written 22,102 duplicate images into the site under
|
||
ids that have no art. §1.1's "32,766 decoded" was counting these.
|
||
|
||
**This is specific to the UOP path** (see §4.6), because `artidx.mul` stores `-1` for an absent
|
||
record where an unmapped UOP slot is a zeroed struct.
|
||
|
||
So the rule, and it is the deliverable phase 1 inherits:
|
||
|
||
> **Validate before calling.** The extractor judges an index entry — and, for statics, the record
|
||
> header and row table behind it — *before* handing the id to `Ultima`. A record that fails is
|
||
> reported as absent, never decoded.
|
||
|
||
The checks, all of which phase 0 implemented and measured as `BridgeAssetValidator`:
|
||
|
||
| Check | The shape it stops |
|
||
|---|---|
|
||
| `lookup >= 0`, `length > 0` | the 22,102 empty slots above |
|
||
| `lookup + length <= <data file>.Length` | **`Seek` checks that a record *starts* inside the file and never that it *ends* inside it**; a short read then decodes the previous asset |
|
||
| the same bound against `verdata.mul` for a patched entry | `Verdata.Seek` is bounds-checked **nowhere at all** |
|
||
| land records are at least 2,024 bytes | `LoadLand` reads exactly that many whatever the length says |
|
||
| declared width and height within a ceiling | `LoadStatic` allocates `new Bitmap(width, height)` from two bytes in the file — phase 0 got a ~128 MB allocation out of an edit, and the same field can ask for 8 GB |
|
||
| walk the row table and every run, bounded | `LoadStatic`'s two guards bound the **write** into the bitmap and nothing bounds the **read** out of the record |
|
||
|
||
Measured against the patched client, this refused all eight record-level defects, seven of which
|
||
the library rendered without raising anything. Measured against the stock client it refused
|
||
**nothing** across 49,151 statics and 16,384 land tiles. That second number is the one that makes
|
||
the boundary defensible: a checker that refuses real art would be worse than no checker.
|
||
|
||
Two more ways an id with no art yields a picture, both of which the extractor must handle itself:
|
||
`Art.GetStatic(id, false)` **throws** `IndexOutOfRangeException` above the index's own ceiling
|
||
(16,385 ids in a full sweep), and `Art.GetStatic(id)` with the default `checkmaxid: true` is worse
|
||
— `GetLegalItemID` maps an out-of-range id to **0** and returns **item 0's picture**. Take the
|
||
ceiling from the index that was opened, and pass `checkmaxid: false` so an overrun is loud.
|
||
|
||
**The animation path has none of this yet**, and phase 0 proved it needs it: the patched client's
|
||
verdata entry for body 34 points past `verdata.mul`'s end and the wolf still "decoded", counted
|
||
among the 1,144 successes while rendering something else. `GetAnimation` additionally allocates
|
||
`new int[frameCount]` straight from a file-supplied int. Extending the validator to animations is
|
||
phase 1 work, not a phase 9 tidy-up.
|
||
|
||
### 4.6 The UOP wins outright, and `art.mul` is never opened
|
||
|
||
`FileIndex`'s UOP constructor ends with a bare `MulPath = uopPath`. **When `artLegacyMUL.uop` is
|
||
present it wins, and `art.mul` / `artidx.mul` are not opened at all.** Every current client ships
|
||
the UOP, so this is the normal case and not an edge one.
|
||
|
||
It matters twice.
|
||
|
||
**For us:** an index entry's `lookup` is an offset into whichever file `FileIndex` resolved, so any
|
||
bound taken against `art.mul` while the index holds UOP offsets is not approximate — it is
|
||
meaningless. Phase 0's first run refused **34,299 perfectly good statics** for "declaring
|
||
10533x2085" on exactly that mistake, and every one of those refusals read like a real finding. The
|
||
extractor must resolve the data file with `FileIndex`'s own precedence, not by name.
|
||
|
||
**For operators:** a shard that adds custom art to `art.mul` while the UOP is still in place gets
|
||
nothing, silently. Worth a line in the admin surface's diagnostics (§14) rather than leaving an
|
||
operator to discover it as "my art did not import".
|
||
|
||
### 4.7 The gump crash, reproduced where it counts
|
||
|
||
§4.1 argued from source that the access violation lives on the `hasExtra: true` branch only `Gumps`
|
||
reaches. Phase 0 called `Ultima.Gumps.GetGump(2)` **once, from inside a running shard**: the ServUO
|
||
process disappeared. No exception line, no `catch` reached, no shutdown, nothing in the console —
|
||
the probe's checkpoint file, written before the call, was the entire record of what happened.
|
||
|
||
`AccessViolationException` is a corrupted-state exception that .NET Framework 4.8 does not deliver
|
||
to ordinary handlers, so on a live shard this is a crash with players on it and **there is no
|
||
in-process defence**. §4.1's "nothing in this protocol calls `Ultima.Gumps`" is therefore an earned
|
||
safety rule rather than a scoping preference, and adding gump art later means fixing that path
|
||
first, deliberately.
|
||
|
||
---
|
||
|
||
### 4.8 What phase 1 measured: the animation path has the same defect, and it is worse
|
||
|
||
§4.5 ended by saying the animation path had no validator and that extending it was phase 1's work
|
||
rather than a phase 9 tidy-up. Phase 1 built it, ran it, and the reason that sentence was right is
|
||
larger than the verdata entry that prompted it.
|
||
|
||
**357 of the 1,144 "decodable" bodies are wrong pictures, on the stock client.** Their index entry
|
||
reads `length 0` — no record at all — and `GetAnimation` returns a real bitmap anyway, for the same
|
||
reason `LoadStatic` does: `m_StreamBuffer` is reused, only ever grown, and filled by a
|
||
`stream.Read` whose return value is discarded.
|
||
|
||
Measured directly, because a count could not tell:
|
||
|
||
| Decode body 320 (`lookup 22638982, length 0`) straight after… | What comes back |
|
||
|---|---|
|
||
| body 12, the dragon | the dragon's bitmap, 176×167, identical hash |
|
||
| body 34, the wolf | the wolf's dimensions, 35×34 |
|
||
| body 400, the human male | the human's bitmap, 27×63, identical hash |
|
||
|
||
That is not a near miss or a misaligned fragment. Body 320 has no art, and it renders **whichever
|
||
creature was decoded before it** — which means the picture a bestiary page got would depend on the
|
||
order the importer happened to walk the catalogue in.
|
||
|
||
So the working set is **787 bodies, not 1,144**:
|
||
|
||
| | Bodies 0–2047, direction 1, first frame |
|
||
|---|---|
|
||
| Real art | **787** |
|
||
| Wrong pictures (empty record, library returned a bitmap) | **357** |
|
||
| Absent, and the library agreed | 903 |
|
||
| `bodyconv` resolves nowhere, nothing swept (§4.3) | 1 |
|
||
| Refused by the record walk | **0** |
|
||
| Threw | 0 |
|
||
|
||
**That last-but-one row is the number that matters as much as the first.** The record-level
|
||
animation checks — palette, frame count, frame table, and every run header walked against both the
|
||
record's own length and the bitmap it locks — refused **nothing** across every real body on a stock
|
||
client. §4.5's rule holds: a checker that refuses real art is worse than no checker, and this one
|
||
does not.
|
||
|
||
#### The player bodies: four of twelve, not six
|
||
|
||
§5.2's table was built from the library's answer alone, and two of the six bodies it listed as
|
||
decoding do not have art:
|
||
|
||
| Body | Index entry | Library | Actually |
|
||
|---|---|---|---|
|
||
| Human male/female (400, 401) | real | 24×64, 24×63 | **art** |
|
||
| Elf male/female (605, 606) | real | 24×64, 24×63 | **art** |
|
||
| **Elf ghosts (607, 608)** | `lookup 27221378, length 0` | 24×63 | **the previous body's picture** |
|
||
| Human ghosts (402, 403) | `lookup -1` | nothing | honestly absent |
|
||
| Gargoyle (666, 667, 694, 695) | `lookup -1` / past the index | nothing | honestly absent |
|
||
|
||
Confirmed the same way: body 607 decoded after the dragon is the dragon, after the wolf is the
|
||
wolf, after the human male is the human male. Its 24×63 was the elf *female*'s dimensions, because
|
||
606 is what the catalogue walk decoded immediately before it.
|
||
|
||
Two consequences, both of which change work elsewhere in this document:
|
||
|
||
- **§4.3's UOP decoder covers eight player bodies, not six.** The elf ghosts join the human ghosts
|
||
and the four gargoyle bodies. Phase 4's scope grows by two ids and its argument does not change.
|
||
— *Phase 4 then found that six of those eight are in no client file at all, and recovered the
|
||
other two (§4.9). The reasoning above is why they were in scope; the answer is there.*
|
||
- **The `lookup -1` / `length 0` distinction is the whole difference between an honest absence and
|
||
a wrong picture**, and it is not visible from outside the index. `artidx.mul` and the legacy
|
||
`anim*.idx` write `-1` for a record that is not there; an empty UOP slot, and evidently a
|
||
deliberately blanked legacy entry, is a zeroed struct. `FileIndex.Seek` rejects the first and
|
||
accepts the second, and the second is 357 creatures and two playable ghosts.
|
||
|
||
#### Why this could not have been found any other way
|
||
|
||
Phase 0 ran this exact sweep and reported "1,144 decoded, 904 empty, 0 faults", and every one of
|
||
those numbers is true. The library raised nothing, returned bitmaps of plausible sizes, and agreed
|
||
with itself. Nothing downstream of the decode — not a count, not an exception, not a hash of the
|
||
output, not a look at one picture in isolation — distinguishes body 320's dragon from body 12's.
|
||
The only things that did were **validating the index entry before the call** and **decoding the
|
||
same id twice after different neighbours**.
|
||
|
||
That is the same method note §4.3 ended on, and this is its second confirmed catch. Any time a UO
|
||
file lookup is addressed by index, a success count is evidence of nothing.
|
||
|
||
---
|
||
|
||
### 4.9 What phase 4 measured: two of eight, and 233 nobody was looking for
|
||
|
||
§4.3 was scoped to eight player-character bodies. Before writing the reader, phase 4 opened the
|
||
five packages and asked which of the eight were in them.
|
||
|
||
**Two.** Gargoyles 666 and 667, in `AnimationFrame3.uop`. The other six are in no package at all.
|
||
|
||
| Body | Legacy index | In a UOP package? |
|
||
|---|---|---|
|
||
| Gargoyle 666, 667 | index position runs **past the end of `anim5.idx`** | **yes — `AnimationFrame3.uop`** |
|
||
| Human ghosts 402, 403 | `lookup -1` — honest absence | no |
|
||
| Elf ghosts 607, 608 | `length 0` — the §4.8 shape | no |
|
||
| Gargoyle ghosts 694, 695 | `lookup -1` | no |
|
||
|
||
**That absence is established, not merely unfound**, which matters because "I looked and did not
|
||
see it" is exactly how a name scheme that is subtly wrong presents. The five packages hold
|
||
**10,724 entries** between them, and hashing `build/animationlegacyframe/%06d/%02d.bin` over bodies
|
||
0–4095 and actions 0–99 claims **all 10,724** — every entry accounted for, none left over for
|
||
another naming to hide in. No client ships ghost art. §5.2 is where that lands, and §17.9 is the
|
||
decision it produced.
|
||
|
||
#### The 233 the phase was not asked for
|
||
|
||
The rule "legacy first, UOP when it has nothing" is the same code whether it is applied to six
|
||
bodies or to all of them, and applied to all of them it finds **244 bodies with a group 0** in the
|
||
packages, of which only **8** also have legacy art. So the catalogue's working set goes from 787
|
||
to **1,022**, measured on the live rig:
|
||
|
||
| | Rows | Of which |
|
||
|---|---|---|
|
||
| `legacy` — vendored `Animations` over `anim*.mul` | **787** | 366 Equipment, 240 Monster, 95 Animal, 65 unlisted, 17 Human, 4 Sea |
|
||
| `uop` — §4.3's reader | **235** | 97 Equipment, 57 Monster, 50 unlisted, 26 Animal, 3 Human, 2 Sea |
|
||
|
||
The mix is worth a sentence because "add every body" sounds like it changes what a catalogue *is*:
|
||
it does not. The legacy 787 was **already** 366 equipment bodies — weapons and tools drawn alone
|
||
for the client to composite — so the UOP's 97 more change the proportion by a point. What a
|
||
bestiary sees is 83 creatures it could not show before, plus the gargoyles.
|
||
|
||
#### The format, as read rather than as assumed
|
||
|
||
A group file is one action of one body, every direction concatenated, zlib-compressed inside the
|
||
package (`flag == 1`, `78 9c`). The payload is not a repackaged mul record:
|
||
|
||
```
|
||
'AMOU' version decompressedSize bodyId … frameCount frameTableOffset
|
||
frame table: frameCount × 16 bytes (group, frameId, two unknowns, pixel offset)
|
||
each frame: its OWN 512-byte ARGB1555 palette, then centreX/centreY/width/height,
|
||
then the same run-length rows the legacy Frame decoder walks
|
||
```
|
||
|
||
Two consequences the reader is built on. The per-frame palette means **no `Ultima.Frame` and no
|
||
`Bitmap`**: the runs are written into a `ushort[]` of our own, which is what §4.4's promise about
|
||
this reader and `System.Drawing` was always about — and it leaves the encode as the only GDI+ step,
|
||
which phase 4 replaced with a ~200-line PNG writer (zlib around net48's raw-deflate-only
|
||
`DeflateStream`, CRC32, one IDAT, filter 0).
|
||
|
||
And **direction is a slice of the frame table**, not an index into five records: direction *d*
|
||
starts at `d * (frameCount / 5)`. On **nine** of the 244 bodies the frame count is not a multiple
|
||
of five (41, 42, 46…), where integer division lands slightly early in the run. That is what the
|
||
reference implementations do and it is the right trade here, because the failure §4.5 and §4.8 are
|
||
written against is a picture of the **wrong creature**, and this cannot produce one — the worst
|
||
case is the right creature at a marginally different angle, on nine bodies, against losing nine
|
||
bodies outright.
|
||
|
||
#### Validate-as-we-go, and the measurement that says it is in the right place
|
||
|
||
§4.5's rule is "validate before calling", because `Ultima`'s decoders take their bounds from the
|
||
file they read. Here there is no library to validate ahead of: this code **is** the decode. So the
|
||
same discipline appears as a bound on every read — the block chain against the file length, a
|
||
record against the file, the inflated length against the declared one, the frame table against the
|
||
payload, and every run header against **both** the record's remaining bytes and the bitmap it
|
||
writes into, which is the bound `Frame` itself does not have.
|
||
|
||
Measured exactly the way §4.5 was measured, because the second number is what makes a checker
|
||
defensible: across every UOP body on a stock client it refused **nothing** that carries art. The
|
||
one body it refuses, 286, declares a 0×0 frame — which the vendored decoder also treats as no art
|
||
rather than as damage, so it reports absent silently rather than logging a defect on every scan.
|
||
|
||
#### The live rig
|
||
|
||
Real sidecar, real ServUO, this machine's client:
|
||
|
||
| | |
|
||
|---|---|
|
||
| Catalogue | **1,022 rows in ONE page, 1,409 ms cold** |
|
||
| Player bodies | **six, and all six have art** — 400/401/605/606 `legacy`, 666/667 `uop` — every one at direction 0 |
|
||
| Direction split | 6 at direction 0, 1,016 at direction 1 |
|
||
| The six ghost bodies | `absent` |
|
||
| Duplicate-sha256 scan | 45 groups, of which **one** is new |
|
||
|
||
That last row is phase 3's cheap detector for the §4.8 bug, and it earned its keep again. The new
|
||
group is bodies 1531 and 1532 — two **distinct** records, 48,281 bytes each, whose first frames are
|
||
identical pixels. What separates that from a reused-buffer defect is not the hash: it is that each
|
||
payload declares its own body id, and the reader checked. A duplicate you can explain at the source
|
||
is data; one you cannot is the bug.
|
||
|
||
Manifest and fetch rows now carry `source` (`legacy` / `uop`) for exactly this reason — an operator
|
||
looking at a wrong picture can say which half of the extractor to doubt, and an acceptance walk can
|
||
prove the fallback fired rather than infer it from a count.
|
||
|
||
**`EXTRACTOR_VERSION` is 2.** Every client file is byte-identical and the answer is different,
|
||
which is the whole of what §7's number exists to say. The UOP packages join `assets.sources` and
|
||
the catalogue id with it, so patching one is drift rather than a silent no-op — and resolving them
|
||
needed its own lookup, because `Ultima.Files`' table of known client files predates UOP animations
|
||
and answers null for every `AnimationFrame*.uop` however present they are. The replacement matches
|
||
**case-insensitively by enumeration**, which is a Linux-host concern rather than a tidiness one.
|
||
|
||
### 4.10 What phase 6 measured: the ceiling, and 452 validated pictures of the wrong body
|
||
|
||
§4.8 found the library returning the previously-decoded body's bitmap for an index entry that
|
||
reads `length 0`. Phase 6 went looking one axis over — along **actions** rather than bodies — and
|
||
found the same class of failure with none of the tells.
|
||
|
||
An index entry for an animation is `bodyBase + action * 5 + direction`, and `bodyBase` comes from a
|
||
**band**: 110 slots for a high-detail body (22 actions), 65 for a low-detail one (13), 175 for a
|
||
people body (35). The bands are contiguous, so the slots immediately after a body's band are the
|
||
**next body's**. Ask for one action past the ceiling and the arithmetic lands on a real entry, at a
|
||
real offset, holding a real animation record — of a different creature.
|
||
|
||
Measured on this machine's stock client, over the 795 bodies the legacy path serves:
|
||
|
||
| Asking one action past the band | Bodies |
|
||
|---|---|
|
||
| Refused by §4.5's validator (the entry is absent or unreadable) | 152 |
|
||
| **Passes `CheckEntry` and `AnimationSane` and decodes** | **643** |
|
||
| ...and the picture is **byte-identical to body+1's action 0** | **452** |
|
||
|
||
Body 1 action 22 is an ettin. Body 3 action 22 is an imp. Both confirmed by rendering them beside
|
||
the body they belong to, because a count would have said the walk was fine — phase 0's validator
|
||
cannot catch this and is not wrong to miss it: there is nothing defective about the record. The
|
||
only defence is to **refuse the address**, so the ceiling lives inside
|
||
`BridgeAssetValidator.ResolveAnimation`, where every caller already goes and no caller can skip it.
|
||
|
||
**The ceiling is the banding, not the library's own `GetAnimLength`.** That function exists, looks
|
||
authoritative and disagrees with the index arithmetic on exactly one body of this client: a body
|
||
reaching file type 5 as id 34 is excluded from the first band by `Animations.GetFileIndex`'s own
|
||
"looks strange, though it works" special case, so it owns **13** actions while `GetAnimLength`
|
||
answers **22**. Taking the larger number is nine actions of somebody else's art, reached on this
|
||
client by translation from body 276. So the count is derived from the same switch that produces the
|
||
offset, in the same file, where the two cannot drift apart.
|
||
|
||
This is the §4.3 never-sweep rule again in a third disguise. Sweeping file types puts a spider on
|
||
the gargoyle page; trusting a `length 0` entry puts the last creature decoded on this one; walking
|
||
past a band puts the *next* creature on it. All three decode cleanly, all three report success, and
|
||
all three are caught by refusing to ask rather than by checking the answer.
|
||
|
||
---
|
||
|
||
## 5. Addressing: one key for every asset
|
||
|
||
Every asset the bridge can serve is named by a single string key, and the key is the cache key,
|
||
the hash key, the filename stem and the manifest row id:
|
||
|
||
```
|
||
static/3922 one item graphic
|
||
static/3922/h33 the same graphic, hue 33 applied
|
||
land/3 one land tile
|
||
body/34/a0 creature body 34, action 0, first frame
|
||
body/820/a23 body 820, whose action 0 is empty — a horse (§11.2)
|
||
body/400/a0/f0..f9 human male, action 0, all ten frames — NOT SERVED (§11.2)
|
||
cliloc/enu the whole converted string table (not an image)
|
||
tree/Spawns/Trammel.xml a ServUO tree file (§10)
|
||
```
|
||
|
||
**The catalogue's key names whichever action its picture came from.** One row per body either way,
|
||
and for all but 73 of this client's bodies that action is 0 — but a body with no art there is
|
||
catalogued at the first action that has any, and the key says which (§11.2). Calling it `a0`
|
||
regardless would have been fewer changes downstream and a key that lies about its content, which
|
||
is the failure §4.5, §4.8 and §4.10 each describe from a different direction.
|
||
|
||
**The frame depth is defined and not served.** `body/400/a0/f0..f9` is well-formed under this
|
||
scheme and every request for it is refused `unsupported` — see §11.2 for the decision and what it
|
||
would cost to change.
|
||
|
||
Three properties this shape buys:
|
||
|
||
- **Hue is part of the key, not a transform.** `itemId` and `hue` are already on the wire together
|
||
(`BridgeMarket.cs:582`, `BridgeProfile.cs:314`), so a marketplace listing already knows the exact
|
||
key for its own picture. Applying hues website-side would mean shipping `Hues.mul` semantics into
|
||
Node for no gain.
|
||
- **Depth is expressible without being mandatory.** `body/400/a0` and `body/400/a0/f0..f9` are the
|
||
same addressing scheme at two depths, which is what lets §11 bulk-import thumbnails and, if
|
||
anything ever wants them, fetch full animations on demand without a second protocol. Phase 6
|
||
measured what that second depth costs and deferred it (§11.2); the point of the scheme is that
|
||
deferring it is a decision about *what to serve*, not about what can be named.
|
||
- **Nothing in the key is client-version-specific**, so a client patch changes an asset's *bytes*,
|
||
not its name — which is what makes §7's delta work.
|
||
|
||
### 5.1 There is no direction segment, because only one direction is wanted
|
||
|
||
Bodies are stored in **five** directions and the client mirrors three of them to reach eight. Only
|
||
one is needed here, so **direction is fixed by the extractor and is not part of the key**. Leaving
|
||
it in would advertise a choice nobody is going to vary and would five-fold every count in §11 for
|
||
nothing.
|
||
|
||
**Which one depends on whether the body is a player character:**
|
||
|
||
| Body | Direction | Why |
|
||
|---|---|---|
|
||
| A player character body | **0** — head-on, facing the viewer | A character is a portrait; it should look at you |
|
||
| Everything else | **1** — front three-quarter | The view that actually reads as a creature (see the caveat below) |
|
||
|
||
Which index is which was determined by **rendering all five** for a human, a wolf and a dragon
|
||
rather than from a table, because the answer is not obvious and the small-thumbnail version of the
|
||
same test suggested the exact opposite:
|
||
|
||
| Index | View |
|
||
|---|---|
|
||
| **0** | **Head-on, facing the viewer** — face, chest and front legs visible |
|
||
| 1 | Front three-quarter |
|
||
| 2 | Full side profile |
|
||
| 3 | Rear three-quarter |
|
||
| 4 | Directly away — back of the head, and a quadruped's tail toward the camera |
|
||
|
||
The caveat the render made obvious is what produced the split: **index 0 is the least legible view
|
||
for four-legged and long-bodied creatures.** A wolf seen head-on is a dark blob; the same wolf at
|
||
index 1 is unmistakably a wolf, which is also why UOFiddler's own thumbnail list picks that view. A
|
||
humanoid has no such problem — it reads fine head-on, and head-on is what a character portrait
|
||
wants.
|
||
|
||
Both indices stay **configuration values** (defaulting to 0 and 1), so changing the catalogue's mind
|
||
later is a setting and a re-import, not a protocol change.
|
||
|
||
### 5.2 "Player character body" is asked of the shard, never hardcoded
|
||
|
||
`Server.Race.AllRaces` gives every registered race, and each carries `MaleBody`, `FemaleBody`,
|
||
`MaleGhostBody` and `FemaleGhostBody`. The plugin enumerates the **living pair** per race, and that
|
||
set — nothing else — takes index 0. On stock ServUO 57.4 that is six ids:
|
||
|
||
| Race | Male | Female | Male ghost | Female ghost |
|
||
|---|---|---|---|---|
|
||
| Human | **400** | **401** | 402 | 403 |
|
||
| Elf | **605** | **606** | 607 | 608 |
|
||
| Gargoyle | **666** | **667** | 695 | 694 |
|
||
|
||
This is the §8 argument again in miniature: only code inside ServUO can answer it, and asking is
|
||
the only thing that works on a shard with a custom race. Two details make the case that a
|
||
hardcoded list would have been wrong — `RaceDefinitions.cs` passes the gargoyle's ghost bodies in
|
||
the **opposite order** to the other two races (695 male, 694 female), and a shard that calls
|
||
`RegisterRace` adds ids no table of ours would contain.
|
||
|
||
**The ghost columns are greyed above because no client has art for any of them, anywhere.** That is
|
||
phase 4's measurement and it is what removed them from the set (§17.9). The evidence is in §4.9:
|
||
402/403 and 694/695 read `lookup -1` in the legacy index, 607/608 read `length 0` — the §4.8 shape,
|
||
where the library hands back the previously-decoded body's picture — and **none of the six appears
|
||
in any `AnimationFrame*.uop`**, established by claiming all 10,724 entries of the five packages
|
||
with a single name scheme rather than by failing to find them.
|
||
|
||
So the set is asked of the shard exactly as before; only the question changed, from four ids per
|
||
race to two. A shard whose client *does* ship ghost art loses nothing: the body is catalogued like
|
||
any other, at direction 1 rather than 0.
|
||
|
||
**And with phase 4 in, all six have art.** The gargoyles were the last gap, and they were never a
|
||
missing-art problem: `Bodyconv.def` sends 666 and 667 to `anim5`, at an index past the end of
|
||
`anim5.idx`, while `AnimationFrame3.uop` has held both all along.
|
||
|
||
| Has art | Via |
|
||
|---|---|
|
||
| Human male/female (400, 401) | vendored `Animations`, `anim.mul` |
|
||
| Elf male/female (605, 606) | vendored `Animations`, `anim5.mul` |
|
||
| **Gargoyle male/female (666, 667)** | **§4.3's UOP reader, `AnimationFrame3.uop`** |
|
||
|
||
The catalogue must still **not** treat a missing player body as an error. It is now an unlikely
|
||
answer rather than the expected one, but a client that ships fewer bodies than this one is an
|
||
ordinary thing to meet, and a status screen that flags failures on every import teaches an operator
|
||
to ignore it. `shard_spawn_creatures.art` staying NULL remains a first-class state everywhere it is
|
||
consumed, which it already is.
|
||
|
||
---
|
||
|
||
## 6. The manifest, and what the two buttons actually do
|
||
|
||
Two stages, which is where **Import** and **Update** come from.
|
||
|
||
**Stage 1 — the source gate.** The shard reports a manifest of the client files themselves:
|
||
size, mtime and content hash of `Cliloc.enu`, `anim*.idx`/`anim*.mul`, `art.mul`/`artidx.mul`,
|
||
`Body.def`, `Bodyconv.def`, `Hues.mul`. Unchanged since the last import, and nothing else happens.
|
||
This is the same hash gate the spawn atlas and the cliloc table already use, and for the same
|
||
reason: the normal case is a restart that changed nothing, and it must cost nothing.
|
||
|
||
`anim.mul` is 195 MB and `art.mul` is 148 MB, so the gate is **(size, mtime) first, content hash
|
||
only when those differ** — a full hash of 343 MB on every status poll would make the admin panel
|
||
feel broken.
|
||
|
||
**Phase 1 found that the rule is not sufficient on its own, because of §3.3's other ceiling.** The
|
||
first time those hashes are needed there is nothing cached to compare against, so "hash only when
|
||
(size, mtime) differ" still means hashing 343 MB — inside a 10 s reply timeout it will not fit,
|
||
and the call would 504 rather than answer. So the shard's hashes are computed **off the request
|
||
path entirely**:
|
||
|
||
- `assets.sources` always answers immediately, with `size` and `mtime` for every file and
|
||
`sha256: null` for any file whose hash is not cached against exactly that (size, mtime).
|
||
- A file with a missing hash starts a background pass on its **own** thread — deliberately not a
|
||
job on §3.2's single-slot worker, which would answer every status poll `bridge.busy` for the
|
||
minutes the pass takes, at exactly the moment an operator is watching the panel.
|
||
- The reply carries `hashing` and `complete`, so the website knows to poll again rather than to
|
||
treat a null hash as a changed file.
|
||
|
||
The gate is unchanged; what changed is that "the normal case must cost nothing" now also means
|
||
"and the abnormal case must not time out".
|
||
|
||
**Stage 2 — the asset manifest.** For the working set (§11), the shard streams
|
||
`[{ key, sha256, bytes }]` — no pixels. The website diffs that against what it holds and requests
|
||
**only the keys whose hash changed**.
|
||
|
||
- **Update** = stage 1, then stage 2, then fetch the diff.
|
||
- **Import** = the same path with the diff skipped and every key fetched.
|
||
- **A key that has vanished** from the manifest is staged for review, never applied silently —
|
||
the same rule, and the same reasoning, as a vanished cliloc source or a disappearing atlas
|
||
facet. An unmounted volume and a deliberate client downgrade look identical from here.
|
||
|
||
Clilocs are the exception and stay a **whole-table replace** whenever the file hash changes:
|
||
the measured cost is 663 ms for 67,496 rows, so per-entry deltas would be complexity bought for
|
||
nothing.
|
||
|
||
---
|
||
|
||
## 7. The parser version applies here too
|
||
|
||
`spawnAtlasSource.js` carries `PARSER_VERSION` (currently 6) and the cliloc source carries its own,
|
||
both counted as drift so that a corrected parse reaches an install whose files never change. The
|
||
asset pipeline inherits the rule and needs it more, not less: a fixed hue application or a
|
||
corrected frame offset changes the bytes we derive from files that are byte-identical.
|
||
|
||
**`EXTRACTOR_VERSION` lives in the plugin**, because the plugin is what derives the bytes, and it
|
||
is folded into stage 1's gate. Bumping it makes every asset drift, which is correct.
|
||
|
||
---
|
||
|
||
## 8. Body ids: the part only the shard can do
|
||
|
||
The atlas knows creatures by **slug**, derived from type names in `Spawns/*.xml`. The client knows
|
||
them by **body id**. Nothing in the ServUO tree declares the mapping as data — today an operator
|
||
bridges it by grepping `Scripts/Mobiles/Normal/<Name>.cs` for `Body =`, which appears variously as
|
||
a decimal, as hex (`0xD1`), as `Utility.RandomList(35, 36)`, and as an `m_IDs[]` table.
|
||
|
||
Inside ServUO the problem does not exist. `BridgeWorld.cs:350` already does exactly the required
|
||
thing for a different feature:
|
||
|
||
```csharp
|
||
var type = ScriptCompiler.FindTypeByName(name, true);
|
||
var creature = Activator.CreateInstance(type) as BaseCreature;
|
||
```
|
||
|
||
Construct, read `creature.Body.BodyID`, `Delete()`. Authoritative, no source parsing, and correct
|
||
for custom creatures a grep would never find.
|
||
|
||
**This pass must run on the Core thread** — it constructs and deletes mobiles, which is world
|
||
mutation — while the decode in §4 must run **off** it. That split is the one genuinely new
|
||
threading shape in this protocol, and it is why slug→body resolution is its own request kind with
|
||
its own (small) batch size rather than a step inside asset extraction.
|
||
|
||
Constructing arbitrary creature types has side effects: constructors pack items, set skills, start
|
||
timers. The mitigations are per-type `try`/`catch`, immediate `Delete()`, small batches, and the
|
||
fact that the whole pass is admin-triggered rather than something that runs at boot.
|
||
|
||
### 8.1 What phase 3 measured
|
||
|
||
**Built and walked 2026-09-10** against a live ServUO with a real world (43,000 mobiles, 210,000
|
||
items) and the real sidecar.
|
||
|
||
**The catalogue is 787, exactly as §4.8 predicted**, and it arrives in one page: the whole scan of
|
||
bodies 1–2047 — index validation, 787 decodes, 787 PNG encodes and 787 SHA-256s — took **734 ms**
|
||
cold. That is well inside the 3 s scan budget, so the wall-clock paging §11 was designed for never
|
||
fired on this client. It stays, because the budget is what keeps a slower host or a larger family
|
||
inside the 10 s reply timeout rather than producing replies that are always thrown away.
|
||
|
||
Every prediction in §4.8 and §5.2 held when the bytes were actually rendered and looked at:
|
||
|
||
| Asked for | Answer | Why it matters |
|
||
|---|---|---|
|
||
| `body/320/a0` (`length 0`) | **absent** | The 357-class bug. The library would have returned the previously-decoded creature |
|
||
| `body/607/a0`, `body/608/a0` (elf ghosts) | **absent** | §4.8's two rows that moved; the library returns the elf female here |
|
||
| `body/666/a0` (gargoyle → `anim5`) | **absent** | The spider trap. Nothing swept, nothing found |
|
||
| `body/400/a0/…` deeper key | **unsupported** | Well-formed under §5, not served until phase 6 |
|
||
| direction distribution across 787 | **783 at index 1, 4 at index 0** | Four player bodies, not six — §4.8 again, from `Race.AllRaces` rather than a table. *Phase 4 made it six of six and 1,016/6 across 1,022 rows: §4.9* |
|
||
|
||
**44 of the 787 hashes are shared by two or three bodies, and that is correct.** It is the exact
|
||
signature the wrong-picture bug produces, so it was chased rather than assumed: the client's own
|
||
`Body.def` says `83 {1}`, `84 {1}`, `138 {7}`, `139 {7}`, `106 {12, 59}`, and the sharing groups
|
||
match those lines rather than being runs of consecutive ids (which is what a reused stream buffer
|
||
produces). The distinguishing check is at the source: `Animations.Translate(ref body, ref hue)`
|
||
rewrites `body` **only when bit 31 of the table entry is set**, unlike the one-argument overload
|
||
which always does — and `BridgeAssetValidator.ResolveAnimation` calls that same two-argument
|
||
overload, matching `GetAnimation(…, preserveHue: false, …)`. Validator and decoder therefore
|
||
resolve the identical record, which is the property the whole §4.5 design rests on.
|
||
|
||
**The Core-thread pass costs about 190 ms per 100 types.** All 455 stock `Scripts/Mobiles/Normal`
|
||
classes were constructed and deleted in five chunks, producing every status the protocol defines
|
||
(`ok` 436, `unknown` 11, `notCreature` 4, `failed` 4). The world's mobile count went from 43,000 at
|
||
boot to 42,924 afterwards and its item count fell too — so `Delete()` is reclaiming the packed
|
||
inventory as well as the mobile, and nothing leaked.
|
||
|
||
The three refusals answer as designed end to end: an unknown `family` is **400**, a stale `catalog`
|
||
on a fetch is **422**, and 101 types in one `assets.bodies` is **400** naming the cap.
|
||
|
||
---
|
||
|
||
## 9. The cliloc decompressor is ours now
|
||
|
||
Every modern client ships `Cliloc.*` in the Mythic compressed container — this machine's
|
||
`Cliloc.enu` is 4,989,921 bytes beginning `E8 79 67 8E`, high byte `0x8E`. ServUO's bundled
|
||
`Ultima.StringList` implements only the plain layout and throws on it (§1.1), which is also why
|
||
the shard's own `VendorSearch.GetItemName` is already inert.
|
||
|
||
**UOFiddler is released under the Beerware licence**, so porting its decompressor into our
|
||
GPL-3.0-or-later tree is clean. It lands in the overlay as ordinary C# — the **only** decoder
|
||
Protocol 8 writes rather than calls (§4) — and from that point:
|
||
|
||
- No operator installs UOFiddler.
|
||
- No operator runs `dotnet build` on a converter.
|
||
- No operator copies a 5 MB file to a server.
|
||
- `website/server/tools/cliloc-export/` is retired, and `UOFIDDLER.md` is deleted rather than
|
||
rewritten. (Phase 2 deleted its Part 1; **phase 3 deleted the page**, a phase earlier than that
|
||
section predicted, because creature art was the only thing left on it.)
|
||
|
||
### 9.1 What phase 2 built, and what the port cost
|
||
|
||
**Built 2026-09-10.** `BridgeCliloc.cs` in the overlay, `GET /cliloc` in the sidecar, and a paging
|
||
walk in module-uo that merges the shard's table under the `custom/` overlays.
|
||
|
||
The port is `MythicDecompress` + `MoveToFront` rewritten against plain arrays: the upstream is
|
||
`Span<T>` / `ArrayPool<T>` / `BinaryPrimitives` code and ServUO targets `net48`, which has none of
|
||
them without a package this tree does not vendor. The algorithm is unchanged, **including the parts
|
||
that read oddly** — the three-region count/cursor/end table and the symbol-table shifts are
|
||
upstream's, deliberately, because a tidier rewrite of somebody else's format decoder is a chance to
|
||
be subtly wrong in a way that produces plausible text.
|
||
|
||
Two bounds checks were added, and they are the only behavioural change: the upstream indexes its
|
||
payload without checking, which is safe for a file the client wrote and is not safe for a file this
|
||
shard was handed. A truncated container now reports `UNREADABLE` with the byte it wanted, instead of
|
||
throwing an `IndexOutOfRangeException` from inside a decoder.
|
||
|
||
**Measured on this machine's stock client:** 4,989,921 bytes read, decompressed and parsed in
|
||
**290 ms**, yielding **67,496** non-blank rows in id order, ~5.4 MB on the wire, ~11 pages.
|
||
|
||
That 67,496 is the acceptance test, and it is worth saying why it is a strong one: it is the number
|
||
[`../website/CLILOCS.md`](../website/CLILOCS.md) already recorded for this same client, measured
|
||
through **UOFiddler's own `Ultima.dll`** by the converter this phase deletes. An independent
|
||
implementation agreeing to the row is not something a subtly-wrong decoder produces. Checked
|
||
alongside it: zero U+FFFD, so the UTF-8 survived; 696 rows with non-ASCII text, spot-checked as
|
||
correct curly quotes; the longest row is a 12,149-character EULA, which is also why the record
|
||
length is read **unsigned** (a signed read turns anything over 32 KB negative).
|
||
|
||
**The shard drops blanks before they reach the wire.** ~56,000 of the 123,490 entries are empty
|
||
strings the client reserves, the website discards them at import anyway, and sending them would
|
||
double the transfer for data thrown away on arrival.
|
||
|
||
**Language is a parameter (`enu` by default) but not a free one.** `Ultima.Files` resolves only the
|
||
names in its own table, which for clilocs is `enu`, `deu`, `custom1` and `custom2`; anything else is
|
||
refused `NOT_FOUND` rather than answered with an empty table. `custom1`/`custom2` are the
|
||
client-side custom cliloc files a shard ships to its players — readable here, and deliberately not
|
||
wired into the website's import, because `custom/` on the site is the supported answer for
|
||
shard-added ids.
|
||
|
||
**`assets.error` gained a `code`.** Phase 1 chose between 403 and 400 by looking for the word
|
||
"disabled" in the operator-facing sentence, which makes prose load-bearing; the codes are
|
||
`DISABLED` (403), `NOT_FOUND` (404), `UNREADABLE` (422), `UNAVAILABLE` (503) and `BAD_REQUEST`
|
||
(400). The substring check survives as a fallback, because an overlay and a sidecar are deployed
|
||
separately and a phase-1 shard must keep its 403.
|
||
|
||
### 9.2 Where the base comes from now, and when it is read
|
||
|
||
**The shard wins whenever uo-link is configured and enabled.** No mode setting: there is no version
|
||
of that question an operator benefits from answering. A file on disk remains the source only where
|
||
there is no shard link, plus a one-off explicit path — the deprecated pipeline, kept for installs
|
||
with no bridge and for development.
|
||
|
||
**Boot no longer imports on the bridge.** The file path could hash 5 MB locally and skip in 14 ms; a
|
||
shard round trip in the boot sequence would be spent answering "no" on every restart but the one
|
||
after a client patch — and patching a client is an operator action, so importing became one:
|
||
Admin → Shard → Import. Whatever table is loaded keeps serving until then.
|
||
|
||
Three things about the walk are worth recording because each is a way a shard can hand back a table
|
||
that *looks* complete:
|
||
|
||
- **Only `cut: "end"` finishes it** (§3.4). A short page can equally be a spent budget.
|
||
- **The cursor must advance**, or the walk stops rather than spinning.
|
||
- **Every page echoes the source's size and mtime.** A client patched mid-import is refused
|
||
outright (`SOURCE_CHANGED`) rather than stitched together — half of what arrived came from a file
|
||
that no longer exists and nothing later can tell which half.
|
||
|
||
**The base is exempt from the vanished-source rule**, and that is an upgrade detail rather than a
|
||
design preference: an install that used the file pipeline carries its base file's label in the
|
||
stored fingerprint, and on the bridge that label is *supposed* to disappear. Counting it as vanished
|
||
would make the first import after the upgrade demand approval for a change the upgrade itself made.
|
||
Overlays keep the rule in full.
|
||
|
||
**module-uo's protocol pin moved 7 → 8 in this phase**, which is the third declaration site §15
|
||
names and the one nothing enforces. Phase 1 moved the sidecar and the overlay together because the
|
||
installer refuses to pair a mismatched bundle; this one had to be moved by hand, in the phase that
|
||
first calls a protocol-8 route — the same trap that left the pin at 5 for two phases of the Event
|
||
System while every REST call was answered `409`.
|
||
|
||
**What survives untouched is the `custom/` overlay mechanism.** Shard-added items carry cliloc ids
|
||
no client table has, and ServUO has no server-side notion of a custom cliloc — that is a real gap
|
||
in the *game*, not an artefact of the manual pipeline, and `CLILOCS.md`'s reasoning for it stands.
|
||
The base table now arrives over the bridge; overlays still come from a directory the site reads.
|
||
Measured on the live shard: 16,434 cliloc ids referenced by the script tree, 37 absent from stock.
|
||
|
||
---
|
||
|
||
## 10. The atlas stops needing a shared filesystem
|
||
|
||
Today `SPAWN_ATLAS.md` requires the **website** to read the ServUO tree — "same host, a bind mount,
|
||
or a shared volume". That is the one place the platform's own rule (only the sidecar bridges the
|
||
shard) is broken, and it is broken by the component that faces the internet.
|
||
|
||
The same transport closes it. `spawnAtlasSource.js` already labels every file it reads with a
|
||
portable key:
|
||
|
||
| Label | Count (stock 57.4) |
|
||
|---|---|
|
||
| `Data/Regions.xml` | 1 |
|
||
| `Data/Locations/*.xml` | 6 |
|
||
| `Spawns/*.xml` | 13, ~10.5 MB |
|
||
| `Config/ChampionSpawns.xml` | 1 |
|
||
| `Data/Decoration/**` | tree |
|
||
|
||
So the shard serves these files over the same batched request/reply path, and
|
||
`spawnAtlasSource.js` gains a second backend behind its existing interface: **filesystem** (today,
|
||
kept for same-host installs and for development) or **sidecar** (new, and the default once
|
||
configured).
|
||
|
||
**The parsers do not move.** `spawnAtlasParse.js` is pure, fs-free and CI-covered without a ServUO
|
||
tree, and every quirk it handles — the two respawn delay units, `:OBJ=` splitting, facet-name
|
||
reconciliation, the XmlSpawner directive stripping — stays exactly where it is. The shard sends
|
||
bytes; the website still decides what they mean. That is the same division as §2, and it keeps the
|
||
sidecar a forwarder here too.
|
||
|
||
`SERVUO_PATH` and the `spawn_atlas_servuo_path` setting remain, and select the filesystem backend.
|
||
|
||
### 10.1 What phase 7 measured: `tree/<label>` → bytes cannot work
|
||
|
||
The sentence above said the shard would serve `tree/<label>` → bytes. Measured against a stock 57.4
|
||
tree, **it cannot**. `Spawns/trammel.xml` is **4.03 MB**; the sidecar discards any inbound line over
|
||
**1 MiB** (`shard.rs` `MAX_INBOUND_LINE_BYTES`); that file as one base64 row is 5.4 MiB. The reply
|
||
would be dropped, the request would time out, the import would re-request it, forever — a failure
|
||
with no error anywhere in it. **Two files on a stock tree are in that state**, and a shard with
|
||
hand-built spawn tables has more.
|
||
|
||
| Label group | Files | Bytes |
|
||
|---|---|---|
|
||
| `Spawns/*.xml` | 13 | 10,445,607 |
|
||
| `Data/Decoration/**/*.cfg` | 120 | 1,278,425 |
|
||
| `Data/Regions.xml` | 1 | 129,008 |
|
||
| `Data/Locations/*.xml` | 6 | 37,549 |
|
||
| `Config/ChampionSpawns.xml` | 1 | 4,838 |
|
||
| **Total** | **141** | **11,895,427** |
|
||
|
||
Three carriages were measured before anything was written:
|
||
|
||
| | Wire | Pages | Undeliverable |
|
||
|---|---|---|---|
|
||
| Raw base64 | 15.1 MB | 31 | **2 files** |
|
||
| Whole-file gzip | 1.21 MB | 3 | none *on this tree* |
|
||
| **512 KiB chunks, each gzipped** | **1.26 MB** | **3** | **none, by construction** |
|
||
|
||
**The chunk is the bound and the compression is the saving**, and which is which is the whole
|
||
decision. Compression is what makes it cheap — the tree gzips 12.5x, so the atlas source arrives in
|
||
three pages instead of thirty-one. But nothing guarantees an operator's files compress at all, so
|
||
the ceiling has to hold when they do not: a 512 KiB chunk that refuses to compress is ~683 KiB of
|
||
base64, still inside the wire cap that §3.3's deliberate factor of two leaves room for. Whole-file
|
||
gzip works on every tree anyone would test and fails on the first one nobody did — the difference
|
||
between the second and third rows is entirely a difference in what is *guaranteed*.
|
||
|
||
Chunking costs 0.2 of the ratio (12.3x against 12.5x) and adds a depth segment to the key:
|
||
|
||
```
|
||
tree/Spawns/trammel.xml the manifest row — size, hash, chunk count
|
||
tree/Spawns/trammel.xml/c0 the first 512 KiB of it, gzipped
|
||
```
|
||
|
||
which is §5's scheme doing the same job it does for `body/400/a0/f0`, and needing no protocol change
|
||
to do it — the second time that has paid for itself (§11.2 was the first).
|
||
|
||
### 10.2 It is a family, not two new commands
|
||
|
||
§14 planned `tree.manifest` and `tree.fetch` with `/tree/*` REST beside them. Phase 5 had since
|
||
built a **family registry** on `assets.fetch` — "the command is the transport and the family is a
|
||
property of the key" — and §5 had already written `tree/Spawns/Trammel.xml` as a key. So phase 7
|
||
registered a `tree` family instead, and generalised `assets.manifest` the same way, which is that
|
||
lesson landing one level up.
|
||
|
||
What that bought: the single slot, the byte budget, the paging envelope, the key-count ceiling, the
|
||
`catalog` mid-import guard and the 425 backoff, none of them written twice — **and `link` needed
|
||
nothing at all**, for the third phase running, because `/assets/manifest?family=` and
|
||
`POST /assets/fetch` forward verbatim. `tree.manifest`, `tree.fetch` and `/tree/*` are **not built**.
|
||
|
||
**It has its own consent, though: `Bridge.TreeEnabled`.** Reusing the command is not the same as
|
||
reusing the switch. `Bridge.AssetsEnabled` is an operator agreeing that the website may read *their
|
||
UO client* — art and animations licensed from EA. This is the operator's *own configuration*, which
|
||
they wrote, and which the public bestiary is built from. One switch could not express both, and the
|
||
thing that would have silently disappeared for an operator who declined the first is their spawn
|
||
atlas. So the consent check moved from the front of `assets.fetch` into the family lookup, and
|
||
`assets.sources` now answers whenever *either* plane is on, reporting `families` filtered to what is
|
||
actually enabled — which is how a tree-only shard's website discovers there is anything to ask for.
|
||
|
||
### 10.3 The three things the walk checks, and the two defects they found
|
||
|
||
Every check is a way this ends in a tree that *looks* imported. XML is forgiving about what it
|
||
skips, so a mis-assembled spawn file parses cleanly and simply has fewer spawns in it.
|
||
|
||
- **Each chunk re-declares its own address** — label, index, offset — and carries the hash of its own
|
||
uncompressed bytes. The reader places chunks by declared index rather than arrival order.
|
||
- **The whole file is hashed after reassembly** against the manifest row, which is also the
|
||
fingerprint the drift gate stores.
|
||
- **The `catalog` must not move mid-walk**, or the import is refused rather than stitched out of two
|
||
trees.
|
||
|
||
Two defects were found, and *which harness found which* is the part worth keeping.
|
||
|
||
**An empty `catalog` is not an absent one.** The shard compared `expected != null`, so a caller
|
||
sending `""` — a serialisation of "I have no fingerprint to assert" — had every fetch refused, with
|
||
a sentence naming no catalog at all ("catalog is now 8159778b"). Found by an offline probe that
|
||
passed one by accident.
|
||
|
||
It was **inline in three places** by the time it was found: the body catalogue (phase 3), statics
|
||
and land (phase 5) and the tree (phase 7). All three now go through one
|
||
`BridgeAssets.CatalogMismatch`, because three copies of a comparison are three chances for the next
|
||
family to get it wrong in a way only a differently-written client would ever reveal. It is
|
||
deliberately *not* the shape `BridgeLeases` uses for its own `expected`: there the value is a world
|
||
property, where an empty string is a legitimate thing to expect and `!= null` is correct.
|
||
|
||
Both defects are the same kind of thing and worth naming as one: **a value that means "I am not
|
||
asserting anything" has two spellings on a wire, and a stock install finds whichever one you did not
|
||
think about.** The plane is now checked against a stock ServUO install as a whole — every family,
|
||
each asked with a real catalog, an absent one and an empty one:
|
||
|
||
```
|
||
cliloc.table walk 67,496 rows, 12 pages
|
||
body manifest / fetch 1,095 rows; ok with a real, absent and empty catalog
|
||
static + land fetch ok with a real, absent and empty catalog
|
||
static/land carry their OWN catalog art 66a112c1 vs body 323f284f — a cross-family id is refused
|
||
tree manifest / fetch 141 files incl. BOTH empty ones; all three ways
|
||
empty files carry a VALID gzip member 2 rows gunzip to 0 bytes
|
||
a STALE catalog is still refused body, static and tree all 422
|
||
```
|
||
|
||
**`GZipStream` writes nothing for zero bytes of input.** The gzip header is emitted lazily on the
|
||
first write, so a stream opened and closed without one yields a zero-length buffer rather than the
|
||
20-byte empty member — which is not a valid gzip stream. **Stock ServUO 57.4 ships two empty
|
||
decoration files**, so this broke every import off an untouched tree. The offline probe reassembled
|
||
all 141 files and reported success, because .NET's own decompressor treats an empty stream as empty
|
||
data and the chunk's declared length (0) and hash (of nothing) both agreed. **Only the live walk,
|
||
through a reader on a different runtime, disagreed.** The overlay now answers a literal empty gzip
|
||
member; teaching the reader to accept an empty payload was rejected as putting a special case on the
|
||
wire, where every future reader would have to know it.
|
||
|
||
### 10.4 One canonical order, because the parse is order-sensitive
|
||
|
||
The atlas parse keeps the **first** item id it sees for a decoration type, and writes `meta.source`
|
||
in iteration order. The two readers agreed on a stock tree and agreed *by coincidence*: the
|
||
filesystem reader walks each decoration directory with `localeCompare` while the shard sorts whole
|
||
relative paths, and those diverge the moment a directory mixes cases.
|
||
|
||
So `buildFromFiles` sorts by label, ordinally, once, whatever order the files arrived in — and
|
||
`PARSER_VERSION` bumps, because that is identical input producing a different answer for a
|
||
handful of types, which is exactly what that number exists to push through the hash gate. It was
|
||
written here as **4 → 5** and the cutover renumbered it to **6**: `main` released a *different* 5
|
||
meanwhile (a spawn point keeping its `UniqueId`, v1.2.2), and an install already storing 5 would
|
||
have read a phase-7 build declaring 5 as current and never re-read. See §17.15. The
|
||
**source fingerprint also moved to raw bytes** at both ends: hashing decoded text hashes a UTF-8
|
||
*re-encoding*, which is identical for valid UTF-8 and different for a file that is not, and one
|
||
Latin-1 character in a creature name would have made the drift gate report a change on every import
|
||
forever with the tree untouched.
|
||
|
||
This was found by the live walk too — the unit parity test used `deepEqual`, which ignores key
|
||
order. It now asserts serialised equality as well.
|
||
|
||
### 10.5 Measured end to end
|
||
|
||
Against a live ServUO shard, the real sidecar and module-uo's own reader:
|
||
|
||
| | |
|
||
|---|---|
|
||
| Manifest | 141 rows, one page, 32 KB, **65-92 ms** |
|
||
| Full read | 158 chunks, 3 pages, **1.33 MB on the wire**, **512 ms** |
|
||
| Verification | all 141 files byte-identical to the tree on disk |
|
||
| Atlas parity | **identical** — 6,455 points, 800 creatures, 387 regions, 558 landmarks, 25 champions, 309 decoration types, built from the bridge and from the disk |
|
||
| Drift re-check | 141 rows, **~70 ms**, no file bytes |
|
||
| Traversal | `tree/../../Scripts/...`, `tree/Config/Bridge.cfg`, `tree/Saves/Accounts/accounts.xml` → all `absent` |
|
||
|
||
---
|
||
|
||
## 11. What is bulk and what is on demand
|
||
|
||
The scope approved is creature art, item art, player models "and everything", against a future
|
||
project. §1.1's measurements make the sizing question concrete:
|
||
|
||
| Kind | Addressable | Bulk? |
|
||
|---|---|---|
|
||
| Item statics | **49,152** addressable, **39,189** with art (§11.1, phase 5) | No — on demand, cached, keyed by `itemId` (+ hue) |
|
||
| Land tiles | **16,384** addressable, **4,244** with art | No — on demand |
|
||
| Creature/player bodies, first frame | **1,095** — 787 legacy (§4.8, *not* the 1,144 the library reports) + 235 UOP (§4.9) + 73 at a later action (§11.2) | **Yes** — this is the catalogue |
|
||
| One body, every action, one direction | **210 frames** (body 400); median 118 KB per body, max 9.6 MB (§11.2) | **Not served** (§11.2) |
|
||
| All bodies, every action, one direction | **174,453 frames, 281.5 MB** — measured, not the ~119,000 estimated here | **Not served** (§11.2) |
|
||
| The same at five directions | ~865,000 frames | Not built (§5.1) |
|
||
| Cliloc table | 123,490 entries → 67,496 rows | **Yes** — whole-table replace |
|
||
| ServUO tree files (§10) | ~21 files, ~10.6 MB | **Yes** |
|
||
|
||
**The working set is one thumbnail per body, plus the atlas's own creatures.** 1,022 sprites at
|
||
roughly a kilobyte each is about a megabyte — trivial to import, trivial to re-hash, and it is the
|
||
set that makes a bestiary, a marketplace listing and a character sheet render.
|
||
|
||
Phase 1 took that count down from 1,144, and the 357 it removed are the important part: those are
|
||
ids with **no art** that the library returns a picture for (§4.8). Importing them would have
|
||
written 357 duplicate creature portraits into the site, each one showing whichever body the walk
|
||
happened to decode before it. The count went down; what the catalogue is worth went up.
|
||
|
||
Phase 4 then took it **up** to 1,022, and the direction of travel is the same argument rather than
|
||
its opposite: the 235 it added are bodies with real art in a file the vendored decoder does not
|
||
open (§4.9), each one validated at every bound and identified by a name that carries its body id.
|
||
Both numbers moved because something was measured rather than reported.
|
||
|
||
Everything deeper is the *same protocol at a deeper key* (§5) — a viewer that wants body 400's full
|
||
walk cycle would ask for `body/400/a2/f0..f9`. **That depth is not implemented** (§11.2): the site
|
||
displays still pictures, so the frames have no consumer, and the design's value here is that a
|
||
future one costs a reader and a store rather than a protocol.
|
||
|
||
Because §5.1 dropped four of the five directions, a **complete** one-direction animation set for
|
||
every body looked like ~119,000 frames rather than ~865,000. Phase 6 measured it at **174,453
|
||
frames and 281.5 MB** (§11.2) and, more to the point, established that nothing on this site would
|
||
ever read them: the site shows still pictures. So neither the deep keys nor the
|
||
bulk-fill-everything switch this paragraph anticipated was built, and the numbers above are what a
|
||
future consumer would be choosing to pay.
|
||
|
||
**Hued variants are on demand, always.** `static/3922/h33` is generated when something on the wire
|
||
actually carries hue 33. The cross product of 49,152 statics and 3,000 hues is not a set anyone
|
||
enumerates.
|
||
|
||
**Everything deeper than the catalogue is the same protocol at a deeper key, and is refused rather
|
||
than served** — §11.2. That sentence was written expecting phase 6 to serve it; what phase 6
|
||
actually found was that no consumer exists, so the depth stays named and unserved.
|
||
|
||
### 11.1 What phase 5 measured, and the two traps it found
|
||
|
||
The sizing above was an estimate taken off `art.mul`'s length. Measured through the reader itself,
|
||
against this machine's stock client:
|
||
|
||
| | Measured |
|
||
|---|---|
|
||
| Static ids addressable | **49,152** |
|
||
| ...with real art | **39,189** |
|
||
| ...empty index slots (§4.5's shape) | **9,963** |
|
||
| Land tiles addressable / with art | 16,384 / **4,244** |
|
||
| Whole static + land set, as PNG | 43,433 files, **81 MB**; mean 1.9 KB, max 30.9 KB (id 18213) |
|
||
| Time to decode and encode all of it | **34 s** |
|
||
| Hue slots in `hues.mul` | 3,000 (2,062 named) |
|
||
| Item ids flagged `PartialHue` | **13,259** of 65,536 |
|
||
|
||
Two of those need saying out loud.
|
||
|
||
**49,152, not the 81,884 entries `artidx.mul` declares.** `FileIndex` sizes its index table from the
|
||
**length argument it is constructed with** (`0x10000`), not from the idx file, so the addressable
|
||
static range is `0x10000 - 0x4000`. A ceiling read off the file instead would invent 16,348 ids and
|
||
answer every one of them out of an array nobody bounded. (The first probe of this phase made exactly
|
||
that mistake and reported 65,500 — PowerShell returns `$null` for an out-of-range array index rather
|
||
than throwing, so the over-run counted silently as "empty slots".)
|
||
|
||
**81 MB is small enough to reopen the bulk question, and the answer is still no.** Not on size — on
|
||
what the transfer buys. Base64 puts it at 108 MB through a 512 KB single-slot channel, roughly 210
|
||
round trips, to store 43,433 pictures of which a live shard displays a few hundred. On-demand stays
|
||
right, and phase 6 did not build the bulk-fill switch this sentence promised — see §11.2.
|
||
|
||
#### The library's cache poisons a hued sprite
|
||
|
||
`Art.GetStatic` and `Art.GetLand` memoise into a static `Bitmap[0xFFFF]` and hand back **the same
|
||
instance** on every call; `Hue.ApplyTo` repaints a bitmap **in place**. So the obvious implementation
|
||
— ask the library, apply the hue, encode — edits the library's own copy. Measured before the fix:
|
||
hue item 3922 once, and every later request for the **plain** 3922 comes back hued, with a second hue
|
||
stacking on the first.
|
||
|
||
This is §4.5's failure mode exactly — a confident, plausible, correctly-sized wrong picture that
|
||
every success count agrees with — reached through a door §4.5 never looked at, because phase 0 was
|
||
auditing *records* and this is the library's *cache*. Nothing downstream can see it: the key is
|
||
right, the dimensions are right, the hash is stable.
|
||
|
||
The fix is `Files.CacheData = false` for the life of the process, set once when the asset plane
|
||
initialises, and it pays twice: the same array is never trimmed, so decoding this client's 39,189
|
||
statics would otherwise leave **74 MB of `Bitmap` in a static field of a game server** to serve
|
||
pictures nobody asks for twice. `Animations` does not consult the flag at all, so the body catalogue
|
||
is untouched, and each reader keeps its own cache of *encoded PNG bytes* instead — a tenth of the
|
||
size, already hashed, released when it goes idle.
|
||
|
||
The obvious alternative, copying each bitmap before hueing, was rejected for the retention alone —
|
||
but also because `new Bitmap(src)` **throws** on the `Format16bppArgb1555` these decoders produce.
|
||
The copy has to name the source pixel format explicitly, which is a subtlety on the wrong side of a
|
||
correctness boundary. The invariant is instead re-checked before every hue: a hue is refused if the
|
||
cache is somehow on, because an invariant nothing verifies is a comment.
|
||
|
||
#### `PartialHue` decides the picture, and only the shard can read it
|
||
|
||
A hue is not a tint. It is a 32-entry colour ramp out of `hues.mul` indexed by each pixel's own red
|
||
channel — and whether it replaces **every** pixel or only the grey ones is a per-item-id flag in
|
||
`tiledata.mul`. On this client **13,259 of 65,536 item ids carry it**.
|
||
|
||
Item 597 is a wooden screen with painted flowers. Hued 33 the right way the flowers turn red; the
|
||
wrong way the whole screen turns red. Both decode, both are 44×112, both report success. This is why
|
||
§5 put hue in the key rather than leaving it to the website: shipping `Hues.mul` semantics and a
|
||
65,536-row flag table into Node, to answer a question the shard can answer for free, is the trade
|
||
§2.1 already refused.
|
||
|
||
Two consequences fall out of it. **Land takes no hue segment** — the mode is an *item* flag and land
|
||
has no equivalent, so `land/3/h33` is refused rather than guessed; nothing on the wire carries one
|
||
today, and if something ever does it will arrive with a reason to choose. And **`h0` is not a key**:
|
||
hue 0 on the wire means "not hued", so the plain key already names that picture, and accepting both
|
||
would store one PNG twice under two names and diff them separately forever.
|
||
|
||
#### The namespace trap that compiled
|
||
|
||
The first cut of the reader wrote `TileData.ItemTable` and `TileFlag.PartialHue` unqualified. ServUO
|
||
declares its **own** `Server.TileData`, `Server.ItemData` and `Server.TileFlag` — with a
|
||
`PartialHue` member — in `Server/TileData.cs`, and the reader lives in `Server.Custom.Bridge`, where
|
||
the enclosing namespace beats `using Ultima;`. It compiled. At runtime it read a file resolved
|
||
through `Core.DataDirectories` rather than through `Ultima.Files`, which is §4.6's rule broken in a
|
||
new place: deciding a picture with a file other than the one the pixels came out of. The live rig
|
||
caught it as a `TypeInitializationException` refusing every hued key, from a class the code never
|
||
meant to name.
|
||
|
||
**Gump art is out of scope for Protocol 8, and that is now a safety rule rather than a priority
|
||
call** — §4.1. It is the only decoder that reaches the `hasExtra: true` branch, and that branch
|
||
corrupts the process on the second id. Adding gump art later means fixing that path first,
|
||
deliberately; it is additive under the same key scheme (`gump/<id>`), which is the point of §5.
|
||
|
||
### 11.2 What phase 6 measured, and why the deep keys are not built
|
||
|
||
Everything above about full animations was an estimate — ~119,000 frames, ~117 MB, from a
|
||
151-frame average over six bodies. Phase 6 decoded and PNG-encoded **every action of every body at
|
||
the catalogue's direction**, which is the set a complete one-direction animation store would hold:
|
||
|
||
| | Estimated (§11) | Measured |
|
||
|---|---|---|
|
||
| Bodies with art at *some* action | 1,022 | **1,096** |
|
||
| Action keys (`body/<id>/a<n>`) with frames | — | **26,274** (mean 25.4 per body, max 35) |
|
||
| Frames | ~119,000 | **174,453** |
|
||
| As PNG | ~117 MB | **281.5 MB** (mean 1,692 B, largest single frame 71.5 KB) |
|
||
| Per body | — | median **118 KB** / 210 frames; p90 536 KB; max **9.6 MB** (body 826) |
|
||
| Decode + encode the lot | — | **104 s** |
|
||
|
||
Two facts fall out of that walk that any frame-serving design has to answer.
|
||
|
||
**Frames are not a sequence of pictures; they are a sequence of *placed* pictures.** Every frame
|
||
carries its own centre offset and its own dimensions, and **23,818 of the 26,274 actions have
|
||
frames that differ in size** (centres span x −51..270, y −217..184). A key that returned only a PNG
|
||
would produce an animation that jitters, and nothing downstream could tell — the same shape of
|
||
silent wrongness as every other trap in this document.
|
||
|
||
**At 281.5 MB the transfer is ~375 MB of base64 through a 512 KiB single-slot channel** (§3.2),
|
||
which is upward of 750 round trips for a store of 174,453 files.
|
||
|
||
### The decision: the site shows still pictures, so frames wait for a consumer (org lead, 2026-09-11)
|
||
|
||
§11 justified the deep keys with "the future project", not with anything on this site — and the
|
||
site does not display animation anywhere, nor is it planned to. So phase 6 builds **none** of it:
|
||
no frame keys, no per-body frames manifest, no bulk-fill switch, no store sized for 281 MB, and no
|
||
admin surface for any of that. What would have been the consumer-side machinery is exactly the part
|
||
that would have been guessed at, and when a consumer does exist it can say what shape it wants.
|
||
|
||
`body/<id>/a<n>/f<k>` stays defined in §5 and refused in practice. Reviving it costs the frames
|
||
manifest, the alignment fields and a store — not a protocol change, which is the whole point of
|
||
§5's key scheme.
|
||
|
||
### What phase 6 built instead: the 73 bodies nobody could see
|
||
|
||
The measurement did turn up something the still-picture site wants. **73 bodies have no art at
|
||
action 0 and real art at a later one** — 66 reached through the UOP packages, 7 through the legacy
|
||
files. Body 820's first drawn action is 23, and it is a horse. Until this phase they were absent
|
||
from the catalogue and rendered as text on the bestiary.
|
||
|
||
So the catalogue now falls back to **the first action that has art**, and the key names it
|
||
(§5). The count goes **1,022 → 1,095**, a full cold scan of all 2,047 bodies goes 1,409 ms → 2,090 ms,
|
||
and `EXTRACTOR_VERSION` goes **2 → 3** — unchanged input, a different answer, which is what that
|
||
number is for.
|
||
|
||
Two honesties about those 73. The seven legacy ones and roughly a dozen of the UOP ones are
|
||
recognisable creatures — a horse, a spider, a dragon, a phoenix, a turkey. **Most of the rest are
|
||
thin flat sprites** with entries at actions 1 and 3 only, all anchored at centre x = 63: object or
|
||
equipment art rather than creatures. They are correct by construction (a UOP entry is addressed by
|
||
the hash of a name carrying body *and* action, and the payload re-declares the body), they cost
|
||
about 200 bytes of row each, and nothing surfaces them unless a spawn file names a creature class
|
||
that resolves to one of those ids. And the fallback is the one walk in this protocol that moves
|
||
along the action axis, which is why §4.10's ceiling had to ship with it rather than after it.
|
||
|
||
---
|
||
|
||
## 12. Where it lands on the website
|
||
|
||
Images are written by module-uo into the upload directory. `ctx.uploads`
|
||
(`{ upload, UPLOAD_DIR, MIME_EXT }`) is **already** exposed to modules and
|
||
[`MODULE_API.md:626`](../website/MODULE_API.md) already names its consumer as "atlas art import",
|
||
so no `MODULE_API_VERSION` bump is needed to store them.
|
||
|
||
- `shard_spawn_creatures.art` stops being NULL-by-default and starts being filled by the import.
|
||
- A new asset table carries `key`, `sha256`, `bytes`, `width`, `height`, `imported_at` — the
|
||
manifest side of §6, and what makes an Update a diff rather than a re-download.
|
||
- **The operator-supplied `spawnAtlas.art.json` map stays supported** and continues to win over an
|
||
imported asset. An operator who has drawn their own creature portraits must not have them
|
||
overwritten by a sprite rip on the next Update.
|
||
- **Item and land pictures land in `uploads/items/`, beside the creature portraits and not among
|
||
them** (phase 5). Same content-addressed naming (`uo-static-3922-h33-<sha8>.png`), same "the API
|
||
returns a filename, the client builds the URL" contract, and the same rule that a missing picture
|
||
is a first-class state rather than an error. Separate directories because they have different
|
||
lifetimes: the catalogue is imported as a set and re-imported as a set, while these arrive one at
|
||
a time because something asked for them.
|
||
|
||
Licensing is unchanged and the reasoning is unchanged: these are the operator's own client files,
|
||
extracted on their own host, for their own shard. Nothing is committed, nothing ships in a repo,
|
||
and nothing is redistributed. What changes is only that the extraction stopped requiring a GUI on a
|
||
desktop.
|
||
|
||
### 12.1 The one thing above that phase 3 had to build differently
|
||
|
||
"`shard_spawn_creatures.art` … starts being filled by the import" is right about the outcome and
|
||
wrong about the mechanism, and the difference is not cosmetic.
|
||
|
||
**That table is emptied and refilled by every atlas refresh** — `shardAtlas.db.js`'s `replaceAtlas`
|
||
DELETEs all six atlas tables inside one transaction — and a refresh runs on **every boot**. Before
|
||
protocol 8 that cost nothing, because `art` came from a file on disk and was simply re-read each
|
||
time. An imported sprite is the opposite: expensive to obtain, and gated on client-file hashes that
|
||
would say "unchanged" for weeks afterwards. Writing it onto the creature row would mean an ordinary
|
||
re-parse of the ServUO tree silently deleting every portrait, with the next Update reporting
|
||
*nothing to do* and never restoring them. Nothing would report a fault; the pictures would just be
|
||
gone.
|
||
|
||
So phase 3 built three tables outside that blast radius, and the atlas import reads them on the way
|
||
past (org lead, 2026-09-10):
|
||
|
||
| Table | Holds | Lifetime |
|
||
|---|---|---|
|
||
| `shard_assets` | `asset_key`, `sha256`, `bytes`, `width`, `height`, `body`, `direction`, `file` | Upserted per key; only an approved vanish deletes |
|
||
| `shard_creature_bodies` | `slug`, `type_name`, `body`, `status` | Replaced whole — it is derived from the atlas's creature list, so a slug that has left has no meaning |
|
||
| `shard_asset_meta` | The singleton an Update compares against | Replaced |
|
||
|
||
`replaceAtlas` now takes `{ ...derived, ...operatorMap }`, which is where "the operator's map wins"
|
||
is actually enforced — one spread, in one place, applied on every rebuild rather than only at
|
||
import time.
|
||
|
||
**Two details worth not rediscovering.** The derivation joins on the catalogue **key**, not on
|
||
`a.body = b.body`: today one body has exactly one asset and the simpler join is correct, and it
|
||
stops being correct the day a deeper key (`body/400/a2/f0`) is stored, at which point one slug
|
||
matches dozens of rows and whichever the engine returned last becomes the portrait. And the stored
|
||
filename is **content-addressed** (`uo-body-34-a0-<sha8>.png`), because a stable name overwritten
|
||
in place leaves every browser and CDN serving last month's client's sprite from cache with the
|
||
database row perfectly correct.
|
||
|
||
Phase 6 changed that join, and the way it changed is the point. It read
|
||
`a.asset_key = CONCAT('body/', b.body, '/a0')`, which stopped being right the moment a body could
|
||
be catalogued at another action — it would have dropped exactly the 73 creatures §11.2 added, a
|
||
horse among them, silently. It now reads the row's own action:
|
||
|
||
```sql
|
||
JOIN shard_assets a ON a.body = b.body AND a.family = 'body'
|
||
AND a.asset_key = CONCAT('body/', b.body, '/a', COALESCE(a.action, 0))
|
||
```
|
||
|
||
`COALESCE` because a row written before the column existed has NULL there, and a NULL inside
|
||
`CONCAT` makes the whole comparison NULL — which would have taken every portrait off the site on
|
||
upgrade, with the database perfectly correct and nothing to see in a log. The `action` is stored
|
||
rather than parsed back out of the key because this join needs it in SQL, and re-deriving it there
|
||
would put a second, weaker parser of §5's key scheme in the schema.
|
||
|
||
`shard_creature_bodies` also answers §8 without a schema change on the atlas side:
|
||
`shard_spawn_creatures.name` already holds the ServUO **class name** — the atlas build picks the
|
||
winning spelling of the spawn type token rather than inventing a display label — so the import has
|
||
something `ScriptCompiler.FindTypeByName` will resolve without storing it twice.
|
||
|
||
### 12.2 What phase 8 built, and the two deletions the panel exposed
|
||
|
||
The admin surface is one page, **Admin → Client Files**, covering all three things that come off the
|
||
operator's UO client: the creature catalogue, item and land art, and the cliloc table. One page
|
||
rather than three because they are one job — they live in the same client install, cross the same
|
||
bridge, and all change at the same moment, when the operator patches that client (org lead,
|
||
2026-09-14). The cliloc pair had had no UI at all since phase 2, which meant that on an install
|
||
where boot deliberately never calls the shard, the only way to import 67,496 names was `curl`.
|
||
|
||
Four decisions were put before any code:
|
||
|
||
| | Decided |
|
||
|---|---|
|
||
| Scope | **One page**, three sections, one nav row |
|
||
| The "activity view" (§14) | **The last import's own summary**, not a filtered feed. Rejected: a `prefix` filter on core's activity list plus a `ctx.activity.list` read helper — it would have made a `module-uo`-only phase a three-repo one with a `MODULE_API` bump, to re-read rows the panel can keep itself |
|
||
| A vanished key | **Held in the page, not a table** — see below |
|
||
| Import vs Update | **Two named buttons**, §6's own vocabulary, rather than one button and a `force` checkbox |
|
||
|
||
**The review is in-session by construction, and that is why it needs no table.** The atlas persists
|
||
its equivalent (`shard_atlas_pending`) because *boot* re-parses the tree and would otherwise
|
||
re-prompt on every restart forever. An asset import only ever happens because somebody pressed a
|
||
button on this page, so the review is already in front of the person who caused it; declining is
|
||
not a decision to remember, it is not pressing the other button. What the panel adds is the
|
||
**pictures**: `body/820/a23` names nothing a human recognises, and "is it right that these
|
||
disappear?" cannot be answered from a list of keys. So `needsReview` now returns
|
||
`[{ key, file }]` and the page renders each one.
|
||
|
||
`shard_asset_meta` gained a `last` block — what the import did, and who pressed it. It is written
|
||
**after** the rows commit, because the interesting half (how many atlas creatures resolved) does not
|
||
exist until another shard round trip has happened, and widening the rows-and-meta transaction to
|
||
cover a network call is how an import holds a write lock for the length of a timeout. It is
|
||
cosmetic, so a failure to write it is logged and swallowed rather than failing an import that has
|
||
already applied.
|
||
|
||
#### The two defects, and why only this phase could find them
|
||
|
||
Both are **deletions**, both predate phase 8, and both were invisible until a screen displayed the
|
||
numbers side by side on a stock rig.
|
||
|
||
**One: every item picture reported as vanished.** The import diffs what the site holds against the
|
||
manifest, and a manifest is of one family by construction (§14 — one reply, one catalogue id). The
|
||
held side read the **whole table**. That was correct when phase 3 wrote it and stopped being correct
|
||
in phase 5, which put `static` and `land` rows in the same table: the body manifest never mentions
|
||
item art, so on the walk's rig **313 item pictures were staged for deletion** with a sentence saying
|
||
the shard had stopped offering them. The gate hid it for a while — an Update whose client files are
|
||
unchanged never reads a manifest — so the state that exposes it is the ordinary one: import, let the
|
||
warm pass run for a day, then patch the client.
|
||
|
||
**Two: an approved vanish deleted the picture and kept the row.** `removeSprite` unlinked the file;
|
||
nothing deleted the row. So the catalogue still counted a portrait that was gone, the atlas
|
||
derivation could still point a creature page at a missing file, and the next forced import staged
|
||
the same key for review **again** — reporting "nothing was changed" about a file it had already
|
||
deleted. The removals now go into `saveAssets`'s own transaction, beside the rows and the meta.
|
||
|
||
The third finding is smaller and the same shape: `countAssets()` was whole-table too, so the panel's
|
||
first render of a working install announced a **1,408-row creature catalogue** (1,095 portraits plus
|
||
313 item pictures) on the one screen an operator checks to see whether the import worked.
|
||
|
||
**The rule worth keeping: after phase 5, `shard_assets` is three collections in one table, and every
|
||
read of it that feeds a decision has to say which.** Two of the three that did not were deletions.
|
||
|
||
---
|
||
|
||
## 13. Visibility
|
||
|
||
New surfaces over shard data are admin-toggleable with an operator-set audience, and this is no
|
||
exception. Asset serving is gated like every other shard read: a `requireFeature` gate, an
|
||
audience, and 404-not-403 when the feature is off, so a disabled feature does not advertise itself.
|
||
|
||
The default is the least surprising one: assets are as public as the page that uses them. A
|
||
bestiary that is already anonymous does not become staff-only because its pictures arrived over a
|
||
new pipe.
|
||
|
||
---
|
||
|
||
## 14. Routes and commands added
|
||
|
||
**Loopback (shard ↔ sidecar), all request/reply:**
|
||
|
||
| Command | Reply | Purpose | Built |
|
||
|---|---|---|---|
|
||
| `assets.sources` | `assets.sources.ok` | Stage 1: client file manifest + `EXTRACTOR_VERSION` | **phase 1** |
|
||
| `assets.manifest` | `assets.manifest.ok` | Stage 2: `[{key, sha256, bytes, width, height, body, direction, source}]`, paged | **phase 3** (`source` phase 4) |
|
||
| `assets.fetch` | `assets.fetch.ok` | Content for an explicit key list, paged; base64 PNG per row | **phase 3** (`source` phase 4; `static`/`land` families phase 5) |
|
||
| `assets.bodies` | `assets.bodies.ok` | Slug → body id (§8, Core thread) | **phase 3** |
|
||
| `cliloc.table` | `cliloc.table.ok` | The decompressed table, paged (`?lang=`, `?cursor=`) | **phase 2** |
|
||
| ~~`tree.manifest` / `tree.fetch`~~ | — | §10's ServUO tree files — **not built**; they are a `tree` family on the two commands above (§10.2) | phase 7 |
|
||
|
||
Every one of them is refused outright when `Bridge.AssetsEnabled` is off, and every one of them
|
||
requires a `reqId` — a request without one is refused rather than answered, because an uncorrelated
|
||
reply is by definition an event, and §3.1 is the reason none of this may ever be one.
|
||
|
||
**Sidecar REST** mirrors those one for one under `/assets/*` and `/cliloc`, carrying
|
||
`X-UOLink-Version: 8` and forwarding verbatim. `GET /assets/sources` exists as of phase 1 and
|
||
`GET /cliloc` as of phase 2; their responder maps `bridge.busy` to **425** (flow control, and the
|
||
ordinary answer mid-import rather than a rare one) and reads the refusal's `code` for the rest —
|
||
**403** disabled, **404** no such file, **422** a file the shard cannot decode, **503** a shard that
|
||
cannot do this now (§9.1).
|
||
|
||
Phase 3 added the other three: `GET /assets/manifest?family=&cursor=`, `POST /assets/fetch` and
|
||
`POST /assets/bodies`. **The two POSTs are reads**, and the method is the request body rather than a
|
||
side effect — a few hundred asset keys do not belong in a query string. They are the only reads on
|
||
this link that take one. `422` gains a second meaning on this plane alongside "cannot decode": the
|
||
mid-import guard, a `catalog` that no longer describes the files on disk.
|
||
|
||
**Phase 5 added two families, one field, and no command.** `assets.fetch` grew `static` and `land`
|
||
(§5, §11.1) and `assets.sources` grew **`families`** — which key families this overlay serves. Both
|
||
are additive, so **the protocol stays 8**, and `EXTRACTOR_VERSION` stays **2**: no existing key's
|
||
bytes change, and a new key is not a re-derivation of an old one.
|
||
|
||
The command itself became shared plumbing. Phase 3 gave `assets.fetch` to the body catalogue
|
||
outright, which was right with one family and wrong with three: the command is the *transport* and
|
||
the family is a property of the key. So the correlation id, the operator's consent, the key-count
|
||
ceiling and the family decision now happen once, and a reader only ever sees keys it owns.
|
||
|
||
**The family is derived from the keys and is not a request field.** §5 made the key the address of an
|
||
asset; a request that also named its family would have two places to be wrong and one of them
|
||
silent. A batch must be of **one** family — mixing them is refused (400) rather than split — because
|
||
the reply carries a single `catalog` id, and two families have two fingerprints. A reply claiming one
|
||
of them would be lying about the other.
|
||
|
||
`families` matters more than it looks. Without it, a website talking to a phase-3 or phase-4 overlay
|
||
discovers the gap as a refusal *per key, per pass, forever*, with no picture ever appearing and a
|
||
warning in the log every few minutes. With it, that is one reported state carrying a sentence naming
|
||
the fix.
|
||
|
||
**Phase 7 added a family, a switch and no command** (§10.2). The ServUO tree is served as the
|
||
`tree` key family on `assets.fetch`, and `assets.manifest` became family-dispatched the way
|
||
`assets.fetch` did in phase 5 — so the tree lists its files through the same envelope the body
|
||
catalogue lists its bodies through. Additive, so **the protocol stays 8**, and `EXTRACTOR_VERSION`
|
||
stays **3**: this family derives nothing, it forwards an operator's own file unchanged, so the
|
||
number that tracks *our derivation* has nothing to say about it. `link` needed nothing for the
|
||
third phase running.
|
||
|
||
Two things about it are not shared with the asset families. It answers to **`Bridge.TreeEnabled`**
|
||
rather than `Bridge.AssetsEnabled`, because a UO client and a shard's own configuration are
|
||
different consents; the gate therefore moved out of the front of `assets.fetch` and into the family
|
||
lookup, and `assets.sources` now answers whenever either plane is enabled, with `families` filtered
|
||
to what is actually on. And its rows carry `gzip` rather than `png`, chunked — see §10.1 for the
|
||
measurement that forced that and §10.3 for the two defects the checks around it caught.
|
||
|
||
**Phase 6 added one field and no command.** Manifest and fetch rows carry `action` — which action
|
||
of the body the thumbnail came from (§11.2). Additive, so **the protocol stays 8**; a consumer that
|
||
ignores it sees the catalogue it always saw plus 73 rows, and one that reads it can build the right
|
||
URL for a body catalogued at `a23`. `EXTRACTOR_VERSION` goes **2 → 3**, which is the change every
|
||
consumer does see. A fetch for a key naming an action the catalogue did not choose is answered
|
||
`unsupported` with the chosen action alongside it — never by decoding the asked-for action, which
|
||
is §4.10's wrong picture reached politely.
|
||
|
||
**Phase 4 added one field and no command.** `source` on a manifest or fetch row is `legacy` or
|
||
`uop` — which reader produced the bytes (§4.9). It is additive, so **the protocol stays 8**: a
|
||
consumer that does not read it is unaffected, and one that does can say which half of the extractor
|
||
to doubt when a picture is wrong. `EXTRACTOR_VERSION` went to **2** in the same phase, which is the
|
||
change every consumer *does* see, exactly as §7 intends.
|
||
|
||
**§16 listed phase 3 as `servuo-plugins, module-uo` and that was wrong** — `web.rs` routes every
|
||
command explicitly and has no generic `/assets/*` forwarder, so `link` is in the phase too. The
|
||
table now says so.
|
||
|
||
**Website admin** — `Admin → Client Files`, admin-only, **built in phase 8** (§12.2). Status,
|
||
**Update**, **Re-import everything**, approve/decline for a vanished key, the item-art warm pass and
|
||
the cliloc path setting, in three sections on one page. Every action goes to the admin activity log
|
||
as `shard.assets.*` / `shard.clilocs.*`, and the panel additionally keeps the **last import's own
|
||
summary** beside the catalogue, because that log is one unfiltered list of every admin action on the
|
||
site and an import from three client patches ago is not findable in it.
|
||
|
||
Phase 3 shipped the two routes that make it reachable — `GET /admin/shard/assets` and
|
||
`POST /admin/shard/assets/import`, mirroring the cliloc pair — so that phase could be accepted on a
|
||
real rig. The route surface did not grow in phase 8; what changed is that `needsReview` returns each
|
||
vanished key **with the picture the site currently serves for it**, and the status carries `linked`
|
||
and the `last` block.
|
||
|
||
---
|
||
|
||
## 15. Cross-repo obligations
|
||
|
||
`PROTOCOL_VERSION` goes **7 → 8** in `link/sidecar/src/main.rs`, and in the **same PR**
|
||
`servuo-plugins/overlay.toml` — the installer refuses to pair a sidecar and an overlay that
|
||
disagree, so a split bump means the next bundle silently fails to compose.
|
||
|
||
| Repo | Work |
|
||
|---|---|
|
||
| `servuo-plugins/` | Extraction over ServUO's own `Ultima` (§4), the cliloc decompressor (§9), body resolution (§8), the request handlers, `overlay.toml` |
|
||
| `link/` | Six command families forwarded, the REST surface, **the inbound line cap (§3.3)**, `PROTOCOL_VERSION`. **Nothing in phases 4 or 5** — `assets_call` forwards a request body verbatim and `respond_assets` returns the reply verbatim, so a new key family and a new reply field both pass through untouched |
|
||
| `module-uo/` | Client calls, asset store, the atlas source backend (§10), cliloc ingest, admin surface. Phase 7 also bumped `PARSER_VERSION`, written as 4 → 5 and renumbered to **6** at the cutover (§10.4, §17.15) |
|
||
| `website/` | Almost none — `ctx.uploads` already suffices (§12). Phase 2 deleted `server/tools/cliloc-export/`, the converter this protocol retires |
|
||
| `docs/` | This file; rewrite `CLILOCS.md` §Converting and `SPAWN_ATLAS.md` §Artwork + §Configuring; **delete `UOFIDDLER.md`**; add the libgdiplus prerequisite to `SHARD_PREREQS.md` (§4.4). Phase 9a added the rest: `INTEGRATION.md` to protocol 8, an operator entry in `UPGRADE_NOTES.md`, the index rows, and the prerequisite + `doctor` row in `installer/INSTALL.md` |
|
||
| `installer/` | A `doctor` check for libgdiplus on Linux hosts (§4.4) — **built in phase 9a**, Linux-only (a Windows host has nothing to check, so there is no row rather than a row saying "not applicable") and a `⚠` rather than a `✗`, because names and the spawn atlas have no pixels in them. Bundle pairing already enforces §15 |
|
||
| `android-app/` | Consumes images by URL; no parity gate expected until a screen shows one |
|
||
| `integration-kit/` | A chapter note only — this is UO-specific and teaches nothing about the module contract. **Built in phase 9a** as chapter 3 §2b: the *pattern* (the game host already has the files; route them over the channel you already have; request/reply, one at a time, two stages, version your derivation, never on boot) with no contract re-specified |
|
||
|
||
---
|
||
|
||
## 16. Phases
|
||
|
||
| # | Scope | Repos |
|
||
|---|---|---|
|
||
| 0 | **DONE 2026-09-10.** Spike: the vendored decoders driven from inside a running ServUO, over a client broken 21 ways. §4 stands; the finding was 22,102 wrong pictures on a *stock* client, and the validator that answers them (§4.5-§4.7) | servuo-plugins |
|
||
| 1 | **DONE 2026-09-10.** The transport: `assets.sources`, the single-slot gate (§3.2), the paging envelope (§3.4), the sidecar line cap (§3.3), `EXTRACTOR_VERSION`, `NO_IMAGING`, protocol 7→8. Plus §4.5's validator promoted into the overlay and extended to animations — which found **357 wrong pictures in the body catalogue on a stock client** and cut it from 1,144 to 787 (§4.8) | servuo-plugins, link |
|
||
| 2 | **DONE 2026-09-10.** Clilocs end to end (§9.1, §9.2): the Mythic decompressor ported into the overlay, `cliloc.table` + `GET /cliloc`, the paging walk and the source switch on the website, module-uo's protocol pin 7→8. `cliloc-export/` deleted and `UOFIDDLER.md` §Part 1 with it. **67,496 rows, 290 ms, ~11 pages** — the same count UOFiddler's own DLL produced from this client | all |
|
||
| 3 | **DONE 2026-09-10.** Body resolution (§8) + the **787**-body catalogue (§4.8), `assets.manifest` / `assets.fetch` / `assets.bodies` and their REST mirrors, `shard_spawn_creatures.art` filled and rendered (§8.1, §12.1). **787 rows in one 734 ms page; 455 types resolved at ~190 ms per 100 on the Core thread; zero mobiles leaked.** `UOFIDDLER.md` deleted, two phases early | servuo-plugins, **link**, module-uo |
|
||
| 4 | **DONE 2026-09-11.** The UOP animation decoder (§4.3, §4.9): `BridgeUop` + a PNG encoder that never touches `System.Drawing`, wired in beneath the legacy reader. **Two of the eight player bodies turned out to exist** (gargoyles 666/667); the other six are in no client file, and ghost ids left the player-body set (§5.2, §17.9). The same fallback added **233 other bodies**: the catalogue is **1,022 rows, 1,409 ms cold**, and all six player bodies have art for the first time. `EXTRACTOR_VERSION` 1 → 2 | servuo-plugins |
|
||
| 5 | **DONE 2026-09-11.** Item statics and land on demand (§11.1): the `static` and `land` families, hue applied on the shard from `tiledata.mul`, the byte-bounded art cache, `assets.fetch` made family-aware, `families` on `assets.sources`. Website side: the warm pass, per-row `catalog` staleness, and pictures on the marketplace and the character sheet. **39,189 statics and 4,244 land tiles served; the only refusals are the 9,963 + 12,140 empty index slots §4.5 predicted.** Two traps found — the library's bitmap cache poisons a hued sprite, and `PartialHue` decides the picture from a file only the shard has. Protocol stays 8; `EXTRACTOR_VERSION` stays 2 | servuo-plugins, module-uo |
|
||
| 6 | **DONE 2026-09-11, and not what this row said.** The measurement came first and changed the phase: a complete one-direction animation set is **174,453 frames / 281.5 MB**, not the ~119,000 estimated, and **the site displays still pictures** — so the deep keys and the bulk-fill switch were **not built** (§11.2, org lead 2026-09-11). What shipped is what the still-picture site was missing: the **73 bodies with no art at action 0 and real art deeper** (a horse at `body/820/a23`), the catalogue key carrying its action, the atlas join that reads it, and §4.10's per-body **action ceiling** — without which the fallback walk itself would serve **452 validated pictures of the next body**. Catalogue **1,022 → 1,095**; `EXTRACTOR_VERSION` 2 → 3; protocol stays 8 | servuo-plugins, module-uo |
|
||
| 7 | **DONE 2026-09-14.** The atlas over the sidecar (§10); shared-filesystem requirement retired. The measurement came first again and changed the shape: `tree/<label>` → bytes **cannot work** — `Spawns/trammel.xml` is 4.03 MB against a 1 MiB line cap — so a file crosses as **512 KiB chunks, each gzipped**, which is §5's depth scheme paying for itself a second time (§10.1). It is a `tree` **family** on `assets.fetch` rather than §14's separate commands, with `assets.manifest` generalised to match and its **own consent, `Bridge.TreeEnabled`** (§10.2) — so `link` needed nothing for the third phase running. **141 files / 11.9 MB / 158 chunks / 3 pages / 1.33 MB on the wire / 512 ms**, and the atlas built over the bridge is identical to the one built off the disk. Two defects, each found by a different harness: an empty `catalog` refusing every fetch, and `GZipStream` emitting **nothing** for the two empty files stock ServUO ships (§10.3). `PARSER_VERSION` bumped for one canonical read order — written as 4 → 5, renumbered to **6** at the cutover (§10.4, §17.15); protocol stays 8; `EXTRACTOR_VERSION` stays 3 | servuo-plugins, module-uo |
|
||
| 8 | **DONE 2026-09-14.** The admin surface (§12.2): `Admin → Client Files` — one page over all three planes, because they come off one client and change on one event. The cliloc pair had had no UI since phase 2, which on a bridge install meant `curl` was the only way to import 67,496 names. §14's "activity view" is the **last import's own summary** rather than a filtered feed, which kept the phase to one repo (org lead, 2026-09-14). The walk imported **1,095 portraits in 3.5 s, warmed 313 item pictures in 0.6 s and reloaded 67,496 cliloc rows in 1.7 s** against a real shard — and found **two deletions nobody could see before a screen put the numbers together**: the body import diffing its manifest against *every* family's rows, which staged all 313 item pictures for deletion, and an approved vanish that unlinked the sprite and kept the row, so the key came back for review forever. Both fixed here; `EXTRACTOR_VERSION` and the protocol are untouched | module-uo |
|
||
| 9a | **DONE 2026-09-14.** The docs pass and the acceptance walk (§17.13). `INTEGRATION.md` carried to protocol 8 — it had stopped at **6**, contradicted itself, and documented none of this plane; the operator-facing upgrade note; the index rows; the installer's `libgdiplus` `doctor` check and the kit's chapter note, both promised by §15 and built by no phase. The walk wiped every asset row and every imported sprite and went through it as a new operator: **1,095 portraits in 3.18 s, 67,496 names in 1.42 s, 313 item pictures in 1.38 s, the atlas over the bridge in ~2.0 s, and an Update with no drift answered in 0.99 s**. Two defects (§17.14), the larger of them **not ours and released** — Events phase 12b's `unique_id` has been failing every spawn-atlas import on every upgraded install since v1.2.0, which only a walk on a rig whose tables predate it could see | docs, module-uo, installer, integration-kit, servuo-plugins |
|
||
| 9b | **DONE 2026-09-14.** The `edge → main` cutover, and it is **five repos, not four** — `installer`'s `doctor` check was on `edge` too, and `SHARD_PREREQS.md` on `main` had been describing it as if it shipped (§17.15). Order: the `module-uo` sync first, then `link` + `servuo-plugins` **together** because `bundle.yml` Gate 1 refuses a protocol-mismatched pair, then `website`, `module-uo`, `installer`. The sync found the one real conflict in the whole cutover: both branches had bumped `PARSER_VERSION` **4 → 5** for different reasons and main's 5 was **released**, so keeping it would have made phase 7's re-read unreachable on every existing install — the merged value is **6** (§10.4). Core's entire share of eight phases is a **deletion**: `MODULE_API` never moved, because `ctx.uploads` was already exposed. Published: sidecar **v2.3.0**, overlay **v1.3.0**, module **v1.3.0**, installer **v0.2.0**, bundle **2026.09.15 (protocol 8)**. `edge` stays standing in every repo | link, servuo-plugins, module-uo, website, installer |
|
||
| 9c | **DONE 2026-09-15.** runicgateway.com, and the last phase of the workstream. `checkFacts` was red in **ten** places the moment 9b landed (the protocol in all three reads, the bundle triple, three releases) and `checkReference` in **nine nobody had planned for** — the `Bridge.cfg` keys phases 5, 6 and 7 added, which that check calls *lying by omission* (§17.17). The site gained a page for **Admin → Client files**, the third way into the bridge, `libgdiplus` on the requirements table, and the failure mode a bestiary of text now is. The atlas and marketplace screenshots were **retaken against a real shard** because both now carry artwork, and the new admin screen with them. One defect found on the way: the site's own screenshot rig pinned **protocol 4** | runicgateway.com |
|
||
|
||
Phase 0 exists because §4 chose to call code that can take the shard down if it is wrong, and the
|
||
honest way to hold that choice is to try to break it on purpose — in the real host process, against
|
||
a client that has been patched — *before* building eight phases on top of it. The probes behind
|
||
§1.1 were run from PowerShell against a stock client; neither of those is the environment this will
|
||
actually run in.
|
||
|
||
**Phase 6's deep keys are not deferred to a later phase; they are out of the plan** until something
|
||
wants them. §11.2 has the measurement a future consumer would be choosing to pay and the two facts
|
||
it would have to answer (per-frame centres, and 750 round trips through a single-slot channel).
|
||
Nothing about reviving them needs a protocol change — that is what §5's key scheme bought.
|
||
|
||
Phase 4 sat **after** the catalogue rather than inside it on purpose. The catalogue was useful with
|
||
783 of its 787 bodies, the UOP reader is the one piece of genuinely new format work in this
|
||
protocol, and putting it on the critical path would have held up every website-side phase behind
|
||
it. Its acceptance test was not "it decodes" — it was that a gargoyle looks like a gargoyle,
|
||
checked by eye, because §4.3's whole point is that this failure mode produces confident, wrong
|
||
pictures. It does, and it was.
|
||
|
||
---
|
||
|
||
## 17. Decisions
|
||
|
||
Every item here is settled. Each is recorded because it changes numbers or obligations elsewhere
|
||
in the document.
|
||
|
||
1. **§4: settled 2026-09-10 — call ServUO's vendored `Ultima`,** with one exception added the same
|
||
day: **§4.3's UOP animation decoder** (built in phase 4; §17.9 widened its scope from the
|
||
player bodies to every body the legacy path cannot reach). The crash is confined to the `hasExtra: true` branch that only
|
||
`Gumps` reaches, and nothing here calls `Gumps`. **Phase 0 confirmed both halves in the real
|
||
host process** — nothing faulted on a path we call, and one `GetGump(2)` killed the shard
|
||
outright (§4.7).
|
||
2. **§4.4: settled 2026-09-10 — libgdiplus is a stated requirement** on Linux shard hosts, with
|
||
all three answers taken rather than one: it goes in `SHARD_PREREQS.md`, the installer's `doctor`
|
||
checks it, and its absence degrades to a named `NO_IMAGING` status instead of an error. Windows
|
||
hosts are unaffected.
|
||
3. **§5.1/§5.2: direction — settled 2026-09-10.** Player character bodies use index 0, everything
|
||
else index 1, direction is not in the key, and the player-body set is enumerated from
|
||
`Race.AllRaces` rather than hardcoded. Recorded here because it changes every count in §11.
|
||
Phase 0 reproduced the twelve stock ids and the six that do not decode, exactly.
|
||
4. **§13: the default audience — settled 2026-09-10. An asset inherits the audience of the page
|
||
that uses it.** A bestiary that is already anonymous keeps anonymous pictures; a staff-only
|
||
screen's art is staff-only. The operator can still set the policy explicitly, and the
|
||
`requireFeature` gate with its 404-not-403 behaviour is unchanged. The alternative — one flat
|
||
audience for all asset serving — was rejected because it necessarily disagrees with some page
|
||
that uses it, in one direction or the other, and the disagreement is silent either way.
|
||
5. **§4.5: the response to a malformed record — settled 2026-09-10, and now measured. Validate
|
||
before calling.** The extractor judges an index entry, and for statics the record behind it,
|
||
before handing the id to `Ultima`; a record that fails is reported absent and never decoded.
|
||
Chosen ahead of phase 0 over two alternatives — extracting in a child process (much stronger
|
||
containment, a much larger change to §2 and phase 1) and reversing §4 to write our own decoders
|
||
(weeks, per §2.1). Phase 0 then found the shape that settles it: the dangerous failure is not a
|
||
crash a child process would contain, it is a **wrong picture** that no containment strategy
|
||
would have caught, on **22,102 ids of a stock client**. See §4.5 for the checks and the
|
||
false-refusal measurement that says the boundary is in the right place.
|
||
6. **§3.2/§3.3/§3.4: the transport's three numbers and one shape — settled 2026-09-10, phase 1.**
|
||
A **512 KiB** batch budget under a **1 MiB** inbound line cap, with the factor of two load-bearing
|
||
(a page always admits its first item, so it may overshoot by one). Flow control is enforced
|
||
**on the shard**, as a single slot answering `bridge.busy`, rather than serialised in the
|
||
sidecar or left to the website as a convention — the bound belongs where the memory is. And one
|
||
**paging envelope** (`more` / `cursor` / `cut`) is defined now, with `assets.sources` as its
|
||
first user, rather than left for whichever family pages first to invent.
|
||
7. **§9: the cliloc pipeline's four shapes — settled 2026-09-10, phase 2.** All four were put to
|
||
the org lead before any of it was written, and two of them departed from the recommendation:
|
||
|
||
- **The wire carries rows, not bytes.** `[{n, f, t}]` inside the paging envelope, rather than a
|
||
reconstituted plain-binary file in base64. The container decompresses into *records*; the
|
||
plain layout is a file shape that would have had to be synthesised on the shard and re-parsed
|
||
on the website, at a 33% base64 premium, to reach the same place.
|
||
- **The shard omits blank entries** — ~56,000 of 123,490 — because the website discards them at
|
||
import anyway. The transfer halves and nothing observable changes.
|
||
- **The bridge always wins; the file upload is deprecated** (org lead, departing from the
|
||
proposed `auto`/`bridge`/`file` setting). There is no version of "which source?" an operator
|
||
benefits from answering, so there is no setting to answer it with. A file remains the source
|
||
only where there is no shard link, plus a one-off explicit path.
|
||
- **Import is admin-triggered on the bridge** (org lead, departing from the proposed boot
|
||
refresh). Boot does not call the shard at all — see §9.2 for why that is the right trade and
|
||
what it costs.
|
||
|
||
8. **§4.8: the catalogue is 787 bodies, not 1,144 — measured 2026-09-10, phase 1.** Recorded here
|
||
because it changes §11's sizing, phase 3's scope and phase 4's, and because of *how* it was
|
||
found: the animation path has §4.5's shared-buffer defect too, and 357 ids with no art were
|
||
returning the previously-decoded body's bitmap. The elf ghosts moved from "decodes" to "no art"
|
||
in §5.2 for the same reason, taking phase 4's set from six player bodies to eight.
|
||
|
||
10. **§11.1: phase 5's four, settled 2026-09-11.** Put to the org lead after the client was measured
|
||
and before the reader was written, because the first measurement changed what the risk was:
|
||
|
||
- **Ingest warms; the route only serves.** A page renders the pictures already on disk and
|
||
leaves out the ones that are not; fetching happens behind it, on a timer, from the keys the
|
||
site's own rows name. Chosen over fetching on first request, on one number: the asset plane
|
||
serves **one request at a time** (§3.2), so a URL that fetched would let any anonymous visitor
|
||
walk 49,152 ids × 3,000 hues through that single slot and park an operator's own import behind
|
||
it. Warming from the site's own data has no such surface — the ceiling is the number of
|
||
distinct (item, hue) pairs the shard has already told the site about.
|
||
- **Staleness is a per-row catalogue id, not a manifest.** A client patch changes the shard's
|
||
`catalog` fingerprint and a restart does not, so "is this out of date?" is a column
|
||
comparison. The alternative — a `static` manifest family enumerating 39,189 rows with hashes —
|
||
would cost a 34-second scan of the whole art file per Update to answer a question about maybe
|
||
three hundred pictures, and would re-fetch art nobody looks at any more. Lazy costs nothing
|
||
for the ones nobody wants.
|
||
- **The pictures appear on the marketplace and the character sheet**, the two places the data
|
||
already existed and rendered as `id 1234, hue 33`. That gives the phase an acceptance test
|
||
checkable by eye, which §11.1 says is the only kind that catches this failure mode.
|
||
- **`Files.CacheData` goes off process-wide** at asset init, rather than copying each bitmap
|
||
before hueing. See §11.1: the copy does not solve the 74 MB retention, and the ordinary copy
|
||
constructor throws on ARGB1555 anyway.
|
||
|
||
9. **§4.3/§5.2: phase 4's four, settled 2026-09-11.** Put to the org lead after the packages were
|
||
opened and before the reader was written, because the first measurement changed what the phase
|
||
was worth:
|
||
|
||
- **The UOP fallback applies to every body, not only to player bodies.** It is the same rule
|
||
either way — "legacy first, UOP when it has nothing" — and restricting it would have needed an
|
||
extra filter to *refuse* art the client has. 787 → 1,022 rather than 787 → 789.
|
||
- **Ghost ids leave the player-body set** (org lead, over keeping them with a new status or
|
||
leaving them as `absent`). No client has art for any of the six, so listing them advertised
|
||
keys that cannot exist. The shard is still asked rather than told — the question is now the
|
||
living pair per race.
|
||
- **The UOP path gets its own PNG encoder** rather than `Bitmap.Save`, which is what §4.4's
|
||
sentence about this reader and `System.Drawing` was always promising. Phase 3's `ToPng` is
|
||
untouched: its input comes from a decoder that needs GDI+ to produce a pixel, so encoding it
|
||
without GDI+ buys nothing.
|
||
- **A host without libgdiplus keeps the flat `NO_IMAGING` refusal** (org lead, over serving the
|
||
~238 bodies that no longer need it). One answer beats a quarter-full catalogue nobody can tell
|
||
from a complete one. The door §4.4 wanted open stays open; phase 4 simply does not walk
|
||
through it.
|
||
|
||
11. **§11.2: phase 6 stopped being the phase it was planned as — settled 2026-09-11.** Put to the
|
||
org lead with the measurement in hand, and the answer removed most of the phase:
|
||
|
||
- **The deep animation keys and the bulk-fill switch are not built.** The org lead's own scope
|
||
note — "I hadn't planned to use the animations on the site, just static images" — is the
|
||
whole argument: §11 justified them with a future project rather than with anything on this
|
||
site, and 281.5 MB of store, a warm pass and an admin surface for a consumer that does not
|
||
exist yet would all have been guesses. `body/<id>/a<n>/f<k>` stays named in §5 and refused in
|
||
practice; reviving it is a reader and a store, not a protocol change.
|
||
- **The catalogue falls back to the first action that has art, and the key names that action.**
|
||
73 bodies on a stock client have nothing at action 0 and real art deeper — body 820's action
|
||
23 is a horse — and they rendered as text on the bestiary. Keeping the `a0` spelling for
|
||
them was rejected for the reason §5 gives: a key that lies about its content is this
|
||
protocol's recurring failure, not a shortcut.
|
||
- **Only that one exception**, over a per-body pose override that would let an operator pick a
|
||
better-looking action. One rule with one mechanical exception; the setting can come the day
|
||
somebody wants a specific creature to look different.
|
||
- **§4.10's action ceiling ships in the same phase**, because the fallback is the walk that
|
||
would otherwise hit it: one action past a body's band is 643 validated pictures and 452
|
||
byte-identical copies of the next body.
|
||
|
||
12. **§10: the atlas's three shapes — settled 2026-09-14, phase 7.** Put to the org lead after the
|
||
tree was measured and before a line was written, because the first measurement said the phase as
|
||
written could not work:
|
||
|
||
- **A file crosses as 512 KiB chunks, each gzipped.** Not raw base64 (15.1 MB, 31 pages, and two
|
||
files on a *stock* tree that can never arrive under a 1 MiB line cap), and not whole-file gzip
|
||
— which is smaller on this tree and bounded by nothing, so it works everywhere anyone would
|
||
test and fails on the first tree nobody did. The chunk is the guarantee; the compression is
|
||
only the saving. See §10.1.
|
||
- **It is a `tree` family on `assets.fetch`, with its own switch.** §14's separate `tree.*`
|
||
commands and `/tree/*` REST are not built: phase 5's family registry already owns the single
|
||
slot, the paging envelope, the key ceiling and the mid-import guard, and reusing it left
|
||
`link` with nothing to do for the third phase running. But the *consent* is its own —
|
||
`Bridge.TreeEnabled` — because an operator declining to serve their EA-licensed client is not
|
||
the same as declining to serve the spawn files they wrote, and the atlas would have been the
|
||
casualty. See §10.2.
|
||
- **Boot never calls the shard; importing is an admin action** (departing from the
|
||
recommendation). The same answer §17.7 gave the cliloc table, and the same reasoning, with one
|
||
thing accepted in exchange: an install whose atlas comes over the bridge has *no* automatic
|
||
refresh at all, so an edited spawn file stays invisible until someone presses Import. The
|
||
alternative — one manifest round trip per boot, ~70 ms and no file bytes — was on the table and
|
||
was declined in favour of consistency. The panel and the CLI say so in as many words, and the
|
||
skip is logged rather than silent.
|
||
|
||
13. **Phase 9's shape and its two inherited obligations — settled 2026-09-14.** Put to the org lead
|
||
at the start of the phase, because §16 described phase 9 as a docs pass in one repo and three
|
||
things it did not cover had fallen between the phases:
|
||
|
||
- **Phase 9 is three legs, not one** — `9a` the docs pass plus the acceptance walk on the real
|
||
rig, `9b` the `edge → main` cutover across the four code repos, `9c` the runicgateway.com
|
||
facts. The cutover was in no phase row at all, and the site cannot move until `link`'s
|
||
`main` carries protocol 8 — `runicgateway.com`'s `checkFacts.mjs` reads it from there and goes
|
||
red the moment it changes. The Events and Engagement workstreams split their last phase the
|
||
same way, for the same reason.
|
||
- **The two §15 obligations nobody built are built here**, rather than dropped: the installer's
|
||
`doctor` check for `libgdiplus` and the integration kit's chapter note. The doctor check was
|
||
the third of the three answers §17.2 took, and `SHARD_PREREQS.md` had been *claiming* it
|
||
existed since phase 1.
|
||
- **`INTEGRATION.md` is repaired the whole way to 8, protocol 7 included.** It had stopped at 6
|
||
and contradicted itself — prose saying "the current version is 6" above examples already
|
||
carrying `X-UOLink-Version: 8` — with no protocol-7 paragraph and not one mention of
|
||
`assets.*`. Protocol 7's absence is the Events workstream's debt rather than this one's, but
|
||
it cannot be stepped over on the way to 8, and the file is the only thing an integrator
|
||
outside this org has.
|
||
|
||
14. **The two defects the phase-9 walk found, and where they ship — settled 2026-09-14.** Both were
|
||
put to the org lead with the walk's evidence:
|
||
|
||
- **A busy shard gets its own sentence, and the panel re-reads once.** The status call exhausts
|
||
its 425 backoff whenever something else holds the single slot — an import, or the item-art
|
||
warm pass refilling itself — and phase 8's panel rendered that as *"The shard is not answering
|
||
for client files"*, the same banner as a shard that is down or switched off, and left it
|
||
standing because the page never re-reads. On the rig that banner was up for a quarter of an
|
||
hour over a perfectly healthy shard. `BUSY` now says what it is, and one automatic re-read
|
||
four seconds later clears the ordinary case; the panel still does not poll.
|
||
- **`shard_spawn_points.unique_id` ships on `edge` with this phase** rather than as a hotfix to
|
||
`main` (org lead, weighing that it is already released). Events phase 12b added the column to
|
||
the `CREATE TABLE` and nowhere else, so it reached fresh installs and no existing one —
|
||
`CREATE TABLE IF NOT EXISTS` does not add a column, which is what the twenty-odd
|
||
`ADD COLUMN IF NOT EXISTS` lines in that same file exist to do. Every spawn-atlas import on an
|
||
upgraded install has failed outright since **v1.2.0** with `Unknown column 'unique_id'`: no
|
||
bestiary refresh, no spawn map, no champion altars. The cost of the decision is that operators
|
||
on a released v1.2.x keep that until the Asset Bridge cuts over.
|
||
|
||
15. **The cutover's four questions — settled 2026-09-14.** Put to the org lead before anything
|
||
merged, because the survey of `edge` against `main` turned up two things §16's row did not
|
||
anticipate and two the row could not decide.
|
||
|
||
- **Five repos, not four.** `installer`'s `edge` carried phase 9a's `libgdiplus` `doctor`
|
||
check, and `SHARD_PREREQS.md` on `main` had been describing that check as existing since
|
||
phase 1. A four-repo cutover leaves an operator-facing doc on `main` naming a check that is in
|
||
no released binary, so the installer goes over with the rest and publishes a release — which
|
||
is the only way the check reaches a host.
|
||
- **`PARSER_VERSION` resolves to 6, with both notes kept.** This is the one genuine conflict in
|
||
the cutover and the most instructive thing in the phase. `edge` bumped 4 → 5 for phase 7's
|
||
canonical read order; `main` bumped 4 → 5 for the `UniqueId` fix and **released it as
|
||
v1.2.2**. The two numbers are the same and the two derivations are not. Keeping 5 would have
|
||
left every install that imported under v1.2.2 storing 5, `currentParser` calling a phase-7
|
||
build current, and phase 7's change reaching nobody already running — the exact failure the
|
||
constant exists to prevent, arrived at through a merge rather than through forgetting to bump.
|
||
**Two long-lived branches bumping one counter for different reasons is a defect the counter
|
||
cannot see; only the merge can.** Both notes stay in the version log, 5 as main's released
|
||
meaning and 6 as phase 7's, with the renumber explained in place rather than collapsed.
|
||
- **The re-verify is the bundle, not a second walk** (departing from the Event System's
|
||
cutover, which rebuilt the whole released-artefact rig). 9a had already walked the feature end
|
||
to end on a real shard one day earlier, against the same commits; what a release path can
|
||
break that a dev rig cannot is the sidecar/overlay **pairing**, so that is what gets checked.
|
||
- **`module-uo`'s `ci/core-ref.json` stays where the sync puts it** (`655fbf3f`). `website`'s
|
||
`main` moves in this cutover but `MODULE_API_VERSION` does not, and the pin asserts an API
|
||
version, not a tip. Leaving it alone also keeps the frozen-manifest job — the one thing that
|
||
can answer whether the module's claimed URLs are the URLs it serves — from going red in the
|
||
middle of a five-repo window for a reason unrelated to any of it.
|
||
|
||
The ordering itself was not a question: `link` and `servuo-plugins` must land **together**,
|
||
because `bundle.yml` Gate 1 refuses a sidecar/overlay pair that disagrees on protocol and
|
||
whichever lands alone leaves a compose that cannot run. That is not a prediction — it is what
|
||
happened at the Event System's cutover, compose run 102, and it happened again here: the overlay
|
||
released nine minutes ahead of the sidecar and **compose run 111 refused the pair in exactly
|
||
those terms** — *"sidecar v2.2.0 speaks 7, overlay v1.3.0 declares 8. Refusing to publish a
|
||
bundle that would install a shard whose events the sidecar rejects with 409."* The gate is not
|
||
a formality; it is the only thing standing between a half-landed cutover and an operator
|
||
installing a shard that cannot talk to its own sidecar.
|
||
|
||
16. **What the cutover published, and the one thing that needed a hand.**
|
||
|
||
| Component | Release | Protocol |
|
||
|---|---|---|
|
||
| `link` sidecar | **v2.3.0** | 8 |
|
||
| `servuo-plugins` overlay | **v1.3.0** | 8 |
|
||
| `module-uo` | **v1.3.0** | — |
|
||
| `installer` | **v0.2.0** | — |
|
||
| bundle | **2026.09.15** | 8 |
|
||
|
||
`edge` stays standing in every repo.
|
||
|
||
The re-verify §17.15 chose — check the pairing rather than rebuild the whole rig — is what
|
||
found the one thing that did not happen by itself. **The bundle did not recompose on its own.**
|
||
Two compose runs failed for two unrelated reasons, and only reading both logs separates them:
|
||
|
||
- **Run 111 was Gate 1**, correctly refusing the mismatched pair in the window between the two
|
||
releases. Expected, and the reason the pair lands together.
|
||
- **Run 114 was the runner**, after both releases were out: `Set up job` took **14m33s** and
|
||
then every subsequent step failed at **0s** with **no log blob stored** (its log route 500s).
|
||
That is the same signature as the 2026-09-09 outage and has nothing to do with the bundle's
|
||
content.
|
||
|
||
A failed compose is silent from the outside: the bundle simply stays at its previous value, so
|
||
operators keep installing the **previous protocol** while `main` carries the new one, and
|
||
nothing says so. `bundle.yml`'s nightly cron exists to self-heal exactly this, but that is up to
|
||
a day of operators fetching a stale pair. Recovery is
|
||
`POST /api/v1/repos/RunicGateway/installer/actions/workflows/bundle.yml/dispatches {"ref":"main"}`
|
||
→ 204, and the recomposed bundle published within the minute.
|
||
|
||
**The lesson for any future cutover: publishing the releases is not the last step — verifying
|
||
the bundle actually moved is.** A green release run does not imply a recomposed bundle, and the
|
||
gap between them is invisible to everything except a direct read of `current.json` (through
|
||
`/contents/`, never `/raw/`, which is CDN-cached for six hours).
|
||
|
||
17. **What the site leg found, and the one thing that was still lying — 2026-09-15, phase 9c.**
|
||
The site is the only repo whose checks read the platform over the API rather than describing
|
||
it, so 9c is the one phase whose starting state is a measurement. It was red in two places:
|
||
|
||
- **`checkFacts`, ten failures, all expected.** Protocol 7 → 8 in three independent reads, the
|
||
bundle triple, and the three releases the cutover cut. `moduleApi` and the capability list
|
||
did not move, which is the cutover's own finding restated by a different mechanism: core's
|
||
whole share of eight phases was a deletion.
|
||
- **`checkReference`, nine failures nobody planned for.** `AssetsEnabled`, `AssetBatchBytes`,
|
||
`AssetBodyBatch`, `AssetFetchKeys`, `AssetScanMs`, `AssetPlayerDirection`,
|
||
`AssetCreatureDirection`, `TreeEnabled` and `TreeChunkBytes` reached `Bridge.cfg` across
|
||
phases 5, 6 and 7 and the public reference page never mentioned one of them. **Nothing in
|
||
this plan's own phase rows would ever have caught that** — the rows name the repos a change
|
||
edits, and this was a repo that only *describes* them. The check that catches it is the one
|
||
that compares in both directions; the direction that earned its keep here is the one asking
|
||
what the source has that the page does not.
|
||
|
||
Two corrections the checks could not make, because both are prose:
|
||
|
||
- **§17.15's "five repositories" had already become a page on the site**, which named
|
||
`website` among them and explained why core was reached. For protocol 8 the honest table has
|
||
four rows and core is absent — and the absence is the more interesting half, because what
|
||
the asset plane needed (somewhere to put a picture) core already offered every module.
|
||
- **The site said the bridge has "two ways in".** It has three. The asset plane is neither an
|
||
event nor a point-in-time read: it is a paged bulk read on the reply path, one request in
|
||
flight, and the reason it must not ride the event stream is structural rather than about
|
||
throughput — `app.rs` persists every event and broadcasts it to every client, which is right
|
||
for "a house went IDOC" and wrong for hundreds of megabytes of artwork.
|
||
|
||
**The defect the phase found is a version pinned in a rig.** `seedDemo.mjs` — the script that
|
||
stands up the deployment the screenshots are taken of — pinned **protocol 4**, under a careful
|
||
comment explaining a `module-uo` debt that phases since have paid. A literal written down with
|
||
a good reason outlives the reason, and nothing checked it because the rig is not the product.
|
||
Its default now reads `platform.json`, which `checkFacts` verifies against `link`'s `main`, so
|
||
the number cannot drift again without the build going red.
|
||
|
||
**The screenshots were retaken against the real thing** (D42 holds: no mock-ups). On the demo
|
||
deployment, against the running shard and sidecar: **1,095 portraits in 2.1 s, 67,496 names in
|
||
1.4 s, 313 item pictures in 0.8 s**, all matching the 9a walk on a different database. The
|
||
atlas shot needed its scroll offset moved 36 px, because a row with a portrait in it is taller
|
||
than a row of text — a screenshot's framing is a fact about the product, and it moved.
|