§16 planned deep animation keys and a bulk-fill switch. Measuring first changed the phase: a complete one-direction animation set is 174,453 frames / 281.5 MB (not the ~119,000 / ~117 MB §11 estimated), every frame carries its own centre and 23,818 of 26,274 actions have frames of differing size — and the site displays still pictures. So none of it was built (§11.2, org lead 2026-09-11); `body/<id>/a<n>/f<k>` stays named in §5 and refused in practice. §4.10 is what the walk found instead. A body's actions are a contiguous band and the next body's begin immediately after, so one action past the ceiling is a real record of another creature: 643 of 795 legacy bodies pass every check there and 452 are byte-identical to body+1's action 0. `Animations.GetAnimLength` is not the ceiling — it disagrees with the index arithmetic on one body of this client, by nine actions. What shipped: the 73 bodies with no art at action 0 (a horse at `body/820/a23`), the catalogue key carrying its action, the atlas join that reads it instead of hardcoding `a0`, and the ceiling that makes the fallback walk safe. Catalogue 1,022 -> 1,095; EXTRACTOR_VERSION 2 -> 3; protocol stays 8. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
1620 lines
104 KiB
Markdown
1620 lines
104 KiB
Markdown
# Asset Bridge (Protocol 8) — client assets without UOFiddler
|
||
|
||
**Call it the Asset Bridge.** That is the name for this work everywhere — commits, PR titles,
|
||
branches (`feat/asset-bridge-p<n>`), and conversation. The protocol number is **8** and this file is
|
||
`docs/link/v8.md`.
|
||
|
||
**Status:** Design of record, **phases 0-4 complete** (§16). Approved in principle 2026-09-09
|
||
(architecture, asset scope, built-in cliloc decoder, atlas cleanup); refined 2026-09-10 (one
|
||
direction not five, ServUO's own decoders, the UOP reader, the libgdiplus requirement). The phase 0
|
||
spike ran on 2026-09-10 and its findings are §4.5-§4.7 -- §4's decision stands, and the response to
|
||
a malformed record is now measured rather than proposed. Phase 4 built the one decoder §4.3
|
||
reserved, and §4.9 is what it measured: **two** of the eight player bodies it was scoped to exist
|
||
at all, and the same fallback found **233** other bodies, taking the catalogue to **1,022**.
|
||
**Nothing in §17 is open.**
|
||
**Supersedes the manual half of:** `../website/UOFIDDLER.md` (deleted in phase 3, once creature
|
||
art stopped needing it), [`../website/CLILOCS.md`](../website/CLILOCS.md) §Converting,
|
||
[`../website/SPAWN_ATLAS.md`](../website/SPAWN_ATLAS.md) §Artwork and §Configuring the tree.
|
||
|
||
Two features on this platform read data that only exists inside a UO client, and today both reach
|
||
the site by hand: the operator installs UOFiddler, converts `Cliloc.enu` on their own desktop,
|
||
exports sprites one at a time from a GUI, hand-writes a slug→filename JSON map, and copies the
|
||
result to the server. A third — the spawn atlas — avoids UOFiddler but pays a different price: the
|
||
**website** must be able to read the shard's ServUO tree directly, over a bind mount or a shared
|
||
volume.
|
||
|
||
This protocol deletes all three arrangements. The shard already has everything, and the bridge
|
||
already goes to the website.
|
||
|
||
---
|
||
|
||
## 1. The premise, which turns out to be free
|
||
|
||
**A ServUO shard cannot boot without a UO client installation.** It reads maps, statics, tiledata
|
||
and multis out of `.mul`/`.uop` files, and `Config/DataPath.cfg` is where an operator declares
|
||
where those live — *required* on Linux, auto-detected from the registry on Windows. At runtime the
|
||
resolved directories sit in `Server.Core.DataDirectories`, a public static the plugin can read on
|
||
any shard, with no new configuration and nothing for an operator to set up.
|
||
|
||
So the files the operator has been converting on their desktop are already on the shard host, in a
|
||
directory the shard already knows the path of, in a process the bridge already runs inside.
|
||
|
||
Everything below follows from that.
|
||
|
||
### 1.1 What was measured, not assumed
|
||
|
||
Against this machine's ServUO 57.4 tree (`C:\Users\colby\Desktop\ServUO`) and client
|
||
(`D:\Games\Electronic Arts\Ultima Online Classic`, 3.5 GB), loading **ServUO's own
|
||
`Ultima.dll`** — the assembly `overlay/Scripts/Scripts.csproj:39` already carries a
|
||
`<ProjectReference>` to:
|
||
|
||
| Call | Result |
|
||
|---|---|
|
||
| `Art.GetStatic(0…16383)` | 16,384 decoded, 0 errors |
|
||
| `Art.GetStatic(16384…65535)` | 32,766 decoded, 1 empty, 16,385 clean out-of-range errors |
|
||
| `Art.GetLand(0…16383)` | 16,384 decoded, 0 errors |
|
||
| `Animations.GetAnimation(0…2047, 0, 1)` | **1,144** bodies with a decodable first frame, 904 empty, 0 errors — of which **357 are wrong pictures**; see §4.8 |
|
||
| `Hues.GetHue(33)` | loads |
|
||
| `Bitmap.Save(…, Png)` | 852-byte PNG from one creature frame |
|
||
| `Gumps.GetGump(2)` | **hard crash** — `AccessViolationException`, process exit `0xC0000005` |
|
||
| `new StringList("enu", "Cliloc.enu")` | throws — `Non-negative number required` |
|
||
|
||
Two of those rows are load-bearing and are dealt with in §4 and §9. The rest say the same thing:
|
||
**most of the extraction this protocol needs is already implemented, already compiled, and already
|
||
referenced by the plugin's own build.**
|
||
|
||
**Every "decoded" count in that table is an upper bound, not a measurement**, and phases 0 and 1
|
||
spent themselves establishing by how much. `Ultima` reports success for records that do not exist
|
||
(§4.5, §4.8), so the honest reading of the animation row is 787 bodies with art, 357 that return
|
||
the previously-decoded body's bitmap, and 904 that return nothing. The static and land rows are
|
||
overstated the same way. This is not a table to size anything from any more — §11 is.
|
||
|
||
Depth, for §11's sizing: body 400 (human male) has **35 actions × 5 directions = 1,050 frames**.
|
||
One body. §5.1 cuts that by exactly 5×.
|
||
|
||
---
|
||
|
||
## 2. Architecture: the shard extracts, the sidecar forwards, the website decides
|
||
|
||
```
|
||
UO client files (operator's own, on the shard host)
|
||
│ read by the plugin, off the Core thread
|
||
▼
|
||
ServUO shard (servuo-plugins/) ← decodes; resolves body ids; hashes
|
||
│ loopback JSON, request/reply, one batch outstanding at a time
|
||
▼
|
||
uo-link sidecar (link/) ← forwards bytes; decides nothing
|
||
│ REST, bearer-token auth, X-UOLink-Version: 8
|
||
▼
|
||
website (module-uo/) ← stores, names, gates, serves
|
||
```
|
||
|
||
This is deliberately the *only* arrangement that keeps
|
||
[the bridge's standing rules](PLAN.md) intact:
|
||
|
||
- **The sidecar stays a dumb forwarder.** It moves opaque assets and decides nothing about them —
|
||
no audience, no projection, no capability advertisement. Putting the decoders in Rust would have
|
||
meant the sidecar deciding what an asset *is*, on top of re-deriving in Rust what is already
|
||
compiled next door in C#.
|
||
- **Access control stays on the website**, which has the auth machinery and the admin forms.
|
||
- **The shard is still never network-reachable.** Nothing here opens a port; the plugin answers
|
||
requests on the connection it already dialled out on.
|
||
|
||
### 2.1 Why not the sidecar, and why not the operator's desktop
|
||
|
||
A Rust extractor in the sidecar would need ports of: the Mythic cliloc decompressor, `FileIndex`
|
||
(including UOP), the ARGB1555 run-length frame decoder, `Body.def`/`Bodyconv.def` translation,
|
||
`Hues.mul`, and a PNG encoder — weeks of work to re-derive what §1.1 shows already runs. It also
|
||
cannot do §8: resolving a creature slug to a body id requires being inside ServUO.
|
||
|
||
Automating on the operator's desktop (shipping the converter with the installer) removes UOFiddler
|
||
but keeps a manual step and still cannot do §8. It was considered and rejected.
|
||
|
||
---
|
||
|
||
## 3. The transport, and the three traps in it
|
||
|
||
### 3.1 Assets go over the request/reply path, never the event path
|
||
|
||
`link/sidecar/src/app.rs:122` persists **every** non-`pong` event into the SQLite store *and*
|
||
broadcasts it to every WebSocket subscriber. An asset stream on that path would grow the sidecar's
|
||
store without bound and fan megabytes out to every connected client, forever.
|
||
|
||
`rpc.rs`'s `try_route` consumes a correlated reply and `continue`s **before** either of those
|
||
happens. So an asset batch is a reply, not an event. This is not a new mechanism — it is the one
|
||
`char.request`, `account.roster` and `vendor.snapshot` already use.
|
||
|
||
### 3.2 One batch outstanding, always
|
||
|
||
`BridgeLink.Emit()` enqueues onto a **bounded drop-oldest** queue (`Bridge.QueueCap`, default
|
||
10,000). It counts **lines, not bytes** — a design that is correct for live events and dangerous
|
||
for bulk transfer, because 10,000 queued 200 KB replies is 2 GB of shard memory.
|
||
|
||
The rule that makes this safe is flow control, not a bigger queue: **the website requests batch
|
||
*n+1* only after batch *n* has arrived.** Queue depth stays at approximately one. A dropped or
|
||
lost reply simply times out and the batch is re-requested, which is safe because reading a client
|
||
file is idempotent and has no world side effects.
|
||
|
||
**Phase 1 made that a rule the shard enforces rather than one the website is trusted to follow.**
|
||
The asset plane has a **single slot**: a request arriving while one is in flight is answered
|
||
`bridge.busy` — which the sidecar already maps to 425 — and runs nothing. The bound belongs on the
|
||
side where the memory actually is; a documented convention would have held right up until the
|
||
first website bug, and its failure mode is the 2 GB above.
|
||
|
||
The alternative considered was serialising in the sidecar, so a second caller waits instead of
|
||
being refused. It was rejected because a waiter spends the website's own 12 s timeout doing
|
||
nothing, and because it leaves the shard itself unguarded against anything that is not that one
|
||
sidecar.
|
||
|
||
What this costs, and it is deliberate: **a status poll shares the slot with a batch**, so polling
|
||
during a long import is answered 425 until the batch lands. That is honest — this plane really
|
||
does do one thing at a time — and it is why the one genuinely long operation on it, hashing 343 MB
|
||
of client files, is explicitly *not* a job on this worker (§6).
|
||
|
||
### 3.3 The size ceilings are already fixed, and one of them is missing
|
||
|
||
| Limit | Value | Where |
|
||
|---|---|---|
|
||
| Sidecar waits for a shard reply | **10 s** | `rpc.rs` `REPLY_TIMEOUT` |
|
||
| Website waits for the sidecar | **12 s** | `module-uo/server/utils/uoLinkClient.js` `TIMEOUT_MS` |
|
||
| Sidecar → shard line | 1 MiB | `BridgeLink.cs:283` |
|
||
| **Shard → sidecar line** | **1 MiB** | `shard.rs` `MAX_INBOUND_LINE_BYTES` — **added in phase 1**; it was unbounded |
|
||
| **Batch budget** | **512 KiB encoded** | `Bridge.AssetBatchBytes` — **added in phase 1** |
|
||
|
||
The first two bound a batch: it must decode, encode, serialise and cross the wire inside ten
|
||
seconds. The last is a gap this protocol must close — an unbounded `read_line` facing a component
|
||
that is now deliberately sending large lines is a memory-exhaustion shape we would be inventing
|
||
ourselves. **Protocol 8 adds an explicit inbound line cap to the sidecar**, set above the largest
|
||
legal batch and rejecting rather than buffering past it.
|
||
|
||
Batches are therefore sized by bytes, not by count, with the emitter cutting a batch short when it
|
||
would exceed the cap. Base64 costs 33%; the budget must be stated in encoded bytes.
|
||
|
||
**The two numbers, settled in phase 1: a 512 KiB batch budget under a 1 MiB line cap.** The cap is
|
||
symmetric with the one the shard has always applied to its own inbound lines, so both directions
|
||
of this link read the same. The factor of two between them is load-bearing rather than cautious: a
|
||
page always admits its **first** item even when that item alone exceeds the budget, because the
|
||
alternative is an oversized item being skipped for the budget on every page forever and its family
|
||
never making progress. The headroom is what makes that overshoot land on the wire instead of being
|
||
rejected by the cap.
|
||
|
||
An over-long line is **discarded and the connection kept**, which is what `BridgeLink.cs` has
|
||
always done in the other direction. Tearing the link down would take the live event feed with it
|
||
over a single malformed frame, and the reply that was lost simply times out and is re-requested —
|
||
safe, because reading a client file is idempotent.
|
||
|
||
### 3.4 One paging envelope, defined once
|
||
|
||
Five of the families in §14 page: clilocs (phase 2), the body catalogue (3), statics and land (5),
|
||
deep animation keys (6) and the ServUO tree files (7). Left to themselves that is five chances to
|
||
invent five slightly different shapes, and the website would have to learn each one — so phase 1
|
||
defines the envelope before the first family needs it, and `assets.sources` is its first user even
|
||
though it has nothing to page.
|
||
|
||
```
|
||
"files": [ … ], // the array, named by the family
|
||
"more": true, // ask again, with this cursor
|
||
"cursor": "s:4104", // opaque to everyone but the shard; absent when more is false
|
||
"cut": "budget" // budget | end | limit — WHY the page stopped
|
||
```
|
||
|
||
`cut` is the field that is easy to leave out and expensive not to have. "This page is short" has
|
||
three different meanings — the source ran out (`end`), the byte budget was spent (`budget`), or
|
||
the family stopped at a limit of its own (`limit`) — and **only the first means the import is
|
||
finished.** A website that had to infer completion from an item count would resume from the wrong
|
||
place the first time a page happened to land exactly on a boundary.
|
||
|
||
The cursor is deliberately **opaque and shard-defined**. The shard is the only side that knows how
|
||
its own walk is ordered, and a cursor the website could parse is a cursor the website would
|
||
eventually construct.
|
||
|
||
The budget is counted in **UTF-8 bytes**, not characters and not items. Characters would undercount
|
||
every non-ASCII byte in a cliloc row, and the ceiling this has to live inside — §3.3's line cap —
|
||
is measured in bytes.
|
||
|
||
---
|
||
|
||
## 4. The decoders are ServUO's own — decided, and the crash is narrower than it looked
|
||
|
||
**We call ServUO's vendored `Ultima` (decided 2026-09-10).** No decoders are reimplemented.
|
||
`overlay/Scripts/Scripts.csproj:39` already references the project, so the art half of this protocol
|
||
costs plumbing rather than pixel code, and only §9's cliloc decompressor is written from scratch.
|
||
|
||
The reason that is safe, rather than merely cheap, is a distinction §1.1 did not draw at first.
|
||
|
||
### 4.1 The crash lives on one code path, and nothing we call uses it
|
||
|
||
`Gumps.GetGump(2)` does not fail — it **corrupts the process**: `AccessViolationException`, exit
|
||
`0xC0000005`. That is a corrupted-state exception, uncatchable by an ordinary `try/catch` on .NET
|
||
Framework 4.8, so in-process on a live shard it is a crash with players on it. That much is
|
||
alarming, and on its own it looked like an argument against using this library at all.
|
||
|
||
It is not, because of how the three decoders construct their `FileIndex`:
|
||
|
||
| Decoder | UOP file | `hasExtra` | Probed |
|
||
|---|---|---|---|
|
||
| `Art` | `artLegacyMUL.uop` | **false** | 49,150 statics + 16,384 land tiles, **0 faults** |
|
||
| `Animations` | *none — legacy `anim*.mul` only* | — | 1,144 bodies, **0 faults** |
|
||
| `Gumps` | `gumpartLegacyMUL.uop` | **true** | **faults on the second id** |
|
||
|
||
`FileIndex.cs`'s own comment says the extra-field handling exists *for* `gumpartlegacy.uop` — it is
|
||
the one UOP layout carrying an extra field, and `hasExtra: true` is the branch written to cope with
|
||
it. **Gumps is the only caller that sets it.** So the fault is not a general fragility in this
|
||
library's `unsafe` code; it is a bug on a branch that exactly one decoder reaches, and that decoder
|
||
is already out of scope (§11).
|
||
|
||
The rule this turns into is a safety rule, not a preference: **nothing in this protocol calls
|
||
`Ultima.Gumps`.** Adding gump art later means fixing or replacing that path first, deliberately,
|
||
not discovering it in production.
|
||
|
||
### 4.2 What the decision accepts
|
||
|
||
Three costs come with it, all known and none of them blocking:
|
||
|
||
1. **Six of the twelve stock player-character bodies have no art on this path** — both human ghosts
|
||
and every gargoyle body (§5.2). `Animations` never reads `AnimationFrame*.uop`. **This one is
|
||
not merely accepted: §4.3 adds a decoder for it**, because it is the player character and the
|
||
scope says player models.
|
||
2. **`System.Drawing` is a hard dependency, in the decode and not just the encode.** `Frame`
|
||
writes ARGB1555 straight through a `LockBits` pointer, so a Linux shard needs **libgdiplus** to
|
||
read a sprite at all. That is a stated prerequisite — §4.4.
|
||
3. **We inherit whatever `Ultima` a given ServUO vendors**, which can change under a shard upgrade.
|
||
`EXTRACTOR_VERSION` (§7) is the mitigation: it already counts as drift, so a shard whose library
|
||
changed re-derives on the next import.
|
||
|
||
The residual risk that remains is a patched or custom client tripping an out-of-bounds read on a
|
||
path we *do* call. §16's phase 0 is where that gets exercised rather than assumed.
|
||
|
||
### 4.3 One decoder we do write: UOP animation
|
||
|
||
**Built in phase 4. What it reaches is not what this section predicted, and §4.9 is the
|
||
measurement.** The section stands as the argument; the numbers below are the corrected ones.
|
||
|
||
The bodies the legacy path cannot resolve get a decoder rather than a caveat. It is deliberately
|
||
the **narrowest possible** addition: a reader for `AnimationFrame*.uop`, used *only* where the
|
||
vendored code has nothing. Everything it can already decode keeps going through it — which after
|
||
phase 4 is still 787 of the catalogue's 1,022 bodies.
|
||
|
||
This client ships `AnimationFrame1/2/3/4/6.uop` (107, 118, 253, 115 and 24 MB) plus
|
||
`AnimationSequence.uop`. ServUO's `FileIndex` already contains a UOP reader — but `Animations`
|
||
never constructs one, and the UOP animation *payload* is its own format rather than a repackaged
|
||
mul record, so wiring the existing `FileIndex` in is not sufficient. Two license-compatible
|
||
references exist to work from: ClassicUO's animation loader (GPL-3, and we are GPL-3-or-later) and
|
||
UOFiddler 4.22 (Beerware, already established in §9).
|
||
|
||
**What it was for, and what it actually delivers.** The phase was scoped to the eight player
|
||
bodies §4.8 left undecodable. **Two of the eight are in the client at all** — gargoyles 666 and
|
||
667, in `AnimationFrame3.uop`. The six ghost bodies are in no package, and §4.9 shows how that was
|
||
established rather than assumed. What the same fallback *does* deliver is **233 further bodies**
|
||
the catalogue had nothing for, so the phase's real effect is 787 → **1,022**.
|
||
|
||
**The trap this must not fall into, and it is why the phase exists at all.** `Bodyconv.def` maps
|
||
gargoyle 666 to `anim5`, and `BodyConverter.Convert` faithfully returns fileType 5 — where this
|
||
client's index runs out before the entry. Asking the *other* anim files for index 666 does not
|
||
fail. It returns 175 decodable action/direction combinations of **a giant spider**, because
|
||
something unrelated occupies that index in `anim2.mul`; fileTypes 3 and 4 return misaligned colour
|
||
fragments. Rendered and confirmed.
|
||
|
||
So the extractor takes `BodyConverter.Convert`'s answer and, if that yields nothing, reports
|
||
nothing. **It must never sweep file types looking for a hit** — that does not find missing art, it
|
||
silently puts a spider on the gargoyle page, and no error is raised anywhere. A "0 rows" outcome is
|
||
correct behaviour; a plausible wrong picture is the failure this protocol most needs to avoid,
|
||
because nothing downstream can detect it.
|
||
|
||
**Searching all five UOP packages for one body is not that, and the difference is structural.** A
|
||
legacy index is addressed **by position**: nothing in the file says which body a record belongs to,
|
||
so a wrong lookup returns a confident picture of something else. A UOP entry is addressed by the
|
||
**hash of a name that contains the body id** — `build/animationlegacyframe/000666/00.bin` — and the
|
||
payload then declares that id again in its own header, which the reader checks against the id it
|
||
asked for. A hit is proof of identity rather than a coincidence of position. Measured: no hash
|
||
appears in two packages.
|
||
|
||
### 4.4 Requirement: a Linux shard host needs libgdiplus
|
||
|
||
**Stated prerequisite, not a soft recommendation.** ServUO targets `net48`, so on Linux it runs under
|
||
Mono, and Mono's `System.Drawing` is a thin layer over **libgdiplus**. §4.2 put `System.Drawing` in
|
||
the decode path, so without that library a Linux shard cannot extract art at all — the cliloc table
|
||
(§9) and the atlas files (§10) are unaffected, since neither touches pixels.
|
||
|
||
**Windows shard hosts need nothing.** `System.Drawing` ships with .NET Framework. This is a
|
||
Linux-only prerequisite and most shards will never read this section.
|
||
|
||
| Host | Get it with |
|
||
|---|---|
|
||
| Debian / Ubuntu | `sudo apt-get install libgdiplus` — in Debian since bullseye (6.0.4) and bookworm/trixie (6.1), and in Ubuntu universe |
|
||
| Fedora / RHEL | `sudo dnf install libgdiplus` (EPEL or the Mono repository) |
|
||
| Docker | `RUN apt-get update && apt-get install -y libgdiplus` in the shard image |
|
||
| Alpine, or a distro with no package | Build from source — see the repository below. This is the awkward case and is worth avoiding by choosing a Debian-based image |
|
||
|
||
Upstream is <https://github.com/mono/libgdiplus>, with the Mono project's own page at
|
||
<https://www.mono-project.com/docs/gui/libgdiplus/>.
|
||
|
||
**One thing to know before depending on it: that repository was archived in March 2025 and is
|
||
read-only.** Distributions still package and patch it, so `apt-get install libgdiplus` is a normal,
|
||
supported thing to do today — but upstream is not maintained. It is the strongest long-term argument
|
||
for eventually moving extraction off `System.Drawing`, and phase 4's UOP reader (§4.3) is written
|
||
without it precisely so that door stays open — decode and encode both (§4.9). It does not change
|
||
the decision now, and §17.9 is the decision not to walk through that door yet: a host with no
|
||
libgdiplus still gets the flat refusal below rather than the ~238 bodies that no longer need it.
|
||
|
||
**How its absence must present.** Never a stack trace and never a 500. Missing libgdiplus is a
|
||
named, actionable outcome in the same family as the cliloc reader's `COMPRESSED`:
|
||
|
||
```
|
||
status: unavailable
|
||
code: NO_IMAGING
|
||
reason: This shard host cannot render images — Mono's System.Drawing needs libgdiplus.
|
||
Install it (apt-get install libgdiplus) and re-run the import. Cliloc and
|
||
atlas import are unaffected.
|
||
```
|
||
|
||
The installer's `doctor` checks for it and reports it alongside its other host checks, so an
|
||
operator learns about this while setting the shard up rather than from an empty bestiary weeks
|
||
later.
|
||
|
||
---
|
||
|
||
### 4.5 What phase 0 measured, and the rule it produced
|
||
|
||
Phase 0 ran §4's decoders from inside a live ServUO 57.4 against a client broken in 21 catalogued
|
||
ways ([`servuo-plugins/tools/scaffolding/README.md`](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/src/branch/main/tools/scaffolding/README.md)
|
||
carries the full results). §4's decision **stands** — nothing faulted on a path this protocol
|
||
calls, and §9's cliloc reader reproduced UOFiddler's 123,490-entry table byte for byte in 218 ms.
|
||
|
||
But the spike was looking for the wrong kind of failure, and found a worse one.
|
||
|
||
**A malformed record does not usually throw. It renders the previous asset.** `LoadStatic` and
|
||
`LoadLand` decode out of `m_StreamBuffer`, which is reused across calls, only ever grown, and
|
||
filled by a `stream.Read` whose return value is discarded. So a record that is short, absent or
|
||
out of bounds produces a real bitmap of whatever was decoded last — reported as success by every
|
||
count in the library and undetectable by anything downstream.
|
||
|
||
On the **stock, unmodified** client on this machine that is **22,102 ids**: 9,962 statics and
|
||
12,140 land tiles whose index entry reads `lookup 0, length 0`. `FileIndex.Seek` rejects
|
||
`lookup < 0` and `length < 0`, and zero is neither, so it treats an empty slot as a hit. A bulk
|
||
import that trusted the library would have written 22,102 duplicate images into the site under
|
||
ids that have no art. §1.1's "32,766 decoded" was counting these.
|
||
|
||
**This is specific to the UOP path** (see §4.6), because `artidx.mul` stores `-1` for an absent
|
||
record where an unmapped UOP slot is a zeroed struct.
|
||
|
||
So the rule, and it is the deliverable phase 1 inherits:
|
||
|
||
> **Validate before calling.** The extractor judges an index entry — and, for statics, the record
|
||
> header and row table behind it — *before* handing the id to `Ultima`. A record that fails is
|
||
> reported as absent, never decoded.
|
||
|
||
The checks, all of which phase 0 implemented and measured as `BridgeAssetValidator`:
|
||
|
||
| Check | The shape it stops |
|
||
|---|---|
|
||
| `lookup >= 0`, `length > 0` | the 22,102 empty slots above |
|
||
| `lookup + length <= <data file>.Length` | **`Seek` checks that a record *starts* inside the file and never that it *ends* inside it**; a short read then decodes the previous asset |
|
||
| the same bound against `verdata.mul` for a patched entry | `Verdata.Seek` is bounds-checked **nowhere at all** |
|
||
| land records are at least 2,024 bytes | `LoadLand` reads exactly that many whatever the length says |
|
||
| declared width and height within a ceiling | `LoadStatic` allocates `new Bitmap(width, height)` from two bytes in the file — phase 0 got a ~128 MB allocation out of an edit, and the same field can ask for 8 GB |
|
||
| walk the row table and every run, bounded | `LoadStatic`'s two guards bound the **write** into the bitmap and nothing bounds the **read** out of the record |
|
||
|
||
Measured against the patched client, this refused all eight record-level defects, seven of which
|
||
the library rendered without raising anything. Measured against the stock client it refused
|
||
**nothing** across 49,151 statics and 16,384 land tiles. That second number is the one that makes
|
||
the boundary defensible: a checker that refuses real art would be worse than no checker.
|
||
|
||
Two more ways an id with no art yields a picture, both of which the extractor must handle itself:
|
||
`Art.GetStatic(id, false)` **throws** `IndexOutOfRangeException` above the index's own ceiling
|
||
(16,385 ids in a full sweep), and `Art.GetStatic(id)` with the default `checkmaxid: true` is worse
|
||
— `GetLegalItemID` maps an out-of-range id to **0** and returns **item 0's picture**. Take the
|
||
ceiling from the index that was opened, and pass `checkmaxid: false` so an overrun is loud.
|
||
|
||
**The animation path has none of this yet**, and phase 0 proved it needs it: the patched client's
|
||
verdata entry for body 34 points past `verdata.mul`'s end and the wolf still "decoded", counted
|
||
among the 1,144 successes while rendering something else. `GetAnimation` additionally allocates
|
||
`new int[frameCount]` straight from a file-supplied int. Extending the validator to animations is
|
||
phase 1 work, not a phase 9 tidy-up.
|
||
|
||
### 4.6 The UOP wins outright, and `art.mul` is never opened
|
||
|
||
`FileIndex`'s UOP constructor ends with a bare `MulPath = uopPath`. **When `artLegacyMUL.uop` is
|
||
present it wins, and `art.mul` / `artidx.mul` are not opened at all.** Every current client ships
|
||
the UOP, so this is the normal case and not an edge one.
|
||
|
||
It matters twice.
|
||
|
||
**For us:** an index entry's `lookup` is an offset into whichever file `FileIndex` resolved, so any
|
||
bound taken against `art.mul` while the index holds UOP offsets is not approximate — it is
|
||
meaningless. Phase 0's first run refused **34,299 perfectly good statics** for "declaring
|
||
10533x2085" on exactly that mistake, and every one of those refusals read like a real finding. The
|
||
extractor must resolve the data file with `FileIndex`'s own precedence, not by name.
|
||
|
||
**For operators:** a shard that adds custom art to `art.mul` while the UOP is still in place gets
|
||
nothing, silently. Worth a line in the admin surface's diagnostics (§14) rather than leaving an
|
||
operator to discover it as "my art did not import".
|
||
|
||
### 4.7 The gump crash, reproduced where it counts
|
||
|
||
§4.1 argued from source that the access violation lives on the `hasExtra: true` branch only `Gumps`
|
||
reaches. Phase 0 called `Ultima.Gumps.GetGump(2)` **once, from inside a running shard**: the ServUO
|
||
process disappeared. No exception line, no `catch` reached, no shutdown, nothing in the console —
|
||
the probe's checkpoint file, written before the call, was the entire record of what happened.
|
||
|
||
`AccessViolationException` is a corrupted-state exception that .NET Framework 4.8 does not deliver
|
||
to ordinary handlers, so on a live shard this is a crash with players on it and **there is no
|
||
in-process defence**. §4.1's "nothing in this protocol calls `Ultima.Gumps`" is therefore an earned
|
||
safety rule rather than a scoping preference, and adding gump art later means fixing that path
|
||
first, deliberately.
|
||
|
||
---
|
||
|
||
### 4.8 What phase 1 measured: the animation path has the same defect, and it is worse
|
||
|
||
§4.5 ended by saying the animation path had no validator and that extending it was phase 1's work
|
||
rather than a phase 9 tidy-up. Phase 1 built it, ran it, and the reason that sentence was right is
|
||
larger than the verdata entry that prompted it.
|
||
|
||
**357 of the 1,144 "decodable" bodies are wrong pictures, on the stock client.** Their index entry
|
||
reads `length 0` — no record at all — and `GetAnimation` returns a real bitmap anyway, for the same
|
||
reason `LoadStatic` does: `m_StreamBuffer` is reused, only ever grown, and filled by a
|
||
`stream.Read` whose return value is discarded.
|
||
|
||
Measured directly, because a count could not tell:
|
||
|
||
| Decode body 320 (`lookup 22638982, length 0`) straight after… | What comes back |
|
||
|---|---|
|
||
| body 12, the dragon | the dragon's bitmap, 176×167, identical hash |
|
||
| body 34, the wolf | the wolf's dimensions, 35×34 |
|
||
| body 400, the human male | the human's bitmap, 27×63, identical hash |
|
||
|
||
That is not a near miss or a misaligned fragment. Body 320 has no art, and it renders **whichever
|
||
creature was decoded before it** — which means the picture a bestiary page got would depend on the
|
||
order the importer happened to walk the catalogue in.
|
||
|
||
So the working set is **787 bodies, not 1,144**:
|
||
|
||
| | Bodies 0–2047, direction 1, first frame |
|
||
|---|---|
|
||
| Real art | **787** |
|
||
| Wrong pictures (empty record, library returned a bitmap) | **357** |
|
||
| Absent, and the library agreed | 903 |
|
||
| `bodyconv` resolves nowhere, nothing swept (§4.3) | 1 |
|
||
| Refused by the record walk | **0** |
|
||
| Threw | 0 |
|
||
|
||
**That last-but-one row is the number that matters as much as the first.** The record-level
|
||
animation checks — palette, frame count, frame table, and every run header walked against both the
|
||
record's own length and the bitmap it locks — refused **nothing** across every real body on a stock
|
||
client. §4.5's rule holds: a checker that refuses real art is worse than no checker, and this one
|
||
does not.
|
||
|
||
#### The player bodies: four of twelve, not six
|
||
|
||
§5.2's table was built from the library's answer alone, and two of the six bodies it listed as
|
||
decoding do not have art:
|
||
|
||
| Body | Index entry | Library | Actually |
|
||
|---|---|---|---|
|
||
| Human male/female (400, 401) | real | 24×64, 24×63 | **art** |
|
||
| Elf male/female (605, 606) | real | 24×64, 24×63 | **art** |
|
||
| **Elf ghosts (607, 608)** | `lookup 27221378, length 0` | 24×63 | **the previous body's picture** |
|
||
| Human ghosts (402, 403) | `lookup -1` | nothing | honestly absent |
|
||
| Gargoyle (666, 667, 694, 695) | `lookup -1` / past the index | nothing | honestly absent |
|
||
|
||
Confirmed the same way: body 607 decoded after the dragon is the dragon, after the wolf is the
|
||
wolf, after the human male is the human male. Its 24×63 was the elf *female*'s dimensions, because
|
||
606 is what the catalogue walk decoded immediately before it.
|
||
|
||
Two consequences, both of which change work elsewhere in this document:
|
||
|
||
- **§4.3's UOP decoder covers eight player bodies, not six.** The elf ghosts join the human ghosts
|
||
and the four gargoyle bodies. Phase 4's scope grows by two ids and its argument does not change.
|
||
— *Phase 4 then found that six of those eight are in no client file at all, and recovered the
|
||
other two (§4.9). The reasoning above is why they were in scope; the answer is there.*
|
||
- **The `lookup -1` / `length 0` distinction is the whole difference between an honest absence and
|
||
a wrong picture**, and it is not visible from outside the index. `artidx.mul` and the legacy
|
||
`anim*.idx` write `-1` for a record that is not there; an empty UOP slot, and evidently a
|
||
deliberately blanked legacy entry, is a zeroed struct. `FileIndex.Seek` rejects the first and
|
||
accepts the second, and the second is 357 creatures and two playable ghosts.
|
||
|
||
#### Why this could not have been found any other way
|
||
|
||
Phase 0 ran this exact sweep and reported "1,144 decoded, 904 empty, 0 faults", and every one of
|
||
those numbers is true. The library raised nothing, returned bitmaps of plausible sizes, and agreed
|
||
with itself. Nothing downstream of the decode — not a count, not an exception, not a hash of the
|
||
output, not a look at one picture in isolation — distinguishes body 320's dragon from body 12's.
|
||
The only things that did were **validating the index entry before the call** and **decoding the
|
||
same id twice after different neighbours**.
|
||
|
||
That is the same method note §4.3 ended on, and this is its second confirmed catch. Any time a UO
|
||
file lookup is addressed by index, a success count is evidence of nothing.
|
||
|
||
---
|
||
|
||
### 4.9 What phase 4 measured: two of eight, and 233 nobody was looking for
|
||
|
||
§4.3 was scoped to eight player-character bodies. Before writing the reader, phase 4 opened the
|
||
five packages and asked which of the eight were in them.
|
||
|
||
**Two.** Gargoyles 666 and 667, in `AnimationFrame3.uop`. The other six are in no package at all.
|
||
|
||
| Body | Legacy index | In a UOP package? |
|
||
|---|---|---|
|
||
| Gargoyle 666, 667 | index position runs **past the end of `anim5.idx`** | **yes — `AnimationFrame3.uop`** |
|
||
| Human ghosts 402, 403 | `lookup -1` — honest absence | no |
|
||
| Elf ghosts 607, 608 | `length 0` — the §4.8 shape | no |
|
||
| Gargoyle ghosts 694, 695 | `lookup -1` | no |
|
||
|
||
**That absence is established, not merely unfound**, which matters because "I looked and did not
|
||
see it" is exactly how a name scheme that is subtly wrong presents. The five packages hold
|
||
**10,724 entries** between them, and hashing `build/animationlegacyframe/%06d/%02d.bin` over bodies
|
||
0–4095 and actions 0–99 claims **all 10,724** — every entry accounted for, none left over for
|
||
another naming to hide in. No client ships ghost art. §5.2 is where that lands, and §17.9 is the
|
||
decision it produced.
|
||
|
||
#### The 233 the phase was not asked for
|
||
|
||
The rule "legacy first, UOP when it has nothing" is the same code whether it is applied to six
|
||
bodies or to all of them, and applied to all of them it finds **244 bodies with a group 0** in the
|
||
packages, of which only **8** also have legacy art. So the catalogue's working set goes from 787
|
||
to **1,022**, measured on the live rig:
|
||
|
||
| | Rows | Of which |
|
||
|---|---|---|
|
||
| `legacy` — vendored `Animations` over `anim*.mul` | **787** | 366 Equipment, 240 Monster, 95 Animal, 65 unlisted, 17 Human, 4 Sea |
|
||
| `uop` — §4.3's reader | **235** | 97 Equipment, 57 Monster, 50 unlisted, 26 Animal, 3 Human, 2 Sea |
|
||
|
||
The mix is worth a sentence because "add every body" sounds like it changes what a catalogue *is*:
|
||
it does not. The legacy 787 was **already** 366 equipment bodies — weapons and tools drawn alone
|
||
for the client to composite — so the UOP's 97 more change the proportion by a point. What a
|
||
bestiary sees is 83 creatures it could not show before, plus the gargoyles.
|
||
|
||
#### The format, as read rather than as assumed
|
||
|
||
A group file is one action of one body, every direction concatenated, zlib-compressed inside the
|
||
package (`flag == 1`, `78 9c`). The payload is not a repackaged mul record:
|
||
|
||
```
|
||
'AMOU' version decompressedSize bodyId … frameCount frameTableOffset
|
||
frame table: frameCount × 16 bytes (group, frameId, two unknowns, pixel offset)
|
||
each frame: its OWN 512-byte ARGB1555 palette, then centreX/centreY/width/height,
|
||
then the same run-length rows the legacy Frame decoder walks
|
||
```
|
||
|
||
Two consequences the reader is built on. The per-frame palette means **no `Ultima.Frame` and no
|
||
`Bitmap`**: the runs are written into a `ushort[]` of our own, which is what §4.4's promise about
|
||
this reader and `System.Drawing` was always about — and it leaves the encode as the only GDI+ step,
|
||
which phase 4 replaced with a ~200-line PNG writer (zlib around net48's raw-deflate-only
|
||
`DeflateStream`, CRC32, one IDAT, filter 0).
|
||
|
||
And **direction is a slice of the frame table**, not an index into five records: direction *d*
|
||
starts at `d * (frameCount / 5)`. On **nine** of the 244 bodies the frame count is not a multiple
|
||
of five (41, 42, 46…), where integer division lands slightly early in the run. That is what the
|
||
reference implementations do and it is the right trade here, because the failure §4.5 and §4.8 are
|
||
written against is a picture of the **wrong creature**, and this cannot produce one — the worst
|
||
case is the right creature at a marginally different angle, on nine bodies, against losing nine
|
||
bodies outright.
|
||
|
||
#### Validate-as-we-go, and the measurement that says it is in the right place
|
||
|
||
§4.5's rule is "validate before calling", because `Ultima`'s decoders take their bounds from the
|
||
file they read. Here there is no library to validate ahead of: this code **is** the decode. So the
|
||
same discipline appears as a bound on every read — the block chain against the file length, a
|
||
record against the file, the inflated length against the declared one, the frame table against the
|
||
payload, and every run header against **both** the record's remaining bytes and the bitmap it
|
||
writes into, which is the bound `Frame` itself does not have.
|
||
|
||
Measured exactly the way §4.5 was measured, because the second number is what makes a checker
|
||
defensible: across every UOP body on a stock client it refused **nothing** that carries art. The
|
||
one body it refuses, 286, declares a 0×0 frame — which the vendored decoder also treats as no art
|
||
rather than as damage, so it reports absent silently rather than logging a defect on every scan.
|
||
|
||
#### The live rig
|
||
|
||
Real sidecar, real ServUO, this machine's client:
|
||
|
||
| | |
|
||
|---|---|
|
||
| Catalogue | **1,022 rows in ONE page, 1,409 ms cold** |
|
||
| Player bodies | **six, and all six have art** — 400/401/605/606 `legacy`, 666/667 `uop` — every one at direction 0 |
|
||
| Direction split | 6 at direction 0, 1,016 at direction 1 |
|
||
| The six ghost bodies | `absent` |
|
||
| Duplicate-sha256 scan | 45 groups, of which **one** is new |
|
||
|
||
That last row is phase 3's cheap detector for the §4.8 bug, and it earned its keep again. The new
|
||
group is bodies 1531 and 1532 — two **distinct** records, 48,281 bytes each, whose first frames are
|
||
identical pixels. What separates that from a reused-buffer defect is not the hash: it is that each
|
||
payload declares its own body id, and the reader checked. A duplicate you can explain at the source
|
||
is data; one you cannot is the bug.
|
||
|
||
Manifest and fetch rows now carry `source` (`legacy` / `uop`) for exactly this reason — an operator
|
||
looking at a wrong picture can say which half of the extractor to doubt, and an acceptance walk can
|
||
prove the fallback fired rather than infer it from a count.
|
||
|
||
**`EXTRACTOR_VERSION` is 2.** Every client file is byte-identical and the answer is different,
|
||
which is the whole of what §7's number exists to say. The UOP packages join `assets.sources` and
|
||
the catalogue id with it, so patching one is drift rather than a silent no-op — and resolving them
|
||
needed its own lookup, because `Ultima.Files`' table of known client files predates UOP animations
|
||
and answers null for every `AnimationFrame*.uop` however present they are. The replacement matches
|
||
**case-insensitively by enumeration**, which is a Linux-host concern rather than a tidiness one.
|
||
|
||
### 4.10 What phase 6 measured: the ceiling, and 452 validated pictures of the wrong body
|
||
|
||
§4.8 found the library returning the previously-decoded body's bitmap for an index entry that
|
||
reads `length 0`. Phase 6 went looking one axis over — along **actions** rather than bodies — and
|
||
found the same class of failure with none of the tells.
|
||
|
||
An index entry for an animation is `bodyBase + action * 5 + direction`, and `bodyBase` comes from a
|
||
**band**: 110 slots for a high-detail body (22 actions), 65 for a low-detail one (13), 175 for a
|
||
people body (35). The bands are contiguous, so the slots immediately after a body's band are the
|
||
**next body's**. Ask for one action past the ceiling and the arithmetic lands on a real entry, at a
|
||
real offset, holding a real animation record — of a different creature.
|
||
|
||
Measured on this machine's stock client, over the 795 bodies the legacy path serves:
|
||
|
||
| Asking one action past the band | Bodies |
|
||
|---|---|
|
||
| Refused by §4.5's validator (the entry is absent or unreadable) | 152 |
|
||
| **Passes `CheckEntry` and `AnimationSane` and decodes** | **643** |
|
||
| ...and the picture is **byte-identical to body+1's action 0** | **452** |
|
||
|
||
Body 1 action 22 is an ettin. Body 3 action 22 is an imp. Both confirmed by rendering them beside
|
||
the body they belong to, because a count would have said the walk was fine — phase 0's validator
|
||
cannot catch this and is not wrong to miss it: there is nothing defective about the record. The
|
||
only defence is to **refuse the address**, so the ceiling lives inside
|
||
`BridgeAssetValidator.ResolveAnimation`, where every caller already goes and no caller can skip it.
|
||
|
||
**The ceiling is the banding, not the library's own `GetAnimLength`.** That function exists, looks
|
||
authoritative and disagrees with the index arithmetic on exactly one body of this client: a body
|
||
reaching file type 5 as id 34 is excluded from the first band by `Animations.GetFileIndex`'s own
|
||
"looks strange, though it works" special case, so it owns **13** actions while `GetAnimLength`
|
||
answers **22**. Taking the larger number is nine actions of somebody else's art, reached on this
|
||
client by translation from body 276. So the count is derived from the same switch that produces the
|
||
offset, in the same file, where the two cannot drift apart.
|
||
|
||
This is the §4.3 never-sweep rule again in a third disguise. Sweeping file types puts a spider on
|
||
the gargoyle page; trusting a `length 0` entry puts the last creature decoded on this one; walking
|
||
past a band puts the *next* creature on it. All three decode cleanly, all three report success, and
|
||
all three are caught by refusing to ask rather than by checking the answer.
|
||
|
||
---
|
||
|
||
## 5. Addressing: one key for every asset
|
||
|
||
Every asset the bridge can serve is named by a single string key, and the key is the cache key,
|
||
the hash key, the filename stem and the manifest row id:
|
||
|
||
```
|
||
static/3922 one item graphic
|
||
static/3922/h33 the same graphic, hue 33 applied
|
||
land/3 one land tile
|
||
body/34/a0 creature body 34, action 0, first frame
|
||
body/820/a23 body 820, whose action 0 is empty — a horse (§11.2)
|
||
body/400/a0/f0..f9 human male, action 0, all ten frames — NOT SERVED (§11.2)
|
||
cliloc/enu the whole converted string table (not an image)
|
||
tree/Spawns/Trammel.xml a ServUO tree file (§10)
|
||
```
|
||
|
||
**The catalogue's key names whichever action its picture came from.** One row per body either way,
|
||
and for all but 73 of this client's bodies that action is 0 — but a body with no art there is
|
||
catalogued at the first action that has any, and the key says which (§11.2). Calling it `a0`
|
||
regardless would have been fewer changes downstream and a key that lies about its content, which
|
||
is the failure §4.5, §4.8 and §4.10 each describe from a different direction.
|
||
|
||
**The frame depth is defined and not served.** `body/400/a0/f0..f9` is well-formed under this
|
||
scheme and every request for it is refused `unsupported` — see §11.2 for the decision and what it
|
||
would cost to change.
|
||
|
||
Three properties this shape buys:
|
||
|
||
- **Hue is part of the key, not a transform.** `itemId` and `hue` are already on the wire together
|
||
(`BridgeMarket.cs:582`, `BridgeProfile.cs:314`), so a marketplace listing already knows the exact
|
||
key for its own picture. Applying hues website-side would mean shipping `Hues.mul` semantics into
|
||
Node for no gain.
|
||
- **Depth is expressible without being mandatory.** `body/400/a0` and `body/400/a0/f0..f9` are the
|
||
same addressing scheme at two depths, which is what lets §11 bulk-import thumbnails and, if
|
||
anything ever wants them, fetch full animations on demand without a second protocol. Phase 6
|
||
measured what that second depth costs and deferred it (§11.2); the point of the scheme is that
|
||
deferring it is a decision about *what to serve*, not about what can be named.
|
||
- **Nothing in the key is client-version-specific**, so a client patch changes an asset's *bytes*,
|
||
not its name — which is what makes §7's delta work.
|
||
|
||
### 5.1 There is no direction segment, because only one direction is wanted
|
||
|
||
Bodies are stored in **five** directions and the client mirrors three of them to reach eight. Only
|
||
one is needed here, so **direction is fixed by the extractor and is not part of the key**. Leaving
|
||
it in would advertise a choice nobody is going to vary and would five-fold every count in §11 for
|
||
nothing.
|
||
|
||
**Which one depends on whether the body is a player character:**
|
||
|
||
| Body | Direction | Why |
|
||
|---|---|---|
|
||
| A player character body | **0** — head-on, facing the viewer | A character is a portrait; it should look at you |
|
||
| Everything else | **1** — front three-quarter | The view that actually reads as a creature (see the caveat below) |
|
||
|
||
Which index is which was determined by **rendering all five** for a human, a wolf and a dragon
|
||
rather than from a table, because the answer is not obvious and the small-thumbnail version of the
|
||
same test suggested the exact opposite:
|
||
|
||
| Index | View |
|
||
|---|---|
|
||
| **0** | **Head-on, facing the viewer** — face, chest and front legs visible |
|
||
| 1 | Front three-quarter |
|
||
| 2 | Full side profile |
|
||
| 3 | Rear three-quarter |
|
||
| 4 | Directly away — back of the head, and a quadruped's tail toward the camera |
|
||
|
||
The caveat the render made obvious is what produced the split: **index 0 is the least legible view
|
||
for four-legged and long-bodied creatures.** A wolf seen head-on is a dark blob; the same wolf at
|
||
index 1 is unmistakably a wolf, which is also why UOFiddler's own thumbnail list picks that view. A
|
||
humanoid has no such problem — it reads fine head-on, and head-on is what a character portrait
|
||
wants.
|
||
|
||
Both indices stay **configuration values** (defaulting to 0 and 1), so changing the catalogue's mind
|
||
later is a setting and a re-import, not a protocol change.
|
||
|
||
### 5.2 "Player character body" is asked of the shard, never hardcoded
|
||
|
||
`Server.Race.AllRaces` gives every registered race, and each carries `MaleBody`, `FemaleBody`,
|
||
`MaleGhostBody` and `FemaleGhostBody`. The plugin enumerates the **living pair** per race, and that
|
||
set — nothing else — takes index 0. On stock ServUO 57.4 that is six ids:
|
||
|
||
| Race | Male | Female | Male ghost | Female ghost |
|
||
|---|---|---|---|---|
|
||
| Human | **400** | **401** | 402 | 403 |
|
||
| Elf | **605** | **606** | 607 | 608 |
|
||
| Gargoyle | **666** | **667** | 695 | 694 |
|
||
|
||
This is the §8 argument again in miniature: only code inside ServUO can answer it, and asking is
|
||
the only thing that works on a shard with a custom race. Two details make the case that a
|
||
hardcoded list would have been wrong — `RaceDefinitions.cs` passes the gargoyle's ghost bodies in
|
||
the **opposite order** to the other two races (695 male, 694 female), and a shard that calls
|
||
`RegisterRace` adds ids no table of ours would contain.
|
||
|
||
**The ghost columns are greyed above because no client has art for any of them, anywhere.** That is
|
||
phase 4's measurement and it is what removed them from the set (§17.9). The evidence is in §4.9:
|
||
402/403 and 694/695 read `lookup -1` in the legacy index, 607/608 read `length 0` — the §4.8 shape,
|
||
where the library hands back the previously-decoded body's picture — and **none of the six appears
|
||
in any `AnimationFrame*.uop`**, established by claiming all 10,724 entries of the five packages
|
||
with a single name scheme rather than by failing to find them.
|
||
|
||
So the set is asked of the shard exactly as before; only the question changed, from four ids per
|
||
race to two. A shard whose client *does* ship ghost art loses nothing: the body is catalogued like
|
||
any other, at direction 1 rather than 0.
|
||
|
||
**And with phase 4 in, all six have art.** The gargoyles were the last gap, and they were never a
|
||
missing-art problem: `Bodyconv.def` sends 666 and 667 to `anim5`, at an index past the end of
|
||
`anim5.idx`, while `AnimationFrame3.uop` has held both all along.
|
||
|
||
| Has art | Via |
|
||
|---|---|
|
||
| Human male/female (400, 401) | vendored `Animations`, `anim.mul` |
|
||
| Elf male/female (605, 606) | vendored `Animations`, `anim5.mul` |
|
||
| **Gargoyle male/female (666, 667)** | **§4.3's UOP reader, `AnimationFrame3.uop`** |
|
||
|
||
The catalogue must still **not** treat a missing player body as an error. It is now an unlikely
|
||
answer rather than the expected one, but a client that ships fewer bodies than this one is an
|
||
ordinary thing to meet, and a status screen that flags failures on every import teaches an operator
|
||
to ignore it. `shard_spawn_creatures.art` staying NULL remains a first-class state everywhere it is
|
||
consumed, which it already is.
|
||
|
||
---
|
||
|
||
## 6. The manifest, and what the two buttons actually do
|
||
|
||
Two stages, which is where **Import** and **Update** come from.
|
||
|
||
**Stage 1 — the source gate.** The shard reports a manifest of the client files themselves:
|
||
size, mtime and content hash of `Cliloc.enu`, `anim*.idx`/`anim*.mul`, `art.mul`/`artidx.mul`,
|
||
`Body.def`, `Bodyconv.def`, `Hues.mul`. Unchanged since the last import, and nothing else happens.
|
||
This is the same hash gate the spawn atlas and the cliloc table already use, and for the same
|
||
reason: the normal case is a restart that changed nothing, and it must cost nothing.
|
||
|
||
`anim.mul` is 195 MB and `art.mul` is 148 MB, so the gate is **(size, mtime) first, content hash
|
||
only when those differ** — a full hash of 343 MB on every status poll would make the admin panel
|
||
feel broken.
|
||
|
||
**Phase 1 found that the rule is not sufficient on its own, because of §3.3's other ceiling.** The
|
||
first time those hashes are needed there is nothing cached to compare against, so "hash only when
|
||
(size, mtime) differ" still means hashing 343 MB — inside a 10 s reply timeout it will not fit,
|
||
and the call would 504 rather than answer. So the shard's hashes are computed **off the request
|
||
path entirely**:
|
||
|
||
- `assets.sources` always answers immediately, with `size` and `mtime` for every file and
|
||
`sha256: null` for any file whose hash is not cached against exactly that (size, mtime).
|
||
- A file with a missing hash starts a background pass on its **own** thread — deliberately not a
|
||
job on §3.2's single-slot worker, which would answer every status poll `bridge.busy` for the
|
||
minutes the pass takes, at exactly the moment an operator is watching the panel.
|
||
- The reply carries `hashing` and `complete`, so the website knows to poll again rather than to
|
||
treat a null hash as a changed file.
|
||
|
||
The gate is unchanged; what changed is that "the normal case must cost nothing" now also means
|
||
"and the abnormal case must not time out".
|
||
|
||
**Stage 2 — the asset manifest.** For the working set (§11), the shard streams
|
||
`[{ key, sha256, bytes }]` — no pixels. The website diffs that against what it holds and requests
|
||
**only the keys whose hash changed**.
|
||
|
||
- **Update** = stage 1, then stage 2, then fetch the diff.
|
||
- **Import** = the same path with the diff skipped and every key fetched.
|
||
- **A key that has vanished** from the manifest is staged for review, never applied silently —
|
||
the same rule, and the same reasoning, as a vanished cliloc source or a disappearing atlas
|
||
facet. An unmounted volume and a deliberate client downgrade look identical from here.
|
||
|
||
Clilocs are the exception and stay a **whole-table replace** whenever the file hash changes:
|
||
the measured cost is 663 ms for 67,496 rows, so per-entry deltas would be complexity bought for
|
||
nothing.
|
||
|
||
---
|
||
|
||
## 7. The parser version applies here too
|
||
|
||
`spawnAtlasSource.js` carries `PARSER_VERSION` (currently 5) and the cliloc source carries its own,
|
||
both counted as drift so that a corrected parse reaches an install whose files never change. The
|
||
asset pipeline inherits the rule and needs it more, not less: a fixed hue application or a
|
||
corrected frame offset changes the bytes we derive from files that are byte-identical.
|
||
|
||
**`EXTRACTOR_VERSION` lives in the plugin**, because the plugin is what derives the bytes, and it
|
||
is folded into stage 1's gate. Bumping it makes every asset drift, which is correct.
|
||
|
||
---
|
||
|
||
## 8. Body ids: the part only the shard can do
|
||
|
||
The atlas knows creatures by **slug**, derived from type names in `Spawns/*.xml`. The client knows
|
||
them by **body id**. Nothing in the ServUO tree declares the mapping as data — today an operator
|
||
bridges it by grepping `Scripts/Mobiles/Normal/<Name>.cs` for `Body =`, which appears variously as
|
||
a decimal, as hex (`0xD1`), as `Utility.RandomList(35, 36)`, and as an `m_IDs[]` table.
|
||
|
||
Inside ServUO the problem does not exist. `BridgeWorld.cs:350` already does exactly the required
|
||
thing for a different feature:
|
||
|
||
```csharp
|
||
var type = ScriptCompiler.FindTypeByName(name, true);
|
||
var creature = Activator.CreateInstance(type) as BaseCreature;
|
||
```
|
||
|
||
Construct, read `creature.Body.BodyID`, `Delete()`. Authoritative, no source parsing, and correct
|
||
for custom creatures a grep would never find.
|
||
|
||
**This pass must run on the Core thread** — it constructs and deletes mobiles, which is world
|
||
mutation — while the decode in §4 must run **off** it. That split is the one genuinely new
|
||
threading shape in this protocol, and it is why slug→body resolution is its own request kind with
|
||
its own (small) batch size rather than a step inside asset extraction.
|
||
|
||
Constructing arbitrary creature types has side effects: constructors pack items, set skills, start
|
||
timers. The mitigations are per-type `try`/`catch`, immediate `Delete()`, small batches, and the
|
||
fact that the whole pass is admin-triggered rather than something that runs at boot.
|
||
|
||
### 8.1 What phase 3 measured
|
||
|
||
**Built and walked 2026-09-10** against a live ServUO with a real world (43,000 mobiles, 210,000
|
||
items) and the real sidecar.
|
||
|
||
**The catalogue is 787, exactly as §4.8 predicted**, and it arrives in one page: the whole scan of
|
||
bodies 1–2047 — index validation, 787 decodes, 787 PNG encodes and 787 SHA-256s — took **734 ms**
|
||
cold. That is well inside the 3 s scan budget, so the wall-clock paging §11 was designed for never
|
||
fired on this client. It stays, because the budget is what keeps a slower host or a larger family
|
||
inside the 10 s reply timeout rather than producing replies that are always thrown away.
|
||
|
||
Every prediction in §4.8 and §5.2 held when the bytes were actually rendered and looked at:
|
||
|
||
| Asked for | Answer | Why it matters |
|
||
|---|---|---|
|
||
| `body/320/a0` (`length 0`) | **absent** | The 357-class bug. The library would have returned the previously-decoded creature |
|
||
| `body/607/a0`, `body/608/a0` (elf ghosts) | **absent** | §4.8's two rows that moved; the library returns the elf female here |
|
||
| `body/666/a0` (gargoyle → `anim5`) | **absent** | The spider trap. Nothing swept, nothing found |
|
||
| `body/400/a0/…` deeper key | **unsupported** | Well-formed under §5, not served until phase 6 |
|
||
| direction distribution across 787 | **783 at index 1, 4 at index 0** | Four player bodies, not six — §4.8 again, from `Race.AllRaces` rather than a table. *Phase 4 made it six of six and 1,016/6 across 1,022 rows: §4.9* |
|
||
|
||
**44 of the 787 hashes are shared by two or three bodies, and that is correct.** It is the exact
|
||
signature the wrong-picture bug produces, so it was chased rather than assumed: the client's own
|
||
`Body.def` says `83 {1}`, `84 {1}`, `138 {7}`, `139 {7}`, `106 {12, 59}`, and the sharing groups
|
||
match those lines rather than being runs of consecutive ids (which is what a reused stream buffer
|
||
produces). The distinguishing check is at the source: `Animations.Translate(ref body, ref hue)`
|
||
rewrites `body` **only when bit 31 of the table entry is set**, unlike the one-argument overload
|
||
which always does — and `BridgeAssetValidator.ResolveAnimation` calls that same two-argument
|
||
overload, matching `GetAnimation(…, preserveHue: false, …)`. Validator and decoder therefore
|
||
resolve the identical record, which is the property the whole §4.5 design rests on.
|
||
|
||
**The Core-thread pass costs about 190 ms per 100 types.** All 455 stock `Scripts/Mobiles/Normal`
|
||
classes were constructed and deleted in five chunks, producing every status the protocol defines
|
||
(`ok` 436, `unknown` 11, `notCreature` 4, `failed` 4). The world's mobile count went from 43,000 at
|
||
boot to 42,924 afterwards and its item count fell too — so `Delete()` is reclaiming the packed
|
||
inventory as well as the mobile, and nothing leaked.
|
||
|
||
The three refusals answer as designed end to end: an unknown `family` is **400**, a stale `catalog`
|
||
on a fetch is **422**, and 101 types in one `assets.bodies` is **400** naming the cap.
|
||
|
||
---
|
||
|
||
## 9. The cliloc decompressor is ours now
|
||
|
||
Every modern client ships `Cliloc.*` in the Mythic compressed container — this machine's
|
||
`Cliloc.enu` is 4,989,921 bytes beginning `E8 79 67 8E`, high byte `0x8E`. ServUO's bundled
|
||
`Ultima.StringList` implements only the plain layout and throws on it (§1.1), which is also why
|
||
the shard's own `VendorSearch.GetItemName` is already inert.
|
||
|
||
**UOFiddler is released under the Beerware licence**, so porting its decompressor into our
|
||
GPL-3.0-or-later tree is clean. It lands in the overlay as ordinary C# — the **only** decoder
|
||
Protocol 8 writes rather than calls (§4) — and from that point:
|
||
|
||
- No operator installs UOFiddler.
|
||
- No operator runs `dotnet build` on a converter.
|
||
- No operator copies a 5 MB file to a server.
|
||
- `website/server/tools/cliloc-export/` is retired, and `UOFIDDLER.md` is deleted rather than
|
||
rewritten. (Phase 2 deleted its Part 1; **phase 3 deleted the page**, a phase earlier than that
|
||
section predicted, because creature art was the only thing left on it.)
|
||
|
||
### 9.1 What phase 2 built, and what the port cost
|
||
|
||
**Built 2026-09-10.** `BridgeCliloc.cs` in the overlay, `GET /cliloc` in the sidecar, and a paging
|
||
walk in module-uo that merges the shard's table under the `custom/` overlays.
|
||
|
||
The port is `MythicDecompress` + `MoveToFront` rewritten against plain arrays: the upstream is
|
||
`Span<T>` / `ArrayPool<T>` / `BinaryPrimitives` code and ServUO targets `net48`, which has none of
|
||
them without a package this tree does not vendor. The algorithm is unchanged, **including the parts
|
||
that read oddly** — the three-region count/cursor/end table and the symbol-table shifts are
|
||
upstream's, deliberately, because a tidier rewrite of somebody else's format decoder is a chance to
|
||
be subtly wrong in a way that produces plausible text.
|
||
|
||
Two bounds checks were added, and they are the only behavioural change: the upstream indexes its
|
||
payload without checking, which is safe for a file the client wrote and is not safe for a file this
|
||
shard was handed. A truncated container now reports `UNREADABLE` with the byte it wanted, instead of
|
||
throwing an `IndexOutOfRangeException` from inside a decoder.
|
||
|
||
**Measured on this machine's stock client:** 4,989,921 bytes read, decompressed and parsed in
|
||
**290 ms**, yielding **67,496** non-blank rows in id order, ~5.4 MB on the wire, ~11 pages.
|
||
|
||
That 67,496 is the acceptance test, and it is worth saying why it is a strong one: it is the number
|
||
[`../website/CLILOCS.md`](../website/CLILOCS.md) already recorded for this same client, measured
|
||
through **UOFiddler's own `Ultima.dll`** by the converter this phase deletes. An independent
|
||
implementation agreeing to the row is not something a subtly-wrong decoder produces. Checked
|
||
alongside it: zero U+FFFD, so the UTF-8 survived; 696 rows with non-ASCII text, spot-checked as
|
||
correct curly quotes; the longest row is a 12,149-character EULA, which is also why the record
|
||
length is read **unsigned** (a signed read turns anything over 32 KB negative).
|
||
|
||
**The shard drops blanks before they reach the wire.** ~56,000 of the 123,490 entries are empty
|
||
strings the client reserves, the website discards them at import anyway, and sending them would
|
||
double the transfer for data thrown away on arrival.
|
||
|
||
**Language is a parameter (`enu` by default) but not a free one.** `Ultima.Files` resolves only the
|
||
names in its own table, which for clilocs is `enu`, `deu`, `custom1` and `custom2`; anything else is
|
||
refused `NOT_FOUND` rather than answered with an empty table. `custom1`/`custom2` are the
|
||
client-side custom cliloc files a shard ships to its players — readable here, and deliberately not
|
||
wired into the website's import, because `custom/` on the site is the supported answer for
|
||
shard-added ids.
|
||
|
||
**`assets.error` gained a `code`.** Phase 1 chose between 403 and 400 by looking for the word
|
||
"disabled" in the operator-facing sentence, which makes prose load-bearing; the codes are
|
||
`DISABLED` (403), `NOT_FOUND` (404), `UNREADABLE` (422), `UNAVAILABLE` (503) and `BAD_REQUEST`
|
||
(400). The substring check survives as a fallback, because an overlay and a sidecar are deployed
|
||
separately and a phase-1 shard must keep its 403.
|
||
|
||
### 9.2 Where the base comes from now, and when it is read
|
||
|
||
**The shard wins whenever uo-link is configured and enabled.** No mode setting: there is no version
|
||
of that question an operator benefits from answering. A file on disk remains the source only where
|
||
there is no shard link, plus a one-off explicit path — the deprecated pipeline, kept for installs
|
||
with no bridge and for development.
|
||
|
||
**Boot no longer imports on the bridge.** The file path could hash 5 MB locally and skip in 14 ms; a
|
||
shard round trip in the boot sequence would be spent answering "no" on every restart but the one
|
||
after a client patch — and patching a client is an operator action, so importing became one:
|
||
Admin → Shard → Import. Whatever table is loaded keeps serving until then.
|
||
|
||
Three things about the walk are worth recording because each is a way a shard can hand back a table
|
||
that *looks* complete:
|
||
|
||
- **Only `cut: "end"` finishes it** (§3.4). A short page can equally be a spent budget.
|
||
- **The cursor must advance**, or the walk stops rather than spinning.
|
||
- **Every page echoes the source's size and mtime.** A client patched mid-import is refused
|
||
outright (`SOURCE_CHANGED`) rather than stitched together — half of what arrived came from a file
|
||
that no longer exists and nothing later can tell which half.
|
||
|
||
**The base is exempt from the vanished-source rule**, and that is an upgrade detail rather than a
|
||
design preference: an install that used the file pipeline carries its base file's label in the
|
||
stored fingerprint, and on the bridge that label is *supposed* to disappear. Counting it as vanished
|
||
would make the first import after the upgrade demand approval for a change the upgrade itself made.
|
||
Overlays keep the rule in full.
|
||
|
||
**module-uo's protocol pin moved 7 → 8 in this phase**, which is the third declaration site §15
|
||
names and the one nothing enforces. Phase 1 moved the sidecar and the overlay together because the
|
||
installer refuses to pair a mismatched bundle; this one had to be moved by hand, in the phase that
|
||
first calls a protocol-8 route — the same trap that left the pin at 5 for two phases of the Event
|
||
System while every REST call was answered `409`.
|
||
|
||
**What survives untouched is the `custom/` overlay mechanism.** Shard-added items carry cliloc ids
|
||
no client table has, and ServUO has no server-side notion of a custom cliloc — that is a real gap
|
||
in the *game*, not an artefact of the manual pipeline, and `CLILOCS.md`'s reasoning for it stands.
|
||
The base table now arrives over the bridge; overlays still come from a directory the site reads.
|
||
Measured on the live shard: 16,434 cliloc ids referenced by the script tree, 37 absent from stock.
|
||
|
||
---
|
||
|
||
## 10. The atlas stops needing a shared filesystem
|
||
|
||
Today `SPAWN_ATLAS.md` requires the **website** to read the ServUO tree — "same host, a bind mount,
|
||
or a shared volume". That is the one place the platform's own rule (only the sidecar bridges the
|
||
shard) is broken, and it is broken by the component that faces the internet.
|
||
|
||
The same transport closes it. `spawnAtlasSource.js` already labels every file it reads with a
|
||
portable key:
|
||
|
||
| Label | Count (stock 57.4) |
|
||
|---|---|
|
||
| `Data/Regions.xml` | 1 |
|
||
| `Data/Locations/*.xml` | 6 |
|
||
| `Spawns/*.xml` | 13, ~10.5 MB |
|
||
| `Config/ChampionSpawns.xml` | 1 |
|
||
| `Data/Decoration/**` | tree |
|
||
|
||
So the shard serves `tree/<label>` → bytes over the same batched request/reply path, and
|
||
`spawnAtlasSource.js` gains a second backend behind its existing interface: **filesystem** (today,
|
||
kept for same-host installs and for development) or **sidecar** (new, and the default once
|
||
configured).
|
||
|
||
**The parsers do not move.** `spawnAtlasParse.js` is pure, fs-free and CI-covered without a ServUO
|
||
tree, and every quirk it handles — the two respawn delay units, `:OBJ=` splitting, facet-name
|
||
reconciliation, the XmlSpawner directive stripping — stays exactly where it is. The shard sends
|
||
bytes; the website still decides what they mean. That is the same division as §2, and it keeps the
|
||
sidecar a forwarder here too.
|
||
|
||
`SERVUO_PATH` and the `spawn_atlas_servuo_path` setting remain, and select the filesystem backend.
|
||
|
||
---
|
||
|
||
## 11. What is bulk and what is on demand
|
||
|
||
The scope approved is creature art, item art, player models "and everything", against a future
|
||
project. §1.1's measurements make the sizing question concrete:
|
||
|
||
| Kind | Addressable | Bulk? |
|
||
|---|---|---|
|
||
| Item statics | **49,152** addressable, **39,189** with art (§11.1, phase 5) | No — on demand, cached, keyed by `itemId` (+ hue) |
|
||
| Land tiles | **16,384** addressable, **4,244** with art | No — on demand |
|
||
| Creature/player bodies, first frame | **1,095** — 787 legacy (§4.8, *not* the 1,144 the library reports) + 235 UOP (§4.9) + 73 at a later action (§11.2) | **Yes** — this is the catalogue |
|
||
| One body, every action, one direction | **210 frames** (body 400); median 118 KB per body, max 9.6 MB (§11.2) | **Not served** (§11.2) |
|
||
| All bodies, every action, one direction | **174,453 frames, 281.5 MB** — measured, not the ~119,000 estimated here | **Not served** (§11.2) |
|
||
| The same at five directions | ~865,000 frames | Not built (§5.1) |
|
||
| Cliloc table | 123,490 entries → 67,496 rows | **Yes** — whole-table replace |
|
||
| ServUO tree files (§10) | ~21 files, ~10.6 MB | **Yes** |
|
||
|
||
**The working set is one thumbnail per body, plus the atlas's own creatures.** 1,022 sprites at
|
||
roughly a kilobyte each is about a megabyte — trivial to import, trivial to re-hash, and it is the
|
||
set that makes a bestiary, a marketplace listing and a character sheet render.
|
||
|
||
Phase 1 took that count down from 1,144, and the 357 it removed are the important part: those are
|
||
ids with **no art** that the library returns a picture for (§4.8). Importing them would have
|
||
written 357 duplicate creature portraits into the site, each one showing whichever body the walk
|
||
happened to decode before it. The count went down; what the catalogue is worth went up.
|
||
|
||
Phase 4 then took it **up** to 1,022, and the direction of travel is the same argument rather than
|
||
its opposite: the 235 it added are bodies with real art in a file the vendored decoder does not
|
||
open (§4.9), each one validated at every bound and identified by a name that carries its body id.
|
||
Both numbers moved because something was measured rather than reported.
|
||
|
||
Everything deeper is the *same protocol at a deeper key* (§5) — a viewer that wants body 400's full
|
||
walk cycle would ask for `body/400/a2/f0..f9`. **That depth is not implemented** (§11.2): the site
|
||
displays still pictures, so the frames have no consumer, and the design's value here is that a
|
||
future one costs a reader and a store rather than a protocol.
|
||
|
||
Because §5.1 dropped four of the five directions, a **complete** one-direction animation set for
|
||
every body looked like ~119,000 frames rather than ~865,000. Phase 6 measured it at **174,453
|
||
frames and 281.5 MB** (§11.2) and, more to the point, established that nothing on this site would
|
||
ever read them: the site shows still pictures. So neither the deep keys nor the
|
||
bulk-fill-everything switch this paragraph anticipated was built, and the numbers above are what a
|
||
future consumer would be choosing to pay.
|
||
|
||
**Hued variants are on demand, always.** `static/3922/h33` is generated when something on the wire
|
||
actually carries hue 33. The cross product of 49,152 statics and 3,000 hues is not a set anyone
|
||
enumerates.
|
||
|
||
**Everything deeper than the catalogue is the same protocol at a deeper key, and is refused rather
|
||
than served** — §11.2. That sentence was written expecting phase 6 to serve it; what phase 6
|
||
actually found was that no consumer exists, so the depth stays named and unserved.
|
||
|
||
### 11.1 What phase 5 measured, and the two traps it found
|
||
|
||
The sizing above was an estimate taken off `art.mul`'s length. Measured through the reader itself,
|
||
against this machine's stock client:
|
||
|
||
| | Measured |
|
||
|---|---|
|
||
| Static ids addressable | **49,152** |
|
||
| ...with real art | **39,189** |
|
||
| ...empty index slots (§4.5's shape) | **9,963** |
|
||
| Land tiles addressable / with art | 16,384 / **4,244** |
|
||
| Whole static + land set, as PNG | 43,433 files, **81 MB**; mean 1.9 KB, max 30.9 KB (id 18213) |
|
||
| Time to decode and encode all of it | **34 s** |
|
||
| Hue slots in `hues.mul` | 3,000 (2,062 named) |
|
||
| Item ids flagged `PartialHue` | **13,259** of 65,536 |
|
||
|
||
Two of those need saying out loud.
|
||
|
||
**49,152, not the 81,884 entries `artidx.mul` declares.** `FileIndex` sizes its index table from the
|
||
**length argument it is constructed with** (`0x10000`), not from the idx file, so the addressable
|
||
static range is `0x10000 - 0x4000`. A ceiling read off the file instead would invent 16,348 ids and
|
||
answer every one of them out of an array nobody bounded. (The first probe of this phase made exactly
|
||
that mistake and reported 65,500 — PowerShell returns `$null` for an out-of-range array index rather
|
||
than throwing, so the over-run counted silently as "empty slots".)
|
||
|
||
**81 MB is small enough to reopen the bulk question, and the answer is still no.** Not on size — on
|
||
what the transfer buys. Base64 puts it at 108 MB through a 512 KB single-slot channel, roughly 210
|
||
round trips, to store 43,433 pictures of which a live shard displays a few hundred. On-demand stays
|
||
right, and phase 6 did not build the bulk-fill switch this sentence promised — see §11.2.
|
||
|
||
#### The library's cache poisons a hued sprite
|
||
|
||
`Art.GetStatic` and `Art.GetLand` memoise into a static `Bitmap[0xFFFF]` and hand back **the same
|
||
instance** on every call; `Hue.ApplyTo` repaints a bitmap **in place**. So the obvious implementation
|
||
— ask the library, apply the hue, encode — edits the library's own copy. Measured before the fix:
|
||
hue item 3922 once, and every later request for the **plain** 3922 comes back hued, with a second hue
|
||
stacking on the first.
|
||
|
||
This is §4.5's failure mode exactly — a confident, plausible, correctly-sized wrong picture that
|
||
every success count agrees with — reached through a door §4.5 never looked at, because phase 0 was
|
||
auditing *records* and this is the library's *cache*. Nothing downstream can see it: the key is
|
||
right, the dimensions are right, the hash is stable.
|
||
|
||
The fix is `Files.CacheData = false` for the life of the process, set once when the asset plane
|
||
initialises, and it pays twice: the same array is never trimmed, so decoding this client's 39,189
|
||
statics would otherwise leave **74 MB of `Bitmap` in a static field of a game server** to serve
|
||
pictures nobody asks for twice. `Animations` does not consult the flag at all, so the body catalogue
|
||
is untouched, and each reader keeps its own cache of *encoded PNG bytes* instead — a tenth of the
|
||
size, already hashed, released when it goes idle.
|
||
|
||
The obvious alternative, copying each bitmap before hueing, was rejected for the retention alone —
|
||
but also because `new Bitmap(src)` **throws** on the `Format16bppArgb1555` these decoders produce.
|
||
The copy has to name the source pixel format explicitly, which is a subtlety on the wrong side of a
|
||
correctness boundary. The invariant is instead re-checked before every hue: a hue is refused if the
|
||
cache is somehow on, because an invariant nothing verifies is a comment.
|
||
|
||
#### `PartialHue` decides the picture, and only the shard can read it
|
||
|
||
A hue is not a tint. It is a 32-entry colour ramp out of `hues.mul` indexed by each pixel's own red
|
||
channel — and whether it replaces **every** pixel or only the grey ones is a per-item-id flag in
|
||
`tiledata.mul`. On this client **13,259 of 65,536 item ids carry it**.
|
||
|
||
Item 597 is a wooden screen with painted flowers. Hued 33 the right way the flowers turn red; the
|
||
wrong way the whole screen turns red. Both decode, both are 44×112, both report success. This is why
|
||
§5 put hue in the key rather than leaving it to the website: shipping `Hues.mul` semantics and a
|
||
65,536-row flag table into Node, to answer a question the shard can answer for free, is the trade
|
||
§2.1 already refused.
|
||
|
||
Two consequences fall out of it. **Land takes no hue segment** — the mode is an *item* flag and land
|
||
has no equivalent, so `land/3/h33` is refused rather than guessed; nothing on the wire carries one
|
||
today, and if something ever does it will arrive with a reason to choose. And **`h0` is not a key**:
|
||
hue 0 on the wire means "not hued", so the plain key already names that picture, and accepting both
|
||
would store one PNG twice under two names and diff them separately forever.
|
||
|
||
#### The namespace trap that compiled
|
||
|
||
The first cut of the reader wrote `TileData.ItemTable` and `TileFlag.PartialHue` unqualified. ServUO
|
||
declares its **own** `Server.TileData`, `Server.ItemData` and `Server.TileFlag` — with a
|
||
`PartialHue` member — in `Server/TileData.cs`, and the reader lives in `Server.Custom.Bridge`, where
|
||
the enclosing namespace beats `using Ultima;`. It compiled. At runtime it read a file resolved
|
||
through `Core.DataDirectories` rather than through `Ultima.Files`, which is §4.6's rule broken in a
|
||
new place: deciding a picture with a file other than the one the pixels came out of. The live rig
|
||
caught it as a `TypeInitializationException` refusing every hued key, from a class the code never
|
||
meant to name.
|
||
|
||
**Gump art is out of scope for Protocol 8, and that is now a safety rule rather than a priority
|
||
call** — §4.1. It is the only decoder that reaches the `hasExtra: true` branch, and that branch
|
||
corrupts the process on the second id. Adding gump art later means fixing that path first,
|
||
deliberately; it is additive under the same key scheme (`gump/<id>`), which is the point of §5.
|
||
|
||
### 11.2 What phase 6 measured, and why the deep keys are not built
|
||
|
||
Everything above about full animations was an estimate — ~119,000 frames, ~117 MB, from a
|
||
151-frame average over six bodies. Phase 6 decoded and PNG-encoded **every action of every body at
|
||
the catalogue's direction**, which is the set a complete one-direction animation store would hold:
|
||
|
||
| | Estimated (§11) | Measured |
|
||
|---|---|---|
|
||
| Bodies with art at *some* action | 1,022 | **1,096** |
|
||
| Action keys (`body/<id>/a<n>`) with frames | — | **26,274** (mean 25.4 per body, max 35) |
|
||
| Frames | ~119,000 | **174,453** |
|
||
| As PNG | ~117 MB | **281.5 MB** (mean 1,692 B, largest single frame 71.5 KB) |
|
||
| Per body | — | median **118 KB** / 210 frames; p90 536 KB; max **9.6 MB** (body 826) |
|
||
| Decode + encode the lot | — | **104 s** |
|
||
|
||
Two facts fall out of that walk that any frame-serving design has to answer.
|
||
|
||
**Frames are not a sequence of pictures; they are a sequence of *placed* pictures.** Every frame
|
||
carries its own centre offset and its own dimensions, and **23,818 of the 26,274 actions have
|
||
frames that differ in size** (centres span x −51..270, y −217..184). A key that returned only a PNG
|
||
would produce an animation that jitters, and nothing downstream could tell — the same shape of
|
||
silent wrongness as every other trap in this document.
|
||
|
||
**At 281.5 MB the transfer is ~375 MB of base64 through a 512 KiB single-slot channel** (§3.2),
|
||
which is upward of 750 round trips for a store of 174,453 files.
|
||
|
||
### The decision: the site shows still pictures, so frames wait for a consumer (org lead, 2026-09-11)
|
||
|
||
§11 justified the deep keys with "the future project", not with anything on this site — and the
|
||
site does not display animation anywhere, nor is it planned to. So phase 6 builds **none** of it:
|
||
no frame keys, no per-body frames manifest, no bulk-fill switch, no store sized for 281 MB, and no
|
||
admin surface for any of that. What would have been the consumer-side machinery is exactly the part
|
||
that would have been guessed at, and when a consumer does exist it can say what shape it wants.
|
||
|
||
`body/<id>/a<n>/f<k>` stays defined in §5 and refused in practice. Reviving it costs the frames
|
||
manifest, the alignment fields and a store — not a protocol change, which is the whole point of
|
||
§5's key scheme.
|
||
|
||
### What phase 6 built instead: the 73 bodies nobody could see
|
||
|
||
The measurement did turn up something the still-picture site wants. **73 bodies have no art at
|
||
action 0 and real art at a later one** — 66 reached through the UOP packages, 7 through the legacy
|
||
files. Body 820's first drawn action is 23, and it is a horse. Until this phase they were absent
|
||
from the catalogue and rendered as text on the bestiary.
|
||
|
||
So the catalogue now falls back to **the first action that has art**, and the key names it
|
||
(§5). The count goes **1,022 → 1,095**, a full cold scan of all 2,047 bodies goes 1,409 ms → 2,090 ms,
|
||
and `EXTRACTOR_VERSION` goes **2 → 3** — unchanged input, a different answer, which is what that
|
||
number is for.
|
||
|
||
Two honesties about those 73. The seven legacy ones and roughly a dozen of the UOP ones are
|
||
recognisable creatures — a horse, a spider, a dragon, a phoenix, a turkey. **Most of the rest are
|
||
thin flat sprites** with entries at actions 1 and 3 only, all anchored at centre x = 63: object or
|
||
equipment art rather than creatures. They are correct by construction (a UOP entry is addressed by
|
||
the hash of a name carrying body *and* action, and the payload re-declares the body), they cost
|
||
about 200 bytes of row each, and nothing surfaces them unless a spawn file names a creature class
|
||
that resolves to one of those ids. And the fallback is the one walk in this protocol that moves
|
||
along the action axis, which is why §4.10's ceiling had to ship with it rather than after it.
|
||
|
||
---
|
||
|
||
## 12. Where it lands on the website
|
||
|
||
Images are written by module-uo into the upload directory. `ctx.uploads`
|
||
(`{ upload, UPLOAD_DIR, MIME_EXT }`) is **already** exposed to modules and
|
||
[`MODULE_API.md:626`](../website/MODULE_API.md) already names its consumer as "atlas art import",
|
||
so no `MODULE_API_VERSION` bump is needed to store them.
|
||
|
||
- `shard_spawn_creatures.art` stops being NULL-by-default and starts being filled by the import.
|
||
- A new asset table carries `key`, `sha256`, `bytes`, `width`, `height`, `imported_at` — the
|
||
manifest side of §6, and what makes an Update a diff rather than a re-download.
|
||
- **The operator-supplied `spawnAtlas.art.json` map stays supported** and continues to win over an
|
||
imported asset. An operator who has drawn their own creature portraits must not have them
|
||
overwritten by a sprite rip on the next Update.
|
||
- **Item and land pictures land in `uploads/items/`, beside the creature portraits and not among
|
||
them** (phase 5). Same content-addressed naming (`uo-static-3922-h33-<sha8>.png`), same "the API
|
||
returns a filename, the client builds the URL" contract, and the same rule that a missing picture
|
||
is a first-class state rather than an error. Separate directories because they have different
|
||
lifetimes: the catalogue is imported as a set and re-imported as a set, while these arrive one at
|
||
a time because something asked for them.
|
||
|
||
Licensing is unchanged and the reasoning is unchanged: these are the operator's own client files,
|
||
extracted on their own host, for their own shard. Nothing is committed, nothing ships in a repo,
|
||
and nothing is redistributed. What changes is only that the extraction stopped requiring a GUI on a
|
||
desktop.
|
||
|
||
### 12.1 The one thing above that phase 3 had to build differently
|
||
|
||
"`shard_spawn_creatures.art` … starts being filled by the import" is right about the outcome and
|
||
wrong about the mechanism, and the difference is not cosmetic.
|
||
|
||
**That table is emptied and refilled by every atlas refresh** — `shardAtlas.db.js`'s `replaceAtlas`
|
||
DELETEs all six atlas tables inside one transaction — and a refresh runs on **every boot**. Before
|
||
protocol 8 that cost nothing, because `art` came from a file on disk and was simply re-read each
|
||
time. An imported sprite is the opposite: expensive to obtain, and gated on client-file hashes that
|
||
would say "unchanged" for weeks afterwards. Writing it onto the creature row would mean an ordinary
|
||
re-parse of the ServUO tree silently deleting every portrait, with the next Update reporting
|
||
*nothing to do* and never restoring them. Nothing would report a fault; the pictures would just be
|
||
gone.
|
||
|
||
So phase 3 built three tables outside that blast radius, and the atlas import reads them on the way
|
||
past (org lead, 2026-09-10):
|
||
|
||
| Table | Holds | Lifetime |
|
||
|---|---|---|
|
||
| `shard_assets` | `asset_key`, `sha256`, `bytes`, `width`, `height`, `body`, `direction`, `file` | Upserted per key; only an approved vanish deletes |
|
||
| `shard_creature_bodies` | `slug`, `type_name`, `body`, `status` | Replaced whole — it is derived from the atlas's creature list, so a slug that has left has no meaning |
|
||
| `shard_asset_meta` | The singleton an Update compares against | Replaced |
|
||
|
||
`replaceAtlas` now takes `{ ...derived, ...operatorMap }`, which is where "the operator's map wins"
|
||
is actually enforced — one spread, in one place, applied on every rebuild rather than only at
|
||
import time.
|
||
|
||
**Two details worth not rediscovering.** The derivation joins on the catalogue **key**, not on
|
||
`a.body = b.body`: today one body has exactly one asset and the simpler join is correct, and it
|
||
stops being correct the day a deeper key (`body/400/a2/f0`) is stored, at which point one slug
|
||
matches dozens of rows and whichever the engine returned last becomes the portrait. And the stored
|
||
filename is **content-addressed** (`uo-body-34-a0-<sha8>.png`), because a stable name overwritten
|
||
in place leaves every browser and CDN serving last month's client's sprite from cache with the
|
||
database row perfectly correct.
|
||
|
||
Phase 6 changed that join, and the way it changed is the point. It read
|
||
`a.asset_key = CONCAT('body/', b.body, '/a0')`, which stopped being right the moment a body could
|
||
be catalogued at another action — it would have dropped exactly the 73 creatures §11.2 added, a
|
||
horse among them, silently. It now reads the row's own action:
|
||
|
||
```sql
|
||
JOIN shard_assets a ON a.body = b.body AND a.family = 'body'
|
||
AND a.asset_key = CONCAT('body/', b.body, '/a', COALESCE(a.action, 0))
|
||
```
|
||
|
||
`COALESCE` because a row written before the column existed has NULL there, and a NULL inside
|
||
`CONCAT` makes the whole comparison NULL — which would have taken every portrait off the site on
|
||
upgrade, with the database perfectly correct and nothing to see in a log. The `action` is stored
|
||
rather than parsed back out of the key because this join needs it in SQL, and re-deriving it there
|
||
would put a second, weaker parser of §5's key scheme in the schema.
|
||
|
||
`shard_creature_bodies` also answers §8 without a schema change on the atlas side:
|
||
`shard_spawn_creatures.name` already holds the ServUO **class name** — the atlas build picks the
|
||
winning spelling of the spawn type token rather than inventing a display label — so the import has
|
||
something `ScriptCompiler.FindTypeByName` will resolve without storing it twice.
|
||
|
||
---
|
||
|
||
## 13. Visibility
|
||
|
||
New surfaces over shard data are admin-toggleable with an operator-set audience, and this is no
|
||
exception. Asset serving is gated like every other shard read: a `requireFeature` gate, an
|
||
audience, and 404-not-403 when the feature is off, so a disabled feature does not advertise itself.
|
||
|
||
The default is the least surprising one: assets are as public as the page that uses them. A
|
||
bestiary that is already anonymous does not become staff-only because its pictures arrived over a
|
||
new pipe.
|
||
|
||
---
|
||
|
||
## 14. Routes and commands added
|
||
|
||
**Loopback (shard ↔ sidecar), all request/reply:**
|
||
|
||
| Command | Reply | Purpose | Built |
|
||
|---|---|---|---|
|
||
| `assets.sources` | `assets.sources.ok` | Stage 1: client file manifest + `EXTRACTOR_VERSION` | **phase 1** |
|
||
| `assets.manifest` | `assets.manifest.ok` | Stage 2: `[{key, sha256, bytes, width, height, body, direction, source}]`, paged | **phase 3** (`source` phase 4) |
|
||
| `assets.fetch` | `assets.fetch.ok` | Content for an explicit key list, paged; base64 PNG per row | **phase 3** (`source` phase 4; `static`/`land` families phase 5) |
|
||
| `assets.bodies` | `assets.bodies.ok` | Slug → body id (§8, Core thread) | **phase 3** |
|
||
| `cliloc.table` | `cliloc.table.ok` | The decompressed table, paged (`?lang=`, `?cursor=`) | **phase 2** |
|
||
| `tree.manifest` / `tree.fetch` | `.ok` | §10, the ServUO tree files | phase 7 |
|
||
|
||
Every one of them is refused outright when `Bridge.AssetsEnabled` is off, and every one of them
|
||
requires a `reqId` — a request without one is refused rather than answered, because an uncorrelated
|
||
reply is by definition an event, and §3.1 is the reason none of this may ever be one.
|
||
|
||
**Sidecar REST** mirrors those one for one under `/assets/*`, `/cliloc`, `/tree/*`, carrying
|
||
`X-UOLink-Version: 8` and forwarding verbatim. `GET /assets/sources` exists as of phase 1 and
|
||
`GET /cliloc` as of phase 2; their responder maps `bridge.busy` to **425** (flow control, and the
|
||
ordinary answer mid-import rather than a rare one) and reads the refusal's `code` for the rest —
|
||
**403** disabled, **404** no such file, **422** a file the shard cannot decode, **503** a shard that
|
||
cannot do this now (§9.1).
|
||
|
||
Phase 3 added the other three: `GET /assets/manifest?family=&cursor=`, `POST /assets/fetch` and
|
||
`POST /assets/bodies`. **The two POSTs are reads**, and the method is the request body rather than a
|
||
side effect — a few hundred asset keys do not belong in a query string. They are the only reads on
|
||
this link that take one. `422` gains a second meaning on this plane alongside "cannot decode": the
|
||
mid-import guard, a `catalog` that no longer describes the files on disk.
|
||
|
||
**Phase 5 added two families, one field, and no command.** `assets.fetch` grew `static` and `land`
|
||
(§5, §11.1) and `assets.sources` grew **`families`** — which key families this overlay serves. Both
|
||
are additive, so **the protocol stays 8**, and `EXTRACTOR_VERSION` stays **2**: no existing key's
|
||
bytes change, and a new key is not a re-derivation of an old one.
|
||
|
||
The command itself became shared plumbing. Phase 3 gave `assets.fetch` to the body catalogue
|
||
outright, which was right with one family and wrong with three: the command is the *transport* and
|
||
the family is a property of the key. So the correlation id, the operator's consent, the key-count
|
||
ceiling and the family decision now happen once, and a reader only ever sees keys it owns.
|
||
|
||
**The family is derived from the keys and is not a request field.** §5 made the key the address of an
|
||
asset; a request that also named its family would have two places to be wrong and one of them
|
||
silent. A batch must be of **one** family — mixing them is refused (400) rather than split — because
|
||
the reply carries a single `catalog` id, and two families have two fingerprints. A reply claiming one
|
||
of them would be lying about the other.
|
||
|
||
`families` matters more than it looks. Without it, a website talking to a phase-3 or phase-4 overlay
|
||
discovers the gap as a refusal *per key, per pass, forever*, with no picture ever appearing and a
|
||
warning in the log every few minutes. With it, that is one reported state carrying a sentence naming
|
||
the fix.
|
||
|
||
**Phase 6 added one field and no command.** Manifest and fetch rows carry `action` — which action
|
||
of the body the thumbnail came from (§11.2). Additive, so **the protocol stays 8**; a consumer that
|
||
ignores it sees the catalogue it always saw plus 73 rows, and one that reads it can build the right
|
||
URL for a body catalogued at `a23`. `EXTRACTOR_VERSION` goes **2 → 3**, which is the change every
|
||
consumer does see. A fetch for a key naming an action the catalogue did not choose is answered
|
||
`unsupported` with the chosen action alongside it — never by decoding the asked-for action, which
|
||
is §4.10's wrong picture reached politely.
|
||
|
||
**Phase 4 added one field and no command.** `source` on a manifest or fetch row is `legacy` or
|
||
`uop` — which reader produced the bytes (§4.9). It is additive, so **the protocol stays 8**: a
|
||
consumer that does not read it is unaffected, and one that does can say which half of the extractor
|
||
to doubt when a picture is wrong. `EXTRACTOR_VERSION` went to **2** in the same phase, which is the
|
||
change every consumer *does* see, exactly as §7 intends.
|
||
|
||
**§16 listed phase 3 as `servuo-plugins, module-uo` and that was wrong** — `web.rs` routes every
|
||
command explicitly and has no generic `/assets/*` forwarder, so `link` is in the phase too. The
|
||
table now says so.
|
||
|
||
**Website admin** (`Admin → Shard`, admin-only): status, **Import**, **Update**, approve/reject for
|
||
a vanished key, and the existing path settings. Every action to the admin activity log, as
|
||
`shard.assets.*`. Phase 3 shipped the two that make it reachable — `GET /admin/shard/assets` and
|
||
`POST /admin/shard/assets/import`, mirroring the cliloc pair — so the phase could be accepted on a
|
||
real rig; the full panel is phase 8 (org lead, 2026-09-10).
|
||
|
||
---
|
||
|
||
## 15. Cross-repo obligations
|
||
|
||
`PROTOCOL_VERSION` goes **7 → 8** in `link/sidecar/src/main.rs`, and in the **same PR**
|
||
`servuo-plugins/overlay.toml` — the installer refuses to pair a sidecar and an overlay that
|
||
disagree, so a split bump means the next bundle silently fails to compose.
|
||
|
||
| Repo | Work |
|
||
|---|---|
|
||
| `servuo-plugins/` | Extraction over ServUO's own `Ultima` (§4), the cliloc decompressor (§9), body resolution (§8), the request handlers, `overlay.toml` |
|
||
| `link/` | Six command families forwarded, the REST surface, **the inbound line cap (§3.3)**, `PROTOCOL_VERSION`. **Nothing in phases 4 or 5** — `assets_call` forwards a request body verbatim and `respond_assets` returns the reply verbatim, so a new key family and a new reply field both pass through untouched |
|
||
| `module-uo/` | Client calls, asset store, the atlas source backend (§10), cliloc ingest, admin surface |
|
||
| `website/` | Almost none — `ctx.uploads` already suffices (§12). Phase 2 deleted `server/tools/cliloc-export/`, the converter this protocol retires |
|
||
| `docs/` | This file; rewrite `CLILOCS.md` §Converting and `SPAWN_ATLAS.md` §Artwork + §Configuring; **delete `UOFIDDLER.md`**; add the libgdiplus prerequisite to `SHARD_PREREQS.md` (§4.4) |
|
||
| `installer/` | A `doctor` check for libgdiplus on Linux hosts (§4.4). Bundle pairing already enforces §15 |
|
||
| `android-app/` | Consumes images by URL; no parity gate expected until a screen shows one |
|
||
| `integration-kit/` | A chapter note only — this is UO-specific and teaches nothing about the module contract |
|
||
|
||
---
|
||
|
||
## 16. Phases
|
||
|
||
| # | Scope | Repos |
|
||
|---|---|---|
|
||
| 0 | **DONE 2026-09-10.** Spike: the vendored decoders driven from inside a running ServUO, over a client broken 21 ways. §4 stands; the finding was 22,102 wrong pictures on a *stock* client, and the validator that answers them (§4.5-§4.7) | servuo-plugins |
|
||
| 1 | **DONE 2026-09-10.** The transport: `assets.sources`, the single-slot gate (§3.2), the paging envelope (§3.4), the sidecar line cap (§3.3), `EXTRACTOR_VERSION`, `NO_IMAGING`, protocol 7→8. Plus §4.5's validator promoted into the overlay and extended to animations — which found **357 wrong pictures in the body catalogue on a stock client** and cut it from 1,144 to 787 (§4.8) | servuo-plugins, link |
|
||
| 2 | **DONE 2026-09-10.** Clilocs end to end (§9.1, §9.2): the Mythic decompressor ported into the overlay, `cliloc.table` + `GET /cliloc`, the paging walk and the source switch on the website, module-uo's protocol pin 7→8. `cliloc-export/` deleted and `UOFIDDLER.md` §Part 1 with it. **67,496 rows, 290 ms, ~11 pages** — the same count UOFiddler's own DLL produced from this client | all |
|
||
| 3 | **DONE 2026-09-10.** Body resolution (§8) + the **787**-body catalogue (§4.8), `assets.manifest` / `assets.fetch` / `assets.bodies` and their REST mirrors, `shard_spawn_creatures.art` filled and rendered (§8.1, §12.1). **787 rows in one 734 ms page; 455 types resolved at ~190 ms per 100 on the Core thread; zero mobiles leaked.** `UOFIDDLER.md` deleted, two phases early | servuo-plugins, **link**, module-uo |
|
||
| 4 | **DONE 2026-09-11.** The UOP animation decoder (§4.3, §4.9): `BridgeUop` + a PNG encoder that never touches `System.Drawing`, wired in beneath the legacy reader. **Two of the eight player bodies turned out to exist** (gargoyles 666/667); the other six are in no client file, and ghost ids left the player-body set (§5.2, §17.9). The same fallback added **233 other bodies**: the catalogue is **1,022 rows, 1,409 ms cold**, and all six player bodies have art for the first time. `EXTRACTOR_VERSION` 1 → 2 | servuo-plugins |
|
||
| 5 | **DONE 2026-09-11.** Item statics and land on demand (§11.1): the `static` and `land` families, hue applied on the shard from `tiledata.mul`, the byte-bounded art cache, `assets.fetch` made family-aware, `families` on `assets.sources`. Website side: the warm pass, per-row `catalog` staleness, and pictures on the marketplace and the character sheet. **39,189 statics and 4,244 land tiles served; the only refusals are the 9,963 + 12,140 empty index slots §4.5 predicted.** Two traps found — the library's bitmap cache poisons a hued sprite, and `PartialHue` decides the picture from a file only the shard has. Protocol stays 8; `EXTRACTOR_VERSION` stays 2 | servuo-plugins, module-uo |
|
||
| 6 | **DONE 2026-09-11, and not what this row said.** The measurement came first and changed the phase: a complete one-direction animation set is **174,453 frames / 281.5 MB**, not the ~119,000 estimated, and **the site displays still pictures** — so the deep keys and the bulk-fill switch were **not built** (§11.2, org lead 2026-09-11). What shipped is what the still-picture site was missing: the **73 bodies with no art at action 0 and real art deeper** (a horse at `body/820/a23`), the catalogue key carrying its action, the atlas join that reads it, and §4.10's per-body **action ceiling** — without which the fallback walk itself would serve **452 validated pictures of the next body**. Catalogue **1,022 → 1,095**; `EXTRACTOR_VERSION` 2 → 3; protocol stays 8 | servuo-plugins, module-uo |
|
||
| 7 | The atlas over the sidecar (§10); shared-filesystem requirement retired | module-uo |
|
||
| 8 | Admin surface, Import/Update, approve/reject, activity log | module-uo |
|
||
| 9 | Docs pass across five repos; live walk on the real rig | docs |
|
||
|
||
Phase 0 exists because §4 chose to call code that can take the shard down if it is wrong, and the
|
||
honest way to hold that choice is to try to break it on purpose — in the real host process, against
|
||
a client that has been patched — *before* building eight phases on top of it. The probes behind
|
||
§1.1 were run from PowerShell against a stock client; neither of those is the environment this will
|
||
actually run in.
|
||
|
||
**Phase 6's deep keys are not deferred to a later phase; they are out of the plan** until something
|
||
wants them. §11.2 has the measurement a future consumer would be choosing to pay and the two facts
|
||
it would have to answer (per-frame centres, and 750 round trips through a single-slot channel).
|
||
Nothing about reviving them needs a protocol change — that is what §5's key scheme bought.
|
||
|
||
Phase 4 sat **after** the catalogue rather than inside it on purpose. The catalogue was useful with
|
||
783 of its 787 bodies, the UOP reader is the one piece of genuinely new format work in this
|
||
protocol, and putting it on the critical path would have held up every website-side phase behind
|
||
it. Its acceptance test was not "it decodes" — it was that a gargoyle looks like a gargoyle,
|
||
checked by eye, because §4.3's whole point is that this failure mode produces confident, wrong
|
||
pictures. It does, and it was.
|
||
|
||
---
|
||
|
||
## 17. Decisions
|
||
|
||
Every item here is settled. Each is recorded because it changes numbers or obligations elsewhere
|
||
in the document.
|
||
|
||
1. **§4: settled 2026-09-10 — call ServUO's vendored `Ultima`,** with one exception added the same
|
||
day: **§4.3's UOP animation decoder** (built in phase 4; §17.9 widened its scope from the
|
||
player bodies to every body the legacy path cannot reach). The crash is confined to the `hasExtra: true` branch that only
|
||
`Gumps` reaches, and nothing here calls `Gumps`. **Phase 0 confirmed both halves in the real
|
||
host process** — nothing faulted on a path we call, and one `GetGump(2)` killed the shard
|
||
outright (§4.7).
|
||
2. **§4.4: settled 2026-09-10 — libgdiplus is a stated requirement** on Linux shard hosts, with
|
||
all three answers taken rather than one: it goes in `SHARD_PREREQS.md`, the installer's `doctor`
|
||
checks it, and its absence degrades to a named `NO_IMAGING` status instead of an error. Windows
|
||
hosts are unaffected.
|
||
3. **§5.1/§5.2: direction — settled 2026-09-10.** Player character bodies use index 0, everything
|
||
else index 1, direction is not in the key, and the player-body set is enumerated from
|
||
`Race.AllRaces` rather than hardcoded. Recorded here because it changes every count in §11.
|
||
Phase 0 reproduced the twelve stock ids and the six that do not decode, exactly.
|
||
4. **§13: the default audience — settled 2026-09-10. An asset inherits the audience of the page
|
||
that uses it.** A bestiary that is already anonymous keeps anonymous pictures; a staff-only
|
||
screen's art is staff-only. The operator can still set the policy explicitly, and the
|
||
`requireFeature` gate with its 404-not-403 behaviour is unchanged. The alternative — one flat
|
||
audience for all asset serving — was rejected because it necessarily disagrees with some page
|
||
that uses it, in one direction or the other, and the disagreement is silent either way.
|
||
5. **§4.5: the response to a malformed record — settled 2026-09-10, and now measured. Validate
|
||
before calling.** The extractor judges an index entry, and for statics the record behind it,
|
||
before handing the id to `Ultima`; a record that fails is reported absent and never decoded.
|
||
Chosen ahead of phase 0 over two alternatives — extracting in a child process (much stronger
|
||
containment, a much larger change to §2 and phase 1) and reversing §4 to write our own decoders
|
||
(weeks, per §2.1). Phase 0 then found the shape that settles it: the dangerous failure is not a
|
||
crash a child process would contain, it is a **wrong picture** that no containment strategy
|
||
would have caught, on **22,102 ids of a stock client**. See §4.5 for the checks and the
|
||
false-refusal measurement that says the boundary is in the right place.
|
||
6. **§3.2/§3.3/§3.4: the transport's three numbers and one shape — settled 2026-09-10, phase 1.**
|
||
A **512 KiB** batch budget under a **1 MiB** inbound line cap, with the factor of two load-bearing
|
||
(a page always admits its first item, so it may overshoot by one). Flow control is enforced
|
||
**on the shard**, as a single slot answering `bridge.busy`, rather than serialised in the
|
||
sidecar or left to the website as a convention — the bound belongs where the memory is. And one
|
||
**paging envelope** (`more` / `cursor` / `cut`) is defined now, with `assets.sources` as its
|
||
first user, rather than left for whichever family pages first to invent.
|
||
7. **§9: the cliloc pipeline's four shapes — settled 2026-09-10, phase 2.** All four were put to
|
||
the org lead before any of it was written, and two of them departed from the recommendation:
|
||
|
||
- **The wire carries rows, not bytes.** `[{n, f, t}]` inside the paging envelope, rather than a
|
||
reconstituted plain-binary file in base64. The container decompresses into *records*; the
|
||
plain layout is a file shape that would have had to be synthesised on the shard and re-parsed
|
||
on the website, at a 33% base64 premium, to reach the same place.
|
||
- **The shard omits blank entries** — ~56,000 of 123,490 — because the website discards them at
|
||
import anyway. The transfer halves and nothing observable changes.
|
||
- **The bridge always wins; the file upload is deprecated** (org lead, departing from the
|
||
proposed `auto`/`bridge`/`file` setting). There is no version of "which source?" an operator
|
||
benefits from answering, so there is no setting to answer it with. A file remains the source
|
||
only where there is no shard link, plus a one-off explicit path.
|
||
- **Import is admin-triggered on the bridge** (org lead, departing from the proposed boot
|
||
refresh). Boot does not call the shard at all — see §9.2 for why that is the right trade and
|
||
what it costs.
|
||
|
||
8. **§4.8: the catalogue is 787 bodies, not 1,144 — measured 2026-09-10, phase 1.** Recorded here
|
||
because it changes §11's sizing, phase 3's scope and phase 4's, and because of *how* it was
|
||
found: the animation path has §4.5's shared-buffer defect too, and 357 ids with no art were
|
||
returning the previously-decoded body's bitmap. The elf ghosts moved from "decodes" to "no art"
|
||
in §5.2 for the same reason, taking phase 4's set from six player bodies to eight.
|
||
|
||
10. **§11.1: phase 5's four, settled 2026-09-11.** Put to the org lead after the client was measured
|
||
and before the reader was written, because the first measurement changed what the risk was:
|
||
|
||
- **Ingest warms; the route only serves.** A page renders the pictures already on disk and
|
||
leaves out the ones that are not; fetching happens behind it, on a timer, from the keys the
|
||
site's own rows name. Chosen over fetching on first request, on one number: the asset plane
|
||
serves **one request at a time** (§3.2), so a URL that fetched would let any anonymous visitor
|
||
walk 49,152 ids × 3,000 hues through that single slot and park an operator's own import behind
|
||
it. Warming from the site's own data has no such surface — the ceiling is the number of
|
||
distinct (item, hue) pairs the shard has already told the site about.
|
||
- **Staleness is a per-row catalogue id, not a manifest.** A client patch changes the shard's
|
||
`catalog` fingerprint and a restart does not, so "is this out of date?" is a column
|
||
comparison. The alternative — a `static` manifest family enumerating 39,189 rows with hashes —
|
||
would cost a 34-second scan of the whole art file per Update to answer a question about maybe
|
||
three hundred pictures, and would re-fetch art nobody looks at any more. Lazy costs nothing
|
||
for the ones nobody wants.
|
||
- **The pictures appear on the marketplace and the character sheet**, the two places the data
|
||
already existed and rendered as `id 1234, hue 33`. That gives the phase an acceptance test
|
||
checkable by eye, which §11.1 says is the only kind that catches this failure mode.
|
||
- **`Files.CacheData` goes off process-wide** at asset init, rather than copying each bitmap
|
||
before hueing. See §11.1: the copy does not solve the 74 MB retention, and the ordinary copy
|
||
constructor throws on ARGB1555 anyway.
|
||
|
||
9. **§4.3/§5.2: phase 4's four, settled 2026-09-11.** Put to the org lead after the packages were
|
||
opened and before the reader was written, because the first measurement changed what the phase
|
||
was worth:
|
||
|
||
- **The UOP fallback applies to every body, not only to player bodies.** It is the same rule
|
||
either way — "legacy first, UOP when it has nothing" — and restricting it would have needed an
|
||
extra filter to *refuse* art the client has. 787 → 1,022 rather than 787 → 789.
|
||
- **Ghost ids leave the player-body set** (org lead, over keeping them with a new status or
|
||
leaving them as `absent`). No client has art for any of the six, so listing them advertised
|
||
keys that cannot exist. The shard is still asked rather than told — the question is now the
|
||
living pair per race.
|
||
- **The UOP path gets its own PNG encoder** rather than `Bitmap.Save`, which is what §4.4's
|
||
sentence about this reader and `System.Drawing` was always promising. Phase 3's `ToPng` is
|
||
untouched: its input comes from a decoder that needs GDI+ to produce a pixel, so encoding it
|
||
without GDI+ buys nothing.
|
||
- **A host without libgdiplus keeps the flat `NO_IMAGING` refusal** (org lead, over serving the
|
||
~238 bodies that no longer need it). One answer beats a quarter-full catalogue nobody can tell
|
||
from a complete one. The door §4.4 wanted open stays open; phase 4 simply does not walk
|
||
through it.
|
||
|
||
11. **§11.2: phase 6 stopped being the phase it was planned as — settled 2026-09-11.** Put to the
|
||
org lead with the measurement in hand, and the answer removed most of the phase:
|
||
|
||
- **The deep animation keys and the bulk-fill switch are not built.** The org lead's own scope
|
||
note — "I hadn't planned to use the animations on the site, just static images" — is the
|
||
whole argument: §11 justified them with a future project rather than with anything on this
|
||
site, and 281.5 MB of store, a warm pass and an admin surface for a consumer that does not
|
||
exist yet would all have been guesses. `body/<id>/a<n>/f<k>` stays named in §5 and refused in
|
||
practice; reviving it is a reader and a store, not a protocol change.
|
||
- **The catalogue falls back to the first action that has art, and the key names that action.**
|
||
73 bodies on a stock client have nothing at action 0 and real art deeper — body 820's action
|
||
23 is a horse — and they rendered as text on the bestiary. Keeping the `a0` spelling for
|
||
them was rejected for the reason §5 gives: a key that lies about its content is this
|
||
protocol's recurring failure, not a shortcut.
|
||
- **Only that one exception**, over a per-body pose override that would let an operator pick a
|
||
better-looking action. One rule with one mechanical exception; the setting can come the day
|
||
somebody wants a specific creature to look different.
|
||
- **§4.10's action ceiling ships in the same phase**, because the fallback is the walk that
|
||
would otherwise hit it: one action past a body's band is 643 validated pictures and 452
|
||
byte-identical copies of the next body.
|
||
|