diff --git a/link/v8.md b/link/v8.md new file mode 100644 index 0000000..2964ad7 --- /dev/null +++ b/link/v8.md @@ -0,0 +1,660 @@ +# Asset Bridge (Protocol 8) — client assets without UOFiddler + +**Call it the Asset Bridge.** That is the name for this work everywhere — commits, PR titles, +branches (`feat/asset-bridge-p`), and conversation. The protocol number is **8** and this file is +`docs/link/v8.md`. + +**Status:** Design of record. Approved in principle 2026-09-09 (architecture, asset scope, +built-in cliloc decoder, atlas cleanup); refined 2026-09-10 (one direction not five, ServUO's own +decoders, the UOP reader for player bodies, the libgdiplus requirement). §17 lists what is still +open. +**Supersedes the manual half of:** [`../website/UOFIDDLER.md`](../website/UOFIDDLER.md), +[`../website/CLILOCS.md`](../website/CLILOCS.md) §Converting, +[`../website/SPAWN_ATLAS.md`](../website/SPAWN_ATLAS.md) §Artwork and §Configuring the tree. + +Two features on this platform read data that only exists inside a UO client, and today both reach +the site by hand: the operator installs UOFiddler, converts `Cliloc.enu` on their own desktop, +exports sprites one at a time from a GUI, hand-writes a slug→filename JSON map, and copies the +result to the server. A third — the spawn atlas — avoids UOFiddler but pays a different price: the +**website** must be able to read the shard's ServUO tree directly, over a bind mount or a shared +volume. + +This protocol deletes all three arrangements. The shard already has everything, and the bridge +already goes to the website. + +--- + +## 1. The premise, which turns out to be free + +**A ServUO shard cannot boot without a UO client installation.** It reads maps, statics, tiledata +and multis out of `.mul`/`.uop` files, and `Config/DataPath.cfg` is where an operator declares +where those live — *required* on Linux, auto-detected from the registry on Windows. At runtime the +resolved directories sit in `Server.Core.DataDirectories`, a public static the plugin can read on +any shard, with no new configuration and nothing for an operator to set up. + +So the files the operator has been converting on their desktop are already on the shard host, in a +directory the shard already knows the path of, in a process the bridge already runs inside. + +Everything below follows from that. + +### 1.1 What was measured, not assumed + +Against this machine's ServUO 57.4 tree (`C:\Users\colby\Desktop\ServUO`) and client +(`D:\Games\Electronic Arts\Ultima Online Classic`, 3.5 GB), loading **ServUO's own +`Ultima.dll`** — the assembly `overlay/Scripts/Scripts.csproj:39` already carries a +`` to: + +| Call | Result | +|---|---| +| `Art.GetStatic(0…16383)` | 16,384 decoded, 0 errors | +| `Art.GetStatic(16384…65535)` | 32,766 decoded, 1 empty, 16,385 clean out-of-range errors | +| `Art.GetLand(0…16383)` | 16,384 decoded, 0 errors | +| `Animations.GetAnimation(0…2047, 0, 1)` | **1,144** bodies with a decodable first frame, 904 empty, 0 errors | +| `Hues.GetHue(33)` | loads | +| `Bitmap.Save(…, Png)` | 852-byte PNG from one creature frame | +| `Gumps.GetGump(2)` | **hard crash** — `AccessViolationException`, process exit `0xC0000005` | +| `new StringList("enu", "Cliloc.enu")` | throws — `Non-negative number required` | + +Two of those rows are load-bearing and are dealt with in §4 and §9. The rest say the same thing: +**most of the extraction this protocol needs is already implemented, already compiled, and already +referenced by the plugin's own build.** + +Depth, for §11's sizing: body 400 (human male) has **35 actions × 5 directions = 1,050 frames**. +One body. §5.1 cuts that by exactly 5×. + +--- + +## 2. Architecture: the shard extracts, the sidecar forwards, the website decides + +``` +UO client files (operator's own, on the shard host) + │ read by the plugin, off the Core thread + ▼ +ServUO shard (servuo-plugins/) ← decodes; resolves body ids; hashes + │ loopback JSON, request/reply, one batch outstanding at a time + ▼ +uo-link sidecar (link/) ← forwards bytes; decides nothing + │ REST, bearer-token auth, X-UOLink-Version: 8 + ▼ +website (module-uo/) ← stores, names, gates, serves +``` + +This is deliberately the *only* arrangement that keeps +[the bridge's standing rules](PLAN.md) intact: + +- **The sidecar stays a dumb forwarder.** It moves opaque assets and decides nothing about them — + no audience, no projection, no capability advertisement. Putting the decoders in Rust would have + meant the sidecar deciding what an asset *is*, on top of re-deriving in Rust what is already + compiled next door in C#. +- **Access control stays on the website**, which has the auth machinery and the admin forms. +- **The shard is still never network-reachable.** Nothing here opens a port; the plugin answers + requests on the connection it already dialled out on. + +### 2.1 Why not the sidecar, and why not the operator's desktop + +A Rust extractor in the sidecar would need ports of: the Mythic cliloc decompressor, `FileIndex` +(including UOP), the ARGB1555 run-length frame decoder, `Body.def`/`Bodyconv.def` translation, +`Hues.mul`, and a PNG encoder — weeks of work to re-derive what §1.1 shows already runs. It also +cannot do §8: resolving a creature slug to a body id requires being inside ServUO. + +Automating on the operator's desktop (shipping the converter with the installer) removes UOFiddler +but keeps a manual step and still cannot do §8. It was considered and rejected. + +--- + +## 3. The transport, and the three traps in it + +### 3.1 Assets go over the request/reply path, never the event path + +`link/sidecar/src/app.rs:122` persists **every** non-`pong` event into the SQLite store *and* +broadcasts it to every WebSocket subscriber. An asset stream on that path would grow the sidecar's +store without bound and fan megabytes out to every connected client, forever. + +`rpc.rs`'s `try_route` consumes a correlated reply and `continue`s **before** either of those +happens. So an asset batch is a reply, not an event. This is not a new mechanism — it is the one +`char.request`, `account.roster` and `vendor.snapshot` already use. + +### 3.2 One batch outstanding, always + +`BridgeLink.Emit()` enqueues onto a **bounded drop-oldest** queue (`Bridge.QueueCap`, default +10,000). It counts **lines, not bytes** — a design that is correct for live events and dangerous +for bulk transfer, because 10,000 queued 200 KB replies is 2 GB of shard memory. + +The rule that makes this safe is flow control, not a bigger queue: **the website requests batch +*n+1* only after batch *n* has arrived.** Queue depth stays at approximately one. A dropped or +lost reply simply times out and the batch is re-requested, which is safe because reading a client +file is idempotent and has no world side effects. + +### 3.3 The size ceilings are already fixed, and one of them is missing + +| Limit | Value | Where | +|---|---|---| +| Sidecar waits for a shard reply | **10 s** | `rpc.rs` `REPLY_TIMEOUT` | +| Website waits for the sidecar | **12 s** | `module-uo/server/utils/uoLinkClient.js` `TIMEOUT_MS` | +| Sidecar → shard line | 1 MiB | `BridgeLink.cs:283` | +| **Shard → sidecar line** | **none** | `shard.rs` uses `read_line` unbounded | + +The first two bound a batch: it must decode, encode, serialise and cross the wire inside ten +seconds. The last is a gap this protocol must close — an unbounded `read_line` facing a component +that is now deliberately sending large lines is a memory-exhaustion shape we would be inventing +ourselves. **Protocol 8 adds an explicit inbound line cap to the sidecar**, set above the largest +legal batch and rejecting rather than buffering past it. + +Batches are therefore sized by bytes, not by count, with the emitter cutting a batch short when it +would exceed the cap. Base64 costs 33%; the budget must be stated in encoded bytes. + +--- + +## 4. The decoders are ServUO's own — decided, and the crash is narrower than it looked + +**We call ServUO's vendored `Ultima` (decided 2026-09-10).** No decoders are reimplemented. +`overlay/Scripts/Scripts.csproj:39` already references the project, so the art half of this protocol +costs plumbing rather than pixel code, and only §9's cliloc decompressor is written from scratch. + +The reason that is safe, rather than merely cheap, is a distinction §1.1 did not draw at first. + +### 4.1 The crash lives on one code path, and nothing we call uses it + +`Gumps.GetGump(2)` does not fail — it **corrupts the process**: `AccessViolationException`, exit +`0xC0000005`. That is a corrupted-state exception, uncatchable by an ordinary `try/catch` on .NET +Framework 4.8, so in-process on a live shard it is a crash with players on it. That much is +alarming, and on its own it looked like an argument against using this library at all. + +It is not, because of how the three decoders construct their `FileIndex`: + +| Decoder | UOP file | `hasExtra` | Probed | +|---|---|---|---| +| `Art` | `artLegacyMUL.uop` | **false** | 49,150 statics + 16,384 land tiles, **0 faults** | +| `Animations` | *none — legacy `anim*.mul` only* | — | 1,144 bodies, **0 faults** | +| `Gumps` | `gumpartLegacyMUL.uop` | **true** | **faults on the second id** | + +`FileIndex.cs`'s own comment says the extra-field handling exists *for* `gumpartlegacy.uop` — it is +the one UOP layout carrying an extra field, and `hasExtra: true` is the branch written to cope with +it. **Gumps is the only caller that sets it.** So the fault is not a general fragility in this +library's `unsafe` code; it is a bug on a branch that exactly one decoder reaches, and that decoder +is already out of scope (§11). + +The rule this turns into is a safety rule, not a preference: **nothing in this protocol calls +`Ultima.Gumps`.** Adding gump art later means fixing or replacing that path first, deliberately, +not discovering it in production. + +### 4.2 What the decision accepts + +Three costs come with it, all known and none of them blocking: + +1. **Six of the twelve stock player-character bodies have no art on this path** — both human ghosts + and every gargoyle body (§5.2). `Animations` never reads `AnimationFrame*.uop`. **This one is + not merely accepted: §4.3 adds a decoder for it**, because it is the player character and the + scope says player models. +2. **`System.Drawing` is a hard dependency, in the decode and not just the encode.** `Frame` + writes ARGB1555 straight through a `LockBits` pointer, so a Linux shard needs **libgdiplus** to + read a sprite at all. That is a stated prerequisite — §4.4. +3. **We inherit whatever `Ultima` a given ServUO vendors**, which can change under a shard upgrade. + `EXTRACTOR_VERSION` (§7) is the mitigation: it already counts as drift, so a shard whose library + changed re-derives on the next import. + +The residual risk that remains is a patched or custom client tripping an out-of-bounds read on a +path we *do* call. §16's phase 0 is where that gets exercised rather than assumed. + +### 4.3 One decoder we do write: UOP animation, for the player bodies + +The six missing bodies are the player character, and the scope is player models, so they get a +decoder rather than a caveat. It is deliberately the **narrowest possible** addition: a reader for +`AnimationFrame*.uop`, used *only* for bodies the legacy path cannot resolve. Everything the +vendored code can already decode keeps going through the vendored code. + +This client ships `AnimationFrame1/2/3/4/6.uop` (107, 118, 253, 115 and 24 MB) plus +`AnimationSequence.uop`. ServUO's `FileIndex` already contains a UOP reader — but `Animations` +never constructs one, and the UOP animation *payload* is its own format rather than a repackaged +mul record, so wiring the existing `FileIndex` in is not sufficient. Two license-compatible +references exist to work from: ClassicUO's animation loader (GPL-3, and we are GPL-3-or-later) and +UOFiddler 4.22 (Beerware, already established in §9). + +**The trap this must not fall into, and it is why the phase exists at all.** `Bodyconv.def` maps +gargoyle 666 to `anim5`, and `BodyConverter.Convert` faithfully returns fileType 5 — where the +client has nothing. Asking the *other* anim files for index 666 does not fail. It returns 175 +decodable action/direction combinations of **a giant spider**, because something unrelated occupies +that index in `anim2.mul`; fileTypes 3 and 4 return misaligned colour fragments. Rendered and +confirmed. + +So the extractor takes `BodyConverter.Convert`'s answer and, if that yields nothing, reports +nothing. **It must never sweep file types looking for a hit** — that does not find missing art, it +silently puts a spider on the gargoyle page, and no error is raised anywhere. A "0 rows" outcome is +correct behaviour; a plausible wrong picture is the failure this protocol most needs to avoid, +because nothing downstream can detect it. + +### 4.4 Requirement: a Linux shard host needs libgdiplus + +**Stated prerequisite, not a soft recommendation.** ServUO targets `net48`, so on Linux it runs under +Mono, and Mono's `System.Drawing` is a thin layer over **libgdiplus**. §4.2 put `System.Drawing` in +the decode path, so without that library a Linux shard cannot extract art at all — the cliloc table +(§9) and the atlas files (§10) are unaffected, since neither touches pixels. + +**Windows shard hosts need nothing.** `System.Drawing` ships with .NET Framework. This is a +Linux-only prerequisite and most shards will never read this section. + +| Host | Get it with | +|---|---| +| Debian / Ubuntu | `sudo apt-get install libgdiplus` — in Debian since bullseye (6.0.4) and bookworm/trixie (6.1), and in Ubuntu universe | +| Fedora / RHEL | `sudo dnf install libgdiplus` (EPEL or the Mono repository) | +| Docker | `RUN apt-get update && apt-get install -y libgdiplus` in the shard image | +| Alpine, or a distro with no package | Build from source — see the repository below. This is the awkward case and is worth avoiding by choosing a Debian-based image | + +Upstream is , with the Mono project's own page at +. + +**One thing to know before depending on it: that repository was archived in March 2025 and is +read-only.** Distributions still package and patch it, so `apt-get install libgdiplus` is a normal, +supported thing to do today — but upstream is not maintained. It is the strongest long-term argument +for eventually moving extraction off `System.Drawing`, and phase 4's UOP reader (§4.3) is written +without it precisely so that door stays open. It does not change the decision now. + +**How its absence must present.** Never a stack trace and never a 500. Missing libgdiplus is a +named, actionable outcome in the same family as the cliloc reader's `COMPRESSED`: + +``` +status: unavailable +code: NO_IMAGING +reason: This shard host cannot render images — Mono's System.Drawing needs libgdiplus. + Install it (apt-get install libgdiplus) and re-run the import. Cliloc and + atlas import are unaffected. +``` + +The installer's `doctor` checks for it and reports it alongside its other host checks, so an +operator learns about this while setting the shard up rather than from an empty bestiary weeks +later. + +--- + +## 5. Addressing: one key for every asset + +Every asset the bridge can serve is named by a single string key, and the key is the cache key, +the hash key, the filename stem and the manifest row id: + +``` +static/3922 one item graphic +static/3922/h33 the same graphic, hue 33 applied +land/3 one land tile +body/34/a0 creature body 34, action 0, first frame +body/400/a0/f0..f9 human male, action 0, all ten frames +cliloc/enu the whole converted string table (not an image) +tree/Spawns/Trammel.xml a ServUO tree file (§10) +``` + +Three properties this shape buys: + +- **Hue is part of the key, not a transform.** `itemId` and `hue` are already on the wire together + (`BridgeMarket.cs:582`, `BridgeProfile.cs:314`), so a marketplace listing already knows the exact + key for its own picture. Applying hues website-side would mean shipping `Hues.mul` semantics into + Node for no gain. +- **Depth is expressible without being mandatory.** `body/400/a0` and `body/400/a0/f0..f9` are the + same addressing scheme at two depths, which is what lets §11 bulk-import thumbnails and fetch full + animations on demand without a second protocol. +- **Nothing in the key is client-version-specific**, so a client patch changes an asset's *bytes*, + not its name — which is what makes §7's delta work. + +### 5.1 There is no direction segment, because only one direction is wanted + +Bodies are stored in **five** directions and the client mirrors three of them to reach eight. Only +one is needed here, so **direction is fixed by the extractor and is not part of the key**. Leaving +it in would advertise a choice nobody is going to vary and would five-fold every count in §11 for +nothing. + +**Which one depends on whether the body is a player character:** + +| Body | Direction | Why | +|---|---|---| +| A player character body | **0** — head-on, facing the viewer | A character is a portrait; it should look at you | +| Everything else | **1** — front three-quarter | The view that actually reads as a creature (see the caveat below) | + +Which index is which was determined by **rendering all five** for a human, a wolf and a dragon +rather than from a table, because the answer is not obvious and the small-thumbnail version of the +same test suggested the exact opposite: + +| Index | View | +|---|---| +| **0** | **Head-on, facing the viewer** — face, chest and front legs visible | +| 1 | Front three-quarter | +| 2 | Full side profile | +| 3 | Rear three-quarter | +| 4 | Directly away — back of the head, and a quadruped's tail toward the camera | + +The caveat the render made obvious is what produced the split: **index 0 is the least legible view +for four-legged and long-bodied creatures.** A wolf seen head-on is a dark blob; the same wolf at +index 1 is unmistakably a wolf, which is also why UOFiddler's own thumbnail list picks that view. A +humanoid has no such problem — it reads fine head-on, and head-on is what a character portrait +wants. + +Both indices stay **configuration values** (defaulting to 0 and 1), so changing the catalogue's mind +later is a setting and a re-import, not a protocol change. + +### 5.2 "Player character body" is asked of the shard, never hardcoded + +`Server.Race.AllRaces` gives every registered race, and each carries `MaleBody`, `FemaleBody`, +`MaleGhostBody` and `FemaleGhostBody`. The plugin enumerates those four ids per race and that set — +nothing else — takes index 0. On stock ServUO 57.4 that is twelve ids: + +| Race | Male | Female | Male ghost | Female ghost | +|---|---|---|---|---| +| Human | 400 | 401 | 402 | 403 | +| Elf | 605 | 606 | 607 | 608 | +| Gargoyle | 666 | 667 | **695** | **694** | + +This is the §8 argument again in miniature: only code inside ServUO can answer it, and asking is +the only thing that works on a shard with a custom race. Two details make the case that a +hardcoded list would have been wrong — `RaceDefinitions.cs` passes the gargoyle's ghost bodies in +the **opposite order** to the other two races (695 male, 694 female), and a shard that calls +`RegisterRace` adds ids no table of ours would contain. + +**Half of that set does not decode with ServUO's vendored library.** Measured: + +| Decodes | Does not | +|---|---| +| Human male/female (400, 401) | **Human ghosts (402, 403)** | +| Elf male/female (605, 606) | **Every gargoyle body (666, 667, 694, 695)** | +| Elf ghosts (607, 608) | | + +Six of twelve, including a whole playable race. These are UOP-only and **§4.3 adds a decoder for +them** — verified genuinely absent from the legacy files rather than merely mis-addressed, by +rendering what the other anim files hold at those indices and finding spiders. + +Until that phase lands, and on any client that lacks them entirely, the catalogue must **not** treat +a missing player body as an error. It is the expected answer for half the set, and a status screen +that flags six failures on every import teaches an operator to ignore it. +`shard_spawn_creatures.art` staying NULL remains a first-class state everywhere it is consumed, +which it already is. + +--- + +## 6. The manifest, and what the two buttons actually do + +Two stages, which is where **Import** and **Update** come from. + +**Stage 1 — the source gate.** The shard reports a manifest of the client files themselves: +size, mtime and content hash of `Cliloc.enu`, `anim*.idx`/`anim*.mul`, `art.mul`/`artidx.mul`, +`Body.def`, `Bodyconv.def`, `Hues.mul`. Unchanged since the last import, and nothing else happens. +This is the same hash gate the spawn atlas and the cliloc table already use, and for the same +reason: the normal case is a restart that changed nothing, and it must cost nothing. + +`anim.mul` is 195 MB and `art.mul` is 148 MB, so the gate is **(size, mtime) first, content hash +only when those differ** — a full hash of 343 MB on every status poll would make the admin panel +feel broken. + +**Stage 2 — the asset manifest.** For the working set (§11), the shard streams +`[{ key, sha256, bytes }]` — no pixels. The website diffs that against what it holds and requests +**only the keys whose hash changed**. + +- **Update** = stage 1, then stage 2, then fetch the diff. +- **Import** = the same path with the diff skipped and every key fetched. +- **A key that has vanished** from the manifest is staged for review, never applied silently — + the same rule, and the same reasoning, as a vanished cliloc source or a disappearing atlas + facet. An unmounted volume and a deliberate client downgrade look identical from here. + +Clilocs are the exception and stay a **whole-table replace** whenever the file hash changes: +the measured cost is 663 ms for 67,496 rows, so per-entry deltas would be complexity bought for +nothing. + +--- + +## 7. The parser version applies here too + +`spawnAtlasSource.js` carries `PARSER_VERSION` (currently 5) and the cliloc source carries its own, +both counted as drift so that a corrected parse reaches an install whose files never change. The +asset pipeline inherits the rule and needs it more, not less: a fixed hue application or a +corrected frame offset changes the bytes we derive from files that are byte-identical. + +**`EXTRACTOR_VERSION` lives in the plugin**, because the plugin is what derives the bytes, and it +is folded into stage 1's gate. Bumping it makes every asset drift, which is correct. + +--- + +## 8. Body ids: the part only the shard can do + +The atlas knows creatures by **slug**, derived from type names in `Spawns/*.xml`. The client knows +them by **body id**. Nothing in the ServUO tree declares the mapping as data — today an operator +bridges it by grepping `Scripts/Mobiles/Normal/.cs` for `Body =`, which appears variously as +a decimal, as hex (`0xD1`), as `Utility.RandomList(35, 36)`, and as an `m_IDs[]` table. + +Inside ServUO the problem does not exist. `BridgeWorld.cs:350` already does exactly the required +thing for a different feature: + +```csharp +var type = ScriptCompiler.FindTypeByName(name, true); +var creature = Activator.CreateInstance(type) as BaseCreature; +``` + +Construct, read `creature.Body.BodyID`, `Delete()`. Authoritative, no source parsing, and correct +for custom creatures a grep would never find. + +**This pass must run on the Core thread** — it constructs and deletes mobiles, which is world +mutation — while the decode in §4 must run **off** it. That split is the one genuinely new +threading shape in this protocol, and it is why slug→body resolution is its own request kind with +its own (small) batch size rather than a step inside asset extraction. + +Constructing arbitrary creature types has side effects: constructors pack items, set skills, start +timers. The mitigations are per-type `try`/`catch`, immediate `Delete()`, small batches, and the +fact that the whole pass is admin-triggered rather than something that runs at boot. + +--- + +## 9. The cliloc decompressor is ours now + +Every modern client ships `Cliloc.*` in the Mythic compressed container — this machine's +`Cliloc.enu` is 4,989,921 bytes beginning `E8 79 67 8E`, high byte `0x8E`. ServUO's bundled +`Ultima.StringList` implements only the plain layout and throws on it (§1.1), which is also why +the shard's own `VendorSearch.GetItemName` is already inert. + +**UOFiddler is released under the Beerware licence**, so porting its decompressor into our +GPL-3.0-or-later tree is clean. It lands in the overlay as ordinary C# — the **only** decoder +Protocol 8 writes rather than calls (§4) — and from that point: + +- No operator installs UOFiddler. +- No operator runs `dotnet build` on a converter. +- No operator copies a 5 MB file to a server. +- `website/server/tools/cliloc-export/` is retired, and `UOFIDDLER.md` is deleted rather than + rewritten. + +**What survives untouched is the `custom/` overlay mechanism.** Shard-added items carry cliloc ids +no client table has, and ServUO has no server-side notion of a custom cliloc — that is a real gap +in the *game*, not an artefact of the manual pipeline, and `CLILOCS.md`'s reasoning for it stands. +The base table now arrives over the bridge; overlays still come from a directory the site reads. +Measured on the live shard: 16,434 cliloc ids referenced by the script tree, 37 absent from stock. + +--- + +## 10. The atlas stops needing a shared filesystem + +Today `SPAWN_ATLAS.md` requires the **website** to read the ServUO tree — "same host, a bind mount, +or a shared volume". That is the one place the platform's own rule (only the sidecar bridges the +shard) is broken, and it is broken by the component that faces the internet. + +The same transport closes it. `spawnAtlasSource.js` already labels every file it reads with a +portable key: + +| Label | Count (stock 57.4) | +|---|---| +| `Data/Regions.xml` | 1 | +| `Data/Locations/*.xml` | 6 | +| `Spawns/*.xml` | 13, ~10.5 MB | +| `Config/ChampionSpawns.xml` | 1 | +| `Data/Decoration/**` | tree | + +So the shard serves `tree/