diff --git a/link/v8.md b/link/v8.md new file mode 100644 index 0000000..8bf247e --- /dev/null +++ b/link/v8.md @@ -0,0 +1,478 @@ +# Protocol 8 — Client assets over the bridge + +**Status:** Design of record. Approved in principle 2026-09-09 (architecture, asset scope, +built-in cliloc decoder, atlas cleanup); §17 lists what is still open. +**Supersedes the manual half of:** [`../website/UOFIDDLER.md`](../website/UOFIDDLER.md), +[`../website/CLILOCS.md`](../website/CLILOCS.md) §Converting, +[`../website/SPAWN_ATLAS.md`](../website/SPAWN_ATLAS.md) §Artwork and §Configuring the tree. + +Two features on this platform read data that only exists inside a UO client, and today both reach +the site by hand: the operator installs UOFiddler, converts `Cliloc.enu` on their own desktop, +exports sprites one at a time from a GUI, hand-writes a slug→filename JSON map, and copies the +result to the server. A third — the spawn atlas — avoids UOFiddler but pays a different price: the +**website** must be able to read the shard's ServUO tree directly, over a bind mount or a shared +volume. + +This protocol deletes all three arrangements. The shard already has everything, and the bridge +already goes to the website. + +--- + +## 1. The premise, which turns out to be free + +**A ServUO shard cannot boot without a UO client installation.** It reads maps, statics, tiledata +and multis out of `.mul`/`.uop` files, and `Config/DataPath.cfg` is where an operator declares +where those live — *required* on Linux, auto-detected from the registry on Windows. At runtime the +resolved directories sit in `Server.Core.DataDirectories`, a public static the plugin can read on +any shard, with no new configuration and nothing for an operator to set up. + +So the files the operator has been converting on their desktop are already on the shard host, in a +directory the shard already knows the path of, in a process the bridge already runs inside. + +Everything below follows from that. + +### 1.1 What was measured, not assumed + +Against this machine's ServUO 57.4 tree (`C:\Users\colby\Desktop\ServUO`) and client +(`D:\Games\Electronic Arts\Ultima Online Classic`, 3.5 GB), loading **ServUO's own +`Ultima.dll`** — the assembly `overlay/Scripts/Scripts.csproj:39` already carries a +`` to: + +| Call | Result | +|---|---| +| `Art.GetStatic(0…16383)` | 16,384 decoded, 0 errors | +| `Art.GetStatic(16384…65535)` | 32,766 decoded, 1 empty, 16,385 clean out-of-range errors | +| `Art.GetLand(0…16383)` | 16,384 decoded, 0 errors | +| `Animations.GetAnimation(0…2047, 0, 1)` | **1,144** bodies with a decodable first frame, 904 empty, 0 errors | +| `Hues.GetHue(33)` | loads | +| `Bitmap.Save(…, Png)` | 852-byte PNG from one creature frame | +| `Gumps.GetGump(2)` | **hard crash** — `AccessViolationException`, process exit `0xC0000005` | +| `new StringList("enu", "Cliloc.enu")` | throws — `Non-negative number required` | + +Two of those rows are load-bearing and are dealt with in §4 and §9. The rest say the same thing: +**most of the extraction this protocol needs is already implemented, already compiled, and already +referenced by the plugin's own build.** + +Depth, for §11's sizing: body 400 (human male) has **35 actions × 5 directions = 1,050 frames**. +One body. + +--- + +## 2. Architecture: the shard extracts, the sidecar forwards, the website decides + +``` +UO client files (operator's own, on the shard host) + │ read by the plugin, off the Core thread + ▼ +ServUO shard (servuo-plugins/) ← decodes; resolves body ids; hashes + │ loopback JSON, request/reply, one batch outstanding at a time + ▼ +uo-link sidecar (link/) ← forwards bytes; decides nothing + │ REST, bearer-token auth, X-UOLink-Version: 8 + ▼ +website (module-uo/) ← stores, names, gates, serves +``` + +This is deliberately the *only* arrangement that keeps +[the bridge's standing rules](PLAN.md) intact: + +- **The sidecar stays a dumb forwarder.** It moves opaque assets and decides nothing about them — + no audience, no projection, no capability advertisement. Putting the decoders in Rust would have + meant the sidecar deciding what an asset *is*, on top of re-deriving in Rust what is already + compiled next door in C#. +- **Access control stays on the website**, which has the auth machinery and the admin forms. +- **The shard is still never network-reachable.** Nothing here opens a port; the plugin answers + requests on the connection it already dialled out on. + +### 2.1 Why not the sidecar, and why not the operator's desktop + +A Rust extractor in the sidecar would need ports of: the Mythic cliloc decompressor, `FileIndex` +(including UOP), the ARGB1555 run-length frame decoder, `Body.def`/`Bodyconv.def` translation, +`Hues.mul`, and a PNG encoder — weeks of work to re-derive what §1.1 shows already runs. It also +cannot do §8: resolving a creature slug to a body id requires being inside ServUO. + +Automating on the operator's desktop (shipping the converter with the installer) removes UOFiddler +but keeps a manual step and still cannot do §8. It was considered and rejected. + +--- + +## 3. The transport, and the three traps in it + +### 3.1 Assets go over the request/reply path, never the event path + +`link/sidecar/src/app.rs:122` persists **every** non-`pong` event into the SQLite store *and* +broadcasts it to every WebSocket subscriber. An asset stream on that path would grow the sidecar's +store without bound and fan megabytes out to every connected client, forever. + +`rpc.rs`'s `try_route` consumes a correlated reply and `continue`s **before** either of those +happens. So an asset batch is a reply, not an event. This is not a new mechanism — it is the one +`char.request`, `account.roster` and `vendor.snapshot` already use. + +### 3.2 One batch outstanding, always + +`BridgeLink.Emit()` enqueues onto a **bounded drop-oldest** queue (`Bridge.QueueCap`, default +10,000). It counts **lines, not bytes** — a design that is correct for live events and dangerous +for bulk transfer, because 10,000 queued 200 KB replies is 2 GB of shard memory. + +The rule that makes this safe is flow control, not a bigger queue: **the website requests batch +*n+1* only after batch *n* has arrived.** Queue depth stays at approximately one. A dropped or +lost reply simply times out and the batch is re-requested, which is safe because reading a client +file is idempotent and has no world side effects. + +### 3.3 The size ceilings are already fixed, and one of them is missing + +| Limit | Value | Where | +|---|---|---| +| Sidecar waits for a shard reply | **10 s** | `rpc.rs` `REPLY_TIMEOUT` | +| Website waits for the sidecar | **12 s** | `module-uo/server/utils/uoLinkClient.js` `TIMEOUT_MS` | +| Sidecar → shard line | 1 MiB | `BridgeLink.cs:283` | +| **Shard → sidecar line** | **none** | `shard.rs` uses `read_line` unbounded | + +The first two bound a batch: it must decode, encode, serialise and cross the wire inside ten +seconds. The last is a gap this protocol must close — an unbounded `read_line` facing a component +that is now deliberately sending large lines is a memory-exhaustion shape we would be inventing +ourselves. **Protocol 8 adds an explicit inbound line cap to the sidecar**, set above the largest +legal batch and rejecting rather than buffering past it. + +Batches are therefore sized by bytes, not by count, with the emitter cutting a batch short when it +would exceed the cap. Base64 costs 33%; the budget must be stated in encoded bytes. + +--- + +## 4. The decoders: whose code, and the crash that decides it + +§1.1 found that `Gumps.GetGump(2)` does not fail — it **corrupts the process**. An +`AccessViolationException` from `unsafe` pointer code is a corrupted-state exception; on .NET +Framework 4.8 it is *not catchable* by an ordinary `try/catch`. In-process, on a live shard, that +is a shard crash with players on it. + +Art and animation probed clean across ~66,000 and ~2,000 ids respectively. That is reassuring and +it is not a guarantee: the inputs we have not tested are exactly the ones that matter — a shard's +own **patched or custom** client files, which is the population this feature exists to serve. + +Two ways to hold this safely: + +**A1 — call ServUO's vendored `Ultima`, isolate the fault.** Fastest to build, and proven for the +kinds we need. Requires running the decode where a fault costs one batch rather than the shard: +a short-lived child process. That means a binary to deploy, which the overlay (deployed as +*source*, compiled by ServUO at boot) has no mechanism for. + +**A2 — own bounds-checked decoders in the overlay.** Roughly 600–900 lines of ordinary safe C#: +`FileIndex` (~150), the RLE frame decoder (~60, and it is the same loop, writing to a `ushort[]` +instead of through a `LockBits` pointer), the static-art decoder, `Body.def`/`Bodyconv.def`, hue +application, and a minimal PNG writer over `System.IO.Compression.DeflateStream` (~80). + +**A2 is recommended**, for four reasons that compound: + +1. **It removes the crash class**, rather than containing it. +2. **It removes `System.Drawing` entirely.** ServUO's `Frame` decoder writes ARGB1555 straight + into a `Bitmap` via `LockBits`, so System.Drawing is in the *decode*, not just the encode — a + Linux/Mono shard needs libgdiplus even to read a sprite. Writing our own pixels and our own PNG + makes the feature portable by construction. +3. **It removes the UOP gap.** ServUO's vendored `Animations` reads legacy `anim*.mul` only. This + client has a full 195 MB `anim.mul` so most bodies resolve — but **gargoyle bodies 666/667 + returned nothing**, because gargoyles live in `AnimationFrame*.uop`. A player race missing from + an asset store built for "player models and everything" is not a caveat, it is a defect. +4. **We are already in this business.** §9 writes a cliloc decompressor from scratch regardless. + +A2 also ends the dependency on whatever version of `Ultima` a given ServUO happens to vendor, +which is the kind of thing that silently changes under a shard upgrade. + +--- + +## 5. Addressing: one key for every asset + +Every asset the bridge can serve is named by a single string key, and the key is the cache key, +the hash key, the filename stem and the manifest row id: + +``` +static/3922 one item graphic +static/3922/h33 the same graphic, hue 33 applied +land/3 one land tile +body/34/a0/d1 creature body 34, action 0, direction 1, first frame +body/400/a0/d1/f0..f9 human male, action 0, direction 1, all ten frames +cliloc/enu the whole converted string table (not an image) +tree/Spawns/Trammel.xml a ServUO tree file (§10) +``` + +Three properties this shape buys: + +- **Hue is part of the key, not a transform.** `itemId` and `hue` are already on the wire together + (`BridgeMarket.cs:582`, `BridgeProfile.cs:314`), so a marketplace listing already knows the exact + key for its own picture. Applying hues website-side would mean shipping `Hues.mul` semantics into + Node for no gain. +- **Depth is expressible without being mandatory.** `body/400/a0/d1` and `body/400/a0/d1/f0..f9` + are the same addressing scheme at two depths, which is what lets §11 bulk-import thumbnails and + fetch full animations on demand without a second protocol. +- **Nothing in the key is client-version-specific**, so a client patch changes an asset's *bytes*, + not its name — which is what makes §7's delta work. + +--- + +## 6. The manifest, and what the two buttons actually do + +Two stages, which is where **Import** and **Update** come from. + +**Stage 1 — the source gate.** The shard reports a manifest of the client files themselves: +size, mtime and content hash of `Cliloc.enu`, `anim*.idx`/`anim*.mul`, `art.mul`/`artidx.mul`, +`Body.def`, `Bodyconv.def`, `Hues.mul`. Unchanged since the last import, and nothing else happens. +This is the same hash gate the spawn atlas and the cliloc table already use, and for the same +reason: the normal case is a restart that changed nothing, and it must cost nothing. + +`anim.mul` is 195 MB and `art.mul` is 148 MB, so the gate is **(size, mtime) first, content hash +only when those differ** — a full hash of 343 MB on every status poll would make the admin panel +feel broken. + +**Stage 2 — the asset manifest.** For the working set (§11), the shard streams +`[{ key, sha256, bytes }]` — no pixels. The website diffs that against what it holds and requests +**only the keys whose hash changed**. + +- **Update** = stage 1, then stage 2, then fetch the diff. +- **Import** = the same path with the diff skipped and every key fetched. +- **A key that has vanished** from the manifest is staged for review, never applied silently — + the same rule, and the same reasoning, as a vanished cliloc source or a disappearing atlas + facet. An unmounted volume and a deliberate client downgrade look identical from here. + +Clilocs are the exception and stay a **whole-table replace** whenever the file hash changes: +the measured cost is 663 ms for 67,496 rows, so per-entry deltas would be complexity bought for +nothing. + +--- + +## 7. The parser version applies here too + +`spawnAtlasSource.js` carries `PARSER_VERSION` (currently 5) and the cliloc source carries its own, +both counted as drift so that a corrected parse reaches an install whose files never change. The +asset pipeline inherits the rule and needs it more, not less: a fixed hue application or a +corrected frame offset changes the bytes we derive from files that are byte-identical. + +**`EXTRACTOR_VERSION` lives in the plugin**, because the plugin is what derives the bytes, and it +is folded into stage 1's gate. Bumping it makes every asset drift, which is correct. + +--- + +## 8. Body ids: the part only the shard can do + +The atlas knows creatures by **slug**, derived from type names in `Spawns/*.xml`. The client knows +them by **body id**. Nothing in the ServUO tree declares the mapping as data — today an operator +bridges it by grepping `Scripts/Mobiles/Normal/.cs` for `Body =`, which appears variously as +a decimal, as hex (`0xD1`), as `Utility.RandomList(35, 36)`, and as an `m_IDs[]` table. + +Inside ServUO the problem does not exist. `BridgeWorld.cs:350` already does exactly the required +thing for a different feature: + +```csharp +var type = ScriptCompiler.FindTypeByName(name, true); +var creature = Activator.CreateInstance(type) as BaseCreature; +``` + +Construct, read `creature.Body.BodyID`, `Delete()`. Authoritative, no source parsing, and correct +for custom creatures a grep would never find. + +**This pass must run on the Core thread** — it constructs and deletes mobiles, which is world +mutation — while the decode in §4 must run **off** it. That split is the one genuinely new +threading shape in this protocol, and it is why slug→body resolution is its own request kind with +its own (small) batch size rather than a step inside asset extraction. + +Constructing arbitrary creature types has side effects: constructors pack items, set skills, start +timers. The mitigations are per-type `try`/`catch`, immediate `Delete()`, small batches, and the +fact that the whole pass is admin-triggered rather than something that runs at boot. + +--- + +## 9. The cliloc decompressor is ours now + +Every modern client ships `Cliloc.*` in the Mythic compressed container — this machine's +`Cliloc.enu` is 4,989,921 bytes beginning `E8 79 67 8E`, high byte `0x8E`. ServUO's bundled +`Ultima.StringList` implements only the plain layout and throws on it (§1.1), which is also why +the shard's own `VendorSearch.GetItemName` is already inert. + +**UOFiddler is released under the Beerware licence**, so porting its decompressor into our +GPL-3.0-or-later tree is clean. It lands in the overlay as ordinary C#, alongside §4's decoders, +and from that point: + +- No operator installs UOFiddler. +- No operator runs `dotnet build` on a converter. +- No operator copies a 5 MB file to a server. +- `website/server/tools/cliloc-export/` is retired, and `UOFIDDLER.md` is deleted rather than + rewritten. + +**What survives untouched is the `custom/` overlay mechanism.** Shard-added items carry cliloc ids +no client table has, and ServUO has no server-side notion of a custom cliloc — that is a real gap +in the *game*, not an artefact of the manual pipeline, and `CLILOCS.md`'s reasoning for it stands. +The base table now arrives over the bridge; overlays still come from a directory the site reads. +Measured on the live shard: 16,434 cliloc ids referenced by the script tree, 37 absent from stock. + +--- + +## 10. The atlas stops needing a shared filesystem + +Today `SPAWN_ATLAS.md` requires the **website** to read the ServUO tree — "same host, a bind mount, +or a shared volume". That is the one place the platform's own rule (only the sidecar bridges the +shard) is broken, and it is broken by the component that faces the internet. + +The same transport closes it. `spawnAtlasSource.js` already labels every file it reads with a +portable key: + +| Label | Count (stock 57.4) | +|---|---| +| `Data/Regions.xml` | 1 | +| `Data/Locations/*.xml` | 6 | +| `Spawns/*.xml` | 13, ~10.5 MB | +| `Config/ChampionSpawns.xml` | 1 | +| `Data/Decoration/**` | tree | + +So the shard serves `tree/