feat(bridge): protocol 8 — the shard reads its own client files (Asset Bridge cutover, 1 of 5) #36
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: RunicGateway/servuo-plugins#36
Reference in New Issue
Block a user
No description provided.
Delete Branch "edge"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Step 1 of 5 of the Asset Bridge
edge → maincutover (docs/link/v8.md§16 row 9b).Land this together with
link#44installer'sbundle.ymlGate 1 refuses a sidecar/overlay pair that disagrees on protocol.This PR moves
overlay.tomlto 8;link#44moves the sidecar'sPROTOCOL_VERSIONto 8.Whichever lands first leaves a compose that cannot run until the other does. Merge them in the same
window.
What comes over
Eight phases, all already merged to
edgeand reviewed there — twelve overlay files, +7,221lines, and the whole shard half of the Asset Bridge:
Ultima.Gumpsis a rule not a preference (AccessViolationExceptionis uncatchable on .NET Framework 4.8)artcolumn filled and renderedSystem.Drawingso the libgdiplus door stays opentrammel.xmlagainst a 1 MiB line cap, so 512 KiB gzipped chunks. 141 files / 11.9 MB → 1.33 MB on the wire in 512 msEXTRACTOR_VERSIONis 3.overlay.tomlprotocol7 → 8.The two safety rules this codifies
BodyConverter.Convert's answer; if it yields nothing, report nothing. Sweeping filetypes for a hit does not find missing art — it puts a giant spider on the gargoyle page, and
every one of those reads reports success. Nothing downstream can detect it.
Ultima.Gumps. It is the one decoder that passeshasExtra: true, the branchthe access violation lives on, and a corrupted-state exception has no in-process defence.
Verification
that booted and served phase 9a's acceptance walk — so this exact overlay is known to compile and
run on a stock 57.4 tree. The shard was left running rather than stopped for a
-Verifypass.67,496 names in 1.42 s, 313 item pictures in 1.38 s, the atlas over the bridge in ~2 s,
and an update with no drift answered in 0.99 s. Portraits verified by eye — panther, boar,
troll and bears all distinct.
After the merge
Release fires on push to
main; expect a minor overlay release, thenbundle.ymlrecomposingthe bundle at protocol 8 once
link's release is out too.edgestays standing.AI disclosure
Authored with Claude Code (Claude Opus 5). Commits carry the
Co-Authored-Bytrailer.🤖 Generated with Claude Code
https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
docs/link/v8.md §16 phase 0. §4 chose to CALL ServUO's vendored `Ultima` rather than reimplement it, on the evidence of a PowerShell probe against a stock client — neither the process nor the client the extractor will run in. This runs the same decoders from inside a running ServUO 57.4 against a client broken in 21 catalogued ways, and it found more than a crash. Adds, all under tools/ and therefore never deployed: * BridgeAssetProbe.cs — the sweep, plus BridgeAssetValidator, a prototype of the validate-before-calling response chosen for §4.2's residual risk. Runs off the Core thread, snapshots Race.AllRaces on it, and writes the id it is ABOUT to touch to a checkpoint file before every call. * BridgeMythicCliloc.cs — the §9 Mythic cliloc reader in net48 C#, ported from UOFiddler (Beerware) with every file-derived index bounds-checked. Phase 2 promotes this into overlay/. * patch_client.ps1 — builds the patched client in five tiers. Hashes every file it touches in the SOURCE before and after and aborts on a change. * an `assetprobe` verb on BridgeRigDriver, so stock and patched can be run against one boot rather than two shard processes. The findings are written up in tools/scaffolding/README.md. The four that change what phase 1 has to build: * FileIndex's UOP constructor ends `MulPath = uopPath`, so artLegacyMUL.uop wins outright and art.mul/artidx.mul are never opened on a current client. A validator bounding offsets against art.mul is not approximate, it is nonsense — the first run refused 34,299 good statics on that mistake, and every refusal looked like a real finding. * 22,102 WRONG PICTURES on a stock, unmodified client. Empty UOP index slots read `lookup 0, length 0`; Seek treats that as a hit, and LoadStatic decodes zero bytes into a shared buffer it reuses, only ever grows, and fills from a Read whose return value is discarded — so the id renders the previously-decoded asset. The mul path does not do this (artidx stores -1), which is why the earlier probe counted 32,766 of them as "ok". A bulk import that trusted the library would have written 22,102 duplicate images under ids that have no art. * The validator caught all 8 record-level defects — 7 of which the library rendered without raising anything, including a verdata lookup past verdata.mul's own end (Verdata.Seek is bounds-checked nowhere) and an 8000x8000 bitmap allocated from two bytes in a file. It refused NOTHING across 49,151 statics and 16,384 land tiles on the stock client, which is the number that makes the boundary defensible. * §4.1's crash reproduces in-process: one Ultima.Gumps.GetGump(2) and the ServUO process disappeared — no catch reached, no console line, the checkpoint file the only record. "Nothing calls Ultima.Gumps" is now an earned safety rule. §9 is proven: 123,490 entries in 218 ms, byte-identical to UOFiddler's own output, with no UOFiddler installed and nothing copied to a server. Not covered, and named as phase 1 work: the animation path has no validator at all, and the patched wolf decoded something else in silence. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4The shard reads its own client's `Cliloc.enu` and serves it over the bridge, so the operator stops installing UOFiddler, building a converter against its `Ultima.dll`, and copying a 5 MB file to the web host every time they patch. `BridgeCliloc.cs` is the one decoder protocol 8 writes rather than calls (docs/link/v8.md §4, §9): a port of UOFiddler's `MythicDecompress` + `MoveToFront` — Beerware, so clean to bring into a GPL-3.0-or-later tree — rewritten against plain arrays, because the upstream is `Span<T>` / `ArrayPool<T>` / `BinaryPrimitives` code and ServUO targets `net48`. The algorithm is deliberately unchanged, including the parts that read oddly. The three-region count/cursor/end table and the symbol-table shifts are upstream's, because a tidier rewrite of somebody else's format decoder is a chance to be subtly wrong in a way that produces plausible text. Two bounds checks were added and they are the only behavioural difference: the upstream indexes its payload unchecked, which is safe for a file the client wrote and is not safe for a file this shard was handed. Measured on a stock client: 4,989,921 bytes read, decompressed and parsed in **290 ms**, yielding **67,496** non-blank rows in id order. That number is the acceptance test — it is what UOFiddler's own DLL produced from this same client through the converter this phase deletes, so an independent implementation agrees to the row. Zero U+FFFD; the 696 non-ASCII rows carry correct curly quotes; the longest row is a 12,149-character EULA, which is why the record length is read unsigned. Blanks never reach the wire — ~56,000 of the 123,490 entries are empty strings the client reserves, and the website discards them at import anyway. Also on this plane: * `assets.error` gains a `code`. Phase 1 chose between 403 and 400 by looking for the word "disabled" in an operator-facing sentence, which makes prose load-bearing; `DISABLED` / `NOT_FOUND` / `UNREADABLE` / `UNAVAILABLE` / `BAD_REQUEST` say it directly. * `Accept` and `Fail` are internal rather than private, because the asset plane's single slot and its refusal frame are shared by every family on it. The cursor is a cliloc NUMBER, not an offset: the decoded table is cached for five idle minutes and released after the last page, so it can be dropped and rebuilt between two pages of one import, and an index would then silently mean something else. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4frombefore the page opens, not after it closes cbdbc9fe5cTwo request families, and they run on opposite threads on purpose. `assets.bodies` (BridgeBodies) answers the question only code inside ServUO can: the atlas knows a creature by the class name in Spawns/*.xml, the client knows it by a body id, and nothing in the tree declares the mapping. Construct the type, read Body.BodyID, Delete(). That is world mutation, so it answers on the CORE thread and is the one family here that does not take the asset worker's slot -- and the batch is capped at 100 names, REFUSED rather than truncated, because a truncated answer is indistinguishable from a complete one from the website's side. `assets.manifest` / `assets.fetch` (BridgeCatalog) are the catalogue, on the worker. The manifest carries { key, sha256, bytes, width, height } and no pixels, so an Update fetches only what moved; the fetch carries base64 PNG. The scan keeps the bytes it hashed rather than decoding all 787 sprites twice. Three things worth stating about the shapes: - It pages on the WALL CLOCK as well as on bytes. The rows are ~90 bytes and the whole catalogue is one page by the byte budget, but building it means decoding hundreds of sprites against a 10 s reply timeout. - `catalog` is derived from the client files (sizes, mtimes, both direction settings, EXTRACTOR_VERSION), not minted per build -- the cache is released when idle, and a fresh id per build would force a restart mid-import although nothing about the client moved. - ARGB1555 is expanded to 32bpp here rather than handed to GDI+, because what it does with a one-bit alpha channel varies by platform and a black rectangle behind every sprite would pass any test that only checked the bytes decoded. Nothing trusts the library's success. Every body goes through CheckEntry and AnimationRecordSane before it is decoded, which is what keeps the 357 bodies whose index entry reads `length 0` -- and which the decoder hands back the PREVIOUS creature's bitmap for -- out of the catalogue. Walked on a live shard: 787 rows in one 734 ms page; bodies 320, 607, 666 all absent rather than wrong; 783 at direction 1 and 4 at direction 0; all 455 stock creature classes resolved at ~190 ms per 100 with zero mobiles leaked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4ServUO's vendored `Ultima.Animations` reads legacy `anim*.mul` only -- it builds its five FileIndexes with the constructor that passes `uopFile: null` -- so everything a modern client moved into `AnimationFrame*.uop` is invisible to it. This adds the one reader docs/link/v8.md 4.3 reserved for phase 4, and wires it in as a fallback beneath the legacy path. What it actually recovers is not what the plan expected, and the difference was measured before any of this was written: - Of the EIGHT player-character bodies 4.8 assigned to this phase, only TWO are in the client at all: gargoyles 666 and 667, in AnimationFrame3.uop. The six ghost bodies (human 402/403, elf 607/608, gargoyle 694/695) are in no package. The five packages hold 10,724 entries between them and the `build/animationlegacyframe/%06d/%02d.bin` name scheme claims every one, so there is no other naming they could be hiding under. - The same fallback reaches 233 further bodies the catalogue had nothing for, so the working set goes from 787 to 1,022 (57 Monster, 26 Animal, 97 Equipment, 50 unlisted, 3 Human, 2 Sea). The catalogue was already 366 Equipment bodies before this, so its character does not change. Decided with the org lead before building: the fallback applies to every body rather than to player bodies alone; ghost ids leave the player-body set entirely (no client has art for any of them, and listing them only advertised keys that cannot exist); the UOP path gets its own PNG encoder rather than Bitmap.Save; and a host without libgdiplus keeps the flat NO_IMAGING refusal rather than serving a partial catalogue. Three things about the reader: - It is not the never-sweep rule being broken. That rule exists because a legacy index is addressed by POSITION, so asking the wrong file returns a giant spider that decodes cleanly. A UOP entry is addressed by the hash of a name containing the body id, and the payload declares that id again for `Group.TryOpen` to check, so a hit is proof of identity. Measured: no hash appears in two packages. - Validate-as-we-go replaces validate-before-calling, because here we ARE the library: the block chain is bounded against the file, the record against the file, the inflated length against the declared one, the frame table against the payload, and every run header against both the record's remaining bytes and the bitmap it writes into. Measured the way 4.5 was -- across every UOP body on a stock client it refuses nothing that carries art. The one body it refuses (286) declares a 0x0 frame, which the vendored decoder treats as absent too. - No System.Drawing anywhere in it, which is what 4.4 promised: the decode fills a ushort[] of ARGB1555 and BridgePng encodes that directly (zlib around net48's raw-deflate-only DeflateStream, CRC32, one IDAT, filter 0). EXTRACTOR_VERSION 1 -> 2: every client file is byte-identical and the answer is different, which is exactly what that number exists to say. The UOP packages join `assets.sources` and the catalogue id, so patching one is drift; `Ultima.Files` cannot resolve them (its table predates UOP animations) so BridgeUop.FindClientFile does it, case-insensitively by enumeration for Linux hosts. Manifest and fetch rows carry a new `source` field (`legacy` / `uop`). Protocol stays 8 -- no message shape changed, only fields added. Measured on the live rig (real sidecar, real ServUO, this machine's client): 1,022 rows in ONE page, 1,409 ms cold; six player bodies, all six with art for the first time (400/401/605/606 legacy, 666/667 uop), all at direction 0; 1,016 at direction 1; the six ghost bodies absent; 45 duplicate-hash groups of which exactly one is new, bodies 1531/1532, two distinct records whose first frames match -- legitimate, and provable only because each payload declares its own body id. The gargoyles were rendered and looked at, because 4.3's whole point is that this failure mode produces confident, wrong pictures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4The spawn atlas was the one place the platform's rule -- only the sidecar bridges the shard -- was broken, and it was broken by the component that faces the internet: SPAWN_ATLAS.md required the website to read the ServUO tree off a bind mount or a shared volume. This serves those files over the loopback link instead (docs/link/v8.md 10). The measurement came first and changed the shape. 10 said the shard would serve `tree/<label>` -> bytes; against a stock 57.4 tree it cannot. Spawns/trammel.xml is 4.03 MB, the sidecar discards any inbound line over 1 MiB, and that file as one base64 row is 5.4 MiB -- it would be dropped, time out, and be re-requested forever with no error anywhere. Two files on a STOCK tree are in that state. So a file crosses as 512 KiB chunks, each gzipped: tree/Spawns/trammel.xml/c0 and so on, which is 5's depth scheme doing the same job it does for body/400/a0/f0 and needing no protocol change to do it. The chunk is the bound and the compression is only the saving -- nothing guarantees an operator's files compress, so the ceiling has to hold when they do not, and a 512 KiB chunk that refuses to compress is still ~683 KiB of base64, inside the wire cap that AssetBatchBytes' deliberate factor of two leaves room for. It is a `tree` FAMILY on assets.fetch rather than 14's separate tree.* commands: phase 5 had already learned that the command is the transport and the family is a property of the key, and assets.manifest is generalised here the same way. That reuses the single slot, the paging envelope, the key ceiling and the mid-import guard -- and leaves `link` with nothing to do for the third phase running. But it gets its OWN consent, Bridge.TreeEnabled. AssetsEnabled is an operator agreeing the website may read their EA-licensed UO client; this is the shard's own configuration, which they wrote, and which the public bestiary is built from. One switch could not express both, and the thing that would silently disappear for an operator who declined the first is their spawn atlas. So the consent check moved into the family lookup, and assets.sources answers whenever either plane is on, reporting `families` filtered to what is actually enabled -- which is how a tree-only shard's website discovers there is anything to ask for. Two defects found, and which harness found which is the part worth keeping: - An empty `catalog` is not an absent one. `expected != null` refused every fetch from a caller that sent "", with a sentence naming no catalog at all. Found by an offline probe that passed one by accident. - GZipStream writes NOTHING for zero bytes of input -- the header is emitted lazily, so a stream opened and closed without a write yields a zero-length buffer rather than the 20-byte empty member. Stock ServUO ships two empty decoration files, so this broke every import off an untouched tree. The offline probe reassembled all 141 files and reported success, because .NET's own decompressor reads an empty stream as empty data and the chunk's declared length (0) and hash (of nothing) both agreed. Only the live walk, through a reader on another runtime, disagreed. Measured end to end against a live shard, the real sidecar and the website's own reader: 141 files, 11,895,427 bytes, 158 chunks, 3 pages, 1.33 MB on the wire, 512 ms; every file byte-identical to disk; the atlas built over the bridge identical to the one built off it. A drift check is the manifest alone -- 32 KB, ~70 ms, no file bytes. The label set is this shard's, never the caller's: a fetch resolves against the set the shard itself enumerated, and tree/../../Scripts/..., Config/Bridge.cfg and Saves/Accounts/accounts.xml are all answered `absent` before a path is built out of them. Protocol stays 8 and EXTRACTOR_VERSION stays 3 -- this family derives nothing, it forwards an operator's own file unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4Phase 7 found this on the tree family and fixed it there. It was inline in THREE places: the body catalogue (phase 3), statics and land (phase 5), and the tree. `expected != null` treats "" as a real fingerprint, so a caller that serialises a missing value as an empty string has EVERY fetch refused -- with a sentence that names no catalog at all ("catalog is now 8159778b"), which reads as a shard fault rather than a caller one. All three now go through one BridgeAssets.CatalogMismatch. Three copies of a comparison are three chances for the next family to get it wrong in a way only a differently-written client would ever reveal. BridgeLeases keeps its own `expected != null` and is deliberately untouched: there the value is a world property, where an empty string is a legitimate thing to expect. Verified against a live shard on a stock ServUO install, every family asked three ways -- with a real catalog, with the field absent, and with an empty string: cliloc.table walk 67,496 rows, 12 pages body manifest / fetch 1,095 rows; ok all three ways static + land fetch ok all three ways static/land carry their OWN catalog art 66a112c1 vs body 323f284f a cross-family catalog refused 422 tree manifest / fetch 141 files incl. BOTH empty ones, all three ways empty files carry a VALID gzip member 2 rows gunzip to 0 bytes a STALE catalog still refused on body, static and tree Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4