21 Commits

Author SHA1 Message Date
59a6c446c6 Merge pull request 'feat(bridge): protocol 8 — the shard reads its own client files (Asset Bridge cutover, 1 of 5)' (#36) from edge into main
All checks were successful
Release overlay / release (push) Successful in -1m10s
Reviewed-on: #36
Reviewed-by: Colby Whitlock <whitlocktech@gmail.com>
2026-09-14 23:09:54 +00:00
9ecc469a5b Merge pull request 'docs(readme): the nine files the Asset Bridge added (Phase 9a)' (#35) from docs/asset-bridge-p9 into edge
Reviewed-on: #35
2026-09-14 22:25:17 +00:00
5050425b0b docs(readme): the nine files the Asset Bridge added (Phase 9a)
The file table in this README stopped at Phase 6's town crier -- ten rows for a
directory that now holds 38 files, stale across four workstreams. Filling all of
it is not this phase's job; documenting the nine files this workstream added is,
and the table now says plainly what it covers so a reader does not take it for an
inventory.

The rows carry the reasoning worth having at a glance: the validator is the
boundary that turned 22,102 confident wrong pictures into honest absences, the
catalogue's action ceiling is what stops the fallback walk serving the next
body's art, `BridgeBodies` is the one question no code outside ServUO can answer,
and `BridgeUop`/`BridgePng` are written without `System.Drawing` on purpose.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
2026-09-14 13:14:26 -05:00
2539764cf7 Merge pull request 'feat(asset-bridge): the shard's own files stop needing a shared filesystem (Phase 7)' (#34) from feat/asset-bridge-p7 into edge
Reviewed-on: #34
2026-09-14 07:36:54 +00:00
936a922487 fix(asset-bridge): an empty catalog is an absent one on every family, not just the tree
Phase 7 found this on the tree family and fixed it there. It was inline in THREE
places: the body catalogue (phase 3), statics and land (phase 5), and the tree.
`expected != null` treats "" as a real fingerprint, so a caller that serialises a
missing value as an empty string has EVERY fetch refused -- with a sentence that
names no catalog at all ("catalog  is now 8159778b"), which reads as a shard
fault rather than a caller one.

All three now go through one BridgeAssets.CatalogMismatch. Three copies of a
comparison are three chances for the next family to get it wrong in a way only a
differently-written client would ever reveal.

BridgeLeases keeps its own `expected != null` and is deliberately untouched:
there the value is a world property, where an empty string is a legitimate thing
to expect.

Verified against a live shard on a stock ServUO install, every family asked three
ways -- with a real catalog, with the field absent, and with an empty string:

  cliloc.table walk                     67,496 rows, 12 pages
  body manifest / fetch                 1,095 rows; ok all three ways
  static + land fetch                   ok all three ways
  static/land carry their OWN catalog   art 66a112c1 vs body 323f284f
  a cross-family catalog                refused 422
  tree manifest / fetch                 141 files incl. BOTH empty ones, all three ways
  empty files carry a VALID gzip member 2 rows gunzip to 0 bytes
  a STALE catalog                       still refused on body, static and tree

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
2026-09-14 02:33:15 -05:00
13b6fc02a4 feat(asset-bridge): the shard's own files stop needing a shared filesystem (Phase 7)
The spawn atlas was the one place the platform's rule -- only the sidecar
bridges the shard -- was broken, and it was broken by the component that faces
the internet: SPAWN_ATLAS.md required the website to read the ServUO tree off a
bind mount or a shared volume. This serves those files over the loopback link
instead (docs/link/v8.md 10).

The measurement came first and changed the shape. 10 said the shard would serve
`tree/<label>` -> bytes; against a stock 57.4 tree it cannot. Spawns/trammel.xml
is 4.03 MB, the sidecar discards any inbound line over 1 MiB, and that file as
one base64 row is 5.4 MiB -- it would be dropped, time out, and be re-requested
forever with no error anywhere. Two files on a STOCK tree are in that state.

So a file crosses as 512 KiB chunks, each gzipped: tree/Spawns/trammel.xml/c0
and so on, which is 5's depth scheme doing the same job it does for
body/400/a0/f0 and needing no protocol change to do it. The chunk is the bound
and the compression is only the saving -- nothing guarantees an operator's files
compress, so the ceiling has to hold when they do not, and a 512 KiB chunk that
refuses to compress is still ~683 KiB of base64, inside the wire cap that
AssetBatchBytes' deliberate factor of two leaves room for.

It is a `tree` FAMILY on assets.fetch rather than 14's separate tree.* commands:
phase 5 had already learned that the command is the transport and the family is
a property of the key, and assets.manifest is generalised here the same way.
That reuses the single slot, the paging envelope, the key ceiling and the
mid-import guard -- and leaves `link` with nothing to do for the third phase
running.

But it gets its OWN consent, Bridge.TreeEnabled. AssetsEnabled is an operator
agreeing the website may read their EA-licensed UO client; this is the shard's
own configuration, which they wrote, and which the public bestiary is built
from. One switch could not express both, and the thing that would silently
disappear for an operator who declined the first is their spawn atlas. So the
consent check moved into the family lookup, and assets.sources answers whenever
either plane is on, reporting `families` filtered to what is actually enabled --
which is how a tree-only shard's website discovers there is anything to ask for.

Two defects found, and which harness found which is the part worth keeping:

  - An empty `catalog` is not an absent one. `expected != null` refused every
    fetch from a caller that sent "", with a sentence naming no catalog at all.
    Found by an offline probe that passed one by accident.
  - GZipStream writes NOTHING for zero bytes of input -- the header is emitted
    lazily, so a stream opened and closed without a write yields a zero-length
    buffer rather than the 20-byte empty member. Stock ServUO ships two empty
    decoration files, so this broke every import off an untouched tree. The
    offline probe reassembled all 141 files and reported success, because .NET's
    own decompressor reads an empty stream as empty data and the chunk's
    declared length (0) and hash (of nothing) both agreed. Only the live walk,
    through a reader on another runtime, disagreed.

Measured end to end against a live shard, the real sidecar and the website's own
reader: 141 files, 11,895,427 bytes, 158 chunks, 3 pages, 1.33 MB on the wire,
512 ms; every file byte-identical to disk; the atlas built over the bridge
identical to the one built off it. A drift check is the manifest alone -- 32 KB,
~70 ms, no file bytes.

The label set is this shard's, never the caller's: a fetch resolves against the
set the shard itself enumerated, and tree/../../Scripts/..., Config/Bridge.cfg
and Saves/Accounts/accounts.xml are all answered `absent` before a path is built
out of them.

Protocol stays 8 and EXTRACTOR_VERSION stays 3 -- this family derives nothing,
it forwards an operator's own file unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
2026-09-14 02:00:22 -05:00
577688b993 Merge pull request 'feat(asset-bridge): the 73 bodies action 0 could not see, and the ceiling that makes looking safe (Phase 6)' (#33) from feat/asset-bridge-p6 into edge
Reviewed-on: #33
2026-09-14 06:15:52 +00:00
a9bd18e48e feat(asset-bridge): the 73 bodies action 0 could not see, and the ceiling that makes looking safe (Phase 6)
The catalogue asked every body for action 0 and reported the rest absent. 73 of
this client's bodies have no art there and real art deeper — body 820's first
drawn action is 23, and it is a horse — so they rendered as text on the bestiary.
The catalogue now falls back to the first action that has art, and the key names
that action (`body/820/a23`). 1,022 -> 1,095 rows.

Walking the action axis is the one thing that can walk off the end of a body's
slots, and the slots after a body's band are the NEXT BODY'S. Measured here: one
action past the band, 643 of 795 legacy bodies return a fully validated picture
and 452 of those are byte-identical to body+1's action 0 (body 1 action 22 is an
ettin; body 3's is an imp, both confirmed by rendering them). Phase 0's validator
cannot catch that — the record is real — so the ceiling refuses the ADDRESS, in
ResolveAnimation where every caller already goes.

The ceiling is the index banding, never `Animations.GetAnimLength`: for a body
reaching file type 5 as id 34 that function answers 22 while the arithmetic gives
13, and the difference is nine actions of another creature's art.

A fetch serves only the key the catalogue chose for that body. `body/820/a0` and
`body/400/a2` come back `unsupported` with the chosen action alongside, never by
decoding what was asked for.

`EXTRACTOR_VERSION` 2 -> 3 (unchanged input, a different answer). Protocol stays
8 — `action` on a manifest/fetch row is additive.

Deep frame keys and the bulk-fill switch that §16 planned for this phase were
NOT built: the site displays still pictures, and a complete one-direction
animation set measures 174,453 frames / 281.5 MB against no consumer (docs
§11.2, org lead 2026-09-11).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
2026-09-14 01:09:38 -05:00
b68aac41c6 Merge pull request 'feat(asset-bridge): item and land art on demand, hued where the files are (Phase 5)' (#32) from feat/asset-bridge-p5 into edge
Reviewed-on: #32
2026-09-11 11:26:40 +00:00
1be1f24562 feat(asset-bridge): item and land art on demand, hued where the files are (Phase 5)
The body catalogue is a set; this is not. This client addresses 49,152 static
ids and has art for 39,189 of them, plus 4,244 land tiles of 16,384 -- and hues
multiply the statics by three thousand. So there is no manifest and no scan:
`assets.fetch` grows two more families (`static`, `land`) and answers the keys
the website's own data names.

`assets.fetch` becomes shared plumbing. BridgeAssets now owns the command, does
the reqId/consent/key-ceiling checks once, derives the family from the keys
themselves (§5 made the key the address; a request that also named its family
would have two places to be wrong and one of them silent) and dispatches to the
reader that registered it. A batch must be of one family, because the reply
carries one `catalog` id. `assets.sources` gains `families` -- additive, so the
protocol stays 8, and EXTRACTOR_VERSION stays 2 because no existing key's bytes
change.

Two traps, both in §4.5's family -- a confident, plausible, wrong picture:

- `Art.GetStatic` memoises into a static Bitmap[0xFFFF] and hands back the SAME
  instance, while `Hue.ApplyTo` repaints in place. Hue a static once and the
  plain key comes back hued from then on, and the next hue stacks on the last.
  Measured on this client before the fix. `Files.CacheData` is now off for the
  life of the process; `TryHue` re-checks it and refuses rather than risk it,
  and the same flag decides whether a bitmap is ours to dispose.

- `PartialHue` decides whether a hue repaints every pixel or only the grey ones,
  per item id, out of `tiledata.mul` -- 13,259 of 65,536 ids on this client.
  Item 597 is a wooden screen with painted flowers: one mode reddens the
  flowers, the other the whole screen. Both decode. The first cut of this reader
  bound `TileData` to ServUO's OWN `Server.TileData` (the enclosing namespace
  beats `using Ultima;`, and it has a PartialHue flag too), which compiled and
  refused every hued key at runtime. Every such type is spelled `Ultima.` now.

Land takes no hue segment: the mode that decides how is an item flag and land
has no equivalent, so `land/3/h33` is refused rather than guessed. `h0` is not a
key either -- hue 0 means "not hued", and the plain key already names it.

Measured through the reader over the whole range: 39,189 statics and 4,244 land
tiles served, and the only refusals are the 9,963 + 12,140 empty index slots
§4.5 predicted. Nothing that carries art is refused.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
2026-09-11 05:58:45 -05:00
452be696df Merge pull request 'feat(asset-bridge): the UOP animation reader, and 235 bodies the legacy path cannot see (Phase 4)' (#31) from feat/asset-bridge-p4 into edge
Reviewed-on: #31
2026-09-11 10:09:53 +00:00
efbd45685c feat(asset-bridge): the UOP animation reader, and 235 bodies the legacy path cannot see (Phase 4)
ServUO's vendored `Ultima.Animations` reads legacy `anim*.mul` only -- it builds its
five FileIndexes with the constructor that passes `uopFile: null` -- so everything a
modern client moved into `AnimationFrame*.uop` is invisible to it. This adds the one
reader docs/link/v8.md 4.3 reserved for phase 4, and wires it in as a fallback beneath
the legacy path.

What it actually recovers is not what the plan expected, and the difference was
measured before any of this was written:

  - Of the EIGHT player-character bodies 4.8 assigned to this phase, only TWO are in
    the client at all: gargoyles 666 and 667, in AnimationFrame3.uop. The six ghost
    bodies (human 402/403, elf 607/608, gargoyle 694/695) are in no package. The five
    packages hold 10,724 entries between them and the
    `build/animationlegacyframe/%06d/%02d.bin` name scheme claims every one, so there
    is no other naming they could be hiding under.
  - The same fallback reaches 233 further bodies the catalogue had nothing for, so the
    working set goes from 787 to 1,022 (57 Monster, 26 Animal, 97 Equipment, 50
    unlisted, 3 Human, 2 Sea). The catalogue was already 366 Equipment bodies before
    this, so its character does not change.

Decided with the org lead before building: the fallback applies to every body rather
than to player bodies alone; ghost ids leave the player-body set entirely (no client
has art for any of them, and listing them only advertised keys that cannot exist); the
UOP path gets its own PNG encoder rather than Bitmap.Save; and a host without
libgdiplus keeps the flat NO_IMAGING refusal rather than serving a partial catalogue.

Three things about the reader:

  - It is not the never-sweep rule being broken. That rule exists because a legacy
    index is addressed by POSITION, so asking the wrong file returns a giant spider
    that decodes cleanly. A UOP entry is addressed by the hash of a name containing the
    body id, and the payload declares that id again for `Group.TryOpen` to check, so a
    hit is proof of identity. Measured: no hash appears in two packages.
  - Validate-as-we-go replaces validate-before-calling, because here we ARE the
    library: the block chain is bounded against the file, the record against the file,
    the inflated length against the declared one, the frame table against the payload,
    and every run header against both the record's remaining bytes and the bitmap it
    writes into. Measured the way 4.5 was -- across every UOP body on a stock client it
    refuses nothing that carries art. The one body it refuses (286) declares a 0x0
    frame, which the vendored decoder treats as absent too.
  - No System.Drawing anywhere in it, which is what 4.4 promised: the decode fills a
    ushort[] of ARGB1555 and BridgePng encodes that directly (zlib around net48's
    raw-deflate-only DeflateStream, CRC32, one IDAT, filter 0).

EXTRACTOR_VERSION 1 -> 2: every client file is byte-identical and the answer is
different, which is exactly what that number exists to say. The UOP packages join
`assets.sources` and the catalogue id, so patching one is drift; `Ultima.Files` cannot
resolve them (its table predates UOP animations) so BridgeUop.FindClientFile does it,
case-insensitively by enumeration for Linux hosts. Manifest and fetch rows carry a new
`source` field (`legacy` / `uop`).

Protocol stays 8 -- no message shape changed, only fields added.

Measured on the live rig (real sidecar, real ServUO, this machine's client):
1,022 rows in ONE page, 1,409 ms cold; six player bodies, all six with art for the
first time (400/401/605/606 legacy, 666/667 uop), all at direction 0; 1,016 at
direction 1; the six ghost bodies absent; 45 duplicate-hash groups of which exactly one
is new, bodies 1531/1532, two distinct records whose first frames match -- legitimate,
and provable only because each payload declares its own body id. The gargoyles were
rendered and looked at, because 4.3's whole point is that this failure mode produces
confident, wrong pictures.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
2026-09-11 04:55:11 -05:00
c71712c734 Merge pull request 'feat(asset-bridge): the body catalogue and slug → body id (Phase 3)' (#30) from feat/asset-bridge-p3 into edge
Reviewed-on: #30
2026-09-10 23:57:33 +00:00
64c0ec00b1 feat(asset-bridge): the body catalogue and slug to body id (Phase 3)
Two request families, and they run on opposite threads on purpose.

`assets.bodies` (BridgeBodies) answers the question only code inside ServUO
can: the atlas knows a creature by the class name in Spawns/*.xml, the client
knows it by a body id, and nothing in the tree declares the mapping. Construct
the type, read Body.BodyID, Delete(). That is world mutation, so it answers on
the CORE thread and is the one family here that does not take the asset
worker's slot -- and the batch is capped at 100 names, REFUSED rather than
truncated, because a truncated answer is indistinguishable from a complete one
from the website's side.

`assets.manifest` / `assets.fetch` (BridgeCatalog) are the catalogue, on the
worker. The manifest carries { key, sha256, bytes, width, height } and no
pixels, so an Update fetches only what moved; the fetch carries base64 PNG.
The scan keeps the bytes it hashed rather than decoding all 787 sprites twice.

Three things worth stating about the shapes:

- It pages on the WALL CLOCK as well as on bytes. The rows are ~90 bytes and
  the whole catalogue is one page by the byte budget, but building it means
  decoding hundreds of sprites against a 10 s reply timeout.
- `catalog` is derived from the client files (sizes, mtimes, both direction
  settings, EXTRACTOR_VERSION), not minted per build -- the cache is released
  when idle, and a fresh id per build would force a restart mid-import although
  nothing about the client moved.
- ARGB1555 is expanded to 32bpp here rather than handed to GDI+, because what
  it does with a one-bit alpha channel varies by platform and a black rectangle
  behind every sprite would pass any test that only checked the bytes decoded.

Nothing trusts the library's success. Every body goes through CheckEntry and
AnimationRecordSane before it is decoded, which is what keeps the 357 bodies
whose index entry reads `length 0` -- and which the decoder hands back the
PREVIOUS creature's bitmap for -- out of the catalogue.

Walked on a live shard: 787 rows in one 734 ms page; bodies 320, 607, 666 all
absent rather than wrong; 783 at direction 1 and 4 at direction 0; all 455 stock
creature classes resolved at ~190 ms per 100 with zero mobiles leaked.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
2026-09-10 18:40:12 -05:00
c79a2a3b2b Merge pull request 'feat(asset-bridge): the cliloc table, decompressed on the shard (Phase 2)' (#29) from feat/asset-bridge-p2 into edge
Reviewed-on: #29
2026-09-10 16:19:49 +00:00
cbdbc9fe5c refactor(asset-bridge): write from before the page opens, not after it closes
`PageBuilder` keeps 256 bytes back for the envelope it still has to write, so a
field appended after `Close()` is spent outside that reserve. It fits today by a
wide margin — the largest measured page was 524,086 of a 524,288 budget, and the
budget is half the line cap on purpose — but "nothing is written after Close()"
is the invariant worth having, because the next family to page will copy this.

No behaviour change; the field moves earlier in the same object.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
2026-09-10 11:18:02 -05:00
73b07eed22 feat(asset-bridge): the cliloc table, decompressed on the shard (Phase 2)
The shard reads its own client's `Cliloc.enu` and serves it over the bridge, so
the operator stops installing UOFiddler, building a converter against its
`Ultima.dll`, and copying a 5 MB file to the web host every time they patch.

`BridgeCliloc.cs` is the one decoder protocol 8 writes rather than calls
(docs/link/v8.md §4, §9): a port of UOFiddler's `MythicDecompress` +
`MoveToFront` — Beerware, so clean to bring into a GPL-3.0-or-later tree —
rewritten against plain arrays, because the upstream is `Span<T>` /
`ArrayPool<T>` / `BinaryPrimitives` code and ServUO targets `net48`.

The algorithm is deliberately unchanged, including the parts that read oddly.
The three-region count/cursor/end table and the symbol-table shifts are
upstream's, because a tidier rewrite of somebody else's format decoder is a
chance to be subtly wrong in a way that produces plausible text. Two bounds
checks were added and they are the only behavioural difference: the upstream
indexes its payload unchecked, which is safe for a file the client wrote and is
not safe for a file this shard was handed.

Measured on a stock client: 4,989,921 bytes read, decompressed and parsed in
**290 ms**, yielding **67,496** non-blank rows in id order. That number is the
acceptance test — it is what UOFiddler's own DLL produced from this same client
through the converter this phase deletes, so an independent implementation
agrees to the row. Zero U+FFFD; the 696 non-ASCII rows carry correct curly
quotes; the longest row is a 12,149-character EULA, which is why the record
length is read unsigned.

Blanks never reach the wire — ~56,000 of the 123,490 entries are empty strings
the client reserves, and the website discards them at import anyway.

Also on this plane:

  * `assets.error` gains a `code`. Phase 1 chose between 403 and 400 by looking
    for the word "disabled" in an operator-facing sentence, which makes prose
    load-bearing; `DISABLED` / `NOT_FOUND` / `UNREADABLE` / `UNAVAILABLE` /
    `BAD_REQUEST` say it directly.
  * `Accept` and `Fail` are internal rather than private, because the asset
    plane's single slot and its refusal frame are shared by every family on it.

The cursor is a cliloc NUMBER, not an offset: the decoded table is cached for
five idle minutes and released after the last page, so it can be dropped and
rebuilt between two pages of one import, and an index would then silently mean
something else.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
2026-09-10 11:12:43 -05:00
e87c103406 Merge pull request 'feat(asset-bridge): the transport, and the 357 wrong pictures it found' (#28) from feat/asset-bridge-p1 into edge
Reviewed-on: #28
2026-09-10 15:02:53 +00:00
c89e818dbf feat(asset-bridge): the transport, and the 357 wrong pictures it found
Asset Bridge phase 1 (docs/link/v8.md §16). Sidecar half: RunicGateway/link#41.
Docs half: RunicGateway/docs#236.

The transport for protocol 8, plus phase 0's validator promoted into the overlay
and extended to animations — which is where the interesting part is.

## 357 of the 1,144 "decodable" bodies are wrong pictures, on a STOCK client

Phase 0 measured the art path and left the animation half unbuilt. It has the same
defect, and it is worse: `GetAnimation` decodes through
`new MemoryStream(m_StreamBuffer, false)` — the whole shared buffer, not the
`length` bytes just read into it — so a truncated or absent record does not even hit
end-of-stream. It sails on into the previous animation's bytes.

Measured directly, because no count could tell:

| Decode body 320 (`lookup 22638982, length 0`) straight after… | Comes back |
|---|---|
| body 12, the dragon | the dragon, 176x167, identical hash |
| body 34, the wolf | the wolf's dimensions, 35x34 |
| body 400, the human male | the human, 27x63, identical hash |

The catalogue is **787 bodies, not 1,144**. Importing the other 357 would have written
duplicate creature portraits into the site showing whichever body the walk decoded
before them.

The record walk refused **0** real bodies on the stock client — the false-refusal
measurement §4.5 says the boundary depends on.

## And four of the twelve player bodies, not six

§5.2 listed the elf ghosts (607, 608) as decoding. Their index entry is `length 0`;
what came back was the elf female at her exact dimensions, because 606 is what the
walk decoded immediately before. Confirmed the same way — 607 after the dragon is
the dragon. Phase 4's UOP decoder now covers eight ids rather than six.

## What is here

- **`overlay/Scripts/Custom/Bridge/BridgeAssets.cs`** — the plane. Accepts on the Core
  thread, hands off to a dedicated asset worker, returns immediately. Three rules, all
  answering a specific failure:
  - **one slot**, second request answered `bridge.busy` (425). `Emit`'s queue is bounded
    in *lines*, so 10,000 queued 200 KB replies is 2 GB of shard memory; the bound that
    holds is flow control, on the side where the memory is.
  - **byte budgets** (`AssetBatchBytes`, 512 KiB) under the sidecar's new 1 MiB cap. The
    factor of two is load-bearing: a page always admits its first item, so it may
    overshoot by one, and the headroom is what makes that land on the wire.
  - **replies, never events** — no `reqId`, no answer. An uncorrelated frame is an event
    by definition, and §3.1 is why none of this may be one.
- **`PageBuilder`** — one paging envelope (`more`/`cursor`/`cut`) for all five families
  that will page, defined before the first one needs it. `cut` matters: "short page" has
  three meanings and only `end` means finished.
- **`assets.sources`** — stage 1 of the import gate, its first user.
- **`BridgeAssetValidator.cs`** — promoted from `tools/`, plus `ResolveAnimation` (the
  never-sweep-file-types rule as code, with no loop and no fallback),
  `AnimationRecordSane` and the frame walk.
- **`EXTRACTOR_VERSION`**, **`overlay.toml` protocol 7 → 8**, `AssetsEnabled`.

## Hashing had to come off the request path

§6's gate is (size, mtime) first, hash only when those differ. The first call has nothing
cached, so that still means hashing 1.06 GB — inside the sidecar's 10 s reply timeout it
does not fit. So hashes are computed on their own thread (deliberately not the single-slot
worker, which would answer every status poll `bridge.busy` for the whole pass) and the
reply carries `hashing`/`complete`.

Measured on the real rig: first call instant with `sha256: null`, second call **44 ms**
with every hash present.

## Verified on the wire, not just compiled

Real ServUO 57.4 + the real sidecar + the real client. `GET /assets/sources` → 200,
`X-UOLink-Version: 8`, `imaging: {ok: true}`, and §4.6's diagnostic firing on a live
client: `artDataFile: artlegacymul.uop`, with `art.mul` and `artidx.mul` both carrying
`shadowedBy`. Live events kept flowing through the new capped reader with no warnings.

Not exercised live: the disabled-plane 403 and the busy 425 (both unit-tested on the
sidecar side; the shard halves are a config read and a lock).

- [x] AI-assisted — Claude Code (Opus 5)

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
2026-09-10 08:32:18 -05:00
1b7edebd31 Merge pull request 'feat(asset-bridge): phase 0 spike — the decoders, from inside a live shard' (#27) from feat/asset-bridge-p0 into edge
Reviewed-on: #27
2026-09-10 08:07:54 +00:00
0ce92152a1 feat(asset-bridge): phase 0 spike — the decoders, from inside a live shard
docs/link/v8.md §16 phase 0. §4 chose to CALL ServUO's vendored `Ultima`
rather than reimplement it, on the evidence of a PowerShell probe against a
stock client — neither the process nor the client the extractor will run in.
This runs the same decoders from inside a running ServUO 57.4 against a
client broken in 21 catalogued ways, and it found more than a crash.

Adds, all under tools/ and therefore never deployed:

  * BridgeAssetProbe.cs — the sweep, plus BridgeAssetValidator, a prototype
    of the validate-before-calling response chosen for §4.2's residual risk.
    Runs off the Core thread, snapshots Race.AllRaces on it, and writes the
    id it is ABOUT to touch to a checkpoint file before every call.
  * BridgeMythicCliloc.cs — the §9 Mythic cliloc reader in net48 C#, ported
    from UOFiddler (Beerware) with every file-derived index bounds-checked.
    Phase 2 promotes this into overlay/.
  * patch_client.ps1 — builds the patched client in five tiers. Hashes every
    file it touches in the SOURCE before and after and aborts on a change.
  * an `assetprobe` verb on BridgeRigDriver, so stock and patched can be run
    against one boot rather than two shard processes.

The findings are written up in tools/scaffolding/README.md. The four that
change what phase 1 has to build:

  * FileIndex's UOP constructor ends `MulPath = uopPath`, so artLegacyMUL.uop
    wins outright and art.mul/artidx.mul are never opened on a current
    client. A validator bounding offsets against art.mul is not approximate,
    it is nonsense — the first run refused 34,299 good statics on that
    mistake, and every refusal looked like a real finding.

  * 22,102 WRONG PICTURES on a stock, unmodified client. Empty UOP index
    slots read `lookup 0, length 0`; Seek treats that as a hit, and
    LoadStatic decodes zero bytes into a shared buffer it reuses, only ever
    grows, and fills from a Read whose return value is discarded — so the id
    renders the previously-decoded asset. The mul path does not do this
    (artidx stores -1), which is why the earlier probe counted 32,766 of
    them as "ok". A bulk import that trusted the library would have written
    22,102 duplicate images under ids that have no art.

  * The validator caught all 8 record-level defects — 7 of which the library
    rendered without raising anything, including a verdata lookup past
    verdata.mul's own end (Verdata.Seek is bounds-checked nowhere) and an
    8000x8000 bitmap allocated from two bytes in a file. It refused NOTHING
    across 49,151 statics and 16,384 land tiles on the stock client, which
    is the number that makes the boundary defensible.

  * §4.1's crash reproduces in-process: one Ultima.Gumps.GetGump(2) and the
    ServUO process disappeared — no catch reached, no console line, the
    checkpoint file the only record. "Nothing calls Ultima.Gumps" is now an
    earned safety rule.

§9 is proven: 123,490 entries in 218 ms, byte-identical to UOFiddler's own
output, with no UOFiddler installed and nothing copied to a server.

Not covered, and named as phase 1 work: the animation path has no validator
at all, and the patched wolf decoded something else in silence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
2026-09-10 03:05:32 -05:00
19 changed files with 9556 additions and 3 deletions

View File

@@ -178,6 +178,26 @@ Runtime script compilation therefore had no effect, silently. `overlay/Scripts/S
| `BridgeAccountLink.cs` | `[link` account linking (Phase 5): one-time code, `link.confirm`, `WebsiteUserId` account tag. |
| `BridgeTownCrier.cs` | Town-crier news (Phase 6): inbound `towncrier.add` / `remove` into the global crier list, with abuse caps. |
The Asset Bridge's own files (protocol 8, `docs/link/v8.md`) — the shard reading the operator's UO
client and its own ServUO tree, and the only part of this plugin that touches files rather than the
world:
| File | Responsibility |
|------|----------------|
| `BridgeAssets.cs` | The plane's front door: the single-slot gate that answers `bridge.busy`, the 512 KiB batch budget, the paging envelope, the family registry, and the background hashing pass that fingerprints the client files without holding the slot. |
| `BridgeAssetValidator.cs` | Judges an index entry (and, for statics, the record behind it) **before** handing an id to `Ultima`. The boundary that turns 22,102 confident wrong pictures on a stock client into honest absences (§4.5). |
| `BridgeCatalog.cs` | The body catalogue: which bodies have art, at which action, and the per-body action ceiling that stops the fallback walk serving the next body's picture (§4.10). |
| `BridgeArt.cs` | Item statics and land tiles on demand, hued on the shard from `tiledata.mul`, behind a byte-bounded cache. |
| `BridgeUop.cs` | The narrow UOP animation reader, written without `System.Drawing` — the one decoder here that is not ServUO's (§4.3). |
| `BridgePng.cs` | Our own PNG encoder, for the same reason. |
| `BridgeBodies.cs` | Slug → body id, on the **Core thread**: construct the type, read `Body.BodyID`, delete it. The one question no code outside ServUO can answer (§8). |
| `BridgeCliloc.cs` | The Mythic cliloc decompressor, ported from UOFiddler (Beerware) — ServUO's own `Ultima.StringList` cannot read a modern client's compressed table (§9). |
| `BridgeTree.cs` | The shard's own `Spawns/*.xml` and friends as a `tree` key family, in gzipped 512 KiB chunks, behind its own consent `Bridge.TreeEnabled` (§10). |
**The table above is the transport plus the Asset Bridge, not all 38 files** in that directory —
the streams added by protocols 3 through 7 (visibility, leases, participation, the event plane's
world verbs) are documented in their own design docs rather than here.
`Emit()` is called from the Core thread. It enqueues and returns — it never touches the socket, never blocks, never allocates a syscall. **A wedged or absent sidecar cannot stall the shard**, and that is the property everything else depends on.
## Testing

View File

@@ -23,8 +23,9 @@
# manual duty: when the protocol changes, bump it here in the same PR that
# changes the emitters, exactly as link bumps PROTOCOL_VERSION.
#
# Current: 6 — see docs/link/v6.md (idempotency keys on inbound commands, champ.boss.killed).
protocol = 7
# Current: 8 — see docs/link/v8.md (the Asset Bridge: client assets over the loopback link
# instead of a converter on somebody's desktop).
protocol = 8
# ── ServUO compatibility ─────────────────────────────────────────────────────
#

View File

@@ -295,6 +295,75 @@ EventsMaxGrantStack=1000
# would land at a moment nobody chose. Set to 0 to allow a save at any time.
EventsMinSaveIntervalSec=300
# The asset plane (docs/link/v8.md, protocol 8). Its own switch, deliberately: turning
# this on is consenting to the website reading this host's UO CLIENT FILES -- art,
# animations, the string table -- over the link. Nothing on this plane writes anything.
AssetsEnabled=true
# The largest reply the asset plane will build, in encoded bytes. Not an item count:
# the ceiling it lives inside is the sidecar's 1 MiB inbound line cap, and base64 adds
# 33% to every payload. Clamped to [64 KiB, 512 KiB] -- half the wire cap, so that a
# single oversized item (always admitted, or its family could never make progress)
# still fits.
AssetBatchBytes=524288
# How many ServUO class names one `assets.bodies` request may carry (phase 3). The only
# bound on this plane counted in items rather than bytes, because what it bounds is not
# reply size -- it is constructing and deleting that many real mobiles ON THE CORE
# THREAD, between two ticks of the world. A larger request is refused, never truncated.
# Clamped to [1, 500].
AssetBodyBatch=100
# How many keys one `assets.fetch` request may name. The byte budget above still decides
# where a page is cut; this only bounds how large a request the shard will parse at all.
# Clamped to [1, 10000].
AssetFetchKeys=2000
# The wall-clock budget for one catalogue page, in milliseconds. The catalogue's manifest
# rows are ~90 bytes so the byte budget never stops it -- but building them means
# decoding hundreds of animations, and the sidecar waits 10 s for a reply. Kept well
# under that, because the page still has to be serialised and written afterwards.
# Clamped to [250, 5000].
AssetScanMs=3000
# Which direction the catalogue renders. NOT part of the asset key: five directions
# would five-fold every count in the working set to express a choice nobody varies.
#
# The split was found by RENDERING all five, not from a table. 0 is head-on, facing the
# viewer -- what a character portrait wants, and the least legible view there is of a
# four-legged creature (a wolf seen from the front is a dark blob). 1 is the front
# three-quarter, where the same wolf is unmistakably a wolf.
#
# Which bodies count as player bodies is asked of the shard (every registered race's
# male/female/ghost ids), never hardcoded. Clamped to [0, 4]: 5-7 are the client
# mirroring 1-3 through a decode branch this overlay has not verified.
AssetPlayerDirection=0
AssetCreatureDirection=1
# The tree plane (docs/link/v8.md §10, phase 7). A THIRD switch, for a third consent:
# the asset switch above is about this host's UO client, which came from EA. This one is
# about the shard's own configuration -- Spawns/*.xml, Data/Regions.xml,
# Data/Locations/*.xml, Config/ChampionSpawns.xml and Data/Decoration/**.cfg -- which is
# the operator's own work and is what the website's spawn atlas is built from. Before
# protocol 8 the website read those files off a shared filesystem; that was the one place
# the platform's own rule (only the sidecar bridges the shard) was broken, and broken by
# the component that faces the internet. Turning this off closes the bridge route and
# leaves that shared-filesystem path as the only way an atlas can be built.
#
# Reads only, and only those five groups. Nothing here joins a path the website sent: a
# request names a label this shard itself enumerated, or it is refused.
TreeEnabled=true
# How much of a tree file one chunk carries, BEFORE compression. Chunking is not an
# optimisation here, it is what makes a spawn file transferable: a stock trammel.xml is
# 4.03 MB, the sidecar discards any inbound line over 1 MiB, and the whole file as one
# base64 row would time out and be re-requested forever with no error anywhere. Each
# chunk is gzipped (a spawn file compresses ~18x, so a chunk is typically 40 KB on the
# wire), but the BOUND comes from the chunk rather than the compression, because nothing
# guarantees input compresses at all. Clamped to [64 KiB, 512 KiB]: at the ceiling a
# worst-case incompressible chunk is ~683 KiB of base64, which still fits the wire.
TreeChunkBytes=524288
# The test scaffolding in tools/scaffolding/ reads its own flags from this file
# (SeedOnStart, CensusOnStart, ProbeOnStart). They are absent here on purpose:
# Config.Get returns the default of false when a key is missing, so a deployed

View File

@@ -0,0 +1,842 @@
using System;
using System.Collections.Generic;
using System.Drawing;
using System.Globalization;
using System.IO;
using System.Text;
using Ultima;
namespace Server.Custom.Bridge
{
/// <summary>
/// **Item and land art, on demand** (docs/link/v8.md §5, §11 — protocol 8, phase 5).
///
/// The body catalogue is a *set*: 1,022 sprites, enumerated, hashed and imported in one
/// pass because a bestiary needs all of them. This is the opposite shape. This client
/// addresses **49,152 static ids** and has real art for **39,189** of them, plus 4,244 land
/// tiles of 16,384 — and then there are hues, which multiply the statics by three thousand.
/// Nothing enumerates that. So there is no manifest here and no scan: the website asks for
/// the handful of keys its own data actually names, and this answers them.
///
/// (49,152 rather than the 81,884 entries `artidx.mul` declares: <c>FileIndex</c> sizes its
/// table from the **length argument it is constructed with**, `0x10000`, not from the idx
/// file — so the addressable range is `0x10000 - 0x4000`. Reading the ceiling off the file
/// instead would invent 16,348 ids, every one of them answered out of an array nobody
/// bounded.)
///
/// ── **The keys** (§5) ──
///
/// <code>
/// static/3922 one item graphic, as the client files hold it
/// static/3922/h33 the same graphic with hue 33 applied
/// land/3 one land tile
/// </code>
///
/// ── **Why the hue is applied HERE and not on the website** ──
///
/// Because it cannot be applied correctly anywhere else, and the incorrect version looks
/// fine.
///
/// A hue is not a tint. It is a 32-entry colour ramp out of `hues.mul` indexed by a
/// pixel's own red channel — and whether it replaces *every* pixel or only the grey ones
/// is decided by the <c>PartialHue</c> flag in <c>tiledata.mul</c>, per item id. On this
/// client **13,259 of 65,536 item ids carry that flag**. Get it wrong on one of them and
/// you do not get an error: item 597 is a wooden screen with painted flowers, and hued red
/// the right way the flowers turn red, the wrong way the whole screen turns red. Both
/// decode. Both are the right size. One is wrong.
///
/// The website has neither file and never will — shipping `Hues.mul` semantics and a
/// 65,536-row flag table into Node to answer a question the shard can answer for free is
/// the same trade §2.1 already refused. So hue is part of the key, and the key is resolved
/// where the files are.
///
/// ── **The trap this phase existed to find** ──
///
/// <c>Art.GetStatic</c> memoises into a static <c>Bitmap[0xFFFF]</c> and returns **the same
/// instance** every time; <c>Hue.ApplyTo</c> repaints a bitmap **in place**. Hue a static
/// once and the library's own copy is hued from then on — the plain key comes back hued,
/// and the next hue stacks on the last. It is §4.5's failure mode (a confident, plausible,
/// wrong picture that every success count agrees with) reached through a door §4.5 never
/// looked at, because phase 0 was auditing *records* and this is the library's *cache*.
///
/// <see cref="BridgeAssets.Initialize"/> turns <c>Files.CacheData</c> off for the life of
/// the process, which makes every bitmap this file receives its own. That invariant is
/// load-bearing enough that <see cref="Render"/> **re-checks it** before applying a hue and
/// refuses rather than risk it: an invariant nothing verifies is a comment.
///
/// ── **What is validated, and against what** ──
///
/// Everything §4.5 built, reused as-is. An index entry is judged before the id is handed to
/// <c>Ultima</c> (<see cref="BridgeAssetValidator.CheckEntry"/>), a static's record header
/// and row table are walked bounded (<c>StaticSane</c>), a land record is checked against
/// the 2,024 bytes <c>LoadLand</c> reads whatever the length says (<c>LandLengthSane</c>),
/// and the bound is taken against **whichever file <c>FileIndex</c> actually opened** —
/// <c>artLegacyMUL.uop</c> on every current client, never <c>art.mul</c> (§4.6).
///
/// Two of §4.5's measurements are this family's, not the catalogue's, and they are the
/// reason all of it is here: on a **stock** client **9,963 static ids and 12,140 land ids**
/// have an index entry reading `lookup 0, length 0`, which <c>FileIndex.Seek</c> treats as
/// a hit and the decoder answers with whatever was decoded last. Measured through this
/// reader over the whole range, those are the ONLY refusals — every one of the 39,189
/// statics and 4,244 land tiles that carries art is served, which is the half of the
/// measurement that says the boundary is in the right place (§4.5).
/// </summary>
public static class BridgeArt
{
/// <summary>Item graphics. <c>static/&lt;id&gt;</c>, optionally <c>/h&lt;hue&gt;</c>.</summary>
private const string StaticFamily = "static";
/// <summary>Land tiles. <c>land/&lt;id&gt;</c>, and no hue segment — see <see cref="TryParseKey"/>.</summary>
private const string LandFamily = "land";
/// <summary>The art index addresses land at its own id and statics at <c>0x4000 + id</c>.</summary>
private const int StaticBase = 0x4000;
/// <summary>Land is addressed with <c>index &amp; 0x3FFF</c> by the library itself.</summary>
private const int LandCount = 0x4000;
/// <summary><c>hues.mul</c> holds 3,000 slots; the wire's hue 1 is slot 0.</summary>
private const int MaxHue = 3000;
public static void Initialize()
{
if (!BridgeConfig.Enabled)
return;
BridgeAssets.RegisterFamily(StaticFamily, ReplyFetch);
BridgeAssets.RegisterFamily(LandFamily, ReplyFetch);
}
// ── the cache (§11) ──────────────────────────────────────────────────────────────────
private sealed class Rendered
{
public string Key;
public string Status;
public string Reason;
public string Sha256;
public byte[] Png;
public int Width;
public int Height;
public int Hue;
public bool PartialHue;
public string Source;
public int Weight
{
get { return Png == null ? 128 : Png.Length + 128; }
}
}
private sealed class Cache
{
public string Id;
public readonly Dictionary<string, Rendered> ByKey =
new Dictionary<string, Rendered>(StringComparer.Ordinal);
/// <summary>Insertion order, for eviction. See <see cref="Remember"/>.</summary>
public readonly Queue<string> Order = new Queue<string>();
public long Bytes;
public DateTime LastUsed;
}
private static readonly object _sync = new object();
private static Cache _cache;
private static readonly TimeSpan IdleFor = TimeSpan.FromMinutes(5);
// ── assets.fetch, the static and land half ───────────────────────────────────────────
/// <summary>
/// Both families' answer to <c>assets.fetch</c>. The correlation id, the operator's
/// consent, the key ceiling and the family decision were made by
/// <see cref="BridgeAssets.OnFetch"/>; every key here belongs to this reader.
///
/// The paging envelope, the byte budget and the `catalog` guard are §3.4's and
/// phase 3's, unchanged — a caller that already walks the body catalogue walks this
/// with the same loop.
/// </summary>
private static void ReplyFetch(string reqId, List<string> keys, string expected, string cursor)
{
string imagingReason;
if (!BridgeAssets.ImagingOk(out imagingReason))
{
// §17.9: a flat refusal, not a partial answer. Every picture in this family needs
// a decoder that goes through GDI+, so there is no half of it to serve.
BridgeAssets.Fail(reqId, "UNAVAILABLE",
"this shard host cannot render images - Mono's System.Drawing needs "
+ "libgdiplus. (" + imagingReason + ")");
return;
}
string id = SourceId();
if (BridgeAssets.CatalogMismatch(expected, id))
{
BridgeAssets.Fail(reqId, "UNREADABLE",
"the shard's client files changed since that catalogue was read (catalog "
+ expected + " is now " + id + "); ask again");
return;
}
Cache cache;
lock (_sync)
{
if (_cache == null || _cache.Id != id)
_cache = new Cache { Id = id };
cache = _cache;
cache.LastUsed = DateTime.UtcNow;
}
int from = ParseKeyCursor(cursor);
var sb = BridgeJson.Begin("assets.fetch.ok");
sb.Str("reqId", reqId)
.Str("family", BridgeAssets.FamilyOfKey(keys[0]))
.Str("catalog", cache.Id)
.Num("extractorVersion", BridgeAssets.EXTRACTOR_VERSION)
.Num("asked", keys.Count)
.Num("from", from);
var page = new BridgeAssets.PageBuilder(sb, "rows", BridgeConfig.AssetBatchBytes);
using (var readers = new Readers())
{
for (int i = from; i < keys.Count; i++)
{
string row = Row(cache, readers, keys[i]);
if (!page.TryAdd(row, "k:" + (i + 1).ToString(CultureInfo.InvariantCulture)))
break;
}
}
page.Close();
sb.Num("sent", page.Count);
BridgeLink.Emit(sb.End());
Sweep();
}
/// <summary>
/// One key to one JSON row.
///
/// A key this shard cannot serve is a **row**, never a failed request: an item id with
/// no art must not cost the other three hundred on the page. The three outcomes are the
/// ones phase 3 defined, and this family adds a `reason` beside them — additive, and
/// the only way an operator learns that eight of their records are damaged rather than
/// simply absent, which is a difference §4.5 spent a whole phase establishing.
/// </summary>
private static string Row(Cache cache, Readers readers, string key)
{
Rendered item = Resolve(cache, readers, key);
var sb = new StringBuilder(2048);
sb.Append("{\"key\":");
BridgeJson.Text(sb, key);
sb.Append(",\"status\":\"").Append(item.Status).Append('"');
if (item.Reason != null)
{
sb.Append(",\"reason\":");
BridgeJson.Text(sb, item.Reason);
}
if (item.Status != "ok")
{
sb.Append('}');
return sb.ToString();
}
sb.Append(",\"sha256\":\"").Append(item.Sha256).Append('"');
sb.Append(",\"bytes\":").Append(item.Png.Length.ToString(CultureInfo.InvariantCulture));
sb.Append(",\"width\":").Append(item.Width.ToString(CultureInfo.InvariantCulture));
sb.Append(",\"height\":").Append(item.Height.ToString(CultureInfo.InvariantCulture));
if (item.Hue > 0)
{
sb.Append(",\"hue\":").Append(item.Hue.ToString(CultureInfo.InvariantCulture));
sb.Append(",\"partialHue\":").Append(item.PartialHue ? "true" : "false");
}
sb.Append(",\"source\":\"").Append(item.Source).Append('"');
sb.Append(",\"png\":\"").Append(Convert.ToBase64String(item.Png)).Append("\"}");
return sb.ToString();
}
private static Rendered Resolve(Cache cache, Readers readers, string key)
{
lock (_sync)
{
Rendered cached;
if (cache.ByKey.TryGetValue(key, out cached))
return cached;
}
Rendered item;
try
{
item = Render(readers, key);
}
catch (Exception e)
{
Console.WriteLine("[Bridge] art: {0}: {1}: {2}", key, e.GetType().Name, e.Message);
item = new Rendered
{
Key = key,
Status = "absent",
Reason = e.GetType().Name
};
}
if (item.Status == "ok")
Remember(cache, item);
return item;
}
/// <summary>
/// Holds the encoded bytes against a byte budget, evicting oldest-first.
///
/// **Oldest-first rather than least-recently-used, deliberately.** The access pattern
/// this serves is a warm pass: the website asks for the keys it has never held, stores
/// them permanently, and does not ask again. What this cache is actually for is the
/// second page of a batch, a retry after a 425, and the same picture appearing in two
/// of a page's rows — all of which insertion order serves exactly as well as recency,
/// and with no bookkeeping on the hot path. A cache whose hit pattern has no recency in
/// it should not pretend to rank by it.
///
/// Only successes are held. An absent key costs one index lookup, which is cheaper than
/// the dictionary entry that would remember it.
/// </summary>
private static void Remember(Cache cache, Rendered item)
{
lock (_sync)
{
if (cache.ByKey.ContainsKey(item.Key))
return;
cache.ByKey[item.Key] = item;
cache.Order.Enqueue(item.Key);
cache.Bytes += item.Weight;
while (cache.Bytes > BridgeConfig.AssetArtCacheBytes && cache.Order.Count > 0)
{
string oldest = cache.Order.Dequeue();
Rendered evicted;
if (!cache.ByKey.TryGetValue(oldest, out evicted))
continue;
cache.ByKey.Remove(oldest);
cache.Bytes -= evicted.Weight;
}
}
}
// ── decode ───────────────────────────────────────────────────────────────────────────
/// <summary>
/// Validate, decode, hue, encode. In that order, and the order is the point.
/// </summary>
private static Rendered Render(Readers readers, string key)
{
bool land;
int id, hue;
if (!TryParseKey(key, out land, out id, out hue))
return Unsupported(key, "not a key this shard serves");
FileIndex index = readers.Index;
if (index == null || index.Index == null)
return Absent(key, "this shard has no art file");
int at = land ? id : StaticBase + id;
if (at < 0 || at >= index.Index.Length)
return Unsupported(key, "id " + id + " is past the end of this client's art index");
string reason;
BridgeAssetValidator.Verdict verdict =
BridgeAssetValidator.CheckEntry(index, at, readers.DataLength, readers.VerdataLength, out reason);
if (verdict == BridgeAssetValidator.Verdict.Absent)
{
// The 9,962 statics and 12,140 land tiles of §4.5: an index entry that reads
// `lookup 0, length 0`, which the library treats as a hit and answers with the
// previous asset's pixels. Absent is the true answer and the only safe one.
return Absent(key, reason);
}
if (verdict != BridgeAssetValidator.Verdict.Ok)
{
// A damaged record rather than a missing one. Still absent to the website — there
// is no picture either way — but the reason is worth carrying, because this one an
// operator can act on.
Console.WriteLine("[Bridge] art: {0} refused: {1}", key, reason);
return Absent(key, reason);
}
if (land)
{
if (!BridgeAssetValidator.LandLengthSane(index, at, out reason))
{
Console.WriteLine("[Bridge] art: {0} refused: {1}", key, reason);
return Absent(key, reason);
}
}
else if (readers.Reader == null || !readers.Reader.StaticSane(index, at, out reason))
{
Console.WriteLine("[Bridge] art: {0} refused: {1}",
key, reason ?? "the art record could not be read");
return Absent(key, reason ?? "the art record could not be read");
}
// A hue is resolved BEFORE anything is decoded, so a bad one costs no pixels and, more
// to the point, cannot half-apply to a picture that then gets cached and served.
Ultima.Hue applied = null;
bool partial = false;
if (hue > 0)
{
if (!TryHue(id, hue, out applied, out partial, out reason))
return Unsupported(key, reason);
}
Bitmap bitmap = land
? Art.GetLand(id)
// `checkmaxid: false` on purpose (§4.5): the default maps an out-of-range id to 0
// and returns ITEM ZERO'S PICTURE. The id is already bounded against the index
// that was actually opened, so this can only be loud.
: Art.GetStatic(id, false);
// **Whether this bitmap is ours to dispose is the same question as whether it is ours
// to hue**, and it has the same answer. With the library's cache off — which
// `BridgeAssets.Initialize` guarantees and `TryHue` re-checks — every call decodes a
// fresh instance that nothing else holds, so not disposing it would leak one bitmap per
// fetched key. With the cache on, that instance is the library's own copy and disposing
// it would leave a disposed `Bitmap` in a static array for the next caller to fault on.
// Both mistakes are silent; the flag decides, once, here.
bool owned = !Files.CacheData;
try
{
if (bitmap == null || bitmap.Width <= 0 || bitmap.Height <= 0)
return Absent(key, "the decoder returned no picture");
if (applied != null)
applied.ApplyTo(bitmap, partial);
byte[] png = BridgeAssets.BitmapToPng(bitmap);
if (png == null)
return Absent(key, "the picture could not be encoded");
return new Rendered
{
Key = key,
Status = "ok",
Sha256 = BridgeAssets.Sha256Hex(png),
Png = png,
Width = bitmap.Width,
Height = bitmap.Height,
Hue = hue,
PartialHue = partial,
Source = readers.Source
};
}
finally
{
if (owned && bitmap != null)
bitmap.Dispose();
}
}
/// <summary>
/// Resolves one wire hue onto a ramp, and decides whether it repaints the whole sprite
/// or only its grey pixels.
///
/// Four things have to hold, and every one of them has a way of not holding that
/// produces a picture rather than an error:
///
/// **The library's cache is off.** Re-checked here because <c>ApplyTo</c> repaints in
/// place: with the cache on, this would edit the copy <c>Art</c> hands to everyone
/// else. <see cref="BridgeAssets.Initialize"/> turns it off at boot and this refuses
/// if it somehow did not, because the failure is invisible and permanent.
///
/// **`hues.mul` is present.** When it is missing <c>Hues.Initialize</c> does not throw
/// — it fills all 3,000 slots with a <c>new Hue(index)</c> whose ramp is **all zeroes**,
/// and applying one of those paints the sprite black. An all-zero ramp is therefore
/// refused whatever the reason for it; on this client there are none.
///
/// **The index is in range.** The wire's hue is 1-based — <c>Ultima.Map</c> does the
/// same <c>GetHue(hue - 1)</c> at line 450 — and <c>GetHue</c> itself masks with
/// `0x3FFF` and falls back to slot 0 rather than failing, so an out-of-range hue would
/// silently become a different colour. Bound it here instead.
///
/// **The <c>PartialHue</c> flag decides the mode**, per item id, out of
/// <c>tiledata.mul</c>. This is the one that is invisible: both modes decode, both are
/// the right size, and 13,259 of this client's item ids need the grey-only one.
/// **Land has no such flag**, which is why <see cref="TryParseKey"/> does not accept a
/// hue on a land key at all rather than guessing a mode for it.
/// </summary>
private static bool TryHue(int id, int hue, out Ultima.Hue applied, out bool partial, out string reason)
{
applied = null;
partial = false;
reason = null;
if (Files.CacheData)
{
reason = "this shard's art cache is on, so a hue cannot be applied safely";
Console.WriteLine("[Bridge] art: refusing hue {0}: {1}", hue, reason);
return false;
}
if (hue < 1 || hue > MaxHue)
{
reason = "hue " + hue + " is outside 1-" + MaxHue;
return false;
}
Ultima.Hue[] list = Ultima.Hues.List;
if (list == null || hue - 1 >= list.Length || list[hue - 1] == null)
{
reason = "this client has no hue table";
return false;
}
Ultima.Hue candidate = list[hue - 1];
if (candidate.Colors == null || AllZero(candidate.Colors))
{
reason = "hue " + hue + " has no colours in this client's hues.mul";
return false;
}
if (!TryPartialHue(id, out partial, out reason))
return false;
applied = candidate;
return true;
}
private static bool AllZero(short[] colors)
{
for (int i = 0; i < colors.Length; i++)
{
if (colors[i] != 0)
return false;
}
return true;
}
/// <summary>
/// The <c>PartialHue</c> flag for one item id.
///
/// Refuses rather than defaults when <c>tiledata.mul</c> cannot be read. Defaulting
/// either way would be a coin flip on 13,259 ids, and the losing side of it is a
/// picture that looks deliberate.
///
/// **Every type here is spelled <c>Ultima.</c> on purpose, and it is not style.**
/// ServUO declares its own <c>Server.TileData</c>, <c>Server.ItemData</c> and
/// <c>Server.TileFlag</c> — with a <c>PartialHue</c> member — in
/// <c>Server/TileData.cs</c>. This file lives in <c>Server.Custom.Bridge</c>, so the
/// enclosing namespace beats the <c>using Ultima;</c> and the unqualified spelling
/// silently binds to the *server's* table: it compiles, the flag exists, and the answer
/// comes from a file resolved through <c>Core.DataDirectories</c> rather than through
/// <c>Ultima.Files</c>, which is the one thing §4.6 says never to do — decide a picture
/// with a file other than the one the pixels came out of. The first run of this reader
/// did exactly that and refused every hued key with a <c>TypeInitializationException</c>
/// from a class this code never meant to name.
/// </summary>
private static bool TryPartialHue(int id, out bool partial, out string reason)
{
partial = false;
reason = null;
Ultima.ItemData[] table;
try
{
table = Ultima.TileData.ItemTable;
}
catch (Exception e)
{
reason = "this client's tiledata could not be read (" + e.GetType().Name + ")";
return false;
}
if (table == null || id < 0 || id >= table.Length)
{
reason = "this client's tiledata does not describe item " + id;
return false;
}
partial = (table[id].Flags & Ultima.TileFlag.PartialHue) != 0;
return true;
}
private static Rendered Absent(string key, string reason)
{
return new Rendered { Key = key, Status = "absent", Reason = reason };
}
private static Rendered Unsupported(string key, string reason)
{
return new Rendered { Key = key, Status = "unsupported", Reason = reason };
}
// ── keys, cursors and the source id ──────────────────────────────────────────────────
/// <summary>
/// <c>static/&lt;id&gt;</c>, <c>static/&lt;id&gt;/h&lt;hue&gt;</c> and
/// <c>land/&lt;id&gt;</c>.
///
/// **A land key takes no hue segment.** The client can hue a land tile, but the mode
/// that decides how is an *item* flag and land has no equivalent — so the honest answer
/// to `land/3/h33` is that this shard does not serve it, rather than a picture produced
/// by guessing. Nothing on the wire carries a hued land tile today; if something ever
/// does, it arrives with a reason to choose.
/// </summary>
private static bool TryParseKey(string key, out bool land, out int id, out int hue)
{
land = false;
id = 0;
hue = 0;
if (key == null)
return false;
string[] parts = key.Split('/');
if (parts.Length < 2 || parts.Length > 3)
return false;
if (parts[0] == LandFamily)
land = true;
else if (parts[0] != StaticFamily)
return false;
if (!Int32.TryParse(parts[1], NumberStyles.None, CultureInfo.InvariantCulture, out id))
return false;
if (id < 0)
return false;
if (land && id >= LandCount)
return false;
if (parts.Length == 2)
return true;
if (land)
return false;
string segment = parts[2];
if (segment.Length < 2 || segment[0] != 'h')
return false;
if (!Int32.TryParse(segment.Substring(1), NumberStyles.None,
CultureInfo.InvariantCulture, out hue))
return false;
// **`h0` is not a key.** Hue 0 on the wire means "this item is not hued", so the plain
// key already names its picture. Accepting `static/3922/h0` as a synonym would have
// the website store the identical PNG twice under two names, diff them separately on
// every Update, and show whichever row it happened to join against -- for a distinction
// that does not exist. The caller drops the segment instead.
return hue > 0;
}
private static int ParseKeyCursor(string cursor)
{
if (cursor == null)
return 0;
int value;
if (cursor.StartsWith("k:", StringComparison.Ordinal)
&& Int32.TryParse(cursor.Substring(2), NumberStyles.None, CultureInfo.InvariantCulture, out value))
return Math.Max(0, value);
return 0;
}
/// <summary>
/// Everything that decides these bytes, hashed into one short id — the same guard
/// phase 3 built, over this family's inputs.
///
/// Four files, and each earns its place: the art data file holds the pixels,
/// `hues.mul` holds the ramps, `tiledata.mul` decides which of the two hue modes an
/// item gets, and `verdata.mul` can patch any record in any of them. Leaving
/// `tiledata.mul` out would be the subtle one — a client patch that only flipped
/// <c>PartialHue</c> flags changes no pixel in any source file and every hued picture
/// derived from them.
/// </summary>
private static string SourceId()
{
var sb = new StringBuilder(256);
sb.Append(BridgeAssets.EXTRACTOR_VERSION);
foreach (string path in new[]
{
BridgeAssetValidator.ArtDataPath(),
FilePath("hues.mul"),
FilePath("tiledata.mul"),
FilePath("verdata.mul")
})
{
sb.Append('|');
if (path == null)
continue;
try
{
var info = new FileInfo(path);
if (!info.Exists)
continue;
sb.Append(info.Length).Append(',').Append(info.LastWriteTimeUtc.Ticks);
}
catch
{
// An unreadable file is itself a state, and one that must not change from page
// to page without being noticed. Leaving the slot empty does that.
}
}
return BridgeAssets.Sha256Hex(Encoding.UTF8.GetBytes(sb.ToString())).Substring(0, 16);
}
private static string FilePath(string name)
{
try
{
return Files.GetFilePath(name);
}
catch
{
return null;
}
}
// ── shared plumbing ──────────────────────────────────────────────────────────────────
/// <summary>
/// The art index and its record reader, opened for one reply and closed with it — the
/// same lifetime rule phase 3's <c>Readers</c> follows, and for the same reason: a page
/// decodes hundreds of sprites through them and opening them is microseconds, so
/// holding handles on the operator's client files for the life of a cache buys nothing.
/// </summary>
private sealed class Readers : IDisposable
{
public readonly FileIndex Index;
public readonly BridgeAssetValidator.RecordReader Reader;
public readonly long DataLength;
public readonly long VerdataLength;
/// <summary>
/// Which file the pixels came out of — `uop` or `legacy` — carried on every row
/// beside the body catalogue's own `source` (§4.9). On this plane it answers §4.6's
/// operator question: art added to `art.mul` while `artLegacyMUL.uop` is present is
/// never read, and a row that says `uop` is what says so.
/// </summary>
public readonly string Source;
public Readers()
{
string data = BridgeAssetValidator.ArtDataPath();
string verdata = FilePath("verdata.mul");
DataLength = BridgeAssetValidator.MulLength(data);
VerdataLength = BridgeAssetValidator.MulLength(verdata);
Source = data != null && data.EndsWith(".uop", StringComparison.OrdinalIgnoreCase)
? "uop"
: "legacy";
try
{
Index = BridgeAssetValidator.OpenArtIndex();
if (data != null)
Reader = new BridgeAssetValidator.RecordReader(data, verdata);
}
catch (Exception e)
{
Console.WriteLine("[Bridge] art: could not open the art files: {0}", e.Message);
}
}
public void Dispose()
{
if (Reader == null)
return;
try
{
Reader.Dispose();
}
catch
{
// Closing a read-only handle. Nothing useful is left to do.
}
}
}
/// <summary>
/// Lets the held pictures go once nothing has asked for one in five minutes. The id is
/// derived from the client files rather than minted per build, so a walk that spans the
/// drop resumes against the same catalogue instead of starting over.
/// </summary>
private static void Sweep()
{
lock (_sync)
{
if (_cache == null)
return;
if (DateTime.UtcNow - _cache.LastUsed > IdleFor)
_cache = null;
}
}
public static string Status()
{
lock (_sync)
{
if (_cache == null)
return "art(empty)";
return String.Format("art(id={0} held={1} bytes={2} cap={3})",
_cache.Id, _cache.ByKey.Count, _cache.Bytes, BridgeConfig.AssetArtCacheBytes);
}
}
}
}

View File

@@ -0,0 +1,863 @@
using System;
using System.IO;
using Ultima;
namespace Server.Custom.Bridge
{
/// <summary>
/// **Validate before calling** (docs/link/v8.md §4.5) — the boundary between this protocol
/// and ServUO's vendored <c>Ultima</c> decoders. Phase 0 prototyped it in
/// <c>tools/scaffolding/BridgeAssetProbe.cs</c> and measured it both ways; phase 1 promoted
/// it here, into the overlay, and extended it to animations.
///
/// The principle: `Ultima`'s decoders take their bounds from the file they are reading, so
/// the extractor must decide whether a record is worth handing over *before* handing it
/// over. Every check below is against the index entry and the record header — cheap, and
/// enough to turn an uncatchable corrupted-state exception into a skipped asset.
///
/// **The failure this exists for is a wrong picture, not a crash.** `LoadStatic`,
/// `LoadLand` and `GetAnimation` all decode out of a shared <c>m_StreamBuffer</c> that is
/// reused, only ever grown, and filled by a <c>stream.Read</c> whose return value is
/// discarded. A record that is short, absent or out of bounds therefore renders **whatever
/// the previously-decoded asset left behind**, reports success, and is undetectable by
/// anything downstream. On the stock client on the machine phase 0 ran on that is 22,102
/// ids whose index entry reads <c>lookup 0, length 0</c>.
///
/// It cannot be complete and does not claim to be. It closes the shapes that reading the
/// source showed are reachable. What says the boundary is in the right place is the second
/// measurement rather than the first: against a client patched 21 ways it refused all eight
/// record-level defects, and against the **stock** client it refused **nothing** across
/// 49,151 statics and 16,384 land tiles. A checker that refuses real art would be worse
/// than no checker.
/// </summary>
public static class BridgeAssetValidator
{
public enum Verdict
{
/// <summary>Nothing at this id, and the index says so honestly.</summary>
Absent,
/// <summary>The entry is self-consistent and inside its file.</summary>
Ok,
/// <summary>The entry claims something the file cannot support. Do not decode it.</summary>
Refused
}
/// <summary>Land tiles decode a fixed 44×44 diamond: 2 × (2+4+…+44) ushorts.</summary>
public const int LandRecordBytes = 2024;
/// <summary>
/// A ceiling on decoded art dimensions. `LoadStatic` allocates
/// <c>new Bitmap(width, height)</c> straight from two ushorts in the record, so a
/// corrupt header asks for up to 65535×65535 — an 8 GB allocation, from a file. Real
/// art is a couple of hundred pixels at most.
/// </summary>
public const int MaxArtDimension = 1024;
/// <summary>
/// Builds our own index over the same files, with the same constructor arguments
/// <c>Art</c> uses — including <c>hasExtra: false</c>, which is the whole reason the
/// art path is safe where the gump path is not (§4.1).
/// </summary>
public static FileIndex OpenArtIndex()
{
if (ArtDataPath() == null)
return null;
return new FileIndex("Artidx.mul", "Art.mul", "artLegacyMUL.uop", 0x10000, 4, ".tga", 0x13FDC, false);
}
/// <summary>
/// The file an art index entry's <c>lookup</c> is an offset **into** — which is not
/// <c>art.mul</c> on any current client.
///
/// This cost a whole probe run to learn and it is the single most important thing
/// phase 1 must not get wrong. <c>FileIndex</c>'s UOP constructor ends with a bare
/// <c>MulPath = uopPath</c>: **when <c>artLegacyMUL.uop</c> exists it wins outright**,
/// and <c>art.mul</c> / <c>artidx.mul</c> are never opened at all. A validator that
/// bounds offsets against <c>art.mul</c> while the index holds UOP offsets is not
/// merely approximate, it is nonsense — the first run of this probe refused 34,299
/// perfectly good statics for "declaring 10533x2085" because it was reading UOP
/// offsets into the wrong file.
///
/// So the resolution order here mirrors <c>FileIndex</c>'s exactly, and anything that
/// needs the bytes behind an entry must ask this rather than assume.
/// </summary>
public static string ArtDataPath()
{
var uop = Files.GetFilePath("artlegacymul.uop");
if (uop != null)
return uop;
return Files.GetFilePath("art.mul");
}
public static long MulLength(string path)
{
if (path == null)
return 0;
try
{
return new FileInfo(path).Length;
}
catch
{
return 0;
}
}
/// <summary>
/// Judges one index entry.
///
/// The check <c>FileIndex.Seek</c> is missing is the last one: it tests
/// <c>Stream.Length &lt; e.lookup</c> — that the record *starts* inside the file — and
/// never that it *ends* inside it. A record that begins two bytes before EOF and
/// declares a length of 4,000 passes, and <c>stream.Read</c> then returns a short count
/// that the decoders discard, leaving the previous asset's bytes in the shared buffer.
/// </summary>
public static Verdict CheckEntry(FileIndex index, int at, long mulLength, long verdataLength, out string reason)
{
reason = null;
if (index == null || index.Index == null || at < 0 || at >= index.Index.Length)
{
reason = "index " + at + " out of range";
return Verdict.Absent;
}
Entry3D e = index.Index[at];
if (e.lookup < 0)
{
reason = "lookup " + e.lookup;
return Verdict.Absent;
}
bool patched = (e.length & (1 << 31)) != 0;
int length = e.length & 0x7FFFFFFF;
if (!patched && e.length < 0)
{
reason = "length " + e.length;
return Verdict.Absent;
}
if (length == 0)
{
reason = "lookup " + e.lookup + ", length 0";
return Verdict.Absent;
}
long ceiling = patched ? verdataLength : mulLength;
if (ceiling <= 0)
{
reason = (patched ? "verdata.mul" : "the art data file") + " has no length";
return Verdict.Refused;
}
if (e.lookup >= ceiling)
{
reason = "lookup " + e.lookup + " past the end of "
+ (patched ? "verdata.mul" : "the mul") + " (" + ceiling + ")";
return Verdict.Refused;
}
// The missing check. A short read is silent, and its consequence is the PREVIOUS
// asset's picture served under this id.
if (e.lookup + (long)length > ceiling)
{
reason = "record runs " + (e.lookup + (long)length - ceiling) + " bytes past the end of "
+ (patched ? "verdata.mul" : "the mul");
return Verdict.Refused;
}
return Verdict.Ok;
}
/// <summary>
/// `LoadLand` reads 2,024 bytes regardless of the declared length, so a shorter record
/// reads past the end of a buffer sized from that length.
/// </summary>
public static bool LandLengthSane(FileIndex index, int at, out string reason)
{
reason = null;
if (index == null || index.Index == null || at < 0 || at >= index.Index.Length)
return true;
int length = index.Index[at].length & 0x7FFFFFFF;
if (length > 0 && length < LandRecordBytes)
{
reason = "land record is " + length + " bytes; LoadLand always reads " + LandRecordBytes;
return false;
}
return true;
}
/// <summary>
/// Walks a static record's own row table the way <c>LoadStatic</c> will, and refuses
/// it if that walk would read outside the record.
///
/// This is the check with teeth. <c>LoadStatic</c>'s inner loop guards the write into
/// the bitmap (<c>xOffset &gt; delta</c>, <c>xOffset + xRun &gt; delta</c>) and does
/// nothing at all about the read cursor, which advances until it happens to find a
/// zero pair — potentially far outside a pinned array. Simulating the same walk with
/// a bound is the cheapest way to know whether handing the id over is safe.
/// </summary>
public static bool StaticRecordSane(byte[] record, int length, out string reason)
{
reason = null;
if (length < 8)
{
reason = "record is " + length + " bytes; a static header needs 8";
return false;
}
int words = length / 2;
int width = ReadUInt16(record, 4);
int height = ReadUInt16(record, 6);
// LoadStatic returns null for these rather than misbehaving, so it is not a refusal.
if (width <= 0 || height <= 0)
return true;
if (width > MaxArtDimension || height > MaxArtDimension)
{
reason = "declares " + width + "x" + height + ", past the " + MaxArtDimension + "px ceiling";
return false;
}
// The row-lookup table: height ushorts starting at word 4.
if (4 + height > words)
{
reason = "row table (" + height + " entries) does not fit in a " + length + "-byte record";
return false;
}
int start = height + 4;
for (int y = 0; y < height; y++)
{
int cursor = start + ReadUInt16(record, (4 + y) * 2);
while (true)
{
// Two ushorts for the run header, and they must both be inside the record.
if (cursor < 0 || cursor + 1 >= words)
{
reason = "row " + y + " reads at word " + cursor + ", past the record's " + words;
return false;
}
int xOffset = ReadUInt16(record, cursor * 2);
int xRun = ReadUInt16(record, (cursor + 1) * 2);
cursor += 2;
if (xOffset + xRun == 0)
break;
// LoadStatic stops the row here, so the read cursor stops with it.
if (xOffset > width || xOffset + xRun > width)
break;
if (cursor + xRun > words)
{
reason = "row " + y + " declares a " + xRun + "-pixel run running past the record";
return false;
}
cursor += xRun;
}
}
return true;
}
// ── animations (phase 1) ─────────────────────────────────────────────────────────────
//
// Phase 0 measured the art path and left this half unbuilt, and then proved it was
// needed: the patched client's verdata entry for body 34 points past verdata.mul's own
// end, and the wolf still "decoded" — counted among the 1,144 successes while rendering
// something else entirely. `GetAnimation` has every weakness `LoadStatic` has and one
// more, because the buffer it decodes from is longer than the record it read.
/// <summary>The palette every animation record opens with: 0x100 ushorts.</summary>
public const int AnimPaletteBytes = 0x100 * 2;
/// <summary>
/// A ceiling on an animation's declared frame count. <c>GetAnimation</c> does
/// <c>new int[frameCount]</c> straight from four bytes in the file, before it has
/// looked at anything else. Real actions are tens of frames.
/// </summary>
public const int MaxAnimFrames = 1024;
/// <summary>The xor <c>Frame</c> applies to every run header before decoding it.</summary>
private const int DoubleXor = (0x200 << 22) | (0x200 << 12);
/// <summary>
/// The <c>anim*.mul</c> an animation index entry's <c>lookup</c> is an offset into.
///
/// Unlike art (§4.6) there is no UOP precedence to get wrong here, and that is not
/// luck: <c>Animations</c> constructs its five <c>FileIndex</c>es with the four-argument
/// constructor, which passes <c>uopFile: null</c>. It never reads
/// <c>AnimationFrame*.uop</c> at all — which is the same fact that leaves six of the
/// twelve player-character bodies undecodable until §4.3's reader lands in phase 4.
/// </summary>
public static string AnimDataPath(int fileType)
{
switch (fileType)
{
case 1: return Files.GetFilePath("anim.mul");
case 2: return Files.GetFilePath("anim2.mul");
case 3: return Files.GetFilePath("anim3.mul");
case 4: return Files.GetFilePath("anim4.mul");
case 5: return Files.GetFilePath("anim5.mul");
default: return null;
}
}
/// <summary>
/// Builds our own index over one anim file, with the same constructor arguments
/// <c>Animations</c> uses — the entry lengths especially, since they decide how far
/// into the file an index runs.
/// </summary>
public static FileIndex OpenAnimIndex(int fileType)
{
if (AnimDataPath(fileType) == null)
return null;
switch (fileType)
{
case 1: return new FileIndex("Anim.idx", "Anim.mul", 0x40000, 6);
case 2: return new FileIndex("Anim2.idx", "Anim2.mul", 0x10000, -1);
case 3: return new FileIndex("Anim3.idx", "Anim3.mul", 0x20000, -1);
case 4: return new FileIndex("Anim4.idx", "Anim4.mul", 0x20000, -1);
case 5: return new FileIndex("Anim5.idx", "Anim5.mul", 0x20000, -1);
default: return null;
}
}
/// <summary>
/// Where a body's animation actually lives: which anim file, and which index in it.
///
/// **This is the never-sweep-file-types rule, written as code** (§4.3). It asks
/// <c>BodyConverter.Convert</c> once, takes its answer, and if that answer leads
/// nowhere it reports nowhere. There is deliberately no loop here and no fallback,
/// because asking the *other* anim files for an index they do not own does not fail —
/// it returns 175 decodable action/direction combinations of **a giant spider** for
/// gargoyle 666, and misaligned colour fragments for the other two. Every one of those
/// reads reports success, and nothing downstream can tell them from art.
///
/// A false return with <paramref name="reason"/> set is the ordinary, expected answer
/// for a body this client has no art for — the caller reports absent, not an error.
/// </summary>
public static bool ResolveAnimation(
int body, int action, int direction, out int fileType, out int index, out string reason)
{
reason = null;
fileType = 0;
index = -1;
if (body <= 0 || action < 0)
{
reason = "body " + body + " action " + action + " is not addressable";
return false;
}
// Directions 5-7 are the client mirroring 1-3, and `Frame` decodes them through its
// flip branch — different pointer arithmetic, which nothing below has checked.
// §5.1 fixed this protocol at direction 0 or 1, so refusing the rest costs nothing
// and keeps the validator honest about what it has actually verified.
if (direction < 0 || direction > 4)
{
reason = "direction " + direction + " is mirrored; this protocol reads 0-4 only";
return false;
}
int translated = body;
int hue = 0;
try
{
// Exactly what GetAnimation(..., preserveHue: false, ...) does first.
Animations.Translate(ref translated, ref hue);
fileType = BodyConverter.Convert(ref translated);
}
catch (Exception e)
{
reason = "body.def/bodyconv.def lookup failed: " + e.GetType().Name;
return false;
}
if (AnimDataPath(fileType) == null)
{
// Gargoyle 666 lands here: Bodyconv.def maps it to anim5, and this client has no
// anim5. Absent is the correct answer and the ONLY safe one.
reason = "bodyconv sends body " + body + " to file type " + fileType
+ ", which this client does not have";
return false;
}
int actions = ActionsOf(translated, fileType);
if (action >= actions)
{
// §4.10, measured in phase 6: this is the never-sweep rule again, one axis over.
// A body's slots are contiguous and the next body's begin immediately after them,
// so `index + action * 5` past the ceiling addresses ANOTHER BODY'S action — a
// real record, at a real offset, that every check below passes. Measured on this
// client: of 795 legacy bodies, 643 return a fully validated picture one action
// past their band and **452 of those are byte-identical to body+1's action 0**.
// Body 1 action 22 is an ettin; body 3 action 22 is an imp. Nothing downstream
// can tell, which is why the refusal has to be here.
reason = "body " + body + " has " + actions + " actions in file type " + fileType
+ "; action " + action + " belongs to the next body";
return false;
}
index = AnimIndexOf(translated, fileType) + (action * 5) + direction;
return true;
}
/// <summary>
/// How many actions the index reserves for a body — the only safe ceiling, and it is
/// the banding rather than the library's own answer.
///
/// <c>Animations.GetAnimLength</c> exists and looks like the right source. It is not:
/// for a body reaching file type 5 as id 34 it answers **22** while
/// <see cref="AnimIndexOf"/> puts that body in the 65-slot band, which is **13**. The
/// two disagree on exactly one body of this client (reached by translation from body
/// 276), and taking the larger number is nine actions of somebody else's art. So the
/// count is derived from the same arithmetic that produces the offset, in the same
/// file, where the two cannot drift apart.
/// </summary>
public static bool ActionCount(int body, out int actions, out int fileType, out string reason)
{
reason = null;
actions = 0;
fileType = 0;
if (body <= 0)
{
reason = "body " + body + " is not addressable";
return false;
}
int translated = body;
int hue = 0;
try
{
Animations.Translate(ref translated, ref hue);
fileType = BodyConverter.Convert(ref translated);
}
catch (Exception e)
{
reason = "body.def/bodyconv.def lookup failed: " + e.GetType().Name;
return false;
}
if (AnimDataPath(fileType) == null)
{
reason = "bodyconv sends body " + body + " to file type " + fileType
+ ", which this client does not have";
return false;
}
actions = ActionsOf(translated, fileType);
return true;
}
/// <summary>
/// The banding of <see cref="AnimIndexOf"/>, read as an action count: a body's slots
/// are five directions per action, so the band size divided by five is how many
/// actions it owns.
/// </summary>
private static int ActionsOf(int body, int fileType)
{
return SlotsOf(body, fileType) / 5;
}
/// <summary>
/// How many index slots <see cref="AnimIndexOf"/>'s arithmetic gives this body. The
/// bands are transcribed there and their sizes here, from the same source and in the
/// same order, because a ceiling that disagrees with an offset is worse than no
/// ceiling at all.
/// </summary>
private static int SlotsOf(int body, int fileType)
{
switch (fileType)
{
case 2:
return body < 200 ? 110 : 65;
case 3:
if (body < 300)
return 65;
return body < 400 ? 110 : 175;
case 5:
// Body 34's exclusion again — it is in the second band here, so it owns 13
// actions and not 22. This is the one body `GetAnimLength` is wrong about.
if (body < 200 && body != 34)
return 110;
return body < 400 ? 65 : 175;
default: // 1 and 4 share their banding
if (body < 200)
return 110;
return body < 400 ? 65 : 175;
}
}
/// <summary>
/// <c>Animations.GetFileIndex</c>'s own arithmetic, which is private. The banding is
/// per file type and the boundaries differ between them, so this is transcribed rather
/// than generalised — an index that disagrees with the library's by one is a picture
/// of the wrong creature, validated.
/// </summary>
private static int AnimIndexOf(int body, int fileType)
{
switch (fileType)
{
case 2:
return body < 200 ? body * 110 : 22000 + ((body - 200) * 65);
case 3:
if (body < 300)
return body * 65;
return body < 400 ? 33000 + ((body - 300) * 110) : 35000 + ((body - 400) * 175);
case 5:
// "looks strange, though it works" — the library's own comment. Body 34 is
// excluded from the first band here and nowhere else.
if (body < 200 && body != 34)
return body * 110;
return body < 400 ? 22000 + ((body - 200) * 65) : 35000 + ((body - 400) * 175);
default: // 1 and 4 share their banding
if (body < 200)
return body * 110;
return body < 400 ? 22000 + ((body - 200) * 65) : 35000 + ((body - 400) * 175);
}
}
/// <summary>
/// Walks an animation record the way <c>GetAnimation</c> and <c>Frame</c> will, and
/// refuses it if that walk would read outside the record or write outside the bitmap.
///
/// Two things make this stricter than the static walk, and both come from the library:
///
/// <c>GetAnimation</c> decodes through <c>new MemoryStream(m_StreamBuffer, false)</c> —
/// the whole shared buffer, not the <c>length</c> bytes it just read into it. So a
/// truncated record does not hit end-of-stream and throw; the reader sails on into the
/// **previous** animation's bytes and returns a plausible frame. Bounding against
/// <paramref name="length"/> rather than against the buffer is the entire point.
///
/// And <c>Frame</c>'s run loop is a *write* through a <c>LockBits</c> pointer whose
/// origin comes from two signed shorts in the file (<c>xCenter</c>, <c>yCenter</c>),
/// with no bound of any kind. <c>LoadStatic</c> at least guards its writes; this does
/// not, so the destination of every run is checked against the bitmap it locked.
///
/// <paramref name="maxFrames"/> is how many frames the caller will actually decode —
/// 1 for the catalogue's thumbnail (<c>FirstFrame: true</c>), 0 for all of them.
/// Checking frames nobody decodes would invent refusals, which §4.5 costs more than
/// it saves.
/// </summary>
public static bool AnimationRecordSane(byte[] record, int length, int maxFrames, out string reason)
{
reason = null;
if (length < AnimPaletteBytes + 4)
{
reason = "record is " + length + " bytes; an animation needs "
+ (AnimPaletteBytes + 4) + " for its palette and frame count";
return false;
}
int start = AnimPaletteBytes;
int frameCount = ReadInt32(record, start);
if (frameCount <= 0)
{
reason = "declares " + frameCount + " frames";
return false;
}
if (frameCount > MaxAnimFrames)
{
reason = "declares " + frameCount + " frames, past the " + MaxAnimFrames + " ceiling";
return false;
}
// The lookup table is read in full whatever FirstFrame says, so it is bounded in full.
long tableEnd = (long)start + 4 + ((long)frameCount * 4);
if (tableEnd > length)
{
reason = "frame table (" + frameCount + " entries) does not fit in a "
+ length + "-byte record";
return false;
}
int check = maxFrames > 0 && maxFrames < frameCount ? maxFrames : frameCount;
for (int i = 0; i < check; i++)
{
int at = start + ReadInt32(record, start + 4 + (i * 4));
if (!FrameSane(record, length, at, i, out reason))
return false;
}
return true;
}
private static bool FrameSane(byte[] record, int length, int at, int frame, out string reason)
{
reason = null;
if (at < 0 || at + 8 > length)
{
reason = "frame " + frame + " starts at " + at + ", outside the "
+ length + "-byte record";
return false;
}
int xCenter = ReadInt16(record, at);
int yCenter = ReadInt16(record, at + 2);
int width = ReadUInt16(record, at + 4);
int height = ReadUInt16(record, at + 6);
// Frame's constructor returns before locking anything for these, so they are empty
// rather than dangerous — and an empty frame is a real thing in this format.
if (width == 0 || height == 0)
return true;
if (width > MaxArtDimension || height > MaxArtDimension)
{
reason = "frame " + frame + " declares " + width + "x" + height + ", past the "
+ MaxArtDimension + "px ceiling";
return false;
}
// Settings.PixelFormat is 16bpp and GDI+ pads each scanline to four bytes, so a row
// is `delta` ushorts wide and the locked region is height*delta of them. This is the
// same `bd.Stride >> 1` Frame computes.
int delta = (((width * 2) + 3) & ~3) >> 1;
long pixels = (long)height * delta;
long origin = (xCenter - 0x200) + ((long)((yCenter + height) - 0x200) * delta);
int cursor = at + 8;
while (true)
{
if (cursor + 4 > length)
{
reason = "frame " + frame
+ " runs off the end of the record looking for its terminator";
return false;
}
int header = ReadInt32(record, cursor);
cursor += 4;
if (header == 0x7FFF7FFF)
break;
header ^= DoubleXor;
long dy = (header >> 12) & 0x3FF;
long dx = (header >> 22) & 0x3FF;
int run = header & 0xFFF;
long first = origin + (dy * delta) + dx;
if (first < 0 || first + run > pixels)
{
reason = "frame " + frame + " writes pixels " + first + ".." + (first + run)
+ " outside its own " + pixels + "-pixel bitmap";
return false;
}
// One palette byte per pixel, read straight out of the record.
if (cursor + run > length)
{
reason = "frame " + frame + " declares a " + run
+ "-pixel run running past the record";
return false;
}
cursor += run;
}
return true;
}
private static int ReadUInt16(byte[] b, int at)
{
return b[at] | (b[at + 1] << 8);
}
private static int ReadInt16(byte[] b, int at)
{
return (short)(b[at] | (b[at + 1] << 8));
}
private static int ReadInt32(byte[] b, int at)
{
return b[at] | (b[at + 1] << 8) | (b[at + 2] << 16) | (b[at + 3] << 24);
}
/// <summary>
/// Reads a record's actual bytes so <see cref="StaticRecordSane"/> or
/// <see cref="AnimationRecordSane"/> can walk it.
///
/// Holds its own handles rather than borrowing the library's, because <c>FileIndex</c>
/// hands out the stream it decodes from and moving that stream's position underneath
/// the decoder would be its own bug. Opened <c>FileShare.ReadWrite</c> to match how
/// <c>FileIndex</c> opens the same files.
///
/// One reader serves one data file, so an animation sweep wants one per file type,
/// built from <see cref="AnimDataPath"/>.
/// </summary>
public sealed class RecordReader : IDisposable
{
private readonly FileStream _mul;
private readonly FileStream _verdata;
private byte[] _scratch = new byte[64 * 1024];
public RecordReader(string mulPath, string verdataPath)
{
_mul = Open(mulPath);
_verdata = Open(verdataPath);
}
private static FileStream Open(string path)
{
if (path == null || !File.Exists(path))
return null;
try
{
return new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
}
catch
{
return null;
}
}
/// <summary>
/// True when the record at <paramref name="at"/> is safe to hand to
/// <c>Art.GetStatic</c>. A record that cannot be read at all is reported sane —
/// <see cref="CheckEntry"/> has already judged the entry, and this must not
/// invent a second reason to refuse.
/// </summary>
public bool StaticSane(FileIndex index, int at, out string reason)
{
int length = ReadRecord(index, at, out reason);
if (length < 0)
return true;
if (length == 0)
return false;
return StaticRecordSane(_scratch, length, out reason);
}
/// <summary>
/// True when the record at <paramref name="at"/> is safe to hand to
/// <c>Animations.GetAnimation</c>. <paramref name="maxFrames"/> is how many frames
/// the caller will decode — 1 for a <c>FirstFrame</c> call, 0 for all of them.
/// </summary>
public bool AnimationSane(FileIndex index, int at, int maxFrames, out string reason)
{
int length = ReadRecord(index, at, out reason);
if (length < 0)
return true;
if (length == 0)
return false;
return AnimationRecordSane(_scratch, length, maxFrames, out reason);
}
/// <summary>
/// Reads one record into <see cref="_scratch"/>. Returns its length, 0 for a
/// failure (with <paramref name="reason"/> set), or -1 when there is nothing to
/// read at all — <see cref="CheckEntry"/> has already judged the entry, and this
/// must not invent a second reason to refuse.
/// </summary>
private int ReadRecord(FileIndex index, int at, out string reason)
{
reason = null;
if (index == null || index.Index == null || at < 0 || at >= index.Index.Length)
return -1;
Entry3D e = index.Index[at];
bool patched = (e.length & (1 << 31)) != 0;
int length = e.length & 0x7FFFFFFF;
var stream = patched ? _verdata : _mul;
if (stream == null || length <= 0 || e.lookup < 0)
return -1;
if (_scratch.Length < length)
_scratch = new byte[length];
int read;
try
{
stream.Seek(e.lookup, SeekOrigin.Begin);
read = stream.Read(_scratch, 0, length);
}
catch (Exception ex)
{
reason = "cannot read the record: " + ex.GetType().Name;
return 0;
}
// The short read the decoders discard. Refusing here is the whole point: the
// library would decode whatever the shared buffer happened to hold.
if (read < length)
{
reason = "short read — " + read + " of " + length + " bytes available";
return 0;
}
return length;
}
public void Dispose()
{
if (_mul != null)
_mul.Dispose();
if (_verdata != null)
_verdata.Dispose();
}
}
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,254 @@
using System;
using System.Collections.Generic;
using System.Globalization;
using System.Text;
using Server.Mobiles;
namespace Server.Custom.Bridge
{
/// <summary>
/// **Slug to body id — the part only the shard can do** (docs/link/v8.md §8, protocol 8,
/// phase 3).
///
/// The spawn atlas knows a creature by a **slug** derived from the class name it found in
/// `Spawns/*.xml` ("giant-spider"). The client knows the same creature by a **body id**
/// (28). Nothing in the ServUO tree declares that mapping as data. Today an operator
/// bridges it by hand, grepping `Scripts/Mobiles/Normal/&lt;Name&gt;.cs` for `Body =`,
/// which appears as a decimal, as hex (`0xD1`), as `Utility.RandomList(35, 36)` and as an
/// `m_IDs[]` table — a parse that is wrong on the shard's own custom creatures, which is
/// precisely the set an operator most wants pictures for.
///
/// Inside ServUO the problem does not exist: construct the type, read `Body.BodyID`,
/// delete it. <c>BridgeWorld.cs</c> already does exactly that for a different feature.
///
/// **This is the one asset-plane family that does NOT run on the asset worker**, and the
/// reason is the whole point of §8. Constructing and deleting a mobile is world mutation,
/// so it must happen on the Core thread — while the decode in <see cref="BridgeCatalog"/>
/// must happen off it, because it reads hundreds of megabytes and would stop the world for
/// every player on the shard. That split is why body resolution is its own request kind
/// rather than a step inside asset extraction.
///
/// Two consequences follow from answering on the Core thread, and both are bounds:
///
/// **The batch is small and the shard enforces the cap itself.** Every type constructed
/// here runs a real constructor — packing items, rolling skills, starting AI timers — and
/// all of that happens between two ticks of the world. The website chunks its own list;
/// a request over <see cref="BridgeConfig.AssetBodyBatch"/> names is **refused** rather
/// than truncated, so the two sides cannot quietly disagree about what was answered.
///
/// **It does not take the asset plane's single slot.** The slot exists to stop several
/// large replies queueing at once (§3.2); this reply is a few kilobytes and the work is
/// not on the worker, so claiming the slot would only make a body pass and a catalogue
/// page refuse each other for no benefit.
///
/// **A creature whose constructor randomises its body reports one of its variants**, not
/// an error and not a set. Constructing twice to detect that would double every side
/// effect above to learn something the bestiary does not render differently — both ids are
/// the same creature. The answer is stable enough to cache and cheap enough to redo.
/// </summary>
public static class BridgeBodies
{
public static void Initialize()
{
if (!BridgeConfig.Enabled)
return;
BridgeBoot.RegisterHandler("assets.bodies", OnBodies);
}
// ── the request ──────────────────────────────────────────────────────────────────────
private static void OnBodies(Dictionary<string, object> o)
{
var reqId = BridgeJson.GetString(o, "reqId");
if (reqId == null)
{
// Rule 1 of the asset plane: without a correlation id this reply lands on the
// event path, is persisted to the sidecar's store and broadcast to every
// subscriber. Refuse rather than answer.
BridgeAssets.Fail(null, "BAD_REQUEST", "assets.bodies requires a reqId");
return;
}
if (!BridgeConfig.AssetsEnabled)
{
BridgeAssets.Fail(reqId, "DISABLED", "asset extraction is disabled on this shard");
return;
}
var types = BridgeJson.GetStringList(o, "types");
if (types.Count == 0)
{
BridgeAssets.Fail(reqId, "BAD_REQUEST",
"assets.bodies requires a non-empty `types` array of ServUO class names");
return;
}
if (types.Count > BridgeConfig.AssetBodyBatch)
{
// Refuse, never truncate. A silently shortened answer looks identical to a
// complete one from the website's side, and the types that fell off the end would
// be recorded as "asked and unanswerable" rather than "never asked".
BridgeAssets.Fail(reqId, "BAD_REQUEST",
"assets.bodies takes at most " + BridgeConfig.AssetBodyBatch
+ " types per request (asked for " + types.Count + "); send them in chunks");
return;
}
Reply(reqId, types);
}
/// <summary>
/// Core thread. Constructs each type once, reads its body, deletes it.
///
/// Every outcome is a **row**, never a failed request: a shard is expected to be asked
/// about types it does not have (an atlas built from a tree that has since changed, a
/// spawn file naming a creature from a script package the operator removed), and a
/// status screen that fails the whole pass over one of those teaches an operator to
/// stop pressing the button.
/// </summary>
private static void Reply(string reqId, List<string> types)
{
var sb = BridgeJson.Begin("assets.bodies.ok");
sb.Str("reqId", reqId)
.Num("extractorVersion", BridgeAssets.EXTRACTOR_VERSION)
.Num("asked", types.Count);
// The envelope is shared with every other family (§3.4) even though this one never
// pages: the website drives the chunking, so `more` is always false and `cut` always
// "end". Writing it anyway means one reader shape on the other side rather than two.
var page = new BridgeAssets.PageBuilder(sb, "rows", BridgeConfig.AssetBatchBytes);
int resolved = 0;
foreach (var name in types)
{
string status;
int body;
Resolve(name, out body, out status);
if (status == "ok")
resolved++;
var item = new StringBuilder(96);
item.Append("{\"type\":");
BridgeJson.Text(item, name);
item.Append(",\"status\":\"").Append(status).Append('"');
if (status == "ok")
item.Append(",\"body\":").Append(body.ToString(CultureInfo.InvariantCulture));
item.Append('}');
// A chunk this small cannot spend the budget — the cap above is a hundred names
// and the budget is half a megabyte — but the check costs nothing and the day
// someone raises `AssetBodyBatch` it is the difference between a short page and a
// line the sidecar drops.
if (!page.TryAdd(item.ToString(), null))
break;
}
page.Close();
sb.Num("resolved", resolved);
BridgeLink.Emit(sb.End());
}
/// <summary>
/// One type name to one body id.
///
/// `status` is the field the website records, and the four values are four different
/// things an operator can act on:
///
/// <c>ok</c> — constructed, body read.
/// <c>unknown</c> — no such type on this shard. The spawn file names something the
/// scripts do not define, which is a real drift an operator wants to see.
/// <c>notCreature</c> — the type exists but is not a `BaseCreature`. Spawn files
/// legitimately name items and static decorations; those have no body and never will,
/// so this is a permanent answer rather than a retryable failure.
/// <c>failed</c> — the constructor threw, or the type has none that takes no
/// arguments. Caught per type, because one creature whose constructor depends on a
/// script package the operator removed must not cost the other ninety-nine.
/// </summary>
private static void Resolve(string name, out int body, out string status)
{
body = 0;
status = "failed";
Type type;
try
{
// `true` is ignoreCase — spawn files are hand-edited and their casing drifts from
// the class it names far more often than the name itself does.
type = ScriptCompiler.FindTypeByName(name, true);
}
catch
{
status = "failed";
return;
}
if (type == null)
{
status = "unknown";
return;
}
if (!typeof(BaseCreature).IsAssignableFrom(type) || type.IsAbstract)
{
status = "notCreature";
return;
}
BaseCreature creature = null;
try
{
creature = Activator.CreateInstance(type) as BaseCreature;
if (creature == null)
{
status = "failed";
return;
}
body = creature.Body.BodyID;
status = body > 0 ? "ok" : "failed";
}
catch (Exception e)
{
Console.WriteLine("[Bridge] assets.bodies: {0}: {1}: {2}",
name, e.GetType().Name, e.Message);
status = "failed";
}
finally
{
if (creature != null)
{
try
{
// Deleting the mobile deletes the items it packed — `Mobile.Delete` walks
// `Items`, and `Item.Delete` walks what each contains — and stops its AI
// timer. A creature left alive here is a creature standing at (0,0,0) on
// the internal map forever, saved with the world, once per import.
creature.Delete();
}
catch
{
// Nothing useful is left to do, and throwing out of `finally` would lose
// whatever the try block was already reporting.
}
}
}
}
}
}

View File

@@ -260,6 +260,10 @@ namespace Server.Custom.Bridge
e.Mobile.SendMessage("Bridge: {0}", BridgeParticipation.Status());
e.Mobile.SendMessage("Bridge: {0}", BridgeWorld.Status());
e.Mobile.SendMessage("Bridge: {0}", BridgeOneShots.Status());
e.Mobile.SendMessage("Bridge: {0}", BridgeAssets.Status());
e.Mobile.SendMessage("Bridge: {0}", BridgeCatalog.Status());
e.Mobile.SendMessage("Bridge: {0}", BridgeArt.Status());
e.Mobile.SendMessage("Bridge: {0}", BridgeTree.Status());
break;
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,772 @@
using System;
using System.Collections.Generic;
using System.Globalization;
using System.IO;
using System.Text;
using Ultima;
namespace Server.Custom.Bridge
{
/// <summary>
/// **The cliloc table, over the bridge** (docs/link/v8.md §9 — protocol 8, phase 2).
///
/// A "cliloc" is UO's localization table: an integer id mapped to a display string. Items
/// on the wire carry a `LabelNumber`, never a name, so without this table the website can
/// only render `id 1023721` where the game renders "quarter staff". The number was never
/// the missing piece; the table was.
///
/// Until this phase the operator supplied it by hand: install UOFiddler, build a converter
/// against its `Ultima.dll`, run it over their own `Cliloc.enu`, copy a 5 MB file to the
/// web host and point a setting at it. That whole pipeline existed for one reason — the
/// file is compressed and **nothing in this stack could read it**. ServUO's own bundled
/// `Ultima.StringList` implements the plain layout only and throws on a modern client's
/// file, which is also why the shard's `VendorSearch.GetItemName` has always been inert.
///
/// So this class is the one decoder protocol 8 **writes** rather than calls (§4): a port
/// of UOFiddler's Mythic decompressor into the overlay, after which the shard can read its
/// own client's table and hand it to the website over the same request/reply path as
/// everything else. The operator installs nothing.
///
/// **Attribution.** The decompression below is a port of `Ultima/Helpers/MythicDecompress`
/// and `MoveToFront` from UOFiddler (https://github.com/polserver/UOFiddler), which is
/// released under the **Beerware** licence — compatible with this tree's GPL-3.0-or-later.
/// It is rewritten for .NET Framework 4.8: the original is written against `Span&lt;T&gt;`,
/// `ArrayPool&lt;T&gt;` and `BinaryPrimitives`, none of which ServUO's `net48` target has.
///
/// **What is NOT here, deliberately.** Shard-added items carry cliloc ids no client table
/// contains, and ServUO has no server-side notion of a custom cliloc — there is nothing in
/// the tree to read. That gap is in the *game*, not in this pipeline, so the website keeps
/// its `custom/` overlay directory and merges it over whatever arrives here. This class
/// answers exactly one question: what does the client's own table say.
/// </summary>
public static class BridgeCliloc
{
/// <summary>
/// Languages this can serve.
///
/// Not an arbitrary code: <c>Ultima.Files</c> resolves only the names in its own file
/// table, and cliloc files are represented there by these four. Asking for anything
/// else cannot resolve to a path however the client is laid out, so it is refused by
/// name rather than answered with an empty table.
///
/// `custom1` / `custom2` are the *client-side* custom cliloc files a shard ships to
/// its players. Nothing on the website imports them today — its `custom/` overlay
/// directory is the supported answer — but they are the shard's files and they are
/// readable, so they are not artificially excluded.
/// </summary>
private static readonly string[] Languages = { "enu", "deu", "custom1", "custom2" };
private const string DefaultLanguage = "enu";
/// <summary>
/// How long a decoded table is kept in memory after its last page.
///
/// A stock `Cliloc.enu` decodes to ~67,000 live strings; holding that forever on a
/// shard that imports once a month is rude, and decoding it again costs about a
/// second. So it is cached only for as long as an import is plausibly still running:
/// freed when the last page is served, and expired on the next request if one never
/// comes (an import abandoned halfway leaves nothing behind).
/// </summary>
private static readonly TimeSpan CacheIdle = TimeSpan.FromMinutes(5);
private static readonly object _sync = new object();
private static Table _cached;
public static void Initialize()
{
if (!BridgeConfig.Enabled)
return;
BridgeBoot.RegisterHandler("cliloc.table", OnTable);
}
// ── the request plane ────────────────────────────────────────────────────────────────
/// <summary>
/// Core thread. Validates, then hands the decode to the asset worker — reading and
/// decompressing five megabytes is emphatically not something to do while the world
/// is waiting, and <see cref="BridgeAssets"/>'s single slot is what keeps the shard's
/// outbound queue at a depth of about one while it happens.
/// </summary>
private static void OnTable(Dictionary<string, object> o)
{
var reqId = BridgeJson.GetString(o, "reqId");
if (reqId == null)
{
// Without a correlation id this reply would land on the event path, be persisted
// to the sidecar's store and broadcast to every subscriber — a megabyte of
// strings to every connected client, forever. Refuse instead (§3.1).
BridgeAssets.Fail(null, "BAD_REQUEST", "cliloc.table requires a reqId");
return;
}
if (!BridgeConfig.AssetsEnabled)
{
BridgeAssets.Fail(reqId, "DISABLED", "asset extraction is disabled on this shard");
return;
}
var lang = BridgeJson.GetString(o, "lang");
if (String.IsNullOrEmpty(lang))
lang = DefaultLanguage;
lang = lang.ToLowerInvariant();
if (Array.IndexOf(Languages, lang) < 0)
{
BridgeAssets.Fail(reqId, "NOT_FOUND",
"no cliloc file for language '" + lang + "' (this shard can serve: "
+ String.Join(", ", Languages) + ")");
return;
}
// The cursor is this family's own resume point and it is a cliloc NUMBER, not an
// offset into anything. That matters: the cache behind it can be dropped and rebuilt
// between two pages of the same import (idle expiry, a second import, a restart), and
// an index into a list would silently mean something different afterwards. "Resume
// after id N" survives all of it, because the table is served in id order.
int after = -1;
var cursor = BridgeJson.GetString(o, "cursor");
if (!String.IsNullOrEmpty(cursor))
{
if (!TryParseCursor(cursor, out after))
{
BridgeAssets.Fail(reqId, "BAD_REQUEST", "malformed cursor: " + cursor);
return;
}
}
string language = lang;
int resumeAfter = after;
BridgeAssets.Accept(reqId, "cliloc.table", () => ReplyTable(reqId, language, resumeAfter));
}
private static bool TryParseCursor(string cursor, out int after)
{
after = -1;
if (!cursor.StartsWith("n:", StringComparison.Ordinal))
return false;
return Int32.TryParse(
cursor.Substring(2), NumberStyles.Integer, CultureInfo.InvariantCulture, out after);
}
/// <summary>
/// Asset worker. Decodes (or reuses) the table and writes one page of it.
/// </summary>
private static void ReplyTable(string reqId, string lang, int after)
{
string path = ResolvePath(lang);
if (path == null)
{
BridgeAssets.Fail(reqId, "NOT_FOUND",
"this shard's client has no cliloc." + lang + " (looked where ServUO's own "
+ "data path points)");
return;
}
Table table;
string code, reason;
if (!TryLoad(lang, path, out table, out code, out reason))
{
BridgeAssets.Fail(reqId, code, reason);
return;
}
var sb = BridgeJson.Begin("cliloc.table.ok");
sb.Str("reqId", reqId)
.Str("lang", lang)
.Num("extractorVersion", BridgeAssets.EXTRACTOR_VERSION)
.Str("file", Path.GetFileName(path))
// The website pages this table over several round trips and must be able to tell
// that the file changed underneath it — an operator patching their client mid-import
// would otherwise produce one table stitched from two, with no error anywhere. It
// compares these two fields across pages and starts over if they move.
.Num("size", table.Size)
.Num("mtime", table.MTime)
.Num("total", table.Count)
.Bool("compressed", table.Compressed);
// Before the page opens, not after it closes: PageBuilder reserves room for the
// envelope it still has to write, and a field appended past Close() is spent outside
// that reserve. It fits today by a wide margin, and it is the kind of thing the next
// family copies.
int start = table.IndexAfter(after);
sb.Num("from", start);
var page = new BridgeAssets.PageBuilder(sb, "rows", BridgeConfig.AssetBatchBytes);
int i = start;
for (; i < table.Count; i++)
{
var item = new StringBuilder(96);
item.Append("{\"n\":").Append(table.Numbers[i].ToString(CultureInfo.InvariantCulture));
item.Append(",\"f\":").Append(table.Flags[i].ToString(CultureInfo.InvariantCulture));
item.Append(",\"t\":");
BridgeJson.Text(item, table.Texts[i]);
item.Append('}');
if (!page.TryAdd(item.ToString(), "n:" + table.Numbers[i].ToString(CultureInfo.InvariantCulture)))
break;
}
page.Close();
bool finished = i >= table.Count;
BridgeLink.Emit(sb.End());
// The last page is also the end of the import, so let the strings go. A retry of that
// page re-decodes, which costs a second and happens approximately never; holding ~67k
// strings against that is the wrong trade.
if (finished)
Release(lang);
}
private static string ResolvePath(string lang)
{
try
{
// ServUO's own `Scripts/Misc/DataPath.cs` calls `Files.SetMulPath` for every
// configured data directory at Configure time, so this resolves against the
// client the SHARD is running on — including on Linux, where `Ultima.Files`'s
// registry lookup finds nothing on its own.
return Files.GetFilePath("cliloc." + lang);
}
catch
{
return null;
}
}
// ── the decoded table ────────────────────────────────────────────────────────────────
private sealed class Table
{
public string Lang;
public long Size;
public long MTime;
public bool Compressed;
public int[] Numbers;
public byte[] Flags;
public string[] Texts;
public DateTime LastUsed;
public int Count { get { return Numbers.Length; } }
/// <summary>
/// Index of the first row with a number greater than <paramref name="after"/>.
/// Binary search, because the rows are in id order by construction and a page
/// deep into the table would otherwise walk everything before it.
/// </summary>
public int IndexAfter(int after)
{
if (after < 0)
return 0;
int lo = 0, hi = Numbers.Length;
while (lo < hi)
{
int mid = lo + ((hi - lo) >> 1);
if (Numbers[mid] <= after)
lo = mid + 1;
else
hi = mid;
}
return lo;
}
}
private static bool TryLoad(string lang, string path, out Table table, out string code, out string reason)
{
code = null;
reason = null;
long size, mtime;
try
{
var info = new FileInfo(path);
size = info.Length;
mtime = (long)(info.LastWriteTimeUtc - new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc))
.TotalMilliseconds;
}
catch (Exception e)
{
table = null;
code = "UNREADABLE";
reason = "cannot stat " + Path.GetFileName(path) + ": " + e.Message;
return false;
}
lock (_sync)
{
if (_cached != null)
{
bool stale = _cached.Lang != lang
|| _cached.Size != size
|| _cached.MTime != mtime
|| DateTime.UtcNow - _cached.LastUsed > CacheIdle;
if (stale)
_cached = null;
}
if (_cached != null)
{
_cached.LastUsed = DateTime.UtcNow;
table = _cached;
return true;
}
}
byte[] raw;
try
{
raw = File.ReadAllBytes(path);
}
catch (Exception e)
{
table = null;
code = "UNREADABLE";
reason = "cannot read " + Path.GetFileName(path) + ": " + e.Message;
return false;
}
bool compressed = IsCompressed(raw);
byte[] plain;
if (compressed)
{
try
{
plain = Mythic.Decompress(raw);
}
catch (Exception e)
{
table = null;
code = "UNREADABLE";
reason = "cannot decompress " + Path.GetFileName(path) + ": " + e.Message;
return false;
}
}
else
{
plain = raw;
}
var built = new Table
{
Lang = lang,
Size = size,
MTime = mtime,
Compressed = compressed,
LastUsed = DateTime.UtcNow
};
if (!TryParseRecords(plain, built, out reason))
{
table = null;
code = "UNREADABLE";
return false;
}
lock (_sync)
{
_cached = built;
}
table = built;
return true;
}
private static void Release(string lang)
{
lock (_sync)
{
if (_cached != null && _cached.Lang == lang)
_cached = null;
}
}
/// <summary>
/// Every compressed cliloc begins with a DWORD whose high byte is <c>0x8E</c> — the
/// top byte of UOFiddler's `HeaderXorKey`, showing through because the value it hides
/// (a length) is far smaller than the key. That single byte is what tells a modern
/// client's file from the pre-2010 plain layout, and both are accepted here: a shard
/// running an old or hand-built client is not a broken shard.
/// </summary>
private static bool IsCompressed(byte[] buffer)
{
return buffer.Length >= 4 && buffer[3] == 0x8E;
}
// ── the plain layout ─────────────────────────────────────────────────────────────────
private const int HeaderBytes = 6; // int32 version + int16 language marker
private const int RecordHeaderBytes = 7; // int32 number + byte flag + uint16 length
/// <summary>
/// Parses the plain layout into the sorted, blank-free arrays the wire wants.
///
/// **Strict about truncation**, and that strictness is the point: a half-decoded table
/// is indistinguishable from a complete one downstream — you would simply see some
/// items named and some not, which is exactly what "no table at all" looks like. So a
/// record running past the end of the buffer is an error naming its offset, never a
/// short table.
///
/// **Blanks are dropped here rather than on the website.** Roughly 56,000 of a stock
/// table's 123,490 entries are empty strings the client reserves and never uses, the
/// website discards them at import already, and a row that resolves to no name is
/// indistinguishable from no row at all to every caller. Dropping them halves what
/// crosses the wire for data that would be thrown away on arrival.
///
/// **A repeated id is resolved last-wins**, matching the client's own loader (its
/// dictionary assignment overwrites). The plain format permits it, so a file the game
/// itself would load happily must not fail here.
/// </summary>
private static bool TryParseRecords(byte[] data, Table into, out string reason)
{
reason = null;
if (data.Length < HeaderBytes)
{
reason = "cliloc file is shorter than its 6-byte header";
return false;
}
var byNumber = new Dictionary<int, Entry>(140000);
int offset = HeaderBytes;
int read = 0;
while (offset < data.Length)
{
if (offset + RecordHeaderBytes > data.Length)
{
reason = "truncated record header at byte " + offset + " (" + read + " entries read)";
return false;
}
int number = ReadInt32(data, offset);
byte flag = data[offset + 4];
// Unsigned: reading this signed (as ServUO's own SDK does) turns any string over
// 32 KB into a negative length. Real tables top out around 12 KB, so it changes
// nothing today and costs nothing to get right.
int length = data[offset + 5] | (data[offset + 6] << 8);
offset += RecordHeaderBytes;
if (offset + length > data.Length)
{
reason = "truncated record body at byte " + offset + " (" + read + " entries read)";
return false;
}
string text;
try
{
text = Encoding.UTF8.GetString(data, offset, length);
}
catch (Exception e)
{
reason = "entry " + number + " at byte " + offset + " is not valid UTF-8: " + e.Message;
return false;
}
offset += length;
read++;
byNumber[number] = new Entry { Flag = flag, Text = text };
}
var numbers = new List<int>(byNumber.Count);
foreach (var pair in byNumber)
{
if (IsBlank(pair.Value.Text))
continue;
numbers.Add(pair.Key);
}
numbers.Sort();
into.Numbers = numbers.ToArray();
into.Flags = new byte[numbers.Count];
into.Texts = new string[numbers.Count];
for (int i = 0; i < numbers.Count; i++)
{
var entry = byNumber[numbers[i]];
into.Flags[i] = entry.Flag;
into.Texts[i] = entry.Text;
}
return true;
}
private struct Entry
{
public byte Flag;
public string Text;
}
private static bool IsBlank(string text)
{
if (String.IsNullOrEmpty(text))
return true;
for (int i = 0; i < text.Length; i++)
{
if (!Char.IsWhiteSpace(text[i]))
return false;
}
return true;
}
private static int ReadInt32(byte[] data, int at)
{
return data[at] | (data[at + 1] << 8) | (data[at + 2] << 16) | (data[at + 3] << 24);
}
// ── the Mythic container ─────────────────────────────────────────────────────────────
/// <summary>
/// The decompressor, ported from UOFiddler (Beerware; see this class's summary).
///
/// The container is two stages over the plain cliloc bytes, undone in reverse:
///
/// 1. A 4-byte header holding the decompressed length, XORed with `0x8E2C9A3D` —
/// which is where the `0x8E` sniff byte comes from.
/// 2. A **move-to-front** coding of…
/// 3. …a Burrows-Wheeler-style transform whose 1 KB frequency header (256 little-endian
/// counts, one per byte value) is both the table sizes and the total output length.
///
/// Rewritten against plain arrays: the upstream is `Span&lt;T&gt;`/`ArrayPool&lt;T&gt;`
/// code and ServUO targets `net48`, which has neither without a package this tree does
/// not vendor. The algorithm is unchanged, including the parts that read oddly — the
/// three-region `partial` table (counts, cursors, ends) and the symbol-table shifts are
/// the original's, deliberately, because this is a format decoder and a tidier
/// rewrite is a chance to be subtly wrong about someone else's bytes.
/// </summary>
private static class Mythic
{
private const uint HeaderXorKey = 0x8E2C9A3D;
private const int FrequencyHeaderSize = 1024; // 256 little-endian ints
public static byte[] Decompress(byte[] source)
{
if (source.Length < 4)
throw new InvalidDataException("compressed cliloc is shorter than its header");
uint declared = (uint)ReadInt32(source, 0) ^ HeaderXorKey;
if (declared == 0 || declared > Int32.MaxValue)
throw new InvalidDataException("compressed cliloc declares an impossible length");
var mtf = new byte[source.Length - 4];
MoveToFrontDecode(source, 4, mtf);
var output = new byte[(int)declared];
int written = InverseTransform(mtf, output);
if (written != (int)declared)
{
throw new InvalidDataException(
"decompressed length " + written + " does not match the declared " + declared);
}
return output;
}
private static void MoveToFrontDecode(byte[] input, int from, byte[] output)
{
var symbols = new byte[256];
for (int i = 0; i < 256; i++)
symbols[i] = (byte)i;
for (int i = 0; i < output.Length; i++)
{
int index = input[from + i];
byte symbol = symbols[index];
output[i] = symbol;
for (int j = index; j > 0; j--)
symbols[j] = symbols[j - 1];
symbols[0] = symbol;
}
}
private static int InverseTransform(byte[] input, byte[] destination)
{
if (input.Length < FrequencyHeaderSize)
throw new InvalidDataException("compressed cliloc is smaller than its frequency header");
// Three regions of 256: [0..255] the counts read from the header, [256..511] a
// moving cursor per symbol, [512..767] where that symbol's run ends.
var partial = new int[256 * 3];
for (int i = 0; i < 256; i++)
partial[i] = ReadInt32(input, i * 4);
int sum = 0;
for (int i = 0; i < 256; i++)
{
if (partial[i] < 0)
throw new InvalidDataException("compressed cliloc has a negative symbol count");
sum += partial[i];
}
if (sum == 0)
return 0;
if (destination.Length < sum)
throw new InvalidDataException("compressed cliloc's frequency header outruns its declared length");
int nonZero = 0;
for (int i = 0; i < 256; i++)
{
if (partial[i] != 0)
nonZero++;
}
var frequency = new byte[256];
Frequency(partial, frequency);
var symbols = new byte[256];
for (int i = 0; i < 256; i++)
symbols[i] = (byte)i;
for (int i = 0, m = 0; i < nonZero; ++i)
{
byte freq = frequency[i];
Need(input, m + FrequencyHeaderSize);
symbols[input[m + FrequencyHeaderSize]] = freq;
partial[freq + 256] = m + 1;
m += partial[freq];
partial[freq + 512] = m;
}
byte val = symbols[0];
int count = 0;
do
{
destination[count] = val;
if (partial[val + 256] < partial[val + 512])
{
Need(input, partial[val + 256] + FrequencyHeaderSize);
byte idx = input[partial[val + 256] + FrequencyHeaderSize];
partial[val + 256]++;
if (idx != 0)
{
ShiftLeft(symbols, idx);
symbols[idx] = val;
val = symbols[0];
}
}
else if (nonZero-- > 0)
{
ShiftLeft(symbols, nonZero);
val = symbols[0];
}
count++;
}
while (count < sum);
return sum;
}
/// <summary>
/// The upstream indexes the payload without bounds-checking it, which is safe for
/// a file the client wrote and is not safe for a file this shard was handed. A
/// truncated or hand-edited container would otherwise read whatever follows the
/// buffer in memory — or, on .NET, throw an `IndexOutOfRangeException` from inside
/// a decoder, which says nothing useful to an operator. This turns both into one
/// named, reportable failure.
/// </summary>
private static void Need(byte[] input, int at)
{
if (at < 0 || at >= input.Length)
throw new InvalidDataException("compressed cliloc ends mid-stream (wanted byte " + at + ")");
}
/// <summary>
/// Symbol values ordered by descending count — the order the coder assigned its
/// runs in. Repeated max-finding rather than a sort, as upstream: 256 passes over
/// 256 entries is nothing, and it reproduces the original's tie-breaking (the
/// lowest index wins), which a comparison sort would not.
/// </summary>
private static void Frequency(int[] counts, byte[] output)
{
var tmp = new int[256];
Array.Copy(counts, tmp, 256);
for (int i = 0; i < 256; i++)
{
int value = 0;
byte index = 0;
for (int j = 0; j < 256; j++)
{
if (tmp[j] > value)
{
index = (byte)j;
value = tmp[j];
}
}
if (value == 0)
break;
output[i] = index;
tmp[index] = 0;
}
}
private static void ShiftLeft(byte[] symbols, int upTo)
{
for (int i = 0; i < upTo; ++i)
symbols[i] = symbols[i + 1];
}
private static int ReadInt32(byte[] data, int at)
{
return data[at] | (data[at + 1] << 8) | (data[at + 2] << 16) | (data[at + 3] << 24);
}
}
}
}

View File

@@ -87,6 +87,80 @@ namespace Server.Custom.Bridge
// morning. Those are different consents, and one switch cannot express both.
public static bool EventsEnabled { get; private set; }
// ---- the asset plane (docs/link/v8.md §3, protocol 8) ----
//
// Its own gate again, and for the same reason the event plane got one: enabling this is
// an operator consenting to the WEBSITE READING THEIR CLIENT FILES -- art, animations and
// the string table, off the host's disk, over the link. That is a different consent from
// publishing world state, and one switch cannot express both. Reads only: nothing on this
// plane writes anything, anywhere.
public static bool AssetsEnabled { get; private set; }
public static int AssetBatchBytes { get; private set; }
// How many types one `assets.bodies` request may name (§8, phase 3). This is the ONLY
// asset-plane bound counted in items rather than bytes, and deliberately so: the cost it
// bounds is not the size of the reply, it is constructing and deleting that many real
// mobiles ON THE CORE THREAD, between two ticks of the world.
public static int AssetBodyBatch { get; private set; }
// How many keys one `assets.fetch` request may name. Bytes still cut the page; this only
// bounds how large a request the shard will parse and walk at all.
public static int AssetFetchKeys { get; private set; }
// The wall-clock budget for one catalogue page (§4.8, phase 3). The catalogue's rows are
// ninety bytes, so the byte budget never stops it -- but building them means decoding
// hundreds of animations, and the sidecar gives a reply ten seconds. Kept well under that,
// because the reply still has to be built, serialised and cross the wire afterwards.
public static int AssetScanMs { get; private set; }
// Which direction the catalogue renders (§5.1). Both are settings and neither is in the
// asset key, because five directions would five-fold every count in §11 to express a
// choice nobody is going to vary.
//
// The split is not arbitrary and was found by RENDERING all five rather than from a table:
// index 0 is head-on, which is what a character portrait wants and the least legible view
// there is of a four-legged creature. A wolf seen from the front is a dark blob; at index
// 1, the front three-quarter, it is unmistakably a wolf.
public static int AssetPlayerDirection { get; private set; }
public static int AssetCreatureDirection { get; private set; }
// ---- the tree plane (docs/link/v8.md §10, phase 7) ----
//
// Its OWN gate, and the third one on this link for the third kind of consent. The asset
// gate above is the operator agreeing that the website may read THEIR UO CLIENT -- art
// and animations and a string table that came from EA. This one is the operator agreeing
// that it may read THE SHARD'S OWN CONFIGURATION: the spawn files, the region and
// location definitions, the champion table, the decoration lists. Those are the
// operator's own work rather than a licensed client, and they are what the spawn atlas is
// built out of -- so a shard that declines to serve client art must still be able to
// publish where its creatures live. One switch could not have expressed both, and the
// atlas would have been the thing that silently disappeared.
//
// Reads only, and only the five labelled groups SPAWN_ATLAS.md already names. Nothing
// here joins a path the website sent: a request names a label this shard enumerated, or
// it is refused.
public static bool TreeEnabled { get; private set; }
// How much of a tree file one chunk carries, BEFORE compression (§10). The chunk is the
// thing that makes this transferable at all: a stock Spawns/trammel.xml is 4.03 MB and
// the sidecar discards any inbound line over 1 MiB, so the file as a single base64 row
// could never arrive -- it would time out and be re-requested forever, which is a failure
// with no error in it anywhere.
//
// Compression is what makes it cheap (a spawn file gzips ~18x, so a chunk is typically
// 40 KB on the wire) and the chunk is what makes it BOUNDED: gzip cannot be relied on to
// shrink anything, so the ceiling has to hold for input that does not compress at all.
// At 512 KiB a worst-case incompressible chunk is ~683 KiB of base64, which still fits
// the wire under AssetBatchBytes' deliberate factor of two.
public static int TreeChunkBytes { get; private set; }
// How many bytes of rendered item and land art the shard holds between requests (§11,
// phase 5). This is a convenience, not a store: the website keeps every picture it fetches
// and does not ask twice, so what this actually buys is the second page of a batch, a
// retry after a 425, and the same item appearing in two rows of one page. Sized so a
// full 512 KB batch and the one before it both fit with room over.
public static int AssetArtCacheBytes { get; private set; }
public static int LeaseMaxDurationSec { get; private set; }
public static int LeaseGraceSec { get; private set; }
@@ -144,6 +218,70 @@ namespace Server.Custom.Bridge
Port = Config.Get("Bridge.Port", 7788);
QueueCap = Config.Get("Bridge.QueueCap", 10000);
AssetsEnabled = Config.Get("Bridge.AssetsEnabled", true);
// The largest reply this plane will build, in ENCODED bytes -- not items, because the
// ceiling it has to live inside is a byte ceiling. Clamped to half the sidecar's 1 MiB
// inbound line cap, and the halving is load-bearing rather than cautious: a page
// always admits its first item even when that item alone exceeds the budget (the
// alternative is an oversized item being skipped forever and its family never making
// progress), so the wire must still have room for one such overshoot.
AssetBatchBytes = Config.Get("Bridge.AssetBatchBytes", 512 * 1024);
if (AssetBatchBytes < 64 * 1024)
AssetBatchBytes = 64 * 1024;
if (AssetBatchBytes > 512 * 1024)
AssetBatchBytes = 512 * 1024;
AssetBodyBatch = Config.Get("Bridge.AssetBodyBatch", 100);
if (AssetBodyBatch < 1)
AssetBodyBatch = 1;
if (AssetBodyBatch > 500)
AssetBodyBatch = 500;
AssetFetchKeys = Config.Get("Bridge.AssetFetchKeys", 2000);
if (AssetFetchKeys < 1)
AssetFetchKeys = 1;
if (AssetFetchKeys > 10000)
AssetFetchKeys = 10000;
AssetScanMs = Config.Get("Bridge.AssetScanMs", 3000);
if (AssetScanMs < 250)
AssetScanMs = 250;
// Half the sidecar's 10 s reply timeout, so the page still has time to be serialised
// and written after the scan stops. A budget set at the timeout would produce replies
// that are always thrown away.
if (AssetScanMs > 5000)
AssetScanMs = 5000;
// Clamped to 0-4: 5-7 are the client MIRRORING 1-3, which `Frame` decodes through a
// different pointer-arithmetic branch that nothing in BridgeAssetValidator has
// checked. Accepting one would hand an unverified write path a bitmap to fill.
AssetPlayerDirection = Clamp(Config.Get("Bridge.AssetPlayerDirection", 0), 0, 4);
AssetCreatureDirection = Clamp(Config.Get("Bridge.AssetCreatureDirection", 1), 0, 4);
// The floor is one batch: a cache that cannot hold the page being built evicts rows
// while they are still being written, which is a cache that costs and never pays. The
// ceiling is a game server's memory, and 64 MB of PNG is already ~34,000 sprites --
// most of this client's art, held for a working set that is measured in hundreds.
AssetArtCacheBytes = Config.Get("Bridge.AssetArtCacheBytes", 16 * 1024 * 1024);
if (AssetArtCacheBytes < AssetBatchBytes)
AssetArtCacheBytes = AssetBatchBytes;
if (AssetArtCacheBytes > 64 * 1024 * 1024)
AssetArtCacheBytes = 64 * 1024 * 1024;
TreeEnabled = Config.Get("Bridge.TreeEnabled", true);
// Floor and ceiling both matter. Below 64 KiB a stock tree is thousands of chunks and
// the per-row overhead starts to dominate the payload; above 512 KiB an incompressible
// chunk stops fitting inside the sidecar's inbound line cap, which is the one bound
// this number exists to respect. Kept equal to AssetBatchBytes' own ceiling so the two
// budgets cannot drift into disagreeing about the same wire.
TreeChunkBytes = Config.Get("Bridge.TreeChunkBytes", 512 * 1024);
if (TreeChunkBytes < 64 * 1024)
TreeChunkBytes = 64 * 1024;
if (TreeChunkBytes > 512 * 1024)
TreeChunkBytes = 512 * 1024;
StatSweepSeconds = Config.Get("Bridge.StatSweepSeconds", 30);
DecaySweepSeconds = Config.Get("Bridge.DecaySweepSeconds", 60);
EconomySweepSeconds = Config.Get("Bridge.EconomySweepSeconds", 300);
@@ -494,6 +632,14 @@ namespace Server.Custom.Bridge
return fallback;
}
private static int Clamp(int value, int min, int max)
{
if (value < min)
return min;
return value > max ? max : value;
}
public static string Describe()
{
return String.Format(

View File

@@ -0,0 +1,231 @@
using System;
using System.IO;
using System.IO.Compression;
namespace Server.Custom.Bridge
{
/// <summary>
/// **A PNG encoder that does not go through GDI+** (docs/link/v8.md §4.4, §4.9 — phase 4).
///
/// <see cref="BridgeUop"/> decodes into a <c>ushort[]</c> of ARGB1555 rather than into a
/// <c>Bitmap</c>, which is the whole point of §4.4's note that the UOP reader is written
/// without <c>System.Drawing</c>: libgdiplus was archived in March 2025, and every line of
/// extraction that does not depend on it is a line that survives its absence. That leaves
/// the encode, and <c>Bitmap.Save(…, ImageFormat.Png)</c> is GDI+ too — so this is the
/// other half.
///
/// It is deliberately the smallest thing that produces a correct file: 8-bit RGBA, one
/// IDAT, filter type 0 on every row. No interlacing, no palette, no colour-type choice, no
/// filter heuristics. A sprite is a few hundred pixels across and the bytes go straight
/// into a base64 field; the compression difference between this and a tuned encoder is a
/// rounding error against the wire, and every knob not turned is a way this cannot be
/// subtly wrong.
///
/// Phase 3's <c>BridgeCatalog.ToPng</c> is left exactly as it is. It is measured, shipped,
/// and its input really is a <c>Bitmap</c> from the vendored decoder — a path that needs
/// GDI+ to produce the pixels in the first place, so encoding them without it buys nothing.
/// </summary>
public static class BridgePng
{
private static readonly byte[] Signature =
{
0x89, (byte)'P', (byte)'N', (byte)'G', 0x0D, 0x0A, 0x1A, 0x0A
};
private static readonly uint[] CrcTable = BuildCrcTable();
private static readonly byte[] Empty = new byte[0];
/// <summary>
/// ARGB1555 to an RGBA8 PNG with a transparent background.
///
/// The expansion is the same one <c>BridgeCatalog.ToPng</c> documents and for the same
/// reason: alpha bit clear is fully transparent, and each 5-bit channel is widened by
/// repeating its high bits — <c>(c &lt;&lt; 3) | (c &gt;&gt; 2)</c>, not a plain shift,
/// which would cap white at 248 and tint every sprite.
/// </summary>
public static byte[] FromArgb1555(ushort[] pixels, int width, int height)
{
if (pixels == null || width <= 0 || height <= 0)
return null;
if ((long)width * height > pixels.Length)
return null;
// One filter byte per row, then RGBA per pixel. This is the PNG "raw" stream, the
// thing that gets deflated. Bounded by the caller's dimension ceiling
// (BridgeAssetValidator.MaxArtDimension), so the arithmetic cannot overflow an int —
// the check is here anyway, because that ceiling lives in another file.
long size = (((long)width * 4) + 1) * height;
if (size > Int32.MaxValue / 2)
return null;
var raw = new byte[size];
int at = 0;
for (int y = 0; y < height; y++)
{
raw[at++] = 0; // filter: None
int row = y * width;
for (int x = 0; x < width; x++)
{
int p = pixels[row + x];
if ((p & 0x8000) == 0)
{
at += 4; // already zero: transparent black
continue;
}
int r = (p >> 10) & 0x1F;
int g = (p >> 5) & 0x1F;
int b = p & 0x1F;
raw[at++] = (byte)((r << 3) | (r >> 2));
raw[at++] = (byte)((g << 3) | (g >> 2));
raw[at++] = (byte)((b << 3) | (b >> 2));
raw[at++] = 0xFF;
}
}
using (var ms = new MemoryStream(raw.Length / 2))
{
ms.Write(Signature, 0, Signature.Length);
var header = new byte[13];
WriteBigEndian(header, 0, (uint)width);
WriteBigEndian(header, 4, (uint)height);
header[8] = 8; // bit depth
header[9] = 6; // colour type: truecolour with alpha
header[10] = 0; // compression: deflate
header[11] = 0; // filter method 0
header[12] = 0; // no interlace
WriteChunk(ms, "IHDR", header, 0, header.Length);
byte[] deflated = Zlib(raw);
WriteChunk(ms, "IDAT", deflated, 0, deflated.Length);
WriteChunk(ms, "IEND", Empty, 0, 0);
return ms.ToArray();
}
}
/// <summary>
/// A zlib stream around .NET Framework's raw-deflate-only <c>DeflateStream</c>: the
/// two-byte header PNG requires, the deflate data, and the adler32 trailer computed
/// here because nothing in the framework will do it. Written by hand for exactly the
/// same reason <see cref="BridgeUop"/> reads one by hand — net48 exposes deflate and
/// calls it zlib, and the two are not the same format.
/// </summary>
private static byte[] Zlib(byte[] data)
{
using (var ms = new MemoryStream(data.Length / 2))
{
// CMF 0x78 (deflate, 32K window) and FLG 0x9C (default level, no dictionary):
// 0x789C is the pair whose value is divisible by 31, which is the check a decoder
// applies.
ms.WriteByte(0x78);
ms.WriteByte(0x9C);
using (var deflate = new DeflateStream(ms, CompressionMode.Compress, true))
deflate.Write(data, 0, data.Length);
uint adler = Adler32(data);
ms.WriteByte((byte)(adler >> 24));
ms.WriteByte((byte)(adler >> 16));
ms.WriteByte((byte)(adler >> 8));
ms.WriteByte((byte)adler);
return ms.ToArray();
}
}
private static void WriteChunk(Stream to, string type, byte[] data, int offset, int length)
{
var head = new byte[8];
WriteBigEndian(head, 0, (uint)length);
head[4] = (byte)type[0];
head[5] = (byte)type[1];
head[6] = (byte)type[2];
head[7] = (byte)type[3];
to.Write(head, 0, head.Length);
if (length > 0)
to.Write(data, offset, length);
// The CRC covers the type and the data, and not the length.
uint crc = Crc32(head, 4, 4, 0xFFFFFFFF);
if (length > 0)
crc = Crc32(data, offset, length, crc);
crc ^= 0xFFFFFFFF;
var tail = new byte[4];
WriteBigEndian(tail, 0, crc);
to.Write(tail, 0, tail.Length);
}
private static void WriteBigEndian(byte[] into, int at, uint value)
{
into[at] = (byte)(value >> 24);
into[at + 1] = (byte)(value >> 16);
into[at + 2] = (byte)(value >> 8);
into[at + 3] = (byte)value;
}
private static uint[] BuildCrcTable()
{
var table = new uint[256];
for (uint n = 0; n < 256; n++)
{
uint c = n;
for (int k = 0; k < 8; k++)
c = (c & 1) != 0 ? 0xEDB88320 ^ (c >> 1) : c >> 1;
table[n] = c;
}
return table;
}
private static uint Crc32(byte[] data, int offset, int length, uint crc)
{
for (int i = 0; i < length; i++)
crc = CrcTable[(crc ^ data[offset + i]) & 0xFF] ^ (crc >> 8);
return crc;
}
private static uint Adler32(byte[] data)
{
const uint Mod = 65521;
uint a = 1, b = 0;
for (int i = 0; i < data.Length; i++)
{
a = (a + data[i]) % Mod;
b = (b + a) % Mod;
}
return (b << 16) | a;
}
}
}

View File

@@ -0,0 +1,775 @@
using System;
using System.Collections.Generic;
using System.Globalization;
using System.IO;
using System.IO.Compression;
using System.Text;
namespace Server.Custom.Bridge
{
/// <summary>
/// **The shard's own configuration, over the bridge** (docs/link/v8.md §10 — protocol 8,
/// phase 7).
///
/// Everything else on the asset plane reads the operator's UO CLIENT. This family reads
/// the shard's own files: the spawn tables, the region and location definitions, the
/// champion list and the decoration lists. The website parses those into its spawn atlas —
/// where every creature lives, which regions exist, what this shard calls scenery — and
/// until protocol 8 it did so by **reading the ServUO tree off a shared filesystem**:
/// same host, a bind mount, or a shared volume.
///
/// That was the one place the platform's own rule was broken, and broken by the component
/// that faces the internet. This closes it. The parsers do not move — `spawnAtlasParse.js`
/// is pure, fs-free and covered by CI without a ServUO tree anywhere near it, and every
/// quirk it handles stays exactly where it is. The shard sends bytes; the website still
/// decides what they mean.
///
/// ── What phase 7 measured, and the shape it forced ────────────────────────────────
///
/// §10 said "the shard serves `tree/&lt;label&gt;` → bytes". Measured against a stock 57.4
/// tree, it cannot: `Spawns/trammel.xml` is **4.03 MB**, the sidecar discards any inbound
/// line over **1 MiB** (`shard.rs` `MAX_INBOUND_LINE_BYTES`), and that file as a single
/// base64 row is 5.4 MiB. It would never arrive — the reply would be discarded, the
/// request would time out, and the import would retry forever with no error anywhere in
/// it. Two files on a *stock* tree are in that state; a shard with hand-built spawn tables
/// has more.
///
/// So a file crosses as **chunks, each gzipped**:
///
/// <code>
/// tree/Spawns/trammel.xml the manifest row — size, hash, chunk count
/// tree/Spawns/trammel.xml/c0 the first 512 KiB of it, gzipped
/// tree/Spawns/trammel.xml/c1 the next
/// </code>
///
/// which is §5's depth scheme at work a second time, exactly as `body/400/a0/f0` is —
/// and, as there, nothing about it needed a protocol change.
///
/// **The chunk is the bound and the compression is the saving**, and it matters which is
/// which. Compression is what makes this cheap: the stock tree is 11.34 MB and gzips to
/// 927 KB, so the whole atlas source arrives in about three pages instead of thirty-one.
/// But nothing guarantees that an operator's files compress at all, so the ceiling has to
/// hold when they do not — and it does, because a 512 KiB chunk that refuses to compress
/// is still only ~683 KiB of base64, inside the wire cap that
/// <see cref="BridgeConfig.AssetBatchBytes"/>' deliberate factor of two leaves room for.
/// A design that leaned on the ratio would work on every tree anyone tested and fail on
/// the first one nobody did.
///
/// ── Two rules that are not negotiable here ────────────────────────────────────────
///
/// **1. The label set is this shard's, never the caller's.** This is the only family on
/// this link whose keys look like paths, and the website is the internet-facing component.
/// So nothing here joins a path that arrived on the wire: a fetch resolves its label
/// against the set <see cref="Enumerate"/> itself produced, and a label that is not in it
/// is refused — before any file is opened, and whatever it spells. The five groups are
/// fixed in code, the extensions are fixed in code, and the resolved path is checked to be
/// under the tree root even after all of that.
///
/// **2. A row re-declares its own address.** Each chunk carries its label, its index, its
/// byte offset and the hash of its own (uncompressed) bytes, and the manifest carries the
/// hash of the whole file. That is the §4.10 lesson on a fourth axis: a reassembly that
/// silently put chunk 3 where chunk 4 belongs would produce a file that parses — XML is
/// forgiving about what it skips — and a spawn atlas subtly missing a facet. Per-chunk
/// hashes make it a named error instead.
/// </summary>
public static class BridgeTree
{
/// <summary>The §5 key family this serves.</summary>
private const string Family = "tree";
/// <summary>
/// The five labelled groups `spawnAtlasSource.js` reads, and nothing else.
///
/// Fixed in code rather than configured, because a configurable list is a way for the
/// website to ask for a file this shard never meant to publish. An operator who wants
/// a different tree served wants a different feature.
/// </summary>
private static readonly string[] SingleFiles =
{
"Data/Regions.xml",
"Config/ChampionSpawns.xml"
};
private const string LocationsDir = "Data/Locations";
private const string SpawnsDir = "Spawns";
private const string DecorationDir = "Data/Decoration";
public static void Initialize()
{
if (!BridgeConfig.Enabled)
return;
// Its own consent, not the asset plane's (§10, phase 7). An operator who declines to
// serve their UO client still gets a spawn atlas, because these are their own files.
BridgeAssets.RegisterFamily(Family, ReplyFetch, ReplyManifest,
() => BridgeConfig.TreeEnabled,
"the shard's configuration tree is not served (Bridge.TreeEnabled is off)");
}
// ── the file set ─────────────────────────────────────────────────────────────────────
private sealed class TreeFile
{
public string Label;
public string Path;
public long Bytes;
public long MTime;
}
/// <summary>
/// Every atlas source file this shard has, tree-relative and forward-slashed.
///
/// The labels are `spawnAtlasSource.js`'s own, character for character, because they
/// are what the website keys its stored fingerprint on: the same tree read here and
/// read there has to produce the same label or every import looks like a change.
/// Forward slashes for the same reason — a Windows shard and a Linux one must agree.
/// </summary>
private static List<TreeFile> Enumerate()
{
string root = Core.BaseDirectory;
var files = new List<TreeFile>();
foreach (string label in SingleFiles)
Add(files, root, label);
foreach (string label in ListByExtension(root, LocationsDir, ".xml"))
Add(files, root, label);
foreach (string label in ListByExtension(root, SpawnsDir, ".xml"))
Add(files, root, label);
foreach (string label in ListTree(root, DecorationDir, ".cfg"))
Add(files, root, label);
return files;
}
private static void Add(List<TreeFile> files, string root, string label)
{
string path = Resolve(root, label);
if (path == null)
return;
try
{
var info = new FileInfo(path);
if (!info.Exists)
return;
files.Add(new TreeFile
{
Label = label,
Path = path,
Bytes = info.Length,
MTime = ToUnixMs(info.LastWriteTimeUtc)
});
}
catch (Exception e)
{
// A file the shard cannot stat is a file it cannot serve. Say so once, here,
// rather than as a refused row on every import pass forever.
Console.WriteLine("[Bridge] tree: cannot read {0}: {1}", label, e.Message);
}
}
/// <summary>One directory's files with the given extension, sorted, as labels.</summary>
private static List<string> ListByExtension(string root, string dir, string extension)
{
var labels = new List<string>();
string full = Path.Combine(root, dir.Replace('/', Path.DirectorySeparatorChar));
try
{
if (!Directory.Exists(full))
return labels;
foreach (string path in Directory.GetFiles(full))
{
string name = Path.GetFileName(path);
if (name.EndsWith(extension, StringComparison.OrdinalIgnoreCase))
labels.Add(dir + "/" + name);
}
}
catch (Exception e)
{
Console.WriteLine("[Bridge] tree: cannot list {0}: {1}", dir, e.Message);
}
labels.Sort(StringComparer.Ordinal);
return labels;
}
/// <summary>
/// One directory tree's files with the given extension, recursively.
///
/// Recursive because `Data/Decoration` nests two deep in places (`Magincia/Trammel`,
/// `Stygian Abyss/Ter Mur`, `Old/Britannia`), and the website's own reader says why
/// that matters: a flat read indexes a third of what the shard has, and the failure is
/// an authoring dropdown quietly missing whole expansions rather than an error anyone
/// would notice.
/// </summary>
private static List<string> ListTree(string root, string dir, string extension)
{
var labels = new List<string>();
string full = Path.Combine(root, dir.Replace('/', Path.DirectorySeparatorChar));
try
{
if (!Directory.Exists(full))
return labels;
foreach (string path in Directory.GetFiles(full, "*", SearchOption.AllDirectories))
{
if (!path.EndsWith(extension, StringComparison.OrdinalIgnoreCase))
continue;
string rel = path.Substring(full.Length).Replace('\\', '/').TrimStart('/');
if (rel.Length > 0)
labels.Add(dir + "/" + rel);
}
}
catch (Exception e)
{
Console.WriteLine("[Bridge] tree: cannot walk {0}: {1}", dir, e.Message);
}
labels.Sort(StringComparer.Ordinal);
return labels;
}
/// <summary>
/// A label to a path on this host, or null if it is not one this shard serves.
///
/// Rule 1 of the class doc lives here. The label has already been matched against the
/// enumerated set by the time a fetch calls this, and this still refuses anything with
/// a traversal segment, a drive or a root in it, and still checks that what
/// <c>Path.GetFullPath</c> produced is under the tree root. Three checks for one rule
/// because the cost of being wrong once is the website reading an arbitrary file off a
/// game server's disk.
/// </summary>
private static string Resolve(string root, string label)
{
if (String.IsNullOrEmpty(label) || label.IndexOf('\\') >= 0)
return null;
string[] segments = label.Split('/');
foreach (string segment in segments)
{
if (segment.Length == 0 || segment == "." || segment == "..")
return null;
}
if (Path.IsPathRooted(label))
return null;
try
{
string rootFull = Path.GetFullPath(root);
string full = Path.GetFullPath(Path.Combine(rootFull,
label.Replace('/', Path.DirectorySeparatorChar)));
if (!rootFull.EndsWith(Path.DirectorySeparatorChar.ToString(CultureInfo.InvariantCulture),
StringComparison.Ordinal))
{
rootFull += Path.DirectorySeparatorChar;
}
return full.StartsWith(rootFull, StringComparison.OrdinalIgnoreCase) ? full : null;
}
catch
{
return null;
}
}
// ── the fingerprint ──────────────────────────────────────────────────────────────────
/// <summary>
/// What the whole tree currently is, in sixteen hex characters.
///
/// The same job <c>BridgeCatalog.SourceId</c> does for client files, and the same
/// reason: it goes on every page of a walk, and a page whose id differs from the
/// first's means the operator edited a spawn file while it was being read. Half of
/// what arrived then describes a tree that no longer exists and nothing later can tell
/// which half, so the website refuses the import outright rather than stitching one.
///
/// Built from (label, size, mtime) rather than from content hashes, because it is
/// computed on every page and hashing the tree's contents each time would spend a
/// tenth of a second per page to answer a question (size, mtime) answers for free.
/// The CONTENT hashes are still sent — once, per file, on the manifest — which is
/// where the website's own drift gate reads them from.
/// </summary>
private static string FingerprintOf(List<TreeFile> files)
{
var sb = new StringBuilder(256);
sb.Append(files.Count);
foreach (TreeFile file in files)
{
sb.Append('|').Append(file.Label)
.Append(':').Append(file.Bytes.ToString(CultureInfo.InvariantCulture))
.Append(':').Append(file.MTime.ToString(CultureInfo.InvariantCulture));
}
return BridgeAssets.Sha256Hex(Encoding.UTF8.GetBytes(sb.ToString())).Substring(0, 16);
}
// ── assets.manifest, for this family ─────────────────────────────────────────────────
/// <summary>
/// Worker thread. Every file this shard would serve, with its size, its content hash
/// and how many chunks it takes — and no bytes.
///
/// That separation is what makes the normal case free. The website stores these
/// hashes; on the next import it asks for this list again, compares, and fetches
/// nothing at all when nothing moved — which on a shard whose maps are not being
/// edited is every import.
///
/// A stock tree is 141 rows and fits in one page comfortably. It pages anyway, by the
/// same envelope as every other family, because the day a shard has three thousand
/// decoration files is not the day to discover this was the one walk that could not
/// end.
/// </summary>
private static void ReplyManifest(string reqId, string cursor)
{
List<TreeFile> files = Enumerate();
string fingerprint = FingerprintOf(files);
int from = ParseCursor(cursor);
if (from < 0 || from > files.Count)
from = 0;
var sb = BridgeJson.Begin("assets.manifest.ok");
sb.Str("reqId", reqId)
.Str("family", Family)
.Str("catalog", fingerprint)
.Num("chunkBytes", BridgeConfig.TreeChunkBytes)
.Num("total", files.Count)
.Num("from", from);
var page = new BridgeAssets.PageBuilder(sb, "rows", BridgeConfig.AssetBatchBytes);
int i = from;
for (; i < files.Count; i++)
{
TreeFile file = files[i];
string hash = HashFile(file.Path);
var item = new StringBuilder(256);
item.Append("{\"key\":");
BridgeJson.Text(item, Family + "/" + file.Label);
item.Append(",\"label\":");
BridgeJson.Text(item, file.Label);
item.Append(",\"bytes\":").Append(file.Bytes.ToString(CultureInfo.InvariantCulture));
item.Append(",\"mtime\":").Append(file.MTime.ToString(CultureInfo.InvariantCulture));
item.Append(",\"chunks\":").Append(
ChunkCount(file.Bytes).ToString(CultureInfo.InvariantCulture));
item.Append(",\"sha256\":");
BridgeJson.Text(item, hash);
item.Append('}');
if (!page.TryAdd(item.ToString(), "t:" + (i + 1).ToString(CultureInfo.InvariantCulture)))
break;
}
page.Close();
sb.Num("sent", page.Count);
BridgeLink.Emit(sb.End());
}
/// <summary>
/// How many chunks a file of this size takes.
///
/// **An empty file is one chunk, not none.** `Data/Locations` can legitimately hold an
/// empty file, and zero chunks would make it a manifest row the website could never
/// fetch: it would wait for content that has no address, and report the import
/// incomplete forever.
/// </summary>
private static int ChunkCount(long bytes)
{
long chunk = BridgeConfig.TreeChunkBytes;
long count = (bytes + chunk - 1) / chunk;
return count < 1 ? 1 : (int)count;
}
// ── assets.fetch, for this family ────────────────────────────────────────────────────
/// <summary>
/// Worker thread. The bytes for an explicit list of chunk keys.
///
/// Chunks are read with a seek rather than by holding the file, so the memory this
/// costs a running game server is one chunk regardless of how large an operator's
/// spawn tables are. A 4 MB file served eight times over is eight seeks and eight
/// 512 KiB reads — cheaper than caching it would be, and with no cache to invalidate
/// when the operator edits it mid-pass.
/// </summary>
private static void ReplyFetch(string reqId, List<string> keys, string expected, string cursor)
{
List<TreeFile> files = Enumerate();
string fingerprint = FingerprintOf(files);
// Shared with every other family on this plane, because an absent fingerprint and an
// empty one have to mean the same thing here and there — see
// `BridgeAssets.CatalogMismatch` for what treating them differently costs.
if (BridgeAssets.CatalogMismatch(expected, fingerprint))
{
// The tree moved between the manifest and this fetch. The same refusal the
// catalogue makes for a patched client, and for the same reason: these keys were
// chosen against a listing that no longer describes what is on disk.
BridgeAssets.Fail(reqId, "UNREADABLE",
"the shard's configuration tree changed since that manifest was read (catalog "
+ expected + " is now " + fingerprint + "); start the import again");
return;
}
var byLabel = new Dictionary<string, TreeFile>(StringComparer.Ordinal);
foreach (TreeFile file in files)
byLabel[file.Label] = file;
int from = ParseCursor(cursor);
if (from < 0 || from > keys.Count)
from = 0;
var sb = BridgeJson.Begin("assets.fetch.ok");
sb.Str("reqId", reqId)
.Str("family", Family)
.Str("catalog", fingerprint)
.Num("chunkBytes", BridgeConfig.TreeChunkBytes)
.Num("asked", keys.Count)
.Num("from", from);
var page = new BridgeAssets.PageBuilder(sb, "rows", BridgeConfig.AssetBatchBytes);
int i = from;
for (; i < keys.Count; i++)
{
string item = Render(byLabel, keys[i]);
if (!page.TryAdd(item, "t:" + (i + 1).ToString(CultureInfo.InvariantCulture)))
break;
}
page.Close();
sb.Num("sent", page.Count);
BridgeLink.Emit(sb.End());
}
/// <summary>
/// One key to one row.
///
/// A key this shard cannot serve is a row rather than a failed request, exactly as in
/// every other family, and `status` keeps the two kinds apart: `absent` is a file this
/// shard does not have (a tree with no `ChampionSpawns.xml` is a normal tree), and
/// `unsupported` is a key shape this family does not serve — which is a website bug,
/// and is counted separately so it cannot hide inside the expected gaps.
/// </summary>
private static string Render(Dictionary<string, TreeFile> byLabel, string key)
{
string label;
int chunk;
if (!ParseKey(key, out label, out chunk))
return Refusal(key, "unsupported", "not a tree chunk key (tree/<label>/c<n>)");
TreeFile file;
if (!byLabel.TryGetValue(label, out file))
{
// Rule 1: the label has to be one THIS shard enumerated. Anything else is refused
// here, before a path is built out of it, whatever it spells.
return Refusal(key, "absent", "this shard does not serve that file");
}
int chunks = ChunkCount(file.Bytes);
if (chunk < 0 || chunk >= chunks)
{
return Refusal(key, "unsupported",
"chunk " + chunk.ToString(CultureInfo.InvariantCulture) + " of "
+ chunks.ToString(CultureInfo.InvariantCulture));
}
long offset = (long)chunk * BridgeConfig.TreeChunkBytes;
byte[] raw;
try
{
raw = ReadChunk(file.Path, offset, BridgeConfig.TreeChunkBytes);
}
catch (Exception e)
{
Console.WriteLine("[Bridge] tree: cannot read {0} chunk {1}: {2}", label, chunk, e.Message);
return Refusal(key, "absent", e.GetType().Name);
}
byte[] packed;
try
{
packed = Gzip(raw);
}
catch (Exception e)
{
Console.WriteLine("[Bridge] tree: cannot compress {0} chunk {1}: {2}", label, chunk, e.Message);
return Refusal(key, "absent", e.GetType().Name);
}
var item = new StringBuilder(packed.Length * 2);
item.Append("{\"key\":");
BridgeJson.Text(item, key);
item.Append(",\"status\":\"ok\",\"label\":");
BridgeJson.Text(item, label);
item.Append(",\"chunk\":").Append(chunk.ToString(CultureInfo.InvariantCulture));
item.Append(",\"chunks\":").Append(chunks.ToString(CultureInfo.InvariantCulture));
item.Append(",\"offset\":").Append(offset.ToString(CultureInfo.InvariantCulture));
item.Append(",\"bytes\":").Append(raw.Length.ToString(CultureInfo.InvariantCulture));
item.Append(",\"sha256\":");
BridgeJson.Text(item, BridgeAssets.Sha256Hex(raw));
item.Append(",\"gzip\":");
BridgeJson.Text(item, Convert.ToBase64String(packed));
item.Append('}');
return item.ToString();
}
private static string Refusal(string key, string status, string reason)
{
var item = new StringBuilder(128);
item.Append("{\"key\":");
BridgeJson.Text(item, key);
item.Append(",\"status\":");
BridgeJson.Text(item, status);
item.Append(",\"reason\":");
BridgeJson.Text(item, reason);
item.Append('}');
return item.ToString();
}
/// <summary>
/// `tree/&lt;label&gt;/c&lt;n&gt;` into its label and chunk index.
///
/// The label itself contains slashes, so the chunk segment is taken off the END rather
/// than by counting segments from the front. That is unambiguous here and not by
/// luck: every label this family serves ends in `.xml` or `.cfg`, so no label's last
/// segment can be spelled `c` followed by digits.
/// </summary>
private static bool ParseKey(string key, out string label, out int chunk)
{
label = null;
chunk = -1;
if (String.IsNullOrEmpty(key))
return false;
string prefix = Family + "/";
if (!key.StartsWith(prefix, StringComparison.Ordinal))
return false;
int slash = key.LastIndexOf('/');
if (slash <= prefix.Length - 1)
return false;
string last = key.Substring(slash + 1);
if (last.Length < 2 || last[0] != 'c')
return false;
for (int i = 1; i < last.Length; i++)
{
if (last[i] < '0' || last[i] > '9')
return false;
}
if (!Int32.TryParse(last.Substring(1), NumberStyles.None, CultureInfo.InvariantCulture, out chunk))
return false;
label = key.Substring(prefix.Length, slash - prefix.Length);
return label.Length > 0;
}
private static int ParseCursor(string cursor)
{
if (String.IsNullOrEmpty(cursor) || !cursor.StartsWith("t:", StringComparison.Ordinal))
return 0;
int value;
return Int32.TryParse(cursor.Substring(2), NumberStyles.None,
CultureInfo.InvariantCulture, out value) ? value : 0;
}
// ── bytes ────────────────────────────────────────────────────────────────────────────
private static byte[] ReadChunk(string path, long offset, int length)
{
using (var stream = new FileStream(path, FileMode.Open, FileAccess.Read,
FileShare.ReadWrite, 1 << 16))
{
long remaining = stream.Length - offset;
if (remaining < 0)
remaining = 0;
if (remaining > length)
remaining = length;
var buffer = new byte[remaining];
stream.Seek(offset, SeekOrigin.Begin);
int filled = 0;
while (filled < buffer.Length)
{
int read = stream.Read(buffer, filled, buffer.Length - filled);
// A short read is not the end of the file here — the length was taken from the
// stream itself. Stopping on one would hand back a chunk whose declared length
// and real length disagree, which the website would only see as a hash
// mismatch on a file it cannot name a cause for.
if (read <= 0)
break;
filled += read;
}
if (filled == buffer.Length)
return buffer;
var exact = new byte[filled];
Buffer.BlockCopy(buffer, 0, exact, 0, filled);
return exact;
}
}
/// <summary>
/// A complete gzip member for exactly one empty chunk.
///
/// **`GZipStream` writes NOTHING for zero bytes of input**, on .NET Framework and on
/// Mono: the gzip header is emitted lazily on the first write, so a stream that is
/// opened and closed without one produces a zero-length buffer rather than the 20-byte
/// empty member. That is not a valid gzip stream, and the reader at the other end says
/// so — `zlib: unexpected end of file`.
///
/// It is not a hypothetical: **stock ServUO 57.4 ships two empty decoration files**
/// (`Felucca/ambitious solen queen quest.cfg` and
/// `Tokuno/terrible hatchlings quest.cfg`), so every import off an untouched tree hit
/// it. Worth knowing how it was found, because it says something about probes: an
/// offline harness reassembled all 141 files and reported success, since .NET's own
/// decompressor treats an empty stream as empty data and the chunk's declared length
/// (0) and hash (of nothing) both agreed with that. Only the live walk, through a
/// reader on a different runtime, disagreed.
///
/// The alternative — letting an empty chunk carry an empty payload and teaching the
/// reader to expect it — was rejected: it puts a special case on the wire, where every
/// future reader has to know it, instead of in the one place that builds the bytes.
/// Header (magic, deflate, no flags, no mtime, no XFL, unknown OS), one empty stored
/// block, then CRC32 and ISIZE of nothing.
/// </summary>
private static readonly byte[] EmptyGzip =
{
0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xff,
0x03, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
};
private static byte[] Gzip(byte[] raw)
{
if (raw.Length == 0)
return EmptyGzip;
using (var ms = new MemoryStream())
{
using (var gz = new GZipStream(ms, CompressionMode.Compress, true))
gz.Write(raw, 0, raw.Length);
return ms.ToArray();
}
}
/// <summary>
/// The content hash of one file, streamed.
///
/// Streamed rather than <c>File.ReadAllBytes</c> because this runs once per file per
/// manifest, and a stock tree's spawn files are 10 MB between them: reading them whole
/// would put that much through a game server's large object heap to produce 141 short
/// strings.
/// </summary>
private static string HashFile(string path)
{
try
{
using (var sha = System.Security.Cryptography.SHA256.Create())
using (var stream = new FileStream(path, FileMode.Open, FileAccess.Read,
FileShare.ReadWrite, 1 << 16))
{
var buffer = new byte[1 << 16];
int read;
while ((read = stream.Read(buffer, 0, buffer.Length)) > 0)
sha.TransformBlock(buffer, 0, read, null, 0);
sha.TransformFinalBlock(buffer, 0, 0);
var sb = new StringBuilder(64);
foreach (byte b in sha.Hash)
sb.Append(b.ToString("x2", CultureInfo.InvariantCulture));
return sb.ToString();
}
}
catch (Exception e)
{
Console.WriteLine("[Bridge] tree: cannot hash {0}: {1}", path, e.Message);
return null;
}
}
private static long ToUnixMs(DateTime utc)
{
return (long)(utc - new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc)).TotalMilliseconds;
}
/// <summary>For `[Bridge] status`, the same one-line shape every other family reports.</summary>
public static string Status()
{
if (!BridgeConfig.TreeEnabled)
return "tree(disabled)";
List<TreeFile> files = Enumerate();
long bytes = 0;
foreach (TreeFile file in files)
bytes += file.Bytes;
return String.Format("tree(files={0} bytes={1} catalog={2})",
files.Count, bytes, FingerprintOf(files));
}
}
}

View File

@@ -0,0 +1,823 @@
using System;
using System.Collections.Generic;
using System.Globalization;
using System.IO;
using System.IO.Compression;
using System.Text;
using Ultima;
namespace Server.Custom.Bridge
{
/// <summary>
/// **The UOP animation reader** (docs/link/v8.md §4.3, §4.9 — protocol 8, phase 4): the
/// second and last decoder this protocol writes rather than calls.
///
/// ServUO's vendored <c>Ultima.Animations</c> reads legacy <c>anim*.mul</c> only — it
/// constructs its five <c>FileIndex</c>es with the four-argument constructor, which passes
/// <c>uopFile: null</c>, so <c>AnimationFrame*.uop</c> is never opened. Everything a
/// modern client added there is invisible to it. This class opens those packages directly.
///
/// ── **Why this is not the never-sweep rule being broken** ──
///
/// §4.3's rule is that a body's file type comes from <c>BodyConverter.Convert</c> and is
/// never guessed, because asking another <c>anim*.mul</c> for an index it does not own
/// returns a decodable picture of something else — a giant spider on the gargoyle page.
/// That rule exists because a legacy index is addressed **by position**: nothing in the
/// file says which body a record belongs to.
///
/// A UOP package is addressed by the **hash of a name that contains the body id**
/// (<c>build/animationlegacyframe/000666/00.bin</c>). Looking in all five packages for one
/// hash is therefore not a sweep — a hit is proof of identity, not a coincidence of
/// position, and the payload repeats the body id in its own header for us to check against.
/// Measured on this machine's client: 10,724 entries across the five packages, every one
/// of them claimed by that name scheme, and **no hash appears in more than one package**.
///
/// ── **Validate as we go, because here we are the library** ──
///
/// §4.5's rule is "validate before calling", and it exists because <c>Ultima</c>'s decoders
/// take their bounds from the file they are reading. Nothing about this code can be
/// validated from outside — it *is* the decode — so the same discipline appears as a bound
/// on every read: the block chain against the file length, an entry's record against the
/// file, the inflated length against the declared one, the frame table against the
/// payload, and every run header against **both** the record's remaining bytes and the
/// bitmap it is writing into. A record that fails any of them is reported absent and no
/// pixel of it is kept.
///
/// Measured the same way §4.5 was, which is the only measurement that says the boundary is
/// in the right place: across every UOP body on a stock client the walk refused **nothing**
/// that carries art, and the one body it does refuse (286) declares a 0×0 frame, which the
/// legacy decoder treats as absent too.
///
/// ── **No <c>System.Drawing</c>, deliberately** ──
///
/// §4.4 states it: libgdiplus was archived in March 2025, and the long-term argument for
/// moving extraction off <c>System.Drawing</c> is that a Linux shard depends on an
/// unmaintained library to see a sprite. This decoder writes ARGB1555 into a
/// <c>ushort[]</c> of its own and <see cref="BridgePng"/> encodes that directly, so the
/// door stays open. (Phase 4 does not walk through it: the catalogue still refuses the
/// whole family when imaging is unavailable, because most of it genuinely needs GDI+.)
/// </summary>
public static class BridgeUop
{
/// <summary>'MYP\0' — the Mythic package magic, as <c>FileIndex</c> reads it.</summary>
private const int PackageMagic = 0x50594D;
/// <summary>'AMOU' — the animation payload's own magic, little-endian.</summary>
private const int PayloadMagic = 0x554F4D41;
/// <summary>Each frame record opens with its own palette: 0x100 ARGB1555 entries.</summary>
private const int PaletteBytes = 0x100 * 2;
/// <summary>The frame table's row width: group, frame id, two unknowns, pixel offset.</summary>
private const int FrameRowBytes = 16;
/// <summary>One block-chain record: offset, three lengths, hash, adler32, flag.</summary>
private const int BlockEntryBytes = 34;
/// <summary>The xor <c>Frame</c> applies to every run header, and so must this.</summary>
private const int DoubleXor = (0x200 << 22) | (0x200 << 12);
/// <summary>
/// A ceiling on a declared decompressed payload. One group file is a whole action for
/// one body across every direction; the largest on this machine's client is body
/// 1248's at 4.3 MB, so this is two orders of magnitude of headroom over real data and
/// still small enough that a corrupt length cannot ask for the host's memory.
/// </summary>
public const int MaxPayloadBytes = 64 * 1024 * 1024;
/// <summary>
/// A ceiling on the block chain. Five packages hold 10,724 entries between them; this
/// bounds a cyclic or corrupt chain into a refusal rather than a hang.
/// </summary>
private const int MaxEntries = 1 << 20;
/// <summary>The five packages this client ships. There is no AnimationFrame5.uop.</summary>
private static readonly int[] PackageNumbers = { 1, 2, 3, 4, 6 };
public static IEnumerable<int> Packages
{
get { return PackageNumbers; }
}
public static string PackageName(int n)
{
return "AnimationFrame" + n.ToString(CultureInfo.InvariantCulture) + ".uop";
}
/// <summary>
/// Where a UOP animation package lives.
///
/// <c>Ultima.Files.GetFilePath</c> cannot answer this: its table of known client files
/// predates UOP animations and contains no <c>AnimationFrame*.uop</c> entry, so it
/// returns null for every one of them. So the lookup is done here, against the same
/// directories ServUO itself resolved at boot — <c>Files.RootDir</c> first, then
/// <c>Core.DataDirectories</c>, which §1 is built on.
///
/// The comparison is case-insensitive **by enumeration** rather than by trying one
/// spelling. On Windows either would work; on a Linux shard host the client directory
/// is case-sensitive and the file may be shipped as `AnimationFrame1.uop`,
/// `animationframe1.uop` or anything between, which is exactly the shape of bug that
/// presents as "the gargoyles import on my machine and not on the server".
///
/// <see cref="FindClientFile"/> is the general form, and `assets.sources` uses it for
/// the same reason: a file Ultima's table predates has to be found some other way.
/// </summary>
public static string PackagePath(int n)
{
return FindClientFile(PackageName(n));
}
private static readonly object _pathSync = new object();
private static readonly Dictionary<string, string> _paths =
new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
/// <summary>
/// Finds a client file <c>Ultima.Files</c> has never heard of.
///
/// Only successful answers are cached: a file an operator copies in while the shard is
/// up should be found by the next import, and nothing here is hot enough for a
/// negative cache to be worth that.
/// </summary>
public static string FindClientFile(string name)
{
if (String.IsNullOrEmpty(name))
return null;
lock (_pathSync)
{
string cached;
if (_paths.TryGetValue(name, out cached))
return cached;
}
foreach (string dir in Directories())
{
if (String.IsNullOrEmpty(dir))
continue;
try
{
if (!Directory.Exists(dir))
continue;
string direct = Path.Combine(dir, name);
string hit = File.Exists(direct) ? direct : null;
if (hit == null)
{
foreach (string found in Directory.GetFiles(dir))
{
if (String.Equals(Path.GetFileName(found), name,
StringComparison.OrdinalIgnoreCase))
{
hit = found;
break;
}
}
}
if (hit == null)
continue;
lock (_pathSync)
_paths[name] = hit;
return hit;
}
catch (Exception e)
{
Console.WriteLine("[Bridge] uop: cannot look in {0}: {1}", dir, e.Message);
}
}
return null;
}
private static IEnumerable<string> Directories()
{
string root = null;
try
{
root = Files.RootDir;
}
catch
{
// Ultima's static initialiser reads the registry on Windows. A host where that
// throws still has Core.DataDirectories, which is the path ServUO actually booted
// from.
}
if (!String.IsNullOrEmpty(root))
yield return root;
List<string> dirs = null;
try
{
dirs = Core.DataDirectories;
}
catch
{
// Same reasoning; an empty list is a real answer and the caller reports absent.
}
if (dirs == null)
yield break;
foreach (string dir in dirs)
yield return dir;
}
/// <summary>
/// The name a body's action file is stored under, hashed the way the container indexes
/// it. <c>Ultima.FileIndex.HashFileName</c> is pure arithmetic over a string — no file
/// is touched and no decoder is entered — so this is the one place phase 4 leans on
/// the vendored code, and it leans on it precisely so that our lookup cannot disagree
/// with the container's own.
/// </summary>
public static ulong HashOf(int body, int action)
{
string name = String.Format(CultureInfo.InvariantCulture,
"build/animationlegacyframe/{0:D6}/{1:D2}.bin", body, action);
return FileIndex.HashFileName(name);
}
// ── the container ────────────────────────────────────────────────────────────────────
private struct Entry
{
public long At;
public int CompressedLength;
public int DecompressedLength;
public short Flag;
}
/// <summary>
/// One opened <c>AnimationFrame*.uop</c>: its entry table in memory, its bytes on
/// demand. Opening one is a single pass over the block chain — 10,724 entries across
/// all five on this client — and the handle is held for the life of a reply, exactly
/// like the legacy readers next to it.
/// </summary>
public sealed class Package : IDisposable
{
private readonly Dictionary<ulong, Entry> _entries;
private readonly FileStream _stream;
public readonly string Path;
private Package(string path, FileStream stream, Dictionary<ulong, Entry> entries)
{
Path = path;
_stream = stream;
_entries = entries;
}
public int Count
{
get { return _entries.Count; }
}
/// <summary>
/// Reads the block chain, refusing anything that does not fit inside the file.
/// Returns null — never throws — because a client that ships a truncated package
/// is an ordinary thing to survive, not an error to raise.
/// </summary>
public static Package Open(string path)
{
if (String.IsNullOrEmpty(path))
return null;
FileStream stream = null;
try
{
stream = new FileStream(path, FileMode.Open, FileAccess.Read,
FileShare.ReadWrite);
long length = stream.Length;
var entries = new Dictionary<ulong, Entry>();
using (var br = new BinaryReader(stream, Encoding.UTF8, true))
{
if (length < 28 || br.ReadInt32() != PackageMagic)
{
Console.WriteLine("[Bridge] uop: {0} is not a Mythic package", path);
stream.Dispose();
return null;
}
br.ReadInt32(); // version
br.ReadUInt32(); // signature
long nextBlock = br.ReadInt64();
br.ReadInt32(); // block capacity
br.ReadInt32(); // declared file count
while (nextBlock > 0)
{
// A block header is 12 bytes. Anything that does not leave room for
// one is a corrupt or cyclic chain, and this is where it stops.
if (nextBlock + 12 > length)
break;
stream.Seek(nextBlock, SeekOrigin.Begin);
int filesCount = br.ReadInt32();
long following = br.ReadInt64();
if (filesCount < 0
|| nextBlock + 12 + ((long)filesCount * BlockEntryBytes) > length)
{
break;
}
for (int i = 0; i < filesCount; i++)
{
long offset = br.ReadInt64();
int headerLength = br.ReadInt32();
int compressedLength = br.ReadInt32();
int decompressedLength = br.ReadInt32();
ulong hash = br.ReadUInt64();
br.ReadUInt32(); // adler32
short flag = br.ReadInt16();
if (offset <= 0 || headerLength < 0 || compressedLength <= 0)
continue;
if (decompressedLength <= 0 || decompressedLength > MaxPayloadBytes)
continue;
long at = offset + headerLength;
// The check FileIndex.Seek is missing, in the place it matters
// here too: that the record ENDS inside the file, not merely that
// it starts inside it (§4.5).
if (at < 0 || at + compressedLength > length)
continue;
if (entries.Count >= MaxEntries)
break;
// First writer wins. Nothing on this client produces a collision
// — measured: no hash appears in two packages, and none twice in
// one — and if a patched client ever did, taking the first is the
// answer that does not depend on chain order.
if (!entries.ContainsKey(hash))
entries[hash] = new Entry
{
At = at,
CompressedLength = compressedLength,
DecompressedLength = decompressedLength,
Flag = flag
};
}
if (following <= nextBlock)
break; // a chain that does not move forward is a loop
nextBlock = following;
}
}
return new Package(path, stream, entries);
}
catch (Exception e)
{
Console.WriteLine("[Bridge] uop: cannot open {0}: {1}: {2}",
path, e.GetType().Name, e.Message);
if (stream != null)
{
try
{
stream.Dispose();
}
catch
{
// Closing a read-only handle.
}
}
return null;
}
}
public bool Has(ulong hash)
{
return _entries.ContainsKey(hash);
}
/// <summary>
/// The bytes behind one entry, decompressed. False with a reason is the ordinary
/// answer for "this package does not hold it".
/// </summary>
public bool TryRead(ulong hash, out byte[] payload, out string reason)
{
payload = null;
reason = null;
Entry entry;
if (!_entries.TryGetValue(hash, out entry))
{
reason = "not in " + System.IO.Path.GetFileName(Path);
return false;
}
byte[] raw;
try
{
_stream.Seek(entry.At, SeekOrigin.Begin);
raw = new byte[entry.CompressedLength];
if (!Fill(_stream, raw, raw.Length))
{
// The §4.5 failure, in our own code this time: a short read that nobody
// checked is how the library ends up decoding the previous asset.
reason = "record is shorter than the index claims";
return false;
}
}
catch (Exception e)
{
reason = "read failed: " + e.GetType().Name;
return false;
}
if (entry.Flag != 1)
{
if (raw.Length != entry.DecompressedLength)
{
reason = "stored record is " + raw.Length + " bytes, not the declared "
+ entry.DecompressedLength;
return false;
}
payload = raw;
return true;
}
return TryInflate(raw, entry.DecompressedLength, out payload, out reason);
}
public void Dispose()
{
try
{
_stream.Dispose();
}
catch
{
// Closing a read-only handle. Nothing useful is left to do.
}
}
}
private static bool Fill(Stream stream, byte[] into, int count)
{
int read = 0;
while (read < count)
{
int n = stream.Read(into, read, count - read);
if (n <= 0)
return false;
read += n;
}
return true;
}
/// <summary>
/// zlib, which .NET Framework 4.8 does not expose — only raw deflate. The two-byte
/// zlib header is checked and skipped rather than assumed, because handing a
/// <c>DeflateStream</c> a stream that is not deflate produces garbage as readily as an
/// exception, and the trailing adler32 is left to the length check below: a stream
/// that inflates to exactly the declared number of bytes did not silently truncate.
/// </summary>
private static bool TryInflate(byte[] raw, int declared, out byte[] payload, out string reason)
{
payload = null;
reason = null;
if (raw.Length < 3)
{
reason = "compressed record is too short to be zlib";
return false;
}
int cmf = raw[0];
int flg = raw[1];
if ((cmf & 0x0F) != 8 || (((cmf << 8) + flg) % 31) != 0 || (flg & 0x20) != 0)
{
reason = "compressed record is not a zlib stream";
return false;
}
try
{
var output = new byte[declared];
using (var source = new MemoryStream(raw, 2, raw.Length - 2, false))
using (var inflate = new DeflateStream(source, CompressionMode.Decompress))
{
int read = 0;
while (read < declared)
{
int n = inflate.Read(output, read, declared - read);
if (n <= 0)
break;
read += n;
}
if (read != declared)
{
reason = "inflated " + read + " bytes, not the declared " + declared;
return false;
}
// One more byte would mean the record is longer than its own header says,
// which is a different file from the one we were promised.
if (inflate.ReadByte() != -1)
{
reason = "inflated past the declared " + declared + " bytes";
return false;
}
}
payload = output;
return true;
}
catch (Exception e)
{
reason = "inflate failed: " + e.GetType().Name;
return false;
}
}
// ── the payload ──────────────────────────────────────────────────────────────────────
/// <summary>One decoded frame: ARGB1555 in our own array, no <c>Bitmap</c> anywhere.</summary>
public sealed class Pixels
{
public int Width;
public int Height;
public int CenterX;
public int CenterY;
public ushort[] Argb1555;
}
/// <summary>
/// One action of one body — every direction of it, concatenated.
///
/// The legacy files address a frame as <c>index + action * 5 + direction</c>; a UOP
/// group file holds the whole action in one record and the directions are equal-length
/// runs inside its frame table. So <see cref="DirectionAt"/> is where "direction 1" is
/// turned into a frame number, and it is integer division exactly as the reference
/// implementations do it — see the note there for the nine bodies where that matters.
/// </summary>
public sealed class Group
{
private readonly byte[] _buf;
private readonly int _dataStart;
public readonly int FrameCount;
public readonly int Body;
private Group(byte[] buf, int body, int frameCount, int dataStart)
{
_buf = buf;
Body = body;
FrameCount = frameCount;
_dataStart = dataStart;
}
public static bool TryOpen(byte[] buf, int expectedBody, out Group group, out string reason)
{
group = null;
reason = null;
if (buf == null || buf.Length < 40)
{
reason = "payload is too short to carry a header";
return false;
}
if (BitConverter.ToInt32(buf, 0) != PayloadMagic)
{
reason = "payload is not an AMOU animation record";
return false;
}
int body = BitConverter.ToInt32(buf, 12);
// The container said which body this is, by the name it was stored under; the
// payload says it again. They agree on every record of this client, and the day
// they do not is the day something is being read that was not asked for.
if (body != expectedBody)
{
reason = "payload declares body " + body + ", not " + expectedBody;
return false;
}
int frameCount = BitConverter.ToInt32(buf, 32);
int dataStart = BitConverter.ToInt32(buf, 36);
if (frameCount <= 0 || frameCount > BridgeAssetValidator.MaxAnimFrames)
{
reason = "payload declares " + frameCount + " frames";
return false;
}
if (dataStart < 40 || dataStart > buf.Length)
{
reason = "frame table starts at " + dataStart + " of " + buf.Length;
return false;
}
if ((long)dataStart + ((long)frameCount * FrameRowBytes) > buf.Length)
{
reason = "frame table of " + frameCount + " rows runs past the record";
return false;
}
group = new Group(buf, body, frameCount, dataStart);
return true;
}
/// <summary>
/// Which frame of this action faces a given direction.
///
/// Five directions share the action's frames equally, so direction *d* starts at
/// <c>d * (FrameCount / 5)</c>. On nine of this client's 244 UOP bodies the frame
/// count is **not** a multiple of five (41, 42, 46…), and integer division then
/// lands a direction or so early in the run. That is what ClassicUO does, it is
/// the right trade, and the reason is §4.8's: the failure being guarded against is
/// a picture of the **wrong creature**, and this cannot produce one — the worst
/// case is the right creature at a slightly different angle, on nine bodies, where
/// refusing them instead would lose nine creatures outright.
/// </summary>
public int DirectionAt(int direction)
{
int perDirection = FrameCount / 5;
if (perDirection <= 0)
return direction == 0 ? 0 : -1;
if (direction < 0 || direction > 4)
return -1;
int at = direction * perDirection;
return at < FrameCount ? at : -1;
}
/// <summary>
/// Decodes one frame, bounding every read against the record and every write
/// against the bitmap.
///
/// The run loop is <c>Ultima.Frame</c>'s, with the two bounds it does not have.
/// <c>Frame</c> writes through a <c>LockBits</c> pointer whose origin comes from
/// two signed shorts in the file and never checks where a run lands; here a run
/// that would leave the bitmap, or read past the record, refuses the frame. Across
/// every UOP body on a stock client that refuses nothing that carries art.
///
/// A 0×0 frame returns false with <paramref name="empty"/> set: the legacy decoder
/// treats that as no art rather than as damage, and so must this, or body 286
/// would be logged as a defect on every scan.
/// </summary>
public bool TryDecode(int index, out Pixels pixels, out bool empty, out string reason)
{
pixels = null;
empty = false;
reason = null;
if (index < 0 || index >= FrameCount)
{
reason = "frame " + index + " of " + FrameCount;
return false;
}
int row = _dataStart + (index * FrameRowBytes);
long at = (long)row + (uint)BitConverter.ToInt32(_buf, row + 12);
if (at < 0 || at + PaletteBytes + 8 > _buf.Length)
{
reason = "frame " + index + " points outside the record";
return false;
}
int pixelAt = (int)at;
int centerX = BitConverter.ToInt16(_buf, pixelAt + PaletteBytes);
int centerY = BitConverter.ToInt16(_buf, pixelAt + PaletteBytes + 2);
int width = BitConverter.ToUInt16(_buf, pixelAt + PaletteBytes + 4);
int height = BitConverter.ToUInt16(_buf, pixelAt + PaletteBytes + 6);
if (width <= 0 || height <= 0)
{
empty = true;
reason = "frame " + index + " is " + width + "x" + height;
return false;
}
if (width > BridgeAssetValidator.MaxArtDimension
|| height > BridgeAssetValidator.MaxArtDimension)
{
reason = "frame " + index + " declares " + width + "x" + height;
return false;
}
var palette = new ushort[0x100];
for (int i = 0; i < palette.Length; i++)
{
// The library's own xor: the stored entry has its alpha bit clear and every
// palette colour is opaque. A pixel no run covers stays zero, which is how a
// sprite keeps its transparent background.
palette[i] = (ushort)(BitConverter.ToUInt16(_buf, pixelAt + (i * 2)) ^ 0x8000);
}
var canvas = new ushort[width * height];
int p = pixelAt + PaletteBytes + 8;
int xBase = centerX - 0x200;
int yBase = (centerY + height) - 0x200;
while (true)
{
if (p + 4 > _buf.Length)
{
reason = "frame " + index + " has no terminator inside the record";
return false;
}
int header = BitConverter.ToInt32(_buf, p);
p += 4;
if (header == 0x7FFF7FFF)
break;
header ^= DoubleXor;
int x = ((header >> 22) & 0x3FF) + xBase;
int y = ((header >> 12) & 0x3FF) + yBase;
int run = header & 0xFFF;
if (run == 0)
continue;
if (p + run > _buf.Length)
{
reason = "frame " + index + " has a run past the end of the record";
return false;
}
if (y < 0 || y >= height || x < 0 || x + run > width)
{
reason = "frame " + index + " has a run at " + x + "," + y + " of "
+ run + " outside " + width + "x" + height;
return false;
}
int cursor = (y * width) + x;
for (int i = 0; i < run; i++)
canvas[cursor + i] = palette[_buf[p + i]];
p += run;
}
pixels = new Pixels
{
Width = width,
Height = height,
CenterX = centerX,
CenterY = centerY,
Argb1555 = canvas
};
return true;
}
}
}
}

580
tools/patch_client.ps1 Normal file
View File

@@ -0,0 +1,580 @@
<#
.SYNOPSIS
Builds a deliberately patched UO client for the Asset Bridge phase 0 spike.
.DESCRIPTION
docs/link/v8.md section 16 phase 0 drives ServUO's vendored `Ultima` decoders "over a deliberately
patched client". Stock clients are not the interesting case: they are the case the library was
written against, and the whole reason phase 0 exists is that section 4 chose to call code that can
take the shard down if it is wrong. A shard operator's client is patched -- custom art, a
verdata.mul, a hand-edited Bodyconv.def -- and that is what has to be survived.
This copies a client and then breaks the copy in four deliberate, catalogued ways. It NEVER
writes to the source: every file it patches is hashed before and after, and a changed source
hash aborts the run.
Each defect is recorded in `patched-client.manifest.json` next to the copy, so the probe's
report can be read against what was actually done rather than against a memory of it. The
manifest is the answer to "is a nonzero REFUSED-BUT-DECODED count a bug or the point?".
.PARAMETER Source
The client to copy. Defaults to this machine's.
.PARAMETER Dest
Where to build the patched copy. Needs ~3.5 GB.
.PARAMETER Tiers
Which defects to apply. Default: all four.
verdata Author a verdata.mul, which this client does not have. Ultima consults Verdata on
EVERY art and anim lookup (Art's FileIndex is built with verdata file id 4,
Animations' with 6), so on a client with no verdata.mul that entire branch is
dead code that has never been exercised -- the largest untested surface in the
library we are about to depend on. Includes one legitimate patch and one whose
lookup points past verdata.mul's own end, because `FileIndex.Seek` bounds-checks
the mul and does not bounds-check verdata.
customart Fill unused artidx.mul slots with real records appended to art.mul, the way a
custom-art shard does. Tests that our out-of-range accounting comes from the
file rather than from a constant someone wrote down.
corrupt Rewrite index entries and record headers into the shapes that reading Art.cs
says are reachable: a lookup past EOF, a record that starts inside the file and
ends outside it, a length too small for a header, absurd dimensions, a row table
pointing outside its own record, and a land tile shorter than the fixed 2,024
bytes LoadLand always reads.
bodyconv Add Bodyconv.def lines pointing bodies at an anim file that holds nothing, and at
an index in another file that holds something unrelated -- the gargoyle-666 spider
case, reproduced on purpose. Proves the extractor takes BodyConverter.Convert's
answer and stops (v8.md section 4.3).
nouop Move artLegacyMUL.uop aside, so art is read from art.mul/artidx.mul.
This is not cosmetic and it is not optional if you want the customart or corrupt
tiers to mean anything. FileIndex's UOP constructor ends with a bare
`MulPath = uopPath`: when artLegacyMUL.uop is present it wins OUTRIGHT and
art.mul / artidx.mul are never opened. Every index-level defect below writes to
files the library does not read on a modern client, so without this tier those
two tiers are inert while still reporting that they applied.
It is also a real configuration in its own right: plenty of shards run mul-only
clients, and a custom-art shard that adds graphics to art.mul while the UOP is
still there gets nothing at all -- an operator trap worth knowing about.
.PARAMETER SkipCopy
Re-patch an existing copy without re-copying 3.5 GB. Only safe on a copy this script made and
has not patched yet -- patching twice compounds the defects and invalidates the manifest.
.EXAMPLE
.\tools\patch_client.ps1 -Dest D:\uo-patched-client
.NOTES
Test scaffolding. Never deployed. The copy contains EA's client art -- like every other
extraction in this project it stays on the machine that made it and is never committed.
#>
[CmdletBinding()]
param(
[string] $Source = 'D:\Games\Electronic Arts\Ultima Online Classic',
[Parameter(Mandatory = $true)]
[string] $Dest,
[ValidateSet('verdata', 'customart', 'corrupt', 'bodyconv', 'nouop')]
[string[]] $Tiers = @('nouop', 'verdata', 'customart', 'corrupt', 'bodyconv'),
[switch] $SkipCopy,
[switch] $Force
)
$ErrorActionPreference = 'Stop'
# Files this script may write to in the copy. Anything not on this list is a bug in the script,
# and the source-hash check at the end is what proves it.
$PatchTargets = @('artidx.mul', 'art.mul', 'verdata.mul', 'Bodyconv.def', 'artLegacyMUL.uop')
# -- Little-endian helpers (BitConverter is fine, but the intent reads better named) ----------
function Read-Int32LE([byte[]] $Bytes, [int] $Offset) {
return [BitConverter]::ToInt32($Bytes, $Offset)
}
function Write-Int32LE([byte[]] $Bytes, [int] $Offset, [int] $Value) {
[Array]::Copy([BitConverter]::GetBytes([int] $Value), 0, $Bytes, $Offset, 4)
}
function Get-ArtEntry([byte[]] $Idx, [int] $Index) {
$at = $Index * 12
return [pscustomobject]@{
Index = $Index
Lookup = Read-Int32LE $Idx $at
Length = Read-Int32LE $Idx ($at + 4)
Extra = Read-Int32LE $Idx ($at + 8)
}
}
function Set-ArtEntry([byte[]] $Idx, [int] $Index, [int] $Lookup, [int] $Length, [int] $Extra) {
$at = $Index * 12
Write-Int32LE $Idx $at $Lookup
Write-Int32LE $Idx ($at + 4) $Length
Write-Int32LE $Idx ($at + 8) $Extra
}
# The defect catalogue. Every mutation appends to this, and it is written out as the manifest.
$script:Defects = New-Object System.Collections.ArrayList
function Add-Defect([string] $Tier, [string] $Key, [string] $What, [string] $Expect) {
[void] $script:Defects.Add([pscustomobject]@{
tier = $Tier
key = $Key
what = $What
expect = $Expect
})
Write-Host (" {0,-22} {1}" -f $Key, $What)
}
# -- Preflight --------------------------------------------------------------------------------
if (-not (Test-Path -LiteralPath $Source)) {
throw "source client not found: $Source"
}
$sourceFull = (Resolve-Path -LiteralPath $Source).Path
if (Test-Path -LiteralPath $Dest) {
$destFull = (Resolve-Path -LiteralPath $Dest).Path
if ($destFull -eq $sourceFull) {
throw "Dest is the source client. Refusing -- this script destroys what it points at."
}
if (-not $SkipCopy -and -not $Force) {
throw "$Dest already exists. Pass -Force to overwrite it, or -SkipCopy to patch it in place."
}
}
Write-Host "source: $sourceFull"
Write-Host "dest: $Dest"
Write-Host "tiers: $($Tiers -join ', ')"
Write-Host ""
# Hash the source files we are about to touch, so "it never writes to the source" is checked and
# not merely asserted.
$before = @{}
foreach ($name in $PatchTargets) {
$path = Join-Path $sourceFull $name
if (Test-Path -LiteralPath $path) {
$before[$name] = (Get-FileHash -LiteralPath $path -Algorithm SHA256).Hash
}
}
# -- Copy -------------------------------------------------------------------------------------
if ($SkipCopy) {
Write-Host "skipping copy (-SkipCopy)"
if (-not (Test-Path -LiteralPath $Dest)) { throw "-SkipCopy but $Dest does not exist" }
} else {
Write-Host "copying (this is ~3.5 GB; a few minutes)..."
# /MIR so a -Force re-run starts clean rather than merging into an already-patched tree.
# /NJH /NJS /NDL /NFL keep robocopy's output to the errors.
$null = robocopy $sourceFull $Dest /MIR /R:1 /W:1 /NJH /NJS /NDL /NFL /NP
# Robocopy exit codes below 8 are success; 8 and above are real failures.
if ($LASTEXITCODE -ge 8) { throw "robocopy failed with exit code $LASTEXITCODE" }
# Robocopy's "1 = files were copied" would otherwise become this script's exit code and read
# as a failure to anything checking it.
$global:LASTEXITCODE = 0
Write-Host "copied."
}
Write-Host ""
$destFull = (Resolve-Path -LiteralPath $Dest).Path
$artIdxPath = Join-Path $destFull 'artidx.mul'
$artMulPath = Join-Path $destFull 'art.mul'
if (-not (Test-Path -LiteralPath $artIdxPath)) { throw "no artidx.mul in the copy" }
$idx = [System.IO.File]::ReadAllBytes($artIdxPath)
$entryCount = [int] ($idx.Length / 12)
$artMulLength = (Get-Item -LiteralPath $artMulPath).Length
Write-Host ("artidx.mul holds {0:N0} entries; art.mul is {1:N0} bytes" -f $entryCount, $artMulLength)
Write-Host ""
# -- Tier: nouop ------------------------------------------------------------------------------
$uopPath = Join-Path $destFull 'artLegacyMUL.uop'
$uopPresent = Test-Path -LiteralPath $uopPath
if ($Tiers -contains 'nouop') {
Write-Host "tier nouop"
if (-not $uopPresent) {
Write-Host " no artLegacyMUL.uop in the copy -- already a mul-only client"
} else {
Move-Item -LiteralPath $uopPath -Destination "$uopPath.disabled" -Force
$uopPresent = $false
Add-Defect 'nouop' 'artLegacyMUL.uop' 'moved aside so art is read from art.mul/artidx.mul' `
'every index-level defect below becomes reachable; without this they are inert'
}
Write-Host ""
} elseif ($uopPresent -and (($Tiers -contains 'corrupt') -or ($Tiers -contains 'customart'))) {
Write-Host " WARNING: artLegacyMUL.uop is present and the nouop tier was not selected."
Write-Host " FileIndex prefers the UOP outright, so the corrupt and customart tiers"
Write-Host " will write to files the library never opens. Add -Tiers nouop."
Write-Host ""
}
# Static ids are offset by 0x4000 in the index; land tiles occupy 0..0x3FFF.
$StaticBase = 0x4000
# Find donor records to copy and victims to corrupt: real, modestly sized statics, so the defects
# are applied to entries that genuinely work today. Picking arbitrary ids risks landing on slots
# that are already empty, where a "defect" would prove nothing.
$donors = New-Object System.Collections.ArrayList
for ($id = 0x1000; $id -lt 0x3000 -and $donors.Count -lt 24; $id++) {
$e = Get-ArtEntry $idx ($id + $StaticBase)
if ($e.Lookup -ge 0 -and $e.Length -gt 200 -and $e.Length -lt 4000 -and ($e.Lookup + $e.Length) -le $artMulLength) {
[void] $donors.Add([pscustomobject]@{ Id = $id; Entry = $e })
}
}
if ($donors.Count -lt 12) { throw "found only $($donors.Count) usable donor statics -- the copy looks wrong" }
Write-Host "using donor statics: $(($donors | Select-Object -First 12 | ForEach-Object { $_.Id }) -join ', ')"
Write-Host ""
$idxDirty = $false
# -- Tier: customart --------------------------------------------------------------------------
if ($Tiers -contains 'customart') {
Write-Host "tier customart"
# A custom-art client does not fill spare slots -- artidx.mul is exactly sized (62,692
# entries here, not one to spare), so adding art means GROWING the index. `Art` builds its
# FileIndex with length 0x10000, so there is room for 2,844 more ids before the library stops
# looking, and the stock ceiling turns out to be nothing more than the size of a file.
$idxCeiling = 0x10000
if ($entryCount -ge $idxCeiling) {
Write-Host " artidx.mul is already at the 0x10000 ceiling -- skipping tier"
} else {
$addCount = 8
$grown = New-Object byte[] (($entryCount + $addCount) * 12)
[Array]::Copy($idx, 0, $grown, 0, $idx.Length)
$idx = $grown
# Read every donor record BEFORE opening the append handle. Append mode takes an
# exclusive lock, so reading the same file while appending to it fails outright.
$buffers = @()
$reader = [System.IO.File]::Open($artMulPath, 'Open', 'Read', 'ReadWrite')
try {
for ($n = 0; $n -lt $addCount; $n++) {
$donor = $donors[$n]
$buffer = New-Object byte[] $donor.Entry.Length
[void] $reader.Seek($donor.Entry.Lookup, 'Begin')
[void] $reader.Read($buffer, 0, $buffer.Length)
$buffers += , $buffer
}
} finally { $reader.Dispose() }
$appendAt = $artMulLength
$stream = [System.IO.File]::Open($artMulPath, 'Append', 'Write')
try {
for ($n = 0; $n -lt $addCount; $n++) {
$buffer = $buffers[$n]
$stream.Write($buffer, 0, $buffer.Length)
$slot = $entryCount + $n
$newId = $slot - $StaticBase
Set-ArtEntry $idx $slot $appendAt $buffer.Length $donors[$n].Entry.Extra
$appendAt += $buffer.Length
Add-Defect 'customart' "static/$newId" `
"custom art appended past the stock ceiling (a copy of static/$($donors[$n].Id))" `
'decodes cleanly; proves the ceiling is read from the file, not from a constant'
}
} finally { $stream.Dispose() }
$entryCount += $addCount
$idxDirty = $true
}
Write-Host ""
}
# -- Tier: corrupt ----------------------------------------------------------------------------
if ($Tiers -contains 'corrupt') {
Write-Host "tier corrupt"
$artMulLength = (Get-Item -LiteralPath $artMulPath).Length
$v = 8 # donors 0..7 may have been consumed by customart as sources; they are unmodified
# 1. A lookup past the end of art.mul. FileIndex.Seek DOES check this one
# (`Stream.Length < e.lookup`), so the library and the validator should agree.
$victim = $donors[$v++].Id
Set-ArtEntry $idx ($victim + $StaticBase) ([int]($artMulLength + 4096)) 512 0
Add-Defect 'corrupt' "static/$victim" 'lookup 4 KB past the end of art.mul' `
'refused by the validator; Seek also catches this one, so no picture'
# 2. A record that STARTS inside the file and ENDS outside it. This is the gap: Seek checks
# the start and never the end, stream.Read returns short, the decoders ignore the count,
# and m_StreamBuffer still holds the PREVIOUS asset. The expected outcome is a picture of
# something else entirely, reported as a success by every count in the library.
$victim = $donors[$v++].Id
Set-ArtEntry $idx ($victim + $StaticBase) ([int]($artMulLength - 64)) 8192 0
Add-Defect 'corrupt' "static/$victim" 'record starts 64 bytes before EOF and declares 8,192' `
'REFUSED BUT DECODED -- the stale-buffer wrong picture'
# 3. A length too small to hold even the 8-byte header.
$victim = $donors[$v++].Id
$donorEntry = $donors[$v - 1].Entry
Set-ArtEntry $idx ($victim + $StaticBase) $donorEntry.Lookup 4 0
Add-Defect 'corrupt' "static/$victim" 'declared length 4 -- smaller than the static header' `
'refused by the validator'
# 4/5/6 rewrite the record BODY, so they need their own bytes rather than an index edit.
# Appended to art.mul and pointed at, which leaves the donor's real record intact.
$stream = [System.IO.File]::Open($artMulPath, 'Append', 'Write')
try {
$appendAt = (Get-Item -LiteralPath $artMulPath).Length
# 4. Absurd dimensions. LoadStatic allocates new Bitmap(width, height) straight from two
# ushorts in the file. 8000x8000 is ~128 MB -- survivable, and the point is made; the
# same field can ask for 65535x65535, which is 8 GB from a two-byte edit.
$victim = $donors[$v++].Id
$rec = New-Object byte[] 2048
[Array]::Copy([BitConverter]::GetBytes([uint16] 8000), 0, $rec, 4, 2)
[Array]::Copy([BitConverter]::GetBytes([uint16] 8000), 0, $rec, 6, 2)
$stream.Write($rec, 0, $rec.Length)
Set-ArtEntry $idx ($victim + $StaticBase) $appendAt $rec.Length 0
$appendAt += $rec.Length
Add-Defect 'corrupt' "static/$victim" 'header declares 8000x8000 (a ~128 MB allocation from two bytes)' `
'refused by the validator; the library would allocate it'
# 5. A row-lookup table pointing outside the record. This is what LoadStatic's unbounded
# read cursor was written to walk off the end of.
$victim = $donors[$v++].Id
$rec = New-Object byte[] 512
[Array]::Copy([BitConverter]::GetBytes([uint16] 32), 0, $rec, 4, 2) # width
[Array]::Copy([BitConverter]::GetBytes([uint16] 32), 0, $rec, 6, 2) # height
for ($row = 0; $row -lt 32; $row++) {
# Each row's offset is added to (height + 4); 60000 puts every row far outside.
[Array]::Copy([BitConverter]::GetBytes([uint16] 60000), 0, $rec, (8 + $row * 2), 2)
}
$stream.Write($rec, 0, $rec.Length)
Set-ArtEntry $idx ($victim + $StaticBase) $appendAt $rec.Length 0
$appendAt += $rec.Length
Add-Defect 'corrupt' "static/$victim" 'row table points 60,000 words outside a 512-byte record' `
'refused by the validator; the library reads adjacent heap'
# 6. A well-formed row table whose run length overruns the record.
$victim = $donors[$v++].Id
$rec = New-Object byte[] 256
[Array]::Copy([BitConverter]::GetBytes([uint16] 16), 0, $rec, 4, 2)
[Array]::Copy([BitConverter]::GetBytes([uint16] 2), 0, $rec, 6, 2)
[Array]::Copy([BitConverter]::GetBytes([uint16] 0), 0, $rec, 8, 2) # row 0 offset
[Array]::Copy([BitConverter]::GetBytes([uint16] 0), 0, $rec, 10, 2) # row 1 offset
$runAt = (2 + 4) * 2 # (height + 4) words
[Array]::Copy([BitConverter]::GetBytes([uint16] 0), 0, $rec, $runAt, 2) # xOffset
[Array]::Copy([BitConverter]::GetBytes([uint16] 16), 0, $rec, ($runAt + 2), 2) # xRun, but
# the record has nowhere near 16 pixels left after this point.
$stream.Write($rec, 0, $rec.Length)
Set-ArtEntry $idx ($victim + $StaticBase) $appendAt 20 0
$appendAt += $rec.Length
Add-Defect 'corrupt' "static/$victim" 'a 16-pixel run declared in a 20-byte record' `
'refused by the validator'
} finally { $stream.Dispose() }
# 7. A land tile shorter than the 2,024 bytes LoadLand reads unconditionally.
$landVictim = 0x0100
$landEntry = Get-ArtEntry $idx $landVictim
if ($landEntry.Lookup -ge 0 -and $landEntry.Length -gt 0) {
Set-ArtEntry $idx $landVictim $landEntry.Lookup 512 0
Add-Defect 'corrupt' "land/$landVictim" 'land record declared 512 bytes; LoadLand always reads 2,024' `
'refused by the validator; the library reads past the buffer'
}
$idxDirty = $true
Write-Host ""
}
if ($idxDirty) {
[System.IO.File]::WriteAllBytes($artIdxPath, $idx)
Write-Host "wrote artidx.mul"
Write-Host ""
}
# -- Tier: verdata ----------------------------------------------------------------------------
if ($Tiers -contains 'verdata') {
Write-Host "tier verdata"
# Layout: int32 count, then count * 5 int32 (file, index, lookup, length, extra), then the
# payloads. `lookup` is an absolute offset into this file.
$entries = New-Object System.Collections.ArrayList
$payloads = New-Object System.Collections.ArrayList
$donorA = $donors[$donors.Count - 1]
$donorB = $donors[$donors.Count - 2]
$artSource = [System.IO.File]::Open($artMulPath, 'Open', 'Read', 'ReadWrite')
try {
$bufferA = New-Object byte[] $donorA.Entry.Length
[void] $artSource.Seek($donorA.Entry.Lookup, 'Begin')
[void] $artSource.Read($bufferA, 0, $bufferA.Length)
} finally { $artSource.Dispose() }
# The victims: ids whose art will now come from verdata.mul rather than art.mul.
$legitVictim = $donors[$donors.Count - 3].Id
$pastEofVictim = $donors[$donors.Count - 4].Id
# A legitimate patch -- the branch working as designed. Without this the tier only proves the
# failure case, and "verdata is broken" and "verdata is never reached" look identical.
[void] $payloads.Add($bufferA)
[void] $entries.Add([pscustomobject]@{
File = 4; Index = ($legitVictim + $StaticBase); Length = $bufferA.Length; Extra = $donorA.Entry.Extra
PayloadIndex = 0; PastEof = $false
})
# The failure case. FileIndex.Seek bounds-checks the mul stream and calls Verdata.Seek with no
# check at all; seeking a FileStream past EOF is legal, the read returns nothing, and the
# shared decode buffer still holds the previous asset.
[void] $entries.Add([pscustomobject]@{
File = 4; Index = ($pastEofVictim + $StaticBase); Length = 900; Extra = 0
PayloadIndex = -1; PastEof = $true
})
# An anim patch, so the tier covers the other file the verdata branch serves. anim.mul is
# verdata file 6; for body < 200 the record index is body*110 + action*5 + direction.
$animBody = 34 # wolf -- decodes on this client, so a patch to it is observable
$animIndex = ($animBody * 110) + (0 * 5) + 1
[void] $entries.Add([pscustomobject]@{
File = 6; Index = $animIndex; Length = 700; Extra = 0
PayloadIndex = -1; PastEof = $true
})
$headerSize = 4 + ($entries.Count * 20)
$offset = $headerSize
foreach ($entry in $entries) {
if ($entry.PayloadIndex -ge 0) {
$entry | Add-Member -NotePropertyName Lookup -NotePropertyValue $offset -Force
$offset += $payloads[$entry.PayloadIndex].Length
}
}
$totalSize = $offset
# Past-EOF lookups are resolved last, because "past the end" is only meaningful once the end
# is known.
foreach ($entry in $entries) {
if ($entry.PastEof) {
$entry | Add-Member -NotePropertyName Lookup -NotePropertyValue ($totalSize + 8192) -Force
}
}
$verdata = New-Object byte[] $totalSize
Write-Int32LE $verdata 0 $entries.Count
$at = 4
foreach ($entry in $entries) {
Write-Int32LE $verdata $at $entry.File
Write-Int32LE $verdata ($at + 4) $entry.Index
Write-Int32LE $verdata ($at + 8) $entry.Lookup
Write-Int32LE $verdata ($at + 12) $entry.Length
Write-Int32LE $verdata ($at + 16) $entry.Extra
$at += 20
}
foreach ($entry in $entries) {
if ($entry.PayloadIndex -ge 0) {
$payload = $payloads[$entry.PayloadIndex]
[Array]::Copy($payload, 0, $verdata, $entry.Lookup, $payload.Length)
}
}
[System.IO.File]::WriteAllBytes((Join-Path $destFull 'verdata.mul'), $verdata)
Add-Defect 'verdata' "static/$legitVictim" `
"legitimately patched to static/$($donorA.Id)'s art via verdata.mul" `
'decodes; the picture must CHANGE, which is how we know the branch ran'
Add-Defect 'verdata' "static/$pastEofVictim" `
'verdata entry whose lookup is 8 KB past the end of verdata.mul' `
'REFUSED BUT DECODED -- Verdata.Seek is not bounds-checked'
Add-Defect 'verdata' "body/$animBody" `
"anim.mul record $animIndex patched to a verdata offset past EOF" `
'the wolf must not silently become another creature'
Write-Host (" wrote verdata.mul: {0} entries, {1:N0} bytes" -f $entries.Count, $totalSize)
Write-Host ""
}
# -- Tier: bodyconv ---------------------------------------------------------------------------
if ($Tiers -contains 'bodyconv') {
Write-Host "tier bodyconv"
$bodyconvPath = Join-Path $destFull 'Bodyconv.def'
if (-not (Test-Path -LiteralPath $bodyconvPath)) {
Write-Host " no Bodyconv.def in the copy -- skipping tier"
} else {
# Columns are tab-separated: original, anim2, anim3, anim4, anim5. -1 means "not in that
# file". BodyConverter.Convert returns the file type of the FIRST column that is not -1,
# and the extractor must take that answer and stop.
$lines = @(
"",
"# Asset Bridge phase 0 -- deliberate defects (tools/patch_client.ps1)",
"1900`t-1`t-1`t-1`t60000",
"1901`t666`t-1`t-1`t-1"
)
Add-Content -LiteralPath $bodyconvPath -Value ($lines -join "`r`n") -Encoding ASCII
Add-Defect 'bodyconv' 'body/1900' 'mapped to anim5 index 60,000, which does not exist' `
'reports nothing -- and must NOT fall back to another anim file'
Add-Defect 'bodyconv' 'body/1901' 'mapped to anim2 index 666, where something unrelated lives' `
'decodes a picture of the WRONG creature -- the spider case, on purpose'
}
Write-Host ""
}
# -- The source must be untouched -------------------------------------------------------------
$tampered = @()
foreach ($name in $before.Keys) {
$path = Join-Path $sourceFull $name
$now = (Get-FileHash -LiteralPath $path -Algorithm SHA256).Hash
if ($now -ne $before[$name]) { $tampered += $name }
}
if ($tampered.Count -gt 0) {
throw "THE SOURCE CLIENT WAS MODIFIED: $($tampered -join ', '). Restore it from the installer before doing anything else."
}
Write-Host "source client verified unchanged ($($before.Count) files hashed before and after)"
# -- Manifest ---------------------------------------------------------------------------------
$manifest = [pscustomobject]@{
built = (Get-Date).ToUniversalTime().ToString('u')
source = $sourceFull
dest = $destFull
tiers = $Tiers
defects = @($script:Defects)
}
$manifestPath = Join-Path $destFull 'patched-client.manifest.json'
$manifest | ConvertTo-Json -Depth 5 | Set-Content -LiteralPath $manifestPath -Encoding utf8
Write-Host ""
Write-Host ("{0} deliberate defects; manifest at {1}" -f $script:Defects.Count, $manifestPath)
Write-Host ""
Write-Host "Point the probe at it by adding to the shard's Config/Bridge.cfg:"
Write-Host ""
Write-Host " AssetProbeClient=$destFull"
Write-Host ""
Write-Host "then, in game or from the rig driver: [assetprobe all patched"

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,532 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Text;
namespace Server.Custom
{
/// <summary>
/// A reader for the **Mythic compressed** cliloc container, in plain .NET Framework 4.8 C#.
///
/// This is the Asset Bridge's §9 decoder — the ONE decoder Protocol 8 writes rather than
/// calls (docs/link/v8.md §4, §9). ServUO's bundled <c>Ultima.StringList</c> implements only
/// the plain layout and throws <c>Non-negative number required</c> on every modern client's
/// file, which is also why the shard's own <c>VendorSearch.GetItemName</c> is already inert.
///
/// **Provenance.** Ported from UOFiddler's <c>Ultima/Helpers/MythicDecompress.cs</c>,
/// <c>MoveToFront.cs</c> and <c>StringList.TryParse</c> (polserver/UOFiddler). UOFiddler is
/// released under the **Beerware** licence, so carrying its algorithm into this
/// GPL-3.0-or-later tree is clean — see v8.md §9.
///
/// **What the port had to change**, and why the differences are not cosmetic:
///
/// * UOFiddler targets net10.0 and its implementation is written in <c>Span&lt;T&gt;</c>,
/// <c>stackalloc</c>, <c>ArrayPool</c> and <c>BinaryPrimitives</c>. ServUO compiles the
/// overlay against net48 with no package feed, so all of that becomes plain arrays.
/// * Every read of the compressed payload is **bounds-checked here and is not there**.
/// Upstream indexes <c>input[m + 1024]</c> and <c>input[firstVal + 1024]</c> with
/// offsets derived from the file's own frequency header, inside a
/// <c>try { } catch (Exception) { return false; }</c>. That is adequate for a desktop
/// tool and is not adequate for us: this runs inside a live shard, and a corrupt or
/// hostile Cliloc.enu must produce a refusal, not an exception unwinding through the
/// bridge. Every such index is tested before use and returns <c>false</c> instead.
///
/// Phase 0 uses this from <see cref="BridgeAssetProbe"/> to prove the port reproduces
/// UOFiddler's own output exactly. **Phase 2 promotes this file into
/// <c>overlay/Scripts/Custom/Bridge/</c>** — it lives in scaffolding only for as long as it
/// is a spike.
/// </summary>
public static class BridgeMythicCliloc
{
/// <summary>The first DWORD of a compressed file is the decompressed length, XORed with this.</summary>
private const uint HeaderXorKey = 0x8E2C9A3D;
/// <summary>256 little-endian int32 symbol frequencies precede the coded payload.</summary>
private const int FrequencyHeaderSize = 1024;
/// <summary>One decoded cliloc row. Mirrors <c>Ultima.StringEntry</c>'s three fields.</summary>
public struct Entry
{
public int Number;
public byte Flag;
public string Text;
public Entry(int number, byte flag, string text)
{
Number = number;
Flag = flag;
Text = text;
}
}
// ── Container detection ──────────────────────────────────────────────────────────────
/// <summary>
/// True when the file looks like the Mythic container. The marker is the high byte of
/// the first DWORD being <c>0x8E</c> — which is not a magic number in the file so much
/// as a consequence of <see cref="HeaderXorKey"/>: a plausible decompressed length is
/// small enough that its top byte is zero, so the XOR leaves 0x8E showing.
/// </summary>
public static bool LooksCompressed(byte[] buffer)
{
return buffer != null && buffer.Length >= 4 && buffer[3] == 0x8E;
}
// ── The public entry point ───────────────────────────────────────────────────────────
/// <summary>
/// Reads a cliloc file, compressed or plain, and returns its entries.
///
/// Tries the layout the header suggests first and the other one second — the same
/// fallback UOFiddler performs, and the reason an already-converted file passes
/// straight through. <paramref name="warning"/> is non-null when a layout parsed
/// *partially*: that is the case a caller must surface rather than swallow, because a
/// quietly short table is the failure mode the website's importer refuses.
/// </summary>
public static bool TryLoadFile(string path, out List<Entry> entries, out string warning, out string error)
{
entries = null;
warning = null;
error = null;
byte[] buffer;
try
{
buffer = File.ReadAllBytes(path);
}
catch (Exception e)
{
error = "cannot read " + path + ": " + e.Message;
return false;
}
return TryLoad(buffer, out entries, out warning, out error);
}
/// <summary>Reads an in-memory cliloc file. See <see cref="TryLoadFile"/>.</summary>
public static bool TryLoad(byte[] buffer, out List<Entry> entries, out string warning, out string error)
{
entries = null;
warning = null;
error = null;
bool compressedFirst = LooksCompressed(buffer);
List<Entry> primary;
string primaryError;
bool primaryComplete;
if (TryParse(buffer, compressedFirst, out primary, out primaryComplete, out primaryError) && primaryComplete)
{
entries = primary;
return true;
}
List<Entry> fallback;
string fallbackError;
bool fallbackComplete;
if (TryParse(buffer, !compressedFirst, out fallback, out fallbackComplete, out fallbackError) && fallbackComplete)
{
entries = fallback;
return true;
}
// Neither layout parsed to the end. Take whichever salvaged more rows and say so.
int primaryCount = primary == null ? 0 : primary.Count;
int fallbackCount = fallback == null ? 0 : fallback.Count;
if (primaryCount == 0 && fallbackCount == 0)
{
error = "as " + Label(compressedFirst) + ": " + primaryError
+ "; as " + Label(!compressedFirst) + ": " + fallbackError;
return false;
}
if (primaryCount >= fallbackCount)
{
entries = primary;
warning = "parsed partially as " + Label(compressedFirst) + ": " + primaryError
+ " (" + primaryCount + " entries salvaged)";
}
else
{
entries = fallback;
warning = "parsed partially as " + Label(!compressedFirst) + ": " + fallbackError
+ " (" + fallbackCount + " entries salvaged)";
}
return true;
}
private static string Label(bool compressed)
{
return compressed ? "compressed" : "uncompressed";
}
// ── Record layout ────────────────────────────────────────────────────────────────────
/// <summary>
/// Walks the plain record layout: a 4-byte and a 2-byte header, then repeating
/// [int32 number][byte flag][uint16 length][length bytes of UTF-8].
///
/// <paramref name="complete"/> distinguishes "parsed to the end of the file" from
/// "stopped early but salvaged rows", which is the distinction the caller needs and
/// an exception would destroy.
/// </summary>
private static bool TryParse(byte[] buffer, bool decompress, out List<Entry> entries, out bool complete, out string error)
{
entries = new List<Entry>();
complete = false;
error = null;
byte[] data;
if (decompress)
{
if (!TryDecompress(buffer, out data, out error))
return false;
}
else
{
data = buffer;
}
if (data.Length < 6)
{
error = "file is " + data.Length + " bytes, smaller than the 6-byte header";
return false;
}
int cursor = 6; // int32 version marker + int16 language marker
int lastNumber = -1;
while (cursor < data.Length)
{
int entryStart = cursor;
int remaining = data.Length - cursor;
if (remaining < 7)
{
error = "unexpected " + remaining + " trailing byte(s) at 0x" + entryStart.ToString("X")
+ " after entry #" + lastNumber + "; an entry header needs 7";
return true;
}
int number = ReadInt32(data, cursor);
byte flag = data[cursor + 4];
// Deliberately UNSIGNED. Read as Int16, a string of 32768 bytes or more comes back
// negative and corrupts every record after it.
int length = data[cursor + 5] | (data[cursor + 6] << 8);
cursor += 7;
if (length > data.Length - cursor)
{
error = "entry #" + number + " at 0x" + entryStart.ToString("X") + " declares length "
+ length + " but only " + (data.Length - cursor) + " byte(s) remain (parsed "
+ entries.Count + " so far)";
return true;
}
string text;
try
{
text = Encoding.UTF8.GetString(data, cursor, length);
}
catch (Exception e)
{
error = "entry #" + number + " at 0x" + entryStart.ToString("X") + " has " + length
+ " body bytes that are not valid UTF-8: " + e.Message;
return true;
}
cursor += length;
entries.Add(new Entry(number, flag, text));
lastNumber = number;
}
complete = true;
return true;
}
// ── Mythic stage 1: the XOR header and the move-to-front code ────────────────────────
/// <summary>
/// Reads the obfuscated decompressed length from the first DWORD. Public so a caller
/// can size a buffer before committing to the decode.
/// </summary>
public static uint PeekDecompressedLength(byte[] source)
{
if (source == null || source.Length < 4)
return 0;
return ReadUInt32(source, 0) ^ HeaderXorKey;
}
/// <summary>
/// Decompresses the Mythic container: strip the 4-byte length header, undo the
/// move-to-front coding, then run stage 2.
/// </summary>
public static bool TryDecompress(byte[] source, out byte[] output, out string error)
{
output = null;
error = null;
if (source == null || source.Length < 4)
{
error = "payload shorter than the 4-byte length header";
return false;
}
uint dataLength = ReadUInt32(source, 0) ^ HeaderXorKey;
// A wrong guess about the container makes this astronomically large, which is the
// cheapest possible rejection and must happen before any allocation.
if (dataLength == 0 || dataLength > int.MaxValue)
{
error = "implausible decompressed length " + dataLength + " — not the compressed layout";
return false;
}
var mtf = new byte[source.Length - 4];
MoveToFrontDecode(source, 4, mtf);
var destination = new byte[(int)dataLength];
int written;
if (!TryInternalDecompress(mtf, destination, out written, out error))
return false;
if (written != (int)dataLength)
{
error = "decompressed " + written + " bytes, header declared " + dataLength;
return false;
}
output = destination;
return true;
}
/// <summary>
/// Move-to-front decode. Each input byte is an index into a 256-symbol table; the
/// symbol found there is emitted and moved to the front.
/// </summary>
private static void MoveToFrontDecode(byte[] input, int offset, byte[] output)
{
var symbols = new byte[256];
for (int i = 0; i < 256; i++)
symbols[i] = (byte)i;
for (int i = 0; i < output.Length; i++)
{
int index = input[offset + i];
byte symbol = symbols[index];
output[i] = symbol;
for (int j = index; j > 0; j--)
symbols[j] = symbols[j - 1];
symbols[0] = symbol;
}
}
// ── Mythic stage 2 ───────────────────────────────────────────────────────────────────
/// <summary>
/// Turns the MTF-decoded payload back into the original bytes.
///
/// The payload is a 1024-byte frequency header (256 little-endian int32 symbol counts)
/// followed by the coded stream. The counts partition the stream into one run per
/// symbol; <c>cursor[]</c> holds each run's read position and <c>limit[]</c> its end,
/// and the walk emits a symbol, advances that symbol's run, and re-orders the symbol
/// table by the index it reads.
///
/// Every index derived from file content is checked. Upstream's equivalent is wrapped
/// in a blanket catch; here a malformed file is a <c>false</c> with a reason.
/// </summary>
private static bool TryInternalDecompress(byte[] input, byte[] destination, out int written, out string error)
{
written = 0;
error = null;
if (input.Length < FrequencyHeaderSize)
{
error = "payload (" + input.Length + " bytes) is smaller than the 1024-byte frequency header";
return false;
}
var counts = new int[256]; // symbol → number of occurrences
var cursor = new int[256]; // symbol → next unread position in its run
var limit = new int[256]; // symbol → one past the end of its run
int sum = 0;
for (int i = 0; i < 256; i++)
{
counts[i] = ReadInt32(input, i * 4);
if (counts[i] < 0)
{
error = "frequency header declares a negative count for symbol " + i;
return false;
}
sum += counts[i];
if (sum < 0)
{
error = "frequency header sums past int range at symbol " + i;
return false;
}
}
if (sum == 0)
{
written = 0;
return true;
}
if (destination.Length < sum)
{
error = "destination holds " + destination.Length + " bytes, payload needs " + sum;
return false;
}
int nonZeroCount = 0;
for (int i = 0; i < 256; i++)
{
if (counts[i] != 0)
nonZeroCount++;
}
// The coded stream must be long enough to hold one index per emitted byte.
if (input.Length - FrequencyHeaderSize < sum)
{
error = "coded stream holds " + (input.Length - FrequencyHeaderSize) + " bytes, frequency header claims " + sum;
return false;
}
var order = new byte[256];
FrequencyOrder(counts, order);
var symbolTable = new byte[256];
for (int i = 0; i < 256; i++)
symbolTable[i] = (byte)i;
for (int i = 0, m = 0; i < nonZeroCount; ++i)
{
byte symbol = order[i];
// m indexes the coded stream and comes from the file's own counts.
if (m < 0 || m >= input.Length - FrequencyHeaderSize)
{
error = "run table for symbol " + symbol + " starts at " + m + ", past the coded stream";
return false;
}
symbolTable[input[m + FrequencyHeaderSize]] = symbol;
cursor[symbol] = m + 1;
m += counts[symbol];
limit[symbol] = m;
}
byte val = symbolTable[0];
int count = 0;
int liveSymbols = nonZeroCount;
do
{
destination[count] = val;
if (cursor[val] < limit[val])
{
int at = cursor[val] + FrequencyHeaderSize;
if (at < FrequencyHeaderSize || at >= input.Length)
{
error = "run for symbol " + val + " reads at " + at + ", past the " + input.Length + "-byte payload";
return false;
}
byte index = input[at];
cursor[val]++;
if (index != 0)
{
ShiftLeft(symbolTable, index);
symbolTable[index] = val;
val = symbolTable[0];
}
}
else if (liveSymbols-- > 0)
{
ShiftLeft(symbolTable, liveSymbols);
val = symbolTable[0];
}
count++;
}
while (count < sum);
written = sum;
return true;
}
/// <summary>
/// Orders symbols by descending frequency: repeatedly take the largest remaining count
/// and record its symbol. Ties go to the lower symbol, because the scan keeps the first
/// strictly-greater value — matching upstream, and the tie-break is load-bearing.
/// </summary>
private static void FrequencyOrder(int[] counts, byte[] output)
{
var tmp = new int[256];
Array.Copy(counts, tmp, 256);
for (int i = 0; i < 256; i++)
{
int best = 0;
byte index = 0;
for (int j = 0; j < 256; j++)
{
if (tmp[j] > best)
{
index = (byte)j;
best = tmp[j];
}
}
if (best == 0)
break;
output[i] = index;
tmp[index] = 0;
}
}
/// <summary>Shifts <c>[1..element]</c> down one slot, dropping element 0.</summary>
private static void ShiftLeft(byte[] input, int element)
{
for (int i = 0; i < element; ++i)
input[i] = input[i + 1];
}
// ── Little-endian readers (BinaryPrimitives is not available on net48) ───────────────
private static int ReadInt32(byte[] b, int at)
{
return b[at] | (b[at + 1] << 8) | (b[at + 2] << 16) | (b[at + 3] << 24);
}
private static uint ReadUInt32(byte[] b, int at)
{
return (uint)(b[at] | (b[at + 1] << 8) | (b[at + 2] << 16) | (b[at + 3] << 24));
}
}
}

View File

@@ -155,6 +155,14 @@ namespace Server.Custom
case "partprobe":
BridgeParticipationProbe.Run(null, Arg(parts, 1), Int(Arg(parts, 2)), Int(Arg(parts, 3)));
break;
// Asset Bridge phase 0. Here for the same reason as partprobe, and for one more:
// the point of that spike is comparing the STOCK client's answers with a patched
// client's, and `AssetProbeOnStart` can only ever run whichever one the config
// names. Driving it from here runs both against a single boot, so a difference
// between them cannot be a difference between two shard processes.
case "assetprobe":
BridgeAssetProbe.Begin(null, Arg(parts, 1) ?? "all", Arg(parts, 2));
break;
// Phase 12a. `world.despawn` answering `gone` rather than `removed` is the
// path a player takes every time they kill an event creature, and it is the one
// outcome the rig cannot reach by asking the bridge: every bridge verb that

View File

@@ -14,10 +14,12 @@ These two scripts produced the measured budget in [PLAN.md](https://gitea.whitlo
| `BridgeCrierProbe.cs` | `Scripts/Custom/BridgeCrierProbe.cs` | Logs the global town-crier entry list every 3s so `towncrier.add` / `remove` can be seen landing in game state. Flag: `CrierProbeOnStart`. |
| `BridgeVendorSaleProbe.cs` | `Scripts/Custom/BridgeVendorSaleProbe.cs` | Fires `PlayerVendorSale` (Phase 7) with real seeded-vendor data so `vendor.sale` can be verified without a live buy. Requires the Phase 7 patches applied. Flag: `VendorSaleProbeOnStart`. |
| `BridgeDemoDress.cs` | `Scripts/Custom/BridgeDemoDress.cs` | Renames a seeded world so it is presentable in a screenshot: shop signs, vendor and character names, house signs. Also stages a few condemned houses back into IDOC, and sets a known password on `seed_000` so a character can be logged in. Flags: `DemoDressOnStart`, `DemoDressPassword`. In game: `[demodress`. |
| `BridgeRigDriver.cs` | `Scripts/Custom/BridgeRigDriver.cs` | Drives the shard from OUTSIDE the game, one verb per line in `Config/rigcmd.txt`, which the driver polls and truncates. Written for the engagement Phase 11b acceptance walk, where each step's assertion is what happened BETWEEN two steps, so the steps have to be separated by the observer rather than by a hard-coded delay -- and ServUO's console takes a fixed verb set (`Scripts/Misc/ConsoleCommands.cs`), so `[p5probe` cannot be typed at a headless shard at all. Verbs: `decaylist`, `decay`, `vendorlist`, `vendorfunds`, `citylist`, `governor`, `election`, `activate`, `password`, `configset`, `configread`, `partprobe`, `save`, `shutdown`. Flag: `RigDriverEnabled`. `configset` exists because **`Config.Get` is written by exactly ONE caller in the whole of ServUO 57.4** (`Server/ScriptCompiler.cs`): no in-game command, gump or console verb writes a config key, so on a stock shard a GM cannot drift a configuration lease even deliberately, and a lease's compare-and-set restore would have no way to be proved. `configread` reads a key back through `Config.Get` long after every type initialiser has run, which is how a key that TOOK is told from one that only appeared to. **Sets passwords, writes live config and mutates the world.** |
| `BridgeRigDriver.cs` | `Scripts/Custom/BridgeRigDriver.cs` | Drives the shard from OUTSIDE the game, one verb per line in `Config/rigcmd.txt`, which the driver polls and truncates. Written for the engagement Phase 11b acceptance walk, where each step's assertion is what happened BETWEEN two steps, so the steps have to be separated by the observer rather than by a hard-coded delay -- and ServUO's console takes a fixed verb set (`Scripts/Misc/ConsoleCommands.cs`), so `[p5probe` cannot be typed at a headless shard at all. Verbs: `decaylist`, `decay`, `vendorlist`, `vendorfunds`, `citylist`, `governor`, `election`, `activate`, `password`, `configset`, `configread`, `partprobe`, `assetprobe`, `save`, `shutdown`. Flag: `RigDriverEnabled`. `configset` exists because **`Config.Get` is written by exactly ONE caller in the whole of ServUO 57.4** (`Server/ScriptCompiler.cs`): no in-game command, gump or console verb writes a config key, so on a stock shard a GM cannot drift a configuration lease even deliberately, and a lease's compare-and-set restore would have no way to be proved. `configread` reads a key back through `Config.Get` long after every type initialiser has run, which is how a key that TOOK is told from one that only appeared to. **Sets passwords, writes live config and mutates the world.** |
| `BridgeProtocol5Probe.cs` | `Scripts/Custom/BridgeProtocol5Probe.cs` | Drives all three Protocol 5 enrichments so their frames can be observed: walks one house Fairly -> Greatly -> IDOC (the PAIR is the assertion -- `estimatedCollapse` must appear only on the IDOC frame), reports each player vendor's fee state straight off the `PlayerVendor` so the emitted `fees` block can be checked against the shard's own numbers, and fires `EventSink.AccountLogin`. Flags: `Protocol5ProbeOnStart`, `Protocol5ProbeAccount`, `Protocol5ProbePassword`. In game: `[p5probe`. **Sets a password on the named account.** |
| `BridgeProtocol6Probe.cs` | `Scripts/Custom/BridgeProtocol6Probe.cs` | Spawns a real champion boss through the shard's own `SpawnChampion()`, waits two champ sweeps so the boss is attributed to its altar, registers unequal damage from two seeded players and kills it -- so `champ.boss.killed` can be observed with a real damage table. **The wait is the assertion**: without it the kill still emits, but with no `serial`/`type`/`level`, which is the documented fallback rather than the case being tested. The altar is placed inside a NAMED region on purpose (see below). Flag: `Protocol6ProbeOnStart`. In game: `[p6probe`. **Spawns and kills a champion boss; rig only.** Protocol 6's other half, the idempotency key, needs no probe -- it is driven from outside with two identical POSTs to the sidecar. |
| `BridgeParticipationProbe.cs` | `Scripts/Custom/BridgeParticipationProbe.cs` | Produces real kill credit inside a participation area with no game client: moves two player mobiles to the venue, spawns a creature there, damages it unequally from both and kills it. **Presence is the half it cannot drive** -- the sweep credits players with a live `NetState`, which is the correct test and not one a probe should loosen, so presence accrual needs a real login. In game: `[partprobe <map> <x> <y>`; from a headless rig, through `BridgeRigDriver`'s `partprobe` verb (the two ship together for that reason). **Moves players and spawns and kills a creature; rig only.** |
| `BridgeAssetProbe.cs` | `Scripts/Custom/BridgeAssetProbe.cs` | **Asset Bridge phase 0** (docs/link/v8.md §16). Drives ServUO's vendored `Ultima` decoders from inside a running shard against a deliberately patched client, and compares every answer with what a pre-flight validator says about the index entry *before* the call. The interesting column is not the error count, it is **WRONG PICTURES** -- records the validator rejects and the library renders anyway. Sweeps statics, land, all 2,048 bodies, the player-character bodies from `Race.AllRaces`, and the ported Mythic cliloc reader against UOFiddler's own output. In game / from `BridgeRigDriver`: `[assetprobe [section] [stock|patched]`. Flag: `AssetProbeOnStart`. **Its `gump` section deliberately kills the shard** and is never part of `all`. |
| `BridgeMythicCliloc.cs` | `Scripts/Custom/BridgeMythicCliloc.cs` | The §9 reader for the **Mythic compressed** cliloc container -- the one decoder Protocol 8 writes rather than calls. Ported from UOFiddler (Beerware) into net48 C# with every file-derived index bounds-checked, which upstream's blanket `catch` does not do. Reproduces UOFiddler's 123,490-entry table exactly. **Phase 2 promotes this file into `overlay/`**; it is scaffolding only for as long as it is a spike. |
## Deploy overwrites Bridge.cfg
@@ -189,3 +191,127 @@ value that lies, printed next to a frame that disagrees with it.
Emitting from a named region is therefore the test. At a dungeon altar the field is legitimately
absent and the probe proves nothing about it.
## What phase 0 found
`BridgeAssetProbe` exists because [v8.md](https://gitea.whitlocktech.com/RunicGateway/docs/src/branch/main/link/v8.md) §4 chose to **call** ServUO's vendored `Ultima` rather than reimplement it, and the evidence for that choice was a PowerShell probe against a stock client — neither the process nor the client the extractor will actually run in. These are its results, from inside a running ServUO 57.4 against this machine's client, and against a copy broken in 21 catalogued ways by `tools/patch_client.ps1`.
### The UOP wins outright, and it took a whole run to notice
`FileIndex`'s UOP constructor ends with a bare `MulPath = uopPath`. **When `artLegacyMUL.uop` is present it wins, and `art.mul` / `artidx.mul` are never opened at all.** Every current client ships the UOP, so:
- A validator that bounds an index offset against `art.mul` while the index holds UOP offsets is not approximate, it is nonsense. The first run of this probe refused **34,299 perfectly good statics** for "declaring 10533x2085" — and every one of those refusals looked like a real finding. `BridgeAssetValidator.ArtDataPath()` now mirrors `FileIndex`'s own resolution order, and phase 1 must too.
- A custom-art shard that adds graphics to `art.mul` while the UOP is still in place **gets nothing**, silently. That is an operator trap rather than a bug in this protocol, but the extractor is where it will be noticed.
- The `corrupt` and `customart` tiers of `patch_client.ps1` therefore need its `nouop` tier to mean anything at all. Without it they report that they applied, and change nothing.
### 22,102 wrong pictures on a stock, unmodified client
The counts that matter, `assetprobe all stock`:
```
statics 0..65535 ok 39,189 WRONG PICTURES (empty record) 9,962 threw 16,385
land 0..16383 ok 4,244 WRONG PICTURES (empty record) 12,140
```
Those 22,102 ids have an index entry of `lookup 0, length 0`**no record at all**. `FileIndex.Seek` treats that as a hit (it rejects `lookup < 0` and `length < 0`, and zero is neither), hands back the stream, and `LoadStatic` decodes `length` = 0 bytes into `m_StreamBuffer` — which is **reused, only ever grown, and filled by a `stream.Read` whose return value is discarded**. So the id renders whatever the previously-decoded asset left in the buffer.
**It is specific to the UOP path.** Run the same sweep against the mul path and those ids come back empty and honest, because `artidx.mul` stores `-1` for an absent record while unmapped UOP slots are simply zeroed structs. That is also why the earlier PowerShell probe counted 32,766 of these as "ok": they decode, they raise nothing, and no success count can tell them from art.
A bulk import that trusted the library would have written 22,102 duplicate images into the site under ids that have no art. This one measurement is the argument for validate-before-calling.
### Every deliberate defect was caught by the validator and rendered by the library
`assetprobe all patched`, against the 21-defect client:
```
statics ok 39,190 absent 9,954 refused 1 WRONG PICTURES (bad record) 6 threw 16,385
land ok 4,243 absent 12,140 WRONG PICTURES (bad record) 1
```
| id | the defect | what the library did |
|---|---|---|
| `static/4104` | lookup 4 KB past the end of `art.mul` | returns nothing — `Seek` does check the record's **start** |
| `static/4105` | starts 64 bytes before EOF, declares 8,192 | **renders the previous asset**`Seek` never checks the record's **end** |
| `static/4108` | declared length 4, smaller than the header | renders something |
| `static/4109` | header declares 8000x8000 | **allocates it** — a ~128 MB bitmap from two bytes in a file, and the same field can ask for 65535×65535 |
| `static/4111` | row table points 60,000 words outside a 512-byte record | renders — `LoadStatic`'s two guards bound the *write* into the bitmap, and nothing bounds the *read* |
| `static/4112` | a 16-pixel run declared in a 20-byte record | renders |
| `static/4131` | verdata entry whose lookup is past verdata.mul's own end | renders — **`Verdata.Seek` has no bounds check whatsoever** |
| `land/256` | 512-byte land record | renders — `LoadLand` reads a fixed 2,024 bytes whatever the length says |
Seven of the eight produce a confident, wrong picture and raise nothing anywhere.
The validator refused all eight, and refused **nothing** on the stock client across 49,151 statics and 16,384 land tiles. That second number is the one that matters: a checker that refuses real art is worse than no checker, so "zero false refusals on a clean client" is what makes validate-before-calling more than a hopeful phrase.
The eight `customart` ids appended past the stock ceiling all decode cleanly, which is that tier's whole point — the ceiling is a property of a file, not a constant anyone should write down.
### Two more ways to get a wrong answer out of an id that has no art
- **`Art.GetStatic(id, false)` throws `IndexOutOfRangeException` for `id >= 49,152`** rather than returning null — 16,385 of them in a full sweep.
- **`Art.GetStatic(id)` with the default `checkmaxid: true` is worse**: `GetLegalItemID` maps an out-of-range id to **0**, so the call returns **item 0's picture**. An exception is recoverable; a picture of the wrong item is not even detectable.
So the extractor takes its id ceiling from the index it opened, and passes `checkmaxid: false` so an overrun is loud rather than plausible.
### The gump crash reproduces in-process, and nothing catches it
`assetprobe gump` called `Ultima.Gumps.GetGump(2)` once. **The ServUO process disappeared** — no exception line in the report, no `catch` reached, no shutdown, nothing in the console. The report ends mid-section, and `checkpoint.txt` reading `gump 2` is the entire record of what happened. That is exactly why the checkpoint is written *before* the call and flushed.
`AccessViolationException` is a corrupted-state exception and .NET Framework 4.8 does not deliver it to ordinary handlers, so **there is no in-process defence** — on a live shard this is a crash with players on it. "Nothing calls `Ultima.Gumps`" is a safety rule, and phase 0's job was to make sure that sentence had been earned rather than assumed. It has.
### The cliloc port is byte-identical to UOFiddler
```
123,490 entries in 218 ms (55,986 blank, 67,504 would be stored)
vs UOFiddler: 123,490 identical, 0 differ, 0 only ours, 0 only theirs
```
§9 is proven: the shard can produce the whole table with no UOFiddler installed, no `dotnet build`, and no 5 MB file copied to a server.
The reference is what makes this a test rather than a demonstration. A subtly wrong inverse-BWT coder still produces a plausible table — mostly-right strings with a few mangled ones is the *expected* shape of a bug in this algorithm, and a row count alone would sail past it.
Note the blank count is **55,986**, not the 55,994 recorded from the manual pipeline. The difference is eight whitespace-only entries, blank to a `trim()` and not to `IsNullOrEmpty` — a definition rather than a defect, but exactly the sort of eight-row drift that gets investigated as one.
### What phase 0 did not cover, and phase 1 must
**The animation path has no validator.** The patched client's verdata entry for body 34 points past verdata.mul's end and the wolf still "decoded" — counted among the 1,144 successes, silently rendering something else, with nothing in the report to say so. `GetAnimation` also allocates `new int[frameCount]` straight from a file-supplied int. Everything above about statics applies here and none of it is implemented yet.
The deliberate `Bodyconv.def` mis-mappings (bodies 1900 and 1901) produced **nothing** rather than a wrong creature on this client, so they did not reproduce the spider. The gargoyle rows remain the real evidence for the never-sweep-file-types rule: 666, 667, 694 and 695 report nothing, and nothing is the correct answer.
### Reference: the rest of the run
```
bodies 0..2047, direction 1 decoded 1,144 empty 904 faulted 0
by file type: 1=1222, 2=140, 3=244, 4=150, 5=292
player bodies (Race.AllRaces, direction 0) 6 decoded, 6 absent, of 12
Human 400 / 401 decode; ghosts 402 / 403 absent
Elf 605 / 606 / 607 / 608 all decode
Gargoyle 666 / 667 / 694 / 695 all absent
```
Two details worth keeping. The body counts reproduce the PowerShell probe **exactly**, from a different process against the same files, which is what makes the two runs comparable at all. And the gargoyle *ghost* bodies resolve to file type **1**, not 5 like the living gargoyle bodies — so "the gargoyle is an anim5 problem" is not quite the shape of it.
## Building the patched client
```powershell
.\tools\patch_client.ps1 -Dest D:\uo-patched-client
```
Copies a client (~3.5 GB) and breaks the copy in five catalogued tiers — `nouop`, `verdata`, `customart`, `corrupt`, `bodyconv`. **It never writes to the source**: every file it touches is hashed in the source before and after, and a changed hash aborts the run. Each defect is recorded in `patched-client.manifest.json` beside the copy, which is what makes a nonzero WRONG PICTURES count readable as "the tier worked" instead of "something broke".
Then point the shard at it and drive the probe:
```ini
RigDriverEnabled=true
AssetProbeClient=D:\uo-patched-client
AssetProbeClilocRef=<a clilocs.tsv from website/server/tools/cliloc-export --tsv>
```
```
assetprobe all stock # the baseline: the validator must refuse nothing here
assetprobe all patched # the experiment
```
Run both against **one boot**, through `rigcmd.txt`, so a difference between them cannot be a difference between two shard processes. Without `AssetProbeClilocRef` the cliloc section reports a row count, which proves nothing about the strings.
**The copy is EA's client art.** It stays on the machine that made it, exactly like every other extraction in this project, and is never committed.