RunicGateway/servuo-plugins
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

feat(asset-bridge): the transport, and the 357 wrong pictures it found

Browse Source 
Asset Bridge phase 1 (docs/link/v8.md §16). Sidecar half: RunicGateway/link#41.
Docs half: RunicGateway/docs#236.

The transport for protocol 8, plus phase 0's validator promoted into the overlay
and extended to animations — which is where the interesting part is.

## 357 of the 1,144 "decodable" bodies are wrong pictures, on a STOCK client

Phase 0 measured the art path and left the animation half unbuilt. It has the same
defect, and it is worse: `GetAnimation` decodes through
`new MemoryStream(m_StreamBuffer, false)` — the whole shared buffer, not the
`length` bytes just read into it — so a truncated or absent record does not even hit
end-of-stream. It sails on into the previous animation's bytes.

Measured directly, because no count could tell:

| Decode body 320 (`lookup 22638982, length 0`) straight after… | Comes back |
|---|---|
| body 12, the dragon | the dragon, 176x167, identical hash |
| body 34, the wolf | the wolf's dimensions, 35x34 |
| body 400, the human male | the human, 27x63, identical hash |

The catalogue is **787 bodies, not 1,144**. Importing the other 357 would have written
duplicate creature portraits into the site showing whichever body the walk decoded
before them.

The record walk refused **0** real bodies on the stock client — the false-refusal
measurement §4.5 says the boundary depends on.

## And four of the twelve player bodies, not six

§5.2 listed the elf ghosts (607, 608) as decoding. Their index entry is `length 0`;
what came back was the elf female at her exact dimensions, because 606 is what the
walk decoded immediately before. Confirmed the same way — 607 after the dragon is
the dragon. Phase 4's UOP decoder now covers eight ids rather than six.

## What is here

- **`overlay/Scripts/Custom/Bridge/BridgeAssets.cs`** — the plane. Accepts on the Core
  thread, hands off to a dedicated asset worker, returns immediately. Three rules, all
  answering a specific failure:
  - **one slot**, second request answered `bridge.busy` (425). `Emit`'s queue is bounded
    in *lines*, so 10,000 queued 200 KB replies is 2 GB of shard memory; the bound that
    holds is flow control, on the side where the memory is.
  - **byte budgets** (`AssetBatchBytes`, 512 KiB) under the sidecar's new 1 MiB cap. The
    factor of two is load-bearing: a page always admits its first item, so it may
    overshoot by one, and the headroom is what makes that land on the wire.
  - **replies, never events** — no `reqId`, no answer. An uncorrelated frame is an event
    by definition, and §3.1 is why none of this may be one.
- **`PageBuilder`** — one paging envelope (`more`/`cursor`/`cut`) for all five families
  that will page, defined before the first one needs it. `cut` matters: "short page" has
  three meanings and only `end` means finished.
- **`assets.sources`** — stage 1 of the import gate, its first user.
- **`BridgeAssetValidator.cs`** — promoted from `tools/`, plus `ResolveAnimation` (the
  never-sweep-file-types rule as code, with no loop and no fallback),
  `AnimationRecordSane` and the frame walk.
- **`EXTRACTOR_VERSION`**, **`overlay.toml` protocol 7 → 8**, `AssetsEnabled`.

## Hashing had to come off the request path

§6's gate is (size, mtime) first, hash only when those differ. The first call has nothing
cached, so that still means hashing 1.06 GB — inside the sidecar's 10 s reply timeout it
does not fit. So hashes are computed on their own thread (deliberately not the single-slot
worker, which would answer every status poll `bridge.busy` for the whole pass) and the
reply carries `hashing`/`complete`.

Measured on the real rig: first call instant with `sha256: null`, second call **44 ms**
with every hash present.

## Verified on the wire, not just compiled

Real ServUO 57.4 + the real sidecar + the real client. `GET /assets/sources` → 200,
`X-UOLink-Version: 8`, `imaging: {ok: true}`, and §4.6's diagnostic firing on a live
client: `artDataFile: artlegacymul.uop`, with `art.mul` and `artidx.mul` both carrying
`shadowedBy`. Live events kept flowing through the new capped reader with no warnings.

Not exercised live: the disabled-plane 403 and the busy 425 (both unit-tested on the
sidecar side; the shard halves are a config read and a lock).

- [x] AI-assisted — Claude Code (Opus 5)

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
This commit is contained in:
claude
2026-09-10 08:32:18 -05:00
parent 1b7edebd31
commit c89e818dbf
7 changed files with 1683 additions and 428 deletions
Download Patch File Download Diff File Expand all files Collapse all files

571
tools/scaffolding/BridgeAssetProbe.cs
View File

@@ -7,6 +7,7 @@ using System.Text;
using System.Threading;
using Server.Commands;
using Server.Custom.Bridge;
using Ultima;
@@ -475,76 +476,109 @@ namespace Server.Custom
/// nothing downstream can tell. A "0 rows" outcome is the correct answer.
///
/// So the sweep records the file type each body resolved to and whether that file
/// answered — and never a second opinion.
/// answered — and never a second opinion. <c>ResolveAnimation</c> is that rule as
/// code, and this sweep is now one of its callers rather than its own transcription.
///
/// **Phase 1 added the validator to this sweep**, which phase 0 ran without one:
/// it reported "1,144 decoded, 0 faults" while the patched client's verdata entry for
/// body 34 pointed past verdata.mul's own end and the wolf rendered something else,
/// counted among those 1,144. REFUSED-BUT-DECODED is the cell that says so, and it is
/// the same cell the art sweeps have had since phase 0.
/// </summary>
private static void SectionBodies()
{
Head("bodies — Animations.GetAnimation, one direction, first frame");
int direction = Config.Get("Bridge.AssetProbeCreatureDirection", 1);
int decoded = 0, empty = 0, faulted = 0;
var tally = new Tally();
var byFileType = new int[8];
var faults = new List<string>();
int unresolved = 0;
for (int body = 0; body < 2048; body++)
var indexes = new Dictionary<int, FileIndex>();
var readers = new Dictionary<int, BridgeAssetValidator.RecordReader>();
var lengths = new Dictionary<int, long>();
string verdataPath = Files.GetFilePath("verdata.mul");
long verdataLength = BridgeAssetValidator.MulLength(verdataPath);
try
{
Checkpoint("bodies", body);
int translated = body;
int fileType;
try
for (int body = 0; body < 2048; body++)
{
fileType = BodyConverter.Convert(ref translated);
}
catch (Exception e)
{
faulted++;
faults.Add("body " + body + " BodyConverter.Convert: " + e.GetType().Name + ": " + e.Message);
continue;
}
Checkpoint("bodies", body);
if (fileType >= 0 && fileType < byFileType.Length)
byFileType[fileType]++;
int fileType, at;
string reason;
try
{
int hue = 0;
var frames = Animations.GetAnimation(body, 0, direction, ref hue, false, true);
if (frames != null && frames.Length > 0 && frames[0] != null && frames[0].Bitmap != null)
if (!BridgeAssetValidator.ResolveAnimation(body, 0, direction, out fileType, out at, out reason))
{
decoded++;
frames[0].Bitmap.Dispose();
// The never-sweep-file-types rule's own outcome (§4.3): bodyconv sent this
// body to a file this client does not have, so we report nothing and ask
// no other file. Asking anim2 for gargoyle 666 returns a giant spider.
unresolved++;
continue;
}
else
{
empty++;
}
}
catch (Exception e)
{
faulted++;
if (faults.Count < 40)
faults.Add("body " + body + " (fileType " + fileType + "): " + e.GetType().Name + ": " + e.Message);
if (fileType >= 0 && fileType < byFileType.Length)
byFileType[fileType]++;
if (!indexes.ContainsKey(fileType))
{
string dataPath = BridgeAssetValidator.AnimDataPath(fileType);
indexes[fileType] = BridgeAssetValidator.OpenAnimIndex(fileType);
lengths[fileType] = BridgeAssetValidator.MulLength(dataPath);
readers[fileType] = new BridgeAssetValidator.RecordReader(dataPath, verdataPath);
}
var index = indexes[fileType];
var verdict = BridgeAssetValidator.CheckEntry(
index, at, lengths[fileType], verdataLength, out reason);
// Only the entry has been judged so far. The record behind it is where the
// frame table and the unbounded run headers live.
if (verdict == BridgeAssetValidator.Verdict.Ok
&& !readers[fileType].AnimationSane(index, at, 1, out reason))
{
verdict = BridgeAssetValidator.Verdict.Refused;
}
bool decoded = false;
string thrown = null;
try
{
int hue = 0;
var frames = Animations.GetAnimation(body, 0, direction, ref hue, false, true);
if (frames != null && frames.Length > 0 && frames[0] != null && frames[0].Bitmap != null)
{
decoded = true;
frames[0].Bitmap.Dispose();
}
}
catch (Exception e)
{
thrown = e.GetType().Name + ": " + e.Message;
}
Record(tally, verdict, reason, decoded, thrown, "body/" + body + "/a0");
}
}
finally
{
foreach (var reader in readers.Values)
{
if (reader != null)
reader.Dispose();
}
}
Say("direction " + direction + " (creature default — §5.1)");
Say(string.Format(" decoded {0} empty {1} FAULTED {2}", decoded, empty, faulted));
Say(string.Format(" bodyconv resolves nowhere (correct — never swept): {0:N0}", unresolved));
Say(" by file type: " + string.Join(", ", FileTypeCounts(byFileType)));
if (faults.Count > 0)
{
Say("");
Say(" faults:");
foreach (var f in faults)
Say(" " + f);
}
Say("");
tally.Report("bodies 0..2047, action 0, first frame");
}
private static string[] FileTypeCounts(int[] byFileType)
@@ -566,10 +600,16 @@ namespace Server.Custom
/// The twelve (on stock 57.4) player-character bodies, each at direction 0 — head-on,
/// because a character is a portrait and should look at you (§5.1).
///
/// Six of them are expected to report nothing on the legacy path: both human ghosts and
/// every gargoyle body are UOP-only. **That is the measurement, not a failure** — it is
/// what phase 4's UOP reader exists for, and a probe that flagged it red would teach an
/// Most of them are expected to have no art on the legacy path — the ghosts and every
/// gargoyle body are UOP-only. **That is the measurement, not a failure**: it is what
/// phase 4's UOP reader exists for, and a probe that flagged it red would teach an
/// operator to ignore the panel.
///
/// **What is a failure is the middle column.** Phase 0 ran this without the validator
/// and read the library's answer as the truth, which made it report six of twelve
/// decoding. Two of those six — the elf ghosts — have an index entry of `length 0` and
/// were returning whatever body was decoded immediately before them, at that body's
/// exact dimensions. Four of twelve have art on a stock client, not six.
/// </summary>
private static void SectionPlayers()
{
@@ -582,14 +622,42 @@ namespace Server.Custom
}
int direction = Config.Get("Bridge.AssetProbePlayerDirection", 0);
int decoded = 0, absent = 0;
int real = 0, absent = 0, wrong = 0;
string verdataPath = Files.GetFilePath("verdata.mul");
long verdataLength = BridgeAssetValidator.MulLength(verdataPath);
foreach (var pb in _playerBodies)
{
Checkpoint("players", pb.Body);
int translated = pb.Body;
int fileType = BodyConverter.Convert(ref translated);
int fileType, at;
string reason;
bool resolved = BridgeAssetValidator.ResolveAnimation(
pb.Body, 0, direction, out fileType, out at, out reason);
// What the validator says BEFORE the library is asked. This is the whole point of
// the section: phase 0 reported "6 of 12 decode" from the library's answer alone,
// and two of those six were the previous body's picture.
var verdict = BridgeAssetValidator.Verdict.Absent;
if (resolved)
{
string dataPath = BridgeAssetValidator.AnimDataPath(fileType);
var index = BridgeAssetValidator.OpenAnimIndex(fileType);
long length = BridgeAssetValidator.MulLength(dataPath);
using (var reader = new BridgeAssetValidator.RecordReader(dataPath, verdataPath))
{
verdict = BridgeAssetValidator.CheckEntry(index, at, length, verdataLength, out reason);
if (verdict == BridgeAssetValidator.Verdict.Ok
&& !reader.AnimationSane(index, at, 1, out reason))
{
verdict = BridgeAssetValidator.Verdict.Refused;
}
}
}
string outcome;
@@ -597,17 +665,32 @@ namespace Server.Custom
{
int hue = 0;
var frames = Animations.GetAnimation(pb.Body, 0, direction, ref hue, false, true);
bool gotBitmap = frames != null && frames.Length > 0
&& frames[0] != null && frames[0].Bitmap != null;
string size = null;
if (frames != null && frames.Length > 0 && frames[0] != null && frames[0].Bitmap != null)
if (gotBitmap)
{
var bmp = frames[0].Bitmap;
outcome = "decoded " + bmp.Width + "x" + bmp.Height;
size = bmp.Width + "x" + bmp.Height;
bmp.Dispose();
decoded++;
}
if (verdict == BridgeAssetValidator.Verdict.Ok && gotBitmap)
{
outcome = "art, " + size;
real++;
}
else if (gotBitmap)
{
// The elf ghosts land here on a stock client: index entry `length 0`, and
// a bitmap the exact size of whatever was decoded last.
outcome = "WRONG PICTURE " + size + " — " + reason;
wrong++;
}
else
{
outcome = "no art on the legacy path (UOP-only — phase 4)";
outcome = "no art on the legacy path (UOP-only — phase 4): " + reason;
absent++;
}
}
@@ -617,11 +700,12 @@ namespace Server.Custom
}
Say(string.Format(" {0,-10} {1,-14} body {2,-5} fileType {3,-3} {4}",
pb.Race, pb.Slot, pb.Body, fileType, outcome));
pb.Race, pb.Slot, pb.Body, resolved ? fileType.ToString() : "-", outcome));
}
Say("");
Say(string.Format(" {0} decoded, {1} absent, of {2}", decoded, absent, _playerBodies.Count));
Say(string.Format(" {0} with art, {1} absent, {2} WRONG PICTURES, of {3}",
real, absent, wrong, _playerBodies.Count));
Say("");
}
@@ -978,369 +1062,4 @@ namespace Server.Custom
to.SendMessage(text);
}
}
/// <summary>
/// **Validate before calling** — the response the org lead chose for §4.2's residual risk,
/// prototyped here so phase 1 adopts it with measurements rather than on faith.
///
/// The principle: `Ultima`'s decoders take their bounds from the file they are reading, so
/// the extractor must decide whether a record is worth handing over *before* handing it
/// over. Every check below is against the index entry and the record header — cheap, and
/// enough to turn an uncatchable corrupted-state exception into a skipped asset.
///
/// It cannot be complete and does not claim to be. It closes the shapes that reading the
/// source showed are reachable; the probe's REFUSED-BUT-DECODED count is what says whether
/// the boundary is drawn in the right place.
///
/// Promoted into the overlay in phase 1.
/// </summary>
public static class BridgeAssetValidator
{
public enum Verdict
{
/// <summary>Nothing at this id, and the index says so honestly.</summary>
Absent,
/// <summary>The entry is self-consistent and inside its file.</summary>
Ok,
/// <summary>The entry claims something the file cannot support. Do not decode it.</summary>
Refused
}
/// <summary>Land tiles decode a fixed 44×44 diamond: 2 × (2+4+…+44) ushorts.</summary>
public const int LandRecordBytes = 2024;
/// <summary>
/// A ceiling on decoded art dimensions. `LoadStatic` allocates
/// <c>new Bitmap(width, height)</c> straight from two ushorts in the record, so a
/// corrupt header asks for up to 65535×65535 — an 8 GB allocation, from a file. Real
/// art is a couple of hundred pixels at most.
/// </summary>
public const int MaxArtDimension = 1024;
/// <summary>
/// Builds our own index over the same files, with the same constructor arguments
/// <c>Art</c> uses — including <c>hasExtra: false</c>, which is the whole reason the
/// art path is safe where the gump path is not (§4.1).
/// </summary>
public static FileIndex OpenArtIndex()
{
if (ArtDataPath() == null)
return null;
return new FileIndex("Artidx.mul", "Art.mul", "artLegacyMUL.uop", 0x10000, 4, ".tga", 0x13FDC, false);
}
/// <summary>
/// The file an art index entry's <c>lookup</c> is an offset **into** — which is not
/// <c>art.mul</c> on any current client.
///
/// This cost a whole probe run to learn and it is the single most important thing
/// phase 1 must not get wrong. <c>FileIndex</c>'s UOP constructor ends with a bare
/// <c>MulPath = uopPath</c>: **when <c>artLegacyMUL.uop</c> exists it wins outright**,
/// and <c>art.mul</c> / <c>artidx.mul</c> are never opened at all. A validator that
/// bounds offsets against <c>art.mul</c> while the index holds UOP offsets is not
/// merely approximate, it is nonsense — the first run of this probe refused 34,299
/// perfectly good statics for "declaring 10533x2085" because it was reading UOP
/// offsets into the wrong file.
///
/// So the resolution order here mirrors <c>FileIndex</c>'s exactly, and anything that
/// needs the bytes behind an entry must ask this rather than assume.
/// </summary>
public static string ArtDataPath()
{
var uop = Files.GetFilePath("artlegacymul.uop");
if (uop != null)
return uop;
return Files.GetFilePath("art.mul");
}
public static long MulLength(string path)
{
if (path == null)
return 0;
try
{
return new FileInfo(path).Length;
}
catch
{
return 0;
}
}
/// <summary>
/// Judges one index entry.
///
/// The check <c>FileIndex.Seek</c> is missing is the last one: it tests
/// <c>Stream.Length &lt; e.lookup</c> — that the record *starts* inside the file — and
/// never that it *ends* inside it. A record that begins two bytes before EOF and
/// declares a length of 4,000 passes, and <c>stream.Read</c> then returns a short count
/// that the decoders discard, leaving the previous asset's bytes in the shared buffer.
/// </summary>
public static Verdict CheckEntry(FileIndex index, int at, long mulLength, long verdataLength, out string reason)
{
reason = null;
if (index == null || index.Index == null || at < 0 || at >= index.Index.Length)
{
reason = "index " + at + " out of range";
return Verdict.Absent;
}
Entry3D e = index.Index[at];
if (e.lookup < 0)
{
reason = "lookup " + e.lookup;
return Verdict.Absent;
}
bool patched = (e.length & (1 << 31)) != 0;
int length = e.length & 0x7FFFFFFF;
if (!patched && e.length < 0)
{
reason = "length " + e.length;
return Verdict.Absent;
}
if (length == 0)
{
reason = "lookup " + e.lookup + ", length 0";
return Verdict.Absent;
}
long ceiling = patched ? verdataLength : mulLength;
if (ceiling <= 0)
{
reason = (patched ? "verdata.mul" : "the art data file") + " has no length";
return Verdict.Refused;
}
if (e.lookup >= ceiling)
{
reason = "lookup " + e.lookup + " past the end of "
+ (patched ? "verdata.mul" : "the mul") + " (" + ceiling + ")";
return Verdict.Refused;
}
// The missing check. A short read is silent, and its consequence is the PREVIOUS
// asset's picture served under this id.
if (e.lookup + (long)length > ceiling)
{
reason = "record runs " + (e.lookup + (long)length - ceiling) + " bytes past the end of "
+ (patched ? "verdata.mul" : "the mul");
return Verdict.Refused;
}
return Verdict.Ok;
}
/// <summary>
/// `LoadLand` reads 2,024 bytes regardless of the declared length, so a shorter record
/// reads past the end of a buffer sized from that length.
/// </summary>
public static bool LandLengthSane(FileIndex index, int at, out string reason)
{
reason = null;
if (index == null || index.Index == null || at < 0 || at >= index.Index.Length)
return true;
int length = index.Index[at].length & 0x7FFFFFFF;
if (length > 0 && length < LandRecordBytes)
{
reason = "land record is " + length + " bytes; LoadLand always reads " + LandRecordBytes;
return false;
}
return true;
}
/// <summary>
/// Walks a static record's own row table the way <c>LoadStatic</c> will, and refuses
/// it if that walk would read outside the record.
///
/// This is the check with teeth. <c>LoadStatic</c>'s inner loop guards the write into
/// the bitmap (<c>xOffset &gt; delta</c>, <c>xOffset + xRun &gt; delta</c>) and does
/// nothing at all about the read cursor, which advances until it happens to find a
/// zero pair — potentially far outside a pinned array. Simulating the same walk with
/// a bound is the cheapest way to know whether handing the id over is safe.
/// </summary>
public static bool StaticRecordSane(byte[] record, int length, out string reason)
{
reason = null;
if (length < 8)
{
reason = "record is " + length + " bytes; a static header needs 8";
return false;
}
int words = length / 2;
int width = ReadUInt16(record, 4);
int height = ReadUInt16(record, 6);
// LoadStatic returns null for these rather than misbehaving, so it is not a refusal.
if (width <= 0 || height <= 0)
return true;
if (width > MaxArtDimension || height > MaxArtDimension)
{
reason = "declares " + width + "x" + height + ", past the " + MaxArtDimension + "px ceiling";
return false;
}
// The row-lookup table: height ushorts starting at word 4.
if (4 + height > words)
{
reason = "row table (" + height + " entries) does not fit in a " + length + "-byte record";
return false;
}
int start = height + 4;
for (int y = 0; y < height; y++)
{
int cursor = start + ReadUInt16(record, (4 + y) * 2);
while (true)
{
// Two ushorts for the run header, and they must both be inside the record.
if (cursor < 0 || cursor + 1 >= words)
{
reason = "row " + y + " reads at word " + cursor + ", past the record's " + words;
return false;
}
int xOffset = ReadUInt16(record, cursor * 2);
int xRun = ReadUInt16(record, (cursor + 1) * 2);
cursor += 2;
if (xOffset + xRun == 0)
break;
// LoadStatic stops the row here, so the read cursor stops with it.
if (xOffset > width || xOffset + xRun > width)
break;
if (cursor + xRun > words)
{
reason = "row " + y + " declares a " + xRun + "-pixel run running past the record";
return false;
}
cursor += xRun;
}
}
return true;
}
private static int ReadUInt16(byte[] b, int at)
{
return b[at] | (b[at + 1] << 8);
}
/// <summary>
/// Reads a record's actual bytes so <see cref="StaticRecordSane"/> can walk it.
///
/// Holds its own handles rather than borrowing the library's, because <c>FileIndex</c>
/// hands out the stream it decodes from and moving that stream's position underneath
/// the decoder would be its own bug. Opened <c>FileShare.ReadWrite</c> to match how
/// <c>FileIndex</c> opens the same files.
/// </summary>
public sealed class RecordReader : IDisposable
{
private readonly FileStream _mul;
private readonly FileStream _verdata;
private byte[] _scratch = new byte[64 * 1024];
public RecordReader(string mulPath, string verdataPath)
{
_mul = Open(mulPath);
_verdata = Open(verdataPath);
}
private static FileStream Open(string path)
{
if (path == null || !File.Exists(path))
return null;
try
{
return new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
}
catch
{
return null;
}
}
/// <summary>
/// True when the record at <paramref name="at"/> is safe to hand to
/// <c>Art.GetStatic</c>. A record that cannot be read at all is reported sane —
/// <see cref="CheckEntry"/> has already judged the entry, and this must not
/// invent a second reason to refuse.
/// </summary>
public bool StaticSane(FileIndex index, int at, out string reason)
{
reason = null;
if (index == null || index.Index == null || at < 0 || at >= index.Index.Length)
return true;
Entry3D e = index.Index[at];
bool patched = (e.length & (1 << 31)) != 0;
int length = e.length & 0x7FFFFFFF;
var stream = patched ? _verdata : _mul;
if (stream == null || length <= 0 || e.lookup < 0)
return true;
if (_scratch.Length < length)
_scratch = new byte[length];
int read;
try
{
stream.Seek(e.lookup, SeekOrigin.Begin);
read = stream.Read(_scratch, 0, length);
}
catch (Exception ex)
{
reason = "cannot read the record: " + ex.GetType().Name;
return false;
}
// The short read the decoders discard. Refusing here is the whole point: the
// library would decode whatever the shared buffer happened to hold.
if (read < length)
{
reason = "short read — " + read + " of " + length + " bytes available";
return false;
}
return StaticRecordSane(_scratch, length, out reason);
}
public void Dispose()
{
if (_mul != null)
_mul.Dispose();
if (_verdata != null)
_verdata.Dispose();
}
}
}
}