RunicGateway/servuo-plugins
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Compare commits

base: RunicGateway:c71712c734168cdc2730eede74927448ae2fc12b
Branches Tags
RunicGateway:main RunicGateway:edge
RunicGateway:v1.3.0 RunicGateway:v1.2.0 RunicGateway:v1.1.0 RunicGateway:v1.0.0 RunicGateway:v0.2.0 RunicGateway:v0.1.1
..
compare: RunicGateway:v1.2.0
Branches Tags
RunicGateway:main RunicGateway:edge
RunicGateway:v1.3.0 RunicGateway:v1.2.0 RunicGateway:v1.1.0 RunicGateway:v1.0.0 RunicGateway:v0.2.0 RunicGateway:v0.1.1

1 Commits
c71712c734 ... v1.2.0

Author SHA1 Message Date
Colby Whitlock
f23a08d449 Merge pull request 'feat(bridge): protocol 7 — the Event System's shard half (Phase 16b cutover, 1 of 6)' (#26) from edge into main
Some checks failed
Release overlay / release (push) Failing after 11m52s
Details 
Reviewed-on: #26
Reviewed-by: Colby Whitlock <whitlocktech@gmail.com>
 2026-09-09 19:53:39 +00:00
14 changed files with 3 additions and 6010 deletions
Expand all files Collapse all files

5
overlay.toml
View File

@@ -23,9 +23,8 @@
# manual duty: when the protocol changes, bump it here in the same PR that
# changes the emitters, exactly as link bumps PROTOCOL_VERSION.
#
# Current: 8 — see docs/link/v8.md (the Asset Bridge: client assets over the loopback link
# instead of a converter on somebody's desktop).
protocol = 8
# Current: 6 — see docs/link/v6.md (idempotency keys on inbound commands, champ.boss.killed).
protocol = 7
# ── ServUO compatibility ─────────────────────────────────────────────────────
#

45
overlay/Config/Bridge.cfg
View File

@@ -295,51 +295,6 @@ EventsMaxGrantStack=1000
# would land at a moment nobody chose. Set to 0 to allow a save at any time.
EventsMinSaveIntervalSec=300
# The asset plane (docs/link/v8.md, protocol 8). Its own switch, deliberately: turning
# this on is consenting to the website reading this host's UO CLIENT FILES -- art,
# animations, the string table -- over the link. Nothing on this plane writes anything.
AssetsEnabled=true
# The largest reply the asset plane will build, in encoded bytes. Not an item count:
# the ceiling it lives inside is the sidecar's 1 MiB inbound line cap, and base64 adds
# 33% to every payload. Clamped to [64 KiB, 512 KiB] -- half the wire cap, so that a
# single oversized item (always admitted, or its family could never make progress)
# still fits.
AssetBatchBytes=524288
# How many ServUO class names one `assets.bodies` request may carry (phase 3). The only
# bound on this plane counted in items rather than bytes, because what it bounds is not
# reply size -- it is constructing and deleting that many real mobiles ON THE CORE
# THREAD, between two ticks of the world. A larger request is refused, never truncated.
# Clamped to [1, 500].
AssetBodyBatch=100
# How many keys one `assets.fetch` request may name. The byte budget above still decides
# where a page is cut; this only bounds how large a request the shard will parse at all.
# Clamped to [1, 10000].
AssetFetchKeys=2000
# The wall-clock budget for one catalogue page, in milliseconds. The catalogue's manifest
# rows are ~90 bytes so the byte budget never stops it -- but building them means
# decoding hundreds of animations, and the sidecar waits 10 s for a reply. Kept well
# under that, because the page still has to be serialised and written afterwards.
# Clamped to [250, 5000].
AssetScanMs=3000
# Which direction the catalogue renders. NOT part of the asset key: five directions
# would five-fold every count in the working set to express a choice nobody varies.
#
# The split was found by RENDERING all five, not from a table. 0 is head-on, facing the
# viewer -- what a character portrait wants, and the least legible view there is of a
# four-legged creature (a wolf seen from the front is a dark blob). 1 is the front
# three-quarter, where the same wolf is unmistakably a wolf.
#
# Which bodies count as player bodies is asked of the shard (every registered race's
# male/female/ghost ids), never hardcoded. Clamped to [0, 4]: 5-7 are the client
# mirroring 1-3 through a decode branch this overlay has not verified.
AssetPlayerDirection=0
AssetCreatureDirection=1
# The test scaffolding in tools/scaffolding/ reads its own flags from this file
# (SeedOnStart, CensusOnStart, ProbeOnStart). They are absent here on purpose:
# Config.Get returns the default of false when a key is missing, so a deployed

751
overlay/Scripts/Custom/Bridge/BridgeAssetValidator.cs
View File

@@ -1,751 +0,0 @@
using System;
using System.IO;
using Ultima;
namespace Server.Custom.Bridge
{
/// <summary>
/// **Validate before calling** (docs/link/v8.md §4.5) — the boundary between this protocol
/// and ServUO's vendored <c>Ultima</c> decoders. Phase 0 prototyped it in
/// <c>tools/scaffolding/BridgeAssetProbe.cs</c> and measured it both ways; phase 1 promoted
/// it here, into the overlay, and extended it to animations.
///
/// The principle: `Ultima`'s decoders take their bounds from the file they are reading, so
/// the extractor must decide whether a record is worth handing over *before* handing it
/// over. Every check below is against the index entry and the record header — cheap, and
/// enough to turn an uncatchable corrupted-state exception into a skipped asset.
///
/// **The failure this exists for is a wrong picture, not a crash.** `LoadStatic`,
/// `LoadLand` and `GetAnimation` all decode out of a shared <c>m_StreamBuffer</c> that is
/// reused, only ever grown, and filled by a <c>stream.Read</c> whose return value is
/// discarded. A record that is short, absent or out of bounds therefore renders **whatever
/// the previously-decoded asset left behind**, reports success, and is undetectable by
/// anything downstream. On the stock client on the machine phase 0 ran on that is 22,102
/// ids whose index entry reads <c>lookup 0, length 0</c>.
///
/// It cannot be complete and does not claim to be. It closes the shapes that reading the
/// source showed are reachable. What says the boundary is in the right place is the second
/// measurement rather than the first: against a client patched 21 ways it refused all eight
/// record-level defects, and against the **stock** client it refused **nothing** across
/// 49,151 statics and 16,384 land tiles. A checker that refuses real art would be worse
/// than no checker.
/// </summary>
public static class BridgeAssetValidator
{
public enum Verdict
{
/// <summary>Nothing at this id, and the index says so honestly.</summary>
Absent,
/// <summary>The entry is self-consistent and inside its file.</summary>
Ok,
/// <summary>The entry claims something the file cannot support. Do not decode it.</summary>
Refused
}
/// <summary>Land tiles decode a fixed 44×44 diamond: 2 × (2+4+…+44) ushorts.</summary>
public const int LandRecordBytes = 2024;
/// <summary>
/// A ceiling on decoded art dimensions. `LoadStatic` allocates
/// <c>new Bitmap(width, height)</c> straight from two ushorts in the record, so a
/// corrupt header asks for up to 65535×65535 — an 8 GB allocation, from a file. Real
/// art is a couple of hundred pixels at most.
/// </summary>
public const int MaxArtDimension = 1024;
/// <summary>
/// Builds our own index over the same files, with the same constructor arguments
/// <c>Art</c> uses — including <c>hasExtra: false</c>, which is the whole reason the
/// art path is safe where the gump path is not (§4.1).
/// </summary>
public static FileIndex OpenArtIndex()
{
if (ArtDataPath() == null)
return null;
return new FileIndex("Artidx.mul", "Art.mul", "artLegacyMUL.uop", 0x10000, 4, ".tga", 0x13FDC, false);
}
/// <summary>
/// The file an art index entry's <c>lookup</c> is an offset **into** — which is not
/// <c>art.mul</c> on any current client.
///
/// This cost a whole probe run to learn and it is the single most important thing
/// phase 1 must not get wrong. <c>FileIndex</c>'s UOP constructor ends with a bare
/// <c>MulPath = uopPath</c>: **when <c>artLegacyMUL.uop</c> exists it wins outright**,
/// and <c>art.mul</c> / <c>artidx.mul</c> are never opened at all. A validator that
/// bounds offsets against <c>art.mul</c> while the index holds UOP offsets is not
/// merely approximate, it is nonsense — the first run of this probe refused 34,299
/// perfectly good statics for "declaring 10533x2085" because it was reading UOP
/// offsets into the wrong file.
///
/// So the resolution order here mirrors <c>FileIndex</c>'s exactly, and anything that
/// needs the bytes behind an entry must ask this rather than assume.
/// </summary>
public static string ArtDataPath()
{
var uop = Files.GetFilePath("artlegacymul.uop");
if (uop != null)
return uop;
return Files.GetFilePath("art.mul");
}
public static long MulLength(string path)
{
if (path == null)
return 0;
try
{
return new FileInfo(path).Length;
}
catch
{
return 0;
}
}
/// <summary>
/// Judges one index entry.
///
/// The check <c>FileIndex.Seek</c> is missing is the last one: it tests
/// <c>Stream.Length &lt; e.lookup</c> — that the record *starts* inside the file — and
/// never that it *ends* inside it. A record that begins two bytes before EOF and
/// declares a length of 4,000 passes, and <c>stream.Read</c> then returns a short count
/// that the decoders discard, leaving the previous asset's bytes in the shared buffer.
/// </summary>
public static Verdict CheckEntry(FileIndex index, int at, long mulLength, long verdataLength, out string reason)
{
reason = null;
if (index == null || index.Index == null || at < 0 || at >= index.Index.Length)
{
reason = "index " + at + " out of range";
return Verdict.Absent;
}
Entry3D e = index.Index[at];
if (e.lookup < 0)
{
reason = "lookup " + e.lookup;
return Verdict.Absent;
}
bool patched = (e.length & (1 << 31)) != 0;
int length = e.length & 0x7FFFFFFF;
if (!patched && e.length < 0)
{
reason = "length " + e.length;
return Verdict.Absent;
}
if (length == 0)
{
reason = "lookup " + e.lookup + ", length 0";
return Verdict.Absent;
}
long ceiling = patched ? verdataLength : mulLength;
if (ceiling <= 0)
{
reason = (patched ? "verdata.mul" : "the art data file") + " has no length";
return Verdict.Refused;
}
if (e.lookup >= ceiling)
{
reason = "lookup " + e.lookup + " past the end of "
+ (patched ? "verdata.mul" : "the mul") + " (" + ceiling + ")";
return Verdict.Refused;
}
// The missing check. A short read is silent, and its consequence is the PREVIOUS
// asset's picture served under this id.
if (e.lookup + (long)length > ceiling)
{
reason = "record runs " + (e.lookup + (long)length - ceiling) + " bytes past the end of "
+ (patched ? "verdata.mul" : "the mul");
return Verdict.Refused;
}
return Verdict.Ok;
}
/// <summary>
/// `LoadLand` reads 2,024 bytes regardless of the declared length, so a shorter record
/// reads past the end of a buffer sized from that length.
/// </summary>
public static bool LandLengthSane(FileIndex index, int at, out string reason)
{
reason = null;
if (index == null || index.Index == null || at < 0 || at >= index.Index.Length)
return true;
int length = index.Index[at].length & 0x7FFFFFFF;
if (length > 0 && length < LandRecordBytes)
{
reason = "land record is " + length + " bytes; LoadLand always reads " + LandRecordBytes;
return false;
}
return true;
}
/// <summary>
/// Walks a static record's own row table the way <c>LoadStatic</c> will, and refuses
/// it if that walk would read outside the record.
///
/// This is the check with teeth. <c>LoadStatic</c>'s inner loop guards the write into
/// the bitmap (<c>xOffset &gt; delta</c>, <c>xOffset + xRun &gt; delta</c>) and does
/// nothing at all about the read cursor, which advances until it happens to find a
/// zero pair — potentially far outside a pinned array. Simulating the same walk with
/// a bound is the cheapest way to know whether handing the id over is safe.
/// </summary>
public static bool StaticRecordSane(byte[] record, int length, out string reason)
{
reason = null;
if (length < 8)
{
reason = "record is " + length + " bytes; a static header needs 8";
return false;
}
int words = length / 2;
int width = ReadUInt16(record, 4);
int height = ReadUInt16(record, 6);
// LoadStatic returns null for these rather than misbehaving, so it is not a refusal.
if (width <= 0 || height <= 0)
return true;
if (width > MaxArtDimension || height > MaxArtDimension)
{
reason = "declares " + width + "x" + height + ", past the " + MaxArtDimension + "px ceiling";
return false;
}
// The row-lookup table: height ushorts starting at word 4.
if (4 + height > words)
{
reason = "row table (" + height + " entries) does not fit in a " + length + "-byte record";
return false;
}
int start = height + 4;
for (int y = 0; y < height; y++)
{
int cursor = start + ReadUInt16(record, (4 + y) * 2);
while (true)
{
// Two ushorts for the run header, and they must both be inside the record.
if (cursor < 0 || cursor + 1 >= words)
{
reason = "row " + y + " reads at word " + cursor + ", past the record's " + words;
return false;
}
int xOffset = ReadUInt16(record, cursor * 2);
int xRun = ReadUInt16(record, (cursor + 1) * 2);
cursor += 2;
if (xOffset + xRun == 0)
break;
// LoadStatic stops the row here, so the read cursor stops with it.
if (xOffset > width || xOffset + xRun > width)
break;
if (cursor + xRun > words)
{
reason = "row " + y + " declares a " + xRun + "-pixel run running past the record";
return false;
}
cursor += xRun;
}
}
return true;
}
// ── animations (phase 1) ─────────────────────────────────────────────────────────────
//
// Phase 0 measured the art path and left this half unbuilt, and then proved it was
// needed: the patched client's verdata entry for body 34 points past verdata.mul's own
// end, and the wolf still "decoded" — counted among the 1,144 successes while rendering
// something else entirely. `GetAnimation` has every weakness `LoadStatic` has and one
// more, because the buffer it decodes from is longer than the record it read.
/// <summary>The palette every animation record opens with: 0x100 ushorts.</summary>
public const int AnimPaletteBytes = 0x100 * 2;
/// <summary>
/// A ceiling on an animation's declared frame count. <c>GetAnimation</c> does
/// <c>new int[frameCount]</c> straight from four bytes in the file, before it has
/// looked at anything else. Real actions are tens of frames.
/// </summary>
public const int MaxAnimFrames = 1024;
/// <summary>The xor <c>Frame</c> applies to every run header before decoding it.</summary>
private const int DoubleXor = (0x200 << 22) | (0x200 << 12);
/// <summary>
/// The <c>anim*.mul</c> an animation index entry's <c>lookup</c> is an offset into.
///
/// Unlike art (§4.6) there is no UOP precedence to get wrong here, and that is not
/// luck: <c>Animations</c> constructs its five <c>FileIndex</c>es with the four-argument
/// constructor, which passes <c>uopFile: null</c>. It never reads
/// <c>AnimationFrame*.uop</c> at all — which is the same fact that leaves six of the
/// twelve player-character bodies undecodable until §4.3's reader lands in phase 4.
/// </summary>
public static string AnimDataPath(int fileType)
{
switch (fileType)
{
case 1: return Files.GetFilePath("anim.mul");
case 2: return Files.GetFilePath("anim2.mul");
case 3: return Files.GetFilePath("anim3.mul");
case 4: return Files.GetFilePath("anim4.mul");
case 5: return Files.GetFilePath("anim5.mul");
default: return null;
}
}
/// <summary>
/// Builds our own index over one anim file, with the same constructor arguments
/// <c>Animations</c> uses — the entry lengths especially, since they decide how far
/// into the file an index runs.
/// </summary>
public static FileIndex OpenAnimIndex(int fileType)
{
if (AnimDataPath(fileType) == null)
return null;
switch (fileType)
{
case 1: return new FileIndex("Anim.idx", "Anim.mul", 0x40000, 6);
case 2: return new FileIndex("Anim2.idx", "Anim2.mul", 0x10000, -1);
case 3: return new FileIndex("Anim3.idx", "Anim3.mul", 0x20000, -1);
case 4: return new FileIndex("Anim4.idx", "Anim4.mul", 0x20000, -1);
case 5: return new FileIndex("Anim5.idx", "Anim5.mul", 0x20000, -1);
default: return null;
}
}
/// <summary>
/// Where a body's animation actually lives: which anim file, and which index in it.
///
/// **This is the never-sweep-file-types rule, written as code** (§4.3). It asks
/// <c>BodyConverter.Convert</c> once, takes its answer, and if that answer leads
/// nowhere it reports nowhere. There is deliberately no loop here and no fallback,
/// because asking the *other* anim files for an index they do not own does not fail —
/// it returns 175 decodable action/direction combinations of **a giant spider** for
/// gargoyle 666, and misaligned colour fragments for the other two. Every one of those
/// reads reports success, and nothing downstream can tell them from art.
///
/// A false return with <paramref name="reason"/> set is the ordinary, expected answer
/// for a body this client has no art for — the caller reports absent, not an error.
/// </summary>
public static bool ResolveAnimation(
int body, int action, int direction, out int fileType, out int index, out string reason)
{
reason = null;
fileType = 0;
index = -1;
if (body <= 0 || action < 0)
{
reason = "body " + body + " action " + action + " is not addressable";
return false;
}
// Directions 5-7 are the client mirroring 1-3, and `Frame` decodes them through its
// flip branch — different pointer arithmetic, which nothing below has checked.
// §5.1 fixed this protocol at direction 0 or 1, so refusing the rest costs nothing
// and keeps the validator honest about what it has actually verified.
if (direction < 0 || direction > 4)
{
reason = "direction " + direction + " is mirrored; this protocol reads 0-4 only";
return false;
}
int translated = body;
int hue = 0;
try
{
// Exactly what GetAnimation(..., preserveHue: false, ...) does first.
Animations.Translate(ref translated, ref hue);
fileType = BodyConverter.Convert(ref translated);
}
catch (Exception e)
{
reason = "body.def/bodyconv.def lookup failed: " + e.GetType().Name;
return false;
}
if (AnimDataPath(fileType) == null)
{
// Gargoyle 666 lands here: Bodyconv.def maps it to anim5, and this client has no
// anim5. Absent is the correct answer and the ONLY safe one.
reason = "bodyconv sends body " + body + " to file type " + fileType
+ ", which this client does not have";
return false;
}
index = AnimIndexOf(translated, fileType) + (action * 5) + direction;
return true;
}
/// <summary>
/// <c>Animations.GetFileIndex</c>'s own arithmetic, which is private. The banding is
/// per file type and the boundaries differ between them, so this is transcribed rather
/// than generalised — an index that disagrees with the library's by one is a picture
/// of the wrong creature, validated.
/// </summary>
private static int AnimIndexOf(int body, int fileType)
{
switch (fileType)
{
case 2:
return body < 200 ? body * 110 : 22000 + ((body - 200) * 65);
case 3:
if (body < 300)
return body * 65;
return body < 400 ? 33000 + ((body - 300) * 110) : 35000 + ((body - 400) * 175);
case 5:
// "looks strange, though it works" — the library's own comment. Body 34 is
// excluded from the first band here and nowhere else.
if (body < 200 && body != 34)
return body * 110;
return body < 400 ? 22000 + ((body - 200) * 65) : 35000 + ((body - 400) * 175);
default: // 1 and 4 share their banding
if (body < 200)
return body * 110;
return body < 400 ? 22000 + ((body - 200) * 65) : 35000 + ((body - 400) * 175);
}
}
/// <summary>
/// Walks an animation record the way <c>GetAnimation</c> and <c>Frame</c> will, and
/// refuses it if that walk would read outside the record or write outside the bitmap.
///
/// Two things make this stricter than the static walk, and both come from the library:
///
/// <c>GetAnimation</c> decodes through <c>new MemoryStream(m_StreamBuffer, false)</c> —
/// the whole shared buffer, not the <c>length</c> bytes it just read into it. So a
/// truncated record does not hit end-of-stream and throw; the reader sails on into the
/// **previous** animation's bytes and returns a plausible frame. Bounding against
/// <paramref name="length"/> rather than against the buffer is the entire point.
///
/// And <c>Frame</c>'s run loop is a *write* through a <c>LockBits</c> pointer whose
/// origin comes from two signed shorts in the file (<c>xCenter</c>, <c>yCenter</c>),
/// with no bound of any kind. <c>LoadStatic</c> at least guards its writes; this does
/// not, so the destination of every run is checked against the bitmap it locked.
///
/// <paramref name="maxFrames"/> is how many frames the caller will actually decode —
/// 1 for the catalogue's thumbnail (<c>FirstFrame: true</c>), 0 for all of them.
/// Checking frames nobody decodes would invent refusals, which §4.5 costs more than
/// it saves.
/// </summary>
public static bool AnimationRecordSane(byte[] record, int length, int maxFrames, out string reason)
{
reason = null;
if (length < AnimPaletteBytes + 4)
{
reason = "record is " + length + " bytes; an animation needs "
+ (AnimPaletteBytes + 4) + " for its palette and frame count";
return false;
}
int start = AnimPaletteBytes;
int frameCount = ReadInt32(record, start);
if (frameCount <= 0)
{
reason = "declares " + frameCount + " frames";
return false;
}
if (frameCount > MaxAnimFrames)
{
reason = "declares " + frameCount + " frames, past the " + MaxAnimFrames + " ceiling";
return false;
}
// The lookup table is read in full whatever FirstFrame says, so it is bounded in full.
long tableEnd = (long)start + 4 + ((long)frameCount * 4);
if (tableEnd > length)
{
reason = "frame table (" + frameCount + " entries) does not fit in a "
+ length + "-byte record";
return false;
}
int check = maxFrames > 0 && maxFrames < frameCount ? maxFrames : frameCount;
for (int i = 0; i < check; i++)
{
int at = start + ReadInt32(record, start + 4 + (i * 4));
if (!FrameSane(record, length, at, i, out reason))
return false;
}
return true;
}
private static bool FrameSane(byte[] record, int length, int at, int frame, out string reason)
{
reason = null;
if (at < 0 || at + 8 > length)
{
reason = "frame " + frame + " starts at " + at + ", outside the "
+ length + "-byte record";
return false;
}
int xCenter = ReadInt16(record, at);
int yCenter = ReadInt16(record, at + 2);
int width = ReadUInt16(record, at + 4);
int height = ReadUInt16(record, at + 6);
// Frame's constructor returns before locking anything for these, so they are empty
// rather than dangerous — and an empty frame is a real thing in this format.
if (width == 0 || height == 0)
return true;
if (width > MaxArtDimension || height > MaxArtDimension)
{
reason = "frame " + frame + " declares " + width + "x" + height + ", past the "
+ MaxArtDimension + "px ceiling";
return false;
}
// Settings.PixelFormat is 16bpp and GDI+ pads each scanline to four bytes, so a row
// is `delta` ushorts wide and the locked region is height*delta of them. This is the
// same `bd.Stride >> 1` Frame computes.
int delta = (((width * 2) + 3) & ~3) >> 1;
long pixels = (long)height * delta;
long origin = (xCenter - 0x200) + ((long)((yCenter + height) - 0x200) * delta);
int cursor = at + 8;
while (true)
{
if (cursor + 4 > length)
{
reason = "frame " + frame
+ " runs off the end of the record looking for its terminator";
return false;
}
int header = ReadInt32(record, cursor);
cursor += 4;
if (header == 0x7FFF7FFF)
break;
header ^= DoubleXor;
long dy = (header >> 12) & 0x3FF;
long dx = (header >> 22) & 0x3FF;
int run = header & 0xFFF;
long first = origin + (dy * delta) + dx;
if (first < 0 || first + run > pixels)
{
reason = "frame " + frame + " writes pixels " + first + ".." + (first + run)
+ " outside its own " + pixels + "-pixel bitmap";
return false;
}
// One palette byte per pixel, read straight out of the record.
if (cursor + run > length)
{
reason = "frame " + frame + " declares a " + run
+ "-pixel run running past the record";
return false;
}
cursor += run;
}
return true;
}
private static int ReadUInt16(byte[] b, int at)
{
return b[at] | (b[at + 1] << 8);
}
private static int ReadInt16(byte[] b, int at)
{
return (short)(b[at] | (b[at + 1] << 8));
}
private static int ReadInt32(byte[] b, int at)
{
return b[at] | (b[at + 1] << 8) | (b[at + 2] << 16) | (b[at + 3] << 24);
}
/// <summary>
/// Reads a record's actual bytes so <see cref="StaticRecordSane"/> or
/// <see cref="AnimationRecordSane"/> can walk it.
///
/// Holds its own handles rather than borrowing the library's, because <c>FileIndex</c>
/// hands out the stream it decodes from and moving that stream's position underneath
/// the decoder would be its own bug. Opened <c>FileShare.ReadWrite</c> to match how
/// <c>FileIndex</c> opens the same files.
///
/// One reader serves one data file, so an animation sweep wants one per file type,
/// built from <see cref="AnimDataPath"/>.
/// </summary>
public sealed class RecordReader : IDisposable
{
private readonly FileStream _mul;
private readonly FileStream _verdata;
private byte[] _scratch = new byte[64 * 1024];
public RecordReader(string mulPath, string verdataPath)
{
_mul = Open(mulPath);
_verdata = Open(verdataPath);
}
private static FileStream Open(string path)
{
if (path == null || !File.Exists(path))
return null;
try
{
return new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
}
catch
{
return null;
}
}
/// <summary>
/// True when the record at <paramref name="at"/> is safe to hand to
/// <c>Art.GetStatic</c>. A record that cannot be read at all is reported sane —
/// <see cref="CheckEntry"/> has already judged the entry, and this must not
/// invent a second reason to refuse.
/// </summary>
public bool StaticSane(FileIndex index, int at, out string reason)
{
int length = ReadRecord(index, at, out reason);
if (length < 0)
return true;
if (length == 0)
return false;
return StaticRecordSane(_scratch, length, out reason);
}
/// <summary>
/// True when the record at <paramref name="at"/> is safe to hand to
/// <c>Animations.GetAnimation</c>. <paramref name="maxFrames"/> is how many frames
/// the caller will decode — 1 for a <c>FirstFrame</c> call, 0 for all of them.
/// </summary>
public bool AnimationSane(FileIndex index, int at, int maxFrames, out string reason)
{
int length = ReadRecord(index, at, out reason);
if (length < 0)
return true;
if (length == 0)
return false;
return AnimationRecordSane(_scratch, length, maxFrames, out reason);
}
/// <summary>
/// Reads one record into <see cref="_scratch"/>. Returns its length, 0 for a
/// failure (with <paramref name="reason"/> set), or -1 when there is nothing to
/// read at all — <see cref="CheckEntry"/> has already judged the entry, and this
/// must not invent a second reason to refuse.
/// </summary>
private int ReadRecord(FileIndex index, int at, out string reason)
{
reason = null;
if (index == null || index.Index == null || at < 0 || at >= index.Index.Length)
return -1;
Entry3D e = index.Index[at];
bool patched = (e.length & (1 << 31)) != 0;
int length = e.length & 0x7FFFFFFF;
var stream = patched ? _verdata : _mul;
if (stream == null || length <= 0 || e.lookup < 0)
return -1;
if (_scratch.Length < length)
_scratch = new byte[length];
int read;
try
{
stream.Seek(e.lookup, SeekOrigin.Begin);
read = stream.Read(_scratch, 0, length);
}
catch (Exception ex)
{
reason = "cannot read the record: " + ex.GetType().Name;
return 0;
}
// The short read the decoders discard. Refusing here is the whole point: the
// library would decode whatever the shared buffer happened to hold.
if (read < length)
{
reason = "short read — " + read + " of " + length + " bytes available";
return 0;
}
return length;
}
public void Dispose()
{
if (_mul != null)
_mul.Dispose();
if (_verdata != null)
_verdata.Dispose();
}
}
}
}

775
overlay/Scripts/Custom/Bridge/BridgeAssets.cs
View File

@@ -1,775 +0,0 @@
using System;
using System.Collections.Generic;
using System.Globalization;
using System.IO;
using System.Security.Cryptography;
using System.Text;
using System.Threading;
using Ultima;
namespace Server.Custom.Bridge
{
/// <summary>
/// **The Asset Bridge's transport** (docs/link/v8.md §3, §6, §7 — protocol 8, phase 1).
///
/// Everything else on this link answers on the Core thread, reads live world state, and
/// replies in microseconds. The asset plane cannot: it reads hundreds of megabytes of the
/// operator's client files and decodes pictures out of them, and doing either on the Core
/// thread would stop the world for every player on the shard. So this class is the one
/// genuinely new shape in protocol 8 — a handler that accepts on the Core thread, hands
/// the work to a **dedicated asset worker**, and returns immediately.
///
/// Three rules hold it together, and each of them is answering a specific way this could
/// go wrong.
///
/// **1. Replies, never events.** Every asset frame carries the caller's `reqId`, so
/// `rpc.rs`'s `try_route` consumes it before `app.rs` can persist it to SQLite and
/// broadcast it to every WebSocket subscriber. An asset stream on the event path would
/// grow the sidecar's store without bound and fan megabytes at every connected client,
/// forever. Nothing here emits an unsolicited frame — if a request has no `reqId` it is
/// refused rather than answered.
///
/// **2. One request outstanding, always.** <see cref="BridgeLink.Emit"/>'s queue is bounded
/// drop-oldest in **lines, not bytes** — a design that is right for live events and
/// dangerous for bulk transfer, because 10,000 queued 200 KB replies is 2 GB of shard
/// memory. The bound that actually holds is flow control, not a bigger queue: this plane
/// has **one slot**, and a second asset request arriving while one is in flight is
/// answered `bridge.busy` (which the sidecar already maps to 425) rather than queued.
/// Queue depth therefore stays at approximately one by construction. A dropped or lost
/// reply just times out and is re-requested, which is safe because reading a client file
/// is idempotent and touches no world state.
///
/// Note what that costs, deliberately: a status poll shares the slot with a batch, so
/// polling during a long import gets 425 until the batch lands. That is honest — this
/// plane really is doing one thing at a time — and the admin surface (phase 8) is where a
/// separate status lane would have to argue for itself.
///
/// **3. Byte budgets, not counts.** Batches are cut by encoded size
/// (<see cref="BridgeConfig.AssetBatchBytes"/>, 512 KB), not by item count, because the
/// ceilings this has to live inside are byte ceilings: the sidecar refuses an inbound line
/// over 1 MiB, and base64 costs 33% on top of whatever the payload measures.
/// <see cref="PageBuilder"/> is that budget, and every asset family shares it so the
/// envelope cannot drift apart between them.
///
/// **Emitting from off the Core thread is safe here, and it is worth saying why.**
/// <c>BridgeLink.Emit</c> enqueues onto a <c>ConcurrentQueue</c> and never touches the
/// socket, so the enqueue itself is fine. The subtle part is
/// <c>BridgeIdempotency.Observe</c>, which <c>Emit</c> calls while a keyed command is in
/// flight: it captures a line only when that line's correlation field **exactly equals**
/// the open command's correlation value, and correlation values come from one monotonic
/// counter in the sidecar. An asset reply therefore cannot be mistaken for a keyed
/// command's reply, whatever the interleaving.
/// </summary>
public static class BridgeAssets
{
/// <summary>
/// What version of *our derivation* produced these bytes (§7).
///
/// The source gate hashes the operator's client files, which answers "did the inputs
/// change". It cannot answer "did the way we read them change" — and that is the case
/// that bites, because a corrected frame offset or a fixed hue application changes
/// every derived byte while every source file stays identical. So this is folded into
/// stage 1 alongside the hashes, and bumping it makes the whole working set drift,
/// which is the intended and correct blast radius.
///
/// Bump it whenever extraction changes what it produces from unchanged input. It is
/// the same rule <c>spawnAtlasSource.js</c>'s `PARSER_VERSION` follows, and it applies
/// here more rather than less: this pipeline derives far more from far less.
/// </summary>
public const int EXTRACTOR_VERSION = 1;
// ── the one slot (§3.2) ──────────────────────────────────────────────────────────────
private static readonly object _sync = new object();
private static Thread _worker;
private static readonly AutoResetEvent _wake = new AutoResetEvent(false);
private static Action _job;
private static string _inFlight;
private static DateTime _inFlightSince;
private static bool _running;
private static long _served, _busied, _failed;
// ── the hash cache (§6) ──────────────────────────────────────────────────────────────
private static readonly Dictionary<string, CachedHash> _hashes =
new Dictionary<string, CachedHash>(StringComparer.OrdinalIgnoreCase);
private static Thread _hasher;
private static volatile bool _hashing;
private sealed class CachedHash
{
public long Size;
public long MTime;
public string Sha256;
}
// ── imaging (§4.4) ───────────────────────────────────────────────────────────────────
private static bool _imagingChecked;
private static bool _imagingOk;
private static string _imagingReason;
public static void Initialize()
{
if (!BridgeConfig.Enabled)
return;
BridgeBoot.RegisterHandler("assets.sources", OnSources);
}
public static string Status()
{
int cached;
lock (_hashes)
{
cached = _hashes.Count;
}
lock (_sync)
{
return String.Format(
"assets(served={0} busied={1} failed={2} inFlight={3} hashing={4} cached={5})",
_served, _busied, _failed, _inFlight ?? "-", _hashing, cached);
}
}
// ── the request plane ────────────────────────────────────────────────────────────────
/// <summary>
/// Stage 1 of §6: what the shard's client files currently are. No pixels, no assets —
/// just the gate that lets the website decide whether anything needs importing at all,
/// because the normal case is a restart that changed nothing and it must cost nothing.
/// </summary>
private static void OnSources(Dictionary<string, object> o)
{
var reqId = BridgeJson.GetString(o, "reqId");
if (reqId == null)
{
// Rule 1. Without a correlation id this would land on the event path, be persisted
// to the sidecar's store and broadcast to every subscriber. Refuse instead.
Fail(null, "BAD_REQUEST", "assets.sources requires a reqId");
return;
}
if (!BridgeConfig.AssetsEnabled)
{
Fail(reqId, "DISABLED", "asset extraction is disabled on this shard");
return;
}
Accept(reqId, "assets.sources", () => ReplySources(reqId));
}
/// <summary>
/// Claims the single slot and hands the work to the worker, or answers `bridge.busy`.
/// Runs on the Core thread and does nothing expensive; <paramref name="job"/> runs on
/// the worker and must touch no world state.
/// </summary>
internal static void Accept(string reqId, string kind, Action job)
{
lock (_sync)
{
if (_inFlight != null)
{
_busied++;
Busy(reqId, kind);
return;
}
_inFlight = kind;
_inFlightSince = DateTime.UtcNow;
_job = job;
try
{
EnsureWorker();
}
catch (Exception e)
{
// The slot is claimed and there is now nothing that will ever free it. Give it
// back here or this plane answers `bridge.busy` for the life of the process.
_inFlight = null;
_job = null;
Console.WriteLine("[Bridge] cannot start the asset worker: {0}", e.Message);
Fail(reqId, "UNAVAILABLE", "the shard could not start its asset worker");
return;
}
}
_wake.Set();
}
private static void Busy(string reqId, string kind)
{
var held = (DateTime.UtcNow - _inFlightSince).TotalSeconds;
var sb = BridgeJson.Begin("bridge.busy");
sb.Str("reqId", reqId)
// `busyKind`, never a second `kind` — `Begin` has already written this frame's own,
// and a JSON object carrying two makes every parser take the last. Protocol 6 shipped
// that bug once and it made the sidecar answer 200 for a refusal.
.Str("busyKind", kind)
.Num("heldForSec", (long)held)
.Str("reason", "the asset plane serves one request at a time");
BridgeLink.Emit(sb.End());
}
/// <summary>
/// The asset plane's one refusal frame, shared by every family on it.
///
/// <paramref name="code"/> is what the sidecar maps to a status, and it exists because
/// the alternative it replaced — matching on the words in <paramref name="reason"/> —
/// makes an operator-facing sentence load-bearing. Rewording "disabled" would silently
/// turn a 403 into a 400. The codes are `DISABLED` (the operator switched this plane
/// off), `NOT_FOUND` (the shard has no such file), `UNREADABLE` (it has it and cannot
/// decode it), `UNAVAILABLE` (the shard cannot do this right now) and `BAD_REQUEST`
/// (the default, and the caller's fault).
/// </summary>
internal static void Fail(string reqId, string code, string reason)
{
var sb = BridgeJson.Begin("assets.error");
if (reqId != null)
sb.Str("reqId", reqId);
sb.Str("code", code)
.Str("reason", reason);
BridgeLink.Emit(sb.End());
}
// ── the worker ───────────────────────────────────────────────────────────────────────
/// <summary>
/// Started on first use rather than at boot, so a shard that never imports an asset
/// never carries the thread. Caller must hold <see cref="_sync"/>.
/// </summary>
private static void EnsureWorker()
{
if (_worker != null)
return;
_running = true;
_worker = new Thread(WorkLoop)
{
Name = "BridgeAssets",
IsBackground = true
};
_worker.Start();
}
private static void WorkLoop()
{
while (_running)
{
_wake.WaitOne(1000);
Action job;
lock (_sync)
{
job = _job;
_job = null;
}
if (job == null)
continue;
try
{
job();
Interlocked.Increment(ref _served);
}
catch (Exception e)
{
// A handler that throws must still free the slot, or this plane is wedged for
// the life of the process and every later request answers `bridge.busy`.
Interlocked.Increment(ref _failed);
Console.WriteLine("[Bridge] asset worker: {0}: {1}", e.GetType().Name, e.Message);
}
finally
{
lock (_sync)
{
_inFlight = null;
}
}
}
}
// ── assets.sources ───────────────────────────────────────────────────────────────────
/// <summary>
/// The client files whose bytes decide everything downstream.
///
/// Resolved through <c>Ultima.Files</c> rather than by joining a configured directory,
/// because that is what the decoders themselves do — a file this reports is a file
/// they would actually open.
/// </summary>
private static readonly string[] SourceFiles =
{
"cliloc.enu",
"artlegacymul.uop", "art.mul", "artidx.mul",
"anim.idx", "anim.mul",
"anim2.idx", "anim2.mul",
"anim3.idx", "anim3.mul",
"anim4.idx", "anim4.mul",
"anim5.idx", "anim5.mul",
"body.def", "bodyconv.def",
"hues.mul",
"verdata.mul"
};
private static void ReplySources(string reqId)
{
var sb = BridgeJson.Begin("assets.sources.ok");
sb.Str("reqId", reqId)
.Num("extractorVersion", EXTRACTOR_VERSION);
WriteImaging(sb);
// §4.6: whichever of art.mul / artLegacyMUL.uop `FileIndex` would actually open. An
// operator who added custom graphics to art.mul while the UOP is present is getting
// nothing, silently, and this is the only place that can tell them so.
string artData = BridgeAssetValidator.ArtDataPath();
sb.Str("artDataFile", artData == null ? null : Path.GetFileName(artData));
var page = new PageBuilder(sb, "files", BridgeConfig.AssetBatchBytes);
bool anyMissingHash = false;
for (int i = 0; i < SourceFiles.Length; i++)
{
string name = SourceFiles[i];
string path = ResolvePath(name);
if (path == null)
continue;
var item = new StringBuilder(256);
item.Append("{\"name\":");
BridgeJson.Text(item, name);
item.Append(",\"path\":");
BridgeJson.Text(item, path);
long size = 0, mtime = 0;
try
{
var info = new FileInfo(path);
size = info.Length;
mtime = ToUnixMs(info.LastWriteTimeUtc);
}
catch (Exception e)
{
item.Append(",\"unreadable\":");
BridgeJson.Text(item, e.GetType().Name);
}
item.Append(",\"size\":").Append(size.ToString(CultureInfo.InvariantCulture));
item.Append(",\"mtime\":").Append(mtime.ToString(CultureInfo.InvariantCulture));
string hash = CachedHashFor(path, size, mtime);
if (hash == null)
anyMissingHash = true;
item.Append(",\"sha256\":");
BridgeJson.Text(item, hash);
// The one diagnostic §4.6 asks for: art.mul is present, and unread.
if (artData != null
&& (name == "art.mul" || name == "artidx.mul")
&& !artData.EndsWith(".mul", StringComparison.OrdinalIgnoreCase))
{
item.Append(",\"shadowedBy\":");
BridgeJson.Text(item, Path.GetFileName(artData));
}
item.Append('}');
if (!page.TryAdd(item.ToString(), name))
break;
}
page.Close();
// §6's gate is (size, mtime) first and a content hash only when those differ, because
// anim.mul and art.mul are 195 MB and 148 MB and a full hash on every status poll
// would make the admin panel feel broken. It would also blow the sidecar's 10 s reply
// timeout outright on the first call. So a hash that is not cached is reported `null`
// and computed in the background: this reply is always fast, and the next poll — after
// `hashing` goes false — carries the answer.
if (anyMissingHash)
StartHashing();
sb.Bool("hashing", _hashing);
sb.Bool("complete", !anyMissingHash);
BridgeLink.Emit(sb.End());
}
private static string ResolvePath(string name)
{
try
{
return Files.GetFilePath(name);
}
catch
{
return null;
}
}
private static long ToUnixMs(DateTime utc)
{
return (long)(utc - new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc)).TotalMilliseconds;
}
// ── imaging (§4.4) ───────────────────────────────────────────────────────────────────
/// <summary>
/// Whether this host can turn a record into a picture at all.
///
/// ServUO targets net48, so a Linux shard runs it under Mono, where
/// <c>System.Drawing</c> is a thin layer over **libgdiplus** — and §4.2 put
/// <c>System.Drawing</c> in the *decode* path, not merely the encode: <c>Frame</c>
/// writes ARGB1555 through a <c>LockBits</c> pointer. Without that library a Linux
/// shard cannot read a sprite at all, while clilocs and the atlas are unaffected
/// because neither touches pixels.
///
/// It must never present as a stack trace or a 500. It is a named, actionable outcome
/// in the same family as the cliloc reader's `COMPRESSED`, and it is reported here —
/// on the *source gate*, the first call any import makes — so an operator learns it
/// while setting the shard up rather than from an empty bestiary weeks later.
/// </summary>
private static void WriteImaging(StringBuilder sb)
{
CheckImaging();
sb.Append(",\"imaging\":{\"ok\":").Append(_imagingOk ? "true" : "false");
if (!_imagingOk)
{
sb.Append(",\"code\":\"NO_IMAGING\",\"reason\":");
BridgeJson.Text(sb,
"This shard host cannot render images — Mono's System.Drawing needs libgdiplus. "
+ "Install it (apt-get install libgdiplus) and re-run the import. Cliloc and atlas "
+ "import are unaffected. (" + _imagingReason + ")");
}
sb.Append('}');
}
/// <summary>
/// Whether this host can produce a picture, for the families that produce pictures.
///
/// <see cref="WriteImaging"/> reports this on the source gate so an operator learns it
/// while setting the shard up. The catalogue needs the same answer as a *decision* —
/// it must refuse rather than throw a <c>DllNotFoundException</c> out of the middle of
/// a decode loop — so the check itself is shared and this is its one accessor.
/// </summary>
internal static bool ImagingOk(out string reason)
{
CheckImaging();
reason = _imagingReason;
return _imagingOk;
}
private static void CheckImaging()
{
if (_imagingChecked)
return;
_imagingChecked = true;
try
{
TouchImaging();
_imagingOk = true;
}
catch (Exception e)
{
// On a host with no libgdiplus this is a TypeInitializationException wrapping a
// DllNotFoundException, and it can surface as the method failing to JIT rather
// than as a throw from inside it — which is why the construction lives in its own
// method, so the failure is contained here instead of taking this class's
// static initialisation with it.
_imagingOk = false;
_imagingReason = e.GetType().Name + ": " + e.Message;
}
}
private static void TouchImaging()
{
using (var bmp = new System.Drawing.Bitmap(1, 1))
{
bmp.SetPixel(0, 0, System.Drawing.Color.Black);
}
}
// ── the hash cache (§6) ──────────────────────────────────────────────────────────────
private static string CachedHashFor(string path, long size, long mtime)
{
lock (_hashes)
{
CachedHash cached;
if (_hashes.TryGetValue(path, out cached)
&& cached.Size == size
&& cached.MTime == mtime)
{
return cached.Sha256;
}
}
return null;
}
/// <summary>
/// Rehashes whatever the cache is missing, on its own thread.
///
/// Deliberately **not** a job on the asset worker: hashing 343 MB takes seconds to
/// tens of seconds, and holding the single slot for that long would answer every
/// status poll `bridge.busy` for the whole pass — which is exactly the moment an
/// operator is watching the panel. It emits nothing and correlates with nothing; it
/// only fills the cache that the next `assets.sources` reads.
/// </summary>
private static void StartHashing()
{
lock (_sync)
{
if (_hashing)
return;
_hashing = true;
_hasher = new Thread(HashLoop)
{
Name = "BridgeAssetHash",
IsBackground = true
};
_hasher.Start();
}
}
private static void HashLoop()
{
try
{
for (int i = 0; i < SourceFiles.Length; i++)
{
string path = ResolvePath(SourceFiles[i]);
if (path == null)
continue;
long size, mtime;
try
{
var info = new FileInfo(path);
size = info.Length;
mtime = ToUnixMs(info.LastWriteTimeUtc);
}
catch
{
continue;
}
if (CachedHashFor(path, size, mtime) != null)
continue;
string hash = HashFile(path);
if (hash == null)
continue;
lock (_hashes)
{
_hashes[path] = new CachedHash { Size = size, MTime = mtime, Sha256 = hash };
}
}
}
catch (Exception e)
{
Console.WriteLine("[Bridge] asset hash pass: {0}: {1}", e.GetType().Name, e.Message);
}
finally
{
// Under _sync, matching StartHashing: cleared outside it, two passes could both
// pass the guard and hash the same 343 MB twice.
lock (_sync)
{
_hashing = false;
}
}
}
private static string HashFile(string path)
{
try
{
using (var sha = SHA256.Create())
using (var stream = new FileStream(
path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite, 1 << 20))
{
var buffer = new byte[1 << 20];
int read;
while ((read = stream.Read(buffer, 0, buffer.Length)) > 0)
sha.TransformBlock(buffer, 0, read, null, 0);
sha.TransformFinalBlock(buffer, 0, 0);
return Hex(sha.Hash);
}
}
catch (Exception e)
{
Console.WriteLine("[Bridge] cannot hash {0}: {1}", path, e.Message);
return null;
}
}
private static string Hex(byte[] bytes)
{
var sb = new StringBuilder(bytes.Length * 2);
for (int i = 0; i < bytes.Length; i++)
sb.Append(bytes[i].ToString("x2", CultureInfo.InvariantCulture));
return sb.ToString();
}
// ── the paging envelope (§3.3) ───────────────────────────────────────────────────────
/// <summary>
/// **One envelope for every asset family**, defined here in phase 1 so that clilocs
/// (phase 2), the body catalogue (3), statics and land (5), deep animation keys (6)
/// and the ServUO tree files (7) all page the same way. They are otherwise five
/// chances to invent five slightly different shapes, and the website would have to
/// learn each one.
///
/// The envelope a reply closes with:
///
/// <code>
/// "items": [ … ],
/// "more": true, // ask again with this cursor
/// "cursor": "s:4104", // opaque to everyone but the shard; absent when more:false
/// "cut": "budget" // budget | end | limit — WHY this page stopped
/// </code>
///
/// **The budget is bytes, and it is UTF-8 bytes.** Not item count, because the ceiling
/// this lives inside is the sidecar's inbound line cap; and not chars, because a
/// cliloc row is real text and a `StringBuilder`'s Length would undercount every
/// non-ASCII character in it.
///
/// `cut` exists because "the page is short" has three different meanings and the
/// website must not have to guess which: the source ran out (`end`), the byte budget
/// was spent (`budget`), or the family stopped at its own limit (`limit`). Only the
/// first means the import is finished.
///
/// **The first item is always admitted**, even if it alone exceeds the budget.
/// Otherwise an oversized item would make its family unable to make any progress at
/// all — it would be skipped for the budget on every page, forever. That is safe
/// precisely because the budget is set to half the sidecar's line cap
/// (<see cref="BridgeConfig.AssetBatchBytes"/>), so one such item still fits the wire.
/// </summary>
public sealed class PageBuilder
{
private readonly StringBuilder _sb;
private readonly int _budget;
private int _bytes;
private int _count;
private string _cursor;
private string _cut = "end";
/// <summary>
/// Room kept back for the fields the envelope must still be able to write after
/// the last item — `more`, `cursor`, `cut` and the closing brace. Without it a
/// page could fill the budget exactly and then overrun it closing itself.
/// </summary>
private const int Reserve = 256;
public PageBuilder(StringBuilder sb, string arrayName, int budget)
{
_sb = sb;
_budget = budget;
sb.Append(",\"").Append(arrayName).Append("\":[");
// The prefix is already written, and it counts: the cap the sidecar enforces is
// on the whole line, not on the array.
_bytes = Encoding.UTF8.GetByteCount(sb.ToString());
}
public int Count { get { return _count; } }
/// <summary>
/// Adds one already-serialised item. <paramref name="cursorAfter"/> is where the
/// family should resume if this turns out to be the last item on the page.
/// Returns false when the budget is spent — the caller stops, and `more` is true.
/// </summary>
public bool TryAdd(string item, string cursorAfter)
{
if (item == null)
return true;
int cost = Encoding.UTF8.GetByteCount(item) + (_count > 0 ? 1 : 0);
if (_count > 0 && _bytes + cost + Reserve > _budget)
{
_cut = "budget";
return false;
}
if (_count > 0)
_sb.Append(',');
_sb.Append(item);
_bytes += cost;
_count++;
_cursor = cursorAfter;
return true;
}
/// <summary>
/// Stops the page for a reason of the family's own — a per-request limit, say —
/// rather than because the budget ran out.
/// </summary>
public void Cut(string why)
{
_cut = why;
}
public void Close()
{
bool more = _cut != "end";
_sb.Append(']');
_sb.Bool("more", more);
_sb.Str("cut", _cut);
if (more && _cursor != null)
_sb.Str("cursor", _cursor);
}
}
}
}

254
overlay/Scripts/Custom/Bridge/BridgeBodies.cs
View File

@@ -1,254 +0,0 @@
using System;
using System.Collections.Generic;
using System.Globalization;
using System.Text;
using Server.Mobiles;
namespace Server.Custom.Bridge
{
/// <summary>
/// **Slug to body id — the part only the shard can do** (docs/link/v8.md §8, protocol 8,
/// phase 3).
///
/// The spawn atlas knows a creature by a **slug** derived from the class name it found in
/// `Spawns/*.xml` ("giant-spider"). The client knows the same creature by a **body id**
/// (28). Nothing in the ServUO tree declares that mapping as data. Today an operator
/// bridges it by hand, grepping `Scripts/Mobiles/Normal/&lt;Name&gt;.cs` for `Body =`,
/// which appears as a decimal, as hex (`0xD1`), as `Utility.RandomList(35, 36)` and as an
/// `m_IDs[]` table — a parse that is wrong on the shard's own custom creatures, which is
/// precisely the set an operator most wants pictures for.
///
/// Inside ServUO the problem does not exist: construct the type, read `Body.BodyID`,
/// delete it. <c>BridgeWorld.cs</c> already does exactly that for a different feature.
///
/// **This is the one asset-plane family that does NOT run on the asset worker**, and the
/// reason is the whole point of §8. Constructing and deleting a mobile is world mutation,
/// so it must happen on the Core thread — while the decode in <see cref="BridgeCatalog"/>
/// must happen off it, because it reads hundreds of megabytes and would stop the world for
/// every player on the shard. That split is why body resolution is its own request kind
/// rather than a step inside asset extraction.
///
/// Two consequences follow from answering on the Core thread, and both are bounds:
///
/// **The batch is small and the shard enforces the cap itself.** Every type constructed
/// here runs a real constructor — packing items, rolling skills, starting AI timers — and
/// all of that happens between two ticks of the world. The website chunks its own list;
/// a request over <see cref="BridgeConfig.AssetBodyBatch"/> names is **refused** rather
/// than truncated, so the two sides cannot quietly disagree about what was answered.
///
/// **It does not take the asset plane's single slot.** The slot exists to stop several
/// large replies queueing at once (§3.2); this reply is a few kilobytes and the work is
/// not on the worker, so claiming the slot would only make a body pass and a catalogue
/// page refuse each other for no benefit.
///
/// **A creature whose constructor randomises its body reports one of its variants**, not
/// an error and not a set. Constructing twice to detect that would double every side
/// effect above to learn something the bestiary does not render differently — both ids are
/// the same creature. The answer is stable enough to cache and cheap enough to redo.
/// </summary>
public static class BridgeBodies
{
public static void Initialize()
{
if (!BridgeConfig.Enabled)
return;
BridgeBoot.RegisterHandler("assets.bodies", OnBodies);
}
// ── the request ──────────────────────────────────────────────────────────────────────
private static void OnBodies(Dictionary<string, object> o)
{
var reqId = BridgeJson.GetString(o, "reqId");
if (reqId == null)
{
// Rule 1 of the asset plane: without a correlation id this reply lands on the
// event path, is persisted to the sidecar's store and broadcast to every
// subscriber. Refuse rather than answer.
BridgeAssets.Fail(null, "BAD_REQUEST", "assets.bodies requires a reqId");
return;
}
if (!BridgeConfig.AssetsEnabled)
{
BridgeAssets.Fail(reqId, "DISABLED", "asset extraction is disabled on this shard");
return;
}
var types = BridgeJson.GetStringList(o, "types");
if (types.Count == 0)
{
BridgeAssets.Fail(reqId, "BAD_REQUEST",
"assets.bodies requires a non-empty `types` array of ServUO class names");
return;
}
if (types.Count > BridgeConfig.AssetBodyBatch)
{
// Refuse, never truncate. A silently shortened answer looks identical to a
// complete one from the website's side, and the types that fell off the end would
// be recorded as "asked and unanswerable" rather than "never asked".
BridgeAssets.Fail(reqId, "BAD_REQUEST",
"assets.bodies takes at most " + BridgeConfig.AssetBodyBatch
+ " types per request (asked for " + types.Count + "); send them in chunks");
return;
}
Reply(reqId, types);
}
/// <summary>
/// Core thread. Constructs each type once, reads its body, deletes it.
///
/// Every outcome is a **row**, never a failed request: a shard is expected to be asked
/// about types it does not have (an atlas built from a tree that has since changed, a
/// spawn file naming a creature from a script package the operator removed), and a
/// status screen that fails the whole pass over one of those teaches an operator to
/// stop pressing the button.
/// </summary>
private static void Reply(string reqId, List<string> types)
{
var sb = BridgeJson.Begin("assets.bodies.ok");
sb.Str("reqId", reqId)
.Num("extractorVersion", BridgeAssets.EXTRACTOR_VERSION)
.Num("asked", types.Count);
// The envelope is shared with every other family (§3.4) even though this one never
// pages: the website drives the chunking, so `more` is always false and `cut` always
// "end". Writing it anyway means one reader shape on the other side rather than two.
var page = new BridgeAssets.PageBuilder(sb, "rows", BridgeConfig.AssetBatchBytes);
int resolved = 0;
foreach (var name in types)
{
string status;
int body;
Resolve(name, out body, out status);
if (status == "ok")
resolved++;
var item = new StringBuilder(96);
item.Append("{\"type\":");
BridgeJson.Text(item, name);
item.Append(",\"status\":\"").Append(status).Append('"');
if (status == "ok")
item.Append(",\"body\":").Append(body.ToString(CultureInfo.InvariantCulture));
item.Append('}');
// A chunk this small cannot spend the budget — the cap above is a hundred names
// and the budget is half a megabyte — but the check costs nothing and the day
// someone raises `AssetBodyBatch` it is the difference between a short page and a
// line the sidecar drops.
if (!page.TryAdd(item.ToString(), null))
break;
}
page.Close();
sb.Num("resolved", resolved);
BridgeLink.Emit(sb.End());
}
/// <summary>
/// One type name to one body id.
///
/// `status` is the field the website records, and the four values are four different
/// things an operator can act on:
///
/// <c>ok</c> — constructed, body read.
/// <c>unknown</c> — no such type on this shard. The spawn file names something the
/// scripts do not define, which is a real drift an operator wants to see.
/// <c>notCreature</c> — the type exists but is not a `BaseCreature`. Spawn files
/// legitimately name items and static decorations; those have no body and never will,
/// so this is a permanent answer rather than a retryable failure.
/// <c>failed</c> — the constructor threw, or the type has none that takes no
/// arguments. Caught per type, because one creature whose constructor depends on a
/// script package the operator removed must not cost the other ninety-nine.
/// </summary>
private static void Resolve(string name, out int body, out string status)
{
body = 0;
status = "failed";
Type type;
try
{
// `true` is ignoreCase — spawn files are hand-edited and their casing drifts from
// the class it names far more often than the name itself does.
type = ScriptCompiler.FindTypeByName(name, true);
}
catch
{
status = "failed";
return;
}
if (type == null)
{
status = "unknown";
return;
}
if (!typeof(BaseCreature).IsAssignableFrom(type) || type.IsAbstract)
{
status = "notCreature";
return;
}
BaseCreature creature = null;
try
{
creature = Activator.CreateInstance(type) as BaseCreature;
if (creature == null)
{
status = "failed";
return;
}
body = creature.Body.BodyID;
status = body > 0 ? "ok" : "failed";
}
catch (Exception e)
{
Console.WriteLine("[Bridge] assets.bodies: {0}: {1}: {2}",
name, e.GetType().Name, e.Message);
status = "failed";
}
finally
{
if (creature != null)
{
try
{
// Deleting the mobile deletes the items it packed — `Mobile.Delete` walks
// `Items`, and `Item.Delete` walks what each contains — and stops its AI
// timer. A creature left alive here is a creature standing at (0,0,0) on
// the internal map forever, saved with the world, once per import.
creature.Delete();
}
catch
{
// Nothing useful is left to do, and throwing out of `finally` would lose
// whatever the try block was already reporting.
}
}
}
}
}
}

2
overlay/Scripts/Custom/Bridge/BridgeBoot.cs
View File

@@ -260,8 +260,6 @@ namespace Server.Custom.Bridge
e.Mobile.SendMessage("Bridge: {0}", BridgeParticipation.Status());
e.Mobile.SendMessage("Bridge: {0}", BridgeWorld.Status());
e.Mobile.SendMessage("Bridge: {0}", BridgeOneShots.Status());
e.Mobile.SendMessage("Bridge: {0}", BridgeAssets.Status());
e.Mobile.SendMessage("Bridge: {0}", BridgeCatalog.Status());
break;
}
}

1010
overlay/Scripts/Custom/Bridge/BridgeCatalog.cs
View File

File diff suppressed because it is too large Load Diff

772
overlay/Scripts/Custom/Bridge/BridgeCliloc.cs
View File

@@ -1,772 +0,0 @@
using System;
using System.Collections.Generic;
using System.Globalization;
using System.IO;
using System.Text;
using Ultima;
namespace Server.Custom.Bridge
{
/// <summary>
/// **The cliloc table, over the bridge** (docs/link/v8.md §9 — protocol 8, phase 2).
///
/// A "cliloc" is UO's localization table: an integer id mapped to a display string. Items
/// on the wire carry a `LabelNumber`, never a name, so without this table the website can
/// only render `id 1023721` where the game renders "quarter staff". The number was never
/// the missing piece; the table was.
///
/// Until this phase the operator supplied it by hand: install UOFiddler, build a converter
/// against its `Ultima.dll`, run it over their own `Cliloc.enu`, copy a 5 MB file to the
/// web host and point a setting at it. That whole pipeline existed for one reason — the
/// file is compressed and **nothing in this stack could read it**. ServUO's own bundled
/// `Ultima.StringList` implements the plain layout only and throws on a modern client's
/// file, which is also why the shard's `VendorSearch.GetItemName` has always been inert.
///
/// So this class is the one decoder protocol 8 **writes** rather than calls (§4): a port
/// of UOFiddler's Mythic decompressor into the overlay, after which the shard can read its
/// own client's table and hand it to the website over the same request/reply path as
/// everything else. The operator installs nothing.
///
/// **Attribution.** The decompression below is a port of `Ultima/Helpers/MythicDecompress`
/// and `MoveToFront` from UOFiddler (https://github.com/polserver/UOFiddler), which is
/// released under the **Beerware** licence — compatible with this tree's GPL-3.0-or-later.
/// It is rewritten for .NET Framework 4.8: the original is written against `Span&lt;T&gt;`,
/// `ArrayPool&lt;T&gt;` and `BinaryPrimitives`, none of which ServUO's `net48` target has.
///
/// **What is NOT here, deliberately.** Shard-added items carry cliloc ids no client table
/// contains, and ServUO has no server-side notion of a custom cliloc — there is nothing in
/// the tree to read. That gap is in the *game*, not in this pipeline, so the website keeps
/// its `custom/` overlay directory and merges it over whatever arrives here. This class
/// answers exactly one question: what does the client's own table say.
/// </summary>
public static class BridgeCliloc
{
/// <summary>
/// Languages this can serve.
///
/// Not an arbitrary code: <c>Ultima.Files</c> resolves only the names in its own file
/// table, and cliloc files are represented there by these four. Asking for anything
/// else cannot resolve to a path however the client is laid out, so it is refused by
/// name rather than answered with an empty table.
///
/// `custom1` / `custom2` are the *client-side* custom cliloc files a shard ships to
/// its players. Nothing on the website imports them today — its `custom/` overlay
/// directory is the supported answer — but they are the shard's files and they are
/// readable, so they are not artificially excluded.
/// </summary>
private static readonly string[] Languages = { "enu", "deu", "custom1", "custom2" };
private const string DefaultLanguage = "enu";
/// <summary>
/// How long a decoded table is kept in memory after its last page.
///
/// A stock `Cliloc.enu` decodes to ~67,000 live strings; holding that forever on a
/// shard that imports once a month is rude, and decoding it again costs about a
/// second. So it is cached only for as long as an import is plausibly still running:
/// freed when the last page is served, and expired on the next request if one never
/// comes (an import abandoned halfway leaves nothing behind).
/// </summary>
private static readonly TimeSpan CacheIdle = TimeSpan.FromMinutes(5);
private static readonly object _sync = new object();
private static Table _cached;
public static void Initialize()
{
if (!BridgeConfig.Enabled)
return;
BridgeBoot.RegisterHandler("cliloc.table", OnTable);
}
// ── the request plane ────────────────────────────────────────────────────────────────
/// <summary>
/// Core thread. Validates, then hands the decode to the asset worker — reading and
/// decompressing five megabytes is emphatically not something to do while the world
/// is waiting, and <see cref="BridgeAssets"/>'s single slot is what keeps the shard's
/// outbound queue at a depth of about one while it happens.
/// </summary>
private static void OnTable(Dictionary<string, object> o)
{
var reqId = BridgeJson.GetString(o, "reqId");
if (reqId == null)
{
// Without a correlation id this reply would land on the event path, be persisted
// to the sidecar's store and broadcast to every subscriber — a megabyte of
// strings to every connected client, forever. Refuse instead (§3.1).
BridgeAssets.Fail(null, "BAD_REQUEST", "cliloc.table requires a reqId");
return;
}
if (!BridgeConfig.AssetsEnabled)
{
BridgeAssets.Fail(reqId, "DISABLED", "asset extraction is disabled on this shard");
return;
}
var lang = BridgeJson.GetString(o, "lang");
if (String.IsNullOrEmpty(lang))
lang = DefaultLanguage;
lang = lang.ToLowerInvariant();
if (Array.IndexOf(Languages, lang) < 0)
{
BridgeAssets.Fail(reqId, "NOT_FOUND",
"no cliloc file for language '" + lang + "' (this shard can serve: "
+ String.Join(", ", Languages) + ")");
return;
}
// The cursor is this family's own resume point and it is a cliloc NUMBER, not an
// offset into anything. That matters: the cache behind it can be dropped and rebuilt
// between two pages of the same import (idle expiry, a second import, a restart), and
// an index into a list would silently mean something different afterwards. "Resume
// after id N" survives all of it, because the table is served in id order.
int after = -1;
var cursor = BridgeJson.GetString(o, "cursor");
if (!String.IsNullOrEmpty(cursor))
{
if (!TryParseCursor(cursor, out after))
{
BridgeAssets.Fail(reqId, "BAD_REQUEST", "malformed cursor: " + cursor);
return;
}
}
string language = lang;
int resumeAfter = after;
BridgeAssets.Accept(reqId, "cliloc.table", () => ReplyTable(reqId, language, resumeAfter));
}
private static bool TryParseCursor(string cursor, out int after)
{
after = -1;
if (!cursor.StartsWith("n:", StringComparison.Ordinal))
return false;
return Int32.TryParse(
cursor.Substring(2), NumberStyles.Integer, CultureInfo.InvariantCulture, out after);
}
/// <summary>
/// Asset worker. Decodes (or reuses) the table and writes one page of it.
/// </summary>
private static void ReplyTable(string reqId, string lang, int after)
{
string path = ResolvePath(lang);
if (path == null)
{
BridgeAssets.Fail(reqId, "NOT_FOUND",
"this shard's client has no cliloc." + lang + " (looked where ServUO's own "
+ "data path points)");
return;
}
Table table;
string code, reason;
if (!TryLoad(lang, path, out table, out code, out reason))
{
BridgeAssets.Fail(reqId, code, reason);
return;
}
var sb = BridgeJson.Begin("cliloc.table.ok");
sb.Str("reqId", reqId)
.Str("lang", lang)
.Num("extractorVersion", BridgeAssets.EXTRACTOR_VERSION)
.Str("file", Path.GetFileName(path))
// The website pages this table over several round trips and must be able to tell
// that the file changed underneath it — an operator patching their client mid-import
// would otherwise produce one table stitched from two, with no error anywhere. It
// compares these two fields across pages and starts over if they move.
.Num("size", table.Size)
.Num("mtime", table.MTime)
.Num("total", table.Count)
.Bool("compressed", table.Compressed);
// Before the page opens, not after it closes: PageBuilder reserves room for the
// envelope it still has to write, and a field appended past Close() is spent outside
// that reserve. It fits today by a wide margin, and it is the kind of thing the next
// family copies.
int start = table.IndexAfter(after);
sb.Num("from", start);
var page = new BridgeAssets.PageBuilder(sb, "rows", BridgeConfig.AssetBatchBytes);
int i = start;
for (; i < table.Count; i++)
{
var item = new StringBuilder(96);
item.Append("{\"n\":").Append(table.Numbers[i].ToString(CultureInfo.InvariantCulture));
item.Append(",\"f\":").Append(table.Flags[i].ToString(CultureInfo.InvariantCulture));
item.Append(",\"t\":");
BridgeJson.Text(item, table.Texts[i]);
item.Append('}');
if (!page.TryAdd(item.ToString(), "n:" + table.Numbers[i].ToString(CultureInfo.InvariantCulture)))
break;
}
page.Close();
bool finished = i >= table.Count;
BridgeLink.Emit(sb.End());
// The last page is also the end of the import, so let the strings go. A retry of that
// page re-decodes, which costs a second and happens approximately never; holding ~67k
// strings against that is the wrong trade.
if (finished)
Release(lang);
}
private static string ResolvePath(string lang)
{
try
{
// ServUO's own `Scripts/Misc/DataPath.cs` calls `Files.SetMulPath` for every
// configured data directory at Configure time, so this resolves against the
// client the SHARD is running on — including on Linux, where `Ultima.Files`'s
// registry lookup finds nothing on its own.
return Files.GetFilePath("cliloc." + lang);
}
catch
{
return null;
}
}
// ── the decoded table ────────────────────────────────────────────────────────────────
private sealed class Table
{
public string Lang;
public long Size;
public long MTime;
public bool Compressed;
public int[] Numbers;
public byte[] Flags;
public string[] Texts;
public DateTime LastUsed;
public int Count { get { return Numbers.Length; } }
/// <summary>
/// Index of the first row with a number greater than <paramref name="after"/>.
/// Binary search, because the rows are in id order by construction and a page
/// deep into the table would otherwise walk everything before it.
/// </summary>
public int IndexAfter(int after)
{
if (after < 0)
return 0;
int lo = 0, hi = Numbers.Length;
while (lo < hi)
{
int mid = lo + ((hi - lo) >> 1);
if (Numbers[mid] <= after)
lo = mid + 1;
else
hi = mid;
}
return lo;
}
}
private static bool TryLoad(string lang, string path, out Table table, out string code, out string reason)
{
code = null;
reason = null;
long size, mtime;
try
{
var info = new FileInfo(path);
size = info.Length;
mtime = (long)(info.LastWriteTimeUtc - new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc))
.TotalMilliseconds;
}
catch (Exception e)
{
table = null;
code = "UNREADABLE";
reason = "cannot stat " + Path.GetFileName(path) + ": " + e.Message;
return false;
}
lock (_sync)
{
if (_cached != null)
{
bool stale = _cached.Lang != lang
|| _cached.Size != size
|| _cached.MTime != mtime
|| DateTime.UtcNow - _cached.LastUsed > CacheIdle;
if (stale)
_cached = null;
}
if (_cached != null)
{
_cached.LastUsed = DateTime.UtcNow;
table = _cached;
return true;
}
}
byte[] raw;
try
{
raw = File.ReadAllBytes(path);
}
catch (Exception e)
{
table = null;
code = "UNREADABLE";
reason = "cannot read " + Path.GetFileName(path) + ": " + e.Message;
return false;
}
bool compressed = IsCompressed(raw);
byte[] plain;
if (compressed)
{
try
{
plain = Mythic.Decompress(raw);
}
catch (Exception e)
{
table = null;
code = "UNREADABLE";
reason = "cannot decompress " + Path.GetFileName(path) + ": " + e.Message;
return false;
}
}
else
{
plain = raw;
}
var built = new Table
{
Lang = lang,
Size = size,
MTime = mtime,
Compressed = compressed,
LastUsed = DateTime.UtcNow
};
if (!TryParseRecords(plain, built, out reason))
{
table = null;
code = "UNREADABLE";
return false;
}
lock (_sync)
{
_cached = built;
}
table = built;
return true;
}
private static void Release(string lang)
{
lock (_sync)
{
if (_cached != null && _cached.Lang == lang)
_cached = null;
}
}
/// <summary>
/// Every compressed cliloc begins with a DWORD whose high byte is <c>0x8E</c> — the
/// top byte of UOFiddler's `HeaderXorKey`, showing through because the value it hides
/// (a length) is far smaller than the key. That single byte is what tells a modern
/// client's file from the pre-2010 plain layout, and both are accepted here: a shard
/// running an old or hand-built client is not a broken shard.
/// </summary>
private static bool IsCompressed(byte[] buffer)
{
return buffer.Length >= 4 && buffer[3] == 0x8E;
}
// ── the plain layout ─────────────────────────────────────────────────────────────────
private const int HeaderBytes = 6; // int32 version + int16 language marker
private const int RecordHeaderBytes = 7; // int32 number + byte flag + uint16 length
/// <summary>
/// Parses the plain layout into the sorted, blank-free arrays the wire wants.
///
/// **Strict about truncation**, and that strictness is the point: a half-decoded table
/// is indistinguishable from a complete one downstream — you would simply see some
/// items named and some not, which is exactly what "no table at all" looks like. So a
/// record running past the end of the buffer is an error naming its offset, never a
/// short table.
///
/// **Blanks are dropped here rather than on the website.** Roughly 56,000 of a stock
/// table's 123,490 entries are empty strings the client reserves and never uses, the
/// website discards them at import already, and a row that resolves to no name is
/// indistinguishable from no row at all to every caller. Dropping them halves what
/// crosses the wire for data that would be thrown away on arrival.
///
/// **A repeated id is resolved last-wins**, matching the client's own loader (its
/// dictionary assignment overwrites). The plain format permits it, so a file the game
/// itself would load happily must not fail here.
/// </summary>
private static bool TryParseRecords(byte[] data, Table into, out string reason)
{
reason = null;
if (data.Length < HeaderBytes)
{
reason = "cliloc file is shorter than its 6-byte header";
return false;
}
var byNumber = new Dictionary<int, Entry>(140000);
int offset = HeaderBytes;
int read = 0;
while (offset < data.Length)
{
if (offset + RecordHeaderBytes > data.Length)
{
reason = "truncated record header at byte " + offset + " (" + read + " entries read)";
return false;
}
int number = ReadInt32(data, offset);
byte flag = data[offset + 4];
// Unsigned: reading this signed (as ServUO's own SDK does) turns any string over
// 32 KB into a negative length. Real tables top out around 12 KB, so it changes
// nothing today and costs nothing to get right.
int length = data[offset + 5] | (data[offset + 6] << 8);
offset += RecordHeaderBytes;
if (offset + length > data.Length)
{
reason = "truncated record body at byte " + offset + " (" + read + " entries read)";
return false;
}
string text;
try
{
text = Encoding.UTF8.GetString(data, offset, length);
}
catch (Exception e)
{
reason = "entry " + number + " at byte " + offset + " is not valid UTF-8: " + e.Message;
return false;
}
offset += length;
read++;
byNumber[number] = new Entry { Flag = flag, Text = text };
}
var numbers = new List<int>(byNumber.Count);
foreach (var pair in byNumber)
{
if (IsBlank(pair.Value.Text))
continue;
numbers.Add(pair.Key);
}
numbers.Sort();
into.Numbers = numbers.ToArray();
into.Flags = new byte[numbers.Count];
into.Texts = new string[numbers.Count];
for (int i = 0; i < numbers.Count; i++)
{
var entry = byNumber[numbers[i]];
into.Flags[i] = entry.Flag;
into.Texts[i] = entry.Text;
}
return true;
}
private struct Entry
{
public byte Flag;
public string Text;
}
private static bool IsBlank(string text)
{
if (String.IsNullOrEmpty(text))
return true;
for (int i = 0; i < text.Length; i++)
{
if (!Char.IsWhiteSpace(text[i]))
return false;
}
return true;
}
private static int ReadInt32(byte[] data, int at)
{
return data[at] | (data[at + 1] << 8) | (data[at + 2] << 16) | (data[at + 3] << 24);
}
// ── the Mythic container ─────────────────────────────────────────────────────────────
/// <summary>
/// The decompressor, ported from UOFiddler (Beerware; see this class's summary).
///
/// The container is two stages over the plain cliloc bytes, undone in reverse:
///
/// 1. A 4-byte header holding the decompressed length, XORed with `0x8E2C9A3D` —
/// which is where the `0x8E` sniff byte comes from.
/// 2. A **move-to-front** coding of…
/// 3. …a Burrows-Wheeler-style transform whose 1 KB frequency header (256 little-endian
/// counts, one per byte value) is both the table sizes and the total output length.
///
/// Rewritten against plain arrays: the upstream is `Span&lt;T&gt;`/`ArrayPool&lt;T&gt;`
/// code and ServUO targets `net48`, which has neither without a package this tree does
/// not vendor. The algorithm is unchanged, including the parts that read oddly — the
/// three-region `partial` table (counts, cursors, ends) and the symbol-table shifts are
/// the original's, deliberately, because this is a format decoder and a tidier
/// rewrite is a chance to be subtly wrong about someone else's bytes.
/// </summary>
private static class Mythic
{
private const uint HeaderXorKey = 0x8E2C9A3D;
private const int FrequencyHeaderSize = 1024; // 256 little-endian ints
public static byte[] Decompress(byte[] source)
{
if (source.Length < 4)
throw new InvalidDataException("compressed cliloc is shorter than its header");
uint declared = (uint)ReadInt32(source, 0) ^ HeaderXorKey;
if (declared == 0 || declared > Int32.MaxValue)
throw new InvalidDataException("compressed cliloc declares an impossible length");
var mtf = new byte[source.Length - 4];
MoveToFrontDecode(source, 4, mtf);
var output = new byte[(int)declared];
int written = InverseTransform(mtf, output);
if (written != (int)declared)
{
throw new InvalidDataException(
"decompressed length " + written + " does not match the declared " + declared);
}
return output;
}
private static void MoveToFrontDecode(byte[] input, int from, byte[] output)
{
var symbols = new byte[256];
for (int i = 0; i < 256; i++)
symbols[i] = (byte)i;
for (int i = 0; i < output.Length; i++)
{
int index = input[from + i];
byte symbol = symbols[index];
output[i] = symbol;
for (int j = index; j > 0; j--)
symbols[j] = symbols[j - 1];
symbols[0] = symbol;
}
}
private static int InverseTransform(byte[] input, byte[] destination)
{
if (input.Length < FrequencyHeaderSize)
throw new InvalidDataException("compressed cliloc is smaller than its frequency header");
// Three regions of 256: [0..255] the counts read from the header, [256..511] a
// moving cursor per symbol, [512..767] where that symbol's run ends.
var partial = new int[256 * 3];
for (int i = 0; i < 256; i++)
partial[i] = ReadInt32(input, i * 4);
int sum = 0;
for (int i = 0; i < 256; i++)
{
if (partial[i] < 0)
throw new InvalidDataException("compressed cliloc has a negative symbol count");
sum += partial[i];
}
if (sum == 0)
return 0;
if (destination.Length < sum)
throw new InvalidDataException("compressed cliloc's frequency header outruns its declared length");
int nonZero = 0;
for (int i = 0; i < 256; i++)
{
if (partial[i] != 0)
nonZero++;
}
var frequency = new byte[256];
Frequency(partial, frequency);
var symbols = new byte[256];
for (int i = 0; i < 256; i++)
symbols[i] = (byte)i;
for (int i = 0, m = 0; i < nonZero; ++i)
{
byte freq = frequency[i];
Need(input, m + FrequencyHeaderSize);
symbols[input[m + FrequencyHeaderSize]] = freq;
partial[freq + 256] = m + 1;
m += partial[freq];
partial[freq + 512] = m;
}
byte val = symbols[0];
int count = 0;
do
{
destination[count] = val;
if (partial[val + 256] < partial[val + 512])
{
Need(input, partial[val + 256] + FrequencyHeaderSize);
byte idx = input[partial[val + 256] + FrequencyHeaderSize];
partial[val + 256]++;
if (idx != 0)
{
ShiftLeft(symbols, idx);
symbols[idx] = val;
val = symbols[0];
}
}
else if (nonZero-- > 0)
{
ShiftLeft(symbols, nonZero);
val = symbols[0];
}
count++;
}
while (count < sum);
return sum;
}
/// <summary>
/// The upstream indexes the payload without bounds-checking it, which is safe for
/// a file the client wrote and is not safe for a file this shard was handed. A
/// truncated or hand-edited container would otherwise read whatever follows the
/// buffer in memory — or, on .NET, throw an `IndexOutOfRangeException` from inside
/// a decoder, which says nothing useful to an operator. This turns both into one
/// named, reportable failure.
/// </summary>
private static void Need(byte[] input, int at)
{
if (at < 0 || at >= input.Length)
throw new InvalidDataException("compressed cliloc ends mid-stream (wanted byte " + at + ")");
}
/// <summary>
/// Symbol values ordered by descending count — the order the coder assigned its
/// runs in. Repeated max-finding rather than a sort, as upstream: 256 passes over
/// 256 entries is nothing, and it reproduces the original's tie-breaking (the
/// lowest index wins), which a comparison sort would not.
/// </summary>
private static void Frequency(int[] counts, byte[] output)
{
var tmp = new int[256];
Array.Copy(counts, tmp, 256);
for (int i = 0; i < 256; i++)
{
int value = 0;
byte index = 0;
for (int j = 0; j < 256; j++)
{
if (tmp[j] > value)
{
index = (byte)j;
value = tmp[j];
}
}
if (value == 0)
break;
output[i] = index;
tmp[index] = 0;
}
}
private static void ShiftLeft(byte[] symbols, int upTo)
{
for (int i = 0; i < upTo; ++i)
symbols[i] = symbols[i + 1];
}
private static int ReadInt32(byte[] data, int at)
{
return data[at] | (data[at + 1] << 8) | (data[at + 2] << 16) | (data[at + 3] << 24);
}
}
}
}

86
overlay/Scripts/Custom/Bridge/BridgeConfig.cs
View File

@@ -87,43 +87,6 @@ namespace Server.Custom.Bridge
// morning. Those are different consents, and one switch cannot express both.
public static bool EventsEnabled { get; private set; }
// ---- the asset plane (docs/link/v8.md §3, protocol 8) ----
//
// Its own gate again, and for the same reason the event plane got one: enabling this is
// an operator consenting to the WEBSITE READING THEIR CLIENT FILES -- art, animations and
// the string table, off the host's disk, over the link. That is a different consent from
// publishing world state, and one switch cannot express both. Reads only: nothing on this
// plane writes anything, anywhere.
public static bool AssetsEnabled { get; private set; }
public static int AssetBatchBytes { get; private set; }
// How many types one `assets.bodies` request may name (§8, phase 3). This is the ONLY
// asset-plane bound counted in items rather than bytes, and deliberately so: the cost it
// bounds is not the size of the reply, it is constructing and deleting that many real
// mobiles ON THE CORE THREAD, between two ticks of the world.
public static int AssetBodyBatch { get; private set; }
// How many keys one `assets.fetch` request may name. Bytes still cut the page; this only
// bounds how large a request the shard will parse and walk at all.
public static int AssetFetchKeys { get; private set; }
// The wall-clock budget for one catalogue page (§4.8, phase 3). The catalogue's rows are
// ninety bytes, so the byte budget never stops it -- but building them means decoding
// hundreds of animations, and the sidecar gives a reply ten seconds. Kept well under that,
// because the reply still has to be built, serialised and cross the wire afterwards.
public static int AssetScanMs { get; private set; }
// Which direction the catalogue renders (§5.1). Both are settings and neither is in the
// asset key, because five directions would five-fold every count in §11 to express a
// choice nobody is going to vary.
//
// The split is not arbitrary and was found by RENDERING all five rather than from a table:
// index 0 is head-on, which is what a character portrait wants and the least legible view
// there is of a four-legged creature. A wolf seen from the front is a dark blob; at index
// 1, the front three-quarter, it is unmistakably a wolf.
public static int AssetPlayerDirection { get; private set; }
public static int AssetCreatureDirection { get; private set; }
public static int LeaseMaxDurationSec { get; private set; }
public static int LeaseGraceSec { get; private set; }
@@ -181,47 +144,6 @@ namespace Server.Custom.Bridge
Port = Config.Get("Bridge.Port", 7788);
QueueCap = Config.Get("Bridge.QueueCap", 10000);
AssetsEnabled = Config.Get("Bridge.AssetsEnabled", true);
// The largest reply this plane will build, in ENCODED bytes -- not items, because the
// ceiling it has to live inside is a byte ceiling. Clamped to half the sidecar's 1 MiB
// inbound line cap, and the halving is load-bearing rather than cautious: a page
// always admits its first item even when that item alone exceeds the budget (the
// alternative is an oversized item being skipped forever and its family never making
// progress), so the wire must still have room for one such overshoot.
AssetBatchBytes = Config.Get("Bridge.AssetBatchBytes", 512 * 1024);
if (AssetBatchBytes < 64 * 1024)
AssetBatchBytes = 64 * 1024;
if (AssetBatchBytes > 512 * 1024)
AssetBatchBytes = 512 * 1024;
AssetBodyBatch = Config.Get("Bridge.AssetBodyBatch", 100);
if (AssetBodyBatch < 1)
AssetBodyBatch = 1;
if (AssetBodyBatch > 500)
AssetBodyBatch = 500;
AssetFetchKeys = Config.Get("Bridge.AssetFetchKeys", 2000);
if (AssetFetchKeys < 1)
AssetFetchKeys = 1;
if (AssetFetchKeys > 10000)
AssetFetchKeys = 10000;
AssetScanMs = Config.Get("Bridge.AssetScanMs", 3000);
if (AssetScanMs < 250)
AssetScanMs = 250;
// Half the sidecar's 10 s reply timeout, so the page still has time to be serialised
// and written after the scan stops. A budget set at the timeout would produce replies
// that are always thrown away.
if (AssetScanMs > 5000)
AssetScanMs = 5000;
// Clamped to 0-4: 5-7 are the client MIRRORING 1-3, which `Frame` decodes through a
// different pointer-arithmetic branch that nothing in BridgeAssetValidator has
// checked. Accepting one would hand an unverified write path a bitmap to fill.
AssetPlayerDirection = Clamp(Config.Get("Bridge.AssetPlayerDirection", 0), 0, 4);
AssetCreatureDirection = Clamp(Config.Get("Bridge.AssetCreatureDirection", 1), 0, 4);
StatSweepSeconds = Config.Get("Bridge.StatSweepSeconds", 30);
DecaySweepSeconds = Config.Get("Bridge.DecaySweepSeconds", 60);
EconomySweepSeconds = Config.Get("Bridge.EconomySweepSeconds", 300);
@@ -572,14 +494,6 @@ namespace Server.Custom.Bridge
return fallback;
}
private static int Clamp(int value, int min, int max)
{
if (value < min)
return min;
return value > max ? max : value;
}
public static string Describe()
{
return String.Format(

580
tools/patch_client.ps1
View File

@@ -1,580 +0,0 @@
<#
.SYNOPSIS
Builds a deliberately patched UO client for the Asset Bridge phase 0 spike.
.DESCRIPTION
docs/link/v8.md section 16 phase 0 drives ServUO's vendored `Ultima` decoders "over a deliberately
patched client". Stock clients are not the interesting case: they are the case the library was
written against, and the whole reason phase 0 exists is that section 4 chose to call code that can
take the shard down if it is wrong. A shard operator's client is patched -- custom art, a
verdata.mul, a hand-edited Bodyconv.def -- and that is what has to be survived.
This copies a client and then breaks the copy in four deliberate, catalogued ways. It NEVER
writes to the source: every file it patches is hashed before and after, and a changed source
hash aborts the run.
Each defect is recorded in `patched-client.manifest.json` next to the copy, so the probe's
report can be read against what was actually done rather than against a memory of it. The
manifest is the answer to "is a nonzero REFUSED-BUT-DECODED count a bug or the point?".
.PARAMETER Source
The client to copy. Defaults to this machine's.
.PARAMETER Dest
Where to build the patched copy. Needs ~3.5 GB.
.PARAMETER Tiers
Which defects to apply. Default: all four.
verdata Author a verdata.mul, which this client does not have. Ultima consults Verdata on
EVERY art and anim lookup (Art's FileIndex is built with verdata file id 4,
Animations' with 6), so on a client with no verdata.mul that entire branch is
dead code that has never been exercised -- the largest untested surface in the
library we are about to depend on. Includes one legitimate patch and one whose
lookup points past verdata.mul's own end, because `FileIndex.Seek` bounds-checks
the mul and does not bounds-check verdata.
customart Fill unused artidx.mul slots with real records appended to art.mul, the way a
custom-art shard does. Tests that our out-of-range accounting comes from the
file rather than from a constant someone wrote down.
corrupt Rewrite index entries and record headers into the shapes that reading Art.cs
says are reachable: a lookup past EOF, a record that starts inside the file and
ends outside it, a length too small for a header, absurd dimensions, a row table
pointing outside its own record, and a land tile shorter than the fixed 2,024
bytes LoadLand always reads.
bodyconv Add Bodyconv.def lines pointing bodies at an anim file that holds nothing, and at
an index in another file that holds something unrelated -- the gargoyle-666 spider
case, reproduced on purpose. Proves the extractor takes BodyConverter.Convert's
answer and stops (v8.md section 4.3).
nouop Move artLegacyMUL.uop aside, so art is read from art.mul/artidx.mul.
This is not cosmetic and it is not optional if you want the customart or corrupt
tiers to mean anything. FileIndex's UOP constructor ends with a bare
`MulPath = uopPath`: when artLegacyMUL.uop is present it wins OUTRIGHT and
art.mul / artidx.mul are never opened. Every index-level defect below writes to
files the library does not read on a modern client, so without this tier those
two tiers are inert while still reporting that they applied.
It is also a real configuration in its own right: plenty of shards run mul-only
clients, and a custom-art shard that adds graphics to art.mul while the UOP is
still there gets nothing at all -- an operator trap worth knowing about.
.PARAMETER SkipCopy
Re-patch an existing copy without re-copying 3.5 GB. Only safe on a copy this script made and
has not patched yet -- patching twice compounds the defects and invalidates the manifest.
.EXAMPLE
.\tools\patch_client.ps1 -Dest D:\uo-patched-client
.NOTES
Test scaffolding. Never deployed. The copy contains EA's client art -- like every other
extraction in this project it stays on the machine that made it and is never committed.
#>
[CmdletBinding()]
param(
[string] $Source = 'D:\Games\Electronic Arts\Ultima Online Classic',
[Parameter(Mandatory = $true)]
[string] $Dest,
[ValidateSet('verdata', 'customart', 'corrupt', 'bodyconv', 'nouop')]
[string[]] $Tiers = @('nouop', 'verdata', 'customart', 'corrupt', 'bodyconv'),
[switch] $SkipCopy,
[switch] $Force
)
$ErrorActionPreference = 'Stop'
# Files this script may write to in the copy. Anything not on this list is a bug in the script,
# and the source-hash check at the end is what proves it.
$PatchTargets = @('artidx.mul', 'art.mul', 'verdata.mul', 'Bodyconv.def', 'artLegacyMUL.uop')
# -- Little-endian helpers (BitConverter is fine, but the intent reads better named) ----------
function Read-Int32LE([byte[]] $Bytes, [int] $Offset) {
return [BitConverter]::ToInt32($Bytes, $Offset)
}
function Write-Int32LE([byte[]] $Bytes, [int] $Offset, [int] $Value) {
[Array]::Copy([BitConverter]::GetBytes([int] $Value), 0, $Bytes, $Offset, 4)
}
function Get-ArtEntry([byte[]] $Idx, [int] $Index) {
$at = $Index * 12
return [pscustomobject]@{
Index = $Index
Lookup = Read-Int32LE $Idx $at
Length = Read-Int32LE $Idx ($at + 4)
Extra = Read-Int32LE $Idx ($at + 8)
}
}
function Set-ArtEntry([byte[]] $Idx, [int] $Index, [int] $Lookup, [int] $Length, [int] $Extra) {
$at = $Index * 12
Write-Int32LE $Idx $at $Lookup
Write-Int32LE $Idx ($at + 4) $Length
Write-Int32LE $Idx ($at + 8) $Extra
}
# The defect catalogue. Every mutation appends to this, and it is written out as the manifest.
$script:Defects = New-Object System.Collections.ArrayList
function Add-Defect([string] $Tier, [string] $Key, [string] $What, [string] $Expect) {
[void] $script:Defects.Add([pscustomobject]@{
tier = $Tier
key = $Key
what = $What
expect = $Expect
})
Write-Host (" {0,-22} {1}" -f $Key, $What)
}
# -- Preflight --------------------------------------------------------------------------------
if (-not (Test-Path -LiteralPath $Source)) {
throw "source client not found: $Source"
}
$sourceFull = (Resolve-Path -LiteralPath $Source).Path
if (Test-Path -LiteralPath $Dest) {
$destFull = (Resolve-Path -LiteralPath $Dest).Path
if ($destFull -eq $sourceFull) {
throw "Dest is the source client. Refusing -- this script destroys what it points at."
}
if (-not $SkipCopy -and -not $Force) {
throw "$Dest already exists. Pass -Force to overwrite it, or -SkipCopy to patch it in place."
}
}
Write-Host "source: $sourceFull"
Write-Host "dest: $Dest"
Write-Host "tiers: $($Tiers -join ', ')"
Write-Host ""
# Hash the source files we are about to touch, so "it never writes to the source" is checked and
# not merely asserted.
$before = @{}
foreach ($name in $PatchTargets) {
$path = Join-Path $sourceFull $name
if (Test-Path -LiteralPath $path) {
$before[$name] = (Get-FileHash -LiteralPath $path -Algorithm SHA256).Hash
}
}
# -- Copy -------------------------------------------------------------------------------------
if ($SkipCopy) {
Write-Host "skipping copy (-SkipCopy)"
if (-not (Test-Path -LiteralPath $Dest)) { throw "-SkipCopy but $Dest does not exist" }
} else {
Write-Host "copying (this is ~3.5 GB; a few minutes)..."
# /MIR so a -Force re-run starts clean rather than merging into an already-patched tree.
# /NJH /NJS /NDL /NFL keep robocopy's output to the errors.
$null = robocopy $sourceFull $Dest /MIR /R:1 /W:1 /NJH /NJS /NDL /NFL /NP
# Robocopy exit codes below 8 are success; 8 and above are real failures.
if ($LASTEXITCODE -ge 8) { throw "robocopy failed with exit code $LASTEXITCODE" }
# Robocopy's "1 = files were copied" would otherwise become this script's exit code and read
# as a failure to anything checking it.
$global:LASTEXITCODE = 0
Write-Host "copied."
}
Write-Host ""
$destFull = (Resolve-Path -LiteralPath $Dest).Path
$artIdxPath = Join-Path $destFull 'artidx.mul'
$artMulPath = Join-Path $destFull 'art.mul'
if (-not (Test-Path -LiteralPath $artIdxPath)) { throw "no artidx.mul in the copy" }
$idx = [System.IO.File]::ReadAllBytes($artIdxPath)
$entryCount = [int] ($idx.Length / 12)
$artMulLength = (Get-Item -LiteralPath $artMulPath).Length
Write-Host ("artidx.mul holds {0:N0} entries; art.mul is {1:N0} bytes" -f $entryCount, $artMulLength)
Write-Host ""
# -- Tier: nouop ------------------------------------------------------------------------------
$uopPath = Join-Path $destFull 'artLegacyMUL.uop'
$uopPresent = Test-Path -LiteralPath $uopPath
if ($Tiers -contains 'nouop') {
Write-Host "tier nouop"
if (-not $uopPresent) {
Write-Host " no artLegacyMUL.uop in the copy -- already a mul-only client"
} else {
Move-Item -LiteralPath $uopPath -Destination "$uopPath.disabled" -Force
$uopPresent = $false
Add-Defect 'nouop' 'artLegacyMUL.uop' 'moved aside so art is read from art.mul/artidx.mul' `
'every index-level defect below becomes reachable; without this they are inert'
}
Write-Host ""
} elseif ($uopPresent -and (($Tiers -contains 'corrupt') -or ($Tiers -contains 'customart'))) {
Write-Host " WARNING: artLegacyMUL.uop is present and the nouop tier was not selected."
Write-Host " FileIndex prefers the UOP outright, so the corrupt and customart tiers"
Write-Host " will write to files the library never opens. Add -Tiers nouop."
Write-Host ""
}
# Static ids are offset by 0x4000 in the index; land tiles occupy 0..0x3FFF.
$StaticBase = 0x4000
# Find donor records to copy and victims to corrupt: real, modestly sized statics, so the defects
# are applied to entries that genuinely work today. Picking arbitrary ids risks landing on slots
# that are already empty, where a "defect" would prove nothing.
$donors = New-Object System.Collections.ArrayList
for ($id = 0x1000; $id -lt 0x3000 -and $donors.Count -lt 24; $id++) {
$e = Get-ArtEntry $idx ($id + $StaticBase)
if ($e.Lookup -ge 0 -and $e.Length -gt 200 -and $e.Length -lt 4000 -and ($e.Lookup + $e.Length) -le $artMulLength) {
[void] $donors.Add([pscustomobject]@{ Id = $id; Entry = $e })
}
}
if ($donors.Count -lt 12) { throw "found only $($donors.Count) usable donor statics -- the copy looks wrong" }
Write-Host "using donor statics: $(($donors | Select-Object -First 12 | ForEach-Object { $_.Id }) -join ', ')"
Write-Host ""
$idxDirty = $false
# -- Tier: customart --------------------------------------------------------------------------
if ($Tiers -contains 'customart') {
Write-Host "tier customart"
# A custom-art client does not fill spare slots -- artidx.mul is exactly sized (62,692
# entries here, not one to spare), so adding art means GROWING the index. `Art` builds its
# FileIndex with length 0x10000, so there is room for 2,844 more ids before the library stops
# looking, and the stock ceiling turns out to be nothing more than the size of a file.
$idxCeiling = 0x10000
if ($entryCount -ge $idxCeiling) {
Write-Host " artidx.mul is already at the 0x10000 ceiling -- skipping tier"
} else {
$addCount = 8
$grown = New-Object byte[] (($entryCount + $addCount) * 12)
[Array]::Copy($idx, 0, $grown, 0, $idx.Length)
$idx = $grown
# Read every donor record BEFORE opening the append handle. Append mode takes an
# exclusive lock, so reading the same file while appending to it fails outright.
$buffers = @()
$reader = [System.IO.File]::Open($artMulPath, 'Open', 'Read', 'ReadWrite')
try {
for ($n = 0; $n -lt $addCount; $n++) {
$donor = $donors[$n]
$buffer = New-Object byte[] $donor.Entry.Length
[void] $reader.Seek($donor.Entry.Lookup, 'Begin')
[void] $reader.Read($buffer, 0, $buffer.Length)
$buffers += , $buffer
}
} finally { $reader.Dispose() }
$appendAt = $artMulLength
$stream = [System.IO.File]::Open($artMulPath, 'Append', 'Write')
try {
for ($n = 0; $n -lt $addCount; $n++) {
$buffer = $buffers[$n]
$stream.Write($buffer, 0, $buffer.Length)
$slot = $entryCount + $n
$newId = $slot - $StaticBase
Set-ArtEntry $idx $slot $appendAt $buffer.Length $donors[$n].Entry.Extra
$appendAt += $buffer.Length
Add-Defect 'customart' "static/$newId" `
"custom art appended past the stock ceiling (a copy of static/$($donors[$n].Id))" `
'decodes cleanly; proves the ceiling is read from the file, not from a constant'
}
} finally { $stream.Dispose() }
$entryCount += $addCount
$idxDirty = $true
}
Write-Host ""
}
# -- Tier: corrupt ----------------------------------------------------------------------------
if ($Tiers -contains 'corrupt') {
Write-Host "tier corrupt"
$artMulLength = (Get-Item -LiteralPath $artMulPath).Length
$v = 8 # donors 0..7 may have been consumed by customart as sources; they are unmodified
# 1. A lookup past the end of art.mul. FileIndex.Seek DOES check this one
# (`Stream.Length < e.lookup`), so the library and the validator should agree.
$victim = $donors[$v++].Id
Set-ArtEntry $idx ($victim + $StaticBase) ([int]($artMulLength + 4096)) 512 0
Add-Defect 'corrupt' "static/$victim" 'lookup 4 KB past the end of art.mul' `
'refused by the validator; Seek also catches this one, so no picture'
# 2. A record that STARTS inside the file and ENDS outside it. This is the gap: Seek checks
# the start and never the end, stream.Read returns short, the decoders ignore the count,
# and m_StreamBuffer still holds the PREVIOUS asset. The expected outcome is a picture of
# something else entirely, reported as a success by every count in the library.
$victim = $donors[$v++].Id
Set-ArtEntry $idx ($victim + $StaticBase) ([int]($artMulLength - 64)) 8192 0
Add-Defect 'corrupt' "static/$victim" 'record starts 64 bytes before EOF and declares 8,192' `
'REFUSED BUT DECODED -- the stale-buffer wrong picture'
# 3. A length too small to hold even the 8-byte header.
$victim = $donors[$v++].Id
$donorEntry = $donors[$v - 1].Entry
Set-ArtEntry $idx ($victim + $StaticBase) $donorEntry.Lookup 4 0
Add-Defect 'corrupt' "static/$victim" 'declared length 4 -- smaller than the static header' `
'refused by the validator'
# 4/5/6 rewrite the record BODY, so they need their own bytes rather than an index edit.
# Appended to art.mul and pointed at, which leaves the donor's real record intact.
$stream = [System.IO.File]::Open($artMulPath, 'Append', 'Write')
try {
$appendAt = (Get-Item -LiteralPath $artMulPath).Length
# 4. Absurd dimensions. LoadStatic allocates new Bitmap(width, height) straight from two
# ushorts in the file. 8000x8000 is ~128 MB -- survivable, and the point is made; the
# same field can ask for 65535x65535, which is 8 GB from a two-byte edit.
$victim = $donors[$v++].Id
$rec = New-Object byte[] 2048
[Array]::Copy([BitConverter]::GetBytes([uint16] 8000), 0, $rec, 4, 2)
[Array]::Copy([BitConverter]::GetBytes([uint16] 8000), 0, $rec, 6, 2)
$stream.Write($rec, 0, $rec.Length)
Set-ArtEntry $idx ($victim + $StaticBase) $appendAt $rec.Length 0
$appendAt += $rec.Length
Add-Defect 'corrupt' "static/$victim" 'header declares 8000x8000 (a ~128 MB allocation from two bytes)' `
'refused by the validator; the library would allocate it'
# 5. A row-lookup table pointing outside the record. This is what LoadStatic's unbounded
# read cursor was written to walk off the end of.
$victim = $donors[$v++].Id
$rec = New-Object byte[] 512
[Array]::Copy([BitConverter]::GetBytes([uint16] 32), 0, $rec, 4, 2) # width
[Array]::Copy([BitConverter]::GetBytes([uint16] 32), 0, $rec, 6, 2) # height
for ($row = 0; $row -lt 32; $row++) {
# Each row's offset is added to (height + 4); 60000 puts every row far outside.
[Array]::Copy([BitConverter]::GetBytes([uint16] 60000), 0, $rec, (8 + $row * 2), 2)
}
$stream.Write($rec, 0, $rec.Length)
Set-ArtEntry $idx ($victim + $StaticBase) $appendAt $rec.Length 0
$appendAt += $rec.Length
Add-Defect 'corrupt' "static/$victim" 'row table points 60,000 words outside a 512-byte record' `
'refused by the validator; the library reads adjacent heap'
# 6. A well-formed row table whose run length overruns the record.
$victim = $donors[$v++].Id
$rec = New-Object byte[] 256
[Array]::Copy([BitConverter]::GetBytes([uint16] 16), 0, $rec, 4, 2)
[Array]::Copy([BitConverter]::GetBytes([uint16] 2), 0, $rec, 6, 2)
[Array]::Copy([BitConverter]::GetBytes([uint16] 0), 0, $rec, 8, 2) # row 0 offset
[Array]::Copy([BitConverter]::GetBytes([uint16] 0), 0, $rec, 10, 2) # row 1 offset
$runAt = (2 + 4) * 2 # (height + 4) words
[Array]::Copy([BitConverter]::GetBytes([uint16] 0), 0, $rec, $runAt, 2) # xOffset
[Array]::Copy([BitConverter]::GetBytes([uint16] 16), 0, $rec, ($runAt + 2), 2) # xRun, but
# the record has nowhere near 16 pixels left after this point.
$stream.Write($rec, 0, $rec.Length)
Set-ArtEntry $idx ($victim + $StaticBase) $appendAt 20 0
$appendAt += $rec.Length
Add-Defect 'corrupt' "static/$victim" 'a 16-pixel run declared in a 20-byte record' `
'refused by the validator'
} finally { $stream.Dispose() }
# 7. A land tile shorter than the 2,024 bytes LoadLand reads unconditionally.
$landVictim = 0x0100
$landEntry = Get-ArtEntry $idx $landVictim
if ($landEntry.Lookup -ge 0 -and $landEntry.Length -gt 0) {
Set-ArtEntry $idx $landVictim $landEntry.Lookup 512 0
Add-Defect 'corrupt' "land/$landVictim" 'land record declared 512 bytes; LoadLand always reads 2,024' `
'refused by the validator; the library reads past the buffer'
}
$idxDirty = $true
Write-Host ""
}
if ($idxDirty) {
[System.IO.File]::WriteAllBytes($artIdxPath, $idx)
Write-Host "wrote artidx.mul"
Write-Host ""
}
# -- Tier: verdata ----------------------------------------------------------------------------
if ($Tiers -contains 'verdata') {
Write-Host "tier verdata"
# Layout: int32 count, then count * 5 int32 (file, index, lookup, length, extra), then the
# payloads. `lookup` is an absolute offset into this file.
$entries = New-Object System.Collections.ArrayList
$payloads = New-Object System.Collections.ArrayList
$donorA = $donors[$donors.Count - 1]
$donorB = $donors[$donors.Count - 2]
$artSource = [System.IO.File]::Open($artMulPath, 'Open', 'Read', 'ReadWrite')
try {
$bufferA = New-Object byte[] $donorA.Entry.Length
[void] $artSource.Seek($donorA.Entry.Lookup, 'Begin')
[void] $artSource.Read($bufferA, 0, $bufferA.Length)
} finally { $artSource.Dispose() }
# The victims: ids whose art will now come from verdata.mul rather than art.mul.
$legitVictim = $donors[$donors.Count - 3].Id
$pastEofVictim = $donors[$donors.Count - 4].Id
# A legitimate patch -- the branch working as designed. Without this the tier only proves the
# failure case, and "verdata is broken" and "verdata is never reached" look identical.
[void] $payloads.Add($bufferA)
[void] $entries.Add([pscustomobject]@{
File = 4; Index = ($legitVictim + $StaticBase); Length = $bufferA.Length; Extra = $donorA.Entry.Extra
PayloadIndex = 0; PastEof = $false
})
# The failure case. FileIndex.Seek bounds-checks the mul stream and calls Verdata.Seek with no
# check at all; seeking a FileStream past EOF is legal, the read returns nothing, and the
# shared decode buffer still holds the previous asset.
[void] $entries.Add([pscustomobject]@{
File = 4; Index = ($pastEofVictim + $StaticBase); Length = 900; Extra = 0
PayloadIndex = -1; PastEof = $true
})
# An anim patch, so the tier covers the other file the verdata branch serves. anim.mul is
# verdata file 6; for body < 200 the record index is body*110 + action*5 + direction.
$animBody = 34 # wolf -- decodes on this client, so a patch to it is observable
$animIndex = ($animBody * 110) + (0 * 5) + 1
[void] $entries.Add([pscustomobject]@{
File = 6; Index = $animIndex; Length = 700; Extra = 0
PayloadIndex = -1; PastEof = $true
})
$headerSize = 4 + ($entries.Count * 20)
$offset = $headerSize
foreach ($entry in $entries) {
if ($entry.PayloadIndex -ge 0) {
$entry | Add-Member -NotePropertyName Lookup -NotePropertyValue $offset -Force
$offset += $payloads[$entry.PayloadIndex].Length
}
}
$totalSize = $offset
# Past-EOF lookups are resolved last, because "past the end" is only meaningful once the end
# is known.
foreach ($entry in $entries) {
if ($entry.PastEof) {
$entry | Add-Member -NotePropertyName Lookup -NotePropertyValue ($totalSize + 8192) -Force
}
}
$verdata = New-Object byte[] $totalSize
Write-Int32LE $verdata 0 $entries.Count
$at = 4
foreach ($entry in $entries) {
Write-Int32LE $verdata $at $entry.File
Write-Int32LE $verdata ($at + 4) $entry.Index
Write-Int32LE $verdata ($at + 8) $entry.Lookup
Write-Int32LE $verdata ($at + 12) $entry.Length
Write-Int32LE $verdata ($at + 16) $entry.Extra
$at += 20
}
foreach ($entry in $entries) {
if ($entry.PayloadIndex -ge 0) {
$payload = $payloads[$entry.PayloadIndex]
[Array]::Copy($payload, 0, $verdata, $entry.Lookup, $payload.Length)
}
}
[System.IO.File]::WriteAllBytes((Join-Path $destFull 'verdata.mul'), $verdata)
Add-Defect 'verdata' "static/$legitVictim" `
"legitimately patched to static/$($donorA.Id)'s art via verdata.mul" `
'decodes; the picture must CHANGE, which is how we know the branch ran'
Add-Defect 'verdata' "static/$pastEofVictim" `
'verdata entry whose lookup is 8 KB past the end of verdata.mul' `
'REFUSED BUT DECODED -- Verdata.Seek is not bounds-checked'
Add-Defect 'verdata' "body/$animBody" `
"anim.mul record $animIndex patched to a verdata offset past EOF" `
'the wolf must not silently become another creature'
Write-Host (" wrote verdata.mul: {0} entries, {1:N0} bytes" -f $entries.Count, $totalSize)
Write-Host ""
}
# -- Tier: bodyconv ---------------------------------------------------------------------------
if ($Tiers -contains 'bodyconv') {
Write-Host "tier bodyconv"
$bodyconvPath = Join-Path $destFull 'Bodyconv.def'
if (-not (Test-Path -LiteralPath $bodyconvPath)) {
Write-Host " no Bodyconv.def in the copy -- skipping tier"
} else {
# Columns are tab-separated: original, anim2, anim3, anim4, anim5. -1 means "not in that
# file". BodyConverter.Convert returns the file type of the FIRST column that is not -1,
# and the extractor must take that answer and stop.
$lines = @(
"",
"# Asset Bridge phase 0 -- deliberate defects (tools/patch_client.ps1)",
"1900`t-1`t-1`t-1`t60000",
"1901`t666`t-1`t-1`t-1"
)
Add-Content -LiteralPath $bodyconvPath -Value ($lines -join "`r`n") -Encoding ASCII
Add-Defect 'bodyconv' 'body/1900' 'mapped to anim5 index 60,000, which does not exist' `
'reports nothing -- and must NOT fall back to another anim file'
Add-Defect 'bodyconv' 'body/1901' 'mapped to anim2 index 666, where something unrelated lives' `
'decodes a picture of the WRONG creature -- the spider case, on purpose'
}
Write-Host ""
}
# -- The source must be untouched -------------------------------------------------------------
$tampered = @()
foreach ($name in $before.Keys) {
$path = Join-Path $sourceFull $name
$now = (Get-FileHash -LiteralPath $path -Algorithm SHA256).Hash
if ($now -ne $before[$name]) { $tampered += $name }
}
if ($tampered.Count -gt 0) {
throw "THE SOURCE CLIENT WAS MODIFIED: $($tampered -join ', '). Restore it from the installer before doing anything else."
}
Write-Host "source client verified unchanged ($($before.Count) files hashed before and after)"
# -- Manifest ---------------------------------------------------------------------------------
$manifest = [pscustomobject]@{
built = (Get-Date).ToUniversalTime().ToString('u')
source = $sourceFull
dest = $destFull
tiers = $Tiers
defects = @($script:Defects)
}
$manifestPath = Join-Path $destFull 'patched-client.manifest.json'
$manifest | ConvertTo-Json -Depth 5 | Set-Content -LiteralPath $manifestPath -Encoding utf8
Write-Host ""
Write-Host ("{0} deliberate defects; manifest at {1}" -f $script:Defects.Count, $manifestPath)
Write-Host ""
Write-Host "Point the probe at it by adding to the shard's Config/Bridge.cfg:"
Write-Host ""
Write-Host " AssetProbeClient=$destFull"
Write-Host ""
Write-Host "then, in game or from the rig driver: [assetprobe all patched"

1065
tools/scaffolding/BridgeAssetProbe.cs
View File

File diff suppressed because it is too large Load Diff

532
tools/scaffolding/BridgeMythicCliloc.cs
View File

@@ -1,532 +0,0 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Text;
namespace Server.Custom
{
/// <summary>
/// A reader for the **Mythic compressed** cliloc container, in plain .NET Framework 4.8 C#.
///
/// This is the Asset Bridge's §9 decoder — the ONE decoder Protocol 8 writes rather than
/// calls (docs/link/v8.md §4, §9). ServUO's bundled <c>Ultima.StringList</c> implements only
/// the plain layout and throws <c>Non-negative number required</c> on every modern client's
/// file, which is also why the shard's own <c>VendorSearch.GetItemName</c> is already inert.
///
/// **Provenance.** Ported from UOFiddler's <c>Ultima/Helpers/MythicDecompress.cs</c>,
/// <c>MoveToFront.cs</c> and <c>StringList.TryParse</c> (polserver/UOFiddler). UOFiddler is
/// released under the **Beerware** licence, so carrying its algorithm into this
/// GPL-3.0-or-later tree is clean — see v8.md §9.
///
/// **What the port had to change**, and why the differences are not cosmetic:
///
/// * UOFiddler targets net10.0 and its implementation is written in <c>Span&lt;T&gt;</c>,
/// <c>stackalloc</c>, <c>ArrayPool</c> and <c>BinaryPrimitives</c>. ServUO compiles the
/// overlay against net48 with no package feed, so all of that becomes plain arrays.
/// * Every read of the compressed payload is **bounds-checked here and is not there**.
/// Upstream indexes <c>input[m + 1024]</c> and <c>input[firstVal + 1024]</c> with
/// offsets derived from the file's own frequency header, inside a
/// <c>try { } catch (Exception) { return false; }</c>. That is adequate for a desktop
/// tool and is not adequate for us: this runs inside a live shard, and a corrupt or
/// hostile Cliloc.enu must produce a refusal, not an exception unwinding through the
/// bridge. Every such index is tested before use and returns <c>false</c> instead.
///
/// Phase 0 uses this from <see cref="BridgeAssetProbe"/> to prove the port reproduces
/// UOFiddler's own output exactly. **Phase 2 promotes this file into
/// <c>overlay/Scripts/Custom/Bridge/</c>** — it lives in scaffolding only for as long as it
/// is a spike.
/// </summary>
public static class BridgeMythicCliloc
{
/// <summary>The first DWORD of a compressed file is the decompressed length, XORed with this.</summary>
private const uint HeaderXorKey = 0x8E2C9A3D;
/// <summary>256 little-endian int32 symbol frequencies precede the coded payload.</summary>
private const int FrequencyHeaderSize = 1024;
/// <summary>One decoded cliloc row. Mirrors <c>Ultima.StringEntry</c>'s three fields.</summary>
public struct Entry
{
public int Number;
public byte Flag;
public string Text;
public Entry(int number, byte flag, string text)
{
Number = number;
Flag = flag;
Text = text;
}
}
// ── Container detection ──────────────────────────────────────────────────────────────
/// <summary>
/// True when the file looks like the Mythic container. The marker is the high byte of
/// the first DWORD being <c>0x8E</c> — which is not a magic number in the file so much
/// as a consequence of <see cref="HeaderXorKey"/>: a plausible decompressed length is
/// small enough that its top byte is zero, so the XOR leaves 0x8E showing.
/// </summary>
public static bool LooksCompressed(byte[] buffer)
{
return buffer != null && buffer.Length >= 4 && buffer[3] == 0x8E;
}
// ── The public entry point ───────────────────────────────────────────────────────────
/// <summary>
/// Reads a cliloc file, compressed or plain, and returns its entries.
///
/// Tries the layout the header suggests first and the other one second — the same
/// fallback UOFiddler performs, and the reason an already-converted file passes
/// straight through. <paramref name="warning"/> is non-null when a layout parsed
/// *partially*: that is the case a caller must surface rather than swallow, because a
/// quietly short table is the failure mode the website's importer refuses.
/// </summary>
public static bool TryLoadFile(string path, out List<Entry> entries, out string warning, out string error)
{
entries = null;
warning = null;
error = null;
byte[] buffer;
try
{
buffer = File.ReadAllBytes(path);
}
catch (Exception e)
{
error = "cannot read " + path + ": " + e.Message;
return false;
}
return TryLoad(buffer, out entries, out warning, out error);
}
/// <summary>Reads an in-memory cliloc file. See <see cref="TryLoadFile"/>.</summary>
public static bool TryLoad(byte[] buffer, out List<Entry> entries, out string warning, out string error)
{
entries = null;
warning = null;
error = null;
bool compressedFirst = LooksCompressed(buffer);
List<Entry> primary;
string primaryError;
bool primaryComplete;
if (TryParse(buffer, compressedFirst, out primary, out primaryComplete, out primaryError) && primaryComplete)
{
entries = primary;
return true;
}
List<Entry> fallback;
string fallbackError;
bool fallbackComplete;
if (TryParse(buffer, !compressedFirst, out fallback, out fallbackComplete, out fallbackError) && fallbackComplete)
{
entries = fallback;
return true;
}
// Neither layout parsed to the end. Take whichever salvaged more rows and say so.
int primaryCount = primary == null ? 0 : primary.Count;
int fallbackCount = fallback == null ? 0 : fallback.Count;
if (primaryCount == 0 && fallbackCount == 0)
{
error = "as " + Label(compressedFirst) + ": " + primaryError
+ "; as " + Label(!compressedFirst) + ": " + fallbackError;
return false;
}
if (primaryCount >= fallbackCount)
{
entries = primary;
warning = "parsed partially as " + Label(compressedFirst) + ": " + primaryError
+ " (" + primaryCount + " entries salvaged)";
}
else
{
entries = fallback;
warning = "parsed partially as " + Label(!compressedFirst) + ": " + fallbackError
+ " (" + fallbackCount + " entries salvaged)";
}
return true;
}
private static string Label(bool compressed)
{
return compressed ? "compressed" : "uncompressed";
}
// ── Record layout ────────────────────────────────────────────────────────────────────
/// <summary>
/// Walks the plain record layout: a 4-byte and a 2-byte header, then repeating
/// [int32 number][byte flag][uint16 length][length bytes of UTF-8].
///
/// <paramref name="complete"/> distinguishes "parsed to the end of the file" from
/// "stopped early but salvaged rows", which is the distinction the caller needs and
/// an exception would destroy.
/// </summary>
private static bool TryParse(byte[] buffer, bool decompress, out List<Entry> entries, out bool complete, out string error)
{
entries = new List<Entry>();
complete = false;
error = null;
byte[] data;
if (decompress)
{
if (!TryDecompress(buffer, out data, out error))
return false;
}
else
{
data = buffer;
}
if (data.Length < 6)
{
error = "file is " + data.Length + " bytes, smaller than the 6-byte header";
return false;
}
int cursor = 6; // int32 version marker + int16 language marker
int lastNumber = -1;
while (cursor < data.Length)
{
int entryStart = cursor;
int remaining = data.Length - cursor;
if (remaining < 7)
{
error = "unexpected " + remaining + " trailing byte(s) at 0x" + entryStart.ToString("X")
+ " after entry #" + lastNumber + "; an entry header needs 7";
return true;
}
int number = ReadInt32(data, cursor);
byte flag = data[cursor + 4];
// Deliberately UNSIGNED. Read as Int16, a string of 32768 bytes or more comes back
// negative and corrupts every record after it.
int length = data[cursor + 5] | (data[cursor + 6] << 8);
cursor += 7;
if (length > data.Length - cursor)
{
error = "entry #" + number + " at 0x" + entryStart.ToString("X") + " declares length "
+ length + " but only " + (data.Length - cursor) + " byte(s) remain (parsed "
+ entries.Count + " so far)";
return true;
}
string text;
try
{
text = Encoding.UTF8.GetString(data, cursor, length);
}
catch (Exception e)
{
error = "entry #" + number + " at 0x" + entryStart.ToString("X") + " has " + length
+ " body bytes that are not valid UTF-8: " + e.Message;
return true;
}
cursor += length;
entries.Add(new Entry(number, flag, text));
lastNumber = number;
}
complete = true;
return true;
}
// ── Mythic stage 1: the XOR header and the move-to-front code ────────────────────────
/// <summary>
/// Reads the obfuscated decompressed length from the first DWORD. Public so a caller
/// can size a buffer before committing to the decode.
/// </summary>
public static uint PeekDecompressedLength(byte[] source)
{
if (source == null || source.Length < 4)
return 0;
return ReadUInt32(source, 0) ^ HeaderXorKey;
}
/// <summary>
/// Decompresses the Mythic container: strip the 4-byte length header, undo the
/// move-to-front coding, then run stage 2.
/// </summary>
public static bool TryDecompress(byte[] source, out byte[] output, out string error)
{
output = null;
error = null;
if (source == null || source.Length < 4)
{
error = "payload shorter than the 4-byte length header";
return false;
}
uint dataLength = ReadUInt32(source, 0) ^ HeaderXorKey;
// A wrong guess about the container makes this astronomically large, which is the
// cheapest possible rejection and must happen before any allocation.
if (dataLength == 0 || dataLength > int.MaxValue)
{
error = "implausible decompressed length " + dataLength + " — not the compressed layout";
return false;
}
var mtf = new byte[source.Length - 4];
MoveToFrontDecode(source, 4, mtf);
var destination = new byte[(int)dataLength];
int written;
if (!TryInternalDecompress(mtf, destination, out written, out error))
return false;
if (written != (int)dataLength)
{
error = "decompressed " + written + " bytes, header declared " + dataLength;
return false;
}
output = destination;
return true;
}
/// <summary>
/// Move-to-front decode. Each input byte is an index into a 256-symbol table; the
/// symbol found there is emitted and moved to the front.
/// </summary>
private static void MoveToFrontDecode(byte[] input, int offset, byte[] output)
{
var symbols = new byte[256];
for (int i = 0; i < 256; i++)
symbols[i] = (byte)i;
for (int i = 0; i < output.Length; i++)
{
int index = input[offset + i];
byte symbol = symbols[index];
output[i] = symbol;
for (int j = index; j > 0; j--)
symbols[j] = symbols[j - 1];
symbols[0] = symbol;
}
}
// ── Mythic stage 2 ───────────────────────────────────────────────────────────────────
/// <summary>
/// Turns the MTF-decoded payload back into the original bytes.
///
/// The payload is a 1024-byte frequency header (256 little-endian int32 symbol counts)
/// followed by the coded stream. The counts partition the stream into one run per
/// symbol; <c>cursor[]</c> holds each run's read position and <c>limit[]</c> its end,
/// and the walk emits a symbol, advances that symbol's run, and re-orders the symbol
/// table by the index it reads.
///
/// Every index derived from file content is checked. Upstream's equivalent is wrapped
/// in a blanket catch; here a malformed file is a <c>false</c> with a reason.
/// </summary>
private static bool TryInternalDecompress(byte[] input, byte[] destination, out int written, out string error)
{
written = 0;
error = null;
if (input.Length < FrequencyHeaderSize)
{
error = "payload (" + input.Length + " bytes) is smaller than the 1024-byte frequency header";
return false;
}
var counts = new int[256]; // symbol → number of occurrences
var cursor = new int[256]; // symbol → next unread position in its run
var limit = new int[256]; // symbol → one past the end of its run
int sum = 0;
for (int i = 0; i < 256; i++)
{
counts[i] = ReadInt32(input, i * 4);
if (counts[i] < 0)
{
error = "frequency header declares a negative count for symbol " + i;
return false;
}
sum += counts[i];
if (sum < 0)
{
error = "frequency header sums past int range at symbol " + i;
return false;
}
}
if (sum == 0)
{
written = 0;
return true;
}
if (destination.Length < sum)
{
error = "destination holds " + destination.Length + " bytes, payload needs " + sum;
return false;
}
int nonZeroCount = 0;
for (int i = 0; i < 256; i++)
{
if (counts[i] != 0)
nonZeroCount++;
}
// The coded stream must be long enough to hold one index per emitted byte.
if (input.Length - FrequencyHeaderSize < sum)
{
error = "coded stream holds " + (input.Length - FrequencyHeaderSize) + " bytes, frequency header claims " + sum;
return false;
}
var order = new byte[256];
FrequencyOrder(counts, order);
var symbolTable = new byte[256];
for (int i = 0; i < 256; i++)
symbolTable[i] = (byte)i;
for (int i = 0, m = 0; i < nonZeroCount; ++i)
{
byte symbol = order[i];
// m indexes the coded stream and comes from the file's own counts.
if (m < 0 || m >= input.Length - FrequencyHeaderSize)
{
error = "run table for symbol " + symbol + " starts at " + m + ", past the coded stream";
return false;
}
symbolTable[input[m + FrequencyHeaderSize]] = symbol;
cursor[symbol] = m + 1;
m += counts[symbol];
limit[symbol] = m;
}
byte val = symbolTable[0];
int count = 0;
int liveSymbols = nonZeroCount;
do
{
destination[count] = val;
if (cursor[val] < limit[val])
{
int at = cursor[val] + FrequencyHeaderSize;
if (at < FrequencyHeaderSize || at >= input.Length)
{
error = "run for symbol " + val + " reads at " + at + ", past the " + input.Length + "-byte payload";
return false;
}
byte index = input[at];
cursor[val]++;
if (index != 0)
{
ShiftLeft(symbolTable, index);
symbolTable[index] = val;
val = symbolTable[0];
}
}
else if (liveSymbols-- > 0)
{
ShiftLeft(symbolTable, liveSymbols);
val = symbolTable[0];
}
count++;
}
while (count < sum);
written = sum;
return true;
}
/// <summary>
/// Orders symbols by descending frequency: repeatedly take the largest remaining count
/// and record its symbol. Ties go to the lower symbol, because the scan keeps the first
/// strictly-greater value — matching upstream, and the tie-break is load-bearing.
/// </summary>
private static void FrequencyOrder(int[] counts, byte[] output)
{
var tmp = new int[256];
Array.Copy(counts, tmp, 256);
for (int i = 0; i < 256; i++)
{
int best = 0;
byte index = 0;
for (int j = 0; j < 256; j++)
{
if (tmp[j] > best)
{
index = (byte)j;
best = tmp[j];
}
}
if (best == 0)
break;
output[i] = index;
tmp[index] = 0;
}
}
/// <summary>Shifts <c>[1..element]</c> down one slot, dropping element 0.</summary>
private static void ShiftLeft(byte[] input, int element)
{
for (int i = 0; i < element; ++i)
input[i] = input[i + 1];
}
// ── Little-endian readers (BinaryPrimitives is not available on net48) ───────────────
private static int ReadInt32(byte[] b, int at)
{
return b[at] | (b[at + 1] << 8) | (b[at + 2] << 16) | (b[at + 3] << 24);
}
private static uint ReadUInt32(byte[] b, int at)
{
return (uint)(b[at] | (b[at + 1] << 8) | (b[at + 2] << 16) | (b[at + 3] << 24));
}
}
}

8
tools/scaffolding/BridgeRigDriver.cs
View File

@@ -155,14 +155,6 @@ namespace Server.Custom
case "partprobe":
BridgeParticipationProbe.Run(null, Arg(parts, 1), Int(Arg(parts, 2)), Int(Arg(parts, 3)));
break;
// Asset Bridge phase 0. Here for the same reason as partprobe, and for one more:
// the point of that spike is comparing the STOCK client's answers with a patched
// client's, and `AssetProbeOnStart` can only ever run whichever one the config
// names. Driving it from here runs both against a single boot, so a difference
// between them cannot be a difference between two shard processes.
case "assetprobe":
BridgeAssetProbe.Begin(null, Arg(parts, 1) ?? "all", Arg(parts, 2));
break;
// Phase 12a. `world.despawn` answering `gone` rather than `removed` is the
// path a player takes every time they kill an event creature, and it is the one
// outcome the rig cannot reach by asking the bridge: every bridge verb that

128
tools/scaffolding/README.md
View File

@@ -14,12 +14,10 @@ These two scripts produced the measured budget in [PLAN.md](https://gitea.whitlo
| `BridgeCrierProbe.cs` | `Scripts/Custom/BridgeCrierProbe.cs` | Logs the global town-crier entry list every 3s so `towncrier.add` / `remove` can be seen landing in game state. Flag: `CrierProbeOnStart`. |
| `BridgeVendorSaleProbe.cs` | `Scripts/Custom/BridgeVendorSaleProbe.cs` | Fires `PlayerVendorSale` (Phase 7) with real seeded-vendor data so `vendor.sale` can be verified without a live buy. Requires the Phase 7 patches applied. Flag: `VendorSaleProbeOnStart`. |
| `BridgeDemoDress.cs` | `Scripts/Custom/BridgeDemoDress.cs` | Renames a seeded world so it is presentable in a screenshot: shop signs, vendor and character names, house signs. Also stages a few condemned houses back into IDOC, and sets a known password on `seed_000` so a character can be logged in. Flags: `DemoDressOnStart`, `DemoDressPassword`. In game: `[demodress`. |
| `BridgeRigDriver.cs` | `Scripts/Custom/BridgeRigDriver.cs` | Drives the shard from OUTSIDE the game, one verb per line in `Config/rigcmd.txt`, which the driver polls and truncates. Written for the engagement Phase 11b acceptance walk, where each step's assertion is what happened BETWEEN two steps, so the steps have to be separated by the observer rather than by a hard-coded delay -- and ServUO's console takes a fixed verb set (`Scripts/Misc/ConsoleCommands.cs`), so `[p5probe` cannot be typed at a headless shard at all. Verbs: `decaylist`, `decay`, `vendorlist`, `vendorfunds`, `citylist`, `governor`, `election`, `activate`, `password`, `configset`, `configread`, `partprobe`, `assetprobe`, `save`, `shutdown`. Flag: `RigDriverEnabled`. `configset` exists because **`Config.Get` is written by exactly ONE caller in the whole of ServUO 57.4** (`Server/ScriptCompiler.cs`): no in-game command, gump or console verb writes a config key, so on a stock shard a GM cannot drift a configuration lease even deliberately, and a lease's compare-and-set restore would have no way to be proved. `configread` reads a key back through `Config.Get` long after every type initialiser has run, which is how a key that TOOK is told from one that only appeared to. **Sets passwords, writes live config and mutates the world.** |
| `BridgeRigDriver.cs` | `Scripts/Custom/BridgeRigDriver.cs` | Drives the shard from OUTSIDE the game, one verb per line in `Config/rigcmd.txt`, which the driver polls and truncates. Written for the engagement Phase 11b acceptance walk, where each step's assertion is what happened BETWEEN two steps, so the steps have to be separated by the observer rather than by a hard-coded delay -- and ServUO's console takes a fixed verb set (`Scripts/Misc/ConsoleCommands.cs`), so `[p5probe` cannot be typed at a headless shard at all. Verbs: `decaylist`, `decay`, `vendorlist`, `vendorfunds`, `citylist`, `governor`, `election`, `activate`, `password`, `configset`, `configread`, `partprobe`, `save`, `shutdown`. Flag: `RigDriverEnabled`. `configset` exists because **`Config.Get` is written by exactly ONE caller in the whole of ServUO 57.4** (`Server/ScriptCompiler.cs`): no in-game command, gump or console verb writes a config key, so on a stock shard a GM cannot drift a configuration lease even deliberately, and a lease's compare-and-set restore would have no way to be proved. `configread` reads a key back through `Config.Get` long after every type initialiser has run, which is how a key that TOOK is told from one that only appeared to. **Sets passwords, writes live config and mutates the world.** |
| `BridgeProtocol5Probe.cs` | `Scripts/Custom/BridgeProtocol5Probe.cs` | Drives all three Protocol 5 enrichments so their frames can be observed: walks one house Fairly -> Greatly -> IDOC (the PAIR is the assertion -- `estimatedCollapse` must appear only on the IDOC frame), reports each player vendor's fee state straight off the `PlayerVendor` so the emitted `fees` block can be checked against the shard's own numbers, and fires `EventSink.AccountLogin`. Flags: `Protocol5ProbeOnStart`, `Protocol5ProbeAccount`, `Protocol5ProbePassword`. In game: `[p5probe`. **Sets a password on the named account.** |
| `BridgeProtocol6Probe.cs` | `Scripts/Custom/BridgeProtocol6Probe.cs` | Spawns a real champion boss through the shard's own `SpawnChampion()`, waits two champ sweeps so the boss is attributed to its altar, registers unequal damage from two seeded players and kills it -- so `champ.boss.killed` can be observed with a real damage table. **The wait is the assertion**: without it the kill still emits, but with no `serial`/`type`/`level`, which is the documented fallback rather than the case being tested. The altar is placed inside a NAMED region on purpose (see below). Flag: `Protocol6ProbeOnStart`. In game: `[p6probe`. **Spawns and kills a champion boss; rig only.** Protocol 6's other half, the idempotency key, needs no probe -- it is driven from outside with two identical POSTs to the sidecar. |
| `BridgeParticipationProbe.cs` | `Scripts/Custom/BridgeParticipationProbe.cs` | Produces real kill credit inside a participation area with no game client: moves two player mobiles to the venue, spawns a creature there, damages it unequally from both and kills it. **Presence is the half it cannot drive** -- the sweep credits players with a live `NetState`, which is the correct test and not one a probe should loosen, so presence accrual needs a real login. In game: `[partprobe <map> <x> <y>`; from a headless rig, through `BridgeRigDriver`'s `partprobe` verb (the two ship together for that reason). **Moves players and spawns and kills a creature; rig only.** |
| `BridgeAssetProbe.cs` | `Scripts/Custom/BridgeAssetProbe.cs` | **Asset Bridge phase 0** (docs/link/v8.md §16). Drives ServUO's vendored `Ultima` decoders from inside a running shard against a deliberately patched client, and compares every answer with what a pre-flight validator says about the index entry *before* the call. The interesting column is not the error count, it is **WRONG PICTURES** -- records the validator rejects and the library renders anyway. Sweeps statics, land, all 2,048 bodies, the player-character bodies from `Race.AllRaces`, and the ported Mythic cliloc reader against UOFiddler's own output. In game / from `BridgeRigDriver`: `[assetprobe [section] [stock|patched]`. Flag: `AssetProbeOnStart`. **Its `gump` section deliberately kills the shard** and is never part of `all`. |
| `BridgeMythicCliloc.cs` | `Scripts/Custom/BridgeMythicCliloc.cs` | The §9 reader for the **Mythic compressed** cliloc container -- the one decoder Protocol 8 writes rather than calls. Ported from UOFiddler (Beerware) into net48 C# with every file-derived index bounds-checked, which upstream's blanket `catch` does not do. Reproduces UOFiddler's 123,490-entry table exactly. **Phase 2 promotes this file into `overlay/`**; it is scaffolding only for as long as it is a spike. |
## Deploy overwrites Bridge.cfg
@@ -191,127 +189,3 @@ value that lies, printed next to a frame that disagrees with it.
Emitting from a named region is therefore the test. At a dungeon altar the field is legitimately
absent and the probe proves nothing about it.
## What phase 0 found
`BridgeAssetProbe` exists because [v8.md](https://gitea.whitlocktech.com/RunicGateway/docs/src/branch/main/link/v8.md) §4 chose to **call** ServUO's vendored `Ultima` rather than reimplement it, and the evidence for that choice was a PowerShell probe against a stock client — neither the process nor the client the extractor will actually run in. These are its results, from inside a running ServUO 57.4 against this machine's client, and against a copy broken in 21 catalogued ways by `tools/patch_client.ps1`.
### The UOP wins outright, and it took a whole run to notice
`FileIndex`'s UOP constructor ends with a bare `MulPath = uopPath`. **When `artLegacyMUL.uop` is present it wins, and `art.mul` / `artidx.mul` are never opened at all.** Every current client ships the UOP, so:
- A validator that bounds an index offset against `art.mul` while the index holds UOP offsets is not approximate, it is nonsense. The first run of this probe refused **34,299 perfectly good statics** for "declaring 10533x2085" — and every one of those refusals looked like a real finding. `BridgeAssetValidator.ArtDataPath()` now mirrors `FileIndex`'s own resolution order, and phase 1 must too.
- A custom-art shard that adds graphics to `art.mul` while the UOP is still in place **gets nothing**, silently. That is an operator trap rather than a bug in this protocol, but the extractor is where it will be noticed.
- The `corrupt` and `customart` tiers of `patch_client.ps1` therefore need its `nouop` tier to mean anything at all. Without it they report that they applied, and change nothing.
### 22,102 wrong pictures on a stock, unmodified client
The counts that matter, `assetprobe all stock`:
```
statics 0..65535 ok 39,189 WRONG PICTURES (empty record) 9,962 threw 16,385
land 0..16383 ok 4,244 WRONG PICTURES (empty record) 12,140
```
Those 22,102 ids have an index entry of `lookup 0, length 0` — **no record at all**. `FileIndex.Seek` treats that as a hit (it rejects `lookup < 0` and `length < 0`, and zero is neither), hands back the stream, and `LoadStatic` decodes `length` = 0 bytes into `m_StreamBuffer` — which is **reused, only ever grown, and filled by a `stream.Read` whose return value is discarded**. So the id renders whatever the previously-decoded asset left in the buffer.
**It is specific to the UOP path.** Run the same sweep against the mul path and those ids come back empty and honest, because `artidx.mul` stores `-1` for an absent record while unmapped UOP slots are simply zeroed structs. That is also why the earlier PowerShell probe counted 32,766 of these as "ok": they decode, they raise nothing, and no success count can tell them from art.
A bulk import that trusted the library would have written 22,102 duplicate images into the site under ids that have no art. This one measurement is the argument for validate-before-calling.
### Every deliberate defect was caught by the validator and rendered by the library
`assetprobe all patched`, against the 21-defect client:
```
statics ok 39,190 absent 9,954 refused 1 WRONG PICTURES (bad record) 6 threw 16,385
land ok 4,243 absent 12,140 WRONG PICTURES (bad record) 1
```
| id | the defect | what the library did |
|---|---|---|
| `static/4104` | lookup 4 KB past the end of `art.mul` | returns nothing — `Seek` does check the record's **start** |
| `static/4105` | starts 64 bytes before EOF, declares 8,192 | **renders the previous asset** — `Seek` never checks the record's **end** |
| `static/4108` | declared length 4, smaller than the header | renders something |
| `static/4109` | header declares 8000x8000 | **allocates it** — a ~128 MB bitmap from two bytes in a file, and the same field can ask for 65535×65535 |
| `static/4111` | row table points 60,000 words outside a 512-byte record | renders — `LoadStatic`'s two guards bound the *write* into the bitmap, and nothing bounds the *read* |
| `static/4112` | a 16-pixel run declared in a 20-byte record | renders |
| `static/4131` | verdata entry whose lookup is past verdata.mul's own end | renders — **`Verdata.Seek` has no bounds check whatsoever** |
| `land/256` | 512-byte land record | renders — `LoadLand` reads a fixed 2,024 bytes whatever the length says |
Seven of the eight produce a confident, wrong picture and raise nothing anywhere.
The validator refused all eight, and refused **nothing** on the stock client across 49,151 statics and 16,384 land tiles. That second number is the one that matters: a checker that refuses real art is worse than no checker, so "zero false refusals on a clean client" is what makes validate-before-calling more than a hopeful phrase.
The eight `customart` ids appended past the stock ceiling all decode cleanly, which is that tier's whole point — the ceiling is a property of a file, not a constant anyone should write down.
### Two more ways to get a wrong answer out of an id that has no art
- **`Art.GetStatic(id, false)` throws `IndexOutOfRangeException` for `id >= 49,152`** rather than returning null — 16,385 of them in a full sweep.
- **`Art.GetStatic(id)` with the default `checkmaxid: true` is worse**: `GetLegalItemID` maps an out-of-range id to **0**, so the call returns **item 0's picture**. An exception is recoverable; a picture of the wrong item is not even detectable.
So the extractor takes its id ceiling from the index it opened, and passes `checkmaxid: false` so an overrun is loud rather than plausible.
### The gump crash reproduces in-process, and nothing catches it
`assetprobe gump` called `Ultima.Gumps.GetGump(2)` once. **The ServUO process disappeared** — no exception line in the report, no `catch` reached, no shutdown, nothing in the console. The report ends mid-section, and `checkpoint.txt` reading `gump 2` is the entire record of what happened. That is exactly why the checkpoint is written *before* the call and flushed.
`AccessViolationException` is a corrupted-state exception and .NET Framework 4.8 does not deliver it to ordinary handlers, so **there is no in-process defence** — on a live shard this is a crash with players on it. "Nothing calls `Ultima.Gumps`" is a safety rule, and phase 0's job was to make sure that sentence had been earned rather than assumed. It has.
### The cliloc port is byte-identical to UOFiddler
```
123,490 entries in 218 ms (55,986 blank, 67,504 would be stored)
vs UOFiddler: 123,490 identical, 0 differ, 0 only ours, 0 only theirs
```
§9 is proven: the shard can produce the whole table with no UOFiddler installed, no `dotnet build`, and no 5 MB file copied to a server.
The reference is what makes this a test rather than a demonstration. A subtly wrong inverse-BWT coder still produces a plausible table — mostly-right strings with a few mangled ones is the *expected* shape of a bug in this algorithm, and a row count alone would sail past it.
Note the blank count is **55,986**, not the 55,994 recorded from the manual pipeline. The difference is eight whitespace-only entries, blank to a `trim()` and not to `IsNullOrEmpty` — a definition rather than a defect, but exactly the sort of eight-row drift that gets investigated as one.
### What phase 0 did not cover, and phase 1 must
**The animation path has no validator.** The patched client's verdata entry for body 34 points past verdata.mul's end and the wolf still "decoded" — counted among the 1,144 successes, silently rendering something else, with nothing in the report to say so. `GetAnimation` also allocates `new int[frameCount]` straight from a file-supplied int. Everything above about statics applies here and none of it is implemented yet.
The deliberate `Bodyconv.def` mis-mappings (bodies 1900 and 1901) produced **nothing** rather than a wrong creature on this client, so they did not reproduce the spider. The gargoyle rows remain the real evidence for the never-sweep-file-types rule: 666, 667, 694 and 695 report nothing, and nothing is the correct answer.
### Reference: the rest of the run
```
bodies 0..2047, direction 1 decoded 1,144 empty 904 faulted 0
by file type: 1=1222, 2=140, 3=244, 4=150, 5=292
player bodies (Race.AllRaces, direction 0) 6 decoded, 6 absent, of 12
Human 400 / 401 decode; ghosts 402 / 403 absent
Elf 605 / 606 / 607 / 608 all decode
Gargoyle 666 / 667 / 694 / 695 all absent
```
Two details worth keeping. The body counts reproduce the PowerShell probe **exactly**, from a different process against the same files, which is what makes the two runs comparable at all. And the gargoyle *ghost* bodies resolve to file type **1**, not 5 like the living gargoyle bodies — so "the gargoyle is an anim5 problem" is not quite the shape of it.
## Building the patched client
```powershell
.\tools\patch_client.ps1 -Dest D:\uo-patched-client
```
Copies a client (~3.5 GB) and breaks the copy in five catalogued tiers — `nouop`, `verdata`, `customart`, `corrupt`, `bodyconv`. **It never writes to the source**: every file it touches is hashed in the source before and after, and a changed hash aborts the run. Each defect is recorded in `patched-client.manifest.json` beside the copy, which is what makes a nonzero WRONG PICTURES count readable as "the tier worked" instead of "something broke".
Then point the shard at it and drive the probe:
```ini
RigDriverEnabled=true
AssetProbeClient=D:\uo-patched-client
AssetProbeClilocRef=<a clilocs.tsv from website/server/tools/cliloc-export --tsv>
```
```
assetprobe all stock # the baseline: the validator must refuse nothing here
assetprobe all patched # the experiment
```
Run both against **one boot**, through `rigcmd.txt`, so a difference between them cannot be a difference between two shard processes. Without `AssetProbeClilocRef` the cliloc section reports a row count, which proves nothing about the strings.
**The copy is EA's client art.** It stays on the machine that made it, exactly like every other extraction in this project, and is never committed.