feat(asset-bridge): item and land art on demand, hued where the files are (Phase 5)

The body catalogue is a set; this is not. This client addresses 49,152 static
ids and has art for 39,189 of them, plus 4,244 land tiles of 16,384 -- and hues
multiply the statics by three thousand. So there is no manifest and no scan:
`assets.fetch` grows two more families (`static`, `land`) and answers the keys
the website's own data names.

`assets.fetch` becomes shared plumbing. BridgeAssets now owns the command, does
the reqId/consent/key-ceiling checks once, derives the family from the keys
themselves (§5 made the key the address; a request that also named its family
would have two places to be wrong and one of them silent) and dispatches to the
reader that registered it. A batch must be of one family, because the reply
carries one `catalog` id. `assets.sources` gains `families` -- additive, so the
protocol stays 8, and EXTRACTOR_VERSION stays 2 because no existing key's bytes
change.

Two traps, both in §4.5's family -- a confident, plausible, wrong picture:

- `Art.GetStatic` memoises into a static Bitmap[0xFFFF] and hands back the SAME
  instance, while `Hue.ApplyTo` repaints in place. Hue a static once and the
  plain key comes back hued from then on, and the next hue stacks on the last.
  Measured on this client before the fix. `Files.CacheData` is now off for the
  life of the process; `TryHue` re-checks it and refuses rather than risk it,
  and the same flag decides whether a bitmap is ours to dispose.

- `PartialHue` decides whether a hue repaints every pixel or only the grey ones,
  per item id, out of `tiledata.mul` -- 13,259 of 65,536 ids on this client.
  Item 597 is a wooden screen with painted flowers: one mode reddens the
  flowers, the other the whole screen. Both decode. The first cut of this reader
  bound `TileData` to ServUO's OWN `Server.TileData` (the enclosing namespace
  beats `using Ultima;`, and it has a PartialHue flag too), which compiled and
  refused every hued key at runtime. Every such type is spelled `Ultima.` now.

Land takes no hue segment: the mode that decides how is an item flag and land
has no equivalent, so `land/3/h33` is refused rather than guessed. `h0` is not a
key either -- hue 0 means "not hued", and the plain key already names it.

Measured through the reader over the whole range: 39,189 statics and 4,244 land
tiles served, and the only refusals are the 9,963 + 12,140 empty index slots
§4.5 predicted. Nothing that carries art is refused.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
This commit is contained in:
2026-09-11 05:58:45 -05:00
parent 452be696df
commit 1be1f24562
5 changed files with 1187 additions and 139 deletions

View File

@@ -0,0 +1,842 @@
using System;
using System.Collections.Generic;
using System.Drawing;
using System.Globalization;
using System.IO;
using System.Text;
using Ultima;
namespace Server.Custom.Bridge
{
/// <summary>
/// **Item and land art, on demand** (docs/link/v8.md §5, §11 — protocol 8, phase 5).
///
/// The body catalogue is a *set*: 1,022 sprites, enumerated, hashed and imported in one
/// pass because a bestiary needs all of them. This is the opposite shape. This client
/// addresses **49,152 static ids** and has real art for **39,189** of them, plus 4,244 land
/// tiles of 16,384 — and then there are hues, which multiply the statics by three thousand.
/// Nothing enumerates that. So there is no manifest here and no scan: the website asks for
/// the handful of keys its own data actually names, and this answers them.
///
/// (49,152 rather than the 81,884 entries `artidx.mul` declares: <c>FileIndex</c> sizes its
/// table from the **length argument it is constructed with**, `0x10000`, not from the idx
/// file — so the addressable range is `0x10000 - 0x4000`. Reading the ceiling off the file
/// instead would invent 16,348 ids, every one of them answered out of an array nobody
/// bounded.)
///
/// ── **The keys** (§5) ──
///
/// <code>
/// static/3922 one item graphic, as the client files hold it
/// static/3922/h33 the same graphic with hue 33 applied
/// land/3 one land tile
/// </code>
///
/// ── **Why the hue is applied HERE and not on the website** ──
///
/// Because it cannot be applied correctly anywhere else, and the incorrect version looks
/// fine.
///
/// A hue is not a tint. It is a 32-entry colour ramp out of `hues.mul` indexed by a
/// pixel's own red channel — and whether it replaces *every* pixel or only the grey ones
/// is decided by the <c>PartialHue</c> flag in <c>tiledata.mul</c>, per item id. On this
/// client **13,259 of 65,536 item ids carry that flag**. Get it wrong on one of them and
/// you do not get an error: item 597 is a wooden screen with painted flowers, and hued red
/// the right way the flowers turn red, the wrong way the whole screen turns red. Both
/// decode. Both are the right size. One is wrong.
///
/// The website has neither file and never will — shipping `Hues.mul` semantics and a
/// 65,536-row flag table into Node to answer a question the shard can answer for free is
/// the same trade §2.1 already refused. So hue is part of the key, and the key is resolved
/// where the files are.
///
/// ── **The trap this phase existed to find** ──
///
/// <c>Art.GetStatic</c> memoises into a static <c>Bitmap[0xFFFF]</c> and returns **the same
/// instance** every time; <c>Hue.ApplyTo</c> repaints a bitmap **in place**. Hue a static
/// once and the library's own copy is hued from then on — the plain key comes back hued,
/// and the next hue stacks on the last. It is §4.5's failure mode (a confident, plausible,
/// wrong picture that every success count agrees with) reached through a door §4.5 never
/// looked at, because phase 0 was auditing *records* and this is the library's *cache*.
///
/// <see cref="BridgeAssets.Initialize"/> turns <c>Files.CacheData</c> off for the life of
/// the process, which makes every bitmap this file receives its own. That invariant is
/// load-bearing enough that <see cref="Render"/> **re-checks it** before applying a hue and
/// refuses rather than risk it: an invariant nothing verifies is a comment.
///
/// ── **What is validated, and against what** ──
///
/// Everything §4.5 built, reused as-is. An index entry is judged before the id is handed to
/// <c>Ultima</c> (<see cref="BridgeAssetValidator.CheckEntry"/>), a static's record header
/// and row table are walked bounded (<c>StaticSane</c>), a land record is checked against
/// the 2,024 bytes <c>LoadLand</c> reads whatever the length says (<c>LandLengthSane</c>),
/// and the bound is taken against **whichever file <c>FileIndex</c> actually opened** —
/// <c>artLegacyMUL.uop</c> on every current client, never <c>art.mul</c> (§4.6).
///
/// Two of §4.5's measurements are this family's, not the catalogue's, and they are the
/// reason all of it is here: on a **stock** client **9,963 static ids and 12,140 land ids**
/// have an index entry reading `lookup 0, length 0`, which <c>FileIndex.Seek</c> treats as
/// a hit and the decoder answers with whatever was decoded last. Measured through this
/// reader over the whole range, those are the ONLY refusals — every one of the 39,189
/// statics and 4,244 land tiles that carries art is served, which is the half of the
/// measurement that says the boundary is in the right place (§4.5).
/// </summary>
public static class BridgeArt
{
/// <summary>Item graphics. <c>static/&lt;id&gt;</c>, optionally <c>/h&lt;hue&gt;</c>.</summary>
private const string StaticFamily = "static";
/// <summary>Land tiles. <c>land/&lt;id&gt;</c>, and no hue segment — see <see cref="TryParseKey"/>.</summary>
private const string LandFamily = "land";
/// <summary>The art index addresses land at its own id and statics at <c>0x4000 + id</c>.</summary>
private const int StaticBase = 0x4000;
/// <summary>Land is addressed with <c>index &amp; 0x3FFF</c> by the library itself.</summary>
private const int LandCount = 0x4000;
/// <summary><c>hues.mul</c> holds 3,000 slots; the wire's hue 1 is slot 0.</summary>
private const int MaxHue = 3000;
public static void Initialize()
{
if (!BridgeConfig.Enabled)
return;
BridgeAssets.RegisterFamily(StaticFamily, ReplyFetch);
BridgeAssets.RegisterFamily(LandFamily, ReplyFetch);
}
// ── the cache (§11) ──────────────────────────────────────────────────────────────────
private sealed class Rendered
{
public string Key;
public string Status;
public string Reason;
public string Sha256;
public byte[] Png;
public int Width;
public int Height;
public int Hue;
public bool PartialHue;
public string Source;
public int Weight
{
get { return Png == null ? 128 : Png.Length + 128; }
}
}
private sealed class Cache
{
public string Id;
public readonly Dictionary<string, Rendered> ByKey =
new Dictionary<string, Rendered>(StringComparer.Ordinal);
/// <summary>Insertion order, for eviction. See <see cref="Remember"/>.</summary>
public readonly Queue<string> Order = new Queue<string>();
public long Bytes;
public DateTime LastUsed;
}
private static readonly object _sync = new object();
private static Cache _cache;
private static readonly TimeSpan IdleFor = TimeSpan.FromMinutes(5);
// ── assets.fetch, the static and land half ───────────────────────────────────────────
/// <summary>
/// Both families' answer to <c>assets.fetch</c>. The correlation id, the operator's
/// consent, the key ceiling and the family decision were made by
/// <see cref="BridgeAssets.OnFetch"/>; every key here belongs to this reader.
///
/// The paging envelope, the byte budget and the `catalog` guard are §3.4's and
/// phase 3's, unchanged — a caller that already walks the body catalogue walks this
/// with the same loop.
/// </summary>
private static void ReplyFetch(string reqId, List<string> keys, string expected, string cursor)
{
string imagingReason;
if (!BridgeAssets.ImagingOk(out imagingReason))
{
// §17.9: a flat refusal, not a partial answer. Every picture in this family needs
// a decoder that goes through GDI+, so there is no half of it to serve.
BridgeAssets.Fail(reqId, "UNAVAILABLE",
"this shard host cannot render images - Mono's System.Drawing needs "
+ "libgdiplus. (" + imagingReason + ")");
return;
}
string id = SourceId();
if (expected != null && expected != id)
{
BridgeAssets.Fail(reqId, "UNREADABLE",
"the shard's client files changed since that catalogue was read (catalog "
+ expected + " is now " + id + "); ask again");
return;
}
Cache cache;
lock (_sync)
{
if (_cache == null || _cache.Id != id)
_cache = new Cache { Id = id };
cache = _cache;
cache.LastUsed = DateTime.UtcNow;
}
int from = ParseKeyCursor(cursor);
var sb = BridgeJson.Begin("assets.fetch.ok");
sb.Str("reqId", reqId)
.Str("family", BridgeAssets.FamilyOfKey(keys[0]))
.Str("catalog", cache.Id)
.Num("extractorVersion", BridgeAssets.EXTRACTOR_VERSION)
.Num("asked", keys.Count)
.Num("from", from);
var page = new BridgeAssets.PageBuilder(sb, "rows", BridgeConfig.AssetBatchBytes);
using (var readers = new Readers())
{
for (int i = from; i < keys.Count; i++)
{
string row = Row(cache, readers, keys[i]);
if (!page.TryAdd(row, "k:" + (i + 1).ToString(CultureInfo.InvariantCulture)))
break;
}
}
page.Close();
sb.Num("sent", page.Count);
BridgeLink.Emit(sb.End());
Sweep();
}
/// <summary>
/// One key to one JSON row.
///
/// A key this shard cannot serve is a **row**, never a failed request: an item id with
/// no art must not cost the other three hundred on the page. The three outcomes are the
/// ones phase 3 defined, and this family adds a `reason` beside them — additive, and
/// the only way an operator learns that eight of their records are damaged rather than
/// simply absent, which is a difference §4.5 spent a whole phase establishing.
/// </summary>
private static string Row(Cache cache, Readers readers, string key)
{
Rendered item = Resolve(cache, readers, key);
var sb = new StringBuilder(2048);
sb.Append("{\"key\":");
BridgeJson.Text(sb, key);
sb.Append(",\"status\":\"").Append(item.Status).Append('"');
if (item.Reason != null)
{
sb.Append(",\"reason\":");
BridgeJson.Text(sb, item.Reason);
}
if (item.Status != "ok")
{
sb.Append('}');
return sb.ToString();
}
sb.Append(",\"sha256\":\"").Append(item.Sha256).Append('"');
sb.Append(",\"bytes\":").Append(item.Png.Length.ToString(CultureInfo.InvariantCulture));
sb.Append(",\"width\":").Append(item.Width.ToString(CultureInfo.InvariantCulture));
sb.Append(",\"height\":").Append(item.Height.ToString(CultureInfo.InvariantCulture));
if (item.Hue > 0)
{
sb.Append(",\"hue\":").Append(item.Hue.ToString(CultureInfo.InvariantCulture));
sb.Append(",\"partialHue\":").Append(item.PartialHue ? "true" : "false");
}
sb.Append(",\"source\":\"").Append(item.Source).Append('"');
sb.Append(",\"png\":\"").Append(Convert.ToBase64String(item.Png)).Append("\"}");
return sb.ToString();
}
private static Rendered Resolve(Cache cache, Readers readers, string key)
{
lock (_sync)
{
Rendered cached;
if (cache.ByKey.TryGetValue(key, out cached))
return cached;
}
Rendered item;
try
{
item = Render(readers, key);
}
catch (Exception e)
{
Console.WriteLine("[Bridge] art: {0}: {1}: {2}", key, e.GetType().Name, e.Message);
item = new Rendered
{
Key = key,
Status = "absent",
Reason = e.GetType().Name
};
}
if (item.Status == "ok")
Remember(cache, item);
return item;
}
/// <summary>
/// Holds the encoded bytes against a byte budget, evicting oldest-first.
///
/// **Oldest-first rather than least-recently-used, deliberately.** The access pattern
/// this serves is a warm pass: the website asks for the keys it has never held, stores
/// them permanently, and does not ask again. What this cache is actually for is the
/// second page of a batch, a retry after a 425, and the same picture appearing in two
/// of a page's rows — all of which insertion order serves exactly as well as recency,
/// and with no bookkeeping on the hot path. A cache whose hit pattern has no recency in
/// it should not pretend to rank by it.
///
/// Only successes are held. An absent key costs one index lookup, which is cheaper than
/// the dictionary entry that would remember it.
/// </summary>
private static void Remember(Cache cache, Rendered item)
{
lock (_sync)
{
if (cache.ByKey.ContainsKey(item.Key))
return;
cache.ByKey[item.Key] = item;
cache.Order.Enqueue(item.Key);
cache.Bytes += item.Weight;
while (cache.Bytes > BridgeConfig.AssetArtCacheBytes && cache.Order.Count > 0)
{
string oldest = cache.Order.Dequeue();
Rendered evicted;
if (!cache.ByKey.TryGetValue(oldest, out evicted))
continue;
cache.ByKey.Remove(oldest);
cache.Bytes -= evicted.Weight;
}
}
}
// ── decode ───────────────────────────────────────────────────────────────────────────
/// <summary>
/// Validate, decode, hue, encode. In that order, and the order is the point.
/// </summary>
private static Rendered Render(Readers readers, string key)
{
bool land;
int id, hue;
if (!TryParseKey(key, out land, out id, out hue))
return Unsupported(key, "not a key this shard serves");
FileIndex index = readers.Index;
if (index == null || index.Index == null)
return Absent(key, "this shard has no art file");
int at = land ? id : StaticBase + id;
if (at < 0 || at >= index.Index.Length)
return Unsupported(key, "id " + id + " is past the end of this client's art index");
string reason;
BridgeAssetValidator.Verdict verdict =
BridgeAssetValidator.CheckEntry(index, at, readers.DataLength, readers.VerdataLength, out reason);
if (verdict == BridgeAssetValidator.Verdict.Absent)
{
// The 9,962 statics and 12,140 land tiles of §4.5: an index entry that reads
// `lookup 0, length 0`, which the library treats as a hit and answers with the
// previous asset's pixels. Absent is the true answer and the only safe one.
return Absent(key, reason);
}
if (verdict != BridgeAssetValidator.Verdict.Ok)
{
// A damaged record rather than a missing one. Still absent to the website — there
// is no picture either way — but the reason is worth carrying, because this one an
// operator can act on.
Console.WriteLine("[Bridge] art: {0} refused: {1}", key, reason);
return Absent(key, reason);
}
if (land)
{
if (!BridgeAssetValidator.LandLengthSane(index, at, out reason))
{
Console.WriteLine("[Bridge] art: {0} refused: {1}", key, reason);
return Absent(key, reason);
}
}
else if (readers.Reader == null || !readers.Reader.StaticSane(index, at, out reason))
{
Console.WriteLine("[Bridge] art: {0} refused: {1}",
key, reason ?? "the art record could not be read");
return Absent(key, reason ?? "the art record could not be read");
}
// A hue is resolved BEFORE anything is decoded, so a bad one costs no pixels and, more
// to the point, cannot half-apply to a picture that then gets cached and served.
Ultima.Hue applied = null;
bool partial = false;
if (hue > 0)
{
if (!TryHue(id, hue, out applied, out partial, out reason))
return Unsupported(key, reason);
}
Bitmap bitmap = land
? Art.GetLand(id)
// `checkmaxid: false` on purpose (§4.5): the default maps an out-of-range id to 0
// and returns ITEM ZERO'S PICTURE. The id is already bounded against the index
// that was actually opened, so this can only be loud.
: Art.GetStatic(id, false);
// **Whether this bitmap is ours to dispose is the same question as whether it is ours
// to hue**, and it has the same answer. With the library's cache off — which
// `BridgeAssets.Initialize` guarantees and `TryHue` re-checks — every call decodes a
// fresh instance that nothing else holds, so not disposing it would leak one bitmap per
// fetched key. With the cache on, that instance is the library's own copy and disposing
// it would leave a disposed `Bitmap` in a static array for the next caller to fault on.
// Both mistakes are silent; the flag decides, once, here.
bool owned = !Files.CacheData;
try
{
if (bitmap == null || bitmap.Width <= 0 || bitmap.Height <= 0)
return Absent(key, "the decoder returned no picture");
if (applied != null)
applied.ApplyTo(bitmap, partial);
byte[] png = BridgeAssets.BitmapToPng(bitmap);
if (png == null)
return Absent(key, "the picture could not be encoded");
return new Rendered
{
Key = key,
Status = "ok",
Sha256 = BridgeAssets.Sha256Hex(png),
Png = png,
Width = bitmap.Width,
Height = bitmap.Height,
Hue = hue,
PartialHue = partial,
Source = readers.Source
};
}
finally
{
if (owned && bitmap != null)
bitmap.Dispose();
}
}
/// <summary>
/// Resolves one wire hue onto a ramp, and decides whether it repaints the whole sprite
/// or only its grey pixels.
///
/// Four things have to hold, and every one of them has a way of not holding that
/// produces a picture rather than an error:
///
/// **The library's cache is off.** Re-checked here because <c>ApplyTo</c> repaints in
/// place: with the cache on, this would edit the copy <c>Art</c> hands to everyone
/// else. <see cref="BridgeAssets.Initialize"/> turns it off at boot and this refuses
/// if it somehow did not, because the failure is invisible and permanent.
///
/// **`hues.mul` is present.** When it is missing <c>Hues.Initialize</c> does not throw
/// — it fills all 3,000 slots with a <c>new Hue(index)</c> whose ramp is **all zeroes**,
/// and applying one of those paints the sprite black. An all-zero ramp is therefore
/// refused whatever the reason for it; on this client there are none.
///
/// **The index is in range.** The wire's hue is 1-based — <c>Ultima.Map</c> does the
/// same <c>GetHue(hue - 1)</c> at line 450 — and <c>GetHue</c> itself masks with
/// `0x3FFF` and falls back to slot 0 rather than failing, so an out-of-range hue would
/// silently become a different colour. Bound it here instead.
///
/// **The <c>PartialHue</c> flag decides the mode**, per item id, out of
/// <c>tiledata.mul</c>. This is the one that is invisible: both modes decode, both are
/// the right size, and 13,259 of this client's item ids need the grey-only one.
/// **Land has no such flag**, which is why <see cref="TryParseKey"/> does not accept a
/// hue on a land key at all rather than guessing a mode for it.
/// </summary>
private static bool TryHue(int id, int hue, out Ultima.Hue applied, out bool partial, out string reason)
{
applied = null;
partial = false;
reason = null;
if (Files.CacheData)
{
reason = "this shard's art cache is on, so a hue cannot be applied safely";
Console.WriteLine("[Bridge] art: refusing hue {0}: {1}", hue, reason);
return false;
}
if (hue < 1 || hue > MaxHue)
{
reason = "hue " + hue + " is outside 1-" + MaxHue;
return false;
}
Ultima.Hue[] list = Ultima.Hues.List;
if (list == null || hue - 1 >= list.Length || list[hue - 1] == null)
{
reason = "this client has no hue table";
return false;
}
Ultima.Hue candidate = list[hue - 1];
if (candidate.Colors == null || AllZero(candidate.Colors))
{
reason = "hue " + hue + " has no colours in this client's hues.mul";
return false;
}
if (!TryPartialHue(id, out partial, out reason))
return false;
applied = candidate;
return true;
}
private static bool AllZero(short[] colors)
{
for (int i = 0; i < colors.Length; i++)
{
if (colors[i] != 0)
return false;
}
return true;
}
/// <summary>
/// The <c>PartialHue</c> flag for one item id.
///
/// Refuses rather than defaults when <c>tiledata.mul</c> cannot be read. Defaulting
/// either way would be a coin flip on 13,259 ids, and the losing side of it is a
/// picture that looks deliberate.
///
/// **Every type here is spelled <c>Ultima.</c> on purpose, and it is not style.**
/// ServUO declares its own <c>Server.TileData</c>, <c>Server.ItemData</c> and
/// <c>Server.TileFlag</c> — with a <c>PartialHue</c> member — in
/// <c>Server/TileData.cs</c>. This file lives in <c>Server.Custom.Bridge</c>, so the
/// enclosing namespace beats the <c>using Ultima;</c> and the unqualified spelling
/// silently binds to the *server's* table: it compiles, the flag exists, and the answer
/// comes from a file resolved through <c>Core.DataDirectories</c> rather than through
/// <c>Ultima.Files</c>, which is the one thing §4.6 says never to do — decide a picture
/// with a file other than the one the pixels came out of. The first run of this reader
/// did exactly that and refused every hued key with a <c>TypeInitializationException</c>
/// from a class this code never meant to name.
/// </summary>
private static bool TryPartialHue(int id, out bool partial, out string reason)
{
partial = false;
reason = null;
Ultima.ItemData[] table;
try
{
table = Ultima.TileData.ItemTable;
}
catch (Exception e)
{
reason = "this client's tiledata could not be read (" + e.GetType().Name + ")";
return false;
}
if (table == null || id < 0 || id >= table.Length)
{
reason = "this client's tiledata does not describe item " + id;
return false;
}
partial = (table[id].Flags & Ultima.TileFlag.PartialHue) != 0;
return true;
}
private static Rendered Absent(string key, string reason)
{
return new Rendered { Key = key, Status = "absent", Reason = reason };
}
private static Rendered Unsupported(string key, string reason)
{
return new Rendered { Key = key, Status = "unsupported", Reason = reason };
}
// ── keys, cursors and the source id ──────────────────────────────────────────────────
/// <summary>
/// <c>static/&lt;id&gt;</c>, <c>static/&lt;id&gt;/h&lt;hue&gt;</c> and
/// <c>land/&lt;id&gt;</c>.
///
/// **A land key takes no hue segment.** The client can hue a land tile, but the mode
/// that decides how is an *item* flag and land has no equivalent — so the honest answer
/// to `land/3/h33` is that this shard does not serve it, rather than a picture produced
/// by guessing. Nothing on the wire carries a hued land tile today; if something ever
/// does, it arrives with a reason to choose.
/// </summary>
private static bool TryParseKey(string key, out bool land, out int id, out int hue)
{
land = false;
id = 0;
hue = 0;
if (key == null)
return false;
string[] parts = key.Split('/');
if (parts.Length < 2 || parts.Length > 3)
return false;
if (parts[0] == LandFamily)
land = true;
else if (parts[0] != StaticFamily)
return false;
if (!Int32.TryParse(parts[1], NumberStyles.None, CultureInfo.InvariantCulture, out id))
return false;
if (id < 0)
return false;
if (land && id >= LandCount)
return false;
if (parts.Length == 2)
return true;
if (land)
return false;
string segment = parts[2];
if (segment.Length < 2 || segment[0] != 'h')
return false;
if (!Int32.TryParse(segment.Substring(1), NumberStyles.None,
CultureInfo.InvariantCulture, out hue))
return false;
// **`h0` is not a key.** Hue 0 on the wire means "this item is not hued", so the plain
// key already names its picture. Accepting `static/3922/h0` as a synonym would have
// the website store the identical PNG twice under two names, diff them separately on
// every Update, and show whichever row it happened to join against -- for a distinction
// that does not exist. The caller drops the segment instead.
return hue > 0;
}
private static int ParseKeyCursor(string cursor)
{
if (cursor == null)
return 0;
int value;
if (cursor.StartsWith("k:", StringComparison.Ordinal)
&& Int32.TryParse(cursor.Substring(2), NumberStyles.None, CultureInfo.InvariantCulture, out value))
return Math.Max(0, value);
return 0;
}
/// <summary>
/// Everything that decides these bytes, hashed into one short id — the same guard
/// phase 3 built, over this family's inputs.
///
/// Four files, and each earns its place: the art data file holds the pixels,
/// `hues.mul` holds the ramps, `tiledata.mul` decides which of the two hue modes an
/// item gets, and `verdata.mul` can patch any record in any of them. Leaving
/// `tiledata.mul` out would be the subtle one — a client patch that only flipped
/// <c>PartialHue</c> flags changes no pixel in any source file and every hued picture
/// derived from them.
/// </summary>
private static string SourceId()
{
var sb = new StringBuilder(256);
sb.Append(BridgeAssets.EXTRACTOR_VERSION);
foreach (string path in new[]
{
BridgeAssetValidator.ArtDataPath(),
FilePath("hues.mul"),
FilePath("tiledata.mul"),
FilePath("verdata.mul")
})
{
sb.Append('|');
if (path == null)
continue;
try
{
var info = new FileInfo(path);
if (!info.Exists)
continue;
sb.Append(info.Length).Append(',').Append(info.LastWriteTimeUtc.Ticks);
}
catch
{
// An unreadable file is itself a state, and one that must not change from page
// to page without being noticed. Leaving the slot empty does that.
}
}
return BridgeAssets.Sha256Hex(Encoding.UTF8.GetBytes(sb.ToString())).Substring(0, 16);
}
private static string FilePath(string name)
{
try
{
return Files.GetFilePath(name);
}
catch
{
return null;
}
}
// ── shared plumbing ──────────────────────────────────────────────────────────────────
/// <summary>
/// The art index and its record reader, opened for one reply and closed with it — the
/// same lifetime rule phase 3's <c>Readers</c> follows, and for the same reason: a page
/// decodes hundreds of sprites through them and opening them is microseconds, so
/// holding handles on the operator's client files for the life of a cache buys nothing.
/// </summary>
private sealed class Readers : IDisposable
{
public readonly FileIndex Index;
public readonly BridgeAssetValidator.RecordReader Reader;
public readonly long DataLength;
public readonly long VerdataLength;
/// <summary>
/// Which file the pixels came out of — `uop` or `legacy` — carried on every row
/// beside the body catalogue's own `source` (§4.9). On this plane it answers §4.6's
/// operator question: art added to `art.mul` while `artLegacyMUL.uop` is present is
/// never read, and a row that says `uop` is what says so.
/// </summary>
public readonly string Source;
public Readers()
{
string data = BridgeAssetValidator.ArtDataPath();
string verdata = FilePath("verdata.mul");
DataLength = BridgeAssetValidator.MulLength(data);
VerdataLength = BridgeAssetValidator.MulLength(verdata);
Source = data != null && data.EndsWith(".uop", StringComparison.OrdinalIgnoreCase)
? "uop"
: "legacy";
try
{
Index = BridgeAssetValidator.OpenArtIndex();
if (data != null)
Reader = new BridgeAssetValidator.RecordReader(data, verdata);
}
catch (Exception e)
{
Console.WriteLine("[Bridge] art: could not open the art files: {0}", e.Message);
}
}
public void Dispose()
{
if (Reader == null)
return;
try
{
Reader.Dispose();
}
catch
{
// Closing a read-only handle. Nothing useful is left to do.
}
}
}
/// <summary>
/// Lets the held pictures go once nothing has asked for one in five minutes. The id is
/// derived from the client files rather than minted per build, so a walk that spans the
/// drop resumes against the same catalogue instead of starting over.
/// </summary>
private static void Sweep()
{
lock (_sync)
{
if (_cache == null)
return;
if (DateTime.UtcNow - _cache.LastUsed > IdleFor)
_cache = null;
}
}
public static string Status()
{
lock (_sync)
{
if (_cache == null)
return "art(empty)";
return String.Format("art(id={0} held={1} bytes={2} cap={3})",
_cache.Id, _cache.ByKey.Count, _cache.Bytes, BridgeConfig.AssetArtCacheBytes);
}
}
}
}

View File

@@ -1,7 +1,10 @@
using System;
using System.Collections.Generic;
using System.Drawing;
using System.Drawing.Imaging;
using System.Globalization;
using System.IO;
using System.Runtime.InteropServices;
using System.Security.Cryptography;
using System.Text;
using System.Threading;
@@ -123,7 +126,58 @@ namespace Server.Custom.Bridge
if (!BridgeConfig.Enabled)
return;
DisableTheLibraryCache();
BridgeBoot.RegisterHandler("assets.sources", OnSources);
BridgeBoot.RegisterHandler("assets.fetch", OnFetch);
}
/// <summary>
/// **Turns <c>Ultima.Files.CacheData</c> off for the life of the process** (phase 5,
/// §17.10). One line, and it answers two separate problems that both end in a
/// confident wrong picture or an out-of-memory shard.
///
/// **The poisoning.** <c>Art.GetStatic</c> and <c>Art.GetLand</c> memoise into a
/// <c>Bitmap[0xFFFF]</c> and hand back **the same instance** on every call, while
/// <c>Hue.ApplyTo</c> repaints a bitmap **in place**. So hueing a static edits the
/// library's cached copy: measured on this client, hue item 3922 once and every later
/// request for the *plain* 3922 comes back hued, and a second hue stacks on the first.
/// Nothing downstream can see it — the row is the right size, the right shape and the
/// right id. It is §4.5's failure mode arriving through a completely different door.
///
/// **The retention.** That array is never trimmed. Decoding this client's 39,189
/// statics once would leave 74 MB of <c>Bitmap</c> in a static field of a game server,
/// kept for as long as the process lives, to serve pictures nobody asked for twice.
///
/// The obvious alternative — copy each bitmap before hueing — was rejected, and not
/// only for the retention: <c>new Bitmap(src)</c> **throws** on the
/// <c>Format16bppArgb1555</c> these decoders produce, so the copy has to name the
/// source pixel format explicitly, which is a subtlety on the wrong side of a
/// correctness boundary.
///
/// **What it costs is nothing measurable here.** <c>Animations</c> — the whole of the
/// body catalogue — does not consult this flag at all, and
/// <see cref="BridgeCatalog"/> and <see cref="BridgeArt"/> each keep their own cache of
/// *encoded PNG bytes*, which is the thing worth holding: a tenth of the size, already
/// hashed, and released when it goes idle.
///
/// It is a process-global on a library nothing else in this overlay reads, which is why
/// setting it here rather than saving and restoring it around each decode is safe —
/// and a save/restore would not be, because the asset worker is a thread.
/// </summary>
private static void DisableTheLibraryCache()
{
try
{
Files.CacheData = false;
}
catch (Exception e)
{
// A client this library cannot even open. The families report that for themselves,
// per key, with a reason; it must not stop the plugin booting.
Console.WriteLine("[Bridge] assets: could not disable the Ultima bitmap cache: {0}",
e.Message);
}
}
public static string Status()
@@ -228,6 +282,248 @@ namespace Server.Custom.Bridge
BridgeLink.Emit(sb.End());
}
// ── assets.fetch, and the families behind it (§5, phase 5) ───────────────────────────
/// <summary>
/// One family's answer to a fetch. Runs on the asset worker, never the Core thread.
/// </summary>
internal delegate void FamilyFetch(string reqId, List<string> keys, string catalog, string cursor);
private static readonly Dictionary<string, FamilyFetch> _families =
new Dictionary<string, FamilyFetch>(StringComparer.Ordinal);
/// <summary>
/// Claims one §5 key family for a reader.
///
/// Phase 3 gave <c>assets.fetch</c> to the body catalogue outright, which was right
/// while there was one family and wrong the moment there were three: the command is
/// the *transport*, and the family is a property of the key. So the shared parts — the
/// correlation id, the operator's consent, the key-count ceiling, and deciding which
/// reader a request belongs to — live here once, and a family only ever sees a list of
/// keys it owns.
///
/// Registration is order-independent on purpose: ServUO calls every
/// <c>Initialize</c> in an order nothing here controls, and this fills a dictionary the
/// handler does not read until a request arrives.
/// </summary>
internal static void RegisterFamily(string name, FamilyFetch fetch)
{
lock (_families)
{
_families[name] = fetch;
}
}
/// <summary>The families this shard can serve, for §6's stage 1 and for diagnostics.</summary>
internal static List<string> Families()
{
lock (_families)
{
var names = new List<string>(_families.Keys);
names.Sort(StringComparer.Ordinal);
return names;
}
}
private static FamilyFetch FamilyFor(string name)
{
lock (_families)
{
FamilyFetch fetch;
return _families.TryGetValue(name, out fetch) ? fetch : null;
}
}
/// <summary>
/// The family segment of a §5 key: everything before the first `/`.
/// </summary>
internal static string FamilyOfKey(string key)
{
if (String.IsNullOrEmpty(key))
return null;
int slash = key.IndexOf('/');
return slash <= 0 ? null : key.Substring(0, slash);
}
/// <summary>
/// §14's `assets.fetch`, for every family.
///
/// **The family is derived from the keys and is not a request field.** §5 made the key
/// the address of an asset, so a request that had to name its family as well would have
/// two places to be wrong and one of them silent. A batch must be of one family —
/// mixing them is refused rather than split — because the reply carries a single
/// `catalog` id, and that id is what stops an operator patching their client mid-import
/// from stitching one asset set out of two. Two families, two fingerprints, and a reply
/// that claimed one of them would be lying about the other.
/// </summary>
private static void OnFetch(Dictionary<string, object> o)
{
var reqId = BridgeJson.GetString(o, "reqId");
if (reqId == null)
{
Fail(null, "BAD_REQUEST", "assets.fetch requires a reqId");
return;
}
if (!BridgeConfig.AssetsEnabled)
{
Fail(reqId, "DISABLED", "asset extraction is disabled on this shard");
return;
}
var keys = BridgeJson.GetStringList(o, "keys");
if (keys.Count == 0)
{
Fail(reqId, "BAD_REQUEST", "assets.fetch requires a non-empty `keys` array");
return;
}
if (keys.Count > BridgeConfig.AssetFetchKeys)
{
Fail(reqId, "BAD_REQUEST",
"assets.fetch takes at most " + BridgeConfig.AssetFetchKeys
+ " keys per request (asked for " + keys.Count + ")");
return;
}
string family = FamilyOfKey(keys[0]);
for (int i = 1; i < keys.Count; i++)
{
if (String.Equals(FamilyOfKey(keys[i]), family, StringComparison.Ordinal))
continue;
Fail(reqId, "BAD_REQUEST",
"assets.fetch takes keys of one family per request; this one mixes '"
+ family + "' with '" + FamilyOfKey(keys[i]) + "'");
return;
}
FamilyFetch fetch = FamilyFor(family);
if (fetch == null)
{
Fail(reqId, "BAD_REQUEST",
"this shard serves no '" + family + "' asset family (it serves "
+ String.Join(", ", Families().ToArray()) + ")");
return;
}
var catalog = BridgeJson.GetString(o, "catalog");
var cursor = BridgeJson.GetString(o, "cursor");
Accept(reqId, "assets.fetch", () => fetch(reqId, keys, catalog, cursor));
}
/// <summary>
/// ARGB1555 to a PNG with a transparent background.
///
/// <c>Frame</c> writes 16-bit ARGB1555: a pixel the sprite does not cover is left as
/// zero and a pixel it does cover carries the top bit set. Saving that format straight
/// to PNG asks GDI+ to make the conversion, and what it does with a one-bit alpha
/// channel varies by platform — on Mono it is a different implementation entirely. A
/// sprite that came back with a black rectangle behind it would look fine in a test
/// that only checked the bytes decoded, and wrong on every page that showed it.
///
/// So the expansion is done here, explicitly: alpha bit clear becomes fully
/// transparent, and each 5-bit channel is widened to 8 bits by repeating its high bits
/// (<c>(c &lt;&lt; 3) | (c &gt;&gt; 2)</c>) rather than by shifting alone, which would
/// cap white at 248 and tint the whole catalogue.
/// </summary>
internal static byte[] BitmapToPng(Bitmap source)
{
var rect = new Rectangle(0, 0, source.Width, source.Height);
if (source.PixelFormat != PixelFormat.Format16bppArgb1555)
{
// Not what this library has ever produced. Save it rather than reinterpret it:
// guessing at an unknown layout is how a catalogue fills with confident nonsense.
using (var ms = new MemoryStream())
{
source.Save(ms, ImageFormat.Png);
return ms.ToArray();
}
}
using (var target = new Bitmap(source.Width, source.Height, PixelFormat.Format32bppArgb))
{
BitmapData src = source.LockBits(rect, ImageLockMode.ReadOnly, PixelFormat.Format16bppArgb1555);
BitmapData dst = null;
try
{
dst = target.LockBits(rect, ImageLockMode.WriteOnly, PixelFormat.Format32bppArgb);
var line = new short[source.Width];
var outLine = new int[source.Width];
for (int y = 0; y < source.Height; y++)
{
Marshal.Copy(new IntPtr(src.Scan0.ToInt64() + ((long)y * src.Stride)),
line, 0, source.Width);
for (int x = 0; x < source.Width; x++)
{
int p = line[x] & 0xFFFF;
if ((p & 0x8000) == 0)
{
outLine[x] = 0;
continue;
}
int r = (p >> 10) & 0x1F;
int g = (p >> 5) & 0x1F;
int b = p & 0x1F;
outLine[x] = unchecked((int)0xFF000000)
| (((r << 3) | (r >> 2)) << 16)
| (((g << 3) | (g >> 2)) << 8)
| ((b << 3) | (b >> 2));
}
Marshal.Copy(outLine, 0, new IntPtr(dst.Scan0.ToInt64() + ((long)y * dst.Stride)),
source.Width);
}
}
finally
{
if (dst != null)
target.UnlockBits(dst);
source.UnlockBits(src);
}
using (var ms = new MemoryStream())
{
target.Save(ms, ImageFormat.Png);
return ms.ToArray();
}
}
}
/// <summary>
/// SHA-256, lowercase hex. Shared because the hash in a manifest row, the hash in a
/// fetch row and the hash the website stores must be one function.
/// </summary>
internal static string Sha256Hex(byte[] bytes)
{
using (var sha = SHA256.Create())
{
byte[] digest = sha.ComputeHash(bytes);
var sb = new StringBuilder(digest.Length * 2);
foreach (byte b in digest)
sb.Append(b.ToString("x2", CultureInfo.InvariantCulture));
return sb.ToString();
}
}
/// <summary>
/// The asset plane's one refusal frame, shared by every family on it.
///
@@ -354,6 +650,24 @@ namespace Server.Custom.Bridge
sb.Str("artDataFile", artData == null ? null : Path.GetFileName(artData));
// Which §5 families this shard can be asked for. Additive, so the protocol stays 8: a
// consumer that does not read it behaves exactly as it did. One that does can tell an
// older overlay (bodies only) from this one without discovering it as a refused fetch
// halfway through a warm pass.
var families = Families();
sb.Append(",\"families\":[");
for (int i = 0; i < families.Count; i++)
{
if (i > 0)
sb.Append(',');
BridgeJson.Text(sb, families[i]);
}
sb.Append(']');
var page = new PageBuilder(sb, "files", BridgeConfig.AssetBatchBytes);
bool anyMissingHash = false;

View File

@@ -262,6 +262,7 @@ namespace Server.Custom.Bridge
e.Mobile.SendMessage("Bridge: {0}", BridgeOneShots.Status());
e.Mobile.SendMessage("Bridge: {0}", BridgeAssets.Status());
e.Mobile.SendMessage("Bridge: {0}", BridgeCatalog.Status());
e.Mobile.SendMessage("Bridge: {0}", BridgeArt.Status());
break;
}
}

View File

@@ -1,11 +1,8 @@
using System;
using System.Collections.Generic;
using System.Drawing;
using System.Drawing.Imaging;
using System.Globalization;
using System.IO;
using System.Runtime.InteropServices;
using System.Security.Cryptography;
using System.Text;
using Ultima;
@@ -101,7 +98,10 @@ namespace Server.Custom.Bridge
return;
BridgeBoot.RegisterHandler("assets.manifest", OnManifest);
BridgeBoot.RegisterHandler("assets.fetch", OnFetch);
// `assets.fetch` is shared plumbing as of phase 5 (§5): BridgeAssets owns the command,
// decides which family a batch of keys belongs to, and calls the reader that owns it.
BridgeAssets.RegisterFamily(Family, ReplyFetch);
}
// ── the cache ────────────────────────────────────────────────────────────────────────
@@ -324,36 +324,11 @@ namespace Server.Custom.Bridge
// ── assets.fetch ─────────────────────────────────────────────────────────────────────
private static void OnFetch(Dictionary<string, object> o)
{
string reqId;
if (!Admit(o, "assets.fetch", out reqId))
return;
var keys = BridgeJson.GetStringList(o, "keys");
if (keys.Count == 0)
{
BridgeAssets.Fail(reqId, "BAD_REQUEST",
"assets.fetch requires a non-empty `keys` array");
return;
}
if (keys.Count > BridgeConfig.AssetFetchKeys)
{
BridgeAssets.Fail(reqId, "BAD_REQUEST",
"assets.fetch takes at most " + BridgeConfig.AssetFetchKeys
+ " keys per request (asked for " + keys.Count + ")");
return;
}
var catalog = BridgeJson.GetString(o, "catalog");
var cursor = BridgeJson.GetString(o, "cursor");
BridgeAssets.Accept(reqId, "assets.fetch", () => ReplyFetch(reqId, keys, catalog, cursor));
}
/// <summary>
/// The `body` family's half of <c>assets.fetch</c>. The correlation id, the operator's
/// consent, the key ceiling and the family decision have already been made by
/// <see cref="BridgeAssets.RegisterFamily"/>'s caller; every key here is this family's.
/// </summary>
private static void ReplyFetch(string reqId, List<string> keys, string expected, string cursor)
{
string imagingReason;
@@ -651,7 +626,7 @@ namespace Server.Custom.Bridge
Png = png,
Width = pixels.Width,
Height = pixels.Height,
Sha256 = Hash(png),
Sha256 = BridgeAssets.Sha256Hex(png),
Source = "uop"
};
}
@@ -673,7 +648,7 @@ namespace Server.Custom.Bridge
if (bitmap == null || bitmap.Width <= 0 || bitmap.Height <= 0)
return null;
byte[] png = ToPng(bitmap);
byte[] png = BridgeAssets.BitmapToPng(bitmap);
if (png == null)
return null;
@@ -687,112 +662,11 @@ namespace Server.Custom.Bridge
Png = png,
Width = bitmap.Width,
Height = bitmap.Height,
Sha256 = Hash(png),
Sha256 = BridgeAssets.Sha256Hex(png),
Source = "legacy"
};
}
/// <summary>
/// ARGB1555 to a PNG with a transparent background.
///
/// <c>Frame</c> writes 16-bit ARGB1555: a pixel the sprite does not cover is left as
/// zero and a pixel it does cover carries the top bit set. Saving that format straight
/// to PNG asks GDI+ to make the conversion, and what it does with a one-bit alpha
/// channel varies by platform — on Mono it is a different implementation entirely. A
/// sprite that came back with a black rectangle behind it would look fine in a test
/// that only checked the bytes decoded, and wrong on every page that showed it.
///
/// So the expansion is done here, explicitly: alpha bit clear becomes fully
/// transparent, and each 5-bit channel is widened to 8 bits by repeating its high bits
/// (<c>(c &lt;&lt; 3) | (c &gt;&gt; 2)</c>) rather than by shifting alone, which would
/// cap white at 248 and tint the whole catalogue.
/// </summary>
private static byte[] ToPng(Bitmap source)
{
var rect = new Rectangle(0, 0, source.Width, source.Height);
if (source.PixelFormat != PixelFormat.Format16bppArgb1555)
{
// Not what this library has ever produced. Save it rather than reinterpret it:
// guessing at an unknown layout is how a catalogue fills with confident nonsense.
using (var ms = new MemoryStream())
{
source.Save(ms, ImageFormat.Png);
return ms.ToArray();
}
}
using (var target = new Bitmap(source.Width, source.Height, PixelFormat.Format32bppArgb))
{
BitmapData src = source.LockBits(rect, ImageLockMode.ReadOnly, PixelFormat.Format16bppArgb1555);
BitmapData dst = null;
try
{
dst = target.LockBits(rect, ImageLockMode.WriteOnly, PixelFormat.Format32bppArgb);
var line = new short[source.Width];
var outLine = new int[source.Width];
for (int y = 0; y < source.Height; y++)
{
Marshal.Copy(new IntPtr(src.Scan0.ToInt64() + ((long)y * src.Stride)),
line, 0, source.Width);
for (int x = 0; x < source.Width; x++)
{
int p = line[x] & 0xFFFF;
if ((p & 0x8000) == 0)
{
outLine[x] = 0;
continue;
}
int r = (p >> 10) & 0x1F;
int g = (p >> 5) & 0x1F;
int b = p & 0x1F;
outLine[x] = unchecked((int)0xFF000000)
| (((r << 3) | (r >> 2)) << 16)
| (((g << 3) | (g >> 2)) << 8)
| ((b << 3) | (b >> 2));
}
Marshal.Copy(outLine, 0, new IntPtr(dst.Scan0.ToInt64() + ((long)y * dst.Stride)),
source.Width);
}
}
finally
{
if (dst != null)
target.UnlockBits(dst);
source.UnlockBits(src);
}
using (var ms = new MemoryStream())
{
target.Save(ms, ImageFormat.Png);
return ms.ToArray();
}
}
}
private static string Hash(byte[] bytes)
{
using (var sha = SHA256.Create())
{
byte[] digest = sha.ComputeHash(bytes);
var sb = new StringBuilder(digest.Length * 2);
foreach (byte b in digest)
sb.Append(b.ToString("x2", CultureInfo.InvariantCulture));
return sb.ToString();
}
}
// ── player bodies (§5.2) ─────────────────────────────────────────────────────────────
/// <summary>
@@ -997,7 +871,7 @@ namespace Server.Custom.Bridge
}
}
return Hash(Encoding.UTF8.GetBytes(sb.ToString())).Substring(0, 16);
return BridgeAssets.Sha256Hex(Encoding.UTF8.GetBytes(sb.ToString())).Substring(0, 16);
}
// ── shared plumbing ──────────────────────────────────────────────────────────────────

View File

@@ -124,6 +124,13 @@ namespace Server.Custom.Bridge
public static int AssetPlayerDirection { get; private set; }
public static int AssetCreatureDirection { get; private set; }
// How many bytes of rendered item and land art the shard holds between requests (§11,
// phase 5). This is a convenience, not a store: the website keeps every picture it fetches
// and does not ask twice, so what this actually buys is the second page of a batch, a
// retry after a 425, and the same item appearing in two rows of one page. Sized so a
// full 512 KB batch and the one before it both fit with room over.
public static int AssetArtCacheBytes { get; private set; }
public static int LeaseMaxDurationSec { get; private set; }
public static int LeaseGraceSec { get; private set; }
@@ -222,6 +229,16 @@ namespace Server.Custom.Bridge
AssetPlayerDirection = Clamp(Config.Get("Bridge.AssetPlayerDirection", 0), 0, 4);
AssetCreatureDirection = Clamp(Config.Get("Bridge.AssetCreatureDirection", 1), 0, 4);
// The floor is one batch: a cache that cannot hold the page being built evicts rows
// while they are still being written, which is a cache that costs and never pays. The
// ceiling is a game server's memory, and 64 MB of PNG is already ~34,000 sprites --
// most of this client's art, held for a working set that is measured in hundreds.
AssetArtCacheBytes = Config.Get("Bridge.AssetArtCacheBytes", 16 * 1024 * 1024);
if (AssetArtCacheBytes < AssetBatchBytes)
AssetArtCacheBytes = AssetBatchBytes;
if (AssetArtCacheBytes > 64 * 1024 * 1024)
AssetArtCacheBytes = 64 * 1024 * 1024;
StatSweepSeconds = Config.Get("Bridge.StatSweepSeconds", 30);
DecaySweepSeconds = Config.Get("Bridge.DecaySweepSeconds", 60);
EconomySweepSeconds = Config.Get("Bridge.EconomySweepSeconds", 300);