Claude
13b6fc02a4
The spawn atlas was the one place the platform's rule -- only the sidecar
bridges the shard -- was broken, and it was broken by the component that faces
the internet: SPAWN_ATLAS.md required the website to read the ServUO tree off a
bind mount or a shared volume. This serves those files over the loopback link
instead (docs/link/v8.md 10).
The measurement came first and changed the shape. 10 said the shard would serve
`tree/<label>` -> bytes; against a stock 57.4 tree it cannot. Spawns/trammel.xml
is 4.03 MB, the sidecar discards any inbound line over 1 MiB, and that file as
one base64 row is 5.4 MiB -- it would be dropped, time out, and be re-requested
forever with no error anywhere. Two files on a STOCK tree are in that state.
So a file crosses as 512 KiB chunks, each gzipped: tree/Spawns/trammel.xml/c0
and so on, which is 5's depth scheme doing the same job it does for
body/400/a0/f0 and needing no protocol change to do it. The chunk is the bound
and the compression is only the saving -- nothing guarantees an operator's files
compress, so the ceiling has to hold when they do not, and a 512 KiB chunk that
refuses to compress is still ~683 KiB of base64, inside the wire cap that
AssetBatchBytes' deliberate factor of two leaves room for.
It is a `tree` FAMILY on assets.fetch rather than 14's separate tree.* commands:
phase 5 had already learned that the command is the transport and the family is
a property of the key, and assets.manifest is generalised here the same way.
That reuses the single slot, the paging envelope, the key ceiling and the
mid-import guard -- and leaves `link` with nothing to do for the third phase
running.
But it gets its OWN consent, Bridge.TreeEnabled. AssetsEnabled is an operator
agreeing the website may read their EA-licensed UO client; this is the shard's
own configuration, which they wrote, and which the public bestiary is built
from. One switch could not express both, and the thing that would silently
disappear for an operator who declined the first is their spawn atlas. So the
consent check moved into the family lookup, and assets.sources answers whenever
either plane is on, reporting `families` filtered to what is actually enabled --
which is how a tree-only shard's website discovers there is anything to ask for.
Two defects found, and which harness found which is the part worth keeping:
- An empty `catalog` is not an absent one. `expected != null` refused every
fetch from a caller that sent "", with a sentence naming no catalog at all.
Found by an offline probe that passed one by accident.
- GZipStream writes NOTHING for zero bytes of input -- the header is emitted
lazily, so a stream opened and closed without a write yields a zero-length
buffer rather than the 20-byte empty member. Stock ServUO ships two empty
decoration files, so this broke every import off an untouched tree. The
offline probe reassembled all 141 files and reported success, because .NET's
own decompressor reads an empty stream as empty data and the chunk's
declared length (0) and hash (of nothing) both agreed. Only the live walk,
through a reader on another runtime, disagreed.
Measured end to end against a live shard, the real sidecar and the website's own
reader: 141 files, 11,895,427 bytes, 158 chunks, 3 pages, 1.33 MB on the wire,
512 ms; every file byte-identical to disk; the atlas built over the bridge
identical to the one built off it. A drift check is the manifest alone -- 32 KB,
~70 ms, no file bytes.
The label set is this shard's, never the caller's: a fetch resolves against the
set the shard itself enumerated, and tree/../../Scripts/..., Config/Bridge.cfg
and Saves/Accounts/accounts.xml are all answered `absent` before a path is built
out of them.
Protocol stays 8 and EXTRACTOR_VERSION stays 3 -- this family derives nothing,
it forwards an operator's own file unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
1282 lines
54 KiB
C#
1282 lines
54 KiB
C#
using System;
|
|
using System.Collections.Generic;
|
|
using System.Drawing;
|
|
using System.Drawing.Imaging;
|
|
using System.Globalization;
|
|
using System.IO;
|
|
using System.Runtime.InteropServices;
|
|
using System.Security.Cryptography;
|
|
using System.Text;
|
|
using System.Threading;
|
|
|
|
using Ultima;
|
|
|
|
namespace Server.Custom.Bridge
|
|
{
|
|
/// <summary>
|
|
/// **The Asset Bridge's transport** (docs/link/v8.md §3, §6, §7 — protocol 8, phase 1).
|
|
///
|
|
/// Everything else on this link answers on the Core thread, reads live world state, and
|
|
/// replies in microseconds. The asset plane cannot: it reads hundreds of megabytes of the
|
|
/// operator's client files and decodes pictures out of them, and doing either on the Core
|
|
/// thread would stop the world for every player on the shard. So this class is the one
|
|
/// genuinely new shape in protocol 8 — a handler that accepts on the Core thread, hands
|
|
/// the work to a **dedicated asset worker**, and returns immediately.
|
|
///
|
|
/// Three rules hold it together, and each of them is answering a specific way this could
|
|
/// go wrong.
|
|
///
|
|
/// **1. Replies, never events.** Every asset frame carries the caller's `reqId`, so
|
|
/// `rpc.rs`'s `try_route` consumes it before `app.rs` can persist it to SQLite and
|
|
/// broadcast it to every WebSocket subscriber. An asset stream on the event path would
|
|
/// grow the sidecar's store without bound and fan megabytes at every connected client,
|
|
/// forever. Nothing here emits an unsolicited frame — if a request has no `reqId` it is
|
|
/// refused rather than answered.
|
|
///
|
|
/// **2. One request outstanding, always.** <see cref="BridgeLink.Emit"/>'s queue is bounded
|
|
/// drop-oldest in **lines, not bytes** — a design that is right for live events and
|
|
/// dangerous for bulk transfer, because 10,000 queued 200 KB replies is 2 GB of shard
|
|
/// memory. The bound that actually holds is flow control, not a bigger queue: this plane
|
|
/// has **one slot**, and a second asset request arriving while one is in flight is
|
|
/// answered `bridge.busy` (which the sidecar already maps to 425) rather than queued.
|
|
/// Queue depth therefore stays at approximately one by construction. A dropped or lost
|
|
/// reply just times out and is re-requested, which is safe because reading a client file
|
|
/// is idempotent and touches no world state.
|
|
///
|
|
/// Note what that costs, deliberately: a status poll shares the slot with a batch, so
|
|
/// polling during a long import gets 425 until the batch lands. That is honest — this
|
|
/// plane really is doing one thing at a time — and the admin surface (phase 8) is where a
|
|
/// separate status lane would have to argue for itself.
|
|
///
|
|
/// **3. Byte budgets, not counts.** Batches are cut by encoded size
|
|
/// (<see cref="BridgeConfig.AssetBatchBytes"/>, 512 KB), not by item count, because the
|
|
/// ceilings this has to live inside are byte ceilings: the sidecar refuses an inbound line
|
|
/// over 1 MiB, and base64 costs 33% on top of whatever the payload measures.
|
|
/// <see cref="PageBuilder"/> is that budget, and every asset family shares it so the
|
|
/// envelope cannot drift apart between them.
|
|
///
|
|
/// **Emitting from off the Core thread is safe here, and it is worth saying why.**
|
|
/// <c>BridgeLink.Emit</c> enqueues onto a <c>ConcurrentQueue</c> and never touches the
|
|
/// socket, so the enqueue itself is fine. The subtle part is
|
|
/// <c>BridgeIdempotency.Observe</c>, which <c>Emit</c> calls while a keyed command is in
|
|
/// flight: it captures a line only when that line's correlation field **exactly equals**
|
|
/// the open command's correlation value, and correlation values come from one monotonic
|
|
/// counter in the sidecar. An asset reply therefore cannot be mistaken for a keyed
|
|
/// command's reply, whatever the interleaving.
|
|
/// </summary>
|
|
public static class BridgeAssets
|
|
{
|
|
/// <summary>
|
|
/// What version of *our derivation* produced these bytes (§7).
|
|
///
|
|
/// The source gate hashes the operator's client files, which answers "did the inputs
|
|
/// change". It cannot answer "did the way we read them change" — and that is the case
|
|
/// that bites, because a corrected frame offset or a fixed hue application changes
|
|
/// every derived byte while every source file stays identical. So this is folded into
|
|
/// stage 1 alongside the hashes, and bumping it makes the whole working set drift,
|
|
/// which is the intended and correct blast radius.
|
|
///
|
|
/// Bump it whenever extraction changes what it produces from unchanged input. It is
|
|
/// the same rule <c>spawnAtlasSource.js</c>'s `PARSER_VERSION` follows, and it applies
|
|
/// here more rather than less: this pipeline derives far more from far less.
|
|
///
|
|
/// **2** — phase 4 (§4.3, §4.9). The catalogue now falls back to
|
|
/// <c>AnimationFrame*.uop</c> for bodies the legacy path has nothing for, which on a
|
|
/// stock client is 235 new sprites and two of them player-character bodies; and the
|
|
/// player-body set no longer carries ghost ids. Every client file is byte-identical
|
|
/// and the answer is different, which is precisely what this number exists to say.
|
|
///
|
|
/// **3** — phase 6 (§4.10, §11.2). A body with no art at action 0 is catalogued at
|
|
/// the first action that has any, and its key names that action. 74 more bodies on a
|
|
/// stock client, no existing key's bytes changed — but a body that was absent is now
|
|
/// a row, which is the same "unchanged input, different answer" this number covers.
|
|
/// </summary>
|
|
public const int EXTRACTOR_VERSION = 3;
|
|
|
|
// ── the one slot (§3.2) ──────────────────────────────────────────────────────────────
|
|
|
|
private static readonly object _sync = new object();
|
|
private static Thread _worker;
|
|
private static readonly AutoResetEvent _wake = new AutoResetEvent(false);
|
|
private static Action _job;
|
|
private static string _inFlight;
|
|
private static DateTime _inFlightSince;
|
|
private static bool _running;
|
|
|
|
private static long _served, _busied, _failed;
|
|
|
|
// ── the hash cache (§6) ──────────────────────────────────────────────────────────────
|
|
|
|
private static readonly Dictionary<string, CachedHash> _hashes =
|
|
new Dictionary<string, CachedHash>(StringComparer.OrdinalIgnoreCase);
|
|
|
|
private static Thread _hasher;
|
|
private static volatile bool _hashing;
|
|
|
|
private sealed class CachedHash
|
|
{
|
|
public long Size;
|
|
public long MTime;
|
|
public string Sha256;
|
|
}
|
|
|
|
// ── imaging (§4.4) ───────────────────────────────────────────────────────────────────
|
|
|
|
private static bool _imagingChecked;
|
|
private static bool _imagingOk;
|
|
private static string _imagingReason;
|
|
|
|
public static void Initialize()
|
|
{
|
|
if (!BridgeConfig.Enabled)
|
|
return;
|
|
|
|
DisableTheLibraryCache();
|
|
|
|
BridgeBoot.RegisterHandler("assets.sources", OnSources);
|
|
BridgeBoot.RegisterHandler("assets.fetch", OnFetch);
|
|
|
|
// Owned here since phase 7, for the same reason `assets.fetch` moved here in phase 5:
|
|
// it is the transport, and more than one family has something to enumerate.
|
|
BridgeBoot.RegisterHandler("assets.manifest", OnManifest);
|
|
}
|
|
|
|
/// <summary>
|
|
/// **Turns <c>Ultima.Files.CacheData</c> off for the life of the process** (phase 5,
|
|
/// §17.10). One line, and it answers two separate problems that both end in a
|
|
/// confident wrong picture or an out-of-memory shard.
|
|
///
|
|
/// **The poisoning.** <c>Art.GetStatic</c> and <c>Art.GetLand</c> memoise into a
|
|
/// <c>Bitmap[0xFFFF]</c> and hand back **the same instance** on every call, while
|
|
/// <c>Hue.ApplyTo</c> repaints a bitmap **in place**. So hueing a static edits the
|
|
/// library's cached copy: measured on this client, hue item 3922 once and every later
|
|
/// request for the *plain* 3922 comes back hued, and a second hue stacks on the first.
|
|
/// Nothing downstream can see it — the row is the right size, the right shape and the
|
|
/// right id. It is §4.5's failure mode arriving through a completely different door.
|
|
///
|
|
/// **The retention.** That array is never trimmed. Decoding this client's 39,189
|
|
/// statics once would leave 74 MB of <c>Bitmap</c> in a static field of a game server,
|
|
/// kept for as long as the process lives, to serve pictures nobody asked for twice.
|
|
///
|
|
/// The obvious alternative — copy each bitmap before hueing — was rejected, and not
|
|
/// only for the retention: <c>new Bitmap(src)</c> **throws** on the
|
|
/// <c>Format16bppArgb1555</c> these decoders produce, so the copy has to name the
|
|
/// source pixel format explicitly, which is a subtlety on the wrong side of a
|
|
/// correctness boundary.
|
|
///
|
|
/// **What it costs is nothing measurable here.** <c>Animations</c> — the whole of the
|
|
/// body catalogue — does not consult this flag at all, and
|
|
/// <see cref="BridgeCatalog"/> and <see cref="BridgeArt"/> each keep their own cache of
|
|
/// *encoded PNG bytes*, which is the thing worth holding: a tenth of the size, already
|
|
/// hashed, and released when it goes idle.
|
|
///
|
|
/// It is a process-global on a library nothing else in this overlay reads, which is why
|
|
/// setting it here rather than saving and restoring it around each decode is safe —
|
|
/// and a save/restore would not be, because the asset worker is a thread.
|
|
/// </summary>
|
|
private static void DisableTheLibraryCache()
|
|
{
|
|
try
|
|
{
|
|
Files.CacheData = false;
|
|
}
|
|
catch (Exception e)
|
|
{
|
|
// A client this library cannot even open. The families report that for themselves,
|
|
// per key, with a reason; it must not stop the plugin booting.
|
|
Console.WriteLine("[Bridge] assets: could not disable the Ultima bitmap cache: {0}",
|
|
e.Message);
|
|
}
|
|
}
|
|
|
|
public static string Status()
|
|
{
|
|
int cached;
|
|
|
|
lock (_hashes)
|
|
{
|
|
cached = _hashes.Count;
|
|
}
|
|
|
|
lock (_sync)
|
|
{
|
|
return String.Format(
|
|
"assets(served={0} busied={1} failed={2} inFlight={3} hashing={4} cached={5})",
|
|
_served, _busied, _failed, _inFlight ?? "-", _hashing, cached);
|
|
}
|
|
}
|
|
|
|
// ── the request plane ────────────────────────────────────────────────────────────────
|
|
|
|
/// <summary>
|
|
/// Stage 1 of §6: what the shard's client files currently are. No pixels, no assets —
|
|
/// just the gate that lets the website decide whether anything needs importing at all,
|
|
/// because the normal case is a restart that changed nothing and it must cost nothing.
|
|
/// </summary>
|
|
private static void OnSources(Dictionary<string, object> o)
|
|
{
|
|
var reqId = BridgeJson.GetString(o, "reqId");
|
|
|
|
if (reqId == null)
|
|
{
|
|
// Rule 1. Without a correlation id this would land on the event path, be persisted
|
|
// to the sidecar's store and broadcast to every subscriber. Refuse instead.
|
|
Fail(null, "BAD_REQUEST", "assets.sources requires a reqId");
|
|
return;
|
|
}
|
|
|
|
// Stage 1 answers for the whole plane, not for the client files alone: since phase 7
|
|
// an operator can serve the shard's own configuration tree while declining to serve
|
|
// their UO client, and `families` is where a website discovers which. Refused only
|
|
// when there is nothing at all to report.
|
|
if (Families().Count == 0)
|
|
{
|
|
Fail(reqId, "DISABLED", "asset extraction is disabled on this shard");
|
|
return;
|
|
}
|
|
|
|
Accept(reqId, "assets.sources", () => ReplySources(reqId));
|
|
}
|
|
|
|
/// <summary>
|
|
/// Claims the single slot and hands the work to the worker, or answers `bridge.busy`.
|
|
/// Runs on the Core thread and does nothing expensive; <paramref name="job"/> runs on
|
|
/// the worker and must touch no world state.
|
|
/// </summary>
|
|
internal static void Accept(string reqId, string kind, Action job)
|
|
{
|
|
lock (_sync)
|
|
{
|
|
if (_inFlight != null)
|
|
{
|
|
_busied++;
|
|
Busy(reqId, kind);
|
|
return;
|
|
}
|
|
|
|
_inFlight = kind;
|
|
_inFlightSince = DateTime.UtcNow;
|
|
_job = job;
|
|
|
|
try
|
|
{
|
|
EnsureWorker();
|
|
}
|
|
catch (Exception e)
|
|
{
|
|
// The slot is claimed and there is now nothing that will ever free it. Give it
|
|
// back here or this plane answers `bridge.busy` for the life of the process.
|
|
_inFlight = null;
|
|
_job = null;
|
|
|
|
Console.WriteLine("[Bridge] cannot start the asset worker: {0}", e.Message);
|
|
Fail(reqId, "UNAVAILABLE", "the shard could not start its asset worker");
|
|
return;
|
|
}
|
|
}
|
|
|
|
_wake.Set();
|
|
}
|
|
|
|
private static void Busy(string reqId, string kind)
|
|
{
|
|
var held = (DateTime.UtcNow - _inFlightSince).TotalSeconds;
|
|
|
|
var sb = BridgeJson.Begin("bridge.busy");
|
|
|
|
sb.Str("reqId", reqId)
|
|
// `busyKind`, never a second `kind` — `Begin` has already written this frame's own,
|
|
// and a JSON object carrying two makes every parser take the last. Protocol 6 shipped
|
|
// that bug once and it made the sidecar answer 200 for a refusal.
|
|
.Str("busyKind", kind)
|
|
.Num("heldForSec", (long)held)
|
|
.Str("reason", "the asset plane serves one request at a time");
|
|
|
|
BridgeLink.Emit(sb.End());
|
|
}
|
|
|
|
// ── assets.fetch, and the families behind it (§5, phase 5) ───────────────────────────
|
|
|
|
/// <summary>
|
|
/// One family's answer to a fetch. Runs on the asset worker, never the Core thread.
|
|
/// </summary>
|
|
internal delegate void FamilyFetch(string reqId, List<string> keys, string catalog, string cursor);
|
|
|
|
/// <summary>
|
|
/// One family's answer to a manifest walk — everything it can serve, no payload.
|
|
/// Runs on the asset worker, never the Core thread. A family with nothing to
|
|
/// enumerate (statics and land are addressed, not listed) registers none.
|
|
/// </summary>
|
|
internal delegate void FamilyManifest(string reqId, string cursor);
|
|
|
|
/// <summary>
|
|
/// What one §5 key family registered: how to serve it, how to list it, and — since
|
|
/// phase 7 — which operator consent it answers to.
|
|
///
|
|
/// The gate is per family rather than per plane because the planes are not one
|
|
/// consent. `body`, `static` and `land` are the operator's UO CLIENT, licensed from
|
|
/// EA and read off their disk; `tree` is the shard's OWN configuration, which they
|
|
/// wrote. An operator can reasonably want the second published and not the first, and
|
|
/// before this the atlas would have been what silently disappeared when they said so.
|
|
/// </summary>
|
|
private sealed class FamilyReader
|
|
{
|
|
public FamilyFetch Fetch;
|
|
public FamilyManifest Manifest;
|
|
public Func<bool> Enabled;
|
|
public string DisabledReason;
|
|
}
|
|
|
|
private static readonly Dictionary<string, FamilyReader> _families =
|
|
new Dictionary<string, FamilyReader>(StringComparer.Ordinal);
|
|
|
|
/// <summary>
|
|
/// Claims one §5 key family for a reader.
|
|
///
|
|
/// Phase 3 gave <c>assets.fetch</c> to the body catalogue outright, which was right
|
|
/// while there was one family and wrong the moment there were three: the command is
|
|
/// the *transport*, and the family is a property of the key. So the shared parts — the
|
|
/// correlation id, the operator's consent, the key-count ceiling, and deciding which
|
|
/// reader a request belongs to — live here once, and a family only ever sees a list of
|
|
/// keys it owns.
|
|
///
|
|
/// Registration is order-independent on purpose: ServUO calls every
|
|
/// <c>Initialize</c> in an order nothing here controls, and this fills a dictionary the
|
|
/// handler does not read until a request arrives.
|
|
/// </summary>
|
|
internal static void RegisterFamily(string name, FamilyFetch fetch)
|
|
{
|
|
RegisterFamily(name, fetch, null, null, null);
|
|
}
|
|
|
|
/// <summary>
|
|
/// The full registration: a fetch reader, an optional manifest reader, and the
|
|
/// consent this family answers to.
|
|
///
|
|
/// <paramref name="enabled"/> null means the asset plane's own gate
|
|
/// (<c>Bridge.AssetsEnabled</c>), which is what every client-file family wants.
|
|
/// A family that reads something else entirely passes its own.
|
|
/// </summary>
|
|
internal static void RegisterFamily(string name, FamilyFetch fetch, FamilyManifest manifest,
|
|
Func<bool> enabled, string disabledReason)
|
|
{
|
|
lock (_families)
|
|
{
|
|
_families[name] = new FamilyReader
|
|
{
|
|
Fetch = fetch,
|
|
Manifest = manifest,
|
|
Enabled = enabled,
|
|
DisabledReason = disabledReason
|
|
};
|
|
}
|
|
}
|
|
|
|
/// <summary>
|
|
/// The families this shard can serve **right now**, for §6's stage 1 and for
|
|
/// diagnostics.
|
|
///
|
|
/// Filtered by consent rather than by registration, because that is the question the
|
|
/// website is actually asking: a family it can see in this list is one it can fetch.
|
|
/// Listing a family the operator has switched off would turn one clear refusal at
|
|
/// import time into a per-key refusal on every pass, forever — which is exactly the
|
|
/// failure `families` was added in phase 5 to prevent.
|
|
/// </summary>
|
|
internal static List<string> Families()
|
|
{
|
|
var names = new List<string>();
|
|
|
|
lock (_families)
|
|
{
|
|
foreach (var pair in _families)
|
|
{
|
|
if (EnabledFor(pair.Value))
|
|
names.Add(pair.Key);
|
|
}
|
|
}
|
|
|
|
names.Sort(StringComparer.Ordinal);
|
|
return names;
|
|
}
|
|
|
|
private static bool EnabledFor(FamilyReader reader)
|
|
{
|
|
if (reader == null)
|
|
return false;
|
|
|
|
try
|
|
{
|
|
return reader.Enabled == null ? BridgeConfig.AssetsEnabled : reader.Enabled();
|
|
}
|
|
catch
|
|
{
|
|
// A gate that throws is a gate that has not consented.
|
|
return false;
|
|
}
|
|
}
|
|
|
|
private static FamilyReader FamilyFor(string name)
|
|
{
|
|
lock (_families)
|
|
{
|
|
FamilyReader reader;
|
|
return _families.TryGetValue(name, out reader) ? reader : null;
|
|
}
|
|
}
|
|
|
|
/// <summary>
|
|
/// Resolves a named family and answers the request itself when it cannot.
|
|
///
|
|
/// Shared by <c>assets.fetch</c> and <c>assets.manifest</c> so the two cannot drift
|
|
/// apart about what "this shard does not serve that" means — and so the consent check
|
|
/// happens in exactly one place for both.
|
|
/// </summary>
|
|
private static bool Resolve(string reqId, string family, out FamilyReader reader)
|
|
{
|
|
reader = FamilyFor(family);
|
|
|
|
if (reader == null)
|
|
{
|
|
Fail(reqId, "BAD_REQUEST",
|
|
"this shard serves no '" + family + "' asset family (it serves "
|
|
+ String.Join(", ", Families().ToArray()) + ")");
|
|
return false;
|
|
}
|
|
|
|
if (!EnabledFor(reader))
|
|
{
|
|
Fail(reqId, "DISABLED", reader.DisabledReason
|
|
?? "asset extraction is disabled on this shard");
|
|
return false;
|
|
}
|
|
|
|
return true;
|
|
}
|
|
|
|
/// <summary>
|
|
/// The family segment of a §5 key: everything before the first `/`.
|
|
/// </summary>
|
|
internal static string FamilyOfKey(string key)
|
|
{
|
|
if (String.IsNullOrEmpty(key))
|
|
return null;
|
|
|
|
int slash = key.IndexOf('/');
|
|
|
|
return slash <= 0 ? null : key.Substring(0, slash);
|
|
}
|
|
|
|
/// <summary>
|
|
/// §14's `assets.fetch`, for every family.
|
|
///
|
|
/// **The family is derived from the keys and is not a request field.** §5 made the key
|
|
/// the address of an asset, so a request that had to name its family as well would have
|
|
/// two places to be wrong and one of them silent. A batch must be of one family —
|
|
/// mixing them is refused rather than split — because the reply carries a single
|
|
/// `catalog` id, and that id is what stops an operator patching their client mid-import
|
|
/// from stitching one asset set out of two. Two families, two fingerprints, and a reply
|
|
/// that claimed one of them would be lying about the other.
|
|
/// </summary>
|
|
private static void OnFetch(Dictionary<string, object> o)
|
|
{
|
|
var reqId = BridgeJson.GetString(o, "reqId");
|
|
|
|
if (reqId == null)
|
|
{
|
|
Fail(null, "BAD_REQUEST", "assets.fetch requires a reqId");
|
|
return;
|
|
}
|
|
|
|
// The consent check is NOT here any more (phase 7). It cannot be: which consent this
|
|
// request needs is a property of the keys, and the keys have not been read yet. So the
|
|
// shape checks come first and the gate happens in `Resolve`, once the family is known.
|
|
var keys = BridgeJson.GetStringList(o, "keys");
|
|
|
|
if (keys.Count == 0)
|
|
{
|
|
Fail(reqId, "BAD_REQUEST", "assets.fetch requires a non-empty `keys` array");
|
|
return;
|
|
}
|
|
|
|
if (keys.Count > BridgeConfig.AssetFetchKeys)
|
|
{
|
|
Fail(reqId, "BAD_REQUEST",
|
|
"assets.fetch takes at most " + BridgeConfig.AssetFetchKeys
|
|
+ " keys per request (asked for " + keys.Count + ")");
|
|
return;
|
|
}
|
|
|
|
string family = FamilyOfKey(keys[0]);
|
|
|
|
for (int i = 1; i < keys.Count; i++)
|
|
{
|
|
if (String.Equals(FamilyOfKey(keys[i]), family, StringComparison.Ordinal))
|
|
continue;
|
|
|
|
Fail(reqId, "BAD_REQUEST",
|
|
"assets.fetch takes keys of one family per request; this one mixes '"
|
|
+ family + "' with '" + FamilyOfKey(keys[i]) + "'");
|
|
return;
|
|
}
|
|
|
|
FamilyReader reader;
|
|
|
|
if (!Resolve(reqId, family, out reader))
|
|
return;
|
|
|
|
if (reader.Fetch == null)
|
|
{
|
|
Fail(reqId, "BAD_REQUEST",
|
|
"the '" + family + "' family cannot be fetched by key on this shard");
|
|
return;
|
|
}
|
|
|
|
var catalog = BridgeJson.GetString(o, "catalog");
|
|
var cursor = BridgeJson.GetString(o, "cursor");
|
|
FamilyFetch fetch = reader.Fetch;
|
|
|
|
Accept(reqId, "assets.fetch", () => fetch(reqId, keys, catalog, cursor));
|
|
}
|
|
|
|
/// <summary>
|
|
/// §14's `assets.manifest`, for every family that has one.
|
|
///
|
|
/// Phase 3 gave this command to the body catalogue outright and phase 5 learned, for
|
|
/// `assets.fetch`, that the command is the transport and the family is a property of
|
|
/// the key. Phase 7 is where the same lesson lands one level up: the tree family
|
|
/// enumerates its files exactly the way the catalogue enumerates its bodies, and
|
|
/// nothing about the envelope, the cursor or the consent differs between them.
|
|
///
|
|
/// **`family` still defaults to `body`.** A phase-3 website asks without naming one
|
|
/// and must keep getting the catalogue it asked for.
|
|
/// </summary>
|
|
private static void OnManifest(Dictionary<string, object> o)
|
|
{
|
|
var reqId = BridgeJson.GetString(o, "reqId");
|
|
|
|
if (reqId == null)
|
|
{
|
|
Fail(null, "BAD_REQUEST", "assets.manifest requires a reqId");
|
|
return;
|
|
}
|
|
|
|
var family = BridgeJson.GetString(o, "family") ?? "body";
|
|
|
|
FamilyReader reader;
|
|
|
|
if (!Resolve(reqId, family, out reader))
|
|
return;
|
|
|
|
if (reader.Manifest == null)
|
|
{
|
|
// Named rather than defaulted: statics and land are ADDRESSED (§11.1) rather than
|
|
// listed, and a website that asked for a list of 49,152 item graphics has made a
|
|
// mistake it needs told about rather than an empty page it will read as "none".
|
|
Fail(reqId, "BAD_REQUEST",
|
|
"the '" + family + "' family is fetched by key and has no manifest");
|
|
return;
|
|
}
|
|
|
|
var cursor = BridgeJson.GetString(o, "cursor");
|
|
FamilyManifest manifest = reader.Manifest;
|
|
|
|
Accept(reqId, "assets.manifest", () => manifest(reqId, cursor));
|
|
}
|
|
|
|
/// <summary>
|
|
/// ARGB1555 to a PNG with a transparent background.
|
|
///
|
|
/// <c>Frame</c> writes 16-bit ARGB1555: a pixel the sprite does not cover is left as
|
|
/// zero and a pixel it does cover carries the top bit set. Saving that format straight
|
|
/// to PNG asks GDI+ to make the conversion, and what it does with a one-bit alpha
|
|
/// channel varies by platform — on Mono it is a different implementation entirely. A
|
|
/// sprite that came back with a black rectangle behind it would look fine in a test
|
|
/// that only checked the bytes decoded, and wrong on every page that showed it.
|
|
///
|
|
/// So the expansion is done here, explicitly: alpha bit clear becomes fully
|
|
/// transparent, and each 5-bit channel is widened to 8 bits by repeating its high bits
|
|
/// (<c>(c << 3) | (c >> 2)</c>) rather than by shifting alone, which would
|
|
/// cap white at 248 and tint the whole catalogue.
|
|
/// </summary>
|
|
internal static byte[] BitmapToPng(Bitmap source)
|
|
{
|
|
var rect = new Rectangle(0, 0, source.Width, source.Height);
|
|
|
|
if (source.PixelFormat != PixelFormat.Format16bppArgb1555)
|
|
{
|
|
// Not what this library has ever produced. Save it rather than reinterpret it:
|
|
// guessing at an unknown layout is how a catalogue fills with confident nonsense.
|
|
using (var ms = new MemoryStream())
|
|
{
|
|
source.Save(ms, ImageFormat.Png);
|
|
return ms.ToArray();
|
|
}
|
|
}
|
|
|
|
using (var target = new Bitmap(source.Width, source.Height, PixelFormat.Format32bppArgb))
|
|
{
|
|
BitmapData src = source.LockBits(rect, ImageLockMode.ReadOnly, PixelFormat.Format16bppArgb1555);
|
|
BitmapData dst = null;
|
|
|
|
try
|
|
{
|
|
dst = target.LockBits(rect, ImageLockMode.WriteOnly, PixelFormat.Format32bppArgb);
|
|
|
|
var line = new short[source.Width];
|
|
var outLine = new int[source.Width];
|
|
|
|
for (int y = 0; y < source.Height; y++)
|
|
{
|
|
Marshal.Copy(new IntPtr(src.Scan0.ToInt64() + ((long)y * src.Stride)),
|
|
line, 0, source.Width);
|
|
|
|
for (int x = 0; x < source.Width; x++)
|
|
{
|
|
int p = line[x] & 0xFFFF;
|
|
|
|
if ((p & 0x8000) == 0)
|
|
{
|
|
outLine[x] = 0;
|
|
continue;
|
|
}
|
|
|
|
int r = (p >> 10) & 0x1F;
|
|
int g = (p >> 5) & 0x1F;
|
|
int b = p & 0x1F;
|
|
|
|
outLine[x] = unchecked((int)0xFF000000)
|
|
| (((r << 3) | (r >> 2)) << 16)
|
|
| (((g << 3) | (g >> 2)) << 8)
|
|
| ((b << 3) | (b >> 2));
|
|
}
|
|
|
|
Marshal.Copy(outLine, 0, new IntPtr(dst.Scan0.ToInt64() + ((long)y * dst.Stride)),
|
|
source.Width);
|
|
}
|
|
}
|
|
finally
|
|
{
|
|
if (dst != null)
|
|
target.UnlockBits(dst);
|
|
|
|
source.UnlockBits(src);
|
|
}
|
|
|
|
using (var ms = new MemoryStream())
|
|
{
|
|
target.Save(ms, ImageFormat.Png);
|
|
return ms.ToArray();
|
|
}
|
|
}
|
|
}
|
|
|
|
/// <summary>
|
|
/// SHA-256, lowercase hex. Shared because the hash in a manifest row, the hash in a
|
|
/// fetch row and the hash the website stores must be one function.
|
|
/// </summary>
|
|
internal static string Sha256Hex(byte[] bytes)
|
|
{
|
|
using (var sha = SHA256.Create())
|
|
{
|
|
byte[] digest = sha.ComputeHash(bytes);
|
|
var sb = new StringBuilder(digest.Length * 2);
|
|
|
|
foreach (byte b in digest)
|
|
sb.Append(b.ToString("x2", CultureInfo.InvariantCulture));
|
|
|
|
return sb.ToString();
|
|
}
|
|
}
|
|
|
|
/// <summary>
|
|
/// The asset plane's one refusal frame, shared by every family on it.
|
|
///
|
|
/// <paramref name="code"/> is what the sidecar maps to a status, and it exists because
|
|
/// the alternative it replaced — matching on the words in <paramref name="reason"/> —
|
|
/// makes an operator-facing sentence load-bearing. Rewording "disabled" would silently
|
|
/// turn a 403 into a 400. The codes are `DISABLED` (the operator switched this plane
|
|
/// off), `NOT_FOUND` (the shard has no such file), `UNREADABLE` (it has it and cannot
|
|
/// decode it), `UNAVAILABLE` (the shard cannot do this right now) and `BAD_REQUEST`
|
|
/// (the default, and the caller's fault).
|
|
/// </summary>
|
|
internal static void Fail(string reqId, string code, string reason)
|
|
{
|
|
var sb = BridgeJson.Begin("assets.error");
|
|
|
|
if (reqId != null)
|
|
sb.Str("reqId", reqId);
|
|
|
|
sb.Str("code", code)
|
|
.Str("reason", reason);
|
|
|
|
BridgeLink.Emit(sb.End());
|
|
}
|
|
|
|
// ── the worker ───────────────────────────────────────────────────────────────────────
|
|
|
|
/// <summary>
|
|
/// Started on first use rather than at boot, so a shard that never imports an asset
|
|
/// never carries the thread. Caller must hold <see cref="_sync"/>.
|
|
/// </summary>
|
|
private static void EnsureWorker()
|
|
{
|
|
if (_worker != null)
|
|
return;
|
|
|
|
_running = true;
|
|
|
|
_worker = new Thread(WorkLoop)
|
|
{
|
|
Name = "BridgeAssets",
|
|
IsBackground = true
|
|
};
|
|
|
|
_worker.Start();
|
|
}
|
|
|
|
private static void WorkLoop()
|
|
{
|
|
while (_running)
|
|
{
|
|
_wake.WaitOne(1000);
|
|
|
|
Action job;
|
|
|
|
lock (_sync)
|
|
{
|
|
job = _job;
|
|
_job = null;
|
|
}
|
|
|
|
if (job == null)
|
|
continue;
|
|
|
|
try
|
|
{
|
|
job();
|
|
Interlocked.Increment(ref _served);
|
|
}
|
|
catch (Exception e)
|
|
{
|
|
// A handler that throws must still free the slot, or this plane is wedged for
|
|
// the life of the process and every later request answers `bridge.busy`.
|
|
Interlocked.Increment(ref _failed);
|
|
Console.WriteLine("[Bridge] asset worker: {0}: {1}", e.GetType().Name, e.Message);
|
|
}
|
|
finally
|
|
{
|
|
lock (_sync)
|
|
{
|
|
_inFlight = null;
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
// ── assets.sources ───────────────────────────────────────────────────────────────────
|
|
|
|
/// <summary>
|
|
/// The client files whose bytes decide everything downstream.
|
|
///
|
|
/// Resolved through <c>Ultima.Files</c> rather than by joining a configured directory,
|
|
/// because that is what the decoders themselves do — a file this reports is a file
|
|
/// they would actually open.
|
|
/// </summary>
|
|
private static readonly string[] SourceFiles =
|
|
{
|
|
"cliloc.enu",
|
|
"artlegacymul.uop", "art.mul", "artidx.mul",
|
|
"anim.idx", "anim.mul",
|
|
"anim2.idx", "anim2.mul",
|
|
"anim3.idx", "anim3.mul",
|
|
"anim4.idx", "anim4.mul",
|
|
"anim5.idx", "anim5.mul",
|
|
"animationframe1.uop", "animationframe2.uop", "animationframe3.uop",
|
|
"animationframe4.uop", "animationframe6.uop",
|
|
"body.def", "bodyconv.def",
|
|
"hues.mul",
|
|
"verdata.mul"
|
|
};
|
|
|
|
private static void ReplySources(string reqId)
|
|
{
|
|
var sb = BridgeJson.Begin("assets.sources.ok");
|
|
|
|
sb.Str("reqId", reqId)
|
|
.Num("extractorVersion", EXTRACTOR_VERSION)
|
|
// Which of the two consents this shard has given (phase 7). Without it a website
|
|
// whose operator switched client-file extraction off would read an empty `files`
|
|
// array as "your client has no cliloc.enu" — a sentence that sends them looking at
|
|
// their client install for a setting that lives on their shard.
|
|
.Bool("assetsEnabled", BridgeConfig.AssetsEnabled)
|
|
.Bool("treeEnabled", BridgeConfig.TreeEnabled);
|
|
|
|
WriteImaging(sb);
|
|
|
|
// §4.6: whichever of art.mul / artLegacyMUL.uop `FileIndex` would actually open. An
|
|
// operator who added custom graphics to art.mul while the UOP is present is getting
|
|
// nothing, silently, and this is the only place that can tell them so.
|
|
string artData = BridgeAssetValidator.ArtDataPath();
|
|
|
|
sb.Str("artDataFile", artData == null ? null : Path.GetFileName(artData));
|
|
|
|
// Which §5 families this shard can be asked for. Additive, so the protocol stays 8: a
|
|
// consumer that does not read it behaves exactly as it did. One that does can tell an
|
|
// older overlay (bodies only) from this one without discovering it as a refused fetch
|
|
// halfway through a warm pass.
|
|
var families = Families();
|
|
|
|
sb.Append(",\"families\":[");
|
|
|
|
for (int i = 0; i < families.Count; i++)
|
|
{
|
|
if (i > 0)
|
|
sb.Append(',');
|
|
|
|
BridgeJson.Text(sb, families[i]);
|
|
}
|
|
|
|
sb.Append(']');
|
|
|
|
var page = new PageBuilder(sb, "files", BridgeConfig.AssetBatchBytes);
|
|
bool anyMissingHash = false;
|
|
|
|
// The client files are the asset plane's own subject, so they are listed under the
|
|
// asset plane's own consent. A tree-only shard answers this call — that is how its
|
|
// website learns the `tree` family exists — and reports no client files at all,
|
|
// which is the truthful answer to "what may I read here".
|
|
for (int i = 0; BridgeConfig.AssetsEnabled && i < SourceFiles.Length; i++)
|
|
{
|
|
string name = SourceFiles[i];
|
|
string path = ResolvePath(name);
|
|
|
|
if (path == null)
|
|
continue;
|
|
|
|
var item = new StringBuilder(256);
|
|
|
|
item.Append("{\"name\":");
|
|
BridgeJson.Text(item, name);
|
|
item.Append(",\"path\":");
|
|
BridgeJson.Text(item, path);
|
|
|
|
long size = 0, mtime = 0;
|
|
|
|
try
|
|
{
|
|
var info = new FileInfo(path);
|
|
size = info.Length;
|
|
mtime = ToUnixMs(info.LastWriteTimeUtc);
|
|
}
|
|
catch (Exception e)
|
|
{
|
|
item.Append(",\"unreadable\":");
|
|
BridgeJson.Text(item, e.GetType().Name);
|
|
}
|
|
|
|
item.Append(",\"size\":").Append(size.ToString(CultureInfo.InvariantCulture));
|
|
item.Append(",\"mtime\":").Append(mtime.ToString(CultureInfo.InvariantCulture));
|
|
|
|
string hash = CachedHashFor(path, size, mtime);
|
|
|
|
if (hash == null)
|
|
anyMissingHash = true;
|
|
|
|
item.Append(",\"sha256\":");
|
|
BridgeJson.Text(item, hash);
|
|
|
|
// The one diagnostic §4.6 asks for: art.mul is present, and unread.
|
|
if (artData != null
|
|
&& (name == "art.mul" || name == "artidx.mul")
|
|
&& !artData.EndsWith(".mul", StringComparison.OrdinalIgnoreCase))
|
|
{
|
|
item.Append(",\"shadowedBy\":");
|
|
BridgeJson.Text(item, Path.GetFileName(artData));
|
|
}
|
|
|
|
item.Append('}');
|
|
|
|
if (!page.TryAdd(item.ToString(), name))
|
|
break;
|
|
}
|
|
|
|
page.Close();
|
|
|
|
// §6's gate is (size, mtime) first and a content hash only when those differ, because
|
|
// anim.mul and art.mul are 195 MB and 148 MB and a full hash on every status poll
|
|
// would make the admin panel feel broken. It would also blow the sidecar's 10 s reply
|
|
// timeout outright on the first call. So a hash that is not cached is reported `null`
|
|
// and computed in the background: this reply is always fast, and the next poll — after
|
|
// `hashing` goes false — carries the answer.
|
|
if (anyMissingHash)
|
|
StartHashing();
|
|
|
|
sb.Bool("hashing", _hashing);
|
|
sb.Bool("complete", !anyMissingHash);
|
|
|
|
BridgeLink.Emit(sb.End());
|
|
}
|
|
|
|
private static string ResolvePath(string name)
|
|
{
|
|
try
|
|
{
|
|
string path = Files.GetFilePath(name);
|
|
|
|
if (path != null)
|
|
return path;
|
|
}
|
|
catch
|
|
{
|
|
// Ultima's lookup reads the registry on Windows; a host where that throws still
|
|
// has the directories ServUO itself booted from, which is what the fallback uses.
|
|
}
|
|
|
|
// `Ultima.Files` has a fixed table of file names that predates UOP animations, so it
|
|
// answers null for every `AnimationFrame*.uop` however present they are (§4.3). Phase
|
|
// 4 added those to this list, so the fallback is what makes their size, mtime and hash
|
|
// reachable at all.
|
|
return BridgeUop.FindClientFile(name);
|
|
}
|
|
|
|
private static long ToUnixMs(DateTime utc)
|
|
{
|
|
return (long)(utc - new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc)).TotalMilliseconds;
|
|
}
|
|
|
|
// ── imaging (§4.4) ───────────────────────────────────────────────────────────────────
|
|
|
|
/// <summary>
|
|
/// Whether this host can turn a record into a picture at all.
|
|
///
|
|
/// ServUO targets net48, so a Linux shard runs it under Mono, where
|
|
/// <c>System.Drawing</c> is a thin layer over **libgdiplus** — and §4.2 put
|
|
/// <c>System.Drawing</c> in the *decode* path, not merely the encode: <c>Frame</c>
|
|
/// writes ARGB1555 through a <c>LockBits</c> pointer. Without that library a Linux
|
|
/// shard cannot read a sprite at all, while clilocs and the atlas are unaffected
|
|
/// because neither touches pixels.
|
|
///
|
|
/// It must never present as a stack trace or a 500. It is a named, actionable outcome
|
|
/// in the same family as the cliloc reader's `COMPRESSED`, and it is reported here —
|
|
/// on the *source gate*, the first call any import makes — so an operator learns it
|
|
/// while setting the shard up rather than from an empty bestiary weeks later.
|
|
/// </summary>
|
|
private static void WriteImaging(StringBuilder sb)
|
|
{
|
|
CheckImaging();
|
|
|
|
sb.Append(",\"imaging\":{\"ok\":").Append(_imagingOk ? "true" : "false");
|
|
|
|
if (!_imagingOk)
|
|
{
|
|
sb.Append(",\"code\":\"NO_IMAGING\",\"reason\":");
|
|
BridgeJson.Text(sb,
|
|
"This shard host cannot render images — Mono's System.Drawing needs libgdiplus. "
|
|
+ "Install it (apt-get install libgdiplus) and re-run the import. Cliloc and atlas "
|
|
+ "import are unaffected. (" + _imagingReason + ")");
|
|
}
|
|
|
|
sb.Append('}');
|
|
}
|
|
|
|
/// <summary>
|
|
/// Whether this host can produce a picture, for the families that produce pictures.
|
|
///
|
|
/// <see cref="WriteImaging"/> reports this on the source gate so an operator learns it
|
|
/// while setting the shard up. The catalogue needs the same answer as a *decision* —
|
|
/// it must refuse rather than throw a <c>DllNotFoundException</c> out of the middle of
|
|
/// a decode loop — so the check itself is shared and this is its one accessor.
|
|
/// </summary>
|
|
internal static bool ImagingOk(out string reason)
|
|
{
|
|
CheckImaging();
|
|
reason = _imagingReason;
|
|
return _imagingOk;
|
|
}
|
|
|
|
private static void CheckImaging()
|
|
{
|
|
if (_imagingChecked)
|
|
return;
|
|
|
|
_imagingChecked = true;
|
|
|
|
try
|
|
{
|
|
TouchImaging();
|
|
_imagingOk = true;
|
|
}
|
|
catch (Exception e)
|
|
{
|
|
// On a host with no libgdiplus this is a TypeInitializationException wrapping a
|
|
// DllNotFoundException, and it can surface as the method failing to JIT rather
|
|
// than as a throw from inside it — which is why the construction lives in its own
|
|
// method, so the failure is contained here instead of taking this class's
|
|
// static initialisation with it.
|
|
_imagingOk = false;
|
|
_imagingReason = e.GetType().Name + ": " + e.Message;
|
|
}
|
|
}
|
|
|
|
private static void TouchImaging()
|
|
{
|
|
using (var bmp = new System.Drawing.Bitmap(1, 1))
|
|
{
|
|
bmp.SetPixel(0, 0, System.Drawing.Color.Black);
|
|
}
|
|
}
|
|
|
|
// ── the hash cache (§6) ──────────────────────────────────────────────────────────────
|
|
|
|
private static string CachedHashFor(string path, long size, long mtime)
|
|
{
|
|
lock (_hashes)
|
|
{
|
|
CachedHash cached;
|
|
|
|
if (_hashes.TryGetValue(path, out cached)
|
|
&& cached.Size == size
|
|
&& cached.MTime == mtime)
|
|
{
|
|
return cached.Sha256;
|
|
}
|
|
}
|
|
|
|
return null;
|
|
}
|
|
|
|
/// <summary>
|
|
/// Rehashes whatever the cache is missing, on its own thread.
|
|
///
|
|
/// Deliberately **not** a job on the asset worker: hashing 343 MB takes seconds to
|
|
/// tens of seconds, and holding the single slot for that long would answer every
|
|
/// status poll `bridge.busy` for the whole pass — which is exactly the moment an
|
|
/// operator is watching the panel. It emits nothing and correlates with nothing; it
|
|
/// only fills the cache that the next `assets.sources` reads.
|
|
/// </summary>
|
|
private static void StartHashing()
|
|
{
|
|
lock (_sync)
|
|
{
|
|
if (_hashing)
|
|
return;
|
|
|
|
_hashing = true;
|
|
|
|
_hasher = new Thread(HashLoop)
|
|
{
|
|
Name = "BridgeAssetHash",
|
|
IsBackground = true
|
|
};
|
|
|
|
_hasher.Start();
|
|
}
|
|
}
|
|
|
|
private static void HashLoop()
|
|
{
|
|
try
|
|
{
|
|
for (int i = 0; i < SourceFiles.Length; i++)
|
|
{
|
|
string path = ResolvePath(SourceFiles[i]);
|
|
|
|
if (path == null)
|
|
continue;
|
|
|
|
long size, mtime;
|
|
|
|
try
|
|
{
|
|
var info = new FileInfo(path);
|
|
size = info.Length;
|
|
mtime = ToUnixMs(info.LastWriteTimeUtc);
|
|
}
|
|
catch
|
|
{
|
|
continue;
|
|
}
|
|
|
|
if (CachedHashFor(path, size, mtime) != null)
|
|
continue;
|
|
|
|
string hash = HashFile(path);
|
|
|
|
if (hash == null)
|
|
continue;
|
|
|
|
lock (_hashes)
|
|
{
|
|
_hashes[path] = new CachedHash { Size = size, MTime = mtime, Sha256 = hash };
|
|
}
|
|
}
|
|
}
|
|
catch (Exception e)
|
|
{
|
|
Console.WriteLine("[Bridge] asset hash pass: {0}: {1}", e.GetType().Name, e.Message);
|
|
}
|
|
finally
|
|
{
|
|
// Under _sync, matching StartHashing: cleared outside it, two passes could both
|
|
// pass the guard and hash the same 343 MB twice.
|
|
lock (_sync)
|
|
{
|
|
_hashing = false;
|
|
}
|
|
}
|
|
}
|
|
|
|
private static string HashFile(string path)
|
|
{
|
|
try
|
|
{
|
|
using (var sha = SHA256.Create())
|
|
using (var stream = new FileStream(
|
|
path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite, 1 << 20))
|
|
{
|
|
var buffer = new byte[1 << 20];
|
|
int read;
|
|
|
|
while ((read = stream.Read(buffer, 0, buffer.Length)) > 0)
|
|
sha.TransformBlock(buffer, 0, read, null, 0);
|
|
|
|
sha.TransformFinalBlock(buffer, 0, 0);
|
|
|
|
return Hex(sha.Hash);
|
|
}
|
|
}
|
|
catch (Exception e)
|
|
{
|
|
Console.WriteLine("[Bridge] cannot hash {0}: {1}", path, e.Message);
|
|
return null;
|
|
}
|
|
}
|
|
|
|
private static string Hex(byte[] bytes)
|
|
{
|
|
var sb = new StringBuilder(bytes.Length * 2);
|
|
|
|
for (int i = 0; i < bytes.Length; i++)
|
|
sb.Append(bytes[i].ToString("x2", CultureInfo.InvariantCulture));
|
|
|
|
return sb.ToString();
|
|
}
|
|
|
|
// ── the paging envelope (§3.3) ───────────────────────────────────────────────────────
|
|
|
|
/// <summary>
|
|
/// **One envelope for every asset family**, defined here in phase 1 so that clilocs
|
|
/// (phase 2), the body catalogue (3), statics and land (5), deep animation keys (6)
|
|
/// and the ServUO tree files (7) all page the same way. They are otherwise five
|
|
/// chances to invent five slightly different shapes, and the website would have to
|
|
/// learn each one.
|
|
///
|
|
/// The envelope a reply closes with:
|
|
///
|
|
/// <code>
|
|
/// "items": [ … ],
|
|
/// "more": true, // ask again with this cursor
|
|
/// "cursor": "s:4104", // opaque to everyone but the shard; absent when more:false
|
|
/// "cut": "budget" // budget | end | limit — WHY this page stopped
|
|
/// </code>
|
|
///
|
|
/// **The budget is bytes, and it is UTF-8 bytes.** Not item count, because the ceiling
|
|
/// this lives inside is the sidecar's inbound line cap; and not chars, because a
|
|
/// cliloc row is real text and a `StringBuilder`'s Length would undercount every
|
|
/// non-ASCII character in it.
|
|
///
|
|
/// `cut` exists because "the page is short" has three different meanings and the
|
|
/// website must not have to guess which: the source ran out (`end`), the byte budget
|
|
/// was spent (`budget`), or the family stopped at its own limit (`limit`). Only the
|
|
/// first means the import is finished.
|
|
///
|
|
/// **The first item is always admitted**, even if it alone exceeds the budget.
|
|
/// Otherwise an oversized item would make its family unable to make any progress at
|
|
/// all — it would be skipped for the budget on every page, forever. That is safe
|
|
/// precisely because the budget is set to half the sidecar's line cap
|
|
/// (<see cref="BridgeConfig.AssetBatchBytes"/>), so one such item still fits the wire.
|
|
/// </summary>
|
|
public sealed class PageBuilder
|
|
{
|
|
private readonly StringBuilder _sb;
|
|
private readonly int _budget;
|
|
private int _bytes;
|
|
private int _count;
|
|
private string _cursor;
|
|
private string _cut = "end";
|
|
|
|
/// <summary>
|
|
/// Room kept back for the fields the envelope must still be able to write after
|
|
/// the last item — `more`, `cursor`, `cut` and the closing brace. Without it a
|
|
/// page could fill the budget exactly and then overrun it closing itself.
|
|
/// </summary>
|
|
private const int Reserve = 256;
|
|
|
|
public PageBuilder(StringBuilder sb, string arrayName, int budget)
|
|
{
|
|
_sb = sb;
|
|
_budget = budget;
|
|
|
|
sb.Append(",\"").Append(arrayName).Append("\":[");
|
|
|
|
// The prefix is already written, and it counts: the cap the sidecar enforces is
|
|
// on the whole line, not on the array.
|
|
_bytes = Encoding.UTF8.GetByteCount(sb.ToString());
|
|
}
|
|
|
|
public int Count { get { return _count; } }
|
|
|
|
/// <summary>
|
|
/// Adds one already-serialised item. <paramref name="cursorAfter"/> is where the
|
|
/// family should resume if this turns out to be the last item on the page.
|
|
/// Returns false when the budget is spent — the caller stops, and `more` is true.
|
|
/// </summary>
|
|
public bool TryAdd(string item, string cursorAfter)
|
|
{
|
|
if (item == null)
|
|
return true;
|
|
|
|
int cost = Encoding.UTF8.GetByteCount(item) + (_count > 0 ? 1 : 0);
|
|
|
|
if (_count > 0 && _bytes + cost + Reserve > _budget)
|
|
{
|
|
_cut = "budget";
|
|
return false;
|
|
}
|
|
|
|
if (_count > 0)
|
|
_sb.Append(',');
|
|
|
|
_sb.Append(item);
|
|
|
|
_bytes += cost;
|
|
_count++;
|
|
_cursor = cursorAfter;
|
|
|
|
return true;
|
|
}
|
|
|
|
/// <summary>
|
|
/// Stops the page for a reason of the family's own — a per-request limit, say —
|
|
/// rather than because the budget ran out.
|
|
/// </summary>
|
|
public void Cut(string why)
|
|
{
|
|
_cut = why;
|
|
}
|
|
|
|
public void Close()
|
|
{
|
|
bool more = _cut != "end";
|
|
|
|
_sb.Append(']');
|
|
_sb.Bool("more", more);
|
|
_sb.Str("cut", _cut);
|
|
|
|
if (more && _cursor != null)
|
|
_sb.Str("cursor", _cursor);
|
|
}
|
|
}
|
|
}
|
|
}
|