The spawn atlas was the one place the platform's rule -- only the sidecar
bridges the shard -- was broken, and it was broken by the component that faces
the internet: SPAWN_ATLAS.md required the website to read the ServUO tree off a
bind mount or a shared volume. This serves those files over the loopback link
instead (docs/link/v8.md 10).
The measurement came first and changed the shape. 10 said the shard would serve
`tree/<label>` -> bytes; against a stock 57.4 tree it cannot. Spawns/trammel.xml
is 4.03 MB, the sidecar discards any inbound line over 1 MiB, and that file as
one base64 row is 5.4 MiB -- it would be dropped, time out, and be re-requested
forever with no error anywhere. Two files on a STOCK tree are in that state.
So a file crosses as 512 KiB chunks, each gzipped: tree/Spawns/trammel.xml/c0
and so on, which is 5's depth scheme doing the same job it does for
body/400/a0/f0 and needing no protocol change to do it. The chunk is the bound
and the compression is only the saving -- nothing guarantees an operator's files
compress, so the ceiling has to hold when they do not, and a 512 KiB chunk that
refuses to compress is still ~683 KiB of base64, inside the wire cap that
AssetBatchBytes' deliberate factor of two leaves room for.
It is a `tree` FAMILY on assets.fetch rather than 14's separate tree.* commands:
phase 5 had already learned that the command is the transport and the family is
a property of the key, and assets.manifest is generalised here the same way.
That reuses the single slot, the paging envelope, the key ceiling and the
mid-import guard -- and leaves `link` with nothing to do for the third phase
running.
But it gets its OWN consent, Bridge.TreeEnabled. AssetsEnabled is an operator
agreeing the website may read their EA-licensed UO client; this is the shard's
own configuration, which they wrote, and which the public bestiary is built
from. One switch could not express both, and the thing that would silently
disappear for an operator who declined the first is their spawn atlas. So the
consent check moved into the family lookup, and assets.sources answers whenever
either plane is on, reporting `families` filtered to what is actually enabled --
which is how a tree-only shard's website discovers there is anything to ask for.
Two defects found, and which harness found which is the part worth keeping:
- An empty `catalog` is not an absent one. `expected != null` refused every
fetch from a caller that sent "", with a sentence naming no catalog at all.
Found by an offline probe that passed one by accident.
- GZipStream writes NOTHING for zero bytes of input -- the header is emitted
lazily, so a stream opened and closed without a write yields a zero-length
buffer rather than the 20-byte empty member. Stock ServUO ships two empty
decoration files, so this broke every import off an untouched tree. The
offline probe reassembled all 141 files and reported success, because .NET's
own decompressor reads an empty stream as empty data and the chunk's
declared length (0) and hash (of nothing) both agreed. Only the live walk,
through a reader on another runtime, disagreed.
Measured end to end against a live shard, the real sidecar and the website's own
reader: 141 files, 11,895,427 bytes, 158 chunks, 3 pages, 1.33 MB on the wire,
512 ms; every file byte-identical to disk; the atlas built over the bridge
identical to the one built off it. A drift check is the manifest alone -- 32 KB,
~70 ms, no file bytes.
The label set is this shard's, never the caller's: a fetch resolves against the
set the shard itself enumerated, and tree/../../Scripts/..., Config/Bridge.cfg
and Saves/Accounts/accounts.xml are all answered `absent` before a path is built
out of them.
Protocol stays 8 and EXTRACTOR_VERSION stays 3 -- this family derives nothing,
it forwards an operator's own file unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
778 lines
33 KiB
C#
778 lines
33 KiB
C#
using System;
|
|
using System.Collections.Generic;
|
|
using System.Globalization;
|
|
using System.IO;
|
|
using System.IO.Compression;
|
|
using System.Text;
|
|
|
|
namespace Server.Custom.Bridge
|
|
{
|
|
/// <summary>
|
|
/// **The shard's own configuration, over the bridge** (docs/link/v8.md §10 — protocol 8,
|
|
/// phase 7).
|
|
///
|
|
/// Everything else on the asset plane reads the operator's UO CLIENT. This family reads
|
|
/// the shard's own files: the spawn tables, the region and location definitions, the
|
|
/// champion list and the decoration lists. The website parses those into its spawn atlas —
|
|
/// where every creature lives, which regions exist, what this shard calls scenery — and
|
|
/// until protocol 8 it did so by **reading the ServUO tree off a shared filesystem**:
|
|
/// same host, a bind mount, or a shared volume.
|
|
///
|
|
/// That was the one place the platform's own rule was broken, and broken by the component
|
|
/// that faces the internet. This closes it. The parsers do not move — `spawnAtlasParse.js`
|
|
/// is pure, fs-free and covered by CI without a ServUO tree anywhere near it, and every
|
|
/// quirk it handles stays exactly where it is. The shard sends bytes; the website still
|
|
/// decides what they mean.
|
|
///
|
|
/// ── What phase 7 measured, and the shape it forced ────────────────────────────────
|
|
///
|
|
/// §10 said "the shard serves `tree/<label>` → bytes". Measured against a stock 57.4
|
|
/// tree, it cannot: `Spawns/trammel.xml` is **4.03 MB**, the sidecar discards any inbound
|
|
/// line over **1 MiB** (`shard.rs` `MAX_INBOUND_LINE_BYTES`), and that file as a single
|
|
/// base64 row is 5.4 MiB. It would never arrive — the reply would be discarded, the
|
|
/// request would time out, and the import would retry forever with no error anywhere in
|
|
/// it. Two files on a *stock* tree are in that state; a shard with hand-built spawn tables
|
|
/// has more.
|
|
///
|
|
/// So a file crosses as **chunks, each gzipped**:
|
|
///
|
|
/// <code>
|
|
/// tree/Spawns/trammel.xml the manifest row — size, hash, chunk count
|
|
/// tree/Spawns/trammel.xml/c0 the first 512 KiB of it, gzipped
|
|
/// tree/Spawns/trammel.xml/c1 the next
|
|
/// </code>
|
|
///
|
|
/// which is §5's depth scheme at work a second time, exactly as `body/400/a0/f0` is —
|
|
/// and, as there, nothing about it needed a protocol change.
|
|
///
|
|
/// **The chunk is the bound and the compression is the saving**, and it matters which is
|
|
/// which. Compression is what makes this cheap: the stock tree is 11.34 MB and gzips to
|
|
/// 927 KB, so the whole atlas source arrives in about three pages instead of thirty-one.
|
|
/// But nothing guarantees that an operator's files compress at all, so the ceiling has to
|
|
/// hold when they do not — and it does, because a 512 KiB chunk that refuses to compress
|
|
/// is still only ~683 KiB of base64, inside the wire cap that
|
|
/// <see cref="BridgeConfig.AssetBatchBytes"/>' deliberate factor of two leaves room for.
|
|
/// A design that leaned on the ratio would work on every tree anyone tested and fail on
|
|
/// the first one nobody did.
|
|
///
|
|
/// ── Two rules that are not negotiable here ────────────────────────────────────────
|
|
///
|
|
/// **1. The label set is this shard's, never the caller's.** This is the only family on
|
|
/// this link whose keys look like paths, and the website is the internet-facing component.
|
|
/// So nothing here joins a path that arrived on the wire: a fetch resolves its label
|
|
/// against the set <see cref="Enumerate"/> itself produced, and a label that is not in it
|
|
/// is refused — before any file is opened, and whatever it spells. The five groups are
|
|
/// fixed in code, the extensions are fixed in code, and the resolved path is checked to be
|
|
/// under the tree root even after all of that.
|
|
///
|
|
/// **2. A row re-declares its own address.** Each chunk carries its label, its index, its
|
|
/// byte offset and the hash of its own (uncompressed) bytes, and the manifest carries the
|
|
/// hash of the whole file. That is the §4.10 lesson on a fourth axis: a reassembly that
|
|
/// silently put chunk 3 where chunk 4 belongs would produce a file that parses — XML is
|
|
/// forgiving about what it skips — and a spawn atlas subtly missing a facet. Per-chunk
|
|
/// hashes make it a named error instead.
|
|
/// </summary>
|
|
public static class BridgeTree
|
|
{
|
|
/// <summary>The §5 key family this serves.</summary>
|
|
private const string Family = "tree";
|
|
|
|
/// <summary>
|
|
/// The five labelled groups `spawnAtlasSource.js` reads, and nothing else.
|
|
///
|
|
/// Fixed in code rather than configured, because a configurable list is a way for the
|
|
/// website to ask for a file this shard never meant to publish. An operator who wants
|
|
/// a different tree served wants a different feature.
|
|
/// </summary>
|
|
private static readonly string[] SingleFiles =
|
|
{
|
|
"Data/Regions.xml",
|
|
"Config/ChampionSpawns.xml"
|
|
};
|
|
|
|
private const string LocationsDir = "Data/Locations";
|
|
private const string SpawnsDir = "Spawns";
|
|
private const string DecorationDir = "Data/Decoration";
|
|
|
|
public static void Initialize()
|
|
{
|
|
if (!BridgeConfig.Enabled)
|
|
return;
|
|
|
|
// Its own consent, not the asset plane's (§10, phase 7). An operator who declines to
|
|
// serve their UO client still gets a spawn atlas, because these are their own files.
|
|
BridgeAssets.RegisterFamily(Family, ReplyFetch, ReplyManifest,
|
|
() => BridgeConfig.TreeEnabled,
|
|
"the shard's configuration tree is not served (Bridge.TreeEnabled is off)");
|
|
}
|
|
|
|
// ── the file set ─────────────────────────────────────────────────────────────────────
|
|
|
|
private sealed class TreeFile
|
|
{
|
|
public string Label;
|
|
public string Path;
|
|
public long Bytes;
|
|
public long MTime;
|
|
}
|
|
|
|
/// <summary>
|
|
/// Every atlas source file this shard has, tree-relative and forward-slashed.
|
|
///
|
|
/// The labels are `spawnAtlasSource.js`'s own, character for character, because they
|
|
/// are what the website keys its stored fingerprint on: the same tree read here and
|
|
/// read there has to produce the same label or every import looks like a change.
|
|
/// Forward slashes for the same reason — a Windows shard and a Linux one must agree.
|
|
/// </summary>
|
|
private static List<TreeFile> Enumerate()
|
|
{
|
|
string root = Core.BaseDirectory;
|
|
var files = new List<TreeFile>();
|
|
|
|
foreach (string label in SingleFiles)
|
|
Add(files, root, label);
|
|
|
|
foreach (string label in ListByExtension(root, LocationsDir, ".xml"))
|
|
Add(files, root, label);
|
|
|
|
foreach (string label in ListByExtension(root, SpawnsDir, ".xml"))
|
|
Add(files, root, label);
|
|
|
|
foreach (string label in ListTree(root, DecorationDir, ".cfg"))
|
|
Add(files, root, label);
|
|
|
|
return files;
|
|
}
|
|
|
|
private static void Add(List<TreeFile> files, string root, string label)
|
|
{
|
|
string path = Resolve(root, label);
|
|
|
|
if (path == null)
|
|
return;
|
|
|
|
try
|
|
{
|
|
var info = new FileInfo(path);
|
|
|
|
if (!info.Exists)
|
|
return;
|
|
|
|
files.Add(new TreeFile
|
|
{
|
|
Label = label,
|
|
Path = path,
|
|
Bytes = info.Length,
|
|
MTime = ToUnixMs(info.LastWriteTimeUtc)
|
|
});
|
|
}
|
|
catch (Exception e)
|
|
{
|
|
// A file the shard cannot stat is a file it cannot serve. Say so once, here,
|
|
// rather than as a refused row on every import pass forever.
|
|
Console.WriteLine("[Bridge] tree: cannot read {0}: {1}", label, e.Message);
|
|
}
|
|
}
|
|
|
|
/// <summary>One directory's files with the given extension, sorted, as labels.</summary>
|
|
private static List<string> ListByExtension(string root, string dir, string extension)
|
|
{
|
|
var labels = new List<string>();
|
|
string full = Path.Combine(root, dir.Replace('/', Path.DirectorySeparatorChar));
|
|
|
|
try
|
|
{
|
|
if (!Directory.Exists(full))
|
|
return labels;
|
|
|
|
foreach (string path in Directory.GetFiles(full))
|
|
{
|
|
string name = Path.GetFileName(path);
|
|
|
|
if (name.EndsWith(extension, StringComparison.OrdinalIgnoreCase))
|
|
labels.Add(dir + "/" + name);
|
|
}
|
|
}
|
|
catch (Exception e)
|
|
{
|
|
Console.WriteLine("[Bridge] tree: cannot list {0}: {1}", dir, e.Message);
|
|
}
|
|
|
|
labels.Sort(StringComparer.Ordinal);
|
|
return labels;
|
|
}
|
|
|
|
/// <summary>
|
|
/// One directory tree's files with the given extension, recursively.
|
|
///
|
|
/// Recursive because `Data/Decoration` nests two deep in places (`Magincia/Trammel`,
|
|
/// `Stygian Abyss/Ter Mur`, `Old/Britannia`), and the website's own reader says why
|
|
/// that matters: a flat read indexes a third of what the shard has, and the failure is
|
|
/// an authoring dropdown quietly missing whole expansions rather than an error anyone
|
|
/// would notice.
|
|
/// </summary>
|
|
private static List<string> ListTree(string root, string dir, string extension)
|
|
{
|
|
var labels = new List<string>();
|
|
string full = Path.Combine(root, dir.Replace('/', Path.DirectorySeparatorChar));
|
|
|
|
try
|
|
{
|
|
if (!Directory.Exists(full))
|
|
return labels;
|
|
|
|
foreach (string path in Directory.GetFiles(full, "*", SearchOption.AllDirectories))
|
|
{
|
|
if (!path.EndsWith(extension, StringComparison.OrdinalIgnoreCase))
|
|
continue;
|
|
|
|
string rel = path.Substring(full.Length).Replace('\\', '/').TrimStart('/');
|
|
|
|
if (rel.Length > 0)
|
|
labels.Add(dir + "/" + rel);
|
|
}
|
|
}
|
|
catch (Exception e)
|
|
{
|
|
Console.WriteLine("[Bridge] tree: cannot walk {0}: {1}", dir, e.Message);
|
|
}
|
|
|
|
labels.Sort(StringComparer.Ordinal);
|
|
return labels;
|
|
}
|
|
|
|
/// <summary>
|
|
/// A label to a path on this host, or null if it is not one this shard serves.
|
|
///
|
|
/// Rule 1 of the class doc lives here. The label has already been matched against the
|
|
/// enumerated set by the time a fetch calls this, and this still refuses anything with
|
|
/// a traversal segment, a drive or a root in it, and still checks that what
|
|
/// <c>Path.GetFullPath</c> produced is under the tree root. Three checks for one rule
|
|
/// because the cost of being wrong once is the website reading an arbitrary file off a
|
|
/// game server's disk.
|
|
/// </summary>
|
|
private static string Resolve(string root, string label)
|
|
{
|
|
if (String.IsNullOrEmpty(label) || label.IndexOf('\\') >= 0)
|
|
return null;
|
|
|
|
string[] segments = label.Split('/');
|
|
|
|
foreach (string segment in segments)
|
|
{
|
|
if (segment.Length == 0 || segment == "." || segment == "..")
|
|
return null;
|
|
}
|
|
|
|
if (Path.IsPathRooted(label))
|
|
return null;
|
|
|
|
try
|
|
{
|
|
string rootFull = Path.GetFullPath(root);
|
|
string full = Path.GetFullPath(Path.Combine(rootFull,
|
|
label.Replace('/', Path.DirectorySeparatorChar)));
|
|
|
|
if (!rootFull.EndsWith(Path.DirectorySeparatorChar.ToString(CultureInfo.InvariantCulture),
|
|
StringComparison.Ordinal))
|
|
{
|
|
rootFull += Path.DirectorySeparatorChar;
|
|
}
|
|
|
|
return full.StartsWith(rootFull, StringComparison.OrdinalIgnoreCase) ? full : null;
|
|
}
|
|
catch
|
|
{
|
|
return null;
|
|
}
|
|
}
|
|
|
|
// ── the fingerprint ──────────────────────────────────────────────────────────────────
|
|
|
|
/// <summary>
|
|
/// What the whole tree currently is, in sixteen hex characters.
|
|
///
|
|
/// The same job <c>BridgeCatalog.SourceId</c> does for client files, and the same
|
|
/// reason: it goes on every page of a walk, and a page whose id differs from the
|
|
/// first's means the operator edited a spawn file while it was being read. Half of
|
|
/// what arrived then describes a tree that no longer exists and nothing later can tell
|
|
/// which half, so the website refuses the import outright rather than stitching one.
|
|
///
|
|
/// Built from (label, size, mtime) rather than from content hashes, because it is
|
|
/// computed on every page and hashing the tree's contents each time would spend a
|
|
/// tenth of a second per page to answer a question (size, mtime) answers for free.
|
|
/// The CONTENT hashes are still sent — once, per file, on the manifest — which is
|
|
/// where the website's own drift gate reads them from.
|
|
/// </summary>
|
|
private static string FingerprintOf(List<TreeFile> files)
|
|
{
|
|
var sb = new StringBuilder(256);
|
|
|
|
sb.Append(files.Count);
|
|
|
|
foreach (TreeFile file in files)
|
|
{
|
|
sb.Append('|').Append(file.Label)
|
|
.Append(':').Append(file.Bytes.ToString(CultureInfo.InvariantCulture))
|
|
.Append(':').Append(file.MTime.ToString(CultureInfo.InvariantCulture));
|
|
}
|
|
|
|
return BridgeAssets.Sha256Hex(Encoding.UTF8.GetBytes(sb.ToString())).Substring(0, 16);
|
|
}
|
|
|
|
// ── assets.manifest, for this family ─────────────────────────────────────────────────
|
|
|
|
/// <summary>
|
|
/// Worker thread. Every file this shard would serve, with its size, its content hash
|
|
/// and how many chunks it takes — and no bytes.
|
|
///
|
|
/// That separation is what makes the normal case free. The website stores these
|
|
/// hashes; on the next import it asks for this list again, compares, and fetches
|
|
/// nothing at all when nothing moved — which on a shard whose maps are not being
|
|
/// edited is every import.
|
|
///
|
|
/// A stock tree is 141 rows and fits in one page comfortably. It pages anyway, by the
|
|
/// same envelope as every other family, because the day a shard has three thousand
|
|
/// decoration files is not the day to discover this was the one walk that could not
|
|
/// end.
|
|
/// </summary>
|
|
private static void ReplyManifest(string reqId, string cursor)
|
|
{
|
|
List<TreeFile> files = Enumerate();
|
|
string fingerprint = FingerprintOf(files);
|
|
|
|
int from = ParseCursor(cursor);
|
|
|
|
if (from < 0 || from > files.Count)
|
|
from = 0;
|
|
|
|
var sb = BridgeJson.Begin("assets.manifest.ok");
|
|
|
|
sb.Str("reqId", reqId)
|
|
.Str("family", Family)
|
|
.Str("catalog", fingerprint)
|
|
.Num("chunkBytes", BridgeConfig.TreeChunkBytes)
|
|
.Num("total", files.Count)
|
|
.Num("from", from);
|
|
|
|
var page = new BridgeAssets.PageBuilder(sb, "rows", BridgeConfig.AssetBatchBytes);
|
|
|
|
int i = from;
|
|
|
|
for (; i < files.Count; i++)
|
|
{
|
|
TreeFile file = files[i];
|
|
string hash = HashFile(file.Path);
|
|
|
|
var item = new StringBuilder(256);
|
|
|
|
item.Append("{\"key\":");
|
|
BridgeJson.Text(item, Family + "/" + file.Label);
|
|
item.Append(",\"label\":");
|
|
BridgeJson.Text(item, file.Label);
|
|
item.Append(",\"bytes\":").Append(file.Bytes.ToString(CultureInfo.InvariantCulture));
|
|
item.Append(",\"mtime\":").Append(file.MTime.ToString(CultureInfo.InvariantCulture));
|
|
item.Append(",\"chunks\":").Append(
|
|
ChunkCount(file.Bytes).ToString(CultureInfo.InvariantCulture));
|
|
item.Append(",\"sha256\":");
|
|
BridgeJson.Text(item, hash);
|
|
item.Append('}');
|
|
|
|
if (!page.TryAdd(item.ToString(), "t:" + (i + 1).ToString(CultureInfo.InvariantCulture)))
|
|
break;
|
|
}
|
|
|
|
page.Close();
|
|
|
|
sb.Num("sent", page.Count);
|
|
|
|
BridgeLink.Emit(sb.End());
|
|
}
|
|
|
|
/// <summary>
|
|
/// How many chunks a file of this size takes.
|
|
///
|
|
/// **An empty file is one chunk, not none.** `Data/Locations` can legitimately hold an
|
|
/// empty file, and zero chunks would make it a manifest row the website could never
|
|
/// fetch: it would wait for content that has no address, and report the import
|
|
/// incomplete forever.
|
|
/// </summary>
|
|
private static int ChunkCount(long bytes)
|
|
{
|
|
long chunk = BridgeConfig.TreeChunkBytes;
|
|
long count = (bytes + chunk - 1) / chunk;
|
|
|
|
return count < 1 ? 1 : (int)count;
|
|
}
|
|
|
|
// ── assets.fetch, for this family ────────────────────────────────────────────────────
|
|
|
|
/// <summary>
|
|
/// Worker thread. The bytes for an explicit list of chunk keys.
|
|
///
|
|
/// Chunks are read with a seek rather than by holding the file, so the memory this
|
|
/// costs a running game server is one chunk regardless of how large an operator's
|
|
/// spawn tables are. A 4 MB file served eight times over is eight seeks and eight
|
|
/// 512 KiB reads — cheaper than caching it would be, and with no cache to invalidate
|
|
/// when the operator edits it mid-pass.
|
|
/// </summary>
|
|
private static void ReplyFetch(string reqId, List<string> keys, string expected, string cursor)
|
|
{
|
|
List<TreeFile> files = Enumerate();
|
|
string fingerprint = FingerprintOf(files);
|
|
|
|
// `IsNullOrEmpty`, not `!= null`. A caller that has no fingerprint to assert sends
|
|
// the field absent OR empty depending on how its own client serialises a missing
|
|
// value, and the two must mean the same thing — an empty string compared against a
|
|
// real id refuses every fetch, with a sentence that names no catalog at all
|
|
// ("catalog is now 8159778b"). Found by a probe that passed one.
|
|
if (!String.IsNullOrEmpty(expected) && expected != fingerprint)
|
|
{
|
|
// The tree moved between the manifest and this fetch. The same refusal the
|
|
// catalogue makes for a patched client, and for the same reason: these keys were
|
|
// chosen against a listing that no longer describes what is on disk.
|
|
BridgeAssets.Fail(reqId, "UNREADABLE",
|
|
"the shard's configuration tree changed since that manifest was read (catalog "
|
|
+ expected + " is now " + fingerprint + "); start the import again");
|
|
return;
|
|
}
|
|
|
|
var byLabel = new Dictionary<string, TreeFile>(StringComparer.Ordinal);
|
|
|
|
foreach (TreeFile file in files)
|
|
byLabel[file.Label] = file;
|
|
|
|
int from = ParseCursor(cursor);
|
|
|
|
if (from < 0 || from > keys.Count)
|
|
from = 0;
|
|
|
|
var sb = BridgeJson.Begin("assets.fetch.ok");
|
|
|
|
sb.Str("reqId", reqId)
|
|
.Str("family", Family)
|
|
.Str("catalog", fingerprint)
|
|
.Num("chunkBytes", BridgeConfig.TreeChunkBytes)
|
|
.Num("asked", keys.Count)
|
|
.Num("from", from);
|
|
|
|
var page = new BridgeAssets.PageBuilder(sb, "rows", BridgeConfig.AssetBatchBytes);
|
|
|
|
int i = from;
|
|
|
|
for (; i < keys.Count; i++)
|
|
{
|
|
string item = Render(byLabel, keys[i]);
|
|
|
|
if (!page.TryAdd(item, "t:" + (i + 1).ToString(CultureInfo.InvariantCulture)))
|
|
break;
|
|
}
|
|
|
|
page.Close();
|
|
|
|
sb.Num("sent", page.Count);
|
|
|
|
BridgeLink.Emit(sb.End());
|
|
}
|
|
|
|
/// <summary>
|
|
/// One key to one row.
|
|
///
|
|
/// A key this shard cannot serve is a row rather than a failed request, exactly as in
|
|
/// every other family, and `status` keeps the two kinds apart: `absent` is a file this
|
|
/// shard does not have (a tree with no `ChampionSpawns.xml` is a normal tree), and
|
|
/// `unsupported` is a key shape this family does not serve — which is a website bug,
|
|
/// and is counted separately so it cannot hide inside the expected gaps.
|
|
/// </summary>
|
|
private static string Render(Dictionary<string, TreeFile> byLabel, string key)
|
|
{
|
|
string label;
|
|
int chunk;
|
|
|
|
if (!ParseKey(key, out label, out chunk))
|
|
return Refusal(key, "unsupported", "not a tree chunk key (tree/<label>/c<n>)");
|
|
|
|
TreeFile file;
|
|
|
|
if (!byLabel.TryGetValue(label, out file))
|
|
{
|
|
// Rule 1: the label has to be one THIS shard enumerated. Anything else is refused
|
|
// here, before a path is built out of it, whatever it spells.
|
|
return Refusal(key, "absent", "this shard does not serve that file");
|
|
}
|
|
|
|
int chunks = ChunkCount(file.Bytes);
|
|
|
|
if (chunk < 0 || chunk >= chunks)
|
|
{
|
|
return Refusal(key, "unsupported",
|
|
"chunk " + chunk.ToString(CultureInfo.InvariantCulture) + " of "
|
|
+ chunks.ToString(CultureInfo.InvariantCulture));
|
|
}
|
|
|
|
long offset = (long)chunk * BridgeConfig.TreeChunkBytes;
|
|
byte[] raw;
|
|
|
|
try
|
|
{
|
|
raw = ReadChunk(file.Path, offset, BridgeConfig.TreeChunkBytes);
|
|
}
|
|
catch (Exception e)
|
|
{
|
|
Console.WriteLine("[Bridge] tree: cannot read {0} chunk {1}: {2}", label, chunk, e.Message);
|
|
return Refusal(key, "absent", e.GetType().Name);
|
|
}
|
|
|
|
byte[] packed;
|
|
|
|
try
|
|
{
|
|
packed = Gzip(raw);
|
|
}
|
|
catch (Exception e)
|
|
{
|
|
Console.WriteLine("[Bridge] tree: cannot compress {0} chunk {1}: {2}", label, chunk, e.Message);
|
|
return Refusal(key, "absent", e.GetType().Name);
|
|
}
|
|
|
|
var item = new StringBuilder(packed.Length * 2);
|
|
|
|
item.Append("{\"key\":");
|
|
BridgeJson.Text(item, key);
|
|
item.Append(",\"status\":\"ok\",\"label\":");
|
|
BridgeJson.Text(item, label);
|
|
item.Append(",\"chunk\":").Append(chunk.ToString(CultureInfo.InvariantCulture));
|
|
item.Append(",\"chunks\":").Append(chunks.ToString(CultureInfo.InvariantCulture));
|
|
item.Append(",\"offset\":").Append(offset.ToString(CultureInfo.InvariantCulture));
|
|
item.Append(",\"bytes\":").Append(raw.Length.ToString(CultureInfo.InvariantCulture));
|
|
item.Append(",\"sha256\":");
|
|
BridgeJson.Text(item, BridgeAssets.Sha256Hex(raw));
|
|
item.Append(",\"gzip\":");
|
|
BridgeJson.Text(item, Convert.ToBase64String(packed));
|
|
item.Append('}');
|
|
|
|
return item.ToString();
|
|
}
|
|
|
|
private static string Refusal(string key, string status, string reason)
|
|
{
|
|
var item = new StringBuilder(128);
|
|
|
|
item.Append("{\"key\":");
|
|
BridgeJson.Text(item, key);
|
|
item.Append(",\"status\":");
|
|
BridgeJson.Text(item, status);
|
|
item.Append(",\"reason\":");
|
|
BridgeJson.Text(item, reason);
|
|
item.Append('}');
|
|
|
|
return item.ToString();
|
|
}
|
|
|
|
/// <summary>
|
|
/// `tree/<label>/c<n>` into its label and chunk index.
|
|
///
|
|
/// The label itself contains slashes, so the chunk segment is taken off the END rather
|
|
/// than by counting segments from the front. That is unambiguous here and not by
|
|
/// luck: every label this family serves ends in `.xml` or `.cfg`, so no label's last
|
|
/// segment can be spelled `c` followed by digits.
|
|
/// </summary>
|
|
private static bool ParseKey(string key, out string label, out int chunk)
|
|
{
|
|
label = null;
|
|
chunk = -1;
|
|
|
|
if (String.IsNullOrEmpty(key))
|
|
return false;
|
|
|
|
string prefix = Family + "/";
|
|
|
|
if (!key.StartsWith(prefix, StringComparison.Ordinal))
|
|
return false;
|
|
|
|
int slash = key.LastIndexOf('/');
|
|
|
|
if (slash <= prefix.Length - 1)
|
|
return false;
|
|
|
|
string last = key.Substring(slash + 1);
|
|
|
|
if (last.Length < 2 || last[0] != 'c')
|
|
return false;
|
|
|
|
for (int i = 1; i < last.Length; i++)
|
|
{
|
|
if (last[i] < '0' || last[i] > '9')
|
|
return false;
|
|
}
|
|
|
|
if (!Int32.TryParse(last.Substring(1), NumberStyles.None, CultureInfo.InvariantCulture, out chunk))
|
|
return false;
|
|
|
|
label = key.Substring(prefix.Length, slash - prefix.Length);
|
|
|
|
return label.Length > 0;
|
|
}
|
|
|
|
private static int ParseCursor(string cursor)
|
|
{
|
|
if (String.IsNullOrEmpty(cursor) || !cursor.StartsWith("t:", StringComparison.Ordinal))
|
|
return 0;
|
|
|
|
int value;
|
|
|
|
return Int32.TryParse(cursor.Substring(2), NumberStyles.None,
|
|
CultureInfo.InvariantCulture, out value) ? value : 0;
|
|
}
|
|
|
|
// ── bytes ────────────────────────────────────────────────────────────────────────────
|
|
|
|
private static byte[] ReadChunk(string path, long offset, int length)
|
|
{
|
|
using (var stream = new FileStream(path, FileMode.Open, FileAccess.Read,
|
|
FileShare.ReadWrite, 1 << 16))
|
|
{
|
|
long remaining = stream.Length - offset;
|
|
|
|
if (remaining < 0)
|
|
remaining = 0;
|
|
|
|
if (remaining > length)
|
|
remaining = length;
|
|
|
|
var buffer = new byte[remaining];
|
|
|
|
stream.Seek(offset, SeekOrigin.Begin);
|
|
|
|
int filled = 0;
|
|
|
|
while (filled < buffer.Length)
|
|
{
|
|
int read = stream.Read(buffer, filled, buffer.Length - filled);
|
|
|
|
// A short read is not the end of the file here — the length was taken from the
|
|
// stream itself. Stopping on one would hand back a chunk whose declared length
|
|
// and real length disagree, which the website would only see as a hash
|
|
// mismatch on a file it cannot name a cause for.
|
|
if (read <= 0)
|
|
break;
|
|
|
|
filled += read;
|
|
}
|
|
|
|
if (filled == buffer.Length)
|
|
return buffer;
|
|
|
|
var exact = new byte[filled];
|
|
Buffer.BlockCopy(buffer, 0, exact, 0, filled);
|
|
return exact;
|
|
}
|
|
}
|
|
|
|
/// <summary>
|
|
/// A complete gzip member for exactly one empty chunk.
|
|
///
|
|
/// **`GZipStream` writes NOTHING for zero bytes of input**, on .NET Framework and on
|
|
/// Mono: the gzip header is emitted lazily on the first write, so a stream that is
|
|
/// opened and closed without one produces a zero-length buffer rather than the 20-byte
|
|
/// empty member. That is not a valid gzip stream, and the reader at the other end says
|
|
/// so — `zlib: unexpected end of file`.
|
|
///
|
|
/// It is not a hypothetical: **stock ServUO 57.4 ships two empty decoration files**
|
|
/// (`Felucca/ambitious solen queen quest.cfg` and
|
|
/// `Tokuno/terrible hatchlings quest.cfg`), so every import off an untouched tree hit
|
|
/// it. Worth knowing how it was found, because it says something about probes: an
|
|
/// offline harness reassembled all 141 files and reported success, since .NET's own
|
|
/// decompressor treats an empty stream as empty data and the chunk's declared length
|
|
/// (0) and hash (of nothing) both agreed with that. Only the live walk, through a
|
|
/// reader on a different runtime, disagreed.
|
|
///
|
|
/// The alternative — letting an empty chunk carry an empty payload and teaching the
|
|
/// reader to expect it — was rejected: it puts a special case on the wire, where every
|
|
/// future reader has to know it, instead of in the one place that builds the bytes.
|
|
/// Header (magic, deflate, no flags, no mtime, no XFL, unknown OS), one empty stored
|
|
/// block, then CRC32 and ISIZE of nothing.
|
|
/// </summary>
|
|
private static readonly byte[] EmptyGzip =
|
|
{
|
|
0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xff,
|
|
0x03, 0x00,
|
|
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
|
|
};
|
|
|
|
private static byte[] Gzip(byte[] raw)
|
|
{
|
|
if (raw.Length == 0)
|
|
return EmptyGzip;
|
|
|
|
using (var ms = new MemoryStream())
|
|
{
|
|
using (var gz = new GZipStream(ms, CompressionMode.Compress, true))
|
|
gz.Write(raw, 0, raw.Length);
|
|
|
|
return ms.ToArray();
|
|
}
|
|
}
|
|
|
|
/// <summary>
|
|
/// The content hash of one file, streamed.
|
|
///
|
|
/// Streamed rather than <c>File.ReadAllBytes</c> because this runs once per file per
|
|
/// manifest, and a stock tree's spawn files are 10 MB between them: reading them whole
|
|
/// would put that much through a game server's large object heap to produce 141 short
|
|
/// strings.
|
|
/// </summary>
|
|
private static string HashFile(string path)
|
|
{
|
|
try
|
|
{
|
|
using (var sha = System.Security.Cryptography.SHA256.Create())
|
|
using (var stream = new FileStream(path, FileMode.Open, FileAccess.Read,
|
|
FileShare.ReadWrite, 1 << 16))
|
|
{
|
|
var buffer = new byte[1 << 16];
|
|
int read;
|
|
|
|
while ((read = stream.Read(buffer, 0, buffer.Length)) > 0)
|
|
sha.TransformBlock(buffer, 0, read, null, 0);
|
|
|
|
sha.TransformFinalBlock(buffer, 0, 0);
|
|
|
|
var sb = new StringBuilder(64);
|
|
|
|
foreach (byte b in sha.Hash)
|
|
sb.Append(b.ToString("x2", CultureInfo.InvariantCulture));
|
|
|
|
return sb.ToString();
|
|
}
|
|
}
|
|
catch (Exception e)
|
|
{
|
|
Console.WriteLine("[Bridge] tree: cannot hash {0}: {1}", path, e.Message);
|
|
return null;
|
|
}
|
|
}
|
|
|
|
private static long ToUnixMs(DateTime utc)
|
|
{
|
|
return (long)(utc - new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc)).TotalMilliseconds;
|
|
}
|
|
|
|
/// <summary>For `[Bridge] status`, the same one-line shape every other family reports.</summary>
|
|
public static string Status()
|
|
{
|
|
if (!BridgeConfig.TreeEnabled)
|
|
return "tree(disabled)";
|
|
|
|
List<TreeFile> files = Enumerate();
|
|
long bytes = 0;
|
|
|
|
foreach (TreeFile file in files)
|
|
bytes += file.Bytes;
|
|
|
|
return String.Format("tree(files={0} bytes={1} catalog={2})",
|
|
files.Count, bytes, FingerprintOf(files));
|
|
}
|
|
}
|
|
}
|