feat(shard): resolve cliloc names for items and reward titles
Protocol 3.0 §8.6 (docs/link/v3.md), the dependency order 5 was sequenced
behind. Items on the wire carry a LabelNumber, not a name — the bridge has
always sent it (char.profile.equipment.cliloc, reward titles as a cliloc
number in string form, and one per marketplace listing) but the site had no
table to resolve it against, so a character sheet could only render
`id 1023721` where the game renders "quarter staff".
The number was never the missing piece. The table was.
Sourced from a file the operator converts once from their own client, at a
path from the `cliloc_client_path` setting falling back to UO_CLIENT_PATH.
Nothing client-derived is committed: UO's strings are EA's, exactly as the
creature sprites are. A shard with nothing configured is fully supported —
names render as ids, as they did before.
The conversion step is not avoidable, and that is the substantive finding
here: every current client ships its cliloc files COMPRESSED (first DWORD's
high byte 0x8E, the Mythic container), and ServUO's own bundled
Ultima.StringList cannot read that either — so VendorSearch.GetItemName is
already inert on such a shard and the plugin could not supply names instead.
v3.md's original "read the client's Cliloc.enu" recommendation was therefore
not implementable as written, and its committed db/data/clilocs.json artifact
also predates the Part C corrections (no committed derived snapshots, nothing
EA-derived shipped). Replaced with the spawn-atlas pattern: parse on boot from
an operator-configured path, hash-gated, output gitignored.
- utils/clilocParse.js — pure parsers, fs-free so the suite runs in CI.
Accepts the plain binary layout and delimited text, sniffed by header rather
than extension. Rejects a compressed file BY NAME: without that check the
plain parser reads it as ~19k records of negative ids and 60 KB "strings"
before dying mid-file, and the resulting error names the wrong problem.
displayText() drops the ~1_val~ arguments the bridge never sends.
- utils/clilocSource.js — the fs layer. hashSource reports `compressed` so the
admin panel can flag an unconverted file WITHOUT parsing 5 MB per poll;
otherwise pointing at a client directory reports a healthy file with pending
drift ("ready to import") and the operator only finds out on failure.
- model/shardClilocs — refresh/status/lookup. All-or-nothing replace (DELETE,
not TRUNCATE — TRUNCATE is DDL in MariaDB and implicitly commits). Batched
server-side resolution behind a capped cache; never throws, because a cliloc
lookup is decoration on a character sheet.
- Deliberately NO staged-approval flow, unlike the atlas: the atlas escalates
facet loss because a half-copied tree and a real map change are
indistinguishable from inside the process, whereas a partial cliloc copy
makes the parser fail on a truncated record. The ambiguity the atlas must
escalate is one this parser simply detects.
- No public route. The table is never served AS a table: 67k rows would dwarf
any page using them, and the Android client consumes the same resolved JSON.
Two parser bugs found by building it, both now covered by tests: trimming a
text line before splitting ate the trailing separator on empty-text entries
and silently dropped 55,994 of 123,490 while still reporting success; and
Number('') is 0, not NaN, so a line starting with a separator imported as a
bogus cliloc 0.
Verified against the real client table (123,490 entries) and the live MariaDB:
import 663 ms, hash-gated boot no-op 14 ms, cold resolve 4.2 ms / warm 0.015 ms.
Binary and TSV imports converge on the same 67,496 rows with identical keys
(blank entries — half the table — are dropped at import). A file truncated to
half its length is refused with TRUNCATED and leaves the previous table
serving. Boot logs verified for both the import and the compressed-file
warning; neither blocks startup. All three admin routes exercised over HTTP
with a real session. 629 server tests pass; client builds clean; swagger,
routes.manifest.json and routes.guards.json regenerated.
Not covered by an automated test: the character sheet renders resolved names
in presentational React with no DOM test harness in this repo, and was not
rendered against a live linked-player profile — that needs a logged-in player
with a linked game account and a shard answering a profile RPC.
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
140
server/tools/cliloc-export/Program.cs
Normal file
140
server/tools/cliloc-export/Program.cs
Normal file
@@ -0,0 +1,140 @@
|
||||
// Cliloc export — converts a modern client's COMPRESSED Cliloc.enu into the
|
||||
// plain format the website can read (docs/website/CLILOCS.md).
|
||||
//
|
||||
// Why this exists at all: every current UO client ships its cliloc files in the
|
||||
// compressed "Mythic" format — the first DWORD's high byte is 0x8E — and the
|
||||
// plain layout the website parses is what those files looked like before that
|
||||
// change. Decompressing is a bit-level inverse-BWT coder that the site has no
|
||||
// business carrying at runtime, and ServUO's own bundled `Ultima.StringList`
|
||||
// cannot read it either (which is why `VendorSearch.GetItemName` is already
|
||||
// inert on such a shard, and why the shard cannot supply names instead).
|
||||
//
|
||||
// So the conversion happens ONCE, here, against a decompressor that already
|
||||
// exists and is maintained: UOFiddler's `Ultima.dll`.
|
||||
//
|
||||
// ── Why reflection rather than a project reference ────────────────────────
|
||||
//
|
||||
// UOFiddler ships as net10.0. Referencing it from a project built by an older
|
||||
// SDK fails at COMPILE time with CS1705 ("uses System.Runtime 10.0 which has a
|
||||
// higher version than referenced assembly"). Loading it reflectively moves that
|
||||
// question to run time, where `RollForward: LatestMajor` answers it — so this
|
||||
// builds on whatever SDK an operator happens to have and runs on the newest
|
||||
// runtime installed.
|
||||
//
|
||||
// ── Why not StringList.SaveStringList ────────────────────────────────────
|
||||
//
|
||||
// It looks like exactly the right method and it is not: it RE-COMPRESSES on
|
||||
// save, because its purpose is round-tripping a file back into the client. The
|
||||
// output is byte-identical to the compressed input. The plain records below are
|
||||
// written by hand for that reason.
|
||||
//
|
||||
// Usage:
|
||||
// dotnet run -- <Ultima.dll> <Cliloc.enu> <output> [--tsv]
|
||||
//
|
||||
// Nothing produced by this tool is committed. See docs/website/CLILOCS.md.
|
||||
|
||||
using System;
|
||||
using System.Collections;
|
||||
using System.IO;
|
||||
using System.Reflection;
|
||||
using System.Text;
|
||||
|
||||
internal static class Program
|
||||
{
|
||||
private static int Main(string[] args)
|
||||
{
|
||||
if (args.Length < 3)
|
||||
{
|
||||
Console.Error.WriteLine("usage: clilocexport <path-to-Ultima.dll> <cliloc-file> <output-file> [--tsv]");
|
||||
Console.Error.WriteLine(" Ultima.dll ships with UOFiddler (https://github.com/polserver/UOFiddler).");
|
||||
return 2;
|
||||
}
|
||||
|
||||
var (ultimaDll, input, output) = (args[0], args[1], args[2]);
|
||||
var asTsv = Array.IndexOf(args, "--tsv") >= 0;
|
||||
|
||||
// The language code only names the file when StringList resolves the path
|
||||
// itself; here the path is explicit, so it is cosmetic.
|
||||
var language = Path.GetExtension(input).TrimStart('.');
|
||||
if (string.IsNullOrWhiteSpace(language)) language = "enu";
|
||||
|
||||
var assembly = Assembly.LoadFrom(Path.GetFullPath(ultimaDll));
|
||||
var stringListType = assembly.GetType("Ultima.StringList")
|
||||
?? throw new InvalidOperationException("Ultima.StringList not found — is that really UOFiddler's Ultima.dll?");
|
||||
|
||||
// (language, path, decompress). `decompress: true` is the whole point;
|
||||
// the loader falls back to a plain read on its own if the file turns out
|
||||
// not to be compressed, so an already-converted file passes through.
|
||||
var ctor = stringListType.GetConstructor(new[] { typeof(string), typeof(string), typeof(bool) })
|
||||
?? throw new InvalidOperationException("Unexpected Ultima.StringList API — this tool targets UOFiddler 4.21+.");
|
||||
|
||||
var stringList = ctor.Invoke(new object[] { language, Path.GetFullPath(input), true });
|
||||
|
||||
// A partial parse is reported rather than thrown. Surfacing it matters:
|
||||
// the output would otherwise be a quietly short table, which is exactly
|
||||
// the failure mode the website's parser refuses to import.
|
||||
var warning = stringListType.GetProperty("LoadWarning")?.GetValue(stringList) as string;
|
||||
if (!string.IsNullOrWhiteSpace(warning)) Console.Error.WriteLine("warning: " + warning);
|
||||
|
||||
var entries = (IEnumerable)stringListType.GetProperty("Entries")!.GetValue(stringList)!;
|
||||
var entryType = assembly.GetType("Ultima.StringEntry")!;
|
||||
var numberProp = entryType.GetProperty("Number")!;
|
||||
var textProp = entryType.GetProperty("Text")!;
|
||||
var flagProp = entryType.GetProperty("Flag")!;
|
||||
|
||||
int written = 0, skipped = 0, maxBytes = 0;
|
||||
|
||||
if (asTsv)
|
||||
{
|
||||
using var writer = new StreamWriter(output, false, new UTF8Encoding(false));
|
||||
foreach (var entry in entries)
|
||||
{
|
||||
var number = (int)numberProp.GetValue(entry)!;
|
||||
var text = (string?)textProp.GetValue(entry) ?? "";
|
||||
maxBytes = Math.Max(maxBytes, Encoding.UTF8.GetByteCount(text));
|
||||
// A tab or newline inside a cliloc string would break the row.
|
||||
// Neither occurs in real tables, but silently emitting a broken
|
||||
// file is worse than collapsing the whitespace.
|
||||
writer.WriteLine($"{number}\t{text.Replace('\t', ' ').Replace('\r', ' ').Replace('\n', ' ')}");
|
||||
written++;
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
using var stream = new FileStream(output, FileMode.Create, FileAccess.Write);
|
||||
using var binary = new BinaryWriter(stream);
|
||||
binary.Write(2); // int32 — the plain-format version marker
|
||||
binary.Write((short)1); // int16 — language marker
|
||||
|
||||
foreach (var entry in entries)
|
||||
{
|
||||
var number = (int)numberProp.GetValue(entry)!;
|
||||
var text = (string?)textProp.GetValue(entry) ?? "";
|
||||
var flag = Convert.ToByte(Convert.ToInt32(flagProp.GetValue(entry)));
|
||||
|
||||
var utf8 = Encoding.UTF8.GetBytes(text);
|
||||
maxBytes = Math.Max(maxBytes, utf8.Length);
|
||||
|
||||
// The length field is 16 bits. Real tables peak around 12 KB, so
|
||||
// this has never fired — but writing a truncated length would
|
||||
// corrupt every record after it, so an oversize entry is dropped
|
||||
// and counted instead.
|
||||
if (utf8.Length > ushort.MaxValue) { skipped++; continue; }
|
||||
|
||||
binary.Write(number);
|
||||
binary.Write(flag);
|
||||
binary.Write((ushort)utf8.Length);
|
||||
binary.Write(utf8);
|
||||
written++;
|
||||
}
|
||||
}
|
||||
|
||||
Console.WriteLine($"wrote {written} entries to {output} (maxTextBytes={maxBytes}, skippedOversize={skipped})");
|
||||
if (written == 0)
|
||||
{
|
||||
Console.Error.WriteLine("no entries were written — is that a cliloc file?");
|
||||
return 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user