feat(shard): resolve cliloc names for items and reward titles

Protocol 3.0 §8.6 (docs/link/v3.md), the dependency order 5 was sequenced
behind. Items on the wire carry a LabelNumber, not a name — the bridge has
always sent it (char.profile.equipment.cliloc, reward titles as a cliloc
number in string form, and one per marketplace listing) but the site had no
table to resolve it against, so a character sheet could only render
`id 1023721` where the game renders "quarter staff".

The number was never the missing piece. The table was.

Sourced from a file the operator converts once from their own client, at a
path from the `cliloc_client_path` setting falling back to UO_CLIENT_PATH.
Nothing client-derived is committed: UO's strings are EA's, exactly as the
creature sprites are. A shard with nothing configured is fully supported —
names render as ids, as they did before.

The conversion step is not avoidable, and that is the substantive finding
here: every current client ships its cliloc files COMPRESSED (first DWORD's
high byte 0x8E, the Mythic container), and ServUO's own bundled
Ultima.StringList cannot read that either — so VendorSearch.GetItemName is
already inert on such a shard and the plugin could not supply names instead.
v3.md's original "read the client's Cliloc.enu" recommendation was therefore
not implementable as written, and its committed db/data/clilocs.json artifact
also predates the Part C corrections (no committed derived snapshots, nothing
EA-derived shipped). Replaced with the spawn-atlas pattern: parse on boot from
an operator-configured path, hash-gated, output gitignored.

- utils/clilocParse.js — pure parsers, fs-free so the suite runs in CI.
  Accepts the plain binary layout and delimited text, sniffed by header rather
  than extension. Rejects a compressed file BY NAME: without that check the
  plain parser reads it as ~19k records of negative ids and 60 KB "strings"
  before dying mid-file, and the resulting error names the wrong problem.
  displayText() drops the ~1_val~ arguments the bridge never sends.
- utils/clilocSource.js — the fs layer. hashSource reports `compressed` so the
  admin panel can flag an unconverted file WITHOUT parsing 5 MB per poll;
  otherwise pointing at a client directory reports a healthy file with pending
  drift ("ready to import") and the operator only finds out on failure.
- model/shardClilocs — refresh/status/lookup. All-or-nothing replace (DELETE,
  not TRUNCATE — TRUNCATE is DDL in MariaDB and implicitly commits). Batched
  server-side resolution behind a capped cache; never throws, because a cliloc
  lookup is decoration on a character sheet.
- Deliberately NO staged-approval flow, unlike the atlas: the atlas escalates
  facet loss because a half-copied tree and a real map change are
  indistinguishable from inside the process, whereas a partial cliloc copy
  makes the parser fail on a truncated record. The ambiguity the atlas must
  escalate is one this parser simply detects.
- No public route. The table is never served AS a table: 67k rows would dwarf
  any page using them, and the Android client consumes the same resolved JSON.

Two parser bugs found by building it, both now covered by tests: trimming a
text line before splitting ate the trailing separator on empty-text entries
and silently dropped 55,994 of 123,490 while still reporting success; and
Number('') is 0, not NaN, so a line starting with a separator imported as a
bogus cliloc 0.

Verified against the real client table (123,490 entries) and the live MariaDB:
import 663 ms, hash-gated boot no-op 14 ms, cold resolve 4.2 ms / warm 0.015 ms.
Binary and TSV imports converge on the same 67,496 rows with identical keys
(blank entries — half the table — are dropped at import). A file truncated to
half its length is refused with TRUNCATED and leaves the previous table
serving. Boot logs verified for both the import and the compressed-file
warning; neither blocks startup. All three admin routes exercised over HTTP
with a real session. 629 server tests pass; client builds clean; swagger,
routes.manifest.json and routes.guards.json regenerated.

Not covered by an automated test: the character sheet renders resolved names
in presentational React with no DOM test harness in this repo, and was not
rendered against a live linked-player profile — that needs a logged-in player
with a linked game account and a shard answering a profile RPC.

Co-Authored-By: Claude <noreply@anthropic.com>

server/src/utils/clilocParse.js

@@ -0,0 +1,283 @@
+// Cliloc parsing — the pure half.
+//
+// A "cliloc" is UO's localization table: an integer id mapped to a display
+// string. Items carry a `LabelNumber` rather than a name, so without this table
+// the site can only render `id 1023721` where the game shows "quarter staff".
+// The shard already sends the id on every equipment entry (`char.profile`'s
+// `cliloc` field) and will send one per marketplace listing — the *number* was
+// never the missing piece, the *table* was.
+//
+// This module is fs-free on purpose, exactly like `spawnAtlasParse.js`: the
+// suite runs in CI where there is no UO client, so every parser here is driven
+// from inline fixtures. `clilocSource.js` is the only thing that touches disk.
+//
+// ── Two input formats, and why ─────────────────────────────────────────────
+//
+// The client's own `Cliloc.enu` is COMPRESSED (Mythic format) on any modern
+// client, and decompressing it is a bit-level port of an inverse-BWT coder that
+// nothing in this stack needs at runtime. ServUO's own bundled `Ultima.StringList`
+// cannot read it either — which is why `VendorSearch.GetItemName` is already inert
+// on such a shard and the plugin could not supply names even if we asked it to.
+//
+// So the operator converts once, from their own client, and points the site at
+// the result (see docs/website/CLILOCS.md). Two shapes are accepted because
+// different tools produce different things:
+//
+//   • PLAIN BINARY — the pre-compression cliloc layout: a 6-byte header, then
+//     records of {int32 number, byte flag, uint16 length, UTF-8 bytes}.
+//   • DELIMITED TEXT — `number<TAB|,|;>text` per line, which is what the common
+//     GUI exports emit. Quoted CSV fields and a header row are tolerated.
+//
+// Nothing derived from the client is ever committed: the converted file lives at
+// an operator-supplied path and is gitignored, the same rule the spawn atlas art
+// map already follows.
+
+/** Raised for a file we can identify but deliberately refuse to guess at. */
+class ClilocFormatError extends Error {
+  constructor(message, code) {
+    super(message)
+    this.name = 'ClilocFormatError'
+    this.code = code
+  }
+}
+
+/**
+ * Bumped when this parser produces DIFFERENT data from an IDENTICAL source file.
+ *
+ * Stored beside the source hash so the boot path can tell "same file, but the
+ * parser moved on" from "same file, nothing to do". Without it a corrected parse
+ * would ship and never reach an install whose cliloc file never changes — the
+ * trap `spawnAtlasSource.PARSER_VERSION` documents.
+ */
+const PARSER_VERSION = 1
+
+// The plain layout's header is `02 00 00 00 01 00` — a 4-byte version and a
+// 2-byte language marker. Only the size matters for parsing; the values are
+// checked to sniff the format, not to validate it.
+const HEADER_BYTES = 6
+const RECORD_HEADER_BYTES = 7 // int32 number + byte flag + uint16 length
+
+// Every compressed cliloc file the client ships begins with a DWORD whose high
+// byte is 0x8E (the XOR key UOFiddler calls `HeaderXorKey`, 0x8E2C9A3D). That is
+// the single cheapest way to tell an operator they exported the wrong file —
+// without it, the plain parser happily reads compressed bytes as ~19k records of
+// negative ids and 60 KB "strings" before dying somewhere in the middle, and the
+// resulting error names the wrong problem.
+const MYTHIC_HIGH_BYTE = 0x8e
+
+/** True when `buffer` is a Mythic-compressed cliloc rather than the plain layout. */
+function isCompressedCliloc(buffer) {
+  return buffer.length >= 4 && buffer[3] === MYTHIC_HIGH_BYTE
+}
+
+/**
+ * Parse the plain binary cliloc layout.
+ *
+ * Strict about truncation, and that strictness is load-bearing: a half-copied or
+ * partly-written file is the realistic failure here, and it must fail loudly
+ * rather than import a silently short table that then renders half the world as
+ * `id 1023721`. A record that runs past the end of the buffer throws.
+ */
+function parseClilocBinary(buffer) {
+  if (!Buffer.isBuffer(buffer)) throw new ClilocFormatError('Not a buffer', 'NOT_BUFFER')
+  if (isCompressedCliloc(buffer)) {
+    throw new ClilocFormatError(
+      'This is a compressed (Mythic-format) cliloc file, which the site cannot read. ' +
+        'Convert it to the plain format first — see docs/website/CLILOCS.md.',
+      'COMPRESSED',
+    )
+  }
+  if (buffer.length < HEADER_BYTES) {
+    throw new ClilocFormatError('File is shorter than a cliloc header', 'TRUNCATED')
+  }
+
+  const entries = []
+  let offset = HEADER_BYTES
+
+  while (offset < buffer.length) {
+    if (offset + RECORD_HEADER_BYTES > buffer.length) {
+      throw new ClilocFormatError(
+        `Truncated record header at byte ${offset} (${entries.length} entries read)`,
+        'TRUNCATED',
+      )
+    }
+    const number = buffer.readInt32LE(offset)
+    const flag = buffer.readUInt8(offset + 4)
+    // The length is written by the client as an unsigned 16-bit value. Reading it
+    // signed (as ServUO's own SDK does) turns any string over 32 KB into a
+    // negative length; real tables top out around 12 KB, so this has no effect on
+    // current data and costs nothing to get right.
+    const length = buffer.readUInt16LE(offset + 5)
+    offset += RECORD_HEADER_BYTES
+
+    if (offset + length > buffer.length) {
+      throw new ClilocFormatError(
+        `Truncated record body at byte ${offset} (${entries.length} entries read)`,
+        'TRUNCATED',
+      )
+    }
+    entries.push({ number, flag, text: buffer.toString('utf8', offset, offset + length) })
+    offset += length
+  }
+
+  return entries
+}
+
+// A delimited line splits on the FIRST separator only: cliloc text is full of
+// commas ("a scroll of magery, unfinished") and splitting on all of them would
+// truncate every such entry at its first comma.
+const TEXT_SEPARATORS = ['\t', ',', ';']
+
+/** Unwrap one CSV field: strip surrounding quotes and unescape doubled quotes. */
+function unquote(value) {
+  const trimmed = value.trim()
+  if (trimmed.length >= 2 && trimmed.startsWith('"') && trimmed.endsWith('"')) {
+    return trimmed.slice(1, -1).replace(/""/g, '"')
+  }
+  return trimmed
+}
+
+/**
+ * Parse a delimited text export: `number<sep>text` per line.
+ *
+ * Tolerant by design — this is whatever an operator's GUI tool produced, not a
+ * format we control. A header row, blank lines, `#` comments and a trailing
+ * flags column are all ignored. A line whose first field is not an integer is
+ * skipped rather than fatal, because that is exactly what a header row is.
+ *
+ * The one thing it will NOT do is return an empty table quietly: a file that
+ * yields no entries at all is a wrong file, not an empty one.
+ */
+function parseClilocText(text) {
+  const entries = []
+  for (const line of String(text).split(/\r?\n/)) {
+    // The line is deliberately NOT trimmed before the separator search. Roughly
+    // half of a real cliloc table is empty strings (unused ids), which export as
+    // `1005008<TAB>` — and trimming eats that trailing separator, leaving a bare
+    // number that then looks like a header row and is skipped. That silently
+    // dropped 55,994 of 123,490 entries. Individual FIELDS are trimmed instead,
+    // by `unquote`.
+    if (line.trim() === '' || line.trimStart().startsWith('#')) continue
+
+    // Pick the separator that actually appears first, so a tab-delimited line
+    // whose text contains a comma still splits on the tab.
+    let cut = -1
+    for (const sep of TEXT_SEPARATORS) {
+      const at = line.indexOf(sep)
+      if (at !== -1 && (cut === -1 || at < cut)) cut = at
+    }
+    if (cut === -1) continue
+
+    // An EMPTY first field must not become id 0: `Number('')` is 0, not NaN, so
+    // a line that merely starts with a separator would otherwise import as a
+    // bogus cliloc 0 instead of being skipped.
+    const head = unquote(line.slice(0, cut))
+    if (head === '') continue
+    const number = Number(head)
+    if (!Number.isInteger(number)) continue // header row, or a wrapped line
+
+    let rest = line.slice(cut + 1)
+    // Some exports carry `number,flag,text`. A bare integer in the second field
+    // is a flag; anything else is the text itself (and a text field that IS just
+    // a number is indistinguishable, so it stays as the text — the safer miss).
+    let flag = 0
+    for (const sep of TEXT_SEPARATORS) {
+      const at = rest.indexOf(sep)
+      if (at === -1) continue
+      const head = unquote(rest.slice(0, at))
+      if (/^\d{1,3}$/.test(head) && rest.slice(at + 1).trim() !== '') {
+        flag = Number(head)
+        rest = rest.slice(at + 1)
+      }
+      break
+    }
+
+    entries.push({ number, flag, text: unquote(rest) })
+  }
+
+  if (entries.length === 0) {
+    throw new ClilocFormatError('No cliloc entries found in the text export', 'EMPTY')
+  }
+  return entries
+}
+
+/**
+ * Parse either supported shape, sniffing which one this is.
+ *
+ * The sniff is on the binary header rather than the file extension: operators
+ * name these things whatever they like, and an `.enu` that is really a TSV (or a
+ * `.txt` that is really binary) should still import.
+ */
+function parseCliloc(buffer) {
+  const buf = Buffer.isBuffer(buffer) ? buffer : Buffer.from(buffer)
+
+  if (isCompressedCliloc(buf)) {
+    throw new ClilocFormatError(
+      'This is a compressed (Mythic-format) cliloc file, which the site cannot read. ' +
+        'Convert it to the plain format first — see docs/website/CLILOCS.md.',
+      'COMPRESSED',
+    )
+  }
+
+  // The plain layout always opens with version 2 / language 1. Anything else is
+  // treated as text, which is the recoverable guess: a mis-sniffed text file
+  // yields "no entries found", while a mis-sniffed binary yields nonsense.
+  if (buf.length >= HEADER_BYTES && buf.readInt32LE(0) === 2 && buf.readUInt16LE(4) === 1) {
+    return parseClilocBinary(buf)
+  }
+  return parseClilocText(buf.toString('utf8'))
+}
+
+// ── Display ────────────────────────────────────────────────────────────────
+
+// Cliloc strings interpolate arguments the client supplies out of an item's
+// property list: `~1_val~`, `~2_NAME~`, `~1_ITEM~`. We never have those — the
+// bridge sends the id, not the packet — so a name carrying them must be reduced
+// to what is actually knowable rather than shown with the raw tokens in it.
+const PLACEHOLDER_RE = /~\d+_[^~]*~/g
+
+/**
+ * Reduce a raw cliloc string to something displayable.
+ *
+ * Placeholders are dropped and the leftover punctuation tidied, so
+ * `"[~1_stuff~]"` becomes `""` (correctly nothing — the whole string was the
+ * argument) and `"cold damage ~1_val~%"` becomes `"cold damage"`.
+ *
+ * The trailing `%` in that second example is only stripped BECAUSE a placeholder
+ * was removed — it is the unit belonging to the number we never had. Stripping
+ * `%` unconditionally would corrupt a string that legitimately ends in one.
+ *
+ * Returns `''` when nothing survives, which callers treat as "no name" and fall
+ * back to the item id — better than showing a bracket.
+ */
+function displayText(raw) {
+  if (raw == null) return ''
+  const source = String(raw)
+  const hadPlaceholder = PLACEHOLDER_RE.test(source)
+  PLACEHOLDER_RE.lastIndex = 0 // the regex is global; `test` advances it
+
+  const trailing = hadPlaceholder ? /[\s\-–—,.;:%[\]()]+$/ : /[\s\-–—,.;:[\]()]+$/
+
+  return source
+    .replace(PLACEHOLDER_RE, ' ')
+    .replace(/\s+/g, ' ')
+    .replace(/\s+([,.;:!?])/g, '$1')
+    .replace(/^[\s\-–—,.;:[\]()]+/, '')
+    .replace(trailing, '')
+    .trim()
+}
+
+/** True when a raw cliloc string is nothing but interpolated arguments. */
+const isPlaceholderOnly = (raw) => raw != null && String(raw).trim() !== '' && displayText(raw) === ''
+
+module.exports = {
+  ClilocFormatError,
+  PARSER_VERSION,
+  HEADER_BYTES,
+  isCompressedCliloc,
+  parseCliloc,
+  parseClilocBinary,
+  parseClilocText,
+  displayText,
+  isPlaceholderOnly,
+}

server/src/utils/clilocSource.js

@@ -0,0 +1,184 @@
+// Cliloc table — the filesystem layer.
+//
+// `clilocParse.js` holds the pure parsers; this module is the only thing that
+// touches the converted cliloc file on disk, and it is shared by both callers:
+//
+//   - the server, which refreshes the table on boot (`shardClilocs.model.js`)
+//   - the admin panel, which can force a reimport without a restart
+//
+// The file is the OPERATOR'S, produced once from their own UO client (see
+// docs/website/CLILOCS.md). Nothing derived from it is committed: the repo holds
+// no string table, exactly as it holds no map snapshot and no artwork. That rule
+// is why this module reads a configured path instead of a path inside the repo.
+//
+// Reading and hashing ~5 MB costs a few milliseconds and a full parse ~50 ms, so
+// the boot path hashes first and only parses when something actually changed.
+
+const crypto = require('crypto')
+const fs = require('fs')
+const path = require('path')
+
+const { ClilocFormatError, PARSER_VERSION, parseCliloc, isCompressedCliloc } = require('./clilocParse')
+
+/**
+ * Filenames looked for when the configured path is a DIRECTORY.
+ *
+ * Ordered by how specific they are: an explicitly converted file wins over
+ * something that merely sits in a client folder, so an operator who dropped a
+ * `cliloc.plain.enu` next to the original compressed `cliloc.enu` gets the one
+ * they made rather than the one that will be rejected.
+ *
+ * Matching is case-insensitive against the real directory listing, because the
+ * client ships `Cliloc.enu` on Windows and the site usually runs on Linux, where
+ * a hardcoded lowercase open would simply miss.
+ */
+const CANDIDATE_NAMES = [
+  'clilocs.tsv',
+  'clilocs.csv',
+  'cliloc.plain.enu',
+  'cliloc.enu.plain',
+  'clilocs.txt',
+  'cliloc.enu',
+]
+
+class ClilocSourceError extends Error {
+  constructor(message, code) {
+    super(message)
+    this.name = 'ClilocSourceError'
+    this.code = code
+  }
+}
+
+function sha256(buffer) {
+  return crypto.createHash('sha256').update(buffer).digest('hex')
+}
+
+/**
+ * Resolve the configured path to an actual file.
+ *
+ * Accepts either a direct file path or a directory to search, because operators
+ * reasonably supply both — "here is the file" and "here is the folder I put it
+ * in" are equally natural answers to the admin panel's prompt.
+ */
+function resolveFile(configured) {
+  if (!configured || String(configured).trim() === '') {
+    throw new ClilocSourceError('No cliloc path configured', 'NO_PATH')
+  }
+  const target = String(configured).trim()
+
+  let stat
+  try {
+    stat = fs.statSync(target)
+  } catch {
+    throw new ClilocSourceError(`Cliloc path does not exist: ${target}`, 'NOT_FOUND')
+  }
+
+  if (stat.isFile()) return target
+
+  if (!stat.isDirectory()) {
+    throw new ClilocSourceError(`Cliloc path is neither a file nor a directory: ${target}`, 'NOT_FOUND')
+  }
+
+  let listing
+  try {
+    listing = fs.readdirSync(target)
+  } catch {
+    throw new ClilocSourceError(`Cliloc directory is not readable: ${target}`, 'NOT_FOUND')
+  }
+
+  const byLower = new Map(listing.map((name) => [name.toLowerCase(), name]))
+  for (const candidate of CANDIDATE_NAMES) {
+    const actual = byLower.get(candidate)
+    if (actual) return path.join(target, actual)
+  }
+
+  throw new ClilocSourceError(
+    `No cliloc file found in ${target} (looked for ${CANDIDATE_NAMES.join(', ')})`,
+    'NO_FILE',
+  )
+}
+
+/**
+ * A fingerprint of the source file: `{ file, sha256, bytes, mtime, compressed }`.
+ *
+ * The boot path compares the hash against what was last imported and skips the
+ * parse entirely when it matches — the normal case on every restart that did not
+ * follow a client patch.
+ *
+ * `compressed` is reported here rather than left to the parse because the admin
+ * panel calls this and NOT `readCliloc` (parsing 5 MB on every status poll would
+ * be wasteful). Without it, pointing the setting at an unconverted client
+ * directory reports a perfectly readable file with pending drift — "ready to
+ * import" — and the operator only learns otherwise when the import fails. The
+ * check is four bytes of a buffer already in hand.
+ */
+function hashSource(configured) {
+  const file = resolveFile(configured)
+  let buffer
+  try {
+    buffer = fs.readFileSync(file)
+  } catch {
+    throw new ClilocSourceError(`Cliloc file is not readable: ${file}`, 'UNREADABLE')
+  }
+  let mtime = null
+  try {
+    mtime = fs.statSync(file).mtime.toISOString()
+  } catch {
+    // A missing mtime is cosmetic (it is only shown in the admin panel).
+  }
+  return {
+    file,
+    sha256: sha256(buffer),
+    bytes: buffer.length,
+    mtime,
+    compressed: isCompressedCliloc(buffer),
+  }
+}
+
+/** True when two source fingerprints describe the same file. */
+function sameSource(a, b) {
+  return !!a && !!b && a.sha256 === b.sha256
+}
+
+/**
+ * Read and parse the configured cliloc file.
+ *
+ * Returns `{ entries, source }`. Throws `ClilocSourceError` for anything about
+ * the path and `ClilocFormatError` for anything about the contents — the two are
+ * different problems for an operator (wrong place vs wrong file), and the admin
+ * panel says which.
+ */
+function readCliloc(configured) {
+  const file = resolveFile(configured)
+
+  let buffer
+  try {
+    buffer = fs.readFileSync(file)
+  } catch {
+    throw new ClilocSourceError(`Cliloc file is not readable: ${file}`, 'UNREADABLE')
+  }
+
+  const entries = parseCliloc(buffer)
+
+  return {
+    entries,
+    source: {
+      file,
+      sha256: sha256(buffer),
+      bytes: buffer.length,
+      parserVersion: PARSER_VERSION,
+      count: entries.length,
+    },
+  }
+}
+
+module.exports = {
+  ClilocFormatError,
+  ClilocSourceError,
+  PARSER_VERSION,
+  CANDIDATE_NAMES,
+  resolveFile,
+  hashSource,
+  sameSource,
+  readCliloc,
+}