RunicGateway/website
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Activity

feat(shard): resolve cliloc names for items and reward titles

Browse Source 
Protocol 3.0 §8.6 (docs/link/v3.md), the dependency order 5 was sequenced
behind. Items on the wire carry a LabelNumber, not a name — the bridge has
always sent it (char.profile.equipment.cliloc, reward titles as a cliloc
number in string form, and one per marketplace listing) but the site had no
table to resolve it against, so a character sheet could only render
`id 1023721` where the game renders "quarter staff".

The number was never the missing piece. The table was.

Sourced from a file the operator converts once from their own client, at a
path from the `cliloc_client_path` setting falling back to UO_CLIENT_PATH.
Nothing client-derived is committed: UO's strings are EA's, exactly as the
creature sprites are. A shard with nothing configured is fully supported —
names render as ids, as they did before.

The conversion step is not avoidable, and that is the substantive finding
here: every current client ships its cliloc files COMPRESSED (first DWORD's
high byte 0x8E, the Mythic container), and ServUO's own bundled
Ultima.StringList cannot read that either — so VendorSearch.GetItemName is
already inert on such a shard and the plugin could not supply names instead.
v3.md's original "read the client's Cliloc.enu" recommendation was therefore
not implementable as written, and its committed db/data/clilocs.json artifact
also predates the Part C corrections (no committed derived snapshots, nothing
EA-derived shipped). Replaced with the spawn-atlas pattern: parse on boot from
an operator-configured path, hash-gated, output gitignored.

- utils/clilocParse.js — pure parsers, fs-free so the suite runs in CI.
  Accepts the plain binary layout and delimited text, sniffed by header rather
  than extension. Rejects a compressed file BY NAME: without that check the
  plain parser reads it as ~19k records of negative ids and 60 KB "strings"
  before dying mid-file, and the resulting error names the wrong problem.
  displayText() drops the ~1_val~ arguments the bridge never sends.
- utils/clilocSource.js — the fs layer. hashSource reports `compressed` so the
  admin panel can flag an unconverted file WITHOUT parsing 5 MB per poll;
  otherwise pointing at a client directory reports a healthy file with pending
  drift ("ready to import") and the operator only finds out on failure.
- model/shardClilocs — refresh/status/lookup. All-or-nothing replace (DELETE,
  not TRUNCATE — TRUNCATE is DDL in MariaDB and implicitly commits). Batched
  server-side resolution behind a capped cache; never throws, because a cliloc
  lookup is decoration on a character sheet.
- Deliberately NO staged-approval flow, unlike the atlas: the atlas escalates
  facet loss because a half-copied tree and a real map change are
  indistinguishable from inside the process, whereas a partial cliloc copy
  makes the parser fail on a truncated record. The ambiguity the atlas must
  escalate is one this parser simply detects.
- No public route. The table is never served AS a table: 67k rows would dwarf
  any page using them, and the Android client consumes the same resolved JSON.

Two parser bugs found by building it, both now covered by tests: trimming a
text line before splitting ate the trailing separator on empty-text entries
and silently dropped 55,994 of 123,490 while still reporting success; and
Number('') is 0, not NaN, so a line starting with a separator imported as a
bogus cliloc 0.

Verified against the real client table (123,490 entries) and the live MariaDB:
import 663 ms, hash-gated boot no-op 14 ms, cold resolve 4.2 ms / warm 0.015 ms.
Binary and TSV imports converge on the same 67,496 rows with identical keys
(blank entries — half the table — are dropped at import). A file truncated to
half its length is refused with TRUNCATED and leaves the previous table
serving. Boot logs verified for both the import and the compressed-file
warning; neither blocks startup. All three admin routes exercised over HTTP
with a real session. 629 server tests pass; client builds clean; swagger,
routes.manifest.json and routes.guards.json regenerated.

Not covered by an automated test: the character sheet renders resolved names
in presentational React with no DOM test harness in this repo, and was not
rendered against a live linked-player profile — that needs a logged-in player
with a linked game account and a shard answering a profile RPC.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
claude
2026-07-29 04:21:38 -05:00
parent 1e1a3d67c3
commit b61a4d6721
19 changed files with 2182 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

106
server/src/model/shardClilocs/shardClilocs.db.js Normal file
View File

@@ -0,0 +1,106 @@
const { pool, query } = require('../../utils/db')
// Raw SQL for the cliloc table. `shard_clilocs` is IMPORT-OWNED: `replaceAll`
// empties and refills it inside one transaction, and nothing else in the
// codebase writes to it. No foreign keys, consistent with every other shard_*
// table.
const BATCH = 1000
/**
* Replace the entire cliloc table in one transaction.
*
* All-or-nothing on purpose: a failed reload must leave the previous table
* intact rather than a half-loaded one, because a partially-imported cliloc
* table is indistinguishable from a complete one to anyone reading it — you
* would just see some items named and some not, which is also what "no table at
* all" looks like.
*
* `DELETE`, not `TRUNCATE` — `TRUNCATE` is DDL in MariaDB and implicitly
* commits, which would defeat exactly that guarantee. (The same trap the spawn
* atlas import documents; at ~123k rows `DELETE` is still well under a second.)
*/
async function replaceAll(entries, meta) {
const conn = await pool.getConnection()
try {
await conn.beginTransaction()
await conn.query('DELETE FROM shard_clilocs')
// Blank entries are dropped rather than stored. Roughly HALF of a real
// cliloc table is empty strings — ids the client reserves and never uses —
// and a row that resolves to no name is indistinguishable from no row at
// all to every caller. Dropping them halves the table (123,490 → ~67,500)
// and, more importantly, makes the binary and text imports converge on
// identical content: the binary format carries the blanks explicitly and a
// text export may or may not, depending on the tool.
//
// Later duplicates win. The plain format permits a repeated id and the
// client's own loader resolves it the same way (its dictionary assignment
// overwrites), so collapsing here keeps the batch insert from failing on a
// primary-key collision for a file the game itself would load.
const byNumber = new Map()
let blank = 0
for (const entry of entries) {
if (!Number.isInteger(entry.number)) continue
if (String(entry.text ?? '').trim() === '') {
blank++
continue
}
byNumber.set(entry.number, entry)
}
const rows = [...byNumber.values()].map((e) => [e.number, e.flag ?? 0, e.text])
for (let i = 0; i < rows.length; i += BATCH) {
await conn.batch('INSERT INTO shard_clilocs (number, flag, text) VALUES (?,?,?)', rows.slice(i, i + BATCH))
}
await conn.query(
'INSERT INTO shard_cliloc_meta (id, payload) VALUES (1, ?) ' +
'ON DUPLICATE KEY UPDATE payload = VALUES(payload), imported_at = CURRENT_TIMESTAMP',
[JSON.stringify({ ...meta, count: rows.length })],
)
await conn.commit()
return { count: rows.length, blank, duplicates: entries.length - blank - rows.length }
} catch (err) {
await conn.rollback().catch(() => {})
throw err
} finally {
conn.release()
}
}
async function getMeta() {
const rows = await query('SELECT payload, imported_at FROM shard_cliloc_meta WHERE id = 1')
if (rows.length === 0) return null
const payload = typeof rows[0].payload === 'string' ? JSON.parse(rows[0].payload) : rows[0].payload
return { ...payload, importedAt: rows[0].imported_at }
}
/**
* Look up a batch of ids.
*
* Batched rather than one-at-a-time because every caller has a LIST: a character
* sheet resolves a dozen equipment ids at once, and a page of marketplace
* listings resolves fifty. `IN (...)` with generated placeholders keeps it one
* round trip and one parameterized statement.
*/
async function lookup(numbers) {
if (!Array.isArray(numbers) || numbers.length === 0) return []
const ids = [...new Set(numbers.filter((n) => Number.isInteger(n)))]
if (ids.length === 0) return []
const placeholders = ids.map(() => '?').join(',')
return query(`SELECT number, text FROM shard_clilocs WHERE number IN (${placeholders})`, ids)
}
async function count() {
const rows = await query('SELECT COUNT(*) AS n FROM shard_clilocs')
return Number(rows[0]?.n) || 0
}
module.exports = {
replaceAll,
getMeta,
lookup,
count,
}

307
server/src/model/shardClilocs/shardClilocs.model.js Normal file
View File

@@ -0,0 +1,307 @@
const db = require('./shardClilocs.db')
const settings = require('../settings/settings.model')
const { displayText } = require('../../utils/clilocParse')
const {
ClilocFormatError,
ClilocSourceError,
PARSER_VERSION,
hashSource,
readCliloc,
} = require('../../utils/clilocSource')
const log = require('../../utils/logger')('shardClilocs')
// The cliloc table — UO's id → display-string map, refreshed from a file the
// operator converts once from their own client.
//
// Why the site holds this at all: items on the wire carry a `LabelNumber`, not a
// name. `char.profile.equipment` has always sent `cliloc`, and every marketplace
// listing sends one too. Without the table the UI can only print `id 1023721`
// where the game prints "quarter staff".
//
// Two rules govern the boot path, both inherited from the spawn atlas:
//
// 1. **It never blocks startup.** No configured path, an unreadable file, a
// wrong-format file, a database error — all caught and logged. The site
// comes up either way, serving whatever table it already had (or none, in
// which case the UI falls back to item ids exactly as it did before).
// 2. **Nothing client-derived is committed.** The table is built from the
// operator's own file at a configured path. The repo ships no strings.
//
// Unlike the atlas there is no staged-approval flow, and the difference is
// deliberate: the atlas stages a refresh that would REMOVE a facet because a
// half-copied tree and a real map change look identical from here. A cliloc file
// is a single file with a single hash, and the realistic corruption — a partial
// copy — makes the parser fail on a truncated record rather than yield a
// plausible-but-short table. The failure mode the atlas has to guess about is
// one this parser can simply detect.
const SETTING_KEY = 'cliloc_client_path'
/**
* Where the converted cliloc file lives.
*
* The admin setting wins over the environment so an operator can repoint it
* without a redeploy, matching how the rest of the shard integration is
* admin-managed rather than env-configured. `UO_CLIENT_PATH` remains as the
* deploy-time default, since the path usually describes a mount the deployment
* sets up.
*/
async function getClientPath() {
try {
const configured = await settings.get(SETTING_KEY)
if (configured && String(configured).trim() !== '') return String(configured).trim()
} catch {
// Settings unavailable is not fatal — fall through to the env default.
}
const fromEnv = process.env.UO_CLIENT_PATH
return fromEnv && fromEnv.trim() !== '' ? fromEnv.trim() : ''
}
async function setClientPath(value, updatedBy = null) {
const result = await settings.set(SETTING_KEY, String(value ?? '').trim(), updatedBy)
invalidate()
return result
}
// ── Refresh ────────────────────────────────────────────────────────────────
/** Was the loaded table built by THIS parser? */
const currentParser = (meta) => meta?.parserVersion === PARSER_VERSION
/**
* Refresh the cliloc table from the configured file.
*
* Returns a result describing what happened rather than throwing, so the caller
* — including the boot path — can log it and move on:
*
* `skipped` no path configured
* `unavailable` path configured but missing / unreadable / not a cliloc file
* `unchanged` source hash matches the loaded table; nothing parsed
* `imported` parsed and applied
* `failed` parsed or applied and something went wrong
*
* `force` skips the hash check (an admin asking for a reimport).
*/
async function refresh({ force = false, path: pathOverride = '' } = {}) {
// An explicit override wins outright — a one-off "use this file", which must
// not be silently overruled by the configured path the way an env default is.
const configured = pathOverride.trim() !== '' ? pathOverride.trim() : await getClientPath()
if (configured === '') return { status: 'skipped', reason: 'no cliloc path configured' }
let fingerprint
try {
fingerprint = hashSource(configured)
} catch (err) {
if (err instanceof ClilocSourceError) {
return { status: 'unavailable', reason: err.message, code: err.code, path: configured }
}
return { status: 'failed', reason: err.message, path: configured }
}
const meta = await db.getMeta().catch(() => null)
// Two things make a loaded table stale: the file changed, or the PARSER did.
// Only checking the file would strand an install whose client never patches on
// whatever an older build derived.
if (!force && meta?.sha256 === fingerprint.sha256 && currentParser(meta)) {
return { status: 'unchanged', path: configured, file: fingerprint.file, count: meta.count ?? null }
}
let parsed
try {
parsed = readCliloc(configured)
} catch (err) {
if (err instanceof ClilocFormatError || err instanceof ClilocSourceError) {
return { status: 'unavailable', reason: err.message, code: err.code, path: configured }
}
return { status: 'failed', reason: err.message, path: configured }
}
try {
const applied = await db.replaceAll(parsed.entries, { ...parsed.source, mtime: fingerprint.mtime })
invalidate()
return {
status: 'imported',
path: configured,
file: parsed.source.file,
count: applied.count,
parsed: parsed.entries.length,
blank: applied.blank,
duplicates: applied.duplicates,
}
} catch (err) {
return { status: 'failed', reason: err.message, path: configured }
}
}
/**
* Boot hook. Best-effort by contract: it logs and returns, never throws, so a
* missing or malformed cliloc file can never stop the site coming up.
*/
async function refreshOnBoot() {
try {
const result = await refresh()
switch (result.status) {
case 'imported':
log.info('cliloc table refreshed', { file: result.file, count: result.count })
break
case 'unavailable':
// Deliberately a warning, not an error: an operator who has not supplied
// a cliloc file is in a supported state (the UI shows item ids), and the
// most common cause — pointing at the client's own compressed file —
// needs the reason spelled out rather than a stack trace.
log.warn('cliloc source unavailable (item names will show as ids)', {
reason: result.reason,
code: result.code,
path: result.path,
})
break
case 'failed':
log.warn('cliloc refresh failed', { reason: result.reason })
break
default:
break
}
return result
} catch (err) {
log.warn('cliloc refresh errored', { error: err.message })
return { status: 'failed', reason: err.message }
}
}
/** Everything the admin panel needs to describe cliloc state. */
async function status({ path: pathOverride = '' } = {}) {
const configured = pathOverride.trim() !== '' ? pathOverride.trim() : await getClientPath()
const meta = await db.getMeta().catch(() => null)
const loaded = await db.count().catch(() => 0)
let fileReadable = false
let file = null
let drift = null
let problem = null
let code = null
if (configured !== '') {
try {
const fingerprint = hashSource(configured)
fileReadable = true
file = fingerprint.file
// A compressed file is readable but not importable, and the panel has to
// say so HERE — otherwise pointing at an unconverted client directory
// reports a healthy file with pending drift ("ready to import") and the
// operator only finds out when the import fails. `drift` stays null
// because comparing hashes with an unusable file answers nothing.
if (fingerprint.compressed) {
problem =
'This is a compressed (Mythic-format) cliloc file, which the site cannot read. ' +
'Convert it to the plain format first — see docs/website/CLILOCS.md.'
code = 'COMPRESSED'
} else {
drift = meta?.sha256 !== fingerprint.sha256 || !currentParser(meta)
}
} catch (err) {
fileReadable = false
problem = err.message
code = err.code ?? null
}
}
return {
configured: configured !== '',
path: configured,
file,
fileReadable,
problem,
code,
drift,
count: loaded,
importedAt: meta?.importedAt ?? null,
sourceBytes: meta?.bytes ?? null,
}
}
// ── Lookup ─────────────────────────────────────────────────────────────────
//
// Resolution happens SERVER-SIDE, not in the browser. Two reasons: the table is
// ~123k rows and shipping it to a client would dwarf every page that uses it,
// and the Android app consumes the same JSON and would otherwise need its own
// copy. Callers get names, not ids-plus-a-table.
// A small write-through cache in front of the table. Item ids repeat heavily —
// one page of listings is mostly the same few hundred clilocs, and a character
// sheet re-resolves the same gear on every view — so this turns the steady state
// into zero queries. Capped so a pathological caller cannot grow it without
// bound; on overflow it is dropped wholesale rather than evicted entry-by-entry,
// which is cheap and correct for a table that only changes on reimport.
const CACHE_MAX = 20000
let cache = new Map()
function invalidate() {
cache = new Map()
}
/**
* Resolve a batch of cliloc ids to display strings.
*
* Returns a `Map<number, string>` holding only the ids that resolved to
* something displayable — an id with no row, or one whose text is nothing but
* interpolated arguments we do not have, is simply absent. Callers fall back to
* whatever they had (the item id), so "missing" and "unnamed" collapse into one
* branch at the call site.
*
* Never throws: a cliloc lookup is decoration on someone's character sheet, and
* a database blip must not fail the sheet.
*/
async function resolveMany(numbers) {
const out = new Map()
if (!Array.isArray(numbers)) return out
const wanted = [...new Set(numbers.filter((n) => Number.isInteger(n) && n > 0))]
if (wanted.length === 0) return out
const missing = []
for (const number of wanted) {
if (cache.has(number)) {
const hit = cache.get(number)
if (hit !== '') out.set(number, hit)
} else {
missing.push(number)
}
}
if (missing.length > 0) {
try {
const rows = await db.lookup(missing)
const found = new Map(rows.map((r) => [Number(r.number), displayText(r.text)]))
if (cache.size + missing.length > CACHE_MAX) invalidate()
for (const number of missing) {
// Cache the miss too ('' meaning "no usable name"), so an id absent from
// the table does not re-query on every page view.
const text = found.get(number) ?? ''
cache.set(number, text)
if (text !== '') out.set(number, text)
}
} catch (err) {
log.warn('cliloc lookup failed', { message: err.message })
}
}
return out
}
/** Single-id convenience. Returns `null` when there is no usable name. */
async function resolve(number) {
const found = await resolveMany([number])
return found.get(number) ?? null
}
module.exports = {
SETTING_KEY,
getClientPath,
setClientPath,
refresh,
refreshOnBoot,
status,
resolveMany,
resolve,
invalidate,
}