feat(shard): resolve cliloc names for items and reward titles
Protocol 3.0 §8.6 (docs/link/v3.md), the dependency order 5 was sequenced
behind. Items on the wire carry a LabelNumber, not a name — the bridge has
always sent it (char.profile.equipment.cliloc, reward titles as a cliloc
number in string form, and one per marketplace listing) but the site had no
table to resolve it against, so a character sheet could only render
`id 1023721` where the game renders "quarter staff".
The number was never the missing piece. The table was.
Sourced from a file the operator converts once from their own client, at a
path from the `cliloc_client_path` setting falling back to UO_CLIENT_PATH.
Nothing client-derived is committed: UO's strings are EA's, exactly as the
creature sprites are. A shard with nothing configured is fully supported —
names render as ids, as they did before.
The conversion step is not avoidable, and that is the substantive finding
here: every current client ships its cliloc files COMPRESSED (first DWORD's
high byte 0x8E, the Mythic container), and ServUO's own bundled
Ultima.StringList cannot read that either — so VendorSearch.GetItemName is
already inert on such a shard and the plugin could not supply names instead.
v3.md's original "read the client's Cliloc.enu" recommendation was therefore
not implementable as written, and its committed db/data/clilocs.json artifact
also predates the Part C corrections (no committed derived snapshots, nothing
EA-derived shipped). Replaced with the spawn-atlas pattern: parse on boot from
an operator-configured path, hash-gated, output gitignored.
- utils/clilocParse.js — pure parsers, fs-free so the suite runs in CI.
Accepts the plain binary layout and delimited text, sniffed by header rather
than extension. Rejects a compressed file BY NAME: without that check the
plain parser reads it as ~19k records of negative ids and 60 KB "strings"
before dying mid-file, and the resulting error names the wrong problem.
displayText() drops the ~1_val~ arguments the bridge never sends.
- utils/clilocSource.js — the fs layer. hashSource reports `compressed` so the
admin panel can flag an unconverted file WITHOUT parsing 5 MB per poll;
otherwise pointing at a client directory reports a healthy file with pending
drift ("ready to import") and the operator only finds out on failure.
- model/shardClilocs — refresh/status/lookup. All-or-nothing replace (DELETE,
not TRUNCATE — TRUNCATE is DDL in MariaDB and implicitly commits). Batched
server-side resolution behind a capped cache; never throws, because a cliloc
lookup is decoration on a character sheet.
- Deliberately NO staged-approval flow, unlike the atlas: the atlas escalates
facet loss because a half-copied tree and a real map change are
indistinguishable from inside the process, whereas a partial cliloc copy
makes the parser fail on a truncated record. The ambiguity the atlas must
escalate is one this parser simply detects.
- No public route. The table is never served AS a table: 67k rows would dwarf
any page using them, and the Android client consumes the same resolved JSON.
Two parser bugs found by building it, both now covered by tests: trimming a
text line before splitting ate the trailing separator on empty-text entries
and silently dropped 55,994 of 123,490 while still reporting success; and
Number('') is 0, not NaN, so a line starting with a separator imported as a
bogus cliloc 0.
Verified against the real client table (123,490 entries) and the live MariaDB:
import 663 ms, hash-gated boot no-op 14 ms, cold resolve 4.2 ms / warm 0.015 ms.
Binary and TSV imports converge on the same 67,496 rows with identical keys
(blank entries — half the table — are dropped at import). A file truncated to
half its length is refused with TRUNCATED and leaves the previous table
serving. Boot logs verified for both the import and the compressed-file
warning; neither blocks startup. All three admin routes exercised over HTTP
with a real session. 629 server tests pass; client builds clean; swagger,
routes.manifest.json and routes.guards.json regenerated.
Not covered by an automated test: the character sheet renders resolved names
in presentational React with no DOM test harness in this repo, and was not
rendered against a live linked-player profile — that needs a logged-in player
with a linked game account and a shard answering a profile RPC.
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
106
server/src/model/shardClilocs/shardClilocs.db.js
Normal file
106
server/src/model/shardClilocs/shardClilocs.db.js
Normal file
@@ -0,0 +1,106 @@
|
||||
const { pool, query } = require('../../utils/db')
|
||||
|
||||
// Raw SQL for the cliloc table. `shard_clilocs` is IMPORT-OWNED: `replaceAll`
|
||||
// empties and refills it inside one transaction, and nothing else in the
|
||||
// codebase writes to it. No foreign keys, consistent with every other shard_*
|
||||
// table.
|
||||
|
||||
const BATCH = 1000
|
||||
|
||||
/**
|
||||
* Replace the entire cliloc table in one transaction.
|
||||
*
|
||||
* All-or-nothing on purpose: a failed reload must leave the previous table
|
||||
* intact rather than a half-loaded one, because a partially-imported cliloc
|
||||
* table is indistinguishable from a complete one to anyone reading it — you
|
||||
* would just see some items named and some not, which is also what "no table at
|
||||
* all" looks like.
|
||||
*
|
||||
* `DELETE`, not `TRUNCATE` — `TRUNCATE` is DDL in MariaDB and implicitly
|
||||
* commits, which would defeat exactly that guarantee. (The same trap the spawn
|
||||
* atlas import documents; at ~123k rows `DELETE` is still well under a second.)
|
||||
*/
|
||||
async function replaceAll(entries, meta) {
|
||||
const conn = await pool.getConnection()
|
||||
try {
|
||||
await conn.beginTransaction()
|
||||
await conn.query('DELETE FROM shard_clilocs')
|
||||
|
||||
// Blank entries are dropped rather than stored. Roughly HALF of a real
|
||||
// cliloc table is empty strings — ids the client reserves and never uses —
|
||||
// and a row that resolves to no name is indistinguishable from no row at
|
||||
// all to every caller. Dropping them halves the table (123,490 → ~67,500)
|
||||
// and, more importantly, makes the binary and text imports converge on
|
||||
// identical content: the binary format carries the blanks explicitly and a
|
||||
// text export may or may not, depending on the tool.
|
||||
//
|
||||
// Later duplicates win. The plain format permits a repeated id and the
|
||||
// client's own loader resolves it the same way (its dictionary assignment
|
||||
// overwrites), so collapsing here keeps the batch insert from failing on a
|
||||
// primary-key collision for a file the game itself would load.
|
||||
const byNumber = new Map()
|
||||
let blank = 0
|
||||
for (const entry of entries) {
|
||||
if (!Number.isInteger(entry.number)) continue
|
||||
if (String(entry.text ?? '').trim() === '') {
|
||||
blank++
|
||||
continue
|
||||
}
|
||||
byNumber.set(entry.number, entry)
|
||||
}
|
||||
|
||||
const rows = [...byNumber.values()].map((e) => [e.number, e.flag ?? 0, e.text])
|
||||
for (let i = 0; i < rows.length; i += BATCH) {
|
||||
await conn.batch('INSERT INTO shard_clilocs (number, flag, text) VALUES (?,?,?)', rows.slice(i, i + BATCH))
|
||||
}
|
||||
|
||||
await conn.query(
|
||||
'INSERT INTO shard_cliloc_meta (id, payload) VALUES (1, ?) ' +
|
||||
'ON DUPLICATE KEY UPDATE payload = VALUES(payload), imported_at = CURRENT_TIMESTAMP',
|
||||
[JSON.stringify({ ...meta, count: rows.length })],
|
||||
)
|
||||
|
||||
await conn.commit()
|
||||
return { count: rows.length, blank, duplicates: entries.length - blank - rows.length }
|
||||
} catch (err) {
|
||||
await conn.rollback().catch(() => {})
|
||||
throw err
|
||||
} finally {
|
||||
conn.release()
|
||||
}
|
||||
}
|
||||
|
||||
async function getMeta() {
|
||||
const rows = await query('SELECT payload, imported_at FROM shard_cliloc_meta WHERE id = 1')
|
||||
if (rows.length === 0) return null
|
||||
const payload = typeof rows[0].payload === 'string' ? JSON.parse(rows[0].payload) : rows[0].payload
|
||||
return { ...payload, importedAt: rows[0].imported_at }
|
||||
}
|
||||
|
||||
/**
|
||||
* Look up a batch of ids.
|
||||
*
|
||||
* Batched rather than one-at-a-time because every caller has a LIST: a character
|
||||
* sheet resolves a dozen equipment ids at once, and a page of marketplace
|
||||
* listings resolves fifty. `IN (...)` with generated placeholders keeps it one
|
||||
* round trip and one parameterized statement.
|
||||
*/
|
||||
async function lookup(numbers) {
|
||||
if (!Array.isArray(numbers) || numbers.length === 0) return []
|
||||
const ids = [...new Set(numbers.filter((n) => Number.isInteger(n)))]
|
||||
if (ids.length === 0) return []
|
||||
const placeholders = ids.map(() => '?').join(',')
|
||||
return query(`SELECT number, text FROM shard_clilocs WHERE number IN (${placeholders})`, ids)
|
||||
}
|
||||
|
||||
async function count() {
|
||||
const rows = await query('SELECT COUNT(*) AS n FROM shard_clilocs')
|
||||
return Number(rows[0]?.n) || 0
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
replaceAll,
|
||||
getMeta,
|
||||
lookup,
|
||||
count,
|
||||
}
|
||||
Reference in New Issue
Block a user