RunicGateway/website
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Activity

feat(shard): read clilocs from a source SET so shard items get names

Browse Source 
Shards edit items and add new ones, and those carry cliloc ids no stock client
table has. Reading exactly one converted file meant an operator had to
re-export 5 MB every time they added one item — friction enough that the table
would simply go stale, which is the failure the spawn atlas was redesigned to
avoid in the first place.

So this mirrors spawnAtlasSource.readSources(): a BASE (the converted client
table) plus every operator-maintained overlay under `custom/`, all re-read on
every boot and hash-gated as a SET. Later sources win, so an overlay both adds
ids the client never had and overrides stock ones the shard re-purposed.
Adding, editing or removing any overlay counts as drift.

`custom/` is the one convention here that is ours rather than the shard's, and
deliberately so: ServUO has no server-side notion of a custom cliloc — they
live in the patched client a shard distributes, and nothing in the tree
declares them. There is nothing to discover. (An operator who does patch their
client cliloc needs no overlay: convert the patched file and the edits are in
the base.) Scale, measured on the live shard: its script tree references 16,434
cliloc ids and only 37 are absent from stock — tens against a 67k base, which
is why this is an overlay and not a second table.

The set brings back a hazard a single file did not have, and it gets the
atlas's answer. A corrupt source fails the parse loudly, but a source that has
VANISHED parses perfectly and imports a table quietly missing everything it
contributed — an unmounted volume is indistinguishable from a deliberate
deletion. So it is staged, not applied (`needsReview`), reported by both the
import and status(), and accepted with `{approve:true}`. That is a flag rather
than the atlas's approve/reject pair because the atlas stores a pending
decision SO THAT approving re-parses; here nothing is stored, so re-reading at
approval time is automatic.

Also reports a per-source breakdown (entries/added/overrode) on import and in
status, which is how an operator confirms an overlay took effect — "overrode: 0"
on a file meant to re-label stock items says it did not.

Two bugs this surfaced, both found by running a shard-style overlay rather than
by another stock-table fixture:

- displayText tidied punctuation unconditionally, so a custom
  "Runic Gateway Sigil (v2)" rendered as "(v2". Stripping leftover brackets is
  right after a placeholder is removed and wrong otherwise — the same condition
  the `%` rule already had.
- CANDIDATE_NAMES did not include `clilocs.plain`, which is the exact filename
  CLILOCS.md and the export tool's README tell operators to write. Pointing at
  the directory they were told to create failed with NO_FILE.

Verified end to end against the live MariaDB and a real server boot: base-only
import, overlay adding one id and overriding another (per-source breakdown
correct), unchanged set as a no-op, an edited overlay re-importing and
withdrawing its override, a vanished overlay refused with the table intact,
status reporting missingSources, approve applying it, and a file-path
configuration still finding overlays beside it. All three resolve correctly
through the running server: shard-added, overridden and stock. 646 server tests
pass (16 new in clilocSource.test.js, 3 new in clilocParse.test.js); swagger,
routes.manifest.json and routes.guards.json regenerated.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
claude
2026-07-29 06:46:12 -05:00
parent b61a4d6721
commit bda031566a
11 changed files with 939 additions and 120 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
server/src/model/shardClilocs/shardClilocs.db.js
View File

@@ -34,10 +34,12 @@ async function replaceAll(entries, meta) {
// identical content: the binary format carries the blanks explicitly and a
// text export may or may not, depending on the tool.
//
// Later duplicates win. The plain format permits a repeated id and the
// client's own loader resolves it the same way (its dictionary assignment
// overwrites), so collapsing here keeps the batch insert from failing on a
// primary-key collision for a file the game itself would load.
// Later duplicates win. Merging across sources already happened upstream in
// `readCliloc`, so in practice this collapses nothing — it is kept because
// the plain format permits a repeated id WITHIN one file and the client's
// own loader resolves it the same way (its dictionary assignment
// overwrites). Without it, a file the game itself would load happily would
// fail the batch insert on a primary-key collision.
const byNumber = new Map()
let blank = 0
for (const entry of entries) {

105
server/src/model/shardClilocs/shardClilocs.model.js
View File

@@ -5,7 +5,9 @@ const {
ClilocFormatError,
ClilocSourceError,
PARSER_VERSION,
hashSource,
hashSources,
sameSources,
missingSources,
readCliloc,
} = require('../../utils/clilocSource')
const log = require('../../utils/logger')('shardClilocs')
@@ -27,13 +29,24 @@ const log = require('../../utils/logger')('shardClilocs')
// 2. **Nothing client-derived is committed.** The table is built from the
// operator's own file at a configured path. The repo ships no strings.
//
// Unlike the atlas there is no staged-approval flow, and the difference is
// deliberate: the atlas stages a refresh that would REMOVE a facet because a
// half-copied tree and a real map change look identical from here. A cliloc file
// is a single file with a single hash, and the realistic corruption — a partial
// copy — makes the parser fail on a truncated record rather than yield a
// plausible-but-short table. The failure mode the atlas has to guess about is
// one this parser can simply detect.
// The table is built from a SET of sources — the converted client table plus
// every operator-maintained overlay beside it — because shards edit items and
// add new ones, and those carry cliloc ids no stock client table has. All of
// them are re-read on every boot and hash-gated together, so adding one custom
// item never means re-exporting a 5 MB client file. Later sources win.
//
// That set is also why this has the atlas's escalation, in a lighter form. A
// single corrupt file fails the parse loudly, but a source that has simply
// VANISHED parses perfectly and imports a table quietly missing everything it
// contributed — the same ambiguity (real change vs half-copied mount) the atlas
// stages a facet removal for. So a disappearing source is refused and reported
// rather than applied.
//
// It is lighter than the atlas's because it needs to be: the atlas stores a
// pending decision in its own table and adds approve/reject endpoints, whereas
// here the decision is a single boolean an admin passes to the import they were
// already going to run. Re-parsing at approval time — the property that makes
// the atlas store only the decision — is automatic when there is nothing stored.
const SETTING_KEY = 'cliloc_client_path'
@@ -76,13 +89,15 @@ const currentParser = (meta) => meta?.parserVersion === PARSER_VERSION
*
* `skipped` no path configured
* `unavailable` path configured but missing / unreadable / not a cliloc file
* `unchanged` source hash matches the loaded table; nothing parsed
* `unchanged` source hashes match the loaded table; nothing parsed
* `imported` parsed and applied
* `needsReview` a previously-present source has vanished; NOT applied
* `failed` parsed or applied and something went wrong
*
* `force` skips the hash check (an admin asking for a reimport).
* `force` skips the hash check (an admin asking for a reimport). `approve`
* additionally accepts a vanished source.
*/
async function refresh({ force = false, path: pathOverride = '' } = {}) {
async function refresh({ force = false, approve = false, path: pathOverride = '' } = {}) {
// An explicit override wins outright — a one-off "use this file", which must
// not be silently overruled by the configured path the way an env default is.
const configured = pathOverride.trim() !== '' ? pathOverride.trim() : await getClientPath()
@@ -90,7 +105,7 @@ async function refresh({ force = false, path: pathOverride = '' } = {}) {
let fingerprint
try {
fingerprint = hashSource(configured)
fingerprint = hashSources(configured)
} catch (err) {
if (err instanceof ClilocSourceError) {
return { status: 'unavailable', reason: err.message, code: err.code, path: configured }
@@ -100,11 +115,31 @@ async function refresh({ force = false, path: pathOverride = '' } = {}) {
const meta = await db.getMeta().catch(() => null)
// Two things make a loaded table stale: the file changed, or the PARSER did.
// Only checking the file would strand an install whose client never patches on
// whatever an older build derived.
if (!force && meta?.sha256 === fingerprint.sha256 && currentParser(meta)) {
return { status: 'unchanged', path: configured, file: fingerprint.file, count: meta.count ?? null }
// Two things make a loaded table stale: any source changed, or the PARSER did.
// Only checking the sources would strand an install whose client never patches
// on whatever an older build derived.
if (!force && sameSources(fingerprint.hashes, meta?.hashes) && currentParser(meta)) {
return {
status: 'unchanged',
path: configured,
file: fingerprint.file,
count: meta.count ?? null,
customCount: fingerprint.customCount,
}
}
// A source that was there last import and is not there now is refused, not
// applied — an unmounted volume and a deliberate deletion look identical from
// here, and the wrong guess silently drops every name that file contributed.
const gone = missingSources(fingerprint.hashes, meta?.hashes)
if (gone.length > 0 && !approve) {
return {
status: 'needsReview',
reason: `${gone.length} previously-loaded cliloc source(s) are missing; the existing table is unchanged`,
missingSources: gone,
path: configured,
file: fingerprint.file,
}
}
let parsed
@@ -118,7 +153,7 @@ async function refresh({ force = false, path: pathOverride = '' } = {}) {
}
try {
const applied = await db.replaceAll(parsed.entries, { ...parsed.source, mtime: fingerprint.mtime })
const applied = await db.replaceAll(parsed.entries, parsed.source)
invalidate()
return {
status: 'imported',
@@ -127,7 +162,12 @@ async function refresh({ force = false, path: pathOverride = '' } = {}) {
count: applied.count,
parsed: parsed.entries.length,
blank: applied.blank,
duplicates: applied.duplicates,
// Per-source breakdown: how many entries each file contributed and how
// many of them overrode something already merged. An operator who adds an
// overlay wants to see it took effect, and "overrode: 0" on a file meant
// to re-label stock items says it did not.
sources: parsed.source.sources,
acceptedMissing: gone.length > 0 ? gone : undefined,
}
} catch (err) {
return { status: 'failed', reason: err.message, path: configured }
@@ -143,7 +183,18 @@ async function refreshOnBoot() {
const result = await refresh()
switch (result.status) {
case 'imported':
log.info('cliloc table refreshed', { file: result.file, count: result.count })
log.info('cliloc table refreshed', {
file: result.file,
count: result.count,
overlays: (result.sources || []).filter((s) => s.kind === 'custom').length,
})
break
case 'needsReview':
log.warn(
'cliloc refresh staged for admin review — a previously-loaded source is missing; ' +
'the existing table is unchanged',
{ missing: result.missingSources },
)
break
case 'unavailable':
// Deliberately a warning, not an error: an operator who has not supplied
@@ -180,11 +231,15 @@ async function status({ path: pathOverride = '' } = {}) {
let drift = null
let problem = null
let code = null
let sources = []
let missing = []
if (configured !== '') {
try {
const fingerprint = hashSource(configured)
const fingerprint = hashSources(configured)
fileReadable = true
file = fingerprint.file
sources = Object.keys(fingerprint.hashes)
missing = missingSources(fingerprint.hashes, meta?.hashes)
// A compressed file is readable but not importable, and the panel has to
// say so HERE — otherwise pointing at an unconverted client directory
// reports a healthy file with pending drift ("ready to import") and the
@@ -196,7 +251,7 @@ async function status({ path: pathOverride = '' } = {}) {
'Convert it to the plain format first — see docs/website/CLILOCS.md.'
code = 'COMPRESSED'
} else {
drift = meta?.sha256 !== fingerprint.sha256 || !currentParser(meta)
drift = !sameSources(fingerprint.hashes, meta?.hashes) || !currentParser(meta)
}
} catch (err) {
fileReadable = false
@@ -214,6 +269,12 @@ async function status({ path: pathOverride = '' } = {}) {
code,
drift,
count: loaded,
// Every source found now (base first, then overlays), what each contributed
// at the last import, and any that have since vanished — which is the state
// an import will refuse without `approve`.
sources,
loadedSources: meta?.sources ?? null,
missingSources: missing,
importedAt: meta?.importedAt ?? null,
sourceBytes: meta?.bytes ?? null,
}