docs(link): the cliloc table, and why §8.6's recommendation was not implementable #70

wtclaude · 2026-07-29T09:24:04Z

wtclaude commented

2026-07-29 09:24:04 +00:00

What & why

Protocol 3.0 §8.6 — the cliloc dependency, resolved as its own website-only change and landing ahead of §8 so the marketplace ships with real item names. Matching documentation for website #115.

NEW `website/CLILOCS.md`

Operator-facing: why the conversion step exists at all, how to convert, how to configure the path, the refresh contract, what gets stored, and how names are applied.

`link/v3.md` §8.6 — rewritten, because both halves of the recommendation were wrong

The original said: scripts/buildClilocs.js reads the client's Cliloc.enu → committed db/data/clilocs.json. Neither half survives contact, and both failures generalise.

1. The committed artifact predates the Part C corrections (§6.1) and violates both — no committed snapshot of derived content, and nothing EA-derived ever shipped. UO's strings are EA's, exactly as the creature sprites are. Replaced with the §6 pattern: parse on boot from an operator-configured path, hash-gated, output gitignored, PARSER_VERSION counted as drift.

2. "reads the UO client's Cliloc.enu" is not possible. Every current client ships its cliloc files COMPRESSED — all eight Cliloc.* files open with a DWORD whose high byte is 0x8E, the "Mythic" container. The plain layout is what those files looked like before that change, and parsing one as the other does not fail cleanly: 19,508 "records" with only 1,722 distinct ids, ids into the negatives, one 62 KB "string", and a truncation mid-file.

Two facts closed off the alternatives, and the second is the one worth carrying forward:

Decompressing means porting an inverse-BWT coder with a 1 KB frequency header — a few hundred lines whose failure mode is plausible-looking garbage rather than an error.
ServUO cannot read it either. Its bundled Ultima.StringList implements only the plain layout, so VendorSearch.StringList is null and GetItemName returns item.Name. The work could not be pushed to the plugin — and the in-game Vendor Search gump has exactly the same gap, which retires an open cost in §8.2: the mandatory "never call GetItemName in the market sweep" costs us nothing we could otherwise have had.

⇒ the operator converts once, from their own client, and the site reads the result.

Three traps found by building it, all recorded:

StringList.SaveStringList RE-COMPRESSES on save. It looks exactly like the export path and is not — its output is byte-identical to its compressed input, because its purpose is round-tripping a file back into the client.
Trimming a text line before splitting silently drops half the table. About half a real cliloc table is empty strings (ids the client reserves), exported as 1005008<TAB>; trimming eats the trailing separator and the line then looks like a header row. 55,994 of 123,490 entries vanished and the import still reported success.
Number('') is 0, not NaN — a line starting with a separator imports as a bogus cliloc 0.

Also updated

Progress and §9 sequencing tables — order 5 split into 5a (this, website-only) and 5b (the four-repo wire change), with a note on why.
website/BACKEND_DESIGN.md — shard_clilocs / shard_cliloc_meta, the three admin routes, and the two "why not" decisions: no staged-approval flow (the atlas escalates facet loss because a half-copied tree and a real map change are indistinguishable; a partial cliloc copy makes the parser fail on a truncated record instead — the ambiguity the atlas must escalate is one this parser detects), and no public route (67k rows would dwarf any page, and the Android client consumes the same resolved JSON).
link/INTEGRATION.md — the char.profile field note now says explicitly not to expect the shard to resolve clilocs, with the reason, and points at CLILOCS.md.
§10 documentation obligations now list website/CLILOCS.md.

How it was tested

Documentation only — no code. Every claim here was written after the corresponding behaviour was observed, not from the plan:

the compressed-format finding and the exact garbage counts (19,508 / 1,722 / 62 KB) come from parsing the real client files on this machine;
the entry counts (123,490 parsed → 67,496 stored, 55,994 blank) and the timings quoted in CLILOCS.md (663 ms import, 14 ms hash-gated boot, 4.2 ms cold / 0.015 ms warm resolve) come from importing them into the live MariaDB;
the TRUNCATED behaviour, the COMPRESSED status/import responses and the boot log lines are all copied from actual runs in website #115.

PROJECT_TREE.md files are deliberately untouched — they are CI-generated by the sync-project-tree workflow and say so in their header.

Checklist

I have read CONTRIBUTING.md.
The change builds and existing tests/checks pass locally. (Markdown only.)
I have added or updated tests/docs where it makes sense.
My commits are reasonably scoped with clear messages.

AI-assisted contributions (required)

No AI tools were used to produce this contribution.
AI tools were used. Tool(s): Claude Code (Opus 5). I have reviewed and understand every change, and take responsibility for it. AI-authored commits are marked with a Co-Authored-By trailer.

License

I agree that my contribution is licensed under this project's license (GNU GPL v3.0 or later), and I have the right to contribute it.

Update — shard-added and shard-edited items (second commit)

Review feedback: the table has to be updatable the way the spawn atlas is, because shards edit items and add new ones — those carry cliloc ids no stock client table has.

Reading exactly one converted file meant re-exporting 5 MB every time an operator added one item. That is friction enough that the table goes stale, which is the exact failure Part C was redesigned to avoid.

So it now mirrors spawnAtlasSource.readSources(): a base (the converted client table) plus every overlay under custom/, all re-read on every boot and hash-gated as a set. Later sources win, so an overlay both adds ids and overrides stock ones. Adding, editing or removing any overlay counts as drift.

<cliloc path>/
  clilocs.plain            ← base
  custom/
    01-uomysticmoon.tsv    ← shard additions + overrides

custom/ is the one convention here that is ours rather than the shard's, deliberately. ServUO has no server-side notion of a custom cliloc — they live in the patched client a shard distributes, and nothing in the tree declares them, so there is nothing to discover. (An operator who does patch their client cliloc needs no overlay: convert the patched file and the edits are in the base.)

Scale, measured on the live shard: its script tree references 16,434 cliloc ids and only 37 are absent from stock — tens against a 67k base, which is why this is an overlay and not a second table.

The set brings back the atlas's ambiguity, and gets its answer

A corrupt source fails the parse loudly. But a source that has vanished parses perfectly and imports a table quietly missing everything it contributed — an unmounted volume is indistinguishable from a deliberate deletion. That is precisely what the atlas stages a facet removal for, so it is staged here too: status: "needsReview", nothing applied, missingSources reported by both the import and status(), accepted with {"approve": true}.

It is a flag rather than the atlas's approve/reject pair because the atlas stores a pending decision so that approving re-parses the tree; here nothing is stored, so re-reading at approval time is automatic. (This replaces the "why there is no staged-approval flow" reasoning in the original description — that argument held for a single file and does not survive multiple sources.)

Import and status also report a per-source breakdown, which is how an operator confirms an overlay took effect — overrode: 0 on a file meant to re-label stock items says it did not.

Two bugs this surfaced

Both found by running a shard-style overlay rather than another stock-table fixture:

displayText tidied punctuation unconditionally, so a custom "Runic Gateway Sigil (v2)" rendered as "(v2". Stripping leftover brackets is right after a placeholder is removed and wrong otherwise — the same condition the % rule already had.
CANDIDATE_NAMES omitted clilocs.plain — the exact filename CLILOCS.md and the export tool's README tell operators to write. Pointing at the directory they were told to create failed with NO_FILE.

How the update was tested

Full walk against the live MariaDB and a real server boot: base-only import → overlay adding one id and overriding another (breakdown correct: added: 1, overrode: 1) → unchanged set as a no-op → edited overlay re-importing and withdrawing its override → vanished overlay refused with the table intact → status reporting missingSources → approve applying it → a file-path configuration still finding overlays beside it. The needsReview warning was also confirmed on a real boot log.

Through the running server, all three resolve correctly:

1180001 (shard-added)  -> Runic Gateway Sigil (v2)
1023721 (overridden)   -> gnarled quarter staff
1015012 (stock, base)  -> Greater Heal

646 server tests pass (16 new in clilocSource.test.js driving real temp directories — which file wins, what a listing yields, what happens when one vanishes are exactly the behaviours a mock would define away; 3 new in clilocParse.test.js). Artifacts regenerated.

## What & why Protocol 3.0 **§8.6** — the cliloc dependency, resolved as its own website-only change and landing ahead of §8 so the marketplace ships with real item names. Matching documentation for website [#115](https://gitea.whitlocktech.com/RunicGateway/website/pulls/115). ### NEW `website/CLILOCS.md` Operator-facing: why the conversion step exists at all, how to convert, how to configure the path, the refresh contract, what gets stored, and how names are applied. ### `link/v3.md` §8.6 — rewritten, because both halves of the recommendation were wrong The original said: `scripts/buildClilocs.js` reads the client's `Cliloc.enu` → committed `db/data/clilocs.json`. Neither half survives contact, and both failures generalise. **1. The committed artifact predates the Part C corrections (§6.1) and violates both** — no committed snapshot of derived content, and nothing EA-derived ever shipped. UO's strings are EA's, exactly as the creature sprites are. Replaced with the §6 pattern: parse on boot from an operator-configured path, hash-gated, output gitignored, `PARSER_VERSION` counted as drift. **2. "reads the UO client's `Cliloc.enu`" is not possible.** **Every current client ships its cliloc files COMPRESSED** — all eight `Cliloc.*` files open with a DWORD whose high byte is `0x8E`, the "Mythic" container. The plain layout is what those files looked like *before* that change, and parsing one as the other does not fail cleanly: 19,508 "records" with only 1,722 distinct ids, ids into the negatives, one 62 KB "string", and a truncation mid-file. Two facts closed off the alternatives, and the second is the one worth carrying forward: - Decompressing means porting an inverse-BWT coder with a 1 KB frequency header — a few hundred lines whose failure mode is plausible-looking garbage rather than an error. - **ServUO cannot read it either.** Its bundled `Ultima.StringList` implements only the plain layout, so `VendorSearch.StringList` is null and `GetItemName` returns `item.Name`. The work could not be pushed to the plugin — and **the in-game Vendor Search gump has exactly the same gap**, which retires an open cost in §8.2: the mandatory "never call `GetItemName` in the market sweep" costs us nothing we could otherwise have had. ⇒ the operator converts once, from their own client, and the site reads the result. **Three traps found by building it**, all recorded: - **`StringList.SaveStringList` RE-COMPRESSES on save.** It looks exactly like the export path and is not — its output is byte-identical to its compressed input, because its purpose is round-tripping a file back into the client. - **Trimming a text line before splitting silently drops half the table.** About half a real cliloc table is empty strings (ids the client reserves), exported as `1005008<TAB>`; trimming eats the trailing separator and the line then looks like a header row. 55,994 of 123,490 entries vanished and the import still reported success. - **`Number('')` is `0`, not `NaN`** — a line starting with a separator imports as a bogus cliloc 0. ### Also updated - **Progress and §9 sequencing tables** — order 5 split into **5a** (this, website-only) and **5b** (the four-repo wire change), with a note on why. - **`website/BACKEND_DESIGN.md`** — `shard_clilocs` / `shard_cliloc_meta`, the three admin routes, and the two "why not" decisions: no staged-approval flow (the atlas escalates facet loss because a half-copied tree and a real map change are indistinguishable; a partial cliloc copy makes the parser fail on a truncated record instead — the ambiguity the atlas must escalate is one this parser detects), and no public route (67k rows would dwarf any page, and the Android client consumes the same resolved JSON). - **`link/INTEGRATION.md`** — the `char.profile` field note now says explicitly **not** to expect the shard to resolve clilocs, with the reason, and points at CLILOCS.md. - **§10 documentation obligations** now list `website/CLILOCS.md`. ## How it was tested Documentation only — no code. Every claim here was written *after* the corresponding behaviour was observed, not from the plan: - the compressed-format finding and the exact garbage counts (19,508 / 1,722 / 62 KB) come from parsing the real client files on this machine; - the entry counts (123,490 parsed → **67,496** stored, 55,994 blank) and the timings quoted in CLILOCS.md (663 ms import, 14 ms hash-gated boot, 4.2 ms cold / 0.015 ms warm resolve) come from importing them into the live MariaDB; - the `TRUNCATED` behaviour, the `COMPRESSED` status/import responses and the boot log lines are all copied from actual runs in website #115. `PROJECT_TREE.md` files are deliberately **untouched** — they are CI-generated by the `sync-project-tree` workflow and say so in their header. ## Checklist - [x] I have read [CONTRIBUTING.md](CONTRIBUTING.md). - [x] The change builds and existing tests/checks pass locally. *(Markdown only.)* - [x] I have added or updated tests/docs where it makes sense. - [x] My commits are reasonably scoped with clear messages. ## AI-assisted contributions (required) - [ ] No AI tools were used to produce this contribution. - [x] AI tools were used. Tool(s): `Claude Code (Opus 5)`. I have reviewed and understand every change, and take responsibility for it. AI-authored commits are marked with a `Co-Authored-By` trailer. ## License - [x] I agree that my contribution is licensed under this project's license (**GNU GPL v3.0 or later**), and I have the right to contribute it. --- ## Update — shard-added and shard-edited items (second commit) Review feedback: the table has to be updatable the way the spawn atlas is, because **shards edit items and add new ones** — those carry cliloc ids no stock client table has. Reading exactly one converted file meant re-exporting 5 MB every time an operator added one item. That is friction enough that the table goes stale, which is the exact failure Part C was redesigned to avoid. So it now mirrors `spawnAtlasSource.readSources()`: a **base** (the converted client table) plus every overlay under `custom/`, all re-read on every boot and **hash-gated as a set**. Later sources win, so an overlay both *adds* ids and *overrides* stock ones. Adding, editing or removing any overlay counts as drift. ``` <cliloc path>/ clilocs.plain ← base custom/ 01-uomysticmoon.tsv ← shard additions + overrides ``` **`custom/` is the one convention here that is ours rather than the shard's, deliberately.** ServUO has **no server-side notion of a custom cliloc** — they live in the patched client a shard distributes, and nothing in the tree declares them, so there is nothing to discover. (An operator who *does* patch their client cliloc needs no overlay: convert the patched file and the edits are in the base.) Scale, measured on the live shard: its script tree references **16,434** cliloc ids and only **37** are absent from stock — tens against a 67k base, which is why this is an overlay and not a second table. ### The set brings back the atlas's ambiguity, and gets its answer A corrupt source fails the parse loudly. But a source that has **vanished** parses perfectly and imports a table quietly missing everything it contributed — an unmounted volume is indistinguishable from a deliberate deletion. That is precisely what the atlas stages a facet removal for, so it is staged here too: `status: "needsReview"`, nothing applied, `missingSources` reported by both the import and `status()`, accepted with `{"approve": true}`. It is a **flag** rather than the atlas's approve/reject pair because the atlas stores a pending decision *so that approving re-parses the tree*; here nothing is stored, so re-reading at approval time is automatic. (This replaces the "why there is no staged-approval flow" reasoning in the original description — that argument held for a single file and does not survive multiple sources.) Import and status also report a per-source breakdown, which is how an operator confirms an overlay took effect — `overrode: 0` on a file meant to re-label stock items says it did not. ### Two bugs this surfaced Both found by running a shard-style overlay rather than another stock-table fixture: 1. **`displayText` tidied punctuation unconditionally**, so a custom `"Runic Gateway Sigil (v2)"` rendered as `"(v2"`. Stripping leftover brackets is right after a placeholder is removed and wrong otherwise — the same condition the `%` rule already had. 2. **`CANDIDATE_NAMES` omitted `clilocs.plain`** — the exact filename CLILOCS.md and the export tool's README tell operators to write. Pointing at the directory they were told to create failed with `NO_FILE`. ### How the update was tested Full walk against the live MariaDB and a real server boot: base-only import → overlay adding one id and overriding another (breakdown correct: `added: 1, overrode: 1`) → unchanged set as a no-op → edited overlay re-importing and withdrawing its override → **vanished overlay refused with the table intact** → `status` reporting `missingSources` → `approve` applying it → a file-path configuration still finding overlays beside it. The `needsReview` warning was also confirmed on a real boot log. Through the running server, all three resolve correctly: ``` 1180001 (shard-added) -> Runic Gateway Sigil (v2) 1023721 (overridden) -> gnarled quarter staff 1015012 (stock, base) -> Greater Heal ``` **646 server tests pass** (16 new in `clilocSource.test.js` driving real temp directories — which file wins, what a listing yields, what happens when one vanishes are exactly the behaviours a mock would define away; 3 new in `clilocParse.test.js`). Artifacts regenerated.

wtclaude added 1 commit 2026-07-29 09:24:06 +00:00

docs(link): the cliloc table, and why §8.6's recommendation was not implementable be9f5019fa

Protocol 3.0 §8.6 resolved as its own website-only change, landing ahead of
§8 so the marketplace ships with real item names. Matching documentation for
website #TBD.

NEW website/CLILOCS.md — operator-facing: why the conversion step exists, how
to convert, how to configure the path, the refresh contract, what gets stored
and how names are applied.

link/v3.md §8.6 rewritten. Two things in the original recommendation turned
out to be wrong, and both are recorded because the reasoning generalises:

1. The committed db/data/clilocs.json artifact predates the Part C
   corrections (§6.1) and violates both — no committed snapshot of derived
   content, and nothing EA-derived ever shipped. UO's strings are EA's,
   exactly as the creature sprites are.
2. "scripts/buildClilocs.js reads the UO client's Cliloc.enu" is not
   possible. EVERY current client ships its cliloc files compressed (first
   DWORD's high byte 0x8E, the Mythic container); the plain layout is what
   those files looked like before that change, and parsing one as the other
   does not fail cleanly — it yields ~19k records with negative ids, 1,722
   distinct keys out of 19,508, one 62 KB "string", and a truncation
   somewhere in the middle. ServUO's own Ultima.StringList cannot read it
   either, so VendorSearch.GetItemName is already inert on such a shard and
   the work could not be pushed to the plugin.

That second point also retires an open question in §8.2: the warning never to
call GetItemName in the market sweep costs us nothing we could otherwise have
had, because the in-game Vendor Search gump has the same gap.

Three traps found by building it are recorded: StringList.SaveStringList
RE-COMPRESSES on save (its output is byte-identical to its compressed input,
because its purpose is round-tripping a file back into the client); trimming a
text line before splitting silently drops the ~half of a table that is empty
strings; and Number('') is 0, not NaN.

Also updated:
- Progress and §9 sequencing tables: order 5 split into 5a (this, website
  only) and 5b (the four-repo wire change).
- website/BACKEND_DESIGN.md — shard_clilocs / shard_cliloc_meta, the three
  admin routes, and why there is no staged-approval flow and no public route.
- link/INTEGRATION.md — the char.profile field note now says explicitly not to
  expect the shard to resolve clilocs, and points at CLILOCS.md.
- §10 documentation obligations list CLILOCS.md.

Documentation only. Every claim was written after the corresponding behaviour
was observed running: the compressed-format finding and the parse failures
come from the real client files on this machine, and the counts (123,490
parsed → 67,496 stored) and timings from importing them into the live MariaDB.

PROJECT_TREE.md files are deliberately untouched — they are CI-generated by
the sync-project-tree workflow and say so in their header.

Co-Authored-By: Claude <noreply@anthropic.com>

wtclaude referenced this issue from a commit

2026-07-29 09:25:11 +00:00

docs(link): fill in the cliloc PR numbers, correct the cliloc file count

wtclaude added 1 commit 2026-07-29 09:25:11 +00:00

docs(link): fill in the cliloc PR numbers, correct the cliloc file count e3aabf9e3e

The progress table and CLILOCS.md carried #TBD placeholders until the PRs
existed; they now point at website #115 and docs #70.

Also corrects "all four Cliloc.* files" to eight (chs, cht, deu, enu, esp,
fra, jpn, kor) in both v3.md §8.6 and CLILOCS.md — the compression marker was
checked against every one of them, and understating the sample weakens the
claim it supports.

Co-Authored-By: Claude <noreply@anthropic.com>

wtclaude added 1 commit 2026-07-29 11:46:49 +00:00

docs(link): cliloc overlays for shard-added and shard-edited items ee0c146d7a

Shards edit items and add new ones, carrying cliloc ids no stock client table
has. The cliloc table now reads a SET of sources rather than one file — a base
plus every operator-maintained overlay under `custom/`, hash-gated together and
re-read on every boot, exactly as §6 reads the ServUO tree. Matching docs for
website #115.

website/CLILOCS.md gains a "Shard-added and shard-edited items" section: the
directory layout, merge precedence, the per-source breakdown an operator uses
to confirm an overlay took effect, and why `custom/` is a convention we chose
rather than one discovered from the shard — ServUO has no server-side notion of
a custom cliloc, they live in the patched client a shard distributes and
nothing in the tree declares them. Scale recorded: the live shard's script tree
references 16,434 cliloc ids and only 37 are absent from stock, which is why
this is an overlay and not a second table.

"Why there is no staged-approval flow" is replaced by "Two ways a refresh is
refused", because the set brings back the hazard a single file did not have. A
corrupt source fails the parse loudly; a source that has VANISHED parses
perfectly and imports a table quietly missing everything it contributed. That
is the same ambiguity §6 stages a facet removal for, so it is staged here too
(`needsReview`, `{approve:true}` to accept) — as a flag rather than §6's
approve/reject pair, because the atlas stores a pending decision SO THAT
approving re-parses, and here nothing is stored.

Two more traps recorded in §8.6 (now five), both found by running a shard-style
overlay rather than another stock-table fixture:

- Tidying punctuation unconditionally corrupts real names — a custom
  "Runic Gateway Sigil (v2)" rendered as "(v2". Stripping leftover brackets is
  right after a placeholder is removed and wrong otherwise, the same condition
  the `%` rule already had.
- Source labels must be forward-slashed and root-relative or the same directory
  fingerprints differently on Windows and Linux and every boot looks like a
  change — the identical bug §6 records.

BACKEND_DESIGN.md: the source set, the two refusals, and the updated admin
route notes (`approve`, `missingSources`, overlays found beside a file path).

Documentation only. Every claim was written after the behaviour was observed:
the counts come from the live shard's script tree, and the refusal/approve flow
and per-source breakdown are copied from real runs in website #115.

PROJECT_TREE.md files are deliberately untouched — CI-generated by the
sync-project-tree workflow.

Co-Authored-By: Claude <noreply@anthropic.com>

whitlocktech merged commit 70d49b7792 into edge

2026-07-29 11:58:28 +00:00

whitlocktech deleted branch docs/cliloc-table

2026-07-29 11:58:29 +00:00

whitlocktech referenced this issue from a commit

2026-07-29 11:58:30 +00:00

Merge pull request 'docs(link): the cliloc table, and why §8.6's recommendation was not implementable' (#70) from docs/cliloc-table into edge

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: RunicGateway/docs#70