RunicGateway/docs
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(link): the cliloc table, and why §8.6's recommendation was not implementable

Browse Source 
Protocol 3.0 §8.6 resolved as its own website-only change, landing ahead of
§8 so the marketplace ships with real item names. Matching documentation for
website #TBD.

NEW website/CLILOCS.md — operator-facing: why the conversion step exists, how
to convert, how to configure the path, the refresh contract, what gets stored
and how names are applied.

link/v3.md §8.6 rewritten. Two things in the original recommendation turned
out to be wrong, and both are recorded because the reasoning generalises:

1. The committed db/data/clilocs.json artifact predates the Part C
   corrections (§6.1) and violates both — no committed snapshot of derived
   content, and nothing EA-derived ever shipped. UO's strings are EA's,
   exactly as the creature sprites are.
2. "scripts/buildClilocs.js reads the UO client's Cliloc.enu" is not
   possible. EVERY current client ships its cliloc files compressed (first
   DWORD's high byte 0x8E, the Mythic container); the plain layout is what
   those files looked like before that change, and parsing one as the other
   does not fail cleanly — it yields ~19k records with negative ids, 1,722
   distinct keys out of 19,508, one 62 KB "string", and a truncation
   somewhere in the middle. ServUO's own Ultima.StringList cannot read it
   either, so VendorSearch.GetItemName is already inert on such a shard and
   the work could not be pushed to the plugin.

That second point also retires an open question in §8.2: the warning never to
call GetItemName in the market sweep costs us nothing we could otherwise have
had, because the in-game Vendor Search gump has the same gap.

Three traps found by building it are recorded: StringList.SaveStringList
RE-COMPRESSES on save (its output is byte-identical to its compressed input,
because its purpose is round-tripping a file back into the client); trimming a
text line before splitting silently drops the ~half of a table that is empty
strings; and Number('') is 0, not NaN.

Also updated:
- Progress and §9 sequencing tables: order 5 split into 5a (this, website
  only) and 5b (the four-repo wire change).
- website/BACKEND_DESIGN.md — shard_clilocs / shard_cliloc_meta, the three
  admin routes, and why there is no staged-approval flow and no public route.
- link/INTEGRATION.md — the char.profile field note now says explicitly not to
  expect the shard to resolve clilocs, and points at CLILOCS.md.
- §10 documentation obligations list CLILOCS.md.

Documentation only. Every claim was written after the corresponding behaviour
was observed running: the compressed-format finding and the parse failures
come from the real client files on this machine, and the counts (123,490
parsed → 67,496 stored) and timings from importing them into the live MariaDB.

PROJECT_TREE.md files are deliberately untouched — they are CI-generated by
the sync-project-tree workflow and say so in their header.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
claude
2026-07-29 04:22:05 -05:00
parent f715323aa0
commit be9f5019fa
4 changed files with 327 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files

73
link/v3.md
View File

@@ -15,9 +15,14 @@ Each part is marked off here as it lands on `edge`. §9 carries the same state p
| 2 | **B/1** — `world.ruleset` (§5) | ✅ **Done** | servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66) |
| 3 | **C** — spawn atlas (§6) | ✅ **Done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables) + [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113) (API + pages + admin panel), docs [#67](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/67) + [#68](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/68) |
| 4 | **B/2** — `points.board` (§7) | ✅ **Done** | servuo-plugins [#4](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/4), link [#18](https://gitea.whitlocktech.com/RunicGateway/link/pulls/18), website [#114](https://gitea.whitlocktech.com/RunicGateway/website/pulls/114), docs [#69](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/69) |
| 5 | **B/3** — `vendor.listing` (§8) | ⬜ Not started | — |
| 5a | **B/3 dependency** — cliloc table (§8.6) | 🟨 In review | website [#TBD](https://gitea.whitlocktech.com/RunicGateway/website/pulls), docs [#TBD](https://gitea.whitlocktech.com/RunicGateway/docs/pulls) |
| 5b | **B/3** — `vendor.listing` (§8) | ⬜ Not started | — |
| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | ⬜ Not started | — |
Order 5 split in two once §8.6's cliloc dependency turned out to be a client-format problem rather
than a parser (see §8.6). 5a is website-only and lands first so the marketplace ships with real item
names; 5b is the four-repo wire change.
---
## 1. Why 3.0
@@ -715,17 +720,62 @@ admin can turn the stream on. `uoLinkSocket` paginates `/market` on reconnect, b
`/market/vendors/:serial`, behind `requireFeature('market')`. **Rate-limit it** — this is the first
genuinely expensive public endpoint; `express-rate-limit` is already a dependency.
### 8.6 The open dependency — cliloc names
### 8.6 The open dependency — cliloc names ✅ Resolved (shipped ahead of §8)
`CharacterSheet.jsx:14-15` already documents the gap ("without a cliloc table on the site we can only
show literals") and renders equipment as `id {itemId}`. Search-by-name needs that table.
`CharacterSheet.jsx:14-15` documented the gap ("without a cliloc table on the site we can only
show literals") and rendered equipment as `id {itemId}`. Search-by-name needs that table.
- **Recommended:** `scripts/buildClilocs.js` reads the UO client's `Cliloc.enu` → committed
`db/data/clilocs.json`; ingest denormalizes into `shard_vendor_items.display_name`. Same
build-artifact pattern as §6, and it **also fixes the character sheet**.
- **Fallback:** ship with item-art + price + region filters, and name search only over renamed items.
**Resolved as its own website-only change, landed BEFORE the market so `/site/market` ships with real
item names.** Full design and operator guide: [`docs/website/CLILOCS.md`](../website/CLILOCS.md).
Ingest denormalizes the resolved name into `shard_vendor_items.display_name` as planned.
This decision is the reason §8 is sequenced last.
Two things in the original recommendation above turned out to be wrong, and both are worth recording
because the reasoning generalises.
**1. The committed `db/data/clilocs.json` artifact was dropped.** It predates the two Part C
corrections (§6.1) and violates both: no committed snapshot of derived content, and nothing
EA-derived ever shipped. UO's strings are EA's, exactly as the creature sprites are. Replaced with
the §6 pattern instead — parse on every boot from an operator-configured path, hash-gated, output
gitignored, `PARSER_VERSION` counted as drift.
**2. `scripts/buildClilocs.js reads the UO client's Cliloc.enu` is not possible, and the reason
matters.** **Every current client ships its cliloc files COMPRESSED** — all four `Cliloc.*` files
open with a DWORD whose high byte is `0x8E`, the "Mythic" container. The plain layout (`02 00 00 00
01 00`, then `{int32 number, byte flag, uint16 length, UTF-8}`) is what those files looked like
*before* that change. Parsing a compressed file as plain does not fail cleanly: it yields ~19k
"records" with negative ids, 1,722 distinct keys out of 19,508, one 62 KB "string", and a truncation
somewhere in the middle.
Decompressing means porting an inverse-BWT coder with a 1 KB frequency header — a few hundred lines
whose failure mode is plausible-looking garbage rather than an error. Two facts closed off the
alternatives:
- **ServUO cannot read it either.** Its bundled `Ultima.StringList` implements only the plain layout,
so on a modern client `VendorSearch.StringList` is null and `VendorSearch.GetItemName` returns
`item.Name`. **The in-game Vendor Search gump has the same gap** — which also means §8.2's warning
never to call `GetItemName` in the sweep costs us nothing we could otherwise have had.
- The shard therefore cannot supply names on our behalf, so this could not be pushed to the plugin.
⇒ **the operator converts once, from their own client, and the site reads the result.** Accepted
shapes are the plain binary layout and a `number<TAB|,|;>text` export; the site sniffs which.
`server/tools/cliloc-export/` drives UOFiddler's `Ultima.dll` (the decompressor that already exists)
and writes the plain form. A shard that never converts is fully supported — names render as ids,
exactly as before.
Three traps found by building it, all recorded in `CLILOCS.md`:
- **`StringList.SaveStringList` RE-COMPRESSES on save.** It looks exactly like the export path and is
not; its output is byte-identical to its compressed input, because its purpose is round-tripping a
file back into the client.
- **Trimming a text line before splitting silently drops half the table.** Roughly half of a real
cliloc table is empty strings (ids the client reserves), exported as `1005008<TAB>`. Trimming eats
the trailing separator, leaving a bare number that then looks like a header row — 55,994 of 123,490
entries vanished, and the import still looked successful.
- **`Number('')` is `0`, not `NaN`.** A line starting with a separator imports as a bogus cliloc 0
unless the empty field is rejected explicitly.
Blank entries are dropped at import (123,490 parsed → **67,496** stored), which also makes the binary
and text paths converge on identical content.
### 8.7 Client
@@ -743,7 +793,8 @@ inherently up to one full cycle old, and the UI must say so.
| 2 | **B/1** — `world.ruleset` (§5) | all four | new kind | ✅ Done |
| 3 | **C** — spawn atlas (§6) | website, docs | none | ✅ Done |
| 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | ✅ Done |
| 5 | **B/3** — `vendor.listing` (§8) | all four | new kinds | ⬜ |
| 5a | **B/3 dependency** — cliloc table (§8.6) | website, docs | none | 🟨 In review |
| 5b | **B/3** — `vendor.listing` (§8) | all four | new kinds | ⬜ |
| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | ⬜ |
---
@@ -761,7 +812,7 @@ inherently up to one full cycle old, and the UI must say so.
in the security section.
- NEW `website/SHARD_VISIBILITY.md` — admin-facing: what each feature exposes, what each rung means,
what cannot be loosened.
- NEW `website/SPAWN_ATLAS.md`, NEW `website/MARKETPLACE.md`.
- NEW `website/SPAWN_ATLAS.md`, NEW `website/CLILOCS.md`, NEW `website/MARKETPLACE.md`.
- `PROJECT_TREE.md` in each touched repo.
- `npm run swagger` **and** `npm run routes:manifest` on every route-touching PR — both are committed
artifacts, and `test/routeManifest.test.js` fails on drift.