docs(link): the cliloc table, and why §8.6's recommendation was not implementable
Protocol 3.0 §8.6 resolved as its own website-only change, landing ahead of
§8 so the marketplace ships with real item names. Matching documentation for
website #TBD.
NEW website/CLILOCS.md — operator-facing: why the conversion step exists, how
to convert, how to configure the path, the refresh contract, what gets stored
and how names are applied.
link/v3.md §8.6 rewritten. Two things in the original recommendation turned
out to be wrong, and both are recorded because the reasoning generalises:
1. The committed db/data/clilocs.json artifact predates the Part C
corrections (§6.1) and violates both — no committed snapshot of derived
content, and nothing EA-derived ever shipped. UO's strings are EA's,
exactly as the creature sprites are.
2. "scripts/buildClilocs.js reads the UO client's Cliloc.enu" is not
possible. EVERY current client ships its cliloc files compressed (first
DWORD's high byte 0x8E, the Mythic container); the plain layout is what
those files looked like before that change, and parsing one as the other
does not fail cleanly — it yields ~19k records with negative ids, 1,722
distinct keys out of 19,508, one 62 KB "string", and a truncation
somewhere in the middle. ServUO's own Ultima.StringList cannot read it
either, so VendorSearch.GetItemName is already inert on such a shard and
the work could not be pushed to the plugin.
That second point also retires an open question in §8.2: the warning never to
call GetItemName in the market sweep costs us nothing we could otherwise have
had, because the in-game Vendor Search gump has the same gap.
Three traps found by building it are recorded: StringList.SaveStringList
RE-COMPRESSES on save (its output is byte-identical to its compressed input,
because its purpose is round-tripping a file back into the client); trimming a
text line before splitting silently drops the ~half of a table that is empty
strings; and Number('') is 0, not NaN.
Also updated:
- Progress and §9 sequencing tables: order 5 split into 5a (this, website
only) and 5b (the four-repo wire change).
- website/BACKEND_DESIGN.md — shard_clilocs / shard_cliloc_meta, the three
admin routes, and why there is no staged-approval flow and no public route.
- link/INTEGRATION.md — the char.profile field note now says explicitly not to
expect the shard to resolve clilocs, and points at CLILOCS.md.
- §10 documentation obligations list CLILOCS.md.
Documentation only. Every claim was written after the corresponding behaviour
was observed running: the compressed-format finding and the parse failures
come from the real client files on this machine, and the counts (123,490
parsed → 67,496 stored) and timings from importing them into the live MariaDB.
PROJECT_TREE.md files are deliberately untouched — they are CI-generated by
the sync-project-tree workflow and say so in their header.
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -488,7 +488,7 @@ Full character sheet: stats, all trained skills, worn equipment with flattened i
|
|||||||
Field notes:
|
Field notes:
|
||||||
- `skills[].base` is trained value, `value` includes item/temp bonuses, `cap` is the cap. **Do not assume `base <= cap`** — GM characters can exceed it.
|
- `skills[].base` is trained value, `value` includes item/temp bonuses, `cap` is the cap. **Do not assume `base <= cap`** — GM characters can exceed it.
|
||||||
- `equipment[].mods` is a flattened map of every non-zero AOS attribute on the item (weapon or armor). Empty `{}` for plain items.
|
- `equipment[].mods` is a flattened map of every non-zero AOS attribute on the item (weapon or armor). Empty `{}` for plain items.
|
||||||
- Item names are usually **clilocs**, not strings: use `name` when present, otherwise resolve `cliloc` against a UO cliloc table on the site.
|
- Item names are usually **clilocs**, not strings: use `name` when present, otherwise resolve `cliloc` against a UO cliloc table on the site. **Do not expect the shard to resolve them for you** — on any modern client ServUO's own `Ultima.StringList` cannot read the client's compressed cliloc files, so `VendorSearch.GetItemName` returns `item.Name` and the in-game Vendor Search gump has the same gap. Building that table is a consumer-side job; the website's is described in [`website/CLILOCS.md`](../website/CLILOCS.md).
|
||||||
- `titles` (Protocol 2.0): `selected` is the index into `reward` currently displayed (`-1` if none). `fameKarma`/`skill` are computed display titles, omitted when the character has none. `reward` entries may be a **cliloc number as a string** or a literal string — resolve numeric ones against your cliloc table, same as item names.
|
- `titles` (Protocol 2.0): `selected` is the index into `reward` currently displayed (`-1` if none). `fameKarma`/`skill` are computed display titles, omitted when the character has none. `reward` entries may be a **cliloc number as a string** or a literal string — resolve numeric ones against your cliloc table, same as item names.
|
||||||
- Errors: unknown account → **404** `{"kind":"bridge.error","reason":"unknown account"}`; bad slot → **404**/**400** similarly.
|
- Errors: unknown account → **404** `{"kind":"bridge.error","reason":"unknown account"}`; bad slot → **404**/**400** similarly.
|
||||||
|
|
||||||
|
|||||||
73
link/v3.md
73
link/v3.md
@@ -15,9 +15,14 @@ Each part is marked off here as it lands on `edge`. §9 carries the same state p
|
|||||||
| 2 | **B/1** — `world.ruleset` (§5) | ✅ **Done** | servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66) |
|
| 2 | **B/1** — `world.ruleset` (§5) | ✅ **Done** | servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66) |
|
||||||
| 3 | **C** — spawn atlas (§6) | ✅ **Done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables) + [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113) (API + pages + admin panel), docs [#67](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/67) + [#68](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/68) |
|
| 3 | **C** — spawn atlas (§6) | ✅ **Done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables) + [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113) (API + pages + admin panel), docs [#67](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/67) + [#68](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/68) |
|
||||||
| 4 | **B/2** — `points.board` (§7) | ✅ **Done** | servuo-plugins [#4](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/4), link [#18](https://gitea.whitlocktech.com/RunicGateway/link/pulls/18), website [#114](https://gitea.whitlocktech.com/RunicGateway/website/pulls/114), docs [#69](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/69) |
|
| 4 | **B/2** — `points.board` (§7) | ✅ **Done** | servuo-plugins [#4](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/4), link [#18](https://gitea.whitlocktech.com/RunicGateway/link/pulls/18), website [#114](https://gitea.whitlocktech.com/RunicGateway/website/pulls/114), docs [#69](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/69) |
|
||||||
| 5 | **B/3** — `vendor.listing` (§8) | ⬜ Not started | — |
|
| 5a | **B/3 dependency** — cliloc table (§8.6) | 🟨 In review | website [#TBD](https://gitea.whitlocktech.com/RunicGateway/website/pulls), docs [#TBD](https://gitea.whitlocktech.com/RunicGateway/docs/pulls) |
|
||||||
|
| 5b | **B/3** — `vendor.listing` (§8) | ⬜ Not started | — |
|
||||||
| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | ⬜ Not started | — |
|
| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | ⬜ Not started | — |
|
||||||
|
|
||||||
|
Order 5 split in two once §8.6's cliloc dependency turned out to be a client-format problem rather
|
||||||
|
than a parser (see §8.6). 5a is website-only and lands first so the marketplace ships with real item
|
||||||
|
names; 5b is the four-repo wire change.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 1. Why 3.0
|
## 1. Why 3.0
|
||||||
@@ -715,17 +720,62 @@ admin can turn the stream on. `uoLinkSocket` paginates `/market` on reconnect, b
|
|||||||
`/market/vendors/:serial`, behind `requireFeature('market')`. **Rate-limit it** — this is the first
|
`/market/vendors/:serial`, behind `requireFeature('market')`. **Rate-limit it** — this is the first
|
||||||
genuinely expensive public endpoint; `express-rate-limit` is already a dependency.
|
genuinely expensive public endpoint; `express-rate-limit` is already a dependency.
|
||||||
|
|
||||||
### 8.6 The open dependency — cliloc names
|
### 8.6 The open dependency — cliloc names ✅ Resolved (shipped ahead of §8)
|
||||||
|
|
||||||
`CharacterSheet.jsx:14-15` already documents the gap ("without a cliloc table on the site we can only
|
`CharacterSheet.jsx:14-15` documented the gap ("without a cliloc table on the site we can only
|
||||||
show literals") and renders equipment as `id {itemId}`. Search-by-name needs that table.
|
show literals") and rendered equipment as `id {itemId}`. Search-by-name needs that table.
|
||||||
|
|
||||||
- **Recommended:** `scripts/buildClilocs.js` reads the UO client's `Cliloc.enu` → committed
|
**Resolved as its own website-only change, landed BEFORE the market so `/site/market` ships with real
|
||||||
`db/data/clilocs.json`; ingest denormalizes into `shard_vendor_items.display_name`. Same
|
item names.** Full design and operator guide: [`docs/website/CLILOCS.md`](../website/CLILOCS.md).
|
||||||
build-artifact pattern as §6, and it **also fixes the character sheet**.
|
Ingest denormalizes the resolved name into `shard_vendor_items.display_name` as planned.
|
||||||
- **Fallback:** ship with item-art + price + region filters, and name search only over renamed items.
|
|
||||||
|
|
||||||
This decision is the reason §8 is sequenced last.
|
Two things in the original recommendation above turned out to be wrong, and both are worth recording
|
||||||
|
because the reasoning generalises.
|
||||||
|
|
||||||
|
**1. The committed `db/data/clilocs.json` artifact was dropped.** It predates the two Part C
|
||||||
|
corrections (§6.1) and violates both: no committed snapshot of derived content, and nothing
|
||||||
|
EA-derived ever shipped. UO's strings are EA's, exactly as the creature sprites are. Replaced with
|
||||||
|
the §6 pattern instead — parse on every boot from an operator-configured path, hash-gated, output
|
||||||
|
gitignored, `PARSER_VERSION` counted as drift.
|
||||||
|
|
||||||
|
**2. `scripts/buildClilocs.js reads the UO client's Cliloc.enu` is not possible, and the reason
|
||||||
|
matters.** **Every current client ships its cliloc files COMPRESSED** — all four `Cliloc.*` files
|
||||||
|
open with a DWORD whose high byte is `0x8E`, the "Mythic" container. The plain layout (`02 00 00 00
|
||||||
|
01 00`, then `{int32 number, byte flag, uint16 length, UTF-8}`) is what those files looked like
|
||||||
|
*before* that change. Parsing a compressed file as plain does not fail cleanly: it yields ~19k
|
||||||
|
"records" with negative ids, 1,722 distinct keys out of 19,508, one 62 KB "string", and a truncation
|
||||||
|
somewhere in the middle.
|
||||||
|
|
||||||
|
Decompressing means porting an inverse-BWT coder with a 1 KB frequency header — a few hundred lines
|
||||||
|
whose failure mode is plausible-looking garbage rather than an error. Two facts closed off the
|
||||||
|
alternatives:
|
||||||
|
|
||||||
|
- **ServUO cannot read it either.** Its bundled `Ultima.StringList` implements only the plain layout,
|
||||||
|
so on a modern client `VendorSearch.StringList` is null and `VendorSearch.GetItemName` returns
|
||||||
|
`item.Name`. **The in-game Vendor Search gump has the same gap** — which also means §8.2's warning
|
||||||
|
never to call `GetItemName` in the sweep costs us nothing we could otherwise have had.
|
||||||
|
- The shard therefore cannot supply names on our behalf, so this could not be pushed to the plugin.
|
||||||
|
|
||||||
|
⇒ **the operator converts once, from their own client, and the site reads the result.** Accepted
|
||||||
|
shapes are the plain binary layout and a `number<TAB|,|;>text` export; the site sniffs which.
|
||||||
|
`server/tools/cliloc-export/` drives UOFiddler's `Ultima.dll` (the decompressor that already exists)
|
||||||
|
and writes the plain form. A shard that never converts is fully supported — names render as ids,
|
||||||
|
exactly as before.
|
||||||
|
|
||||||
|
Three traps found by building it, all recorded in `CLILOCS.md`:
|
||||||
|
|
||||||
|
- **`StringList.SaveStringList` RE-COMPRESSES on save.** It looks exactly like the export path and is
|
||||||
|
not; its output is byte-identical to its compressed input, because its purpose is round-tripping a
|
||||||
|
file back into the client.
|
||||||
|
- **Trimming a text line before splitting silently drops half the table.** Roughly half of a real
|
||||||
|
cliloc table is empty strings (ids the client reserves), exported as `1005008<TAB>`. Trimming eats
|
||||||
|
the trailing separator, leaving a bare number that then looks like a header row — 55,994 of 123,490
|
||||||
|
entries vanished, and the import still looked successful.
|
||||||
|
- **`Number('')` is `0`, not `NaN`.** A line starting with a separator imports as a bogus cliloc 0
|
||||||
|
unless the empty field is rejected explicitly.
|
||||||
|
|
||||||
|
Blank entries are dropped at import (123,490 parsed → **67,496** stored), which also makes the binary
|
||||||
|
and text paths converge on identical content.
|
||||||
|
|
||||||
### 8.7 Client
|
### 8.7 Client
|
||||||
|
|
||||||
@@ -743,7 +793,8 @@ inherently up to one full cycle old, and the UI must say so.
|
|||||||
| 2 | **B/1** — `world.ruleset` (§5) | all four | new kind | ✅ Done |
|
| 2 | **B/1** — `world.ruleset` (§5) | all four | new kind | ✅ Done |
|
||||||
| 3 | **C** — spawn atlas (§6) | website, docs | none | ✅ Done |
|
| 3 | **C** — spawn atlas (§6) | website, docs | none | ✅ Done |
|
||||||
| 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | ✅ Done |
|
| 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | ✅ Done |
|
||||||
| 5 | **B/3** — `vendor.listing` (§8) | all four | new kinds | ⬜ |
|
| 5a | **B/3 dependency** — cliloc table (§8.6) | website, docs | none | 🟨 In review |
|
||||||
|
| 5b | **B/3** — `vendor.listing` (§8) | all four | new kinds | ⬜ |
|
||||||
| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | ⬜ |
|
| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | ⬜ |
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -761,7 +812,7 @@ inherently up to one full cycle old, and the UI must say so.
|
|||||||
in the security section.
|
in the security section.
|
||||||
- NEW `website/SHARD_VISIBILITY.md` — admin-facing: what each feature exposes, what each rung means,
|
- NEW `website/SHARD_VISIBILITY.md` — admin-facing: what each feature exposes, what each rung means,
|
||||||
what cannot be loosened.
|
what cannot be loosened.
|
||||||
- NEW `website/SPAWN_ATLAS.md`, NEW `website/MARKETPLACE.md`.
|
- NEW `website/SPAWN_ATLAS.md`, NEW `website/CLILOCS.md`, NEW `website/MARKETPLACE.md`.
|
||||||
- `PROJECT_TREE.md` in each touched repo.
|
- `PROJECT_TREE.md` in each touched repo.
|
||||||
- `npm run swagger` **and** `npm run routes:manifest` on every route-touching PR — both are committed
|
- `npm run swagger` **and** `npm run routes:manifest` on every route-touching PR — both are committed
|
||||||
artifacts, and `test/routeManifest.test.js` fails on drift.
|
artifacts, and `test/routeManifest.test.js` fails on drift.
|
||||||
|
|||||||
@@ -486,6 +486,51 @@ artwork: sprites live in the operator's own client `.mul`/`.uop` files and are t
|
|||||||
redistribute. An operator supplies art via a gitignored map plus images under the (already
|
redistribute. An operator supplies art via a gitignored map plus images under the (already
|
||||||
gitignored) `server/uploads/atlas/`. Text-only is the normal, supported state.
|
gitignored) `server/uploads/atlas/`. Text-only is the normal, supported state.
|
||||||
|
|
||||||
|
### shard_clilocs / shard_cliloc_meta — UO's localization table (Protocol 3.0)
|
||||||
|
|
||||||
|
Items on the wire carry a `LabelNumber`, not a name. The bridge has always sent it —
|
||||||
|
`char.profile.equipment.cliloc`, reward titles as a cliloc number in string form, and one per
|
||||||
|
marketplace listing — but with no table to resolve it against, the character sheet could only render
|
||||||
|
`id 1023721` where the game renders "quarter staff".
|
||||||
|
|
||||||
|
| Table | Shape |
|
||||||
|
|---|---|
|
||||||
|
| `shard_clilocs` | `number` INT PK, `flag`, `text` TEXT |
|
||||||
|
| `shard_cliloc_meta` | Singleton (`id = 1`), `payload` JSON (source file, sha256, count, `parserVersion`), `imported_at` |
|
||||||
|
|
||||||
|
Import-owned and all-or-nothing in one transaction, same contract as the atlas — including **`DELETE`,
|
||||||
|
not `TRUNCATE`**, for the same reason.
|
||||||
|
|
||||||
|
**Sourced from a file the operator converts once from their own UO client**, at a path from the
|
||||||
|
`cliloc_client_path` setting falling back to `UO_CLIENT_PATH`. Nothing client-derived is committed:
|
||||||
|
UO's strings are EA's, exactly as the creature sprites are. A shard with nothing configured is fully
|
||||||
|
supported — names render as ids. Full design and operator guide: [`CLILOCS.md`](CLILOCS.md).
|
||||||
|
|
||||||
|
The conversion step is not avoidable: **every current client ships its cliloc files compressed**
|
||||||
|
(first DWORD's high byte `0x8E`), and ServUO's own bundled `Ultima.StringList` cannot read that
|
||||||
|
either — so the shard cannot supply names on our behalf. The plain layout and a delimited text export
|
||||||
|
are both accepted, sniffed by header rather than extension.
|
||||||
|
|
||||||
|
Three decisions worth stating:
|
||||||
|
|
||||||
|
- **`text` is TEXT, not VARCHAR.** Long property descriptions reach 12 KB. The index that matters for
|
||||||
|
marketplace search is the denormalized `shard_vendor_items.display_name`, not this table.
|
||||||
|
- **Blank entries are dropped at import** — 123,490 parsed → **67,496** stored. Roughly half a cliloc
|
||||||
|
table is empty strings for ids the client reserves and never uses; a row that resolves to no name is
|
||||||
|
indistinguishable from no row at all, and dropping them makes the binary and text imports converge
|
||||||
|
on identical content.
|
||||||
|
- **No staged-approval flow, unlike the atlas.** The atlas escalates facet loss because a half-copied
|
||||||
|
tree and a real map change are indistinguishable from inside the process. A cliloc file is one file
|
||||||
|
with one hash, and a partial copy makes the parser fail on a truncated record — the ambiguity the
|
||||||
|
atlas must escalate is one this parser simply detects, so it refuses the import and leaves the
|
||||||
|
previous table serving.
|
||||||
|
|
||||||
|
**Resolution is server-side and there is no public route.** The table is never served *as* a table:
|
||||||
|
67k rows would dwarf any page using them, and the Android client consumes the same already-resolved
|
||||||
|
JSON. `resolveMany()` returns only ids that resolved to something displayable — placeholders like
|
||||||
|
`~1_val~` are stripped, since the bridge sends the id and never the property packet that carries the
|
||||||
|
arguments — and it never throws, because a cliloc lookup is decoration on a character sheet.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 4. API contract
|
## 4. API contract
|
||||||
@@ -741,6 +786,9 @@ file a route sits in — that is the property the route manifest freezes.
|
|||||||
| POST | `/shard/atlas/import` | re-import without restarting; `{force}` ignores the hash gate. **An unreadable tree answers 200 with `status:"unavailable"`, not 500** — `refresh()` reports outcomes rather than throwing (the boot path must never be blocked by a bad tree) and that contract is preserved at the API. |
|
| POST | `/shard/atlas/import` | re-import without restarting; `{force}` ignores the hash gate. **An unreadable tree answers 200 with `status:"unavailable"`, not 500** — `refresh()` reports outcomes rather than throwing (the boot path must never be blocked by a bad tree) and that contract is preserved at the API. |
|
||||||
| POST | `/shard/atlas/approve` · `/shard/atlas/reject` | answer a refresh staged because it would REMOVE a facet. Approving **re-parses** the tree, so what lands matches it at approval time; rejecting is remembered against those source hashes so it does not re-prompt every restart. 404 when nothing is staged. |
|
| POST | `/shard/atlas/approve` · `/shard/atlas/reject` | answer a refresh staged because it would REMOVE a facet. Approving **re-parses** the tree, so what lands matches it at approval time; rejecting is remembered against those source hashes so it does not re-prompt every restart. 404 when nothing is staged. |
|
||||||
| PUT | `/shard/atlas/path` | point the atlas at a different tree (persisted as `spawn_atlas_servuo_path`, which wins over `SERVUO_PATH`). Blank clears it. Deliberately **does not import** — moving the mount and reloading the world are separate decisions — and returns fresh status so the panel can offer the import next. |
|
| PUT | `/shard/atlas/path` | point the atlas at a different tree (persisted as `spawn_atlas_servuo_path`, which wins over `SERVUO_PATH`). Blank clears it. Deliberately **does not import** — moving the mount and reloading the world are separate decisions — and returns fresh status so the panel can offer the import next. |
|
||||||
|
| GET | `/shard/clilocs` | cliloc-table status (`adminOnly`): the configured path, the file actually resolved (the path may be a directory), readability, drift against what is loaded, and the entry count. `configured:false` is a supported state — item names then render as ids. No public counterpart: the table is never served *as* a table. |
|
||||||
|
| POST | `/shard/clilocs/import` | reload after a client patch; `{force}` ignores the hash gate. **A missing file — or the likely mistake of pointing at the client's own COMPRESSED `Cliloc.enu` — answers 200 with `status:"unavailable"` and a `code`, not 500.** `COMPRESSED` is called out by name: a 500 would say only "something broke", and the operator needs to be told which file to convert. |
|
||||||
|
| PUT | `/shard/clilocs/path` | point the site at a different cliloc file or directory (persisted as `cliloc_client_path`, which wins over `UO_CLIENT_PATH`). Blank clears it. Deliberately **does not import**, same reasoning as the atlas path. |
|
||||||
|
|
||||||
Every admin write logs to `activity_log`.
|
Every admin write logs to `activity_log`.
|
||||||
|
|
||||||
|
|||||||
216
website/CLILOCS.md
Normal file
216
website/CLILOCS.md
Normal file
@@ -0,0 +1,216 @@
|
|||||||
|
# Cliloc table (item and title names)
|
||||||
|
|
||||||
|
**Status:** Complete on `edge` — website [#TBD](https://gitea.whitlocktech.com/RunicGateway/website/pulls).
|
||||||
|
**Design:** [`docs/link/v3.md` §8.6](../link/v3.md) — Protocol 3.0, the dependency Part B/3 was sequenced behind.
|
||||||
|
|
||||||
|
A "cliloc" is UO's localization table: an integer id mapped to a display string.
|
||||||
|
**Items on the wire carry a `LabelNumber`, not a name.** The bridge has always
|
||||||
|
sent that number — `char.profile.equipment` has a `cliloc` field, reward titles
|
||||||
|
arrive as a cliloc number in string form, and every marketplace listing carries
|
||||||
|
one — but the site had no table to look it up in, so a character sheet could only
|
||||||
|
render `id 1023721` where the game renders **"quarter staff"**.
|
||||||
|
|
||||||
|
The number was never the missing piece. The table was.
|
||||||
|
|
||||||
|
## Why the operator has to convert the file
|
||||||
|
|
||||||
|
This is the awkward part, and it is not avoidable:
|
||||||
|
|
||||||
|
**Every current UO client ships its cliloc files compressed.** The four
|
||||||
|
`Cliloc.*` files in a modern client all begin with a DWORD whose high byte is
|
||||||
|
`0x8E` — the "Mythic" compressed container. The plain layout this site parses is
|
||||||
|
what those files looked like *before* that change.
|
||||||
|
|
||||||
|
Decompressing it means an inverse-BWT coder with a frequency header — a few
|
||||||
|
hundred lines of bit-level work whose failure mode is plausible-looking garbage
|
||||||
|
rather than an error. The site has no business carrying that at runtime.
|
||||||
|
|
||||||
|
Two facts make the alternatives worse, not better:
|
||||||
|
|
||||||
|
- **ServUO cannot read it either.** Its bundled `Ultima.StringList` implements
|
||||||
|
only the plain layout, so on a modern client `VendorSearch.StringList` is null
|
||||||
|
and `VendorSearch.GetItemName` returns `item.Name` — usually nothing. The
|
||||||
|
shard cannot supply names on our behalf; the in-game Vendor Search gump has the
|
||||||
|
same gap.
|
||||||
|
- **Nothing client-derived may be committed.** UO's strings are EA's. The repo
|
||||||
|
ships no string table for the same reason it ships no artwork and no map
|
||||||
|
snapshot — see [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md).
|
||||||
|
|
||||||
|
So the conversion happens **once, on the operator's machine, against their own
|
||||||
|
client**, and the site reads the result from a path it is given. A shard that
|
||||||
|
never does this is in a fully supported state: names render as ids, exactly as
|
||||||
|
they did before the table existed.
|
||||||
|
|
||||||
|
## Converting
|
||||||
|
|
||||||
|
Either format below is accepted; the site sniffs which one it was handed.
|
||||||
|
|
||||||
|
| Format | Fidelity | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| **Plain binary** (recommended) | Exact | 6-byte header, then `{int32 number, byte flag, uint16 length, UTF-8}` records |
|
||||||
|
| Delimited text | Loses leading/trailing whitespace | `number<TAB\|,\|;>text` per line; a header row, blank lines and `#` comments are ignored |
|
||||||
|
|
||||||
|
The whitespace caveat is real but cosmetic: ~1,300 of the 123,490 entries in a
|
||||||
|
stock `Cliloc.enu` are label prefixes like `"max = "` whose trailing space is
|
||||||
|
meaningful when the client concatenates a value onto them. Nothing on this site
|
||||||
|
concatenates, and every consumer passes through `displayText()`, which trims.
|
||||||
|
|
||||||
|
### Using the bundled tool
|
||||||
|
|
||||||
|
`server/tools/cliloc-export/` is a small .NET console app that drives
|
||||||
|
[UOFiddler](https://github.com/polserver/UOFiddler)'s `Ultima.dll` — the
|
||||||
|
decompressor that already exists and is already maintained — and writes the plain
|
||||||
|
format. It loads that DLL **reflectively** so it compiles against any SDK, and it
|
||||||
|
writes the records by hand because UOFiddler's own `SaveStringList` *re-compresses*
|
||||||
|
on save (its purpose is round-tripping a file back into the client, so its output
|
||||||
|
is byte-identical to its input — a trap worth knowing about).
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd website/server/tools/cliloc-export
|
||||||
|
dotnet build -c Release
|
||||||
|
|
||||||
|
# binary (recommended)
|
||||||
|
dotnet run -- "<UOFiddler>/Ultima.dll" "<UO client>/Cliloc.enu" /srv/uo-data/clilocs.plain
|
||||||
|
|
||||||
|
# or tab-delimited
|
||||||
|
dotnet run -- "<UOFiddler>/Ultima.dll" "<UO client>/Cliloc.enu" /srv/uo-data/clilocs.tsv --tsv
|
||||||
|
```
|
||||||
|
|
||||||
|
A UOFiddler GUI export works equally well — anything producing one of the two
|
||||||
|
shapes above is fine.
|
||||||
|
|
||||||
|
## Configuring the path
|
||||||
|
|
||||||
|
Two ways to point at the converted file, the setting winning over the
|
||||||
|
environment:
|
||||||
|
|
||||||
|
| Source | Notes |
|
||||||
|
|---|---|
|
||||||
|
| `cliloc_client_path` setting | Admin-editable (Admin → Shard); takes effect on the next refresh without a redeploy |
|
||||||
|
| `UO_CLIENT_PATH` env var | The deploy-time default, since the path usually describes a mount the deployment sets up |
|
||||||
|
|
||||||
|
The value may be **the file itself or a directory to search**, because both are
|
||||||
|
natural answers to "where is it". A directory is searched case-insensitively (the
|
||||||
|
client writes `Cliloc.enu` on Windows; the site usually runs on Linux) for, in
|
||||||
|
order: `clilocs.tsv`, `clilocs.csv`, `cliloc.plain.enu`, `cliloc.enu.plain`,
|
||||||
|
`clilocs.txt`, `cliloc.enu`.
|
||||||
|
|
||||||
|
That ordering puts explicitly-converted names first on purpose. Pointing the
|
||||||
|
setting straight at an unconverted client directory finds `cliloc.enu`, which is
|
||||||
|
compressed — and the site says so by name rather than failing obscurely:
|
||||||
|
|
||||||
|
```
|
||||||
|
status: unavailable
|
||||||
|
code: COMPRESSED
|
||||||
|
reason: This is a compressed (Mythic-format) cliloc file, which the site cannot
|
||||||
|
read. Convert it to the plain format first — see docs/website/CLILOCS.md.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Refresh contract
|
||||||
|
|
||||||
|
Identical in shape to the spawn atlas, and for the same reasons:
|
||||||
|
|
||||||
|
- **It never blocks startup.** No path, an unreadable file, a wrong-format file,
|
||||||
|
a database error — all caught and logged. The site comes up either way.
|
||||||
|
- **Hash-gated.** The boot path hashes the file and skips the parse entirely when
|
||||||
|
it matches what is loaded, which is every restart that did not follow a client
|
||||||
|
patch. Measured on a stock table: **14 ms** for the no-op, **663 ms** for a full
|
||||||
|
parse and replace.
|
||||||
|
- **A `PARSER_VERSION` bump also counts as drift**, so a corrected parse reaches
|
||||||
|
an install whose client never patches.
|
||||||
|
|
||||||
|
### Why there is no staged-approval flow
|
||||||
|
|
||||||
|
The atlas stages a refresh that would *remove a facet*, because a half-copied
|
||||||
|
tree and a real map change are indistinguishable from inside the process. A
|
||||||
|
cliloc file is one file with one hash, and its realistic corruption — a partial
|
||||||
|
copy — makes the parser fail on a truncated record instead of yielding a
|
||||||
|
plausible-but-short table. **The ambiguity the atlas has to escalate to a human is
|
||||||
|
one this parser can simply detect**, so it refuses the import and leaves the
|
||||||
|
previous table serving. Verified: a file truncated to half its length reports
|
||||||
|
|
||||||
|
```
|
||||||
|
code: TRUNCATED
|
||||||
|
reason: Truncated record header at byte 2486759 (74909 entries read)
|
||||||
|
```
|
||||||
|
|
||||||
|
and the 67,496 rows already loaded are untouched.
|
||||||
|
|
||||||
|
## What gets stored
|
||||||
|
|
||||||
|
| | |
|
||||||
|
|---|---|
|
||||||
|
| Parsed from a stock `Cliloc.enu` | **123,490** entries |
|
||||||
|
| Of those, empty strings | **55,994** (ids the client reserves and never uses) |
|
||||||
|
| Stored in `shard_clilocs` | **67,496** |
|
||||||
|
|
||||||
|
Blank entries are dropped at import. A row resolving to no name is
|
||||||
|
indistinguishable from no row at all to every caller, and dropping them makes the
|
||||||
|
binary and text imports converge on **identical** content — the binary format
|
||||||
|
carries the blanks explicitly and a text export may or may not, depending on the
|
||||||
|
tool. Verified: both formats import to the same 67,496 rows with the same keys.
|
||||||
|
|
||||||
|
`text` is `TEXT`, not `VARCHAR`: the long property descriptions reach 12 KB, and
|
||||||
|
silently truncating them would be worse than storing them. The index that matters
|
||||||
|
for marketplace search is on the denormalized `shard_vendor_items.display_name`,
|
||||||
|
not here.
|
||||||
|
|
||||||
|
## How names are applied
|
||||||
|
|
||||||
|
**Resolution happens server-side.** The table is never served *as* a table and
|
||||||
|
there is no public route for it. Two reasons: 67k rows would dwarf any page that
|
||||||
|
used them, and the Android client consumes the same JSON and would otherwise need
|
||||||
|
its own copy.
|
||||||
|
|
||||||
|
`resolveMany()` takes a batch of ids and returns a `Map` holding only those that
|
||||||
|
resolved to something displayable, so "no such id" and "id with no usable name"
|
||||||
|
collapse into one branch at the call site. It never throws — a cliloc lookup is
|
||||||
|
decoration on someone's character sheet, and a database blip must not fail the
|
||||||
|
sheet. A capped in-process cache fronts it; measured cold **4.2 ms**, warm
|
||||||
|
**0.015 ms**.
|
||||||
|
|
||||||
|
### `displayText()`
|
||||||
|
|
||||||
|
Cliloc strings interpolate arguments the client pulls from an item's property
|
||||||
|
list — `~1_val~`, `~2_NAME~`. **We never have those**: the bridge sends the id,
|
||||||
|
not the packet. So a name carrying them is reduced to what is actually knowable.
|
||||||
|
|
||||||
|
| Raw | Displayed |
|
||||||
|
|---|---|
|
||||||
|
| `quarter staff` | `quarter staff` |
|
||||||
|
| `cold damage ~1_val~%` | `cold damage` |
|
||||||
|
| `[~1_stuff~]` | *(nothing — the whole string was the argument)* |
|
||||||
|
| `50%` | `50%` |
|
||||||
|
|
||||||
|
The trailing `%` in row two is dropped **because a placeholder was removed** — it
|
||||||
|
is the unit belonging to the number we never had. Row four shows why that
|
||||||
|
condition matters: stripping `%` unconditionally would corrupt a string that
|
||||||
|
legitimately ends in one.
|
||||||
|
|
||||||
|
### Consumers
|
||||||
|
|
||||||
|
- **Character sheet equipment.** `enrichCharProfile` attaches `clilocName` to each
|
||||||
|
item. A player-given `name` always wins — "Bob's lucky axe" must not be
|
||||||
|
relabelled "hatchet" — and the client re-states that precedence.
|
||||||
|
- **Reward titles.** `titles.rewardResolved` is a parallel array with the numeric
|
||||||
|
entries turned into words (`null` where nothing resolved). The sheet used to
|
||||||
|
*skip* numeric reward titles entirely, having no way to render them.
|
||||||
|
- **Marketplace listings** (Protocol 3.0 §8) denormalize the resolved name into
|
||||||
|
`shard_vendor_items.display_name` so search can index it.
|
||||||
|
|
||||||
|
## Admin surface
|
||||||
|
|
||||||
|
All admin-only, alongside the atlas under Admin → Shard:
|
||||||
|
|
||||||
|
| Route | Purpose |
|
||||||
|
|---|---|
|
||||||
|
| `GET /api/v1/admin/shard/clilocs` | Path, resolved file, readability, drift, entry count |
|
||||||
|
| `POST /api/v1/admin/shard/clilocs/import` | Reload after a client patch; `{ "force": true }` reimports an unchanged file |
|
||||||
|
| `PUT /api/v1/admin/shard/clilocs/path` | Set the path; blank disables resolution |
|
||||||
|
|
||||||
|
A refresh **result is not an exception**: a missing file, or the likely mistake of
|
||||||
|
pointing at the client's own compressed `Cliloc.enu`, answers `200` with
|
||||||
|
`status: "unavailable"` and a reason. A `500` would say only "something broke";
|
||||||
|
the operator needs to be told which file to convert. Setting the path
|
||||||
|
deliberately does **not** import as a side effect — the response carries the
|
||||||
|
refreshed status so the panel can offer that as the next step.
|
||||||
Reference in New Issue
Block a user