# Cliloc table (item and title names) **Status:** Complete on `edge` — website [#115](https://gitea.whitlocktech.com/RunicGateway/website/pulls/115), docs [#70](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/70). **Design:** [`docs/link/v3.md` §8.6](../link/v3.md) — Protocol 3.0, the dependency Part B/3 was sequenced behind. A "cliloc" is UO's localization table: an integer id mapped to a display string. **Items on the wire carry a `LabelNumber`, not a name.** The bridge has always sent that number — `char.profile.equipment` has a `cliloc` field, reward titles arrive as a cliloc number in string form, and every marketplace listing carries one — but the site had no table to look it up in, so a character sheet could only render `id 1023721` where the game renders **"quarter staff"**. The number was never the missing piece. The table was. ## Why the operator has to convert the file This is the awkward part, and it is not avoidable: **Every current UO client ships its cliloc files compressed.** All eight `Cliloc.*` files in a modern client (`chs`, `cht`, `deu`, `enu`, `esp`, `fra`, `jpn`, `kor`) begin with a DWORD whose high byte is `0x8E` — the "Mythic" compressed container. The plain layout this site parses is what those files looked like *before* that change. Decompressing it means an inverse-BWT coder with a frequency header — a few hundred lines of bit-level work whose failure mode is plausible-looking garbage rather than an error. The site has no business carrying that at runtime. Two facts make the alternatives worse, not better: - **ServUO cannot read it either.** Its bundled `Ultima.StringList` implements only the plain layout, so on a modern client `VendorSearch.StringList` is null and `VendorSearch.GetItemName` returns `item.Name` — usually nothing. The shard cannot supply names on our behalf; the in-game Vendor Search gump has the same gap. - **Nothing client-derived may be committed.** UO's strings are EA's. The repo ships no string table for the same reason it ships no artwork and no map snapshot — see [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md). So the conversion happens **once, on the operator's machine, against their own client**, and the site reads the result from a path it is given. A shard that never does this is in a fully supported state: names render as ids, exactly as they did before the table existed. ## Converting > **Step-by-step operator instructions — where to get UOFiddler, where your > client files are, and how to verify the import — are in > [`UOFIDDLER.md`](UOFIDDLER.md).** This section covers the formats and the > reasoning behind them. Either format below is accepted; the site sniffs which one it was handed. | Format | Fidelity | Notes | |---|---|---| | **Plain binary** (recommended) | Exact | 6-byte header, then `{int32 number, byte flag, uint16 length, UTF-8}` records | | Delimited text | Loses leading/trailing whitespace | `numbertext` per line; a header row, blank lines and `#` comments are ignored | The whitespace caveat is real but cosmetic: ~1,300 of the 123,490 entries in a stock `Cliloc.enu` are label prefixes like `"max = "` whose trailing space is meaningful when the client concatenates a value onto them. Nothing on this site concatenates, and every consumer passes through `displayText()`, which trims. ### Using the bundled tool `server/tools/cliloc-export/` is a small .NET console app that drives [UOFiddler](https://github.com/polserver/UOFiddler)'s `Ultima.dll` — the decompressor that already exists and is already maintained — and writes the plain format. It loads that DLL **reflectively** so it compiles against any SDK, and it writes the records by hand because UOFiddler's own `SaveStringList` *re-compresses* on save (its purpose is round-tripping a file back into the client, so its output is byte-identical to its input — a trap worth knowing about). ```bash cd website/server/tools/cliloc-export dotnet build -c Release # binary (recommended) dotnet run -- "/Ultima.dll" "/Cliloc.enu" /srv/uo-data/clilocs.plain # or tab-delimited dotnet run -- "/Ultima.dll" "/Cliloc.enu" /srv/uo-data/clilocs.tsv --tsv ``` A UOFiddler GUI export works too, but **not unmodified**: its Cliloc tab writes `Number;Text;Flag` — three columns, the flag *last* — and the parser reads `numbertext`, so the trailing field is absorbed into the name and every item renders as `quarter staff;0`. Stripping it is one `sed`, given in [`UOFIDDLER.md`](UOFIDDLER.md) §Route B. The parser already tolerates `number,flag,text`, with the flag in the *middle*. It is not extended to cover the trailing form because a final `;0` is indistinguishable from a name that genuinely ends that way — a heuristic there would corrupt real names to save the operator one command. ## Shard-added and shard-edited items **Shards edit items and add new ones**, and those carry cliloc ids no stock client table has. The table is therefore built from a **set** of sources, all re-read on every boot and hash-gated together — the same shape as the spawn atlas, which reads `Regions.xml` + `Locations/*.xml` + `Spawns/*.xml` + `ChampionSpawns.xml` and merges them: ``` / clilocs.plain ← base: the converted client table custom/ 01-uomysticmoon.tsv ← overlays: shard additions and overrides 02-events.tsv ``` Overlays use the same delimited-text format, are read in **sorted order**, and **later sources win** — so an overlay both *adds* ids the client never had and *overrides* stock ones the shard has re-purposed. Any `.tsv`, `.csv`, `.txt`, `.enu` or `.plain` file in `custom/` is picked up; anything else (a `README.md`, say) is ignored. Adding, editing or removing any overlay counts as drift, so a new custom item needs only a file edit and a restart — or the admin panel's Import button. **Adding one item never means re-exporting a 5 MB client file.** The import result reports what each source contributed, which is how you confirm an overlay took effect — `overrode: 0` on a file meant to re-label stock items says it did not: ```json "sources": [ { "label": "clilocs.plain", "kind": "base", "entries": 123490, "added": 123490, "overrode": 0 }, { "label": "custom/uomysticmoon.tsv", "kind": "custom", "entries": 2, "added": 1, "overrode": 1 } ] ``` **Why a convention rather than discovery.** Everywhere else this pipeline follows the shard's own files, but **ServUO has no server-side notion of a custom cliloc** — they live in the patched client a shard distributes to its players, and nothing in the tree declares them. There is nothing to discover, so `custom/` is the one thing here that is our convention rather than the shard's. (An operator who *does* patch their client cliloc needs no overlay at all: convert the patched file and their edits are simply in the base.) Measured on the live shard for scale: its script tree references **16,434** cliloc ids and only **37** are absent from the stock client table — tens of entries against a 67k base, which is what makes an overlay the right shape rather than a second full table. ## Configuring the path Two ways to point at the sources, the setting winning over the environment: | Source | Notes | |---|---| | `cliloc_client_path` setting | Admin-editable (Admin → Shard); takes effect on the next refresh without a redeploy | | `UO_CLIENT_PATH` env var | The deploy-time default, since the path usually describes a mount the deployment sets up | The value may be **the base file itself or a directory to search**, because both are natural answers to "where is it". Overlays are read from a `custom/` directory beside the base **either way** — pointing at a file does not forfeit them. A directory is searched case-insensitively (the client writes `Cliloc.enu` on Windows; the site usually runs on Linux) for, in order: `clilocs.tsv`, `clilocs.csv`, `clilocs.plain`, `cliloc.plain`, `cliloc.plain.enu`, `cliloc.enu.plain`, `clilocs.txt`, `cliloc.enu`. That ordering puts explicitly-converted names first on purpose. Pointing the setting straight at an unconverted client directory finds `cliloc.enu`, which is compressed — and the site says so by name rather than failing obscurely: ``` status: unavailable code: COMPRESSED reason: This is a compressed (Mythic-format) cliloc file, which the site cannot read. Convert it to the plain format first — see docs/website/CLILOCS.md. ``` ## Refresh contract Identical in shape to the spawn atlas, and for the same reasons: - **It never blocks startup.** No path, an unreadable file, a wrong-format file, a database error — all caught and logged. The site comes up either way. - **Hash-gated.** The boot path hashes the file and skips the parse entirely when it matches what is loaded, which is every restart that did not follow a client patch. Measured on a stock table: **14 ms** for the no-op, **663 ms** for a full parse and replace. - **A `PARSER_VERSION` bump also counts as drift**, so a corrected parse reaches an install whose client never patches. ### Two ways a refresh is refused **A corrupt file** — the realistic failure for any single source — makes the parser fail on a truncated record rather than yield a plausible-but-short table, so it is caught outright. Verified: a file truncated to half its length reports ``` code: TRUNCATED reason: Truncated record header at byte 2486759 (74909 entries read) ``` and the rows already loaded are untouched. A malformed overlay names the file it came from (`custom/broken.tsv: No cliloc entries found…`), because "which of my six overlay files is broken" is otherwise a guessing game. **A source that has VANISHED** is the hazard a single file did not have. It parses perfectly and imports a table quietly missing everything that file contributed — and an unmounted volume looks exactly like a deliberate deletion from here. This is the same ambiguity the atlas stages a facet removal for, so it is escalated rather than applied: ``` status: needsReview reason: 1 previously-loaded cliloc source(s) are missing; the existing table is unchanged missingSources: ["custom/uomysticmoon.tsv"] ``` `status()` reports `missingSources` too, so the panel can show it before anyone clicks Import. An admin accepts it by re-running the import with `{ "approve": true }`. **Why that is a flag and not the atlas's approve/reject pair.** The atlas stores a pending decision in its own table so that approving *re-parses the tree*, which is what keeps a multi-megabyte blob out of the database and makes the applied result match the tree at approval time. Here nothing is stored, so re-reading at approval time is automatic — the decision is a single boolean on the import an admin was already going to run. ## What gets stored | | | |---|---| | Parsed from a stock `Cliloc.enu` | **123,490** entries | | Of those, empty strings | **55,994** (ids the client reserves and never uses) | | Stored in `shard_clilocs` | **67,496** | Blank entries are dropped at import. A row resolving to no name is indistinguishable from no row at all to every caller, and dropping them makes the binary and text imports converge on **identical** content — the binary format carries the blanks explicitly and a text export may or may not, depending on the tool. Verified: both formats import to the same 67,496 rows with the same keys. `text` is `TEXT`, not `VARCHAR`: the long property descriptions reach 12 KB, and silently truncating them would be worse than storing them. The index that matters for marketplace search is on the denormalized `shard_vendor_items.display_name`, not here. ## How names are applied **Resolution happens server-side.** The table is never served *as* a table and there is no public route for it. Two reasons: 67k rows would dwarf any page that used them, and the Android client consumes the same JSON and would otherwise need its own copy. `resolveMany()` takes a batch of ids and returns a `Map` holding only those that resolved to something displayable, so "no such id" and "id with no usable name" collapse into one branch at the call site. It never throws — a cliloc lookup is decoration on someone's character sheet, and a database blip must not fail the sheet. A capped in-process cache fronts it; measured cold **4.2 ms**, warm **0.015 ms**. ### `displayText()` Cliloc strings interpolate arguments the client pulls from an item's property list — `~1_val~`, `~2_NAME~`. **We never have those**: the bridge sends the id, not the packet. So a name carrying them is reduced to what is actually knowable. | Raw | Displayed | |---|---| | `quarter staff` | `quarter staff` | | `cold damage ~1_val~%` | `cold damage` | | `[~1_stuff~]` | *(nothing — the whole string was the argument)* | | `50%` | `50%` | | `Runic Gateway Sigil (v2)` | `Runic Gateway Sigil (v2)` | **Punctuation is only tidied when a placeholder was actually removed.** The trailing `%` in row two is the unit belonging to the number we never had, and the brackets in row three only ever wrapped the argument — but a string with no placeholder has no such debris, and trimming it anyway corrupts real names. Rows four and five are the ones that caught it: a shard's custom `"Runic Gateway Sigil (v2)"` rendered as `"(v2"` while the bracket trim was unconditional. ### Consumers - **Character sheet equipment.** `enrichCharProfile` attaches `clilocName` to each item. A player-given `name` always wins — "Bob's lucky axe" must not be relabelled "hatchet" — and the client re-states that precedence. - **Reward titles.** `titles.rewardResolved` is a parallel array with the numeric entries turned into words (`null` where nothing resolved). The sheet used to *skip* numeric reward titles entirely, having no way to render them. - **Marketplace listings** (Protocol 3.0 §8) denormalize the resolved name into `shard_vendor_items.display_name` so search can index it. ## Admin surface All admin-only, alongside the atlas under Admin → Shard: | Route | Purpose | |---|---| | `GET /api/v1/admin/shard/clilocs` | Sources found, what each contributed at the last import, readability, drift, entry count, `missingSources` | | `POST /api/v1/admin/shard/clilocs/import` | Reload after a client patch or an overlay edit; `{ "force": true }` reimports an unchanged set, `{ "approve": true }` accepts a vanished source | | `PUT /api/v1/admin/shard/clilocs/path` | Set the path; blank disables resolution | A refresh **result is not an exception**: a missing file, or the likely mistake of pointing at the client's own compressed `Cliloc.enu`, answers `200` with `status: "unavailable"` and a reason. A `500` would say only "something broke"; the operator needs to be told which file to convert. Setting the path deliberately does **not** import as a side effect — the response carries the refreshed status so the panel can offer that as the next step.