RunicGateway/docs
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(link): clilocs come over the bridge now, and UOFiddler's first job is gone

Browse Source 
Phase 2 of the Asset Bridge is built, so the documentation stops telling an
operator to install a GUI tool.

`v8.md` gains §9.1 and §9.2 — what the port cost, what it measured, and where the
base table comes from now. The measurement worth keeping: **67,496 rows in
290 ms**, which is exactly what UOFiddler's own `Ultima.dll` produced from this
same client through the converter this phase deletes. An independent
implementation agreeing to the row is the strongest check available that a format
decoder is correct, and it is not something a subtly-wrong one produces.

§17 records the four shapes the org lead settled before any of it was written.
Two departed from the recommendation: **the bridge always wins** (no source
setting — there is no version of that question an operator benefits from
answering) and **import is admin-triggered** (boot does not call the shard at
all).

`CLILOCS.md` is rewritten around that: where the table comes from, what arrives
and in how many pieces, the refusals — including the two the file pipeline had no
equivalent of (a client patched mid-import, and the base's exemption from the
vanished-source rule, which exists so an upgraded install is not asked to approve
a change the upgrade itself made).

`UOFIDDLER.md` loses Part 1 entirely rather than having it rewritten. What is
left is creature art, which phase 5 takes, after which the page goes away. `v3.md`
§8.6 keeps its reasoning with a note saying what superseded it, because the
argument for why the manual step existed is still the argument for why this was
worth building.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
This commit is contained in:
claude
2026-09-10 11:13:56 -05:00
parent df9b3fd990
commit bbd69a8e2e
6 changed files with 312 additions and 336 deletions
Download Patch File Download Diff File Expand all files Collapse all files

290
website/CLILOCS.md
View File

@@ -1,7 +1,8 @@
# Cliloc table (item and title names)
**Status:** Complete on `edge` — website [#115](https://gitea.whitlocktech.com/RunicGateway/website/pulls/115), docs [#70](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/70).
**Design:** [`docs/link/v3.md` §8.6](../link/v3.md) — Protocol 3.0, the dependency Part B/3 was sequenced behind.
**Status:** Complete. The base table now arrives **over the bridge** — protocol 8, phase 2.
**Design:** [`docs/link/v3.md` §8.6](../link/v3.md) (the table itself, Protocol 3.0) and
[`docs/link/v8.md` §9](../link/v8.md) (the Asset Bridge, which retired the manual conversion).
A "cliloc" is UO's localization table: an integer id mapped to a display string.
**Items on the wire carry a `LabelNumber`, not a name.** The bridge has always
@@ -12,86 +13,70 @@ render `id 1023721` where the game renders **"quarter staff"**.
The number was never the missing piece. The table was.
## Why the operator has to convert the file
## Where the table comes from
This is the awkward part, and it is not avoidable:
**The shard reads its own client.** A ServUO server cannot boot without a UO
client — `Config/DataPath.cfg` resolves into `Core.DataDirectories` at run time —
so the file this table is made of is already sitting on the shard host. Since
protocol 8 the plugin decompresses it and serves it over the bridge, and the site
imports it like any other shard read:
**Every current UO client ships its cliloc files compressed.** All eight
`Cliloc.*` files in a modern client (`chs`, `cht`, `deu`, `enu`, `esp`, `fra`,
`jpn`, `kor`) begin with a DWORD whose high byte is `0x8E` — the "Mythic"
compressed container. The plain layout this site parses is what those files
looked like *before* that change.
Decompressing it means an inverse-BWT coder with a frequency header — a few
hundred lines of bit-level work whose failure mode is plausible-looking garbage
rather than an error. The site has no business carrying that at runtime.
Two facts make the alternatives worse, not better:
- **ServUO cannot read it either.** Its bundled `Ultima.StringList` implements
only the plain layout, so on a modern client `VendorSearch.StringList` is null
and `VendorSearch.GetItemName` returns `item.Name` — usually nothing. The
shard cannot supply names on our behalf; the in-game Vendor Search gump has the
same gap.
- **Nothing client-derived may be committed.** UO's strings are EA's. The repo
ships no string table for the same reason it ships no artwork and no map
snapshot — see [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md).
So the conversion happens **once, on the operator's machine, against their own
client**, and the site reads the result from a path it is given. A shard that
never does this is in a fully supported state: names render as ids, exactly as
they did before the table existed.
## Converting
> **Step-by-step operator instructions — where to get UOFiddler, where your
> client files are, and how to verify the import — are in
> [`UOFIDDLER.md`](UOFIDDLER.md).** This section covers the formats and the
> reasoning behind them.
Either format below is accepted; the site sniffs which one it was handed.
| Format | Fidelity | Notes |
|---|---|---|
| **Plain binary** (recommended) | Exact | 6-byte header, then `{int32 number, byte flag, uint16 length, UTF-8}` records |
| Delimited text | Loses leading/trailing whitespace | `number<TAB\|,\|;>text` per line; a header row, blank lines and `#` comments are ignored |
The whitespace caveat is real but cosmetic: ~1,300 of the 123,490 entries in a
stock `Cliloc.enu` are label prefixes like `"max = "` whose trailing space is
meaningful when the client concatenates a value onto them. Nothing on this site
concatenates, and every consumer passes through `displayText()`, which trims.
### Using the bundled tool
`server/tools/cliloc-export/` is a small .NET console app that drives
[UOFiddler](https://github.com/polserver/UOFiddler)'s `Ultima.dll` — the
decompressor that already exists and is already maintained — and writes the plain
format. It loads that DLL **reflectively** so it compiles against any SDK, and it
writes the records by hand because UOFiddler's own `SaveStringList` *re-compresses*
on save (its purpose is round-tripping a file back into the client, so its output
is byte-identical to its input — a trap worth knowing about).
```bash
cd website/server/tools/cliloc-export
dotnet build -c Release
# binary (recommended)
dotnet run -- "<UOFiddler>/Ultima.dll" "<UO client>/Cliloc.enu" /srv/uo-data/clilocs.plain
# or tab-delimited
dotnet run -- "<UOFiddler>/Ultima.dll" "<UO client>/Cliloc.enu" /srv/uo-data/clilocs.tsv --tsv
```
ServUO shard ──`cliloc.table`──▶ uo-link sidecar ──`GET /cliloc`──▶ website
reads Cliloc.enu, forwards, keeps merges overlays,
decompresses, pages nothing replaces the table
```
A UOFiddler GUI export works too, but **not unmodified**: its Cliloc tab writes
`Number;Text;Flag` — three columns, the flag *last* — and the parser reads
`number<separator>text`, so the trailing field is absorbed into the name and
every item renders as `quarter staff;0`. Stripping it is one `sed`, given in
[`UOFIDDLER.md`](UOFIDDLER.md) §Route B.
**Why that was worth building.** Every current UO client ships its cliloc files
compressed: all eight `Cliloc.*` files in a modern client (`chs`, `cht`, `deu`,
`enu`, `esp`, `fra`, `jpn`, `kor`) begin with a DWORD whose high byte is `0x8E` —
the "Mythic" container. The plain layout is what those files looked like *before*
that change, and **ServUO's own bundled `Ultima.StringList` cannot read the new
one either**, which is why `VendorSearch.GetItemName` is inert on a modern shard
and the in-game Vendor Search gump has the same gap.
The parser already tolerates `number,flag,text`, with the flag in the *middle*.
It is not extended to cover the trailing form because a final `;0` is
indistinguishable from a name that genuinely ends that way — a heuristic there
would corrupt real names to save the operator one command.
So until protocol 8 an operator had to install UOFiddler, build a converter
against its `Ultima.dll`, run it over their client and copy a 5 MB file to the web
host — every time they patched. The decompressor now lives in the overlay
(`overlay/Scripts/Custom/Bridge/BridgeCliloc.cs`, ported from UOFiddler, which is
Beerware and therefore clean to bring into a GPL tree), so **none of that is a
step any more**.
Two things are unchanged and remain the point:
- **Nothing client-derived is committed.** UO's strings are EA's. The repo ships
no string table for the same reason it ships no artwork and no map snapshot —
see [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md). The extraction happens on the
operator's own host, from their own files, for their own shard.
- **A shard with no table is fully supported.** Names render as ids, exactly as
they did before the table existed.
### What arrives, and in how many pieces
The sidecar's reply timeout is 10 s and its inbound line cap is 1 MiB, so the
table is **paged**: the shard cuts at a 512 KiB byte budget and hands back a
cursor, and the site walks it until a page says `more: false`. Measured on a stock
English client: **67,496 rows in about eleven pages**, decoded on the shard in
**290 ms**.
The rows are `{ n, f, t }` — number, flag, text. **Blanks never leave the shard**:
roughly 56,000 of a stock table's 123,490 entries are empty strings the client
reserves and never uses, the site drops them at import anyway, and sending them
would double the transfer for data that is discarded on arrival.
Only `cut: "end"` means the table finished. A short page can equally mean the byte
budget was spent, and importing a table that stopped early is the one failure that
is invisible downstream — some items named, some not, which is exactly what *no
table* looks like.
### The file pipeline is deprecated, not removed
An install with **no uo-link configured** can still be pointed at a converted file
and works exactly as it did. That path exists for shards with no bridge and for
development without a running ServUO, it accepts the same two formats it always
did (plain binary, or `number<TAB|,|;>text` delimited text), and passing an
explicit path to a refresh still selects it as a one-off. Nothing new should be
built on it.
## Shard-added and shard-edited items
@@ -103,12 +88,16 @@ atlas, which reads `Regions.xml` + `Locations/*.xml` + `Spawns/*.xml` +
```
<cliloc path>/
clilocs.plain ← base: the converted client table
custom/
01-uomysticmoon.tsv ← overlays: shard additions and overrides
02-events.tsv
```
The base is no longer a file in that directory — it comes from the shard — so on
a bridged install the configured path selects **only** where `custom/` is read
from. (Without a shard link, a converted base file sitting beside `custom/` is
still found, which is the deprecated pipeline above.)
Overlays use the same delimited-text format, are read in **sorted order**, and
**later sources win** — so an overlay both *adds* ids the client never had and
*overrides* stock ones the shard has re-purposed. Any `.tsv`, `.csv`, `.txt`,
@@ -116,8 +105,9 @@ Overlays use the same delimited-text format, are read in **sorted order**, and
say) is ignored.
Adding, editing or removing any overlay counts as drift, so a new custom item
needs only a file edit and a restart — or the admin panel's Import button.
**Adding one item never means re-exporting a 5 MB client file.**
needs only a file edit and the admin panel's Import button. **Adding one item
never means re-reading the client table** — though on the bridge that is now
cheap enough not to matter much.
The import result reports what each source contributed, which is how you confirm
an overlay took effect — `overrode: 0` on a file meant to re-label stock items
@@ -125,11 +115,14 @@ says it did not:
```json
"sources": [
{ "label": "clilocs.plain", "kind": "base", "entries": 123490, "added": 123490, "overrode": 0 },
{ "label": "custom/uomysticmoon.tsv", "kind": "custom", "entries": 2, "added": 1, "overrode": 1 }
{ "label": "cliloc.enu", "kind": "shard", "entries": 67496, "added": 67496, "overrode": 0 },
{ "label": "custom/uomysticmoon.tsv", "kind": "custom", "entries": 2, "added": 1, "overrode": 1 }
]
```
`kind` says where a source came from: `shard` over the bridge, `base` a converted
file on disk, `custom` an overlay.
**Why a convention rather than discovery.** Everywhere else this pipeline follows
the shard's own files, but **ServUO has no server-side notion of a custom
cliloc** — they live in the patched client a shard distributes to its players,
@@ -143,28 +136,29 @@ ids and only **37** are absent from the stock client table — tens of entries
against a 67k base, which is what makes an overlay the right shape rather than a
second full table.
## Configuring the path
## Choosing the source
Two ways to point at the sources, the setting winning over the environment:
Nothing to configure: **the shard wins whenever uo-link is configured and
enabled.** There is no mode setting, because there is no version of this question
an operator benefits from answering — a shard that can serve its own client table
is strictly better than a file somebody converted by hand months ago.
| Source | Notes |
The two escape hatches, both deliberate:
| | |
|---|---|
| `cliloc_client_path` setting | Admin-editable (Admin → Shard); takes effect on the next refresh without a redeploy |
| `UO_CLIENT_PATH` env var | The deploy-time default, since the path usually describes a mount the deployment sets up |
| No uo-link configured | The file pipeline, exactly as before |
| An explicit `path` passed to a refresh | A one-off "import from this file", which the shard never overrules |
The value may be **the base file itself or a directory to search**, because both
are natural answers to "where is it". Overlays are read from a `custom/`
directory beside the base **either way** — pointing at a file does not forfeit
them.
A directory is searched case-insensitively (the client writes `Cliloc.enu` on
Windows; the site usually runs on Linux) for, in order: `clilocs.tsv`,
`clilocs.csv`, `clilocs.plain`, `cliloc.plain`, `cliloc.plain.enu`,
`cliloc.enu.plain`, `clilocs.txt`, `cliloc.enu`.
That ordering puts explicitly-converted names first on purpose. Pointing the
setting straight at an unconverted client directory finds `cliloc.enu`, which is
compressed — and the site says so by name rather than failing obscurely:
`cliloc_client_path` (setting, admin-editable) and `UO_CLIENT_PATH` (env
deploy-time default) still name a path, and the setting still wins over the
environment. What that path *means* narrowed: on a bridged install it is where
`custom/` overlays live. Without a link it is also searched for a base file,
case-insensitively (the client writes `Cliloc.enu` on Windows; the site usually
runs on Linux), in order: `clilocs.tsv`, `clilocs.csv`, `clilocs.plain`,
`cliloc.plain`, `cliloc.plain.enu`, `cliloc.enu.plain`, `clilocs.txt`,
`cliloc.enu` — and pointing it straight at an unconverted client directory still
answers by name rather than failing obscurely:
```
status: unavailable
@@ -175,16 +169,25 @@ reason: This is a compressed (Mythic-format) cliloc file, which the site cannot
## Refresh contract
Identical in shape to the spawn atlas, and for the same reasons:
- **It never blocks startup.** No path, an unreadable file, a wrong-format file,
a database error — all caught and logged. The site comes up either way.
- **Hash-gated.** The boot path hashes the file and skips the parse entirely when
it matches what is loaded, which is every restart that did not follow a client
patch. Measured on a stock table: **14 ms** for the no-op, **663 ms** for a full
parse and replace.
- **A `PARSER_VERSION` bump also counts as drift**, so a corrected parse reaches
an install whose client never patches.
- **It never blocks startup**, and on the bridge it never *touches* startup: boot
imports nothing when the shard is the source. A sidecar round trip in the boot
sequence would be spent answering "no" on every restart but the one after a
client patch — and patching a client is an operator action, so importing is an
operator action. **Admin → Shard → Import** is the button. Whatever table is
loaded keeps serving until then.
- **Still hash-gated**, so pressing Import when nothing changed costs one small
call. The gate is the shard's own `assets.sources`: size, mtime and content
hash of `Cliloc.enu` plus the shard's `EXTRACTOR_VERSION`. A `sha256` of `null`
with `hashing: true` means the shard has not computed it yet (it hashes off the
request path, because the art and animation files it also reports are 343 MB)
— that means *ask again*, never *changed*, and the comparison falls back to
(size, mtime) meanwhile.
- **Without a link** the file path is unchanged: boot hashes the local file and
skips the parse when it matches. Measured on a stock table: **14 ms** for the
no-op, **663 ms** for a full parse and replace.
- **A `PARSER_VERSION` bump counts as drift**, and so does an `EXTRACTOR_VERSION`
bump on the shard — same argument at the other end of the wire: a corrected
reader must reach an install whose client never patches.
### Two ways a refresh is refused
@@ -201,7 +204,7 @@ and the rows already loaded are untouched. A malformed overlay names the file it
came from (`custom/broken.tsv: No cliloc entries found…`), because "which of my
six overlay files is broken" is otherwise a guessing game.
**A source that has VANISHED** is the hazard a single file did not have. It
**An OVERLAY that has VANISHED** is the hazard a single file did not have. It
parses perfectly and imports a table quietly missing everything that file
contributed — and an unmounted volume looks exactly like a deliberate deletion
from here. This is the same ambiguity the atlas stages a facet removal for, so it
@@ -209,11 +212,28 @@ is escalated rather than applied:
```
status: needsReview
reason: 1 previously-loaded cliloc source(s) are missing;
reason: 1 previously-loaded cliloc overlay(s) are missing;
the existing table is unchanged
missingSources: ["custom/uomysticmoon.tsv"]
```
**The base is deliberately exempt from that question**, and that is an upgrade
detail worth stating: an install that used the converted-file pipeline carries its
base file's label in the stored fingerprint, and on the bridge that label is
*supposed* to disappear. Counting it as a vanished source would make the first
import after the upgrade demand an approval for a change the upgrade itself made.
**A client patched mid-import** is refused outright rather than staged, because
there is nothing to decide: every page echoes the source file's size and mtime, and
if they move between pages then half of what arrived came from a file that no
longer exists and nothing later can tell which half.
```
status: unavailable
code: SOURCE_CHANGED
reason: The shard's cliloc file changed while it was being read; nothing was imported
```
`status()` reports `missingSources` too, so the panel can show it before anyone
clicks Import. An admin accepts it by re-running the import with
`{ "approve": true }`.
@@ -229,15 +249,20 @@ admin was already going to run.
| | |
|---|---|
| Parsed from a stock `Cliloc.enu` | **123,490** entries |
| Entries in a stock `Cliloc.enu` | **123,490** |
| Of those, empty strings | **55,994** (ids the client reserves and never uses) |
| Stored in `shard_clilocs` | **67,496** |
Blank entries are dropped at import. A row resolving to no name is
indistinguishable from no row at all to every caller, and dropping them makes the
binary and text imports converge on **identical** content — the binary format
carries the blanks explicitly and a text export may or may not, depending on the
tool. Verified: both formats import to the same 67,496 rows with the same keys.
Blank entries are dropped, and since protocol 8 they are dropped **on the shard**,
before they reach the wire. A row resolving to no name is indistinguishable from
no row at all to every caller, so sending 56,000 of them would double the
transfer for data discarded on arrival. The site still filters at import, because
an overlay file can carry one and because the file pipeline still exists.
That the shard's own decoder lands on **exactly 67,496** is also the strongest
check there is that the ported decompressor is correct: the number was measured
first through UOFiddler's `Ultima.dll` against this same client, by an entirely
different implementation.
`text` is `TEXT`, not `VARCHAR`: the long property descriptions reach 12 KB, and
silently truncating them would be worse than storing them. The index that matters
@@ -297,13 +322,24 @@ All admin-only, alongside the atlas under Admin → Shard:
| Route | Purpose |
|---|---|
| `GET /api/v1/admin/shard/clilocs` | Sources found, what each contributed at the last import, readability, drift, entry count, `missingSources` |
| `POST /api/v1/admin/shard/clilocs/import` | Reload after a client patch or an overlay edit; `{ "force": true }` reimports an unchanged set, `{ "approve": true }` accepts a vanished source |
| `PUT /api/v1/admin/shard/clilocs/path` | Set the path; blank disables resolution |
| `GET /api/v1/admin/shard/clilocs` | Which source is in use (`bridge` / `file`), the shard's client-file fingerprint, what each source contributed at the last import, drift, entry count, `missingSources` |
| `POST /api/v1/admin/shard/clilocs/import` | Reload after a client patch or an overlay edit; `{ "force": true }` reimports an unchanged set, `{ "approve": true }` accepts a vanished overlay |
| `PUT /api/v1/admin/shard/clilocs/path` | Set the overlay path (and, with no shard link, the base file's); blank clears it |
A refresh **result is not an exception**: a missing file, or the likely mistake of
pointing at the client's own compressed `Cliloc.enu`, answers `200` with
`status: "unavailable"` and a reason. A `500` would say only "something broke";
the operator needs to be told which file to convert. Setting the path
deliberately does **not** import as a side effect — the response carries the
refreshed status so the panel can offer that as the next step.
**Import is now the only thing that refreshes the table on a bridged install**,
since boot no longer asks the shard. It is the button an operator presses after
patching their client.
A refresh **result is not an exception**, and protocol 8 widened the set of things
that covers: a shard that is down, an asset plane the operator has switched off
(`Bridge.AssetsEnabled`), a client with no cliloc file, a client patched halfway
through the import, plus everything the file pipeline could already report. Each
answers `200` with `status: "unavailable"` and a reason naming what to fix. A
`500` would say only "something broke". Setting the path deliberately does **not**
import as a side effect — the response carries the refreshed status so the panel
can offer that as the next step.
The sidecar's own statuses are worth knowing when reading a log: **425** is the
shard saying it is busy with another asset request (flow control, and the ordinary
answer mid-import — the site retries), **403** the asset plane switched off, **404**
a client with no such file, **422** a file it has and cannot decode.