docs/website/CLILOCS.md

Cliloc table (item and title names)

Status: Complete. The base table now arrives over the bridge — protocol 8, phase 2. Design: docs/link/v3.md §8.6 (the table itself, Protocol 3.0) and docs/link/v8.md §9 (the Asset Bridge, which retired the manual conversion).

A "cliloc" is UO's localization table: an integer id mapped to a display string. Items on the wire carry a LabelNumber, not a name. The bridge has always sent that number — char.profile.equipment has a cliloc field, reward titles arrive as a cliloc number in string form, and every marketplace listing carries one — but the site had no table to look it up in, so a character sheet could only render id 1023721 where the game renders "quarter staff".

The number was never the missing piece. The table was.

Where the table comes from

The shard reads its own client. A ServUO server cannot boot without a UO client — Config/DataPath.cfg resolves into Core.DataDirectories at run time — so the file this table is made of is already sitting on the shard host. Since protocol 8 the plugin decompresses it and serves it over the bridge, and the site imports it like any other shard read:

ServUO shard  ──`cliloc.table`──▶  uo-link sidecar  ──`GET /cliloc`──▶  website
   reads Cliloc.enu,                 forwards, keeps               merges overlays,
   decompresses, pages                  nothing                    replaces the table

Why that was worth building. Every current UO client ships its cliloc files compressed: all eight Cliloc.* files in a modern client (chs, cht, deu, enu, esp, fra, jpn, kor) begin with a DWORD whose high byte is 0x8E — the "Mythic" container. The plain layout is what those files looked like before that change, and ServUO's own bundled Ultima.StringList cannot read the new one either, which is why VendorSearch.GetItemName is inert on a modern shard and the in-game Vendor Search gump has the same gap.

So until protocol 8 an operator had to install UOFiddler, build a converter against its Ultima.dll, run it over their client and copy a 5 MB file to the web host — every time they patched. The decompressor now lives in the overlay (overlay/Scripts/Custom/Bridge/BridgeCliloc.cs, ported from UOFiddler, which is Beerware and therefore clean to bring into a GPL tree), so none of that is a step any more.

Two things are unchanged and remain the point:

• Nothing client-derived is committed. UO's strings are EA's. The repo ships no string table for the same reason it ships no artwork and no map snapshot — see SPAWN_ATLAS.md. The extraction happens on the operator's own host, from their own files, for their own shard.
• A shard with no table is fully supported. Names render as ids, exactly as they did before the table existed.

What arrives, and in how many pieces

The sidecar's reply timeout is 10 s and its inbound line cap is 1 MiB, so the table is paged: the shard cuts at a 512 KiB byte budget and hands back a cursor, and the site walks it until a page says more: false. Measured on a stock English client: 67,496 rows in about eleven pages, decoded on the shard in 290 ms.

The rows are { n, f, t } — number, flag, text. Blanks never leave the shard: roughly 56,000 of a stock table's 123,490 entries are empty strings the client reserves and never uses, the site drops them at import anyway, and sending them would double the transfer for data that is discarded on arrival.

Only cut: "end" means the table finished. A short page can equally mean the byte budget was spent, and importing a table that stopped early is the one failure that is invisible downstream — some items named, some not, which is exactly what no table looks like.

The file pipeline is deprecated, not removed

An install with no uo-link configured can still be pointed at a converted file and works exactly as it did. That path exists for shards with no bridge and for development without a running ServUO, it accepts the same two formats it always did (plain binary, or number<TAB|,|;>text delimited text), and passing an explicit path to a refresh still selects it as a one-off. Nothing new should be built on it.

Shard-added and shard-edited items

Shards edit items and add new ones, and those carry cliloc ids no stock client table has. The table is therefore built from a set of sources, all re-read on every boot and hash-gated together — the same shape as the spawn atlas, which reads Regions.xml + Locations/*.xml + Spawns/*.xml + ChampionSpawns.xml and merges them:

<cliloc path>/
  custom/
    01-uomysticmoon.tsv    ← overlays: shard additions and overrides
    02-events.tsv

The base is no longer a file in that directory — it comes from the shard — so on a bridged install the configured path selects only where custom/ is read from. (Without a shard link, a converted base file sitting beside custom/ is still found, which is the deprecated pipeline above.)

Overlays use the same delimited-text format, are read in sorted order, and later sources win — so an overlay both adds ids the client never had and overrides stock ones the shard has re-purposed. Any .tsv, .csv, .txt, .enu or .plain file in custom/ is picked up; anything else (a README.md, say) is ignored.

Adding, editing or removing any overlay counts as drift, so a new custom item needs only a file edit and the admin panel's Import button. Adding one item never means re-reading the client table — though on the bridge that is now cheap enough not to matter much.

The import result reports what each source contributed, which is how you confirm an overlay took effect — overrode: 0 on a file meant to re-label stock items says it did not:

"sources": [
  { "label": "cliloc.enu",              "kind": "shard",  "entries": 67496, "added": 67496, "overrode": 0 },
  { "label": "custom/uomysticmoon.tsv", "kind": "custom", "entries": 2,     "added": 1,     "overrode": 1 }
]

kind says where a source came from: shard over the bridge, base a converted file on disk, custom an overlay.

Why a convention rather than discovery. Everywhere else this pipeline follows the shard's own files, but ServUO has no server-side notion of a custom cliloc — they live in the patched client a shard distributes to its players, and nothing in the tree declares them. There is nothing to discover, so custom/ is the one thing here that is our convention rather than the shard's. (An operator who does patch their client cliloc needs no overlay at all: convert the patched file and their edits are simply in the base.)

Measured on the live shard for scale: its script tree references 16,434 cliloc ids and only 37 are absent from the stock client table — tens of entries against a 67k base, which is what makes an overlay the right shape rather than a second full table.

Choosing the source

Nothing to configure: the shard wins whenever uo-link is configured and enabled. There is no mode setting, because there is no version of this question an operator benefits from answering — a shard that can serve its own client table is strictly better than a file somebody converted by hand months ago.

The two escape hatches, both deliberate:

No uo-link configured	The file pipeline, exactly as before
An explicit path passed to a refresh	A one-off "import from this file", which the shard never overrules

cliloc_client_path (setting, admin-editable) and UO_CLIENT_PATH (env deploy-time default) still name a path, and the setting still wins over the environment. What that path means narrowed: on a bridged install it is where custom/ overlays live. Without a link it is also searched for a base file, case-insensitively (the client writes Cliloc.enu on Windows; the site usually runs on Linux), in order: clilocs.tsv, clilocs.csv, clilocs.plain, cliloc.plain, cliloc.plain.enu, cliloc.enu.plain, clilocs.txt, cliloc.enu — and pointing it straight at an unconverted client directory still answers by name rather than failing obscurely:

status: unavailable
code:   COMPRESSED
reason: This is a compressed (Mythic-format) cliloc file, which the site cannot
        read. Convert it to the plain format first — see docs/website/CLILOCS.md.

Refresh contract

• It never blocks startup, and on the bridge it never touches startup: boot imports nothing when the shard is the source. A sidecar round trip in the boot sequence would be spent answering "no" on every restart but the one after a client patch — and patching a client is an operator action, so importing is an operator action. Admin → Client Files → Update is the button (the cliloc section of it; the page covers everything that comes off the UO client, because a client patch changes all of it at once). Whatever table is loaded keeps serving until then.
• Still hash-gated, so pressing Import when nothing changed costs one small call. The gate is the shard's own assets.sources: size, mtime and content hash of Cliloc.enu plus the shard's EXTRACTOR_VERSION. A sha256 of null with hashing: true means the shard has not computed it yet (it hashes off the request path, because the art and animation files it also reports are 343 MB) — that means ask again, never changed, and the comparison falls back to (size, mtime) meanwhile.
• Without a link the file path is unchanged: boot hashes the local file and skips the parse when it matches. Measured on a stock table: 14 ms for the no-op, 663 ms for a full parse and replace.
• A PARSER_VERSION bump counts as drift, and so does an EXTRACTOR_VERSION bump on the shard — same argument at the other end of the wire: a corrected reader must reach an install whose client never patches.

Two ways a refresh is refused

A corrupt file — the realistic failure for any single source — makes the parser fail on a truncated record rather than yield a plausible-but-short table, so it is caught outright. Verified: a file truncated to half its length reports

code:   TRUNCATED
reason: Truncated record header at byte 2486759 (74909 entries read)

and the rows already loaded are untouched. A malformed overlay names the file it came from (custom/broken.tsv: No cliloc entries found…), because "which of my six overlay files is broken" is otherwise a guessing game.

An OVERLAY that has VANISHED is the hazard a single file did not have. It parses perfectly and imports a table quietly missing everything that file contributed — and an unmounted volume looks exactly like a deliberate deletion from here. This is the same ambiguity the atlas stages a facet removal for, so it is escalated rather than applied:

status:         needsReview
reason:         1 previously-loaded cliloc overlay(s) are missing;
                the existing table is unchanged
missingSources: ["custom/uomysticmoon.tsv"]

The base is deliberately exempt from that question, and that is an upgrade detail worth stating: an install that used the converted-file pipeline carries its base file's label in the stored fingerprint, and on the bridge that label is supposed to disappear. Counting it as a vanished source would make the first import after the upgrade demand an approval for a change the upgrade itself made.

A client patched mid-import is refused outright rather than staged, because there is nothing to decide: every page echoes the source file's size and mtime, and if they move between pages then half of what arrived came from a file that no longer exists and nothing later can tell which half.

status: unavailable
code:   SOURCE_CHANGED
reason: The shard's cliloc file changed while it was being read; nothing was imported

status() reports missingSources too, so the panel can show it before anyone clicks Import. An admin accepts it by re-running the import with { "approve": true }.

Why that is a flag and not the atlas's approve/reject pair. The atlas stores a pending decision in its own table so that approving re-parses the tree, which is what keeps a multi-megabyte blob out of the database and makes the applied result match the tree at approval time. Here nothing is stored, so re-reading at approval time is automatic — the decision is a single boolean on the import an admin was already going to run.

What gets stored

Entries in a stock Cliloc.enu	123,490
Of those, empty strings	55,994 (ids the client reserves and never uses)
Stored in shard_clilocs	67,496

Blank entries are dropped, and since protocol 8 they are dropped on the shard, before they reach the wire. A row resolving to no name is indistinguishable from no row at all to every caller, so sending 56,000 of them would double the transfer for data discarded on arrival. The site still filters at import, because an overlay file can carry one and because the file pipeline still exists.

That the shard's own decoder lands on exactly 67,496 is also the strongest check there is that the ported decompressor is correct: the number was measured first through UOFiddler's Ultima.dll against this same client, by an entirely different implementation.

text is TEXT, not VARCHAR: the long property descriptions reach 12 KB, and silently truncating them would be worse than storing them. The index that matters for marketplace search is on the denormalized shard_vendor_items.display_name, not here.

How names are applied

Resolution happens server-side. The table is never served as a table and there is no public route for it. Two reasons: 67k rows would dwarf any page that used them, and the Android client consumes the same JSON and would otherwise need its own copy.

resolveMany() takes a batch of ids and returns a Map holding only those that resolved to something displayable, so "no such id" and "id with no usable name" collapse into one branch at the call site. It never throws — a cliloc lookup is decoration on someone's character sheet, and a database blip must not fail the sheet. A capped in-process cache fronts it; measured cold 4.2 ms, warm 0.015 ms.

displayText()

Cliloc strings interpolate arguments the client pulls from an item's property list — ~1_val~, ~2_NAME~. We never have those: the bridge sends the id, not the packet. So a name carrying them is reduced to what is actually knowable.

Raw	Displayed
quarter staff	quarter staff
cold damage ~1_val~%	cold damage
[~1_stuff~]	(nothing — the whole string was the argument)
50%	50%
Runic Gateway Sigil (v2)	Runic Gateway Sigil (v2)

Punctuation is only tidied when a placeholder was actually removed. The trailing % in row two is the unit belonging to the number we never had, and the brackets in row three only ever wrapped the argument — but a string with no placeholder has no such debris, and trimming it anyway corrupts real names. Rows four and five are the ones that caught it: a shard's custom "Runic Gateway Sigil (v2)" rendered as "(v2" while the bracket trim was unconditional.

Consumers

• Character sheet equipment. enrichCharProfile attaches clilocName to each item. A player-given name always wins — "Bob's lucky axe" must not be relabelled "hatchet" — and the client re-states that precedence.
• Reward titles. titles.rewardResolved is a parallel array with the numeric entries turned into words (null where nothing resolved). The sheet used to skip numeric reward titles entirely, having no way to render them.
• Marketplace listings (Protocol 3.0 §8) denormalize the resolved name into shard_vendor_items.display_name so search can index it.

Admin surface

All admin-only, and driven from Admin → Client Files — one page over the cliloc table, the creature catalogue and item art, since all three are read from the same client install (docs/link/v8.md §12.2):

Route	Purpose
GET /api/v1/admin/shard/clilocs	Which source is in use (bridge / file), the shard's client-file fingerprint, what each source contributed at the last import, drift, entry count, missingSources
POST /api/v1/admin/shard/clilocs/import	Reload after a client patch or an overlay edit; { "force": true } reimports an unchanged set, { "approve": true } accepts a vanished overlay
PUT /api/v1/admin/shard/clilocs/path	Set the overlay path (and, with no shard link, the base file's); blank clears it

Import is now the only thing that refreshes the table on a bridged install, since boot no longer asks the shard. It is the button an operator presses after patching their client.

A refresh result is not an exception, and protocol 8 widened the set of things that covers: a shard that is down, an asset plane the operator has switched off (Bridge.AssetsEnabled), a client with no cliloc file, a client patched halfway through the import, plus everything the file pipeline could already report. Each answers 200 with status: "unavailable" and a reason naming what to fix. A 500 would say only "something broke". Setting the path deliberately does not import as a side effect — the response carries the refreshed status so the panel can offer that as the next step.

The sidecar's own statuses are worth knowing when reading a log: 425 is the shard saying it is busy with another asset request (flow control, and the ordinary answer mid-import — the site retries), 403 the asset plane switched off, 404 a client with no such file, 422 a file it has and cannot decode.