docs/website/CLILOCS.md

# Cliloc table (item and title names)

**Status:** Complete. The base table now arrives **over the bridge** — protocol 8, phase 2.
**Design:** [`docs/link/v3.md` §8.6](../link/v3.md) (the table itself, Protocol 3.0) and
[`docs/link/v8.md` §9](../link/v8.md) (the Asset Bridge, which retired the manual conversion).

A "cliloc" is UO's localization table: an integer id mapped to a display string.
**Items on the wire carry a `LabelNumber`, not a name.** The bridge has always
sent that number — `char.profile.equipment` has a `cliloc` field, reward titles
arrive as a cliloc number in string form, and every marketplace listing carries
one — but the site had no table to look it up in, so a character sheet could only
render `id 1023721` where the game renders **"quarter staff"**.

The number was never the missing piece. The table was.

## Where the table comes from

**The shard reads its own client.** A ServUO server cannot boot without a UO
client — `Config/DataPath.cfg` resolves into `Core.DataDirectories` at run time —
so the file this table is made of is already sitting on the shard host. Since
protocol 8 the plugin decompresses it and serves it over the bridge, and the site
imports it like any other shard read:

```
ServUO shard  ──`cliloc.table`──▶  uo-link sidecar  ──`GET /cliloc`──▶  website
   reads Cliloc.enu,                 forwards, keeps               merges overlays,
   decompresses, pages                  nothing                    replaces the table
```

**Why that was worth building.** Every current UO client ships its cliloc files
compressed: all eight `Cliloc.*` files in a modern client (`chs`, `cht`, `deu`,
`enu`, `esp`, `fra`, `jpn`, `kor`) begin with a DWORD whose high byte is `0x8E` —
the "Mythic" container. The plain layout is what those files looked like *before*
that change, and **ServUO's own bundled `Ultima.StringList` cannot read the new
one either**, which is why `VendorSearch.GetItemName` is inert on a modern shard
and the in-game Vendor Search gump has the same gap.

So until protocol 8 an operator had to install UOFiddler, build a converter
against its `Ultima.dll`, run it over their client and copy a 5 MB file to the web
host — every time they patched. The decompressor now lives in the overlay
(`overlay/Scripts/Custom/Bridge/BridgeCliloc.cs`, ported from UOFiddler, which is
Beerware and therefore clean to bring into a GPL tree), so **none of that is a
step any more**.

Two things are unchanged and remain the point:

- **Nothing client-derived is committed.** UO's strings are EA's. The repo ships
  no string table for the same reason it ships no artwork and no map snapshot —
  see [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md). The extraction happens on the
  operator's own host, from their own files, for their own shard.
- **A shard with no table is fully supported.** Names render as ids, exactly as
  they did before the table existed.

### What arrives, and in how many pieces

The sidecar's reply timeout is 10 s and its inbound line cap is 1 MiB, so the
table is **paged**: the shard cuts at a 512 KiB byte budget and hands back a
cursor, and the site walks it until a page says `more: false`. Measured on a stock
English client: **67,496 rows in about eleven pages**, decoded on the shard in
**290 ms**.

The rows are `{ n, f, t }` — number, flag, text. **Blanks never leave the shard**:
roughly 56,000 of a stock table's 123,490 entries are empty strings the client
reserves and never uses, the site drops them at import anyway, and sending them
would double the transfer for data that is discarded on arrival.

Only `cut: "end"` means the table finished. A short page can equally mean the byte
budget was spent, and importing a table that stopped early is the one failure that
is invisible downstream — some items named, some not, which is exactly what *no
table* looks like.

### The file pipeline is deprecated, not removed

An install with **no uo-link configured** can still be pointed at a converted file
and works exactly as it did. That path exists for shards with no bridge and for
development without a running ServUO, it accepts the same two formats it always
did (plain binary, or `number<TAB|,|;>text` delimited text), and passing an
explicit path to a refresh still selects it as a one-off. Nothing new should be
built on it.

## Shard-added and shard-edited items

**Shards edit items and add new ones**, and those carry cliloc ids no stock
client table has. The table is therefore built from a **set** of sources, all
re-read on every boot and hash-gated together — the same shape as the spawn
atlas, which reads `Regions.xml` + `Locations/*.xml` + `Spawns/*.xml` +
`ChampionSpawns.xml` and merges them:

```
<cliloc path>/
  custom/
    01-uomysticmoon.tsv    ← overlays: shard additions and overrides
    02-events.tsv
```

The base is no longer a file in that directory — it comes from the shard — so on
a bridged install the configured path selects **only** where `custom/` is read
from. (Without a shard link, a converted base file sitting beside `custom/` is
still found, which is the deprecated pipeline above.)

Overlays use the same delimited-text format, are read in **sorted order**, and
**later sources win** — so an overlay both *adds* ids the client never had and
*overrides* stock ones the shard has re-purposed. Any `.tsv`, `.csv`, `.txt`,
`.enu` or `.plain` file in `custom/` is picked up; anything else (a `README.md`,
say) is ignored.

Adding, editing or removing any overlay counts as drift, so a new custom item
needs only a file edit and the admin panel's Import button. **Adding one item
never means re-reading the client table** — though on the bridge that is now
cheap enough not to matter much.

The import result reports what each source contributed, which is how you confirm
an overlay took effect — `overrode: 0` on a file meant to re-label stock items
says it did not:

```json
"sources": [
  { "label": "cliloc.enu",              "kind": "shard",  "entries": 67496, "added": 67496, "overrode": 0 },
  { "label": "custom/uomysticmoon.tsv", "kind": "custom", "entries": 2,     "added": 1,     "overrode": 1 }
]
```

`kind` says where a source came from: `shard` over the bridge, `base` a converted
file on disk, `custom` an overlay.

**Why a convention rather than discovery.** Everywhere else this pipeline follows
the shard's own files, but **ServUO has no server-side notion of a custom
cliloc** — they live in the patched client a shard distributes to its players,
and nothing in the tree declares them. There is nothing to discover, so `custom/`
is the one thing here that is our convention rather than the shard's. (An
operator who *does* patch their client cliloc needs no overlay at all: convert
the patched file and their edits are simply in the base.)

Measured on the live shard for scale: its script tree references **16,434** cliloc
ids and only **37** are absent from the stock client table — tens of entries
against a 67k base, which is what makes an overlay the right shape rather than a
second full table.

## Choosing the source

Nothing to configure: **the shard wins whenever uo-link is configured and
enabled.** There is no mode setting, because there is no version of this question
an operator benefits from answering — a shard that can serve its own client table
is strictly better than a file somebody converted by hand months ago.

The two escape hatches, both deliberate:

| | |
|---|---|
| No uo-link configured | The file pipeline, exactly as before |
| An explicit `path` passed to a refresh | A one-off "import from this file", which the shard never overrules |

`cliloc_client_path` (setting, admin-editable) and `UO_CLIENT_PATH` (env
deploy-time default) still name a path, and the setting still wins over the
environment. What that path *means* narrowed: on a bridged install it is where
`custom/` overlays live. Without a link it is also searched for a base file,
case-insensitively (the client writes `Cliloc.enu` on Windows; the site usually
runs on Linux), in order: `clilocs.tsv`, `clilocs.csv`, `clilocs.plain`,
`cliloc.plain`, `cliloc.plain.enu`, `cliloc.enu.plain`, `clilocs.txt`,
`cliloc.enu` — and pointing it straight at an unconverted client directory still
answers by name rather than failing obscurely:

```
status: unavailable
code:   COMPRESSED
reason: This is a compressed (Mythic-format) cliloc file, which the site cannot
        read. Convert it to the plain format first — see docs/website/CLILOCS.md.
```

## Refresh contract

- **It never blocks startup**, and on the bridge it never *touches* startup: boot
  imports nothing when the shard is the source. A sidecar round trip in the boot
  sequence would be spent answering "no" on every restart but the one after a
  client patch — and patching a client is an operator action, so importing is an
  operator action. **Admin → Shard → Import** is the button. Whatever table is
  loaded keeps serving until then.
- **Still hash-gated**, so pressing Import when nothing changed costs one small
  call. The gate is the shard's own `assets.sources`: size, mtime and content
  hash of `Cliloc.enu` plus the shard's `EXTRACTOR_VERSION`. A `sha256` of `null`
  with `hashing: true` means the shard has not computed it yet (it hashes off the
  request path, because the art and animation files it also reports are 343 MB)
  — that means *ask again*, never *changed*, and the comparison falls back to
  (size, mtime) meanwhile.
- **Without a link** the file path is unchanged: boot hashes the local file and
  skips the parse when it matches. Measured on a stock table: **14 ms** for the
  no-op, **663 ms** for a full parse and replace.
- **A `PARSER_VERSION` bump counts as drift**, and so does an `EXTRACTOR_VERSION`
  bump on the shard — same argument at the other end of the wire: a corrected
  reader must reach an install whose client never patches.

### Two ways a refresh is refused

**A corrupt file** — the realistic failure for any single source — makes the
parser fail on a truncated record rather than yield a plausible-but-short table,
so it is caught outright. Verified: a file truncated to half its length reports

```
code:   TRUNCATED
reason: Truncated record header at byte 2486759 (74909 entries read)
```

and the rows already loaded are untouched. A malformed overlay names the file it
came from (`custom/broken.tsv: No cliloc entries found…`), because "which of my
six overlay files is broken" is otherwise a guessing game.

**An OVERLAY that has VANISHED** is the hazard a single file did not have. It
parses perfectly and imports a table quietly missing everything that file
contributed — and an unmounted volume looks exactly like a deliberate deletion
from here. This is the same ambiguity the atlas stages a facet removal for, so it
is escalated rather than applied:

```
status:         needsReview
reason:         1 previously-loaded cliloc overlay(s) are missing;
                the existing table is unchanged
missingSources: ["custom/uomysticmoon.tsv"]
```

**The base is deliberately exempt from that question**, and that is an upgrade
detail worth stating: an install that used the converted-file pipeline carries its
base file's label in the stored fingerprint, and on the bridge that label is
*supposed* to disappear. Counting it as a vanished source would make the first
import after the upgrade demand an approval for a change the upgrade itself made.

**A client patched mid-import** is refused outright rather than staged, because
there is nothing to decide: every page echoes the source file's size and mtime, and
if they move between pages then half of what arrived came from a file that no
longer exists and nothing later can tell which half.

```
status: unavailable
code:   SOURCE_CHANGED
reason: The shard's cliloc file changed while it was being read; nothing was imported
```

`status()` reports `missingSources` too, so the panel can show it before anyone
clicks Import. An admin accepts it by re-running the import with
`{ "approve": true }`.

**Why that is a flag and not the atlas's approve/reject pair.** The atlas stores
a pending decision in its own table so that approving *re-parses the tree*, which
is what keeps a multi-megabyte blob out of the database and makes the applied
result match the tree at approval time. Here nothing is stored, so re-reading at
approval time is automatic — the decision is a single boolean on the import an
admin was already going to run.

## What gets stored

| | |
|---|---|
| Entries in a stock `Cliloc.enu` | **123,490** |
| Of those, empty strings | **55,994** (ids the client reserves and never uses) |
| Stored in `shard_clilocs` | **67,496** |

Blank entries are dropped, and since protocol 8 they are dropped **on the shard**,
before they reach the wire. A row resolving to no name is indistinguishable from
no row at all to every caller, so sending 56,000 of them would double the
transfer for data discarded on arrival. The site still filters at import, because
an overlay file can carry one and because the file pipeline still exists.

That the shard's own decoder lands on **exactly 67,496** is also the strongest
check there is that the ported decompressor is correct: the number was measured
first through UOFiddler's `Ultima.dll` against this same client, by an entirely
different implementation.

`text` is `TEXT`, not `VARCHAR`: the long property descriptions reach 12 KB, and
silently truncating them would be worse than storing them. The index that matters
for marketplace search is on the denormalized `shard_vendor_items.display_name`,
not here.

## How names are applied

**Resolution happens server-side.** The table is never served *as* a table and
there is no public route for it. Two reasons: 67k rows would dwarf any page that
used them, and the Android client consumes the same JSON and would otherwise need
its own copy.

`resolveMany()` takes a batch of ids and returns a `Map` holding only those that
resolved to something displayable, so "no such id" and "id with no usable name"
collapse into one branch at the call site. It never throws — a cliloc lookup is
decoration on someone's character sheet, and a database blip must not fail the
sheet. A capped in-process cache fronts it; measured cold **4.2 ms**, warm
**0.015 ms**.

### `displayText()`

Cliloc strings interpolate arguments the client pulls from an item's property
list — `~1_val~`, `~2_NAME~`. **We never have those**: the bridge sends the id,
not the packet. So a name carrying them is reduced to what is actually knowable.

| Raw | Displayed |
|---|---|
| `quarter staff` | `quarter staff` |
| `cold damage ~1_val~%` | `cold damage` |
| `[~1_stuff~]` | *(nothing — the whole string was the argument)* |
| `50%` | `50%` |
| `Runic Gateway Sigil (v2)` | `Runic Gateway Sigil (v2)` |

**Punctuation is only tidied when a placeholder was actually removed.** The
trailing `%` in row two is the unit belonging to the number we never had, and the
brackets in row three only ever wrapped the argument — but a string with no
placeholder has no such debris, and trimming it anyway corrupts real names. Rows
four and five are the ones that caught it: a shard's custom
`"Runic Gateway Sigil (v2)"` rendered as `"(v2"` while the bracket trim was
unconditional.

### Consumers

- **Character sheet equipment.** `enrichCharProfile` attaches `clilocName` to each
  item. A player-given `name` always wins — "Bob's lucky axe" must not be
  relabelled "hatchet" — and the client re-states that precedence.
- **Reward titles.** `titles.rewardResolved` is a parallel array with the numeric
  entries turned into words (`null` where nothing resolved). The sheet used to
  *skip* numeric reward titles entirely, having no way to render them.
- **Marketplace listings** (Protocol 3.0 §8) denormalize the resolved name into
  `shard_vendor_items.display_name` so search can index it.

## Admin surface

All admin-only, alongside the atlas under Admin → Shard:

| Route | Purpose |
|---|---|
| `GET /api/v1/admin/shard/clilocs` | Which source is in use (`bridge` / `file`), the shard's client-file fingerprint, what each source contributed at the last import, drift, entry count, `missingSources` |
| `POST /api/v1/admin/shard/clilocs/import` | Reload after a client patch or an overlay edit; `{ "force": true }` reimports an unchanged set, `{ "approve": true }` accepts a vanished overlay |
| `PUT /api/v1/admin/shard/clilocs/path` | Set the overlay path (and, with no shard link, the base file's); blank clears it |

**Import is now the only thing that refreshes the table on a bridged install**,
since boot no longer asks the shard. It is the button an operator presses after
patching their client.

A refresh **result is not an exception**, and protocol 8 widened the set of things
that covers: a shard that is down, an asset plane the operator has switched off
(`Bridge.AssetsEnabled`), a client with no cliloc file, a client patched halfway
through the import, plus everything the file pipeline could already report. Each
answers `200` with `status: "unavailable"` and a reason naming what to fix. A
`500` would say only "something broke". Setting the path deliberately does **not**
import as a side effect — the response carries the refreshed status so the panel
can offer that as the next step.

The sidecar's own statuses are worth knowing when reading a log: **425** is the
shard saying it is busy with another asset request (flow control, and the ordinary
answer mid-import — the site retries), **403** the asset plane switched off, **404**
a client with no such file, **422** a file it has and cannot decode.