From be9f5019fa0985edd448ed341dc1effa36753994 Mon Sep 17 00:00:00 2001 From: wtclaude Date: Wed, 29 Jul 2026 04:22:05 -0500 Subject: [PATCH 1/3] =?UTF-8?q?docs(link):=20the=20cliloc=20table,=20and?= =?UTF-8?q?=20why=20=C2=A78.6's=20recommendation=20was=20not=20implementab?= =?UTF-8?q?le?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Protocol 3.0 §8.6 resolved as its own website-only change, landing ahead of §8 so the marketplace ships with real item names. Matching documentation for website #TBD. NEW website/CLILOCS.md — operator-facing: why the conversion step exists, how to convert, how to configure the path, the refresh contract, what gets stored and how names are applied. link/v3.md §8.6 rewritten. Two things in the original recommendation turned out to be wrong, and both are recorded because the reasoning generalises: 1. The committed db/data/clilocs.json artifact predates the Part C corrections (§6.1) and violates both — no committed snapshot of derived content, and nothing EA-derived ever shipped. UO's strings are EA's, exactly as the creature sprites are. 2. "scripts/buildClilocs.js reads the UO client's Cliloc.enu" is not possible. EVERY current client ships its cliloc files compressed (first DWORD's high byte 0x8E, the Mythic container); the plain layout is what those files looked like before that change, and parsing one as the other does not fail cleanly — it yields ~19k records with negative ids, 1,722 distinct keys out of 19,508, one 62 KB "string", and a truncation somewhere in the middle. ServUO's own Ultima.StringList cannot read it either, so VendorSearch.GetItemName is already inert on such a shard and the work could not be pushed to the plugin. That second point also retires an open question in §8.2: the warning never to call GetItemName in the market sweep costs us nothing we could otherwise have had, because the in-game Vendor Search gump has the same gap. Three traps found by building it are recorded: StringList.SaveStringList RE-COMPRESSES on save (its output is byte-identical to its compressed input, because its purpose is round-tripping a file back into the client); trimming a text line before splitting silently drops the ~half of a table that is empty strings; and Number('') is 0, not NaN. Also updated: - Progress and §9 sequencing tables: order 5 split into 5a (this, website only) and 5b (the four-repo wire change). - website/BACKEND_DESIGN.md — shard_clilocs / shard_cliloc_meta, the three admin routes, and why there is no staged-approval flow and no public route. - link/INTEGRATION.md — the char.profile field note now says explicitly not to expect the shard to resolve clilocs, and points at CLILOCS.md. - §10 documentation obligations list CLILOCS.md. Documentation only. Every claim was written after the corresponding behaviour was observed running: the compressed-format finding and the parse failures come from the real client files on this machine, and the counts (123,490 parsed → 67,496 stored) and timings from importing them into the live MariaDB. PROJECT_TREE.md files are deliberately untouched — they are CI-generated by the sync-project-tree workflow and say so in their header. Co-Authored-By: Claude --- link/INTEGRATION.md | 2 +- link/v3.md | 73 +++++++++++-- website/BACKEND_DESIGN.md | 48 +++++++++ website/CLILOCS.md | 216 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 327 insertions(+), 12 deletions(-) create mode 100644 website/CLILOCS.md diff --git a/link/INTEGRATION.md b/link/INTEGRATION.md index 697235f..1b5ba20 100644 --- a/link/INTEGRATION.md +++ b/link/INTEGRATION.md @@ -488,7 +488,7 @@ Full character sheet: stats, all trained skills, worn equipment with flattened i Field notes: - `skills[].base` is trained value, `value` includes item/temp bonuses, `cap` is the cap. **Do not assume `base <= cap`** — GM characters can exceed it. - `equipment[].mods` is a flattened map of every non-zero AOS attribute on the item (weapon or armor). Empty `{}` for plain items. -- Item names are usually **clilocs**, not strings: use `name` when present, otherwise resolve `cliloc` against a UO cliloc table on the site. +- Item names are usually **clilocs**, not strings: use `name` when present, otherwise resolve `cliloc` against a UO cliloc table on the site. **Do not expect the shard to resolve them for you** — on any modern client ServUO's own `Ultima.StringList` cannot read the client's compressed cliloc files, so `VendorSearch.GetItemName` returns `item.Name` and the in-game Vendor Search gump has the same gap. Building that table is a consumer-side job; the website's is described in [`website/CLILOCS.md`](../website/CLILOCS.md). - `titles` (Protocol 2.0): `selected` is the index into `reward` currently displayed (`-1` if none). `fameKarma`/`skill` are computed display titles, omitted when the character has none. `reward` entries may be a **cliloc number as a string** or a literal string — resolve numeric ones against your cliloc table, same as item names. - Errors: unknown account → **404** `{"kind":"bridge.error","reason":"unknown account"}`; bad slot → **404**/**400** similarly. diff --git a/link/v3.md b/link/v3.md index 5b6862d..3ef0cc5 100644 --- a/link/v3.md +++ b/link/v3.md @@ -15,9 +15,14 @@ Each part is marked off here as it lands on `edge`. §9 carries the same state p | 2 | **B/1** — `world.ruleset` (§5) | ✅ **Done** | servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66) | | 3 | **C** — spawn atlas (§6) | ✅ **Done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables) + [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113) (API + pages + admin panel), docs [#67](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/67) + [#68](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/68) | | 4 | **B/2** — `points.board` (§7) | ✅ **Done** | servuo-plugins [#4](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/4), link [#18](https://gitea.whitlocktech.com/RunicGateway/link/pulls/18), website [#114](https://gitea.whitlocktech.com/RunicGateway/website/pulls/114), docs [#69](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/69) | -| 5 | **B/3** — `vendor.listing` (§8) | ⬜ Not started | — | +| 5a | **B/3 dependency** — cliloc table (§8.6) | 🟨 In review | website [#TBD](https://gitea.whitlocktech.com/RunicGateway/website/pulls), docs [#TBD](https://gitea.whitlocktech.com/RunicGateway/docs/pulls) | +| 5b | **B/3** — `vendor.listing` (§8) | ⬜ Not started | — | | 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | ⬜ Not started | — | +Order 5 split in two once §8.6's cliloc dependency turned out to be a client-format problem rather +than a parser (see §8.6). 5a is website-only and lands first so the marketplace ships with real item +names; 5b is the four-repo wire change. + --- ## 1. Why 3.0 @@ -715,17 +720,62 @@ admin can turn the stream on. `uoLinkSocket` paginates `/market` on reconnect, b `/market/vendors/:serial`, behind `requireFeature('market')`. **Rate-limit it** — this is the first genuinely expensive public endpoint; `express-rate-limit` is already a dependency. -### 8.6 The open dependency — cliloc names +### 8.6 The open dependency — cliloc names ✅ Resolved (shipped ahead of §8) -`CharacterSheet.jsx:14-15` already documents the gap ("without a cliloc table on the site we can only -show literals") and renders equipment as `id {itemId}`. Search-by-name needs that table. +`CharacterSheet.jsx:14-15` documented the gap ("without a cliloc table on the site we can only +show literals") and rendered equipment as `id {itemId}`. Search-by-name needs that table. -- **Recommended:** `scripts/buildClilocs.js` reads the UO client's `Cliloc.enu` → committed - `db/data/clilocs.json`; ingest denormalizes into `shard_vendor_items.display_name`. Same - build-artifact pattern as §6, and it **also fixes the character sheet**. -- **Fallback:** ship with item-art + price + region filters, and name search only over renamed items. +**Resolved as its own website-only change, landed BEFORE the market so `/site/market` ships with real +item names.** Full design and operator guide: [`docs/website/CLILOCS.md`](../website/CLILOCS.md). +Ingest denormalizes the resolved name into `shard_vendor_items.display_name` as planned. -This decision is the reason §8 is sequenced last. +Two things in the original recommendation above turned out to be wrong, and both are worth recording +because the reasoning generalises. + +**1. The committed `db/data/clilocs.json` artifact was dropped.** It predates the two Part C +corrections (§6.1) and violates both: no committed snapshot of derived content, and nothing +EA-derived ever shipped. UO's strings are EA's, exactly as the creature sprites are. Replaced with +the §6 pattern instead — parse on every boot from an operator-configured path, hash-gated, output +gitignored, `PARSER_VERSION` counted as drift. + +**2. `scripts/buildClilocs.js reads the UO client's Cliloc.enu` is not possible, and the reason +matters.** **Every current client ships its cliloc files COMPRESSED** — all four `Cliloc.*` files +open with a DWORD whose high byte is `0x8E`, the "Mythic" container. The plain layout (`02 00 00 00 +01 00`, then `{int32 number, byte flag, uint16 length, UTF-8}`) is what those files looked like +*before* that change. Parsing a compressed file as plain does not fail cleanly: it yields ~19k +"records" with negative ids, 1,722 distinct keys out of 19,508, one 62 KB "string", and a truncation +somewhere in the middle. + +Decompressing means porting an inverse-BWT coder with a 1 KB frequency header — a few hundred lines +whose failure mode is plausible-looking garbage rather than an error. Two facts closed off the +alternatives: + +- **ServUO cannot read it either.** Its bundled `Ultima.StringList` implements only the plain layout, + so on a modern client `VendorSearch.StringList` is null and `VendorSearch.GetItemName` returns + `item.Name`. **The in-game Vendor Search gump has the same gap** — which also means §8.2's warning + never to call `GetItemName` in the sweep costs us nothing we could otherwise have had. +- The shard therefore cannot supply names on our behalf, so this could not be pushed to the plugin. + +⇒ **the operator converts once, from their own client, and the site reads the result.** Accepted +shapes are the plain binary layout and a `numbertext` export; the site sniffs which. +`server/tools/cliloc-export/` drives UOFiddler's `Ultima.dll` (the decompressor that already exists) +and writes the plain form. A shard that never converts is fully supported — names render as ids, +exactly as before. + +Three traps found by building it, all recorded in `CLILOCS.md`: + +- **`StringList.SaveStringList` RE-COMPRESSES on save.** It looks exactly like the export path and is + not; its output is byte-identical to its compressed input, because its purpose is round-tripping a + file back into the client. +- **Trimming a text line before splitting silently drops half the table.** Roughly half of a real + cliloc table is empty strings (ids the client reserves), exported as `1005008`. Trimming eats + the trailing separator, leaving a bare number that then looks like a header row — 55,994 of 123,490 + entries vanished, and the import still looked successful. +- **`Number('')` is `0`, not `NaN`.** A line starting with a separator imports as a bogus cliloc 0 + unless the empty field is rejected explicitly. + +Blank entries are dropped at import (123,490 parsed → **67,496** stored), which also makes the binary +and text paths converge on identical content. ### 8.7 Client @@ -743,7 +793,8 @@ inherently up to one full cycle old, and the UI must say so. | 2 | **B/1** — `world.ruleset` (§5) | all four | new kind | ✅ Done | | 3 | **C** — spawn atlas (§6) | website, docs | none | ✅ Done | | 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | ✅ Done | -| 5 | **B/3** — `vendor.listing` (§8) | all four | new kinds | ⬜ | +| 5a | **B/3 dependency** — cliloc table (§8.6) | website, docs | none | 🟨 In review | +| 5b | **B/3** — `vendor.listing` (§8) | all four | new kinds | ⬜ | | 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | ⬜ | --- @@ -761,7 +812,7 @@ inherently up to one full cycle old, and the UI must say so. in the security section. - NEW `website/SHARD_VISIBILITY.md` — admin-facing: what each feature exposes, what each rung means, what cannot be loosened. -- NEW `website/SPAWN_ATLAS.md`, NEW `website/MARKETPLACE.md`. +- NEW `website/SPAWN_ATLAS.md`, NEW `website/CLILOCS.md`, NEW `website/MARKETPLACE.md`. - `PROJECT_TREE.md` in each touched repo. - `npm run swagger` **and** `npm run routes:manifest` on every route-touching PR — both are committed artifacts, and `test/routeManifest.test.js` fails on drift. diff --git a/website/BACKEND_DESIGN.md b/website/BACKEND_DESIGN.md index 4304050..ba13de7 100644 --- a/website/BACKEND_DESIGN.md +++ b/website/BACKEND_DESIGN.md @@ -486,6 +486,51 @@ artwork: sprites live in the operator's own client `.mul`/`.uop` files and are t redistribute. An operator supplies art via a gitignored map plus images under the (already gitignored) `server/uploads/atlas/`. Text-only is the normal, supported state. +### shard_clilocs / shard_cliloc_meta — UO's localization table (Protocol 3.0) + +Items on the wire carry a `LabelNumber`, not a name. The bridge has always sent it — +`char.profile.equipment.cliloc`, reward titles as a cliloc number in string form, and one per +marketplace listing — but with no table to resolve it against, the character sheet could only render +`id 1023721` where the game renders "quarter staff". + +| Table | Shape | +|---|---| +| `shard_clilocs` | `number` INT PK, `flag`, `text` TEXT | +| `shard_cliloc_meta` | Singleton (`id = 1`), `payload` JSON (source file, sha256, count, `parserVersion`), `imported_at` | + +Import-owned and all-or-nothing in one transaction, same contract as the atlas — including **`DELETE`, +not `TRUNCATE`**, for the same reason. + +**Sourced from a file the operator converts once from their own UO client**, at a path from the +`cliloc_client_path` setting falling back to `UO_CLIENT_PATH`. Nothing client-derived is committed: +UO's strings are EA's, exactly as the creature sprites are. A shard with nothing configured is fully +supported — names render as ids. Full design and operator guide: [`CLILOCS.md`](CLILOCS.md). + +The conversion step is not avoidable: **every current client ships its cliloc files compressed** +(first DWORD's high byte `0x8E`), and ServUO's own bundled `Ultima.StringList` cannot read that +either — so the shard cannot supply names on our behalf. The plain layout and a delimited text export +are both accepted, sniffed by header rather than extension. + +Three decisions worth stating: + +- **`text` is TEXT, not VARCHAR.** Long property descriptions reach 12 KB. The index that matters for + marketplace search is the denormalized `shard_vendor_items.display_name`, not this table. +- **Blank entries are dropped at import** — 123,490 parsed → **67,496** stored. Roughly half a cliloc + table is empty strings for ids the client reserves and never uses; a row that resolves to no name is + indistinguishable from no row at all, and dropping them makes the binary and text imports converge + on identical content. +- **No staged-approval flow, unlike the atlas.** The atlas escalates facet loss because a half-copied + tree and a real map change are indistinguishable from inside the process. A cliloc file is one file + with one hash, and a partial copy makes the parser fail on a truncated record — the ambiguity the + atlas must escalate is one this parser simply detects, so it refuses the import and leaves the + previous table serving. + +**Resolution is server-side and there is no public route.** The table is never served *as* a table: +67k rows would dwarf any page using them, and the Android client consumes the same already-resolved +JSON. `resolveMany()` returns only ids that resolved to something displayable — placeholders like +`~1_val~` are stripped, since the bridge sends the id and never the property packet that carries the +arguments — and it never throws, because a cliloc lookup is decoration on a character sheet. + --- ## 4. API contract @@ -741,6 +786,9 @@ file a route sits in — that is the property the route manifest freezes. | POST | `/shard/atlas/import` | re-import without restarting; `{force}` ignores the hash gate. **An unreadable tree answers 200 with `status:"unavailable"`, not 500** — `refresh()` reports outcomes rather than throwing (the boot path must never be blocked by a bad tree) and that contract is preserved at the API. | | POST | `/shard/atlas/approve` · `/shard/atlas/reject` | answer a refresh staged because it would REMOVE a facet. Approving **re-parses** the tree, so what lands matches it at approval time; rejecting is remembered against those source hashes so it does not re-prompt every restart. 404 when nothing is staged. | | PUT | `/shard/atlas/path` | point the atlas at a different tree (persisted as `spawn_atlas_servuo_path`, which wins over `SERVUO_PATH`). Blank clears it. Deliberately **does not import** — moving the mount and reloading the world are separate decisions — and returns fresh status so the panel can offer the import next. | +| GET | `/shard/clilocs` | cliloc-table status (`adminOnly`): the configured path, the file actually resolved (the path may be a directory), readability, drift against what is loaded, and the entry count. `configured:false` is a supported state — item names then render as ids. No public counterpart: the table is never served *as* a table. | +| POST | `/shard/clilocs/import` | reload after a client patch; `{force}` ignores the hash gate. **A missing file — or the likely mistake of pointing at the client's own COMPRESSED `Cliloc.enu` — answers 200 with `status:"unavailable"` and a `code`, not 500.** `COMPRESSED` is called out by name: a 500 would say only "something broke", and the operator needs to be told which file to convert. | +| PUT | `/shard/clilocs/path` | point the site at a different cliloc file or directory (persisted as `cliloc_client_path`, which wins over `UO_CLIENT_PATH`). Blank clears it. Deliberately **does not import**, same reasoning as the atlas path. | Every admin write logs to `activity_log`. diff --git a/website/CLILOCS.md b/website/CLILOCS.md new file mode 100644 index 0000000..a7b5b49 --- /dev/null +++ b/website/CLILOCS.md @@ -0,0 +1,216 @@ +# Cliloc table (item and title names) + +**Status:** Complete on `edge` — website [#TBD](https://gitea.whitlocktech.com/RunicGateway/website/pulls). +**Design:** [`docs/link/v3.md` §8.6](../link/v3.md) — Protocol 3.0, the dependency Part B/3 was sequenced behind. + +A "cliloc" is UO's localization table: an integer id mapped to a display string. +**Items on the wire carry a `LabelNumber`, not a name.** The bridge has always +sent that number — `char.profile.equipment` has a `cliloc` field, reward titles +arrive as a cliloc number in string form, and every marketplace listing carries +one — but the site had no table to look it up in, so a character sheet could only +render `id 1023721` where the game renders **"quarter staff"**. + +The number was never the missing piece. The table was. + +## Why the operator has to convert the file + +This is the awkward part, and it is not avoidable: + +**Every current UO client ships its cliloc files compressed.** The four +`Cliloc.*` files in a modern client all begin with a DWORD whose high byte is +`0x8E` — the "Mythic" compressed container. The plain layout this site parses is +what those files looked like *before* that change. + +Decompressing it means an inverse-BWT coder with a frequency header — a few +hundred lines of bit-level work whose failure mode is plausible-looking garbage +rather than an error. The site has no business carrying that at runtime. + +Two facts make the alternatives worse, not better: + +- **ServUO cannot read it either.** Its bundled `Ultima.StringList` implements + only the plain layout, so on a modern client `VendorSearch.StringList` is null + and `VendorSearch.GetItemName` returns `item.Name` — usually nothing. The + shard cannot supply names on our behalf; the in-game Vendor Search gump has the + same gap. +- **Nothing client-derived may be committed.** UO's strings are EA's. The repo + ships no string table for the same reason it ships no artwork and no map + snapshot — see [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md). + +So the conversion happens **once, on the operator's machine, against their own +client**, and the site reads the result from a path it is given. A shard that +never does this is in a fully supported state: names render as ids, exactly as +they did before the table existed. + +## Converting + +Either format below is accepted; the site sniffs which one it was handed. + +| Format | Fidelity | Notes | +|---|---|---| +| **Plain binary** (recommended) | Exact | 6-byte header, then `{int32 number, byte flag, uint16 length, UTF-8}` records | +| Delimited text | Loses leading/trailing whitespace | `numbertext` per line; a header row, blank lines and `#` comments are ignored | + +The whitespace caveat is real but cosmetic: ~1,300 of the 123,490 entries in a +stock `Cliloc.enu` are label prefixes like `"max = "` whose trailing space is +meaningful when the client concatenates a value onto them. Nothing on this site +concatenates, and every consumer passes through `displayText()`, which trims. + +### Using the bundled tool + +`server/tools/cliloc-export/` is a small .NET console app that drives +[UOFiddler](https://github.com/polserver/UOFiddler)'s `Ultima.dll` — the +decompressor that already exists and is already maintained — and writes the plain +format. It loads that DLL **reflectively** so it compiles against any SDK, and it +writes the records by hand because UOFiddler's own `SaveStringList` *re-compresses* +on save (its purpose is round-tripping a file back into the client, so its output +is byte-identical to its input — a trap worth knowing about). + +```bash +cd website/server/tools/cliloc-export +dotnet build -c Release + +# binary (recommended) +dotnet run -- "/Ultima.dll" "/Cliloc.enu" /srv/uo-data/clilocs.plain + +# or tab-delimited +dotnet run -- "/Ultima.dll" "/Cliloc.enu" /srv/uo-data/clilocs.tsv --tsv +``` + +A UOFiddler GUI export works equally well — anything producing one of the two +shapes above is fine. + +## Configuring the path + +Two ways to point at the converted file, the setting winning over the +environment: + +| Source | Notes | +|---|---| +| `cliloc_client_path` setting | Admin-editable (Admin → Shard); takes effect on the next refresh without a redeploy | +| `UO_CLIENT_PATH` env var | The deploy-time default, since the path usually describes a mount the deployment sets up | + +The value may be **the file itself or a directory to search**, because both are +natural answers to "where is it". A directory is searched case-insensitively (the +client writes `Cliloc.enu` on Windows; the site usually runs on Linux) for, in +order: `clilocs.tsv`, `clilocs.csv`, `cliloc.plain.enu`, `cliloc.enu.plain`, +`clilocs.txt`, `cliloc.enu`. + +That ordering puts explicitly-converted names first on purpose. Pointing the +setting straight at an unconverted client directory finds `cliloc.enu`, which is +compressed — and the site says so by name rather than failing obscurely: + +``` +status: unavailable +code: COMPRESSED +reason: This is a compressed (Mythic-format) cliloc file, which the site cannot + read. Convert it to the plain format first — see docs/website/CLILOCS.md. +``` + +## Refresh contract + +Identical in shape to the spawn atlas, and for the same reasons: + +- **It never blocks startup.** No path, an unreadable file, a wrong-format file, + a database error — all caught and logged. The site comes up either way. +- **Hash-gated.** The boot path hashes the file and skips the parse entirely when + it matches what is loaded, which is every restart that did not follow a client + patch. Measured on a stock table: **14 ms** for the no-op, **663 ms** for a full + parse and replace. +- **A `PARSER_VERSION` bump also counts as drift**, so a corrected parse reaches + an install whose client never patches. + +### Why there is no staged-approval flow + +The atlas stages a refresh that would *remove a facet*, because a half-copied +tree and a real map change are indistinguishable from inside the process. A +cliloc file is one file with one hash, and its realistic corruption — a partial +copy — makes the parser fail on a truncated record instead of yielding a +plausible-but-short table. **The ambiguity the atlas has to escalate to a human is +one this parser can simply detect**, so it refuses the import and leaves the +previous table serving. Verified: a file truncated to half its length reports + +``` +code: TRUNCATED +reason: Truncated record header at byte 2486759 (74909 entries read) +``` + +and the 67,496 rows already loaded are untouched. + +## What gets stored + +| | | +|---|---| +| Parsed from a stock `Cliloc.enu` | **123,490** entries | +| Of those, empty strings | **55,994** (ids the client reserves and never uses) | +| Stored in `shard_clilocs` | **67,496** | + +Blank entries are dropped at import. A row resolving to no name is +indistinguishable from no row at all to every caller, and dropping them makes the +binary and text imports converge on **identical** content — the binary format +carries the blanks explicitly and a text export may or may not, depending on the +tool. Verified: both formats import to the same 67,496 rows with the same keys. + +`text` is `TEXT`, not `VARCHAR`: the long property descriptions reach 12 KB, and +silently truncating them would be worse than storing them. The index that matters +for marketplace search is on the denormalized `shard_vendor_items.display_name`, +not here. + +## How names are applied + +**Resolution happens server-side.** The table is never served *as* a table and +there is no public route for it. Two reasons: 67k rows would dwarf any page that +used them, and the Android client consumes the same JSON and would otherwise need +its own copy. + +`resolveMany()` takes a batch of ids and returns a `Map` holding only those that +resolved to something displayable, so "no such id" and "id with no usable name" +collapse into one branch at the call site. It never throws — a cliloc lookup is +decoration on someone's character sheet, and a database blip must not fail the +sheet. A capped in-process cache fronts it; measured cold **4.2 ms**, warm +**0.015 ms**. + +### `displayText()` + +Cliloc strings interpolate arguments the client pulls from an item's property +list — `~1_val~`, `~2_NAME~`. **We never have those**: the bridge sends the id, +not the packet. So a name carrying them is reduced to what is actually knowable. + +| Raw | Displayed | +|---|---| +| `quarter staff` | `quarter staff` | +| `cold damage ~1_val~%` | `cold damage` | +| `[~1_stuff~]` | *(nothing — the whole string was the argument)* | +| `50%` | `50%` | + +The trailing `%` in row two is dropped **because a placeholder was removed** — it +is the unit belonging to the number we never had. Row four shows why that +condition matters: stripping `%` unconditionally would corrupt a string that +legitimately ends in one. + +### Consumers + +- **Character sheet equipment.** `enrichCharProfile` attaches `clilocName` to each + item. A player-given `name` always wins — "Bob's lucky axe" must not be + relabelled "hatchet" — and the client re-states that precedence. +- **Reward titles.** `titles.rewardResolved` is a parallel array with the numeric + entries turned into words (`null` where nothing resolved). The sheet used to + *skip* numeric reward titles entirely, having no way to render them. +- **Marketplace listings** (Protocol 3.0 §8) denormalize the resolved name into + `shard_vendor_items.display_name` so search can index it. + +## Admin surface + +All admin-only, alongside the atlas under Admin → Shard: + +| Route | Purpose | +|---|---| +| `GET /api/v1/admin/shard/clilocs` | Path, resolved file, readability, drift, entry count | +| `POST /api/v1/admin/shard/clilocs/import` | Reload after a client patch; `{ "force": true }` reimports an unchanged file | +| `PUT /api/v1/admin/shard/clilocs/path` | Set the path; blank disables resolution | + +A refresh **result is not an exception**: a missing file, or the likely mistake of +pointing at the client's own compressed `Cliloc.enu`, answers `200` with +`status: "unavailable"` and a reason. A `500` would say only "something broke"; +the operator needs to be told which file to convert. Setting the path +deliberately does **not** import as a side effect — the response carries the +refreshed status so the panel can offer that as the next step. -- 2.49.1 From e3aabf9e3e85bae733af8b752447764318f33f81 Mon Sep 17 00:00:00 2001 From: wtclaude Date: Wed, 29 Jul 2026 04:25:06 -0500 Subject: [PATCH 2/3] docs(link): fill in the cliloc PR numbers, correct the cliloc file count MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The progress table and CLILOCS.md carried #TBD placeholders until the PRs existed; they now point at website #115 and docs #70. Also corrects "all four Cliloc.* files" to eight (chs, cht, deu, enu, esp, fra, jpn, kor) in both v3.md §8.6 and CLILOCS.md — the compression marker was checked against every one of them, and understating the sample weakens the claim it supports. Co-Authored-By: Claude --- link/v3.md | 4 ++-- website/CLILOCS.md | 11 ++++++----- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/link/v3.md b/link/v3.md index 3ef0cc5..7f108e6 100644 --- a/link/v3.md +++ b/link/v3.md @@ -15,7 +15,7 @@ Each part is marked off here as it lands on `edge`. §9 carries the same state p | 2 | **B/1** — `world.ruleset` (§5) | ✅ **Done** | servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66) | | 3 | **C** — spawn atlas (§6) | ✅ **Done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables) + [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113) (API + pages + admin panel), docs [#67](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/67) + [#68](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/68) | | 4 | **B/2** — `points.board` (§7) | ✅ **Done** | servuo-plugins [#4](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/4), link [#18](https://gitea.whitlocktech.com/RunicGateway/link/pulls/18), website [#114](https://gitea.whitlocktech.com/RunicGateway/website/pulls/114), docs [#69](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/69) | -| 5a | **B/3 dependency** — cliloc table (§8.6) | 🟨 In review | website [#TBD](https://gitea.whitlocktech.com/RunicGateway/website/pulls), docs [#TBD](https://gitea.whitlocktech.com/RunicGateway/docs/pulls) | +| 5a | **B/3 dependency** — cliloc table (§8.6) | 🟨 In review | website [#115](https://gitea.whitlocktech.com/RunicGateway/website/pulls/115), docs [#70](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/70) | | 5b | **B/3** — `vendor.listing` (§8) | ⬜ Not started | — | | 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | ⬜ Not started | — | @@ -739,7 +739,7 @@ the §6 pattern instead — parse on every boot from an operator-configured path gitignored, `PARSER_VERSION` counted as drift. **2. `scripts/buildClilocs.js reads the UO client's Cliloc.enu` is not possible, and the reason -matters.** **Every current client ships its cliloc files COMPRESSED** — all four `Cliloc.*` files +matters.** **Every current client ships its cliloc files COMPRESSED** — all eight `Cliloc.*` files open with a DWORD whose high byte is `0x8E`, the "Mythic" container. The plain layout (`02 00 00 00 01 00`, then `{int32 number, byte flag, uint16 length, UTF-8}`) is what those files looked like *before* that change. Parsing a compressed file as plain does not fail cleanly: it yields ~19k diff --git a/website/CLILOCS.md b/website/CLILOCS.md index a7b5b49..3e97f43 100644 --- a/website/CLILOCS.md +++ b/website/CLILOCS.md @@ -1,6 +1,6 @@ # Cliloc table (item and title names) -**Status:** Complete on `edge` — website [#TBD](https://gitea.whitlocktech.com/RunicGateway/website/pulls). +**Status:** Complete on `edge` — website [#115](https://gitea.whitlocktech.com/RunicGateway/website/pulls/115), docs [#70](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/70). **Design:** [`docs/link/v3.md` §8.6](../link/v3.md) — Protocol 3.0, the dependency Part B/3 was sequenced behind. A "cliloc" is UO's localization table: an integer id mapped to a display string. @@ -16,10 +16,11 @@ The number was never the missing piece. The table was. This is the awkward part, and it is not avoidable: -**Every current UO client ships its cliloc files compressed.** The four -`Cliloc.*` files in a modern client all begin with a DWORD whose high byte is -`0x8E` — the "Mythic" compressed container. The plain layout this site parses is -what those files looked like *before* that change. +**Every current UO client ships its cliloc files compressed.** All eight +`Cliloc.*` files in a modern client (`chs`, `cht`, `deu`, `enu`, `esp`, `fra`, +`jpn`, `kor`) begin with a DWORD whose high byte is `0x8E` — the "Mythic" +compressed container. The plain layout this site parses is what those files +looked like *before* that change. Decompressing it means an inverse-BWT coder with a frequency header — a few hundred lines of bit-level work whose failure mode is plausible-looking garbage -- 2.49.1 From ee0c146d7aa68017d0c43d6a6c4863962180e4f8 Mon Sep 17 00:00:00 2001 From: wtclaude Date: Wed, 29 Jul 2026 06:46:43 -0500 Subject: [PATCH 3/3] docs(link): cliloc overlays for shard-added and shard-edited items MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Shards edit items and add new ones, carrying cliloc ids no stock client table has. The cliloc table now reads a SET of sources rather than one file — a base plus every operator-maintained overlay under `custom/`, hash-gated together and re-read on every boot, exactly as §6 reads the ServUO tree. Matching docs for website #115. website/CLILOCS.md gains a "Shard-added and shard-edited items" section: the directory layout, merge precedence, the per-source breakdown an operator uses to confirm an overlay took effect, and why `custom/` is a convention we chose rather than one discovered from the shard — ServUO has no server-side notion of a custom cliloc, they live in the patched client a shard distributes and nothing in the tree declares them. Scale recorded: the live shard's script tree references 16,434 cliloc ids and only 37 are absent from stock, which is why this is an overlay and not a second table. "Why there is no staged-approval flow" is replaced by "Two ways a refresh is refused", because the set brings back the hazard a single file did not have. A corrupt source fails the parse loudly; a source that has VANISHED parses perfectly and imports a table quietly missing everything it contributed. That is the same ambiguity §6 stages a facet removal for, so it is staged here too (`needsReview`, `{approve:true}` to accept) — as a flag rather than §6's approve/reject pair, because the atlas stores a pending decision SO THAT approving re-parses, and here nothing is stored. Two more traps recorded in §8.6 (now five), both found by running a shard-style overlay rather than another stock-table fixture: - Tidying punctuation unconditionally corrupts real names — a custom "Runic Gateway Sigil (v2)" rendered as "(v2". Stripping leftover brackets is right after a placeholder is removed and wrong otherwise, the same condition the `%` rule already had. - Source labels must be forward-slashed and root-relative or the same directory fingerprints differently on Windows and Linux and every boot looks like a change — the identical bug §6 records. BACKEND_DESIGN.md: the source set, the two refusals, and the updated admin route notes (`approve`, `missingSources`, overlays found beside a file path). Documentation only. Every claim was written after the behaviour was observed: the counts come from the live shard's script tree, and the refusal/approve flow and per-source breakdown are copied from real runs in website #115. PROJECT_TREE.md files are deliberately untouched — CI-generated by the sync-project-tree workflow. Co-Authored-By: Claude --- link/v3.md | 26 +++++++- website/BACKEND_DESIGN.md | 37 ++++++++---- website/CLILOCS.md | 123 +++++++++++++++++++++++++++++++------- 3 files changed, 151 insertions(+), 35 deletions(-) diff --git a/link/v3.md b/link/v3.md index 7f108e6..362718d 100644 --- a/link/v3.md +++ b/link/v3.md @@ -762,7 +762,25 @@ shapes are the plain binary layout and a `numbertext` export; the site and writes the plain form. A shard that never converts is fully supported — names render as ids, exactly as before. -Three traps found by building it, all recorded in `CLILOCS.md`: +**Shards edit items and add new ones**, and those carry ids no stock client table has — so this reads +a **set** of sources, not one file, hash-gated together and re-read on every boot exactly as §6 reads +the ServUO tree: a base (the converted client table) plus every overlay under `custom/`, later +winning. Adding one custom item therefore never means re-exporting a 5 MB client file. Measured on +the live shard for scale: its script tree references **16,434** cliloc ids and only **37** are absent +from stock — tens against a 67k base, which is why an overlay and not a second table. `custom/` is the +one convention here that is ours rather than the shard's, because **ServUO has no server-side notion +of a custom cliloc**: they live in the patched client a shard distributes, and nothing in the tree +declares them. + +That set also brings back a hazard a single file did not have, and §8.6 answers it the way §6 does. A +corrupt source fails the parse loudly, but a source that has **vanished** parses perfectly and imports +a table quietly missing everything it contributed — an unmounted volume is indistinguishable from a +deliberate deletion. So it is **staged, not applied** (`status: 'needsReview'`), reported by both the +import and `status()`, and accepted with `{approve:true}`. It is a flag rather than §6's +approve/reject pair because the atlas stores a pending decision *so that approving re-parses*; here +nothing is stored, so re-reading at approval time is automatic. + +Five traps found by building it, all recorded in `CLILOCS.md`: - **`StringList.SaveStringList` RE-COMPRESSES on save.** It looks exactly like the export path and is not; its output is byte-identical to its compressed input, because its purpose is round-tripping a @@ -773,6 +791,12 @@ Three traps found by building it, all recorded in `CLILOCS.md`: entries vanished, and the import still looked successful. - **`Number('')` is `0`, not `NaN`.** A line starting with a separator imports as a bogus cliloc 0 unless the empty field is rejected explicitly. +- **Tidying punctuation unconditionally corrupts real names.** Stripping leftover brackets is right + after a placeholder is removed (`[~1_stuff~]` → nothing) and wrong otherwise: a shard's custom + `"Runic Gateway Sigil (v2)"` rendered as `"(v2"`. Same shape as the `%` rule. **Found only by + running a shard-style overlay through it** — every stock-table fixture passed. +- **Source labels must be forward-slashed and root-relative**, or the same directory fingerprints + differently on Windows and Linux and every boot looks like a change. The identical bug §6 records. Blank entries are dropped at import (123,490 parsed → **67,496** stored), which also makes the binary and text paths converge on identical content. diff --git a/website/BACKEND_DESIGN.md b/website/BACKEND_DESIGN.md index ba13de7..3172b92 100644 --- a/website/BACKEND_DESIGN.md +++ b/website/BACKEND_DESIGN.md @@ -501,10 +501,19 @@ marketplace listing — but with no table to resolve it against, the character s Import-owned and all-or-nothing in one transaction, same contract as the atlas — including **`DELETE`, not `TRUNCATE`**, for the same reason. -**Sourced from a file the operator converts once from their own UO client**, at a path from the -`cliloc_client_path` setting falling back to `UO_CLIENT_PATH`. Nothing client-derived is committed: -UO's strings are EA's, exactly as the creature sprites are. A shard with nothing configured is fully -supported — names render as ids. Full design and operator guide: [`CLILOCS.md`](CLILOCS.md). +**Sourced from files the operator supplies**, at a path from the `cliloc_client_path` setting falling +back to `UO_CLIENT_PATH`. Nothing client-derived is committed: UO's strings are EA's, exactly as the +creature sprites are. A shard with nothing configured is fully supported — names render as ids. Full +design and operator guide: [`CLILOCS.md`](CLILOCS.md). + +**It reads a SET of sources, not one file**, because shards edit items and add new ones and those +carry cliloc ids no stock client table has. A base (the converted client table) plus every overlay +under `custom/` are re-read on every boot and hash-gated **together**, exactly as the atlas re-reads +`Regions.xml` + `Locations/*.xml` + `Spawns/*.xml` + `ChampionSpawns.xml`. Later sources win, so an +overlay both adds ids and overrides stock ones, and adding one custom item never means re-exporting a +5 MB client file. Scale, measured on the live shard: its script tree references 16,434 cliloc ids and +only 37 are absent from stock — tens of entries against a 67k base, which is why this is an overlay +and not a second table. The conversion step is not avoidable: **every current client ships its cliloc files compressed** (first DWORD's high byte `0x8E`), and ServUO's own bundled `Ultima.StringList` cannot read that @@ -519,11 +528,15 @@ Three decisions worth stating: table is empty strings for ids the client reserves and never uses; a row that resolves to no name is indistinguishable from no row at all, and dropping them makes the binary and text imports converge on identical content. -- **No staged-approval flow, unlike the atlas.** The atlas escalates facet loss because a half-copied - tree and a real map change are indistinguishable from inside the process. A cliloc file is one file - with one hash, and a partial copy makes the parser fail on a truncated record — the ambiguity the - atlas must escalate is one this parser simply detects, so it refuses the import and leaves the - previous table serving. +- **Two refusals, one of them the atlas's.** A corrupt source fails the parse on a truncated record, + so it is caught outright and leaves the previous table serving. But a source that has **vanished** + parses perfectly and imports a table quietly missing everything it contributed — an unmounted volume + and a deliberate deletion are indistinguishable from here, which is precisely the ambiguity the + atlas stages a facet removal for. So it is escalated: `status: 'needsReview'`, nothing applied, + `missingSources` reported by both the import and `status()`, and an admin accepts it with + `{approve:true}`. That is a flag rather than the atlas's approve/reject pair because the atlas + stores a pending decision so that approving **re-parses** the tree; here nothing is stored, so + re-reading at approval time is automatic. **Resolution is server-side and there is no public route.** The table is never served *as* a table: 67k rows would dwarf any page using them, and the Android client consumes the same already-resolved @@ -786,9 +799,9 @@ file a route sits in — that is the property the route manifest freezes. | POST | `/shard/atlas/import` | re-import without restarting; `{force}` ignores the hash gate. **An unreadable tree answers 200 with `status:"unavailable"`, not 500** — `refresh()` reports outcomes rather than throwing (the boot path must never be blocked by a bad tree) and that contract is preserved at the API. | | POST | `/shard/atlas/approve` · `/shard/atlas/reject` | answer a refresh staged because it would REMOVE a facet. Approving **re-parses** the tree, so what lands matches it at approval time; rejecting is remembered against those source hashes so it does not re-prompt every restart. 404 when nothing is staged. | | PUT | `/shard/atlas/path` | point the atlas at a different tree (persisted as `spawn_atlas_servuo_path`, which wins over `SERVUO_PATH`). Blank clears it. Deliberately **does not import** — moving the mount and reloading the world are separate decisions — and returns fresh status so the panel can offer the import next. | -| GET | `/shard/clilocs` | cliloc-table status (`adminOnly`): the configured path, the file actually resolved (the path may be a directory), readability, drift against what is loaded, and the entry count. `configured:false` is a supported state — item names then render as ids. No public counterpart: the table is never served *as* a table. | -| POST | `/shard/clilocs/import` | reload after a client patch; `{force}` ignores the hash gate. **A missing file — or the likely mistake of pointing at the client's own COMPRESSED `Cliloc.enu` — answers 200 with `status:"unavailable"` and a `code`, not 500.** `COMPRESSED` is called out by name: a 500 would say only "something broke", and the operator needs to be told which file to convert. | -| PUT | `/shard/clilocs/path` | point the site at a different cliloc file or directory (persisted as `cliloc_client_path`, which wins over `UO_CLIENT_PATH`). Blank clears it. Deliberately **does not import**, same reasoning as the atlas path. | +| GET | `/shard/clilocs` | cliloc-table status (`adminOnly`): every source found now (base first, then `custom/` overlays in merge order), what each contributed at the last import, readability, drift across the set, the entry count, and `missingSources`. `configured:false` is a supported state — item names then render as ids. No public counterpart: the table is never served *as* a table. | +| POST | `/shard/clilocs/import` | reload after a client patch or an overlay edit; `{force}` ignores the hash gate, `{approve}` accepts a **vanished** source (refused by default — see the table notes above). **A missing path — or the likely mistake of pointing at the client's own COMPRESSED `Cliloc.enu` — answers 200 with `status:"unavailable"` and a `code`, not 500.** `COMPRESSED` is called out by name: a 500 would say only "something broke", and the operator needs to be told which file to convert. | +| PUT | `/shard/clilocs/path` | point the site at a different cliloc base file or directory (persisted as `cliloc_client_path`, which wins over `UO_CLIENT_PATH`). Overlays are read from `custom/` beside it either way. Blank clears it. Deliberately **does not import**, same reasoning as the atlas path. | Every admin write logs to `activity_log`. diff --git a/website/CLILOCS.md b/website/CLILOCS.md index 3e97f43..e0747f5 100644 --- a/website/CLILOCS.md +++ b/website/CLILOCS.md @@ -80,21 +80,74 @@ dotnet run -- "/Ultima.dll" "/Cliloc.enu" /srv/uo-data/cli A UOFiddler GUI export works equally well — anything producing one of the two shapes above is fine. +## Shard-added and shard-edited items + +**Shards edit items and add new ones**, and those carry cliloc ids no stock +client table has. The table is therefore built from a **set** of sources, all +re-read on every boot and hash-gated together — the same shape as the spawn +atlas, which reads `Regions.xml` + `Locations/*.xml` + `Spawns/*.xml` + +`ChampionSpawns.xml` and merges them: + +``` +/ + clilocs.plain ← base: the converted client table + custom/ + 01-uomysticmoon.tsv ← overlays: shard additions and overrides + 02-events.tsv +``` + +Overlays use the same delimited-text format, are read in **sorted order**, and +**later sources win** — so an overlay both *adds* ids the client never had and +*overrides* stock ones the shard has re-purposed. Any `.tsv`, `.csv`, `.txt`, +`.enu` or `.plain` file in `custom/` is picked up; anything else (a `README.md`, +say) is ignored. + +Adding, editing or removing any overlay counts as drift, so a new custom item +needs only a file edit and a restart — or the admin panel's Import button. +**Adding one item never means re-exporting a 5 MB client file.** + +The import result reports what each source contributed, which is how you confirm +an overlay took effect — `overrode: 0` on a file meant to re-label stock items +says it did not: + +```json +"sources": [ + { "label": "clilocs.plain", "kind": "base", "entries": 123490, "added": 123490, "overrode": 0 }, + { "label": "custom/uomysticmoon.tsv", "kind": "custom", "entries": 2, "added": 1, "overrode": 1 } +] +``` + +**Why a convention rather than discovery.** Everywhere else this pipeline follows +the shard's own files, but **ServUO has no server-side notion of a custom +cliloc** — they live in the patched client a shard distributes to its players, +and nothing in the tree declares them. There is nothing to discover, so `custom/` +is the one thing here that is our convention rather than the shard's. (An +operator who *does* patch their client cliloc needs no overlay at all: convert +the patched file and their edits are simply in the base.) + +Measured on the live shard for scale: its script tree references **16,434** cliloc +ids and only **37** are absent from the stock client table — tens of entries +against a 67k base, which is what makes an overlay the right shape rather than a +second full table. + ## Configuring the path -Two ways to point at the converted file, the setting winning over the -environment: +Two ways to point at the sources, the setting winning over the environment: | Source | Notes | |---|---| | `cliloc_client_path` setting | Admin-editable (Admin → Shard); takes effect on the next refresh without a redeploy | | `UO_CLIENT_PATH` env var | The deploy-time default, since the path usually describes a mount the deployment sets up | -The value may be **the file itself or a directory to search**, because both are -natural answers to "where is it". A directory is searched case-insensitively (the -client writes `Cliloc.enu` on Windows; the site usually runs on Linux) for, in -order: `clilocs.tsv`, `clilocs.csv`, `cliloc.plain.enu`, `cliloc.enu.plain`, -`clilocs.txt`, `cliloc.enu`. +The value may be **the base file itself or a directory to search**, because both +are natural answers to "where is it". Overlays are read from a `custom/` +directory beside the base **either way** — pointing at a file does not forfeit +them. + +A directory is searched case-insensitively (the client writes `Cliloc.enu` on +Windows; the site usually runs on Linux) for, in order: `clilocs.tsv`, +`clilocs.csv`, `clilocs.plain`, `cliloc.plain`, `cliloc.plain.enu`, +`cliloc.enu.plain`, `clilocs.txt`, `cliloc.enu`. That ordering puts explicitly-converted names first on purpose. Pointing the setting straight at an unconverted client directory finds `cliloc.enu`, which is @@ -120,22 +173,44 @@ Identical in shape to the spawn atlas, and for the same reasons: - **A `PARSER_VERSION` bump also counts as drift**, so a corrected parse reaches an install whose client never patches. -### Why there is no staged-approval flow +### Two ways a refresh is refused -The atlas stages a refresh that would *remove a facet*, because a half-copied -tree and a real map change are indistinguishable from inside the process. A -cliloc file is one file with one hash, and its realistic corruption — a partial -copy — makes the parser fail on a truncated record instead of yielding a -plausible-but-short table. **The ambiguity the atlas has to escalate to a human is -one this parser can simply detect**, so it refuses the import and leaves the -previous table serving. Verified: a file truncated to half its length reports +**A corrupt file** — the realistic failure for any single source — makes the +parser fail on a truncated record rather than yield a plausible-but-short table, +so it is caught outright. Verified: a file truncated to half its length reports ``` code: TRUNCATED reason: Truncated record header at byte 2486759 (74909 entries read) ``` -and the 67,496 rows already loaded are untouched. +and the rows already loaded are untouched. A malformed overlay names the file it +came from (`custom/broken.tsv: No cliloc entries found…`), because "which of my +six overlay files is broken" is otherwise a guessing game. + +**A source that has VANISHED** is the hazard a single file did not have. It +parses perfectly and imports a table quietly missing everything that file +contributed — and an unmounted volume looks exactly like a deliberate deletion +from here. This is the same ambiguity the atlas stages a facet removal for, so it +is escalated rather than applied: + +``` +status: needsReview +reason: 1 previously-loaded cliloc source(s) are missing; + the existing table is unchanged +missingSources: ["custom/uomysticmoon.tsv"] +``` + +`status()` reports `missingSources` too, so the panel can show it before anyone +clicks Import. An admin accepts it by re-running the import with +`{ "approve": true }`. + +**Why that is a flag and not the atlas's approve/reject pair.** The atlas stores +a pending decision in its own table so that approving *re-parses the tree*, which +is what keeps a multi-megabyte blob out of the database and makes the applied +result match the tree at approval time. Here nothing is stored, so re-reading at +approval time is automatic — the decision is a single boolean on the import an +admin was already going to run. ## What gets stored @@ -182,11 +257,15 @@ not the packet. So a name carrying them is reduced to what is actually knowable. | `cold damage ~1_val~%` | `cold damage` | | `[~1_stuff~]` | *(nothing — the whole string was the argument)* | | `50%` | `50%` | +| `Runic Gateway Sigil (v2)` | `Runic Gateway Sigil (v2)` | -The trailing `%` in row two is dropped **because a placeholder was removed** — it -is the unit belonging to the number we never had. Row four shows why that -condition matters: stripping `%` unconditionally would corrupt a string that -legitimately ends in one. +**Punctuation is only tidied when a placeholder was actually removed.** The +trailing `%` in row two is the unit belonging to the number we never had, and the +brackets in row three only ever wrapped the argument — but a string with no +placeholder has no such debris, and trimming it anyway corrupts real names. Rows +four and five are the ones that caught it: a shard's custom +`"Runic Gateway Sigil (v2)"` rendered as `"(v2"` while the bracket trim was +unconditional. ### Consumers @@ -205,8 +284,8 @@ All admin-only, alongside the atlas under Admin → Shard: | Route | Purpose | |---|---| -| `GET /api/v1/admin/shard/clilocs` | Path, resolved file, readability, drift, entry count | -| `POST /api/v1/admin/shard/clilocs/import` | Reload after a client patch; `{ "force": true }` reimports an unchanged file | +| `GET /api/v1/admin/shard/clilocs` | Sources found, what each contributed at the last import, readability, drift, entry count, `missingSources` | +| `POST /api/v1/admin/shard/clilocs/import` | Reload after a client patch or an overlay edit; `{ "force": true }` reimports an unchanged set, `{ "approve": true }` accepts a vanished source | | `PUT /api/v1/admin/shard/clilocs/path` | Set the path; blank disables resolution | A refresh **result is not an exception**: a missing file, or the likely mistake of -- 2.49.1