From bbd69a8e2eb7870d7cdf817d85117552dad3c8ae Mon Sep 17 00:00:00 2001 From: wtclaude Date: Thu, 10 Sep 2026 11:13:56 -0500 Subject: [PATCH] docs(link): clilocs come over the bridge now, and UOFiddler's first job is gone MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 2 of the Asset Bridge is built, so the documentation stops telling an operator to install a GUI tool. `v8.md` gains §9.1 and §9.2 — what the port cost, what it measured, and where the base table comes from now. The measurement worth keeping: **67,496 rows in 290 ms**, which is exactly what UOFiddler's own `Ultima.dll` produced from this same client through the converter this phase deletes. An independent implementation agreeing to the row is the strongest check available that a format decoder is correct, and it is not something a subtly-wrong one produces. §17 records the four shapes the org lead settled before any of it was written. Two departed from the recommendation: **the bridge always wins** (no source setting — there is no version of that question an operator benefits from answering) and **import is admin-triggered** (boot does not call the shard at all). `CLILOCS.md` is rewritten around that: where the table comes from, what arrives and in how many pieces, the refusals — including the two the file pipeline had no equivalent of (a client patched mid-import, and the base's exemption from the vanished-source rule, which exists so an upgraded install is not asked to approve a change the upgrade itself made). `UOFIDDLER.md` loses Part 1 entirely rather than having it rewritten. What is left is creature art, which phase 5 takes, after which the page goes away. `v3.md` §8.6 keeps its reasoning with a note saying what superseded it, because the argument for why the manual step existed is still the argument for why this was worth building. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4 --- README.md | 2 +- link/v3.md | 5 + link/v8.md | 111 +++++++++++++++-- modules/uo/README.md | 2 +- website/CLILOCS.md | 290 ++++++++++++++++++++++++------------------- website/UOFIDDLER.md | 238 ++++++----------------------------- 6 files changed, 312 insertions(+), 336 deletions(-) diff --git a/README.md b/README.md index d057676..c9e7723 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,7 @@ sidecar as a service, and hands you the values the website needs. | [MODERATION_APPEALS.md](website/MODERATION_APPEALS.md) | Moderation actions, content reports and the appeals flow | | [SPAWN_ATLAS.md](website/SPAWN_ATLAS.md) | The bestiary / spawn atlas: what the shard contains, parsed from its own ServUO tree | | [CLILOCS.md](website/CLILOCS.md) | UO's id → name table: converting one from your client so items have names | -| [UOFIDDLER.md](website/UOFIDDLER.md) | **Operator runbook** — step-by-step extraction from your own UO client (cliloc table, creature art) | +| [UOFIDDLER.md](website/UOFIDDLER.md) | **Operator runbook** — extracting creature art from your own UO client. Shrinking: the cliloc half moved to the [Asset Bridge](link/v8.md) in protocol 8 | | [MARKETPLACE.md](website/MARKETPLACE.md) | The player-vendor index: how it is gathered, what it costs, how to tune it | | [website-README.md](website/website-README.md) | Snapshot of the website repo's README (setup/run reference) | | [test-plan.md](website/test-plan.md) | The website's test strategy and harness | diff --git a/link/v3.md b/link/v3.md index cd79d2a..476e59c 100644 --- a/link/v3.md +++ b/link/v3.md @@ -823,6 +823,11 @@ shapes are the plain binary layout and a `numbertext` export; the site and writes the plain form. A shard that never converts is fully supported — names render as ids, exactly as before. +> **Superseded by [`v8.md`](v8.md) §9 (protocol 8, phase 2).** The decompressor is in the overlay +> now, the shard reads its own client, and `server/tools/cliloc-export/` has been deleted. The +> paragraph above is kept as the record of why the manual step existed; everything else in this +> section — the overlay set, the hash gate, the refusals — still describes what runs. + **Shards edit items and add new ones**, and those carry ids no stock client table has — so this reads a **set** of sources, not one file, hash-gated together and re-read on every boot exactly as §6 reads the ServUO tree: a base (the converted client table) plus every overlay under `custom/`, later diff --git a/link/v8.md b/link/v8.md index 8bbb192..d8dd5c1 100644 --- a/link/v8.md +++ b/link/v8.md @@ -714,6 +714,84 @@ Protocol 8 writes rather than calls (§4) — and from that point: - `website/server/tools/cliloc-export/` is retired, and `UOFIDDLER.md` is deleted rather than rewritten. +### 9.1 What phase 2 built, and what the port cost + +**Built 2026-09-10.** `BridgeCliloc.cs` in the overlay, `GET /cliloc` in the sidecar, and a paging +walk in module-uo that merges the shard's table under the `custom/` overlays. + +The port is `MythicDecompress` + `MoveToFront` rewritten against plain arrays: the upstream is +`Span` / `ArrayPool` / `BinaryPrimitives` code and ServUO targets `net48`, which has none of +them without a package this tree does not vendor. The algorithm is unchanged, **including the parts +that read oddly** — the three-region count/cursor/end table and the symbol-table shifts are +upstream's, deliberately, because a tidier rewrite of somebody else's format decoder is a chance to +be subtly wrong in a way that produces plausible text. + +Two bounds checks were added, and they are the only behavioural change: the upstream indexes its +payload without checking, which is safe for a file the client wrote and is not safe for a file this +shard was handed. A truncated container now reports `UNREADABLE` with the byte it wanted, instead of +throwing an `IndexOutOfRangeException` from inside a decoder. + +**Measured on this machine's stock client:** 4,989,921 bytes read, decompressed and parsed in +**290 ms**, yielding **67,496** non-blank rows in id order, ~5.4 MB on the wire, ~11 pages. + +That 67,496 is the acceptance test, and it is worth saying why it is a strong one: it is the number +[`../website/CLILOCS.md`](../website/CLILOCS.md) already recorded for this same client, measured +through **UOFiddler's own `Ultima.dll`** by the converter this phase deletes. An independent +implementation agreeing to the row is not something a subtly-wrong decoder produces. Checked +alongside it: zero U+FFFD, so the UTF-8 survived; 696 rows with non-ASCII text, spot-checked as +correct curly quotes; the longest row is a 12,149-character EULA, which is also why the record +length is read **unsigned** (a signed read turns anything over 32 KB negative). + +**The shard drops blanks before they reach the wire.** ~56,000 of the 123,490 entries are empty +strings the client reserves, the website discards them at import anyway, and sending them would +double the transfer for data thrown away on arrival. + +**Language is a parameter (`enu` by default) but not a free one.** `Ultima.Files` resolves only the +names in its own table, which for clilocs is `enu`, `deu`, `custom1` and `custom2`; anything else is +refused `NOT_FOUND` rather than answered with an empty table. `custom1`/`custom2` are the +client-side custom cliloc files a shard ships to its players — readable here, and deliberately not +wired into the website's import, because `custom/` on the site is the supported answer for +shard-added ids. + +**`assets.error` gained a `code`.** Phase 1 chose between 403 and 400 by looking for the word +"disabled" in the operator-facing sentence, which makes prose load-bearing; the codes are +`DISABLED` (403), `NOT_FOUND` (404), `UNREADABLE` (422), `UNAVAILABLE` (503) and `BAD_REQUEST` +(400). The substring check survives as a fallback, because an overlay and a sidecar are deployed +separately and a phase-1 shard must keep its 403. + +### 9.2 Where the base comes from now, and when it is read + +**The shard wins whenever uo-link is configured and enabled.** No mode setting: there is no version +of that question an operator benefits from answering. A file on disk remains the source only where +there is no shard link, plus a one-off explicit path — the deprecated pipeline, kept for installs +with no bridge and for development. + +**Boot no longer imports on the bridge.** The file path could hash 5 MB locally and skip in 14 ms; a +shard round trip in the boot sequence would be spent answering "no" on every restart but the one +after a client patch — and patching a client is an operator action, so importing became one: +Admin → Shard → Import. Whatever table is loaded keeps serving until then. + +Three things about the walk are worth recording because each is a way a shard can hand back a table +that *looks* complete: + +- **Only `cut: "end"` finishes it** (§3.4). A short page can equally be a spent budget. +- **The cursor must advance**, or the walk stops rather than spinning. +- **Every page echoes the source's size and mtime.** A client patched mid-import is refused + outright (`SOURCE_CHANGED`) rather than stitched together — half of what arrived came from a file + that no longer exists and nothing later can tell which half. + +**The base is exempt from the vanished-source rule**, and that is an upgrade detail rather than a +design preference: an install that used the file pipeline carries its base file's label in the +stored fingerprint, and on the bridge that label is *supposed* to disappear. Counting it as vanished +would make the first import after the upgrade demand approval for a change the upgrade itself made. +Overlays keep the rule in full. + +**module-uo's protocol pin moved 7 → 8 in this phase**, which is the third declaration site §15 +names and the one nothing enforces. Phase 1 moved the sidecar and the overlay together because the +installer refuses to pair a mismatched bundle; this one had to be moved by hand, in the phase that +first calls a protocol-8 route — the same trap that left the pin at 5 for two phases of the Event +System while every REST call was answered `409`. + **What survives untouched is the `custom/` overlay mechanism.** Shard-added items carry cliloc ids no client table has, and ServUO has no server-side notion of a custom cliloc — that is a real gap in the *game*, not an artefact of the manual pipeline, and `CLILOCS.md`'s reasoning for it stands. @@ -846,7 +924,7 @@ new pipe. | `assets.manifest` | `assets.manifest.ok` | Stage 2: `[{key, sha256, bytes}]`, paged | phase 3 | | `assets.fetch` | `assets.fetch.ok` | Content for an explicit key list, paged | phase 3 | | `assets.bodies` | `assets.bodies.ok` | Slug → body id (§8, Core thread) | phase 3 | -| `cliloc.table` | `cliloc.table.ok` | The converted table, paged | phase 2 | +| `cliloc.table` | `cliloc.table.ok` | The decompressed table, paged (`?lang=`, `?cursor=`) | **phase 2** | | `tree.manifest` / `tree.fetch` | `.ok` | §10, the ServUO tree files | phase 7 | Every one of them is refused outright when `Bridge.AssetsEnabled` is off, and every one of them @@ -854,9 +932,11 @@ requires a `reqId` — a request without one is refused rather than answered, be reply is by definition an event, and §3.1 is the reason none of this may ever be one. **Sidecar REST** mirrors those one for one under `/assets/*`, `/cliloc`, `/tree/*`, carrying -`X-UOLink-Version: 8` and forwarding verbatim. `GET /assets/sources` exists as of phase 1; its -responder maps `bridge.busy` to **425** (flow control, and the ordinary answer mid-import rather -than a rare one) and a disabled plane to **403** (an operator's refusal, not a malformed request). +`X-UOLink-Version: 8` and forwarding verbatim. `GET /assets/sources` exists as of phase 1 and +`GET /cliloc` as of phase 2; their responder maps `bridge.busy` to **425** (flow control, and the +ordinary answer mid-import rather than a rare one) and reads the refusal's `code` for the rest — +**403** disabled, **404** no such file, **422** a file the shard cannot decode, **503** a shard that +cannot do this now (§9.1). **Website admin** (`Admin → Shard`, admin-only): status, **Import**, **Update**, approve/reject for a vanished key, and the existing path settings. Every action to the admin activity log, as @@ -875,7 +955,7 @@ disagree, so a split bump means the next bundle silently fails to compose. | `servuo-plugins/` | Extraction over ServUO's own `Ultima` (§4), the cliloc decompressor (§9), body resolution (§8), the request handlers, `overlay.toml` | | `link/` | Six command families forwarded, the REST surface, **the inbound line cap (§3.3)**, `PROTOCOL_VERSION` | | `module-uo/` | Client calls, asset store, the atlas source backend (§10), cliloc ingest, admin surface | -| `website/` | None expected — `ctx.uploads` already suffices (§12) | +| `website/` | Almost none — `ctx.uploads` already suffices (§12). Phase 2 deleted `server/tools/cliloc-export/`, the converter this protocol retires | | `docs/` | This file; rewrite `CLILOCS.md` §Converting and `SPAWN_ATLAS.md` §Artwork + §Configuring; **delete `UOFIDDLER.md`**; add the libgdiplus prerequisite to `SHARD_PREREQS.md` (§4.4) | | `installer/` | A `doctor` check for libgdiplus on Linux hosts (§4.4). Bundle pairing already enforces §15 | | `android-app/` | Consumes images by URL; no parity gate expected until a screen shows one | @@ -889,7 +969,7 @@ disagree, so a split bump means the next bundle silently fails to compose. |---|---|---| | 0 | **DONE 2026-09-10.** Spike: the vendored decoders driven from inside a running ServUO, over a client broken 21 ways. §4 stands; the finding was 22,102 wrong pictures on a *stock* client, and the validator that answers them (§4.5-§4.7) | servuo-plugins | | 1 | **DONE 2026-09-10.** The transport: `assets.sources`, the single-slot gate (§3.2), the paging envelope (§3.4), the sidecar line cap (§3.3), `EXTRACTOR_VERSION`, `NO_IMAGING`, protocol 7→8. Plus §4.5's validator promoted into the overlay and extended to animations — which found **357 wrong pictures in the body catalogue on a stock client** and cut it from 1,144 to 787 (§4.8) | servuo-plugins, link | -| 2 | Clilocs end to end; retire the converter and `UOFIDDLER.md` §Part 1 | all | +| 2 | **DONE 2026-09-10.** Clilocs end to end (§9.1, §9.2): the Mythic decompressor ported into the overlay, `cliloc.table` + `GET /cliloc`, the paging walk and the source switch on the website, module-uo's protocol pin 7→8. `cliloc-export/` deleted and `UOFIDDLER.md` §Part 1 with it. **67,496 rows, 290 ms, ~11 pages** — the same count UOFiddler's own DLL produced from this client | all | | 3 | Body resolution (§8) + the **787**-body catalogue (§4.8); `shard_spawn_creatures.art` filled | servuo-plugins, module-uo | | 4 | **The UOP animation decoder (§4.3)** — the **eight** player bodies the legacy path cannot reach (§4.8 moved the elf ghosts across), the never-sweep rule already enforced in code by phase 1's `ResolveAnimation` | servuo-plugins | | 5 | Item statics and land on demand, hued keys, the cache | servuo-plugins, module-uo | @@ -953,7 +1033,24 @@ in the document. sidecar or left to the website as a convention — the bound belongs where the memory is. And one **paging envelope** (`more` / `cursor` / `cut`) is defined now, with `assets.sources` as its first user, rather than left for whichever family pages first to invent. -7. **§4.8: the catalogue is 787 bodies, not 1,144 — measured 2026-09-10, phase 1.** Recorded here +7. **§9: the cliloc pipeline's four shapes — settled 2026-09-10, phase 2.** All four were put to + the org lead before any of it was written, and two of them departed from the recommendation: + + - **The wire carries rows, not bytes.** `[{n, f, t}]` inside the paging envelope, rather than a + reconstituted plain-binary file in base64. The container decompresses into *records*; the + plain layout is a file shape that would have had to be synthesised on the shard and re-parsed + on the website, at a 33% base64 premium, to reach the same place. + - **The shard omits blank entries** — ~56,000 of 123,490 — because the website discards them at + import anyway. The transfer halves and nothing observable changes. + - **The bridge always wins; the file upload is deprecated** (org lead, departing from the + proposed `auto`/`bridge`/`file` setting). There is no version of "which source?" an operator + benefits from answering, so there is no setting to answer it with. A file remains the source + only where there is no shard link, plus a one-off explicit path. + - **Import is admin-triggered on the bridge** (org lead, departing from the proposed boot + refresh). Boot does not call the shard at all — see §9.2 for why that is the right trade and + what it costs. + +8. **§4.8: the catalogue is 787 bodies, not 1,144 — measured 2026-09-10, phase 1.** Recorded here because it changes §11's sizing, phase 3's scope and phase 4's, and because of *how* it was found: the animation path has §4.5's shared-buffer defect too, and 357 ids with no art were returning the previously-decoded body's bitmap. The elf ghosts moved from "decodes" to "no art" diff --git a/modules/uo/README.md b/modules/uo/README.md index 9de2ec9..d7b410c 100644 --- a/modules/uo/README.md +++ b/modules/uo/README.md @@ -21,7 +21,7 @@ these routes *mean* are the ones that already existed and did not move: | [`SPAWN_ATLAS.md`](../../website/SPAWN_ATLAS.md) | The bestiary / spawn atlas, parsed from the shard's own ServUO tree | | [`MARKETPLACE.md`](../../website/MARKETPLACE.md) | The player-vendor index | | [`CLILOCS.md`](../../website/CLILOCS.md) | UO's id → name table | -| [`UOFIDDLER.md`](../../website/UOFIDDLER.md) | Operator runbook for extracting the cliloc table and creature art | +| [`UOFIDDLER.md`](../../website/UOFIDDLER.md) | Operator runbook for extracting creature art. The cliloc table now arrives over the bridge ([`link/v8.md`](../../link/v8.md) §9) | | [`../../link/PLAN.md`](../../link/PLAN.md), [`../../link/INTEGRATION.md`](../../link/INTEGRATION.md) | The wire protocol this module speaks to the sidecar | --- diff --git a/website/CLILOCS.md b/website/CLILOCS.md index f2e1413..207de87 100644 --- a/website/CLILOCS.md +++ b/website/CLILOCS.md @@ -1,7 +1,8 @@ # Cliloc table (item and title names) -**Status:** Complete on `edge` — website [#115](https://gitea.whitlocktech.com/RunicGateway/website/pulls/115), docs [#70](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/70). -**Design:** [`docs/link/v3.md` §8.6](../link/v3.md) — Protocol 3.0, the dependency Part B/3 was sequenced behind. +**Status:** Complete. The base table now arrives **over the bridge** — protocol 8, phase 2. +**Design:** [`docs/link/v3.md` §8.6](../link/v3.md) (the table itself, Protocol 3.0) and +[`docs/link/v8.md` §9](../link/v8.md) (the Asset Bridge, which retired the manual conversion). A "cliloc" is UO's localization table: an integer id mapped to a display string. **Items on the wire carry a `LabelNumber`, not a name.** The bridge has always @@ -12,86 +13,70 @@ render `id 1023721` where the game renders **"quarter staff"**. The number was never the missing piece. The table was. -## Why the operator has to convert the file +## Where the table comes from -This is the awkward part, and it is not avoidable: +**The shard reads its own client.** A ServUO server cannot boot without a UO +client — `Config/DataPath.cfg` resolves into `Core.DataDirectories` at run time — +so the file this table is made of is already sitting on the shard host. Since +protocol 8 the plugin decompresses it and serves it over the bridge, and the site +imports it like any other shard read: -**Every current UO client ships its cliloc files compressed.** All eight -`Cliloc.*` files in a modern client (`chs`, `cht`, `deu`, `enu`, `esp`, `fra`, -`jpn`, `kor`) begin with a DWORD whose high byte is `0x8E` — the "Mythic" -compressed container. The plain layout this site parses is what those files -looked like *before* that change. - -Decompressing it means an inverse-BWT coder with a frequency header — a few -hundred lines of bit-level work whose failure mode is plausible-looking garbage -rather than an error. The site has no business carrying that at runtime. - -Two facts make the alternatives worse, not better: - -- **ServUO cannot read it either.** Its bundled `Ultima.StringList` implements - only the plain layout, so on a modern client `VendorSearch.StringList` is null - and `VendorSearch.GetItemName` returns `item.Name` — usually nothing. The - shard cannot supply names on our behalf; the in-game Vendor Search gump has the - same gap. -- **Nothing client-derived may be committed.** UO's strings are EA's. The repo - ships no string table for the same reason it ships no artwork and no map - snapshot — see [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md). - -So the conversion happens **once, on the operator's machine, against their own -client**, and the site reads the result from a path it is given. A shard that -never does this is in a fully supported state: names render as ids, exactly as -they did before the table existed. - -## Converting - -> **Step-by-step operator instructions — where to get UOFiddler, where your -> client files are, and how to verify the import — are in -> [`UOFIDDLER.md`](UOFIDDLER.md).** This section covers the formats and the -> reasoning behind them. - -Either format below is accepted; the site sniffs which one it was handed. - -| Format | Fidelity | Notes | -|---|---|---| -| **Plain binary** (recommended) | Exact | 6-byte header, then `{int32 number, byte flag, uint16 length, UTF-8}` records | -| Delimited text | Loses leading/trailing whitespace | `numbertext` per line; a header row, blank lines and `#` comments are ignored | - -The whitespace caveat is real but cosmetic: ~1,300 of the 123,490 entries in a -stock `Cliloc.enu` are label prefixes like `"max = "` whose trailing space is -meaningful when the client concatenates a value onto them. Nothing on this site -concatenates, and every consumer passes through `displayText()`, which trims. - -### Using the bundled tool - -`server/tools/cliloc-export/` is a small .NET console app that drives -[UOFiddler](https://github.com/polserver/UOFiddler)'s `Ultima.dll` — the -decompressor that already exists and is already maintained — and writes the plain -format. It loads that DLL **reflectively** so it compiles against any SDK, and it -writes the records by hand because UOFiddler's own `SaveStringList` *re-compresses* -on save (its purpose is round-tripping a file back into the client, so its output -is byte-identical to its input — a trap worth knowing about). - -```bash -cd website/server/tools/cliloc-export -dotnet build -c Release - -# binary (recommended) -dotnet run -- "/Ultima.dll" "/Cliloc.enu" /srv/uo-data/clilocs.plain - -# or tab-delimited -dotnet run -- "/Ultima.dll" "/Cliloc.enu" /srv/uo-data/clilocs.tsv --tsv +``` +ServUO shard ──`cliloc.table`──▶ uo-link sidecar ──`GET /cliloc`──▶ website + reads Cliloc.enu, forwards, keeps merges overlays, + decompresses, pages nothing replaces the table ``` -A UOFiddler GUI export works too, but **not unmodified**: its Cliloc tab writes -`Number;Text;Flag` — three columns, the flag *last* — and the parser reads -`numbertext`, so the trailing field is absorbed into the name and -every item renders as `quarter staff;0`. Stripping it is one `sed`, given in -[`UOFIDDLER.md`](UOFIDDLER.md) §Route B. +**Why that was worth building.** Every current UO client ships its cliloc files +compressed: all eight `Cliloc.*` files in a modern client (`chs`, `cht`, `deu`, +`enu`, `esp`, `fra`, `jpn`, `kor`) begin with a DWORD whose high byte is `0x8E` — +the "Mythic" container. The plain layout is what those files looked like *before* +that change, and **ServUO's own bundled `Ultima.StringList` cannot read the new +one either**, which is why `VendorSearch.GetItemName` is inert on a modern shard +and the in-game Vendor Search gump has the same gap. -The parser already tolerates `number,flag,text`, with the flag in the *middle*. -It is not extended to cover the trailing form because a final `;0` is -indistinguishable from a name that genuinely ends that way — a heuristic there -would corrupt real names to save the operator one command. +So until protocol 8 an operator had to install UOFiddler, build a converter +against its `Ultima.dll`, run it over their client and copy a 5 MB file to the web +host — every time they patched. The decompressor now lives in the overlay +(`overlay/Scripts/Custom/Bridge/BridgeCliloc.cs`, ported from UOFiddler, which is +Beerware and therefore clean to bring into a GPL tree), so **none of that is a +step any more**. + +Two things are unchanged and remain the point: + +- **Nothing client-derived is committed.** UO's strings are EA's. The repo ships + no string table for the same reason it ships no artwork and no map snapshot — + see [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md). The extraction happens on the + operator's own host, from their own files, for their own shard. +- **A shard with no table is fully supported.** Names render as ids, exactly as + they did before the table existed. + +### What arrives, and in how many pieces + +The sidecar's reply timeout is 10 s and its inbound line cap is 1 MiB, so the +table is **paged**: the shard cuts at a 512 KiB byte budget and hands back a +cursor, and the site walks it until a page says `more: false`. Measured on a stock +English client: **67,496 rows in about eleven pages**, decoded on the shard in +**290 ms**. + +The rows are `{ n, f, t }` — number, flag, text. **Blanks never leave the shard**: +roughly 56,000 of a stock table's 123,490 entries are empty strings the client +reserves and never uses, the site drops them at import anyway, and sending them +would double the transfer for data that is discarded on arrival. + +Only `cut: "end"` means the table finished. A short page can equally mean the byte +budget was spent, and importing a table that stopped early is the one failure that +is invisible downstream — some items named, some not, which is exactly what *no +table* looks like. + +### The file pipeline is deprecated, not removed + +An install with **no uo-link configured** can still be pointed at a converted file +and works exactly as it did. That path exists for shards with no bridge and for +development without a running ServUO, it accepts the same two formats it always +did (plain binary, or `numbertext` delimited text), and passing an +explicit path to a refresh still selects it as a one-off. Nothing new should be +built on it. ## Shard-added and shard-edited items @@ -103,12 +88,16 @@ atlas, which reads `Regions.xml` + `Locations/*.xml` + `Spawns/*.xml` + ``` / - clilocs.plain ← base: the converted client table custom/ 01-uomysticmoon.tsv ← overlays: shard additions and overrides 02-events.tsv ``` +The base is no longer a file in that directory — it comes from the shard — so on +a bridged install the configured path selects **only** where `custom/` is read +from. (Without a shard link, a converted base file sitting beside `custom/` is +still found, which is the deprecated pipeline above.) + Overlays use the same delimited-text format, are read in **sorted order**, and **later sources win** — so an overlay both *adds* ids the client never had and *overrides* stock ones the shard has re-purposed. Any `.tsv`, `.csv`, `.txt`, @@ -116,8 +105,9 @@ Overlays use the same delimited-text format, are read in **sorted order**, and say) is ignored. Adding, editing or removing any overlay counts as drift, so a new custom item -needs only a file edit and a restart — or the admin panel's Import button. -**Adding one item never means re-exporting a 5 MB client file.** +needs only a file edit and the admin panel's Import button. **Adding one item +never means re-reading the client table** — though on the bridge that is now +cheap enough not to matter much. The import result reports what each source contributed, which is how you confirm an overlay took effect — `overrode: 0` on a file meant to re-label stock items @@ -125,11 +115,14 @@ says it did not: ```json "sources": [ - { "label": "clilocs.plain", "kind": "base", "entries": 123490, "added": 123490, "overrode": 0 }, - { "label": "custom/uomysticmoon.tsv", "kind": "custom", "entries": 2, "added": 1, "overrode": 1 } + { "label": "cliloc.enu", "kind": "shard", "entries": 67496, "added": 67496, "overrode": 0 }, + { "label": "custom/uomysticmoon.tsv", "kind": "custom", "entries": 2, "added": 1, "overrode": 1 } ] ``` +`kind` says where a source came from: `shard` over the bridge, `base` a converted +file on disk, `custom` an overlay. + **Why a convention rather than discovery.** Everywhere else this pipeline follows the shard's own files, but **ServUO has no server-side notion of a custom cliloc** — they live in the patched client a shard distributes to its players, @@ -143,28 +136,29 @@ ids and only **37** are absent from the stock client table — tens of entries against a 67k base, which is what makes an overlay the right shape rather than a second full table. -## Configuring the path +## Choosing the source -Two ways to point at the sources, the setting winning over the environment: +Nothing to configure: **the shard wins whenever uo-link is configured and +enabled.** There is no mode setting, because there is no version of this question +an operator benefits from answering — a shard that can serve its own client table +is strictly better than a file somebody converted by hand months ago. -| Source | Notes | +The two escape hatches, both deliberate: + +| | | |---|---| -| `cliloc_client_path` setting | Admin-editable (Admin → Shard); takes effect on the next refresh without a redeploy | -| `UO_CLIENT_PATH` env var | The deploy-time default, since the path usually describes a mount the deployment sets up | +| No uo-link configured | The file pipeline, exactly as before | +| An explicit `path` passed to a refresh | A one-off "import from this file", which the shard never overrules | -The value may be **the base file itself or a directory to search**, because both -are natural answers to "where is it". Overlays are read from a `custom/` -directory beside the base **either way** — pointing at a file does not forfeit -them. - -A directory is searched case-insensitively (the client writes `Cliloc.enu` on -Windows; the site usually runs on Linux) for, in order: `clilocs.tsv`, -`clilocs.csv`, `clilocs.plain`, `cliloc.plain`, `cliloc.plain.enu`, -`cliloc.enu.plain`, `clilocs.txt`, `cliloc.enu`. - -That ordering puts explicitly-converted names first on purpose. Pointing the -setting straight at an unconverted client directory finds `cliloc.enu`, which is -compressed — and the site says so by name rather than failing obscurely: +`cliloc_client_path` (setting, admin-editable) and `UO_CLIENT_PATH` (env +deploy-time default) still name a path, and the setting still wins over the +environment. What that path *means* narrowed: on a bridged install it is where +`custom/` overlays live. Without a link it is also searched for a base file, +case-insensitively (the client writes `Cliloc.enu` on Windows; the site usually +runs on Linux), in order: `clilocs.tsv`, `clilocs.csv`, `clilocs.plain`, +`cliloc.plain`, `cliloc.plain.enu`, `cliloc.enu.plain`, `clilocs.txt`, +`cliloc.enu` — and pointing it straight at an unconverted client directory still +answers by name rather than failing obscurely: ``` status: unavailable @@ -175,16 +169,25 @@ reason: This is a compressed (Mythic-format) cliloc file, which the site cannot ## Refresh contract -Identical in shape to the spawn atlas, and for the same reasons: - -- **It never blocks startup.** No path, an unreadable file, a wrong-format file, - a database error — all caught and logged. The site comes up either way. -- **Hash-gated.** The boot path hashes the file and skips the parse entirely when - it matches what is loaded, which is every restart that did not follow a client - patch. Measured on a stock table: **14 ms** for the no-op, **663 ms** for a full - parse and replace. -- **A `PARSER_VERSION` bump also counts as drift**, so a corrected parse reaches - an install whose client never patches. +- **It never blocks startup**, and on the bridge it never *touches* startup: boot + imports nothing when the shard is the source. A sidecar round trip in the boot + sequence would be spent answering "no" on every restart but the one after a + client patch — and patching a client is an operator action, so importing is an + operator action. **Admin → Shard → Import** is the button. Whatever table is + loaded keeps serving until then. +- **Still hash-gated**, so pressing Import when nothing changed costs one small + call. The gate is the shard's own `assets.sources`: size, mtime and content + hash of `Cliloc.enu` plus the shard's `EXTRACTOR_VERSION`. A `sha256` of `null` + with `hashing: true` means the shard has not computed it yet (it hashes off the + request path, because the art and animation files it also reports are 343 MB) + — that means *ask again*, never *changed*, and the comparison falls back to + (size, mtime) meanwhile. +- **Without a link** the file path is unchanged: boot hashes the local file and + skips the parse when it matches. Measured on a stock table: **14 ms** for the + no-op, **663 ms** for a full parse and replace. +- **A `PARSER_VERSION` bump counts as drift**, and so does an `EXTRACTOR_VERSION` + bump on the shard — same argument at the other end of the wire: a corrected + reader must reach an install whose client never patches. ### Two ways a refresh is refused @@ -201,7 +204,7 @@ and the rows already loaded are untouched. A malformed overlay names the file it came from (`custom/broken.tsv: No cliloc entries found…`), because "which of my six overlay files is broken" is otherwise a guessing game. -**A source that has VANISHED** is the hazard a single file did not have. It +**An OVERLAY that has VANISHED** is the hazard a single file did not have. It parses perfectly and imports a table quietly missing everything that file contributed — and an unmounted volume looks exactly like a deliberate deletion from here. This is the same ambiguity the atlas stages a facet removal for, so it @@ -209,11 +212,28 @@ is escalated rather than applied: ``` status: needsReview -reason: 1 previously-loaded cliloc source(s) are missing; +reason: 1 previously-loaded cliloc overlay(s) are missing; the existing table is unchanged missingSources: ["custom/uomysticmoon.tsv"] ``` +**The base is deliberately exempt from that question**, and that is an upgrade +detail worth stating: an install that used the converted-file pipeline carries its +base file's label in the stored fingerprint, and on the bridge that label is +*supposed* to disappear. Counting it as a vanished source would make the first +import after the upgrade demand an approval for a change the upgrade itself made. + +**A client patched mid-import** is refused outright rather than staged, because +there is nothing to decide: every page echoes the source file's size and mtime, and +if they move between pages then half of what arrived came from a file that no +longer exists and nothing later can tell which half. + +``` +status: unavailable +code: SOURCE_CHANGED +reason: The shard's cliloc file changed while it was being read; nothing was imported +``` + `status()` reports `missingSources` too, so the panel can show it before anyone clicks Import. An admin accepts it by re-running the import with `{ "approve": true }`. @@ -229,15 +249,20 @@ admin was already going to run. | | | |---|---| -| Parsed from a stock `Cliloc.enu` | **123,490** entries | +| Entries in a stock `Cliloc.enu` | **123,490** | | Of those, empty strings | **55,994** (ids the client reserves and never uses) | | Stored in `shard_clilocs` | **67,496** | -Blank entries are dropped at import. A row resolving to no name is -indistinguishable from no row at all to every caller, and dropping them makes the -binary and text imports converge on **identical** content — the binary format -carries the blanks explicitly and a text export may or may not, depending on the -tool. Verified: both formats import to the same 67,496 rows with the same keys. +Blank entries are dropped, and since protocol 8 they are dropped **on the shard**, +before they reach the wire. A row resolving to no name is indistinguishable from +no row at all to every caller, so sending 56,000 of them would double the +transfer for data discarded on arrival. The site still filters at import, because +an overlay file can carry one and because the file pipeline still exists. + +That the shard's own decoder lands on **exactly 67,496** is also the strongest +check there is that the ported decompressor is correct: the number was measured +first through UOFiddler's `Ultima.dll` against this same client, by an entirely +different implementation. `text` is `TEXT`, not `VARCHAR`: the long property descriptions reach 12 KB, and silently truncating them would be worse than storing them. The index that matters @@ -297,13 +322,24 @@ All admin-only, alongside the atlas under Admin → Shard: | Route | Purpose | |---|---| -| `GET /api/v1/admin/shard/clilocs` | Sources found, what each contributed at the last import, readability, drift, entry count, `missingSources` | -| `POST /api/v1/admin/shard/clilocs/import` | Reload after a client patch or an overlay edit; `{ "force": true }` reimports an unchanged set, `{ "approve": true }` accepts a vanished source | -| `PUT /api/v1/admin/shard/clilocs/path` | Set the path; blank disables resolution | +| `GET /api/v1/admin/shard/clilocs` | Which source is in use (`bridge` / `file`), the shard's client-file fingerprint, what each source contributed at the last import, drift, entry count, `missingSources` | +| `POST /api/v1/admin/shard/clilocs/import` | Reload after a client patch or an overlay edit; `{ "force": true }` reimports an unchanged set, `{ "approve": true }` accepts a vanished overlay | +| `PUT /api/v1/admin/shard/clilocs/path` | Set the overlay path (and, with no shard link, the base file's); blank clears it | -A refresh **result is not an exception**: a missing file, or the likely mistake of -pointing at the client's own compressed `Cliloc.enu`, answers `200` with -`status: "unavailable"` and a reason. A `500` would say only "something broke"; -the operator needs to be told which file to convert. Setting the path -deliberately does **not** import as a side effect — the response carries the -refreshed status so the panel can offer that as the next step. +**Import is now the only thing that refreshes the table on a bridged install**, +since boot no longer asks the shard. It is the button an operator presses after +patching their client. + +A refresh **result is not an exception**, and protocol 8 widened the set of things +that covers: a shard that is down, an asset plane the operator has switched off +(`Bridge.AssetsEnabled`), a client with no cliloc file, a client patched halfway +through the import, plus everything the file pipeline could already report. Each +answers `200` with `status: "unavailable"` and a reason naming what to fix. A +`500` would say only "something broke". Setting the path deliberately does **not** +import as a side effect — the response carries the refreshed status so the panel +can offer that as the next step. + +The sidecar's own statuses are worth knowing when reading a log: **425** is the +shard saying it is busy with another asset request (flow control, and the ordinary +answer mid-import — the site retries), **403** the asset plane switched off, **404** +a client with no such file, **422** a file it has and cannot decode. diff --git a/website/UOFIDDLER.md b/website/UOFIDDLER.md index 96d8e69..f7bd5aa 100644 --- a/website/UOFIDDLER.md +++ b/website/UOFIDDLER.md @@ -1,34 +1,45 @@ # Extracting from your own UO client (UOFiddler) **Audience:** the shard operator, once, at setup time. -**Related:** [`CLILOCS.md`](CLILOCS.md) (why the cliloc conversion is unavoidable), -[`SPAWN_ATLAS.md`](SPAWN_ATLAS.md) (where creature art fits). +**Related:** [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md) (where creature art fits), +[`../link/v8.md`](../link/v8.md) (the Asset Bridge, which is replacing this page). -Two features read data that **only exists inside a UO client**, and a UO client's -files are EA's, not ours to redistribute. So neither this repo nor any image we -publish can ship them — the operator extracts from **their own** client, once, +> ### The cliloc conversion is gone — you no longer do this by hand +> +> **Protocol 8, phase 2.** The shard reads its own client's `Cliloc.enu` and +> serves the table over the bridge, so there is nothing to install, convert or +> copy. Press **Admin → Shard → Import** after you patch your client and that is +> the whole procedure; see [`CLILOCS.md`](CLILOCS.md). +> +> Part 1 of this guide has been deleted rather than rewritten. If you already +> have a converted file it keeps working on an install with no uo-link +> configured, but nobody should make a new one. +> +> **Creature art is next** (phase 5), after which this page goes away entirely. + +What remains here reads data that **only exists inside a UO client**, and a UO +client's files are EA's, not ours to redistribute. So neither this repo nor any +image we publish can ship them — the operator extracts from **their own** client and points the site at the result. | Feature | What it needs | Required? | Without it | |---|---|---|---| -| **Item / title names** ([`CLILOCS.md`](CLILOCS.md)) | `Cliloc.enu`, converted | No | Names render as raw ids — `id 1023721` instead of *quarter staff* | | **Creature art** ([`SPAWN_ATLAS.md`](SPAWN_ATLAS.md)) | Sprites from `.mul`/`.uop` | No | Atlas pages render as text, which is the normal state | -**Both are optional and neither is load-bearing.** A shard that never does any of -this is fully supported. Do part one and skip part two if art is not worth your -time — they share only the tool. +**It is optional and it is not load-bearing.** A shard that never does any of this +is fully supported. -Everything you extract stays **outside the repository**: the converted cliloc -file lives at a path you choose, and `spawnAtlas.art.json` plus `server/uploads/` -are gitignored, so none of it can be committed by accident. +Everything you extract stays **outside the repository**: `spawnAtlas.art.json` and +`server/uploads/` are gitignored, so none of it can be committed by accident. --- ## Part 0 — Get UOFiddler [UOFiddler](https://github.com/polserver/UOFiddler) is the community client-file -editor. We use it because its `Ultima.dll` already contains the cliloc -decompressor, maintained by people who do this for a living. +editor. What is left here uses its GUI to export sprites; the `Ultima.dll` this +guide used to reach for was the cliloc decompressor, and the shard has its own +now. 1. Download the latest release zip from — one asset, named @@ -38,204 +49,30 @@ decompressor, maintained by people who do this for a living. ``` UOFiddler-4.22.2/ - Ultima.dll ← the decompressor (Part 1 needs this path) UoFiddler.exe ← the GUI (Part 2 needs this) + Ultima.dll plugins/ … ``` -3. **Runtime:** UOFiddler 4.22.2 is built for **.NET 10**. Running `UoFiddler.exe` - needs the .NET 10 **Desktop** Runtime (Windows only); loading `Ultima.dll` from - the converter in Part 1 needs the .NET 10 runtime. Install from - . +3. **Runtime:** UOFiddler 4.22.2 is built for **.NET 10**, so running + `UoFiddler.exe` needs the .NET 10 **Desktop** Runtime (Windows only). Install + from . ### Finding your client files -The cliloc file is in your **UO client installation directory**, not in your -ServUO tree — the shard server has no copy of it. Look for `Cliloc.enu` (English; -the other seven are `chs`, `cht`, `deu`, `esp`, `fra`, `jpn`, `kor`) beside -`art.mul` / `artLegacyMUL.uop`. The EA Classic Client's default location is: +The art files are in your **UO client installation directory**. Look for +`art.mul` / `artLegacyMUL.uop` and the `anim*.mul` set. The EA Classic Client's +default location is: ``` C:\Program Files (x86)\Electronic Arts\Ultima Online Classic\ ``` -**If your shard distributes its own patched client to players, use that copy.** -Any cliloc edits you shipped to players are then already in the base table and -you need no overlay for them (see [`CLILOCS.md`](CLILOCS.md) §Shard-added and -shard-edited items). - ---- - -## Part 1 — Convert the cliloc table - -**Goal:** turn the client's compressed `Cliloc.enu` into a file the site can -read, and point the site at it. - -The site cannot read `Cliloc.enu` directly. Every modern client compresses it -(the "Mythic" container), and so does ServUO's own bundled `Ultima.StringList` — -which is why the shard cannot supply names on our behalf either. The full -reasoning is in [`CLILOCS.md`](CLILOCS.md) §Why the operator has to convert the -file; this section is just the procedure. - -Two routes. **The bundled tool is the recommended one** — the GUI export needs a -fixup step, described below. - -### Route A — the bundled converter (recommended) - -Needs a .NET SDK (any version 8 or newer — the project targets `net8.0` and rolls -forward, so whatever you have works) **plus** the .NET 10 runtime from Part 0, -which is what actually loads `Ultima.dll`. - -```bash -cd website/server/tools/cliloc-export -dotnet build -c Release - -# plain binary — recommended, exact -dotnet run -c Release -- \ - "/path/to/UOFiddler-4.22.2/Ultima.dll" \ - "/path/to/UO client/Cliloc.enu" \ - /srv/uo-data/clilocs.plain - -# or tab-delimited text, if you want to eyeball or hand-edit it -dotnet run -c Release -- \ - "/path/to/UOFiddler-4.22.2/Ultima.dll" \ - "/path/to/UO client/Cliloc.enu" \ - /srv/uo-data/clilocs.tsv --tsv -``` - -Expected output for a stock English client: - -``` -wrote 123490 entries to /srv/uo-data/clilocs.plain (maxTextBytes=12150, skippedOversize=0) -``` - -**Sanity-check that number.** A stock `Cliloc.enu` is ~123,000 entries. A few -hundred means it read something else and you should not ship the result. The -tool exits non-zero and says `no entries were written — is that a cliloc file?` -when it gets nothing at all. - -The conversion runs on whatever machine has the client (usually Windows), and the -site reads the output wherever it runs — so **copy the output file to the server** -if those are different machines. It is a single self-contained file (~5 MB); the -`--tsv` form is larger but diff-able. - -
-Errors you may hit - -| Message | Cause | -|---|---| -| `Ultima.StringList not found — is that really UOFiddler's Ultima.dll?` | First argument points at some other `Ultima.dll` (ServUO ships one too — it is **not** the same assembly and cannot do this) | -| `You must install .NET to run this application` | Missing the .NET 10 runtime from Part 0 step 3 | -| `Unexpected Ultima.StringList API` | UOFiddler older than 4.21 | -| `usage: clilocexport …` | Fewer than three arguments | - -
- -### Route B — the UOFiddler GUI - -Use this if you would rather not install a .NET SDK. **It needs one extra step**, -so do not skip the fixup. - -1. Launch `UoFiddler.exe` and point it at your client directory when it asks - (or **Options → Path Settings**). -2. Open the **Cliloc** tab and use its **export to CSV** action. -3. It writes `CliLoc.csv` to UOFiddler's configured output path, in **three** - columns with a header row: - - ``` - Number;Text;Flag - 1023721;quarter staff;0 - ``` - -4. **Strip the trailing flag column.** The site's text parser reads - `numbertext`, so that third field is otherwise absorbed into the name - and every item on the site renders as `quarter staff;0`. - - ```bash - sed -E 's/;[0-9]+$//' CliLoc.csv > clilocs.csv - ``` - - ```powershell - Get-Content CliLoc.csv | - ForEach-Object { $_ -replace ';\d+$','' } | - Set-Content -Encoding utf8 clilocs.csv - ``` - - The header row needs no removal — a line whose first field is not an integer - is skipped. Blank entries (`1005008;`) survive the fixup correctly and are - dropped at import, as intended. - -5. Copy `clilocs.csv` to the server. - -**Why the fixup is not just done for us:** the parser already handles -`number,flag,text` — the flag in the *middle*, which is what several exports -emit. UOFiddler puts it at the *end*, where it is indistinguishable from a name -that genuinely ends in `;0`. One `sed` on the operator's side beats a parser -heuristic that would corrupt real names. - -### Point the site at it - -Two ways, the setting winning over the environment: - -| Where | How | -|---|---| -| **Admin → Shard → cliloc path** | Takes effect on the next refresh, no redeploy | -| `UO_CLIENT_PATH` env var | The deploy-time default | - -The value may be **the file itself or a directory to search** — both are natural -answers to "where is it", and overlays are picked up either way. - -Setting the path deliberately does **not** import as a side effect. Click -**Import** (or `POST /api/v1/admin/shard/clilocs/import`) to load it. - -### Verify - -`GET /api/v1/admin/shard/clilocs`, or the Admin → Shard panel, reports what each -source contributed: - -```json -"sources": [ - { "label": "clilocs.plain", "kind": "base", "entries": 123490, "added": 123490, "overrode": 0 } -] -``` - -Roughly **67,500 rows stored** from a stock table is correct — about half a -cliloc table is empty strings for ids the client reserves and never uses. - -Then load any character sheet with equipment: items should show names rather than -`id 1023721`. - -
-What a refusal means - -A bad file answers `200` with a `status` and a named reason, not a `500` — you -need to be told *which file* to fix. - -| `code` | Meaning | -|---|---| -| `COMPRESSED` | You pointed at the raw client `Cliloc.enu`. Convert it — this whole page. | -| `TRUNCATED` | Half-copied file. Re-copy; the loaded table is untouched. | -| `EMPTY` | A text source with no parseable rows — the file is named in the reason. | -| `status: needsReview` + `missingSources` | A previously-loaded source has vanished (unmounted volume? deliberate deletion?). Nothing changes until you re-import with `{ "approve": true }`. | - -
- -### Custom items — do *not* re-export for these - -Shard-added items carry ids no client table has. Drop a small delimited file in a -`custom/` directory beside the base file and re-import: - -``` -/srv/uo-data/ - clilocs.plain ← base, from this guide - custom/ - 01-uomysticmoon.tsv ← your additions and overrides -``` - -Files are read in sorted order and **later sources win**, so an overlay both adds -new ids and overrides stock ones you have re-purposed. **Adding one item never -means re-exporting a 5 MB client file.** Details in [`CLILOCS.md`](CLILOCS.md). +**If your shard distributes its own patched client to players, use that copy** — +it is what your players actually see. (It is also the copy the shard itself reads +from, since a ServUO server resolves `Config/DataPath.cfg` into its own client +path at boot, which is the premise the Asset Bridge is built on.) --- @@ -246,7 +83,8 @@ functional as text, and `art` is NULL on every fresh import. **This project ships no art and no art-extraction tooling, and never will.** -1. In `UoFiddler.exe` (paths configured as in Route B step 1), open the +1. Open `UoFiddler.exe` and point it at your client directory + (**Settings → Paths**), then open the **Animations** tab for creature sprites — or **Items** for object art — find the creature, and export as PNG. Right-click an entry for its export options, or use the tab's *Export All* action for a batch. (4.22.2 added an export