RunicGateway/docs
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(link): clilocs come over the bridge now, and UOFiddler's first job is gone

Browse Source 
Phase 2 of the Asset Bridge is built, so the documentation stops telling an
operator to install a GUI tool.

`v8.md` gains §9.1 and §9.2 — what the port cost, what it measured, and where the
base table comes from now. The measurement worth keeping: **67,496 rows in
290 ms**, which is exactly what UOFiddler's own `Ultima.dll` produced from this
same client through the converter this phase deletes. An independent
implementation agreeing to the row is the strongest check available that a format
decoder is correct, and it is not something a subtly-wrong one produces.

§17 records the four shapes the org lead settled before any of it was written.
Two departed from the recommendation: **the bridge always wins** (no source
setting — there is no version of that question an operator benefits from
answering) and **import is admin-triggered** (boot does not call the shard at
all).

`CLILOCS.md` is rewritten around that: where the table comes from, what arrives
and in how many pieces, the refusals — including the two the file pipeline had no
equivalent of (a client patched mid-import, and the base's exemption from the
vanished-source rule, which exists so an upgraded install is not asked to approve
a change the upgrade itself made).

`UOFIDDLER.md` loses Part 1 entirely rather than having it rewritten. What is
left is creature art, which phase 5 takes, after which the page goes away. `v3.md`
§8.6 keeps its reasoning with a note saying what superseded it, because the
argument for why the manual step existed is still the argument for why this was
worth building.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
This commit is contained in:
claude
2026-09-10 11:13:56 -05:00
parent df9b3fd990
commit bbd69a8e2e
6 changed files with 312 additions and 336 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@@ -37,7 +37,7 @@ sidecar as a service, and hands you the values the website needs.
| [MODERATION_APPEALS.md](website/MODERATION_APPEALS.md) | Moderation actions, content reports and the appeals flow |
| [SPAWN_ATLAS.md](website/SPAWN_ATLAS.md) | The bestiary / spawn atlas: what the shard contains, parsed from its own ServUO tree |
| [CLILOCS.md](website/CLILOCS.md) | UO's id → name table: converting one from your client so items have names |
| [UOFIDDLER.md](website/UOFIDDLER.md) | **Operator runbook** — step-by-step extraction from your own UO client (cliloc table, creature art) |
| [UOFIDDLER.md](website/UOFIDDLER.md) | **Operator runbook** — extracting creature art from your own UO client. Shrinking: the cliloc half moved to the [Asset Bridge](link/v8.md) in protocol 8 |
| [MARKETPLACE.md](website/MARKETPLACE.md) | The player-vendor index: how it is gathered, what it costs, how to tune it |
| [website-README.md](website/website-README.md) | Snapshot of the website repo's README (setup/run reference) |
| [test-plan.md](website/test-plan.md) | The website's test strategy and harness |

5
link/v3.md
View File

@@ -823,6 +823,11 @@ shapes are the plain binary layout and a `number<TAB|,|;>text` export; the site
and writes the plain form. A shard that never converts is fully supported — names render as ids,
exactly as before.
> **Superseded by [`v8.md`](v8.md) §9 (protocol 8, phase 2).** The decompressor is in the overlay
> now, the shard reads its own client, and `server/tools/cliloc-export/` has been deleted. The
> paragraph above is kept as the record of why the manual step existed; everything else in this
> section — the overlay set, the hash gate, the refusals — still describes what runs.
**Shards edit items and add new ones**, and those carry ids no stock client table has — so this reads
a **set** of sources, not one file, hash-gated together and re-read on every boot exactly as §6 reads
the ServUO tree: a base (the converted client table) plus every overlay under `custom/`, later

111
link/v8.md
View File

@@ -714,6 +714,84 @@ Protocol 8 writes rather than calls (§4) — and from that point:
- `website/server/tools/cliloc-export/` is retired, and `UOFIDDLER.md` is deleted rather than
rewritten.
### 9.1 What phase 2 built, and what the port cost
**Built 2026-09-10.** `BridgeCliloc.cs` in the overlay, `GET /cliloc` in the sidecar, and a paging
walk in module-uo that merges the shard's table under the `custom/` overlays.
The port is `MythicDecompress` + `MoveToFront` rewritten against plain arrays: the upstream is
`Span<T>` / `ArrayPool<T>` / `BinaryPrimitives` code and ServUO targets `net48`, which has none of
them without a package this tree does not vendor. The algorithm is unchanged, **including the parts
that read oddly** — the three-region count/cursor/end table and the symbol-table shifts are
upstream's, deliberately, because a tidier rewrite of somebody else's format decoder is a chance to
be subtly wrong in a way that produces plausible text.
Two bounds checks were added, and they are the only behavioural change: the upstream indexes its
payload without checking, which is safe for a file the client wrote and is not safe for a file this
shard was handed. A truncated container now reports `UNREADABLE` with the byte it wanted, instead of
throwing an `IndexOutOfRangeException` from inside a decoder.
**Measured on this machine's stock client:** 4,989,921 bytes read, decompressed and parsed in
**290 ms**, yielding **67,496** non-blank rows in id order, ~5.4 MB on the wire, ~11 pages.
That 67,496 is the acceptance test, and it is worth saying why it is a strong one: it is the number
[`../website/CLILOCS.md`](../website/CLILOCS.md) already recorded for this same client, measured
through **UOFiddler's own `Ultima.dll`** by the converter this phase deletes. An independent
implementation agreeing to the row is not something a subtly-wrong decoder produces. Checked
alongside it: zero U+FFFD, so the UTF-8 survived; 696 rows with non-ASCII text, spot-checked as
correct curly quotes; the longest row is a 12,149-character EULA, which is also why the record
length is read **unsigned** (a signed read turns anything over 32 KB negative).
**The shard drops blanks before they reach the wire.** ~56,000 of the 123,490 entries are empty
strings the client reserves, the website discards them at import anyway, and sending them would
double the transfer for data thrown away on arrival.
**Language is a parameter (`enu` by default) but not a free one.** `Ultima.Files` resolves only the
names in its own table, which for clilocs is `enu`, `deu`, `custom1` and `custom2`; anything else is
refused `NOT_FOUND` rather than answered with an empty table. `custom1`/`custom2` are the
client-side custom cliloc files a shard ships to its players — readable here, and deliberately not
wired into the website's import, because `custom/` on the site is the supported answer for
shard-added ids.
**`assets.error` gained a `code`.** Phase 1 chose between 403 and 400 by looking for the word
"disabled" in the operator-facing sentence, which makes prose load-bearing; the codes are
`DISABLED` (403), `NOT_FOUND` (404), `UNREADABLE` (422), `UNAVAILABLE` (503) and `BAD_REQUEST`
(400). The substring check survives as a fallback, because an overlay and a sidecar are deployed
separately and a phase-1 shard must keep its 403.
### 9.2 Where the base comes from now, and when it is read
**The shard wins whenever uo-link is configured and enabled.** No mode setting: there is no version
of that question an operator benefits from answering. A file on disk remains the source only where
there is no shard link, plus a one-off explicit path — the deprecated pipeline, kept for installs
with no bridge and for development.
**Boot no longer imports on the bridge.** The file path could hash 5 MB locally and skip in 14 ms; a
shard round trip in the boot sequence would be spent answering "no" on every restart but the one
after a client patch — and patching a client is an operator action, so importing became one:
Admin → Shard → Import. Whatever table is loaded keeps serving until then.
Three things about the walk are worth recording because each is a way a shard can hand back a table
that *looks* complete:
- **Only `cut: "end"` finishes it** (§3.4). A short page can equally be a spent budget.
- **The cursor must advance**, or the walk stops rather than spinning.
- **Every page echoes the source's size and mtime.** A client patched mid-import is refused
outright (`SOURCE_CHANGED`) rather than stitched together — half of what arrived came from a file
that no longer exists and nothing later can tell which half.
**The base is exempt from the vanished-source rule**, and that is an upgrade detail rather than a
design preference: an install that used the file pipeline carries its base file's label in the
stored fingerprint, and on the bridge that label is *supposed* to disappear. Counting it as vanished
would make the first import after the upgrade demand approval for a change the upgrade itself made.
Overlays keep the rule in full.
**module-uo's protocol pin moved 7 → 8 in this phase**, which is the third declaration site §15
names and the one nothing enforces. Phase 1 moved the sidecar and the overlay together because the
installer refuses to pair a mismatched bundle; this one had to be moved by hand, in the phase that
first calls a protocol-8 route — the same trap that left the pin at 5 for two phases of the Event
System while every REST call was answered `409`.
**What survives untouched is the `custom/` overlay mechanism.** Shard-added items carry cliloc ids
no client table has, and ServUO has no server-side notion of a custom cliloc — that is a real gap
in the *game*, not an artefact of the manual pipeline, and `CLILOCS.md`'s reasoning for it stands.
@@ -846,7 +924,7 @@ new pipe.
| `assets.manifest` | `assets.manifest.ok` | Stage 2: `[{key, sha256, bytes}]`, paged | phase 3 |
| `assets.fetch` | `assets.fetch.ok` | Content for an explicit key list, paged | phase 3 |
| `assets.bodies` | `assets.bodies.ok` | Slug → body id (§8, Core thread) | phase 3 |
| `cliloc.table` | `cliloc.table.ok` | The converted table, paged | phase 2 |
| `cliloc.table` | `cliloc.table.ok` | The decompressed table, paged (`?lang=`, `?cursor=`) | **phase 2** |
| `tree.manifest` / `tree.fetch` | `.ok` | §10, the ServUO tree files | phase 7 |
Every one of them is refused outright when `Bridge.AssetsEnabled` is off, and every one of them
@@ -854,9 +932,11 @@ requires a `reqId` — a request without one is refused rather than answered, be
reply is by definition an event, and §3.1 is the reason none of this may ever be one.
**Sidecar REST** mirrors those one for one under `/assets/*`, `/cliloc`, `/tree/*`, carrying
`X-UOLink-Version: 8` and forwarding verbatim. `GET /assets/sources` exists as of phase 1; its
responder maps `bridge.busy` to **425** (flow control, and the ordinary answer mid-import rather
than a rare one) and a disabled plane to **403** (an operator's refusal, not a malformed request).
`X-UOLink-Version: 8` and forwarding verbatim. `GET /assets/sources` exists as of phase 1 and
`GET /cliloc` as of phase 2; their responder maps `bridge.busy` to **425** (flow control, and the
ordinary answer mid-import rather than a rare one) and reads the refusal's `code` for the rest —
**403** disabled, **404** no such file, **422** a file the shard cannot decode, **503** a shard that
cannot do this now (§9.1).
**Website admin** (`Admin → Shard`, admin-only): status, **Import**, **Update**, approve/reject for
a vanished key, and the existing path settings. Every action to the admin activity log, as
@@ -875,7 +955,7 @@ disagree, so a split bump means the next bundle silently fails to compose.
| `servuo-plugins/` | Extraction over ServUO's own `Ultima` (§4), the cliloc decompressor (§9), body resolution (§8), the request handlers, `overlay.toml` |
| `link/` | Six command families forwarded, the REST surface, **the inbound line cap (§3.3)**, `PROTOCOL_VERSION` |
| `module-uo/` | Client calls, asset store, the atlas source backend (§10), cliloc ingest, admin surface |
| `website/` | None expected — `ctx.uploads` already suffices (§12) |
| `website/` | Almost none — `ctx.uploads` already suffices (§12). Phase 2 deleted `server/tools/cliloc-export/`, the converter this protocol retires |
| `docs/` | This file; rewrite `CLILOCS.md` §Converting and `SPAWN_ATLAS.md` §Artwork + §Configuring; **delete `UOFIDDLER.md`**; add the libgdiplus prerequisite to `SHARD_PREREQS.md` (§4.4) |
| `installer/` | A `doctor` check for libgdiplus on Linux hosts (§4.4). Bundle pairing already enforces §15 |
| `android-app/` | Consumes images by URL; no parity gate expected until a screen shows one |
@@ -889,7 +969,7 @@ disagree, so a split bump means the next bundle silently fails to compose.
|---|---|---|
| 0 | **DONE 2026-09-10.** Spike: the vendored decoders driven from inside a running ServUO, over a client broken 21 ways. §4 stands; the finding was 22,102 wrong pictures on a *stock* client, and the validator that answers them (§4.5-§4.7) | servuo-plugins |
| 1 | **DONE 2026-09-10.** The transport: `assets.sources`, the single-slot gate (§3.2), the paging envelope (§3.4), the sidecar line cap (§3.3), `EXTRACTOR_VERSION`, `NO_IMAGING`, protocol 7→8. Plus §4.5's validator promoted into the overlay and extended to animations — which found **357 wrong pictures in the body catalogue on a stock client** and cut it from 1,144 to 787 (§4.8) | servuo-plugins, link |
| 2 | Clilocs end to end; retire the converter and `UOFIDDLER.md` §Part 1 | all |
| 2 | **DONE 2026-09-10.** Clilocs end to end (§9.1, §9.2): the Mythic decompressor ported into the overlay, `cliloc.table` + `GET /cliloc`, the paging walk and the source switch on the website, module-uo's protocol pin 7→8. `cliloc-export/` deleted and `UOFIDDLER.md` §Part 1 with it. **67,496 rows, 290 ms, ~11 pages** — the same count UOFiddler's own DLL produced from this client | all |
| 3 | Body resolution (§8) + the **787**-body catalogue (§4.8); `shard_spawn_creatures.art` filled | servuo-plugins, module-uo |
| 4 | **The UOP animation decoder (§4.3)** — the **eight** player bodies the legacy path cannot reach (§4.8 moved the elf ghosts across), the never-sweep rule already enforced in code by phase 1's `ResolveAnimation` | servuo-plugins |
| 5 | Item statics and land on demand, hued keys, the cache | servuo-plugins, module-uo |
@@ -953,7 +1033,24 @@ in the document.
sidecar or left to the website as a convention — the bound belongs where the memory is. And one
**paging envelope** (`more` / `cursor` / `cut`) is defined now, with `assets.sources` as its
first user, rather than left for whichever family pages first to invent.
7. **§4.8: the catalogue is 787 bodies, not 1,144 — measured 2026-09-10, phase 1.** Recorded here
7. **§9: the cliloc pipeline's four shapes — settled 2026-09-10, phase 2.** All four were put to
the org lead before any of it was written, and two of them departed from the recommendation:
- **The wire carries rows, not bytes.** `[{n, f, t}]` inside the paging envelope, rather than a
reconstituted plain-binary file in base64. The container decompresses into *records*; the
plain layout is a file shape that would have had to be synthesised on the shard and re-parsed
on the website, at a 33% base64 premium, to reach the same place.
- **The shard omits blank entries** — ~56,000 of 123,490 — because the website discards them at
import anyway. The transfer halves and nothing observable changes.
- **The bridge always wins; the file upload is deprecated** (org lead, departing from the
proposed `auto`/`bridge`/`file` setting). There is no version of "which source?" an operator
benefits from answering, so there is no setting to answer it with. A file remains the source
only where there is no shard link, plus a one-off explicit path.
- **Import is admin-triggered on the bridge** (org lead, departing from the proposed boot
refresh). Boot does not call the shard at all — see §9.2 for why that is the right trade and
what it costs.
8. **§4.8: the catalogue is 787 bodies, not 1,144 — measured 2026-09-10, phase 1.** Recorded here
because it changes §11's sizing, phase 3's scope and phase 4's, and because of *how* it was
found: the animation path has §4.5's shared-buffer defect too, and 357 ids with no art were
returning the previously-decoded body's bitmap. The elf ghosts moved from "decodes" to "no art"

2
modules/uo/README.md
View File

@@ -21,7 +21,7 @@ these routes *mean* are the ones that already existed and did not move:
| [`SPAWN_ATLAS.md`](../../website/SPAWN_ATLAS.md) | The bestiary / spawn atlas, parsed from the shard's own ServUO tree |
| [`MARKETPLACE.md`](../../website/MARKETPLACE.md) | The player-vendor index |
| [`CLILOCS.md`](../../website/CLILOCS.md) | UO's id → name table |
| [`UOFIDDLER.md`](../../website/UOFIDDLER.md) | Operator runbook for extracting the cliloc table and creature art |
| [`UOFIDDLER.md`](../../website/UOFIDDLER.md) | Operator runbook for extracting creature art. The cliloc table now arrives over the bridge ([`link/v8.md`](../../link/v8.md) §9) |
| [`../../link/PLAN.md`](../../link/PLAN.md), [`../../link/INTEGRATION.md`](../../link/INTEGRATION.md) | The wire protocol this module speaks to the sidecar |
---

290
website/CLILOCS.md
View File

@@ -1,7 +1,8 @@
# Cliloc table (item and title names)
**Status:** Complete on `edge` — website [#115](https://gitea.whitlocktech.com/RunicGateway/website/pulls/115), docs [#70](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/70).
**Design:** [`docs/link/v3.md` §8.6](../link/v3.md) — Protocol 3.0, the dependency Part B/3 was sequenced behind.
**Status:** Complete. The base table now arrives **over the bridge** — protocol 8, phase 2.
**Design:** [`docs/link/v3.md` §8.6](../link/v3.md) (the table itself, Protocol 3.0) and
[`docs/link/v8.md` §9](../link/v8.md) (the Asset Bridge, which retired the manual conversion).
A "cliloc" is UO's localization table: an integer id mapped to a display string.
**Items on the wire carry a `LabelNumber`, not a name.** The bridge has always
@@ -12,86 +13,70 @@ render `id 1023721` where the game renders **"quarter staff"**.
The number was never the missing piece. The table was.
## Why the operator has to convert the file
## Where the table comes from
This is the awkward part, and it is not avoidable:
**The shard reads its own client.** A ServUO server cannot boot without a UO
client — `Config/DataPath.cfg` resolves into `Core.DataDirectories` at run time —
so the file this table is made of is already sitting on the shard host. Since
protocol 8 the plugin decompresses it and serves it over the bridge, and the site
imports it like any other shard read:
**Every current UO client ships its cliloc files compressed.** All eight
`Cliloc.*` files in a modern client (`chs`, `cht`, `deu`, `enu`, `esp`, `fra`,
`jpn`, `kor`) begin with a DWORD whose high byte is `0x8E` — the "Mythic"
compressed container. The plain layout this site parses is what those files
looked like *before* that change.
Decompressing it means an inverse-BWT coder with a frequency header — a few
hundred lines of bit-level work whose failure mode is plausible-looking garbage
rather than an error. The site has no business carrying that at runtime.
Two facts make the alternatives worse, not better:
- **ServUO cannot read it either.** Its bundled `Ultima.StringList` implements
only the plain layout, so on a modern client `VendorSearch.StringList` is null
and `VendorSearch.GetItemName` returns `item.Name` — usually nothing. The
shard cannot supply names on our behalf; the in-game Vendor Search gump has the
same gap.
- **Nothing client-derived may be committed.** UO's strings are EA's. The repo
ships no string table for the same reason it ships no artwork and no map
snapshot — see [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md).
So the conversion happens **once, on the operator's machine, against their own
client**, and the site reads the result from a path it is given. A shard that
never does this is in a fully supported state: names render as ids, exactly as
they did before the table existed.
## Converting
> **Step-by-step operator instructions — where to get UOFiddler, where your
> client files are, and how to verify the import — are in
> [`UOFIDDLER.md`](UOFIDDLER.md).** This section covers the formats and the
> reasoning behind them.
Either format below is accepted; the site sniffs which one it was handed.
| Format | Fidelity | Notes |
|---|---|---|
| **Plain binary** (recommended) | Exact | 6-byte header, then `{int32 number, byte flag, uint16 length, UTF-8}` records |
| Delimited text | Loses leading/trailing whitespace | `number<TAB\|,\|;>text` per line; a header row, blank lines and `#` comments are ignored |
The whitespace caveat is real but cosmetic: ~1,300 of the 123,490 entries in a
stock `Cliloc.enu` are label prefixes like `"max = "` whose trailing space is
meaningful when the client concatenates a value onto them. Nothing on this site
concatenates, and every consumer passes through `displayText()`, which trims.
### Using the bundled tool
`server/tools/cliloc-export/` is a small .NET console app that drives
[UOFiddler](https://github.com/polserver/UOFiddler)'s `Ultima.dll` — the
decompressor that already exists and is already maintained — and writes the plain
format. It loads that DLL **reflectively** so it compiles against any SDK, and it
writes the records by hand because UOFiddler's own `SaveStringList` *re-compresses*
on save (its purpose is round-tripping a file back into the client, so its output
is byte-identical to its input — a trap worth knowing about).
```bash
cd website/server/tools/cliloc-export
dotnet build -c Release
# binary (recommended)
dotnet run -- "<UOFiddler>/Ultima.dll" "<UO client>/Cliloc.enu" /srv/uo-data/clilocs.plain
# or tab-delimited
dotnet run -- "<UOFiddler>/Ultima.dll" "<UO client>/Cliloc.enu" /srv/uo-data/clilocs.tsv --tsv
```
ServUO shard ──`cliloc.table`──▶ uo-link sidecar ──`GET /cliloc`──▶ website
reads Cliloc.enu, forwards, keeps merges overlays,
decompresses, pages nothing replaces the table
```
A UOFiddler GUI export works too, but **not unmodified**: its Cliloc tab writes
`Number;Text;Flag` — three columns, the flag *last* — and the parser reads
`number<separator>text`, so the trailing field is absorbed into the name and
every item renders as `quarter staff;0`. Stripping it is one `sed`, given in
[`UOFIDDLER.md`](UOFIDDLER.md) §Route B.
**Why that was worth building.** Every current UO client ships its cliloc files
compressed: all eight `Cliloc.*` files in a modern client (`chs`, `cht`, `deu`,
`enu`, `esp`, `fra`, `jpn`, `kor`) begin with a DWORD whose high byte is `0x8E` —
the "Mythic" container. The plain layout is what those files looked like *before*
that change, and **ServUO's own bundled `Ultima.StringList` cannot read the new
one either**, which is why `VendorSearch.GetItemName` is inert on a modern shard
and the in-game Vendor Search gump has the same gap.
The parser already tolerates `number,flag,text`, with the flag in the *middle*.
It is not extended to cover the trailing form because a final `;0` is
indistinguishable from a name that genuinely ends that way — a heuristic there
would corrupt real names to save the operator one command.
So until protocol 8 an operator had to install UOFiddler, build a converter
against its `Ultima.dll`, run it over their client and copy a 5 MB file to the web
host — every time they patched. The decompressor now lives in the overlay
(`overlay/Scripts/Custom/Bridge/BridgeCliloc.cs`, ported from UOFiddler, which is
Beerware and therefore clean to bring into a GPL tree), so **none of that is a
step any more**.
Two things are unchanged and remain the point:
- **Nothing client-derived is committed.** UO's strings are EA's. The repo ships
no string table for the same reason it ships no artwork and no map snapshot —
see [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md). The extraction happens on the
operator's own host, from their own files, for their own shard.
- **A shard with no table is fully supported.** Names render as ids, exactly as
they did before the table existed.
### What arrives, and in how many pieces
The sidecar's reply timeout is 10 s and its inbound line cap is 1 MiB, so the
table is **paged**: the shard cuts at a 512 KiB byte budget and hands back a
cursor, and the site walks it until a page says `more: false`. Measured on a stock
English client: **67,496 rows in about eleven pages**, decoded on the shard in
**290 ms**.
The rows are `{ n, f, t }` — number, flag, text. **Blanks never leave the shard**:
roughly 56,000 of a stock table's 123,490 entries are empty strings the client
reserves and never uses, the site drops them at import anyway, and sending them
would double the transfer for data that is discarded on arrival.
Only `cut: "end"` means the table finished. A short page can equally mean the byte
budget was spent, and importing a table that stopped early is the one failure that
is invisible downstream — some items named, some not, which is exactly what *no
table* looks like.
### The file pipeline is deprecated, not removed
An install with **no uo-link configured** can still be pointed at a converted file
and works exactly as it did. That path exists for shards with no bridge and for
development without a running ServUO, it accepts the same two formats it always
did (plain binary, or `number<TAB|,|;>text` delimited text), and passing an
explicit path to a refresh still selects it as a one-off. Nothing new should be
built on it.
## Shard-added and shard-edited items
@@ -103,12 +88,16 @@ atlas, which reads `Regions.xml` + `Locations/*.xml` + `Spawns/*.xml` +
```
<cliloc path>/
clilocs.plain ← base: the converted client table
custom/
01-uomysticmoon.tsv ← overlays: shard additions and overrides
02-events.tsv
```
The base is no longer a file in that directory — it comes from the shard — so on
a bridged install the configured path selects **only** where `custom/` is read
from. (Without a shard link, a converted base file sitting beside `custom/` is
still found, which is the deprecated pipeline above.)
Overlays use the same delimited-text format, are read in **sorted order**, and
**later sources win** — so an overlay both *adds* ids the client never had and
*overrides* stock ones the shard has re-purposed. Any `.tsv`, `.csv`, `.txt`,
@@ -116,8 +105,9 @@ Overlays use the same delimited-text format, are read in **sorted order**, and
say) is ignored.
Adding, editing or removing any overlay counts as drift, so a new custom item
needs only a file edit and a restart — or the admin panel's Import button.
**Adding one item never means re-exporting a 5 MB client file.**
needs only a file edit and the admin panel's Import button. **Adding one item
never means re-reading the client table** — though on the bridge that is now
cheap enough not to matter much.
The import result reports what each source contributed, which is how you confirm
an overlay took effect — `overrode: 0` on a file meant to re-label stock items
@@ -125,11 +115,14 @@ says it did not:
```json
"sources": [
{ "label": "clilocs.plain", "kind": "base", "entries": 123490, "added": 123490, "overrode": 0 },
{ "label": "custom/uomysticmoon.tsv", "kind": "custom", "entries": 2, "added": 1, "overrode": 1 }
{ "label": "cliloc.enu", "kind": "shard", "entries": 67496, "added": 67496, "overrode": 0 },
{ "label": "custom/uomysticmoon.tsv", "kind": "custom", "entries": 2, "added": 1, "overrode": 1 }
]
```
`kind` says where a source came from: `shard` over the bridge, `base` a converted
file on disk, `custom` an overlay.
**Why a convention rather than discovery.** Everywhere else this pipeline follows
the shard's own files, but **ServUO has no server-side notion of a custom
cliloc** — they live in the patched client a shard distributes to its players,
@@ -143,28 +136,29 @@ ids and only **37** are absent from the stock client table — tens of entries
against a 67k base, which is what makes an overlay the right shape rather than a
second full table.
## Configuring the path
## Choosing the source
Two ways to point at the sources, the setting winning over the environment:
Nothing to configure: **the shard wins whenever uo-link is configured and
enabled.** There is no mode setting, because there is no version of this question
an operator benefits from answering — a shard that can serve its own client table
is strictly better than a file somebody converted by hand months ago.
| Source | Notes |
The two escape hatches, both deliberate:
| | |
|---|---|
| `cliloc_client_path` setting | Admin-editable (Admin → Shard); takes effect on the next refresh without a redeploy |
| `UO_CLIENT_PATH` env var | The deploy-time default, since the path usually describes a mount the deployment sets up |
| No uo-link configured | The file pipeline, exactly as before |
| An explicit `path` passed to a refresh | A one-off "import from this file", which the shard never overrules |
The value may be **the base file itself or a directory to search**, because both
are natural answers to "where is it". Overlays are read from a `custom/`
directory beside the base **either way** — pointing at a file does not forfeit
them.
A directory is searched case-insensitively (the client writes `Cliloc.enu` on
Windows; the site usually runs on Linux) for, in order: `clilocs.tsv`,
`clilocs.csv`, `clilocs.plain`, `cliloc.plain`, `cliloc.plain.enu`,
`cliloc.enu.plain`, `clilocs.txt`, `cliloc.enu`.
That ordering puts explicitly-converted names first on purpose. Pointing the
setting straight at an unconverted client directory finds `cliloc.enu`, which is
compressed — and the site says so by name rather than failing obscurely:
`cliloc_client_path` (setting, admin-editable) and `UO_CLIENT_PATH` (env
deploy-time default) still name a path, and the setting still wins over the
environment. What that path *means* narrowed: on a bridged install it is where
`custom/` overlays live. Without a link it is also searched for a base file,
case-insensitively (the client writes `Cliloc.enu` on Windows; the site usually
runs on Linux), in order: `clilocs.tsv`, `clilocs.csv`, `clilocs.plain`,
`cliloc.plain`, `cliloc.plain.enu`, `cliloc.enu.plain`, `clilocs.txt`,
`cliloc.enu` — and pointing it straight at an unconverted client directory still
answers by name rather than failing obscurely:
```
status: unavailable
@@ -175,16 +169,25 @@ reason: This is a compressed (Mythic-format) cliloc file, which the site cannot
## Refresh contract
Identical in shape to the spawn atlas, and for the same reasons:
- **It never blocks startup.** No path, an unreadable file, a wrong-format file,
a database error — all caught and logged. The site comes up either way.
- **Hash-gated.** The boot path hashes the file and skips the parse entirely when
it matches what is loaded, which is every restart that did not follow a client
patch. Measured on a stock table: **14 ms** for the no-op, **663 ms** for a full
parse and replace.
- **A `PARSER_VERSION` bump also counts as drift**, so a corrected parse reaches
an install whose client never patches.
- **It never blocks startup**, and on the bridge it never *touches* startup: boot
imports nothing when the shard is the source. A sidecar round trip in the boot
sequence would be spent answering "no" on every restart but the one after a
client patch — and patching a client is an operator action, so importing is an
operator action. **Admin → Shard → Import** is the button. Whatever table is
loaded keeps serving until then.
- **Still hash-gated**, so pressing Import when nothing changed costs one small
call. The gate is the shard's own `assets.sources`: size, mtime and content
hash of `Cliloc.enu` plus the shard's `EXTRACTOR_VERSION`. A `sha256` of `null`
with `hashing: true` means the shard has not computed it yet (it hashes off the
request path, because the art and animation files it also reports are 343 MB)
— that means *ask again*, never *changed*, and the comparison falls back to
(size, mtime) meanwhile.
- **Without a link** the file path is unchanged: boot hashes the local file and
skips the parse when it matches. Measured on a stock table: **14 ms** for the
no-op, **663 ms** for a full parse and replace.
- **A `PARSER_VERSION` bump counts as drift**, and so does an `EXTRACTOR_VERSION`
bump on the shard — same argument at the other end of the wire: a corrected
reader must reach an install whose client never patches.
### Two ways a refresh is refused
@@ -201,7 +204,7 @@ and the rows already loaded are untouched. A malformed overlay names the file it
came from (`custom/broken.tsv: No cliloc entries found…`), because "which of my
six overlay files is broken" is otherwise a guessing game.
**A source that has VANISHED** is the hazard a single file did not have. It
**An OVERLAY that has VANISHED** is the hazard a single file did not have. It
parses perfectly and imports a table quietly missing everything that file
contributed — and an unmounted volume looks exactly like a deliberate deletion
from here. This is the same ambiguity the atlas stages a facet removal for, so it
@@ -209,11 +212,28 @@ is escalated rather than applied:
```
status: needsReview
reason: 1 previously-loaded cliloc source(s) are missing;
reason: 1 previously-loaded cliloc overlay(s) are missing;
the existing table is unchanged
missingSources: ["custom/uomysticmoon.tsv"]
```
**The base is deliberately exempt from that question**, and that is an upgrade
detail worth stating: an install that used the converted-file pipeline carries its
base file's label in the stored fingerprint, and on the bridge that label is
*supposed* to disappear. Counting it as a vanished source would make the first
import after the upgrade demand an approval for a change the upgrade itself made.
**A client patched mid-import** is refused outright rather than staged, because
there is nothing to decide: every page echoes the source file's size and mtime, and
if they move between pages then half of what arrived came from a file that no
longer exists and nothing later can tell which half.
```
status: unavailable
code: SOURCE_CHANGED
reason: The shard's cliloc file changed while it was being read; nothing was imported
```
`status()` reports `missingSources` too, so the panel can show it before anyone
clicks Import. An admin accepts it by re-running the import with
`{ "approve": true }`.
@@ -229,15 +249,20 @@ admin was already going to run.
| | |
|---|---|
| Parsed from a stock `Cliloc.enu` | **123,490** entries |
| Entries in a stock `Cliloc.enu` | **123,490** |
| Of those, empty strings | **55,994** (ids the client reserves and never uses) |
| Stored in `shard_clilocs` | **67,496** |
Blank entries are dropped at import. A row resolving to no name is
indistinguishable from no row at all to every caller, and dropping them makes the
binary and text imports converge on **identical** content — the binary format
carries the blanks explicitly and a text export may or may not, depending on the
tool. Verified: both formats import to the same 67,496 rows with the same keys.
Blank entries are dropped, and since protocol 8 they are dropped **on the shard**,
before they reach the wire. A row resolving to no name is indistinguishable from
no row at all to every caller, so sending 56,000 of them would double the
transfer for data discarded on arrival. The site still filters at import, because
an overlay file can carry one and because the file pipeline still exists.
That the shard's own decoder lands on **exactly 67,496** is also the strongest
check there is that the ported decompressor is correct: the number was measured
first through UOFiddler's `Ultima.dll` against this same client, by an entirely
different implementation.
`text` is `TEXT`, not `VARCHAR`: the long property descriptions reach 12 KB, and
silently truncating them would be worse than storing them. The index that matters
@@ -297,13 +322,24 @@ All admin-only, alongside the atlas under Admin → Shard:
| Route | Purpose |
|---|---|
| `GET /api/v1/admin/shard/clilocs` | Sources found, what each contributed at the last import, readability, drift, entry count, `missingSources` |
| `POST /api/v1/admin/shard/clilocs/import` | Reload after a client patch or an overlay edit; `{ "force": true }` reimports an unchanged set, `{ "approve": true }` accepts a vanished source |
| `PUT /api/v1/admin/shard/clilocs/path` | Set the path; blank disables resolution |
| `GET /api/v1/admin/shard/clilocs` | Which source is in use (`bridge` / `file`), the shard's client-file fingerprint, what each source contributed at the last import, drift, entry count, `missingSources` |
| `POST /api/v1/admin/shard/clilocs/import` | Reload after a client patch or an overlay edit; `{ "force": true }` reimports an unchanged set, `{ "approve": true }` accepts a vanished overlay |
| `PUT /api/v1/admin/shard/clilocs/path` | Set the overlay path (and, with no shard link, the base file's); blank clears it |
A refresh **result is not an exception**: a missing file, or the likely mistake of
pointing at the client's own compressed `Cliloc.enu`, answers `200` with
`status: "unavailable"` and a reason. A `500` would say only "something broke";
the operator needs to be told which file to convert. Setting the path
deliberately does **not** import as a side effect — the response carries the
refreshed status so the panel can offer that as the next step.
**Import is now the only thing that refreshes the table on a bridged install**,
since boot no longer asks the shard. It is the button an operator presses after
patching their client.
A refresh **result is not an exception**, and protocol 8 widened the set of things
that covers: a shard that is down, an asset plane the operator has switched off
(`Bridge.AssetsEnabled`), a client with no cliloc file, a client patched halfway
through the import, plus everything the file pipeline could already report. Each
answers `200` with `status: "unavailable"` and a reason naming what to fix. A
`500` would say only "something broke". Setting the path deliberately does **not**
import as a side effect — the response carries the refreshed status so the panel
can offer that as the next step.
The sidecar's own statuses are worth knowing when reading a log: **425** is the
shard saying it is busy with another asset request (flow control, and the ordinary
answer mid-import — the site retries), **403** the asset plane switched off, **404**
a client with no such file, **422** a file it has and cannot decode.

238
website/UOFIDDLER.md
View File

@@ -1,34 +1,45 @@
# Extracting from your own UO client (UOFiddler)
**Audience:** the shard operator, once, at setup time.
**Related:** [`CLILOCS.md`](CLILOCS.md) (why the cliloc conversion is unavoidable),
[`SPAWN_ATLAS.md`](SPAWN_ATLAS.md) (where creature art fits).
**Related:** [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md) (where creature art fits),
[`../link/v8.md`](../link/v8.md) (the Asset Bridge, which is replacing this page).
Two features read data that **only exists inside a UO client**, and a UO client's
files are EA's, not ours to redistribute. So neither this repo nor any image we
publish can ship them — the operator extracts from **their own** client, once,
> ### The cliloc conversion is gone — you no longer do this by hand
>
> **Protocol 8, phase 2.** The shard reads its own client's `Cliloc.enu` and
> serves the table over the bridge, so there is nothing to install, convert or
> copy. Press **Admin → Shard → Import** after you patch your client and that is
> the whole procedure; see [`CLILOCS.md`](CLILOCS.md).
>
> Part 1 of this guide has been deleted rather than rewritten. If you already
> have a converted file it keeps working on an install with no uo-link
> configured, but nobody should make a new one.
>
> **Creature art is next** (phase 5), after which this page goes away entirely.
What remains here reads data that **only exists inside a UO client**, and a UO
client's files are EA's, not ours to redistribute. So neither this repo nor any
image we publish can ship them — the operator extracts from **their own** client
and points the site at the result.
| Feature | What it needs | Required? | Without it |
|---|---|---|---|
| **Item / title names** ([`CLILOCS.md`](CLILOCS.md)) | `Cliloc.enu`, converted | No | Names render as raw ids — `id 1023721` instead of *quarter staff* |
| **Creature art** ([`SPAWN_ATLAS.md`](SPAWN_ATLAS.md)) | Sprites from `.mul`/`.uop` | No | Atlas pages render as text, which is the normal state |
**Both are optional and neither is load-bearing.** A shard that never does any of
this is fully supported. Do part one and skip part two if art is not worth your
time — they share only the tool.
**It is optional and it is not load-bearing.** A shard that never does any of this
is fully supported.
Everything you extract stays **outside the repository**: the converted cliloc
file lives at a path you choose, and `spawnAtlas.art.json` plus `server/uploads/`
are gitignored, so none of it can be committed by accident.
Everything you extract stays **outside the repository**: `spawnAtlas.art.json` and
`server/uploads/` are gitignored, so none of it can be committed by accident.
---
## Part 0 — Get UOFiddler
[UOFiddler](https://github.com/polserver/UOFiddler) is the community client-file
editor. We use it because its `Ultima.dll` already contains the cliloc
decompressor, maintained by people who do this for a living.
editor. What is left here uses its GUI to export sprites; the `Ultima.dll` this
guide used to reach for was the cliloc decompressor, and the shard has its own
now.
1. Download the latest release zip from
<https://github.com/polserver/UOFiddler/releases/latest> — one asset, named
@@ -38,204 +49,30 @@ decompressor, maintained by people who do this for a living.
```
UOFiddler-4.22.2/
Ultima.dll ← the decompressor (Part 1 needs this path)
UoFiddler.exe ← the GUI (Part 2 needs this)
Ultima.dll
plugins/
…
```
3. **Runtime:** UOFiddler 4.22.2 is built for **.NET 10**. Running `UoFiddler.exe`
needs the .NET 10 **Desktop** Runtime (Windows only); loading `Ultima.dll` from
the converter in Part 1 needs the .NET 10 runtime. Install from
<https://dotnet.microsoft.com/download/dotnet/10.0>.
3. **Runtime:** UOFiddler 4.22.2 is built for **.NET 10**, so running
`UoFiddler.exe` needs the .NET 10 **Desktop** Runtime (Windows only). Install
from <https://dotnet.microsoft.com/download/dotnet/10.0>.
### Finding your client files
The cliloc file is in your **UO client installation directory**, not in your
ServUO tree — the shard server has no copy of it. Look for `Cliloc.enu` (English;
the other seven are `chs`, `cht`, `deu`, `esp`, `fra`, `jpn`, `kor`) beside
`art.mul` / `artLegacyMUL.uop`. The EA Classic Client's default location is:
The art files are in your **UO client installation directory**. Look for
`art.mul` / `artLegacyMUL.uop` and the `anim*.mul` set. The EA Classic Client's
default location is:
```
C:\Program Files (x86)\Electronic Arts\Ultima Online Classic\
```
**If your shard distributes its own patched client to players, use that copy.**
Any cliloc edits you shipped to players are then already in the base table and
you need no overlay for them (see [`CLILOCS.md`](CLILOCS.md) §Shard-added and
shard-edited items).
---
## Part 1 — Convert the cliloc table
**Goal:** turn the client's compressed `Cliloc.enu` into a file the site can
read, and point the site at it.
The site cannot read `Cliloc.enu` directly. Every modern client compresses it
(the "Mythic" container), and so does ServUO's own bundled `Ultima.StringList` —
which is why the shard cannot supply names on our behalf either. The full
reasoning is in [`CLILOCS.md`](CLILOCS.md) §Why the operator has to convert the
file; this section is just the procedure.
Two routes. **The bundled tool is the recommended one** — the GUI export needs a
fixup step, described below.
### Route A — the bundled converter (recommended)
Needs a .NET SDK (any version 8 or newer — the project targets `net8.0` and rolls
forward, so whatever you have works) **plus** the .NET 10 runtime from Part 0,
which is what actually loads `Ultima.dll`.
```bash
cd website/server/tools/cliloc-export
dotnet build -c Release
# plain binary — recommended, exact
dotnet run -c Release -- \
"/path/to/UOFiddler-4.22.2/Ultima.dll" \
"/path/to/UO client/Cliloc.enu" \
/srv/uo-data/clilocs.plain
# or tab-delimited text, if you want to eyeball or hand-edit it
dotnet run -c Release -- \
"/path/to/UOFiddler-4.22.2/Ultima.dll" \
"/path/to/UO client/Cliloc.enu" \
/srv/uo-data/clilocs.tsv --tsv
```
Expected output for a stock English client:
```
wrote 123490 entries to /srv/uo-data/clilocs.plain (maxTextBytes=12150, skippedOversize=0)
```
**Sanity-check that number.** A stock `Cliloc.enu` is ~123,000 entries. A few
hundred means it read something else and you should not ship the result. The
tool exits non-zero and says `no entries were written — is that a cliloc file?`
when it gets nothing at all.
The conversion runs on whatever machine has the client (usually Windows), and the
site reads the output wherever it runs — so **copy the output file to the server**
if those are different machines. It is a single self-contained file (~5 MB); the
`--tsv` form is larger but diff-able.
<details>
<summary>Errors you may hit</summary>
| Message | Cause |
|---|---|
| `Ultima.StringList not found — is that really UOFiddler's Ultima.dll?` | First argument points at some other `Ultima.dll` (ServUO ships one too — it is **not** the same assembly and cannot do this) |
| `You must install .NET to run this application` | Missing the .NET 10 runtime from Part 0 step 3 |
| `Unexpected Ultima.StringList API` | UOFiddler older than 4.21 |
| `usage: clilocexport …` | Fewer than three arguments |
</details>
### Route B — the UOFiddler GUI
Use this if you would rather not install a .NET SDK. **It needs one extra step**,
so do not skip the fixup.
1. Launch `UoFiddler.exe` and point it at your client directory when it asks
(or **Options → Path Settings**).
2. Open the **Cliloc** tab and use its **export to CSV** action.
3. It writes `CliLoc.csv` to UOFiddler's configured output path, in **three**
columns with a header row:
```
Number;Text;Flag
1023721;quarter staff;0
```
4. **Strip the trailing flag column.** The site's text parser reads
`number<TAB|,|;>text`, so that third field is otherwise absorbed into the name
and every item on the site renders as `quarter staff;0`.
```bash
sed -E 's/;[0-9]+$//' CliLoc.csv > clilocs.csv
```
```powershell
Get-Content CliLoc.csv |
ForEach-Object { $_ -replace ';\d+$','' } |
Set-Content -Encoding utf8 clilocs.csv
```
The header row needs no removal — a line whose first field is not an integer
is skipped. Blank entries (`1005008;`) survive the fixup correctly and are
dropped at import, as intended.
5. Copy `clilocs.csv` to the server.
**Why the fixup is not just done for us:** the parser already handles
`number,flag,text` — the flag in the *middle*, which is what several exports
emit. UOFiddler puts it at the *end*, where it is indistinguishable from a name
that genuinely ends in `;0`. One `sed` on the operator's side beats a parser
heuristic that would corrupt real names.
### Point the site at it
Two ways, the setting winning over the environment:
| Where | How |
|---|---|
| **Admin → Shard → cliloc path** | Takes effect on the next refresh, no redeploy |
| `UO_CLIENT_PATH` env var | The deploy-time default |
The value may be **the file itself or a directory to search** — both are natural
answers to "where is it", and overlays are picked up either way.
Setting the path deliberately does **not** import as a side effect. Click
**Import** (or `POST /api/v1/admin/shard/clilocs/import`) to load it.
### Verify
`GET /api/v1/admin/shard/clilocs`, or the Admin → Shard panel, reports what each
source contributed:
```json
"sources": [
{ "label": "clilocs.plain", "kind": "base", "entries": 123490, "added": 123490, "overrode": 0 }
]
```
Roughly **67,500 rows stored** from a stock table is correct — about half a
cliloc table is empty strings for ids the client reserves and never uses.
Then load any character sheet with equipment: items should show names rather than
`id 1023721`.
<details>
<summary>What a refusal means</summary>
A bad file answers `200` with a `status` and a named reason, not a `500` — you
need to be told *which file* to fix.
| `code` | Meaning |
|---|---|
| `COMPRESSED` | You pointed at the raw client `Cliloc.enu`. Convert it — this whole page. |
| `TRUNCATED` | Half-copied file. Re-copy; the loaded table is untouched. |
| `EMPTY` | A text source with no parseable rows — the file is named in the reason. |
| `status: needsReview` + `missingSources` | A previously-loaded source has vanished (unmounted volume? deliberate deletion?). Nothing changes until you re-import with `{ "approve": true }`. |
</details>
### Custom items — do *not* re-export for these
Shard-added items carry ids no client table has. Drop a small delimited file in a
`custom/` directory beside the base file and re-import:
```
/srv/uo-data/
clilocs.plain ← base, from this guide
custom/
01-uomysticmoon.tsv ← your additions and overrides
```
Files are read in sorted order and **later sources win**, so an overlay both adds
new ids and overrides stock ones you have re-purposed. **Adding one item never
means re-exporting a 5 MB client file.** Details in [`CLILOCS.md`](CLILOCS.md).
**If your shard distributes its own patched client to players, use that copy** —
it is what your players actually see. (It is also the copy the shard itself reads
from, since a ServUO server resolves `Config/DataPath.cfg` into its own client
path at boot, which is the premise the Asset Bridge is built on.)
---
@@ -246,7 +83,8 @@ functional as text, and `art` is NULL on every fresh import.
**This project ships no art and no art-extraction tooling, and never will.**
1. In `UoFiddler.exe` (paths configured as in Route B step 1), open the
1. Open `UoFiddler.exe` and point it at your client directory
(**Settings → Paths**), then open the
**Animations** tab for creature sprites — or **Items** for object art — find
the creature, and export as PNG. Right-click an entry for its export options,
or use the tab's *Export All* action for a batch. (4.22.2 added an export