From afcdb373eca209df307289e6a4fae1f2113665f5 Mon Sep 17 00:00:00 2001 From: wtclaude Date: Thu, 30 Jul 2026 03:34:29 -0500 Subject: [PATCH] docs(website): an operator runbook for extracting from your own UO client CLILOCS.md and SPAWN_ATLAS.md each explain WHY the operator has to supply something out of their own client, but neither says how. UOFIDDLER.md is the missing procedure: where to get UOFiddler, which two files in the zip matter, which runtime it needs, where Cliloc.enu actually lives, the conversion, how to point the site at the result, and how to confirm it took. Verified end to end on a stock Windows box: UOFiddler 4.22.2 (Ultima.dll is net10.0), .NET SDK 9.0.312 building the net8.0 converter, RollForward carrying it onto runtime 10.0.8, and the site's own parser reading the output back. Corrects one claim while doing it. CLILOCS.md said a UOFiddler GUI export "works equally well"; it does not. Its Cliloc tab writes `Number;Text;Flag` -- three columns, flag LAST -- and parseClilocText splits on the first separator only, so the flag is absorbed into the name and every item renders as `quarter staff;0`. The parser already handles `number,flag,text` with the flag in the middle, but a trailing `;0` is indistinguishable from a name that genuinely ends that way, so this stays a documented `sed` on the operator's side rather than a heuristic that would corrupt real names. Co-Authored-By: Claude --- README.md | 1 + website/CLILOCS.md | 17 ++- website/SPAWN_ATLAS.md | 3 +- website/UOFIDDLER.md | 282 +++++++++++++++++++++++++++++++++++++++++ 4 files changed, 300 insertions(+), 3 deletions(-) create mode 100644 website/UOFIDDLER.md diff --git a/README.md b/README.md index fe4d540..679624b 100644 --- a/README.md +++ b/README.md @@ -22,6 +22,7 @@ ci/ cross-cutting CI/quality notes | [SHARD_VISIBILITY.md](website/SHARD_VISIBILITY.md) | Who sees which shard data — the admin-configurable audience framework | | [SPAWN_ATLAS.md](website/SPAWN_ATLAS.md) | The bestiary / spawn atlas: what the shard contains, parsed from its own ServUO tree | | [CLILOCS.md](website/CLILOCS.md) | UO's id → name table: converting one from your client so items have names | +| [UOFIDDLER.md](website/UOFIDDLER.md) | **Operator runbook** — step-by-step extraction from your own UO client (cliloc table, creature art) | | [MARKETPLACE.md](website/MARKETPLACE.md) | The player-vendor index: how it is gathered, what it costs, how to tune it | | [website-README.md](website/website-README.md) | Snapshot of the website repo's README (setup/run reference) | | [PROJECT_TREE.md](website/PROJECT_TREE.md) | Auto-generated snapshot of the repo's tracked file layout | diff --git a/website/CLILOCS.md b/website/CLILOCS.md index e0747f5..f2e1413 100644 --- a/website/CLILOCS.md +++ b/website/CLILOCS.md @@ -44,6 +44,11 @@ they did before the table existed. ## Converting +> **Step-by-step operator instructions — where to get UOFiddler, where your +> client files are, and how to verify the import — are in +> [`UOFIDDLER.md`](UOFIDDLER.md).** This section covers the formats and the +> reasoning behind them. + Either format below is accepted; the site sniffs which one it was handed. | Format | Fidelity | Notes | @@ -77,8 +82,16 @@ dotnet run -- "/Ultima.dll" "/Cliloc.enu" /srv/uo-data/cli dotnet run -- "/Ultima.dll" "/Cliloc.enu" /srv/uo-data/clilocs.tsv --tsv ``` -A UOFiddler GUI export works equally well — anything producing one of the two -shapes above is fine. +A UOFiddler GUI export works too, but **not unmodified**: its Cliloc tab writes +`Number;Text;Flag` — three columns, the flag *last* — and the parser reads +`numbertext`, so the trailing field is absorbed into the name and +every item renders as `quarter staff;0`. Stripping it is one `sed`, given in +[`UOFIDDLER.md`](UOFIDDLER.md) §Route B. + +The parser already tolerates `number,flag,text`, with the flag in the *middle*. +It is not extended to cover the trailing form because a final `;0` is +indistinguishable from a name that genuinely ends that way — a heuristic there +would corrupt real names to save the operator one command. ## Shard-added and shard-edited items diff --git a/website/SPAWN_ATLAS.md b/website/SPAWN_ATLAS.md index f74d251..343ad62 100644 --- a/website/SPAWN_ATLAS.md +++ b/website/SPAWN_ATLAS.md @@ -223,7 +223,8 @@ The atlas is fully functional as text. `shard_spawn_creatures.art` is nullable and is NULL on every fresh import; pages render without images, which is the normal and supported state, not a degraded one. -An operator who wants art: +An operator who wants art — step-by-step, with the UOFiddler side spelled out, in +[`UOFIDDLER.md`](UOFIDDLER.md) §Part 2: 1. Extracts it from **their own** client files (UOFiddler, ClassicUO tooling, or any art extractor). diff --git a/website/UOFIDDLER.md b/website/UOFIDDLER.md new file mode 100644 index 0000000..96d8e69 --- /dev/null +++ b/website/UOFIDDLER.md @@ -0,0 +1,282 @@ +# Extracting from your own UO client (UOFiddler) + +**Audience:** the shard operator, once, at setup time. +**Related:** [`CLILOCS.md`](CLILOCS.md) (why the cliloc conversion is unavoidable), +[`SPAWN_ATLAS.md`](SPAWN_ATLAS.md) (where creature art fits). + +Two features read data that **only exists inside a UO client**, and a UO client's +files are EA's, not ours to redistribute. So neither this repo nor any image we +publish can ship them — the operator extracts from **their own** client, once, +and points the site at the result. + +| Feature | What it needs | Required? | Without it | +|---|---|---|---| +| **Item / title names** ([`CLILOCS.md`](CLILOCS.md)) | `Cliloc.enu`, converted | No | Names render as raw ids — `id 1023721` instead of *quarter staff* | +| **Creature art** ([`SPAWN_ATLAS.md`](SPAWN_ATLAS.md)) | Sprites from `.mul`/`.uop` | No | Atlas pages render as text, which is the normal state | + +**Both are optional and neither is load-bearing.** A shard that never does any of +this is fully supported. Do part one and skip part two if art is not worth your +time — they share only the tool. + +Everything you extract stays **outside the repository**: the converted cliloc +file lives at a path you choose, and `spawnAtlas.art.json` plus `server/uploads/` +are gitignored, so none of it can be committed by accident. + +--- + +## Part 0 — Get UOFiddler + +[UOFiddler](https://github.com/polserver/UOFiddler) is the community client-file +editor. We use it because its `Ultima.dll` already contains the cliloc +decompressor, maintained by people who do this for a living. + +1. Download the latest release zip from + — one asset, named + `UOFiddler-.zip` (4.22.2 is ~2 MB). +2. Extract it. The zip contains a single top-level folder, and the two files that + matter are at **its root**: + + ``` + UOFiddler-4.22.2/ + Ultima.dll ← the decompressor (Part 1 needs this path) + UoFiddler.exe ← the GUI (Part 2 needs this) + plugins/ + … + ``` + +3. **Runtime:** UOFiddler 4.22.2 is built for **.NET 10**. Running `UoFiddler.exe` + needs the .NET 10 **Desktop** Runtime (Windows only); loading `Ultima.dll` from + the converter in Part 1 needs the .NET 10 runtime. Install from + . + +### Finding your client files + +The cliloc file is in your **UO client installation directory**, not in your +ServUO tree — the shard server has no copy of it. Look for `Cliloc.enu` (English; +the other seven are `chs`, `cht`, `deu`, `esp`, `fra`, `jpn`, `kor`) beside +`art.mul` / `artLegacyMUL.uop`. The EA Classic Client's default location is: + +``` +C:\Program Files (x86)\Electronic Arts\Ultima Online Classic\ +``` + +**If your shard distributes its own patched client to players, use that copy.** +Any cliloc edits you shipped to players are then already in the base table and +you need no overlay for them (see [`CLILOCS.md`](CLILOCS.md) §Shard-added and +shard-edited items). + +--- + +## Part 1 — Convert the cliloc table + +**Goal:** turn the client's compressed `Cliloc.enu` into a file the site can +read, and point the site at it. + +The site cannot read `Cliloc.enu` directly. Every modern client compresses it +(the "Mythic" container), and so does ServUO's own bundled `Ultima.StringList` — +which is why the shard cannot supply names on our behalf either. The full +reasoning is in [`CLILOCS.md`](CLILOCS.md) §Why the operator has to convert the +file; this section is just the procedure. + +Two routes. **The bundled tool is the recommended one** — the GUI export needs a +fixup step, described below. + +### Route A — the bundled converter (recommended) + +Needs a .NET SDK (any version 8 or newer — the project targets `net8.0` and rolls +forward, so whatever you have works) **plus** the .NET 10 runtime from Part 0, +which is what actually loads `Ultima.dll`. + +```bash +cd website/server/tools/cliloc-export +dotnet build -c Release + +# plain binary — recommended, exact +dotnet run -c Release -- \ + "/path/to/UOFiddler-4.22.2/Ultima.dll" \ + "/path/to/UO client/Cliloc.enu" \ + /srv/uo-data/clilocs.plain + +# or tab-delimited text, if you want to eyeball or hand-edit it +dotnet run -c Release -- \ + "/path/to/UOFiddler-4.22.2/Ultima.dll" \ + "/path/to/UO client/Cliloc.enu" \ + /srv/uo-data/clilocs.tsv --tsv +``` + +Expected output for a stock English client: + +``` +wrote 123490 entries to /srv/uo-data/clilocs.plain (maxTextBytes=12150, skippedOversize=0) +``` + +**Sanity-check that number.** A stock `Cliloc.enu` is ~123,000 entries. A few +hundred means it read something else and you should not ship the result. The +tool exits non-zero and says `no entries were written — is that a cliloc file?` +when it gets nothing at all. + +The conversion runs on whatever machine has the client (usually Windows), and the +site reads the output wherever it runs — so **copy the output file to the server** +if those are different machines. It is a single self-contained file (~5 MB); the +`--tsv` form is larger but diff-able. + +
+Errors you may hit + +| Message | Cause | +|---|---| +| `Ultima.StringList not found — is that really UOFiddler's Ultima.dll?` | First argument points at some other `Ultima.dll` (ServUO ships one too — it is **not** the same assembly and cannot do this) | +| `You must install .NET to run this application` | Missing the .NET 10 runtime from Part 0 step 3 | +| `Unexpected Ultima.StringList API` | UOFiddler older than 4.21 | +| `usage: clilocexport …` | Fewer than three arguments | + +
+ +### Route B — the UOFiddler GUI + +Use this if you would rather not install a .NET SDK. **It needs one extra step**, +so do not skip the fixup. + +1. Launch `UoFiddler.exe` and point it at your client directory when it asks + (or **Options → Path Settings**). +2. Open the **Cliloc** tab and use its **export to CSV** action. +3. It writes `CliLoc.csv` to UOFiddler's configured output path, in **three** + columns with a header row: + + ``` + Number;Text;Flag + 1023721;quarter staff;0 + ``` + +4. **Strip the trailing flag column.** The site's text parser reads + `numbertext`, so that third field is otherwise absorbed into the name + and every item on the site renders as `quarter staff;0`. + + ```bash + sed -E 's/;[0-9]+$//' CliLoc.csv > clilocs.csv + ``` + + ```powershell + Get-Content CliLoc.csv | + ForEach-Object { $_ -replace ';\d+$','' } | + Set-Content -Encoding utf8 clilocs.csv + ``` + + The header row needs no removal — a line whose first field is not an integer + is skipped. Blank entries (`1005008;`) survive the fixup correctly and are + dropped at import, as intended. + +5. Copy `clilocs.csv` to the server. + +**Why the fixup is not just done for us:** the parser already handles +`number,flag,text` — the flag in the *middle*, which is what several exports +emit. UOFiddler puts it at the *end*, where it is indistinguishable from a name +that genuinely ends in `;0`. One `sed` on the operator's side beats a parser +heuristic that would corrupt real names. + +### Point the site at it + +Two ways, the setting winning over the environment: + +| Where | How | +|---|---| +| **Admin → Shard → cliloc path** | Takes effect on the next refresh, no redeploy | +| `UO_CLIENT_PATH` env var | The deploy-time default | + +The value may be **the file itself or a directory to search** — both are natural +answers to "where is it", and overlays are picked up either way. + +Setting the path deliberately does **not** import as a side effect. Click +**Import** (or `POST /api/v1/admin/shard/clilocs/import`) to load it. + +### Verify + +`GET /api/v1/admin/shard/clilocs`, or the Admin → Shard panel, reports what each +source contributed: + +```json +"sources": [ + { "label": "clilocs.plain", "kind": "base", "entries": 123490, "added": 123490, "overrode": 0 } +] +``` + +Roughly **67,500 rows stored** from a stock table is correct — about half a +cliloc table is empty strings for ids the client reserves and never uses. + +Then load any character sheet with equipment: items should show names rather than +`id 1023721`. + +
+What a refusal means + +A bad file answers `200` with a `status` and a named reason, not a `500` — you +need to be told *which file* to fix. + +| `code` | Meaning | +|---|---| +| `COMPRESSED` | You pointed at the raw client `Cliloc.enu`. Convert it — this whole page. | +| `TRUNCATED` | Half-copied file. Re-copy; the loaded table is untouched. | +| `EMPTY` | A text source with no parseable rows — the file is named in the reason. | +| `status: needsReview` + `missingSources` | A previously-loaded source has vanished (unmounted volume? deliberate deletion?). Nothing changes until you re-import with `{ "approve": true }`. | + +
+ +### Custom items — do *not* re-export for these + +Shard-added items carry ids no client table has. Drop a small delimited file in a +`custom/` directory beside the base file and re-import: + +``` +/srv/uo-data/ + clilocs.plain ← base, from this guide + custom/ + 01-uomysticmoon.tsv ← your additions and overrides +``` + +Files are read in sorted order and **later sources win**, so an overlay both adds +new ids and overrides stock ones you have re-purposed. **Adding one item never +means re-exporting a 5 MB client file.** Details in [`CLILOCS.md`](CLILOCS.md). + +--- + +## Part 2 — Creature art for the spawn atlas (optional) + +**Goal:** put sprites on atlas pages. Purely cosmetic — the atlas is fully +functional as text, and `art` is NULL on every fresh import. + +**This project ships no art and no art-extraction tooling, and never will.** + +1. In `UoFiddler.exe` (paths configured as in Route B step 1), open the + **Animations** tab for creature sprites — or **Items** for object art — find + the creature, and export as PNG. Right-click an entry for its export options, + or use the tab's *Export All* action for a batch. (4.22.2 added an export + option to the Animation tab's thumbnail list, which is the convenient one + here.) +2. Put the images under `server/uploads/atlas/`. +3. Copy `server/db/data/spawnAtlas.art.example.json` to `spawnAtlas.art.json` in + the same directory and map creature slugs to file names: + + ```json + { + "lizardman": "lizardman.png", + "orc": "orc.png" + } + ``` + + **Keys are the slugs the atlas API reports**, derived from the type names in + your own shard's `Spawns/*.xml` — read them off the atlas rather than guessing. + A creature with no entry renders without art, which is the default. + +4. Restart, or `npm run atlas:import -- --force`. + +The art map is re-read on every atlas refresh, so adding one image is an edit plus +a refresh. Both `spawnAtlas.art.json` and `server/uploads/` are gitignored. + +--- + +## Licensing, briefly + +UO's strings and sprites are EA's. Extracting from **your own** client for +**your own** shard is the arrangement here; redistributing the extracted files is +not something this project does or can advise on. That is the whole reason this +page exists instead of a download link. -- 2.49.1