README.md

@@ -22,6 +22,7 @@ ci/        cross-cutting CI/quality notes
 | [SHARD_VISIBILITY.md](website/SHARD_VISIBILITY.md) | Who sees which shard data — the admin-configurable audience framework |
 | [SPAWN_ATLAS.md](website/SPAWN_ATLAS.md) | The bestiary / spawn atlas: what the shard contains, parsed from its own ServUO tree |
 | [CLILOCS.md](website/CLILOCS.md) | UO's id → name table: converting one from your client so items have names |
+| [UOFIDDLER.md](website/UOFIDDLER.md) | **Operator runbook** — step-by-step extraction from your own UO client (cliloc table, creature art) |
 | [MARKETPLACE.md](website/MARKETPLACE.md) | The player-vendor index: how it is gathered, what it costs, how to tune it |
 | [website-README.md](website/website-README.md) | Snapshot of the website repo's README (setup/run reference) |
 | [PROJECT_TREE.md](website/PROJECT_TREE.md) | Auto-generated snapshot of the repo's tracked file layout |

website/CLILOCS.md

@@ -44,6 +44,11 @@ they did before the table existed.
 
 ## Converting
 
+> **Step-by-step operator instructions — where to get UOFiddler, where your
+> client files are, and how to verify the import — are in
+> [`UOFIDDLER.md`](UOFIDDLER.md).** This section covers the formats and the
+> reasoning behind them.
+
 Either format below is accepted; the site sniffs which one it was handed.
 
 | Format | Fidelity | Notes |
@@ -77,8 +82,16 @@ dotnet run -- "<UOFiddler>/Ultima.dll" "<UO client>/Cliloc.enu" /srv/uo-data/cli
 dotnet run -- "<UOFiddler>/Ultima.dll" "<UO client>/Cliloc.enu" /srv/uo-data/clilocs.tsv --tsv
 ```
 
-A UOFiddler GUI export works equally well — anything producing one of the two
+A UOFiddler GUI export works too, but **not unmodified**: its Cliloc tab writes
-shapes above is fine.
+`Number;Text;Flag` — three columns, the flag *last* — and the parser reads
+`number<separator>text`, so the trailing field is absorbed into the name and
+every item renders as `quarter staff;0`. Stripping it is one `sed`, given in
+[`UOFIDDLER.md`](UOFIDDLER.md) §Route B.
+
+The parser already tolerates `number,flag,text`, with the flag in the *middle*.
+It is not extended to cover the trailing form because a final `;0` is
+indistinguishable from a name that genuinely ends that way — a heuristic there
+would corrupt real names to save the operator one command.
 
 ## Shard-added and shard-edited items
 

website/SPAWN_ATLAS.md

@@ -223,7 +223,8 @@ The atlas is fully functional as text. `shard_spawn_creatures.art` is nullable
 and is NULL on every fresh import; pages render without images, which is the
 normal and supported state, not a degraded one.
 
-An operator who wants art:
+An operator who wants art — step-by-step, with the UOFiddler side spelled out, in
+[`UOFIDDLER.md`](UOFIDDLER.md) §Part 2:
 
 1. Extracts it from **their own** client files (UOFiddler, ClassicUO tooling, or
    any art extractor).

website/UOFIDDLER.md

@@ -0,0 +1,282 @@
+# Extracting from your own UO client (UOFiddler)
+
+**Audience:** the shard operator, once, at setup time.
+**Related:** [`CLILOCS.md`](CLILOCS.md) (why the cliloc conversion is unavoidable),
+[`SPAWN_ATLAS.md`](SPAWN_ATLAS.md) (where creature art fits).
+
+Two features read data that **only exists inside a UO client**, and a UO client's
+files are EA's, not ours to redistribute. So neither this repo nor any image we
+publish can ship them — the operator extracts from **their own** client, once,
+and points the site at the result.
+
+| Feature | What it needs | Required? | Without it |
+|---|---|---|---|
+| **Item / title names** ([`CLILOCS.md`](CLILOCS.md)) | `Cliloc.enu`, converted | No | Names render as raw ids — `id 1023721` instead of *quarter staff* |
+| **Creature art** ([`SPAWN_ATLAS.md`](SPAWN_ATLAS.md)) | Sprites from `.mul`/`.uop` | No | Atlas pages render as text, which is the normal state |
+
+**Both are optional and neither is load-bearing.** A shard that never does any of
+this is fully supported. Do part one and skip part two if art is not worth your
+time — they share only the tool.
+
+Everything you extract stays **outside the repository**: the converted cliloc
+file lives at a path you choose, and `spawnAtlas.art.json` plus `server/uploads/`
+are gitignored, so none of it can be committed by accident.
+
+---
+
+## Part 0 — Get UOFiddler
+
+[UOFiddler](https://github.com/polserver/UOFiddler) is the community client-file
+editor. We use it because its `Ultima.dll` already contains the cliloc
+decompressor, maintained by people who do this for a living.
+
+1. Download the latest release zip from
+   <https://github.com/polserver/UOFiddler/releases/latest> — one asset, named
+   `UOFiddler-<version>.zip` (4.22.2 is ~2 MB).
+2. Extract it. The zip contains a single top-level folder, and the two files that
+   matter are at **its root**:
+
+   ```
+   UOFiddler-4.22.2/
+     Ultima.dll        ← the decompressor (Part 1 needs this path)
+     UoFiddler.exe     ← the GUI (Part 2 needs this)
+     plugins/
+     …
+   ```
+
+3. **Runtime:** UOFiddler 4.22.2 is built for **.NET 10**. Running `UoFiddler.exe`
+   needs the .NET 10 **Desktop** Runtime (Windows only); loading `Ultima.dll` from
+   the converter in Part 1 needs the .NET 10 runtime. Install from
+   <https://dotnet.microsoft.com/download/dotnet/10.0>.
+
+### Finding your client files
+
+The cliloc file is in your **UO client installation directory**, not in your
+ServUO tree — the shard server has no copy of it. Look for `Cliloc.enu` (English;
+the other seven are `chs`, `cht`, `deu`, `esp`, `fra`, `jpn`, `kor`) beside
+`art.mul` / `artLegacyMUL.uop`. The EA Classic Client's default location is:
+
+```
+C:\Program Files (x86)\Electronic Arts\Ultima Online Classic\
+```
+
+**If your shard distributes its own patched client to players, use that copy.**
+Any cliloc edits you shipped to players are then already in the base table and
+you need no overlay for them (see [`CLILOCS.md`](CLILOCS.md) §Shard-added and
+shard-edited items).
+
+---
+
+## Part 1 — Convert the cliloc table
+
+**Goal:** turn the client's compressed `Cliloc.enu` into a file the site can
+read, and point the site at it.
+
+The site cannot read `Cliloc.enu` directly. Every modern client compresses it
+(the "Mythic" container), and so does ServUO's own bundled `Ultima.StringList` —
+which is why the shard cannot supply names on our behalf either. The full
+reasoning is in [`CLILOCS.md`](CLILOCS.md) §Why the operator has to convert the
+file; this section is just the procedure.
+
+Two routes. **The bundled tool is the recommended one** — the GUI export needs a
+fixup step, described below.
+
+### Route A — the bundled converter (recommended)
+
+Needs a .NET SDK (any version 8 or newer — the project targets `net8.0` and rolls
+forward, so whatever you have works) **plus** the .NET 10 runtime from Part 0,
+which is what actually loads `Ultima.dll`.
+
+```bash
+cd website/server/tools/cliloc-export
+dotnet build -c Release
+
+# plain binary — recommended, exact
+dotnet run -c Release -- \
+  "/path/to/UOFiddler-4.22.2/Ultima.dll" \
+  "/path/to/UO client/Cliloc.enu" \
+  /srv/uo-data/clilocs.plain
+
+# or tab-delimited text, if you want to eyeball or hand-edit it
+dotnet run -c Release -- \
+  "/path/to/UOFiddler-4.22.2/Ultima.dll" \
+  "/path/to/UO client/Cliloc.enu" \
+  /srv/uo-data/clilocs.tsv --tsv
+```
+
+Expected output for a stock English client:
+
+```
+wrote 123490 entries to /srv/uo-data/clilocs.plain (maxTextBytes=12150, skippedOversize=0)
+```
+
+**Sanity-check that number.** A stock `Cliloc.enu` is ~123,000 entries. A few
+hundred means it read something else and you should not ship the result. The
+tool exits non-zero and says `no entries were written — is that a cliloc file?`
+when it gets nothing at all.
+
+The conversion runs on whatever machine has the client (usually Windows), and the
+site reads the output wherever it runs — so **copy the output file to the server**
+if those are different machines. It is a single self-contained file (~5 MB); the
+`--tsv` form is larger but diff-able.
+
+<details>
+<summary>Errors you may hit</summary>
+
+| Message | Cause |
+|---|---|
+| `Ultima.StringList not found — is that really UOFiddler's Ultima.dll?` | First argument points at some other `Ultima.dll` (ServUO ships one too — it is **not** the same assembly and cannot do this) |
+| `You must install .NET to run this application` | Missing the .NET 10 runtime from Part 0 step 3 |
+| `Unexpected Ultima.StringList API` | UOFiddler older than 4.21 |
+| `usage: clilocexport …` | Fewer than three arguments |
+
+</details>
+
+### Route B — the UOFiddler GUI
+
+Use this if you would rather not install a .NET SDK. **It needs one extra step**,
+so do not skip the fixup.
+
+1. Launch `UoFiddler.exe` and point it at your client directory when it asks
+   (or **Options → Path Settings**).
+2. Open the **Cliloc** tab and use its **export to CSV** action.
+3. It writes `CliLoc.csv` to UOFiddler's configured output path, in **three**
+   columns with a header row:
+
+   ```
+   Number;Text;Flag
+   1023721;quarter staff;0
+   ```
+
+4. **Strip the trailing flag column.** The site's text parser reads
+   `number<TAB|,|;>text`, so that third field is otherwise absorbed into the name
+   and every item on the site renders as `quarter staff;0`.
+
+   ```bash
+   sed -E 's/;[0-9]+$//' CliLoc.csv > clilocs.csv
+   ```
+
+   ```powershell
+   Get-Content CliLoc.csv |
+     ForEach-Object { $_ -replace ';\d+$','' } |
+     Set-Content -Encoding utf8 clilocs.csv
+   ```
+
+   The header row needs no removal — a line whose first field is not an integer
+   is skipped. Blank entries (`1005008;`) survive the fixup correctly and are
+   dropped at import, as intended.
+
+5. Copy `clilocs.csv` to the server.
+
+**Why the fixup is not just done for us:** the parser already handles
+`number,flag,text` — the flag in the *middle*, which is what several exports
+emit. UOFiddler puts it at the *end*, where it is indistinguishable from a name
+that genuinely ends in `;0`. One `sed` on the operator's side beats a parser
+heuristic that would corrupt real names.
+
+### Point the site at it
+
+Two ways, the setting winning over the environment:
+
+| Where | How |
+|---|---|
+| **Admin → Shard → cliloc path** | Takes effect on the next refresh, no redeploy |
+| `UO_CLIENT_PATH` env var | The deploy-time default |
+
+The value may be **the file itself or a directory to search** — both are natural
+answers to "where is it", and overlays are picked up either way.
+
+Setting the path deliberately does **not** import as a side effect. Click
+**Import** (or `POST /api/v1/admin/shard/clilocs/import`) to load it.
+
+### Verify
+
+`GET /api/v1/admin/shard/clilocs`, or the Admin → Shard panel, reports what each
+source contributed:
+
+```json
+"sources": [
+  { "label": "clilocs.plain", "kind": "base", "entries": 123490, "added": 123490, "overrode": 0 }
+]
+```
+
+Roughly **67,500 rows stored** from a stock table is correct — about half a
+cliloc table is empty strings for ids the client reserves and never uses.
+
+Then load any character sheet with equipment: items should show names rather than
+`id 1023721`.
+
+<details>
+<summary>What a refusal means</summary>
+
+A bad file answers `200` with a `status` and a named reason, not a `500` — you
+need to be told *which file* to fix.
+
+| `code` | Meaning |
+|---|---|
+| `COMPRESSED` | You pointed at the raw client `Cliloc.enu`. Convert it — this whole page. |
+| `TRUNCATED` | Half-copied file. Re-copy; the loaded table is untouched. |
+| `EMPTY` | A text source with no parseable rows — the file is named in the reason. |
+| `status: needsReview` + `missingSources` | A previously-loaded source has vanished (unmounted volume? deliberate deletion?). Nothing changes until you re-import with `{ "approve": true }`. |
+
+</details>
+
+### Custom items — do *not* re-export for these
+
+Shard-added items carry ids no client table has. Drop a small delimited file in a
+`custom/` directory beside the base file and re-import:
+
+```
+/srv/uo-data/
+  clilocs.plain            ← base, from this guide
+  custom/
+    01-uomysticmoon.tsv    ← your additions and overrides
+```
+
+Files are read in sorted order and **later sources win**, so an overlay both adds
+new ids and overrides stock ones you have re-purposed. **Adding one item never
+means re-exporting a 5 MB client file.** Details in [`CLILOCS.md`](CLILOCS.md).
+
+---
+
+## Part 2 — Creature art for the spawn atlas (optional)
+
+**Goal:** put sprites on atlas pages. Purely cosmetic — the atlas is fully
+functional as text, and `art` is NULL on every fresh import.
+
+**This project ships no art and no art-extraction tooling, and never will.**
+
+1. In `UoFiddler.exe` (paths configured as in Route B step 1), open the
+   **Animations** tab for creature sprites — or **Items** for object art — find
+   the creature, and export as PNG. Right-click an entry for its export options,
+   or use the tab's *Export All* action for a batch. (4.22.2 added an export
+   option to the Animation tab's thumbnail list, which is the convenient one
+   here.)
+2. Put the images under `server/uploads/atlas/`.
+3. Copy `server/db/data/spawnAtlas.art.example.json` to `spawnAtlas.art.json` in
+   the same directory and map creature slugs to file names:
+
+   ```json
+   {
+     "lizardman": "lizardman.png",
+     "orc": "orc.png"
+   }
+   ```
+
+   **Keys are the slugs the atlas API reports**, derived from the type names in
+   your own shard's `Spawns/*.xml` — read them off the atlas rather than guessing.
+   A creature with no entry renders without art, which is the default.
+
+4. Restart, or `npm run atlas:import -- --force`.
+
+The art map is re-read on every atlas refresh, so adding one image is an edit plus
+a refresh. Both `spawnAtlas.art.json` and `server/uploads/` are gitignored.
+
+---
+
+## Licensing, briefly
+
+UO's strings and sprites are EA's. Extracting from **your own** client for
+**your own** shard is the arrangement here; redistributing the extracted files is
+not something this project does or can advise on. That is the whole reason this
+page exists instead of a download link.