diff --git a/README.md b/README.md index fe4d540..679624b 100644 --- a/README.md +++ b/README.md @@ -22,6 +22,7 @@ ci/ cross-cutting CI/quality notes | [SHARD_VISIBILITY.md](website/SHARD_VISIBILITY.md) | Who sees which shard data — the admin-configurable audience framework | | [SPAWN_ATLAS.md](website/SPAWN_ATLAS.md) | The bestiary / spawn atlas: what the shard contains, parsed from its own ServUO tree | | [CLILOCS.md](website/CLILOCS.md) | UO's id → name table: converting one from your client so items have names | +| [UOFIDDLER.md](website/UOFIDDLER.md) | **Operator runbook** — step-by-step extraction from your own UO client (cliloc table, creature art) | | [MARKETPLACE.md](website/MARKETPLACE.md) | The player-vendor index: how it is gathered, what it costs, how to tune it | | [website-README.md](website/website-README.md) | Snapshot of the website repo's README (setup/run reference) | | [PROJECT_TREE.md](website/PROJECT_TREE.md) | Auto-generated snapshot of the repo's tracked file layout | diff --git a/website/CLILOCS.md b/website/CLILOCS.md index e0747f5..f2e1413 100644 --- a/website/CLILOCS.md +++ b/website/CLILOCS.md @@ -44,6 +44,11 @@ they did before the table existed. ## Converting +> **Step-by-step operator instructions — where to get UOFiddler, where your +> client files are, and how to verify the import — are in +> [`UOFIDDLER.md`](UOFIDDLER.md).** This section covers the formats and the +> reasoning behind them. + Either format below is accepted; the site sniffs which one it was handed. | Format | Fidelity | Notes | @@ -77,8 +82,16 @@ dotnet run -- "/Ultima.dll" "/Cliloc.enu" /srv/uo-data/cli dotnet run -- "/Ultima.dll" "/Cliloc.enu" /srv/uo-data/clilocs.tsv --tsv ``` -A UOFiddler GUI export works equally well — anything producing one of the two -shapes above is fine. +A UOFiddler GUI export works too, but **not unmodified**: its Cliloc tab writes +`Number;Text;Flag` — three columns, the flag *last* — and the parser reads +`numbertext`, so the trailing field is absorbed into the name and +every item renders as `quarter staff;0`. Stripping it is one `sed`, given in +[`UOFIDDLER.md`](UOFIDDLER.md) §Route B. + +The parser already tolerates `number,flag,text`, with the flag in the *middle*. +It is not extended to cover the trailing form because a final `;0` is +indistinguishable from a name that genuinely ends that way — a heuristic there +would corrupt real names to save the operator one command. ## Shard-added and shard-edited items diff --git a/website/SPAWN_ATLAS.md b/website/SPAWN_ATLAS.md index f74d251..343ad62 100644 --- a/website/SPAWN_ATLAS.md +++ b/website/SPAWN_ATLAS.md @@ -223,7 +223,8 @@ The atlas is fully functional as text. `shard_spawn_creatures.art` is nullable and is NULL on every fresh import; pages render without images, which is the normal and supported state, not a degraded one. -An operator who wants art: +An operator who wants art — step-by-step, with the UOFiddler side spelled out, in +[`UOFIDDLER.md`](UOFIDDLER.md) §Part 2: 1. Extracts it from **their own** client files (UOFiddler, ClassicUO tooling, or any art extractor). diff --git a/website/UOFIDDLER.md b/website/UOFIDDLER.md new file mode 100644 index 0000000..96d8e69 --- /dev/null +++ b/website/UOFIDDLER.md @@ -0,0 +1,282 @@ +# Extracting from your own UO client (UOFiddler) + +**Audience:** the shard operator, once, at setup time. +**Related:** [`CLILOCS.md`](CLILOCS.md) (why the cliloc conversion is unavoidable), +[`SPAWN_ATLAS.md`](SPAWN_ATLAS.md) (where creature art fits). + +Two features read data that **only exists inside a UO client**, and a UO client's +files are EA's, not ours to redistribute. So neither this repo nor any image we +publish can ship them — the operator extracts from **their own** client, once, +and points the site at the result. + +| Feature | What it needs | Required? | Without it | +|---|---|---|---| +| **Item / title names** ([`CLILOCS.md`](CLILOCS.md)) | `Cliloc.enu`, converted | No | Names render as raw ids — `id 1023721` instead of *quarter staff* | +| **Creature art** ([`SPAWN_ATLAS.md`](SPAWN_ATLAS.md)) | Sprites from `.mul`/`.uop` | No | Atlas pages render as text, which is the normal state | + +**Both are optional and neither is load-bearing.** A shard that never does any of +this is fully supported. Do part one and skip part two if art is not worth your +time — they share only the tool. + +Everything you extract stays **outside the repository**: the converted cliloc +file lives at a path you choose, and `spawnAtlas.art.json` plus `server/uploads/` +are gitignored, so none of it can be committed by accident. + +--- + +## Part 0 — Get UOFiddler + +[UOFiddler](https://github.com/polserver/UOFiddler) is the community client-file +editor. We use it because its `Ultima.dll` already contains the cliloc +decompressor, maintained by people who do this for a living. + +1. Download the latest release zip from + — one asset, named + `UOFiddler-.zip` (4.22.2 is ~2 MB). +2. Extract it. The zip contains a single top-level folder, and the two files that + matter are at **its root**: + + ``` + UOFiddler-4.22.2/ + Ultima.dll ← the decompressor (Part 1 needs this path) + UoFiddler.exe ← the GUI (Part 2 needs this) + plugins/ + … + ``` + +3. **Runtime:** UOFiddler 4.22.2 is built for **.NET 10**. Running `UoFiddler.exe` + needs the .NET 10 **Desktop** Runtime (Windows only); loading `Ultima.dll` from + the converter in Part 1 needs the .NET 10 runtime. Install from + . + +### Finding your client files + +The cliloc file is in your **UO client installation directory**, not in your +ServUO tree — the shard server has no copy of it. Look for `Cliloc.enu` (English; +the other seven are `chs`, `cht`, `deu`, `esp`, `fra`, `jpn`, `kor`) beside +`art.mul` / `artLegacyMUL.uop`. The EA Classic Client's default location is: + +``` +C:\Program Files (x86)\Electronic Arts\Ultima Online Classic\ +``` + +**If your shard distributes its own patched client to players, use that copy.** +Any cliloc edits you shipped to players are then already in the base table and +you need no overlay for them (see [`CLILOCS.md`](CLILOCS.md) §Shard-added and +shard-edited items). + +--- + +## Part 1 — Convert the cliloc table + +**Goal:** turn the client's compressed `Cliloc.enu` into a file the site can +read, and point the site at it. + +The site cannot read `Cliloc.enu` directly. Every modern client compresses it +(the "Mythic" container), and so does ServUO's own bundled `Ultima.StringList` — +which is why the shard cannot supply names on our behalf either. The full +reasoning is in [`CLILOCS.md`](CLILOCS.md) §Why the operator has to convert the +file; this section is just the procedure. + +Two routes. **The bundled tool is the recommended one** — the GUI export needs a +fixup step, described below. + +### Route A — the bundled converter (recommended) + +Needs a .NET SDK (any version 8 or newer — the project targets `net8.0` and rolls +forward, so whatever you have works) **plus** the .NET 10 runtime from Part 0, +which is what actually loads `Ultima.dll`. + +```bash +cd website/server/tools/cliloc-export +dotnet build -c Release + +# plain binary — recommended, exact +dotnet run -c Release -- \ + "/path/to/UOFiddler-4.22.2/Ultima.dll" \ + "/path/to/UO client/Cliloc.enu" \ + /srv/uo-data/clilocs.plain + +# or tab-delimited text, if you want to eyeball or hand-edit it +dotnet run -c Release -- \ + "/path/to/UOFiddler-4.22.2/Ultima.dll" \ + "/path/to/UO client/Cliloc.enu" \ + /srv/uo-data/clilocs.tsv --tsv +``` + +Expected output for a stock English client: + +``` +wrote 123490 entries to /srv/uo-data/clilocs.plain (maxTextBytes=12150, skippedOversize=0) +``` + +**Sanity-check that number.** A stock `Cliloc.enu` is ~123,000 entries. A few +hundred means it read something else and you should not ship the result. The +tool exits non-zero and says `no entries were written — is that a cliloc file?` +when it gets nothing at all. + +The conversion runs on whatever machine has the client (usually Windows), and the +site reads the output wherever it runs — so **copy the output file to the server** +if those are different machines. It is a single self-contained file (~5 MB); the +`--tsv` form is larger but diff-able. + +
+Errors you may hit + +| Message | Cause | +|---|---| +| `Ultima.StringList not found — is that really UOFiddler's Ultima.dll?` | First argument points at some other `Ultima.dll` (ServUO ships one too — it is **not** the same assembly and cannot do this) | +| `You must install .NET to run this application` | Missing the .NET 10 runtime from Part 0 step 3 | +| `Unexpected Ultima.StringList API` | UOFiddler older than 4.21 | +| `usage: clilocexport …` | Fewer than three arguments | + +
+ +### Route B — the UOFiddler GUI + +Use this if you would rather not install a .NET SDK. **It needs one extra step**, +so do not skip the fixup. + +1. Launch `UoFiddler.exe` and point it at your client directory when it asks + (or **Options → Path Settings**). +2. Open the **Cliloc** tab and use its **export to CSV** action. +3. It writes `CliLoc.csv` to UOFiddler's configured output path, in **three** + columns with a header row: + + ``` + Number;Text;Flag + 1023721;quarter staff;0 + ``` + +4. **Strip the trailing flag column.** The site's text parser reads + `numbertext`, so that third field is otherwise absorbed into the name + and every item on the site renders as `quarter staff;0`. + + ```bash + sed -E 's/;[0-9]+$//' CliLoc.csv > clilocs.csv + ``` + + ```powershell + Get-Content CliLoc.csv | + ForEach-Object { $_ -replace ';\d+$','' } | + Set-Content -Encoding utf8 clilocs.csv + ``` + + The header row needs no removal — a line whose first field is not an integer + is skipped. Blank entries (`1005008;`) survive the fixup correctly and are + dropped at import, as intended. + +5. Copy `clilocs.csv` to the server. + +**Why the fixup is not just done for us:** the parser already handles +`number,flag,text` — the flag in the *middle*, which is what several exports +emit. UOFiddler puts it at the *end*, where it is indistinguishable from a name +that genuinely ends in `;0`. One `sed` on the operator's side beats a parser +heuristic that would corrupt real names. + +### Point the site at it + +Two ways, the setting winning over the environment: + +| Where | How | +|---|---| +| **Admin → Shard → cliloc path** | Takes effect on the next refresh, no redeploy | +| `UO_CLIENT_PATH` env var | The deploy-time default | + +The value may be **the file itself or a directory to search** — both are natural +answers to "where is it", and overlays are picked up either way. + +Setting the path deliberately does **not** import as a side effect. Click +**Import** (or `POST /api/v1/admin/shard/clilocs/import`) to load it. + +### Verify + +`GET /api/v1/admin/shard/clilocs`, or the Admin → Shard panel, reports what each +source contributed: + +```json +"sources": [ + { "label": "clilocs.plain", "kind": "base", "entries": 123490, "added": 123490, "overrode": 0 } +] +``` + +Roughly **67,500 rows stored** from a stock table is correct — about half a +cliloc table is empty strings for ids the client reserves and never uses. + +Then load any character sheet with equipment: items should show names rather than +`id 1023721`. + +
+What a refusal means + +A bad file answers `200` with a `status` and a named reason, not a `500` — you +need to be told *which file* to fix. + +| `code` | Meaning | +|---|---| +| `COMPRESSED` | You pointed at the raw client `Cliloc.enu`. Convert it — this whole page. | +| `TRUNCATED` | Half-copied file. Re-copy; the loaded table is untouched. | +| `EMPTY` | A text source with no parseable rows — the file is named in the reason. | +| `status: needsReview` + `missingSources` | A previously-loaded source has vanished (unmounted volume? deliberate deletion?). Nothing changes until you re-import with `{ "approve": true }`. | + +
+ +### Custom items — do *not* re-export for these + +Shard-added items carry ids no client table has. Drop a small delimited file in a +`custom/` directory beside the base file and re-import: + +``` +/srv/uo-data/ + clilocs.plain ← base, from this guide + custom/ + 01-uomysticmoon.tsv ← your additions and overrides +``` + +Files are read in sorted order and **later sources win**, so an overlay both adds +new ids and overrides stock ones you have re-purposed. **Adding one item never +means re-exporting a 5 MB client file.** Details in [`CLILOCS.md`](CLILOCS.md). + +--- + +## Part 2 — Creature art for the spawn atlas (optional) + +**Goal:** put sprites on atlas pages. Purely cosmetic — the atlas is fully +functional as text, and `art` is NULL on every fresh import. + +**This project ships no art and no art-extraction tooling, and never will.** + +1. In `UoFiddler.exe` (paths configured as in Route B step 1), open the + **Animations** tab for creature sprites — or **Items** for object art — find + the creature, and export as PNG. Right-click an entry for its export options, + or use the tab's *Export All* action for a batch. (4.22.2 added an export + option to the Animation tab's thumbnail list, which is the convenient one + here.) +2. Put the images under `server/uploads/atlas/`. +3. Copy `server/db/data/spawnAtlas.art.example.json` to `spawnAtlas.art.json` in + the same directory and map creature slugs to file names: + + ```json + { + "lizardman": "lizardman.png", + "orc": "orc.png" + } + ``` + + **Keys are the slugs the atlas API reports**, derived from the type names in + your own shard's `Spawns/*.xml` — read them off the atlas rather than guessing. + A creature with no entry renders without art, which is the default. + +4. Restart, or `npm run atlas:import -- --force`. + +The art map is re-read on every atlas refresh, so adding one image is an edit plus +a refresh. Both `spawnAtlas.art.json` and `server/uploads/` are gitignored. + +--- + +## Licensing, briefly + +UO's strings and sprites are EA's. Extracting from **your own** client for +**your own** shard is the arrangement here; redistributing the extracted files is +not something this project does or can advise on. That is the whole reason this +page exists instead of a download link.