docs(link): the catalogue is real, and UOFiddler's last job is gone (Phase 3) #238

Merged
whitlocktech merged 2 commits from docs/asset-bridge-p3 into main 2026-09-10 23:57:20 +00:00
7 changed files with 226 additions and 160 deletions

View File

@@ -37,7 +37,6 @@ sidecar as a service, and hands you the values the website needs.
| [MODERATION_APPEALS.md](website/MODERATION_APPEALS.md) | Moderation actions, content reports and the appeals flow | | [MODERATION_APPEALS.md](website/MODERATION_APPEALS.md) | Moderation actions, content reports and the appeals flow |
| [SPAWN_ATLAS.md](website/SPAWN_ATLAS.md) | The bestiary / spawn atlas: what the shard contains, parsed from its own ServUO tree | | [SPAWN_ATLAS.md](website/SPAWN_ATLAS.md) | The bestiary / spawn atlas: what the shard contains, parsed from its own ServUO tree |
| [CLILOCS.md](website/CLILOCS.md) | UO's id → name table: converting one from your client so items have names | | [CLILOCS.md](website/CLILOCS.md) | UO's id → name table: converting one from your client so items have names |
| [UOFIDDLER.md](website/UOFIDDLER.md) | **Operator runbook** — extracting creature art from your own UO client. Shrinking: the cliloc half moved to the [Asset Bridge](link/v8.md) in protocol 8 |
| [MARKETPLACE.md](website/MARKETPLACE.md) | The player-vendor index: how it is gathered, what it costs, how to tune it | | [MARKETPLACE.md](website/MARKETPLACE.md) | The player-vendor index: how it is gathered, what it costs, how to tune it |
| [website-README.md](website/website-README.md) | Snapshot of the website repo's README (setup/run reference) | | [website-README.md](website/website-README.md) | Snapshot of the website repo's README (setup/run reference) |
| [test-plan.md](website/test-plan.md) | The website's test strategy and harness | | [test-plan.md](website/test-plan.md) | The website's test strategy and harness |

View File

@@ -10,8 +10,8 @@ direction not five, ServUO's own decoders, the UOP reader for player bodies, the
requirement). The phase 0 spike ran on 2026-09-10 and its findings are §4.5-§4.7 -- §4's decision requirement). The phase 0 spike ran on 2026-09-10 and its findings are §4.5-§4.7 -- §4's decision
stands, and the response to a malformed record is now measured rather than proposed. **Nothing in stands, and the response to a malformed record is now measured rather than proposed. **Nothing in
§17 is open.** §17 is open.**
**Supersedes the manual half of:** [`../website/UOFIDDLER.md`](../website/UOFIDDLER.md), **Supersedes the manual half of:** `../website/UOFIDDLER.md` (deleted in phase 3, once creature
[`../website/CLILOCS.md`](../website/CLILOCS.md) §Converting, art stopped needing it), [`../website/CLILOCS.md`](../website/CLILOCS.md) §Converting,
[`../website/SPAWN_ATLAS.md`](../website/SPAWN_ATLAS.md) §Artwork and §Configuring the tree. [`../website/SPAWN_ATLAS.md`](../website/SPAWN_ATLAS.md) §Artwork and §Configuring the tree.
Two features on this platform read data that only exists inside a UO client, and today both reach Two features on this platform read data that only exists inside a UO client, and today both reach
@@ -695,6 +695,46 @@ Constructing arbitrary creature types has side effects: constructors pack items,
timers. The mitigations are per-type `try`/`catch`, immediate `Delete()`, small batches, and the timers. The mitigations are per-type `try`/`catch`, immediate `Delete()`, small batches, and the
fact that the whole pass is admin-triggered rather than something that runs at boot. fact that the whole pass is admin-triggered rather than something that runs at boot.
### 8.1 What phase 3 measured
**Built and walked 2026-09-10** against a live ServUO with a real world (43,000 mobiles, 210,000
items) and the real sidecar.
**The catalogue is 787, exactly as §4.8 predicted**, and it arrives in one page: the whole scan of
bodies 12047 — index validation, 787 decodes, 787 PNG encodes and 787 SHA-256s — took **734 ms**
cold. That is well inside the 3 s scan budget, so the wall-clock paging §11 was designed for never
fired on this client. It stays, because the budget is what keeps a slower host or a larger family
inside the 10 s reply timeout rather than producing replies that are always thrown away.
Every prediction in §4.8 and §5.2 held when the bytes were actually rendered and looked at:
| Asked for | Answer | Why it matters |
|---|---|---|
| `body/320/a0` (`length 0`) | **absent** | The 357-class bug. The library would have returned the previously-decoded creature |
| `body/607/a0`, `body/608/a0` (elf ghosts) | **absent** | §4.8's two rows that moved; the library returns the elf female here |
| `body/666/a0` (gargoyle → `anim5`) | **absent** | The spider trap. Nothing swept, nothing found |
| `body/400/a0/…` deeper key | **unsupported** | Well-formed under §5, not served until phase 6 |
| direction distribution across 787 | **783 at index 1, 4 at index 0** | Four player bodies, not six — §4.8 again, from `Race.AllRaces` rather than a table |
**44 of the 787 hashes are shared by two or three bodies, and that is correct.** It is the exact
signature the wrong-picture bug produces, so it was chased rather than assumed: the client's own
`Body.def` says `83 {1}`, `84 {1}`, `138 {7}`, `139 {7}`, `106 {12, 59}`, and the sharing groups
match those lines rather than being runs of consecutive ids (which is what a reused stream buffer
produces). The distinguishing check is at the source: `Animations.Translate(ref body, ref hue)`
rewrites `body` **only when bit 31 of the table entry is set**, unlike the one-argument overload
which always does — and `BridgeAssetValidator.ResolveAnimation` calls that same two-argument
overload, matching `GetAnimation(…, preserveHue: false, …)`. Validator and decoder therefore
resolve the identical record, which is the property the whole §4.5 design rests on.
**The Core-thread pass costs about 190 ms per 100 types.** All 455 stock `Scripts/Mobiles/Normal`
classes were constructed and deleted in five chunks, producing every status the protocol defines
(`ok` 436, `unknown` 11, `notCreature` 4, `failed` 4). The world's mobile count went from 43,000 at
boot to 42,924 afterwards and its item count fell too — so `Delete()` is reclaiming the packed
inventory as well as the mobile, and nothing leaked.
The three refusals answer as designed end to end: an unknown `family` is **400**, a stale `catalog`
on a fetch is **422**, and 101 types in one `assets.bodies` is **400** naming the cap.
--- ---
## 9. The cliloc decompressor is ours now ## 9. The cliloc decompressor is ours now
@@ -712,7 +752,8 @@ Protocol 8 writes rather than calls (§4) — and from that point:
- No operator runs `dotnet build` on a converter. - No operator runs `dotnet build` on a converter.
- No operator copies a 5 MB file to a server. - No operator copies a 5 MB file to a server.
- `website/server/tools/cliloc-export/` is retired, and `UOFIDDLER.md` is deleted rather than - `website/server/tools/cliloc-export/` is retired, and `UOFIDDLER.md` is deleted rather than
rewritten. rewritten. (Phase 2 deleted its Part 1; **phase 3 deleted the page**, a phase earlier than that
section predicted, because creature art was the only thing left on it.)
### 9.1 What phase 2 built, and what the port cost ### 9.1 What phase 2 built, and what the port cost
@@ -900,6 +941,46 @@ extracted on their own host, for their own shard. Nothing is committed, nothing
and nothing is redistributed. What changes is only that the extraction stopped requiring a GUI on a and nothing is redistributed. What changes is only that the extraction stopped requiring a GUI on a
desktop. desktop.
### 12.1 The one thing above that phase 3 had to build differently
"`shard_spawn_creatures.art` … starts being filled by the import" is right about the outcome and
wrong about the mechanism, and the difference is not cosmetic.
**That table is emptied and refilled by every atlas refresh** — `shardAtlas.db.js`'s `replaceAtlas`
DELETEs all six atlas tables inside one transaction — and a refresh runs on **every boot**. Before
protocol 8 that cost nothing, because `art` came from a file on disk and was simply re-read each
time. An imported sprite is the opposite: expensive to obtain, and gated on client-file hashes that
would say "unchanged" for weeks afterwards. Writing it onto the creature row would mean an ordinary
re-parse of the ServUO tree silently deleting every portrait, with the next Update reporting
*nothing to do* and never restoring them. Nothing would report a fault; the pictures would just be
gone.
So phase 3 built three tables outside that blast radius, and the atlas import reads them on the way
past (org lead, 2026-09-10):
| Table | Holds | Lifetime |
|---|---|---|
| `shard_assets` | `asset_key`, `sha256`, `bytes`, `width`, `height`, `body`, `direction`, `file` | Upserted per key; only an approved vanish deletes |
| `shard_creature_bodies` | `slug`, `type_name`, `body`, `status` | Replaced whole — it is derived from the atlas's creature list, so a slug that has left has no meaning |
| `shard_asset_meta` | The singleton an Update compares against | Replaced |
`replaceAtlas` now takes `{ ...derived, ...operatorMap }`, which is where "the operator's map wins"
is actually enforced — one spread, in one place, applied on every rebuild rather than only at
import time.
**Two details worth not rediscovering.** The derivation joins on the catalogue key
(`a.asset_key = CONCAT('body/', b.body, '/a0')`), not on `a.body = b.body`: today one body has
exactly one asset and the simpler join is correct, and it stops being correct the moment phase 6
adds `body/400/a2/f0`, at which point one slug matches dozens of rows and whichever the engine
returned last becomes the portrait. And the stored filename is **content-addressed**
(`uo-body-34-a0-<sha8>.png`), because a stable name overwritten in place leaves every browser and
CDN serving last month's client's sprite from cache with the database row perfectly correct.
`shard_creature_bodies` also answers §8 without a schema change on the atlas side:
`shard_spawn_creatures.name` already holds the ServUO **class name** — the atlas build picks the
winning spelling of the spawn type token rather than inventing a display label — so the import has
something `ScriptCompiler.FindTypeByName` will resolve without storing it twice.
--- ---
## 13. Visibility ## 13. Visibility
@@ -921,9 +1002,9 @@ new pipe.
| Command | Reply | Purpose | Built | | Command | Reply | Purpose | Built |
|---|---|---|---| |---|---|---|---|
| `assets.sources` | `assets.sources.ok` | Stage 1: client file manifest + `EXTRACTOR_VERSION` | **phase 1** | | `assets.sources` | `assets.sources.ok` | Stage 1: client file manifest + `EXTRACTOR_VERSION` | **phase 1** |
| `assets.manifest` | `assets.manifest.ok` | Stage 2: `[{key, sha256, bytes}]`, paged | phase 3 | | `assets.manifest` | `assets.manifest.ok` | Stage 2: `[{key, sha256, bytes, width, height, body, direction}]`, paged | **phase 3** |
| `assets.fetch` | `assets.fetch.ok` | Content for an explicit key list, paged | phase 3 | | `assets.fetch` | `assets.fetch.ok` | Content for an explicit key list, paged; base64 PNG per row | **phase 3** |
| `assets.bodies` | `assets.bodies.ok` | Slug → body id (§8, Core thread) | phase 3 | | `assets.bodies` | `assets.bodies.ok` | Slug → body id (§8, Core thread) | **phase 3** |
| `cliloc.table` | `cliloc.table.ok` | The decompressed table, paged (`?lang=`, `?cursor=`) | **phase 2** | | `cliloc.table` | `cliloc.table.ok` | The decompressed table, paged (`?lang=`, `?cursor=`) | **phase 2** |
| `tree.manifest` / `tree.fetch` | `.ok` | §10, the ServUO tree files | phase 7 | | `tree.manifest` / `tree.fetch` | `.ok` | §10, the ServUO tree files | phase 7 |
@@ -938,9 +1019,21 @@ ordinary answer mid-import rather than a rare one) and reads the refusal's `code
**403** disabled, **404** no such file, **422** a file the shard cannot decode, **503** a shard that **403** disabled, **404** no such file, **422** a file the shard cannot decode, **503** a shard that
cannot do this now (§9.1). cannot do this now (§9.1).
Phase 3 added the other three: `GET /assets/manifest?family=&cursor=`, `POST /assets/fetch` and
`POST /assets/bodies`. **The two POSTs are reads**, and the method is the request body rather than a
side effect — a few hundred asset keys do not belong in a query string. They are the only reads on
this link that take one. `422` gains a second meaning on this plane alongside "cannot decode": the
mid-import guard, a `catalog` that no longer describes the files on disk.
**§16 listed phase 3 as `servuo-plugins, module-uo` and that was wrong** — `web.rs` routes every
command explicitly and has no generic `/assets/*` forwarder, so `link` is in the phase too. The
table now says so.
**Website admin** (`Admin → Shard`, admin-only): status, **Import**, **Update**, approve/reject for **Website admin** (`Admin → Shard`, admin-only): status, **Import**, **Update**, approve/reject for
a vanished key, and the existing path settings. Every action to the admin activity log, as a vanished key, and the existing path settings. Every action to the admin activity log, as
`shard.assets.*`. `shard.assets.*`. Phase 3 shipped the two that make it reachable — `GET /admin/shard/assets` and
`POST /admin/shard/assets/import`, mirroring the cliloc pair — so the phase could be accepted on a
real rig; the full panel is phase 8 (org lead, 2026-09-10).
--- ---
@@ -970,7 +1063,7 @@ disagree, so a split bump means the next bundle silently fails to compose.
| 0 | **DONE 2026-09-10.** Spike: the vendored decoders driven from inside a running ServUO, over a client broken 21 ways. §4 stands; the finding was 22,102 wrong pictures on a *stock* client, and the validator that answers them (§4.5-§4.7) | servuo-plugins | | 0 | **DONE 2026-09-10.** Spike: the vendored decoders driven from inside a running ServUO, over a client broken 21 ways. §4 stands; the finding was 22,102 wrong pictures on a *stock* client, and the validator that answers them (§4.5-§4.7) | servuo-plugins |
| 1 | **DONE 2026-09-10.** The transport: `assets.sources`, the single-slot gate (§3.2), the paging envelope (§3.4), the sidecar line cap (§3.3), `EXTRACTOR_VERSION`, `NO_IMAGING`, protocol 7→8. Plus §4.5's validator promoted into the overlay and extended to animations — which found **357 wrong pictures in the body catalogue on a stock client** and cut it from 1,144 to 787 (§4.8) | servuo-plugins, link | | 1 | **DONE 2026-09-10.** The transport: `assets.sources`, the single-slot gate (§3.2), the paging envelope (§3.4), the sidecar line cap (§3.3), `EXTRACTOR_VERSION`, `NO_IMAGING`, protocol 7→8. Plus §4.5's validator promoted into the overlay and extended to animations — which found **357 wrong pictures in the body catalogue on a stock client** and cut it from 1,144 to 787 (§4.8) | servuo-plugins, link |
| 2 | **DONE 2026-09-10.** Clilocs end to end (§9.1, §9.2): the Mythic decompressor ported into the overlay, `cliloc.table` + `GET /cliloc`, the paging walk and the source switch on the website, module-uo's protocol pin 7→8. `cliloc-export/` deleted and `UOFIDDLER.md` §Part 1 with it. **67,496 rows, 290 ms, ~11 pages** — the same count UOFiddler's own DLL produced from this client | all | | 2 | **DONE 2026-09-10.** Clilocs end to end (§9.1, §9.2): the Mythic decompressor ported into the overlay, `cliloc.table` + `GET /cliloc`, the paging walk and the source switch on the website, module-uo's protocol pin 7→8. `cliloc-export/` deleted and `UOFIDDLER.md` §Part 1 with it. **67,496 rows, 290 ms, ~11 pages** — the same count UOFiddler's own DLL produced from this client | all |
| 3 | Body resolution (§8) + the **787**-body catalogue (§4.8); `shard_spawn_creatures.art` filled | servuo-plugins, module-uo | | 3 | **DONE 2026-09-10.** Body resolution (§8) + the **787**-body catalogue (§4.8), `assets.manifest` / `assets.fetch` / `assets.bodies` and their REST mirrors, `shard_spawn_creatures.art` filled and rendered (§8.1, §12.1). **787 rows in one 734 ms page; 455 types resolved at ~190 ms per 100 on the Core thread; zero mobiles leaked.** `UOFIDDLER.md` deleted, two phases early | servuo-plugins, **link**, module-uo |
| 4 | **The UOP animation decoder (§4.3)** — the **eight** player bodies the legacy path cannot reach (§4.8 moved the elf ghosts across), the never-sweep rule already enforced in code by phase 1's `ResolveAnimation` | servuo-plugins | | 4 | **The UOP animation decoder (§4.3)** — the **eight** player bodies the legacy path cannot reach (§4.8 moved the elf ghosts across), the never-sweep rule already enforced in code by phase 1's `ResolveAnimation` | servuo-plugins |
| 5 | Item statics and land on demand, hued keys, the cache | servuo-plugins, module-uo | | 5 | Item statics and land on demand, hued keys, the cache | servuo-plugins, module-uo |
| 6 | Deep animation keys (`body/<id>/a<n>/f<n>`) for the future project, plus the bulk-fill switch | servuo-plugins, module-uo | | 6 | Deep animation keys (`body/<id>/a<n>/f<n>`) for the future project, plus the bulk-fill switch | servuo-plugins, module-uo |

View File

@@ -23,7 +23,7 @@ knowing a module answers now.
|---|---|---| |---|---|---|
| `/api/v1/public/shard` | 19 | Anonymous. **Never site-mode gated** — the shard surface stays readable during maintenance, per feature audience. | | `/api/v1/public/shard` | 19 | Anonymous. **Never site-mode gated** — the shard surface stays readable during maintenance, per feature audience. |
| `/api/v1/public/atlas` | 6 | Anonymous, and unlike `/shard` it **is** site-mode gated: nothing here touches the sidecar, it is parsed shard content. | | `/api/v1/public/atlas` | 6 | Anonymous, and unlike `/shard` it **is** site-mode gated: nothing here touches the sidecar, it is parsed shard content. |
| `/api/v1/admin/shard` | 26 | Behind core's `isLoggedIn + noindex + staffOnly` group gate, then **mixed per route** — see below. | | `/api/v1/admin/shard` | 28 | Behind core's `isLoggedIn + noindex + staffOnly` group gate, then **mixed per route** — see below. |
| `/api/v1/admin/uo-link` | 7 | `adminOnly`. The sidecar connection config, its live status, the admin SSE stream and the town crier. | | `/api/v1/admin/uo-link` | 7 | `adminOnly`. The sidecar connection config, its live status, the admin SSE stream and the town crier. |
| `/api/v1/player/shard` | 8 | `requireAuth`, **any role** — staff are a superset of players — and every handler is self-scoped to `req.user.id`. | | `/api/v1/player/shard` | 8 | `requireAuth`, **any role** — staff are a superset of players — and every handler is self-scoped to `req.user.id`. |
| `/api/v1/admin/users/:id/shard/*` | 6 | `adminOnly`. The module's routes hanging off a **core** resource, through core's `admin.users.detail` extension slot: core owns the user, the module owns what it knows about their game accounts. | | `/api/v1/admin/users/:id/shard/*` | 6 | `adminOnly`. The module's routes hanging off a **core** resource, through core's `admin.users.detail` extension slot: core owns the user, the module owns what it knows about their game accounts. |
@@ -81,6 +81,8 @@ account-linking routes and the sidecar config under `/admin/uo-link` are in the
| PUT | `/shard/atlas/path` | point the atlas at a different tree (persisted as `spawn_atlas_servuo_path`, which wins over `SERVUO_PATH`). Blank clears it. Deliberately **does not import** — moving the mount and reloading the world are separate decisions — and returns fresh status so the panel can offer the import next. | | PUT | `/shard/atlas/path` | point the atlas at a different tree (persisted as `spawn_atlas_servuo_path`, which wins over `SERVUO_PATH`). Blank clears it. Deliberately **does not import** — moving the mount and reloading the world are separate decisions — and returns fresh status so the panel can offer the import next. |
| GET | `/shard/clilocs` | cliloc-table status (`adminOnly`): every source found now (base first, then `custom/` overlays in merge order), what each contributed at the last import, readability, drift across the set, the entry count, and `missingSources`. `configured:false` is a supported state — item names then render as ids. No public counterpart: the table is never served *as* a table. | | GET | `/shard/clilocs` | cliloc-table status (`adminOnly`): every source found now (base first, then `custom/` overlays in merge order), what each contributed at the last import, readability, drift across the set, the entry count, and `missingSources`. `configured:false` is a supported state — item names then render as ids. No public counterpart: the table is never served *as* a table. |
| POST | `/shard/clilocs/import` | reload after a client patch or an overlay edit; `{force}` ignores the hash gate, `{approve}` accepts a **vanished** source (refused by default — see the table notes above). **A missing path — or the likely mistake of pointing at the client's own COMPRESSED `Cliloc.enu` — answers 200 with `status:"unavailable"` and a `code`, not 500.** `COMPRESSED` is called out by name: a 500 would say only "something broke", and the operator needs to be told which file to convert. | | POST | `/shard/clilocs/import` | reload after a client patch or an overlay edit; `{force}` ignores the hash gate, `{approve}` accepts a **vanished** source (refused by default — see the table notes above). **A missing path — or the likely mistake of pointing at the client's own COMPRESSED `Cliloc.enu` — answers 200 with `status:"unavailable"` and a `code`, not 500.** `COMPRESSED` is called out by name: a 500 would say only "something broke", and the operator needs to be told which file to convert. |
| GET | `/shard/assets` | client-asset import status (`adminOnly`, Protocol 8): the imported body catalogue, how many sprites are on disk, how many atlas creatures resolved to a body id, and the shard's own client files beside them. `drift:true` means the client was patched. `shard.imaging.ok:false` is the named `NO_IMAGING` state — a Linux shard host with no `libgdiplus` cannot decode a sprite at all, and the reason names the package. No public counterpart: the pictures are served as ordinary files under `/uploads`. |
| POST | `/shard/assets/import` | import creature artwork from the shard's UO client; `{force}` ignores the hash gate, `{approve}` accepts a catalogue that no longer offers assets this site holds (refused by default — an unmounted client volume and a deliberate downgrade are indistinguishable, and the wrong guess deletes artwork). **This is the ONLY thing that imports** — boot deliberately never calls the shard. An operator's `spawnAtlas.art.json` always wins over an imported sprite. A body this client has no art for is **not** a failure: two thirds of the playable ghost and gargoyle bodies are in that state on a stock client. |
| PUT | `/shard/clilocs/path` | point the site at a different cliloc base file or directory (persisted as `cliloc_client_path`, which wins over `UO_CLIENT_PATH`). Overlays are read from `custom/` beside it either way. Blank clears it. Deliberately **does not import**, same reasoning as the atlas path. | | PUT | `/shard/clilocs/path` | point the site at a different cliloc base file or directory (persisted as `cliloc_client_path`, which wins over `UO_CLIENT_PATH`). Overlays are read from `custom/` beside it either way. Blank clears it. Deliberately **does not import**, same reasoning as the atlas path. |
## 4. Shard visibility — the audience boundary (Protocol 3.0) ## 4. Shard visibility — the audience boundary (Protocol 3.0)

View File

@@ -21,7 +21,6 @@ these routes *mean* are the ones that already existed and did not move:
| [`SPAWN_ATLAS.md`](../../website/SPAWN_ATLAS.md) | The bestiary / spawn atlas, parsed from the shard's own ServUO tree | | [`SPAWN_ATLAS.md`](../../website/SPAWN_ATLAS.md) | The bestiary / spawn atlas, parsed from the shard's own ServUO tree |
| [`MARKETPLACE.md`](../../website/MARKETPLACE.md) | The player-vendor index | | [`MARKETPLACE.md`](../../website/MARKETPLACE.md) | The player-vendor index |
| [`CLILOCS.md`](../../website/CLILOCS.md) | UO's id → name table | | [`CLILOCS.md`](../../website/CLILOCS.md) | UO's id → name table |
| [`UOFIDDLER.md`](../../website/UOFIDDLER.md) | Operator runbook for extracting creature art. The cliloc table now arrives over the bridge ([`link/v8.md`](../../link/v8.md) §9) |
| [`../../link/PLAN.md`](../../link/PLAN.md), [`../../link/INTEGRATION.md`](../../link/INTEGRATION.md) | The wire protocol this module speaks to the sidecar | | [`../../link/PLAN.md`](../../link/PLAN.md), [`../../link/INTEGRATION.md`](../../link/INTEGRATION.md) | The wire protocol this module speaks to the sidecar |
--- ---

View File

@@ -188,10 +188,52 @@ Four column choices worth stating, because each one is a trap:
Deceit"). The live `champ.update` feed in `shard_champs` is the separate answer to "it is on level 3 Deceit"). The live `champ.update` feed in `shard_champs` is the separate answer to "it is on level 3
right now". Both exist; they are not the same data. right now". Both exist; they are not the same data.
**`shard_spawn_creatures.art` is always NULL on a fresh import.** The project ships no creature **`shard_spawn_creatures.art` is DERIVED, never written by the atlas import itself** — see
artwork: sprites live in the operator's own client `.mul`/`.uop` files and are theirs, not ours to `shard_assets` below. The project still ships no creature artwork: sprites live in the operator's own
redistribute. An operator supplies art via a gitignored map plus images under the (already client `.mul`/`.uop` files and are theirs, not ours to redistribute. What changed in Protocol 8 is
gitignored) `server/uploads/atlas/`. Text-only is the normal, supported state. who extracts them: the shard does, from its own client, over the bridge. An operator's hand-drawn map
plus images under the (already gitignored) `server/uploads/atlas/` still wins over anything imported.
Text-only is still the normal, supported state — an install with no shard link never imports one, and
even a complete import leaves two thirds of the playable ghost and gargoyle bodies without art.
## shard_assets / shard_creature_bodies / shard_asset_meta — the Asset Bridge (Protocol 8)
Creature artwork read from the shard's own UO client ([`link/v8.md`](../../link/v8.md) §6, §8, §12).
| Table | Shape |
|---|---|
| `shard_assets` | `asset_key` VARCHAR PK (§5's key, e.g. `body/34/a0`), `family`, `sha256`, `bytes`, `width`, `height`, `body`, `direction`, `file`, `imported_at` |
| `shard_creature_bodies` | `slug` PK, `type_name` (the ServUO class name asked), `body` nullable, `status`, `resolved_at` |
| `shard_asset_meta` | Singleton (`id = 1`), `payload` JSON (catalogue id, extractor version, source fingerprint, counts), `imported_at` |
**These are the one part of this schema that is deliberately NOT import-owned**, and the reason is
the atlas tables sitting directly above them. `replaceAtlas` empties and refills
`shard_spawn_creatures` on every refresh, and a refresh runs on every boot; an imported filename
stored on that row would be destroyed by an ordinary re-parse of the ServUO tree, with the next asset
Update finding the client files unchanged, reporting "nothing to do", and never restoring it. So the
assets live out here, upserted per key, and `replaceAtlas` re-derives `art` from them on the way
past — `{ ...derived, ...operatorMap }`, which is the one place "the operator's map wins" is
enforced.
Four details that are load-bearing rather than incidental:
- **`file` is a filename under the uploads directory, never a path**, and it is content-addressed
(`uo-body-34-a0-<sha8>.png`). A stable name overwritten in place would leave every browser and CDN
serving the previous client's sprite from cache, with the row perfectly correct.
- **The `art` derivation joins on the catalogue key**, `a.asset_key = CONCAT('body/', b.body, '/a0')`,
not `a.body = b.body`. The simpler join is correct today and stops being correct the moment deeper
animation keys (`body/400/a2/f0`) arrive, at which point one slug matches dozens of rows.
- **`shard_creature_bodies` IS replaced whole**, unlike `shard_assets`: it is derived from the atlas's
creature list, so a slug that has left the atlas has no meaning, and the pass that rebuilds it is a
shard round trip rather than a file transfer.
- **`status` keeps the negative answers** — `unknown` (the spawn files name a type this shard's
scripts do not define, which is real drift), `notCreature` (a spawn entry for an item or
decoration, a permanent answer), `failed`. Without them the next pass asks again, and each name
costs a real constructor on the shard's Core thread.
`shard_spawn_creatures.name` already holds the ServUO **class name** — the atlas build picks the
winning spelling of the spawn type token rather than inventing a display label — which is why the
body pass needs no extra column to ask its question.
## shard_clilocs / shard_cliloc_meta — UO's localization table (Protocol 3.0) ## shard_clilocs / shard_cliloc_meta — UO's localization table (Protocol 3.0)
@@ -203,29 +245,49 @@ marketplace listing — but with no table to resolve it against, the character s
| Table | Shape | | Table | Shape |
|---|---| |---|---|
| `shard_clilocs` | `number` INT PK, `flag`, `text` TEXT | | `shard_clilocs` | `number` INT PK, `flag`, `text` TEXT |
| `shard_cliloc_meta` | Singleton (`id = 1`), `payload` JSON (source file, sha256, count, `parserVersion`), `imported_at` | | `shard_cliloc_meta` | Singleton (`id = 1`), `payload` JSON (`source``bridge` or `file` — the base's fingerprint under `base`, the overlay `hashes`, per-source counts, `parserVersion`), `imported_at` |
Import-owned and all-or-nothing in one transaction, same contract as the atlas — including **`DELETE`, Import-owned and all-or-nothing in one transaction, same contract as the atlas — including **`DELETE`,
not `TRUNCATE`**, for the same reason. not `TRUNCATE`**, for the same reason.
**Sourced from files the operator supplies**, at a path from the `cliloc_client_path` setting falling **The base table comes from the SHARD** on any install with uo-link configured (Protocol 8, phase 2):
back to `UO_CLIENT_PATH`. Nothing client-derived is committed: UO's strings are EA's, exactly as the it reads its own client's compressed `Cliloc.enu` and serves the table paged over the bridge, so
creature sprites are. A shard with nothing configured is fully supported — names render as ids. Full nothing is converted and nothing is copied to the web host. Without a shard link it falls back to a
design and operator guide: [`CLILOCS.md`](../../website/CLILOCS.md). converted file on disk at a path from the `cliloc_client_path` setting, then `UO_CLIENT_PATH` — the
pre-protocol-8 pipeline, deprecated rather than removed so an existing install keeps working.
**Overlays are always the filesystem's**, either way: ServUO has no server-side notion of a custom
cliloc, so `custom/` is the only place shard-added ids exist and there is nothing on the shard to ask
for. That gap is in the *game*, not in this pipeline.
Nothing client-derived is committed: UO's strings are EA's, exactly as the creature sprites are. A
shard with nothing configured is fully supported — names render as ids. Full design and operator
guide: [`CLILOCS.md`](../../website/CLILOCS.md).
**It reads a SET of sources, not one file**, because shards edit items and add new ones and those **It reads a SET of sources, not one file**, because shards edit items and add new ones and those
carry cliloc ids no stock client table has. A base (the converted client table) plus every overlay carry cliloc ids no stock client table has. A base (from the shard, or a converted file) plus every
under `custom/` are re-read on every boot and hash-gated **together**, exactly as the atlas re-reads overlay under `custom/` are hash-gated **together**, exactly as the atlas re-reads
`Regions.xml` + `Locations/*.xml` + `Spawns/*.xml` + `ChampionSpawns.xml`. Later sources win, so an `Regions.xml` + `Locations/*.xml` + `Spawns/*.xml` + `ChampionSpawns.xml`. Later sources win, so an
overlay both adds ids and overrides stock ones, and adding one custom item never means re-exporting a overlay both adds ids and overrides stock ones, and adding one custom item never means re-exporting a
5 MB client file. Scale, measured on the live shard: its script tree references 16,434 cliloc ids and 5 MB client file. Scale, measured on the live shard: its script tree references 16,434 cliloc ids and
only 37 are absent from stock — tens of entries against a 67k base, which is why this is an overlay only 37 are absent from stock — tens of entries against a 67k base, which is why this is an overlay
and not a second table. and not a second table.
The conversion step is not avoidable: **every current client ships its cliloc files compressed** **The conversion step used to be unavoidable, and is not any more.** Every current client ships its
(first DWORD's high byte `0x8E`), and ServUO's own bundled `Ultima.StringList` cannot read that cliloc files compressed (first DWORD's high byte `0x8E`) and ServUO's own bundled
either — so the shard cannot supply names on our behalf. The plain layout and a delimited text export `Ultima.StringList` cannot read that either — which is why, for two protocol versions, the operator
are both accepted, sniffed by header rather than extension. had to install UOFiddler, build a converter against its `Ultima.dll` and copy a 5 MB file to the web
host. Protocol 8 phase 2 ported the Mythic decompressor into the overlay, so **the shard reads its
own client and supplies the names**. The file half survives only as the fallback above, where the
plain layout and a delimited text export are both accepted, sniffed by header rather than extension.
Two consequences for what this table holds. `shard_cliloc_meta.payload` carries the base's
fingerprint under `meta.base` (the shard's file size, mtime, hash and `EXTRACTOR_VERSION`) separately
from the overlay hashes, because on the bridge the old `clilocs.plain` label is *supposed* to
disappear and a single hash map would read that upgrade as a vanished source. And **boot does not
import on the bridge path**: a file could be re-hashed locally on every restart, but asking the shard
would put a sidecar round trip in the boot sequence for a table that changes only when an operator
patches their client. Importing is an admin action.
Three decisions worth stating: Three decisions worth stating:

View File

@@ -213,28 +213,59 @@ implicitly commit, defeating the all-or-nothing guarantee. Point ids are assigne
explicitly rather than left to `AUTO_INCREMENT`, because the join rows need them explicitly rather than left to `AUTO_INCREMENT`, because the join rows need them
and `conn.batch()` reports no usable `insertId`. and `conn.batch()` reports no usable `insertId`.
## Artwork — operator-supplied, never shipped ## Artwork — the shard extracts it now (Protocol 8)
**This project ships no creature art and no extraction tooling, and never will.** **This project ships no creature art and no extraction tooling, and never will.**
UO sprites live in the operator's own client `.mul`/`.uop` files. They are the UO sprites live in the operator's own client `.mul`/`.uop` files. They are the
operator's, not ours to redistribute. operator's, not ours to redistribute.
The atlas is fully functional as text. `shard_spawn_creatures.art` is nullable What changed in protocol 8 is not that rule — it is who does the extracting. The
and is NULL on every fresh import; pages render without images, which is the shard already has those files (a ServUO server cannot boot without a UO client),
normal and supported state, not a degraded one. so as of [`v8.md`](../link/v8.md) phase 3 it decodes them itself and hands the
pictures to the website over the bridge. Nobody installs UOFiddler and nobody
copies images to a web host.
An operator who wants art — step-by-step, with the UOFiddler side spelled out, in **Admin → Shard → Import.** The import walks the shard's asset manifest, fetches
[`UOFIDDLER.md`](UOFIDDLER.md) §Part 2: only the sprites whose hash changed, writes them under `uploads/atlas/`, asks the
shard for a body id per creature (§8 — the shard constructs the creature and
reads `Body.BodyID`, which is the only thing that is right for a shard's own
custom creatures) and points each `shard_spawn_creatures.art` at its picture.
Boot never calls the shard for this: the files change when an operator patches
their client, which is an event they know about and the site does not.
1. Extracts it from **their own** client files (UOFiddler, ClassicUO tooling, or On this machine's stock client that is **787 creature portraits**, about a
any art extractor). megabyte in total.
2. Drops the images under `server/uploads/atlas/`.
3. Copies `server/db/data/spawnAtlas.art.example.json` to `spawnAtlas.art.json`
and maps creature slugs to file names.
4. Restarts, or runs `npm run atlas:import -- --force`.
Both `spawnAtlas.art.json` and `server/uploads/` are gitignored, so neither the **NULL stays a first-class state, and always will be.** An install with no shard
map nor the images can be committed by accident. link has never imported one; a Linux shard host without `libgdiplus` cannot
render a sprite at all (a named `NO_IMAGING` status, not an error); and even
after a complete import, two thirds of the playable ghost and gargoyle bodies
have no art in the client. Pages render without images, which is normal and
supported, not degraded.
### The operator's own artwork still wins
An operator who has drawn their own portraits keeps them. The map is unchanged:
1. Drop the images under `server/uploads/atlas/`.
2. Copy `server/db/data/spawnAtlas.art.example.json` to `spawnAtlas.art.json` and
map creature slugs to file names.
3. Restart, or run the import.
`spawnAtlas.art.json` is applied **over** anything imported, per slug, so a sprite
rip never replaces a hand-drawn portrait on the next Update. Both it and
`server/uploads/` are gitignored, so neither the map nor the images can be
committed by accident.
### Why the imported art is not stored on the creature row
`shard_spawn_creatures` is emptied and refilled by every atlas refresh, and a
refresh happens on every boot. So the body ids and the imported files live in
`shard_creature_bodies` and `shard_assets`, outside that blast radius, and the
atlas import re-derives `art` from them on the way past. Storing it on the row
would mean an ordinary re-parse of the ServUO tree silently deleting every
portrait — with the next asset Update finding the client files unchanged,
reporting "nothing to do", and never putting them back.
## Code layout ## Code layout

View File

@@ -1,120 +0,0 @@
# Extracting from your own UO client (UOFiddler)
**Audience:** the shard operator, once, at setup time.
**Related:** [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md) (where creature art fits),
[`../link/v8.md`](../link/v8.md) (the Asset Bridge, which is replacing this page).
> ### The cliloc conversion is gone — you no longer do this by hand
>
> **Protocol 8, phase 2.** The shard reads its own client's `Cliloc.enu` and
> serves the table over the bridge, so there is nothing to install, convert or
> copy. Press **Admin → Shard → Import** after you patch your client and that is
> the whole procedure; see [`CLILOCS.md`](CLILOCS.md).
>
> Part 1 of this guide has been deleted rather than rewritten. If you already
> have a converted file it keeps working on an install with no uo-link
> configured, but nobody should make a new one.
>
> **Creature art is next** (phase 5), after which this page goes away entirely.
What remains here reads data that **only exists inside a UO client**, and a UO
client's files are EA's, not ours to redistribute. So neither this repo nor any
image we publish can ship them — the operator extracts from **their own** client
and points the site at the result.
| Feature | What it needs | Required? | Without it |
|---|---|---|---|
| **Creature art** ([`SPAWN_ATLAS.md`](SPAWN_ATLAS.md)) | Sprites from `.mul`/`.uop` | No | Atlas pages render as text, which is the normal state |
**It is optional and it is not load-bearing.** A shard that never does any of this
is fully supported.
Everything you extract stays **outside the repository**: `spawnAtlas.art.json` and
`server/uploads/` are gitignored, so none of it can be committed by accident.
---
## Part 0 — Get UOFiddler
[UOFiddler](https://github.com/polserver/UOFiddler) is the community client-file
editor. What is left here uses its GUI to export sprites; the `Ultima.dll` this
guide used to reach for was the cliloc decompressor, and the shard has its own
now.
1. Download the latest release zip from
<https://github.com/polserver/UOFiddler/releases/latest> — one asset, named
`UOFiddler-<version>.zip` (4.22.2 is ~2 MB).
2. Extract it. The zip contains a single top-level folder, and the two files that
matter are at **its root**:
```
UOFiddler-4.22.2/
UoFiddler.exe ← the GUI (Part 2 needs this)
Ultima.dll
plugins/
```
3. **Runtime:** UOFiddler 4.22.2 is built for **.NET 10**, so running
`UoFiddler.exe` needs the .NET 10 **Desktop** Runtime (Windows only). Install
from <https://dotnet.microsoft.com/download/dotnet/10.0>.
### Finding your client files
The art files are in your **UO client installation directory**. Look for
`art.mul` / `artLegacyMUL.uop` and the `anim*.mul` set. The EA Classic Client's
default location is:
```
C:\Program Files (x86)\Electronic Arts\Ultima Online Classic\
```
**If your shard distributes its own patched client to players, use that copy** —
it is what your players actually see. (It is also the copy the shard itself reads
from, since a ServUO server resolves `Config/DataPath.cfg` into its own client
path at boot, which is the premise the Asset Bridge is built on.)
---
## Part 2 — Creature art for the spawn atlas (optional)
**Goal:** put sprites on atlas pages. Purely cosmetic — the atlas is fully
functional as text, and `art` is NULL on every fresh import.
**This project ships no art and no art-extraction tooling, and never will.**
1. Open `UoFiddler.exe` and point it at your client directory
(**Settings → Paths**), then open the
**Animations** tab for creature sprites — or **Items** for object art — find
the creature, and export as PNG. Right-click an entry for its export options,
or use the tab's *Export All* action for a batch. (4.22.2 added an export
option to the Animation tab's thumbnail list, which is the convenient one
here.)
2. Put the images under `server/uploads/atlas/`.
3. Copy `server/db/data/spawnAtlas.art.example.json` to `spawnAtlas.art.json` in
the same directory and map creature slugs to file names:
```json
{
"lizardman": "lizardman.png",
"orc": "orc.png"
}
```
**Keys are the slugs the atlas API reports**, derived from the type names in
your own shard's `Spawns/*.xml` — read them off the atlas rather than guessing.
A creature with no entry renders without art, which is the default.
4. Restart, or `npm run atlas:import -- --force`.
The art map is re-read on every atlas refresh, so adding one image is an edit plus
a refresh. Both `spawnAtlas.art.json` and `server/uploads/` are gitignored.
---
## Licensing, briefly
UO's strings and sprites are EA's. Extracting from **your own** client for
**your own** shard is the arrangement here; redistributing the extracted files is
not something this project does or can advise on. That is the whole reason this
page exists instead of a download link.