From 1a7481e9f4c35137f8d3a461334ec1af6994fd58 Mon Sep 17 00:00:00 2001 From: wtclaude Date: Thu, 10 Sep 2026 18:41:08 -0500 Subject: [PATCH 1/2] docs(link): the catalogue is real, and UOFiddler's last job is gone (Phase 3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 3 is built and walked on a live shard. What the walk measured, and the two places the design of record needed correcting: §8.1, new: the catalogue is 787 exactly as §4.8 predicted, and the whole scan of bodies 1-2047 takes 734 ms cold -- so the wall-clock paging §11 designed never fires on this client. Every §4.8/§5.2 prediction held when the bytes were rendered and LOOKED at: 320, 607, 608 and 666 come back absent rather than as another creature's picture, and the direction split is 783 at index 1 against 4 at index 0 -- four player bodies, not six. 44 of the 787 hashes are shared by two or three bodies, which is the exact signature of the wrong-picture bug, so it was chased rather than assumed. It is the client's own Body.def aliasing (83 {1}, 84 {1}, 106 {12, 59}), and the check that settles it is at the source: Translate(ref body, ref hue) rewrites `body` only when bit 31 is set, unlike the one-argument overload -- and ResolveAnimation calls that same two-argument overload, so validator and decoder resolve the identical record. §12.1, new: **§12 is right about the outcome and wrong about the mechanism.** `shard_spawn_creatures` is emptied and refilled by every atlas refresh, and a refresh runs on every boot -- so an imported filename written to that row is destroyed by an ordinary re-parse of the ServUO tree, and the next Update finds the client files unchanged and never restores it. Three tables outside that blast radius, and the atlas import re-derives `art` on the way past. §14: **§16 listed phase 3 as servuo-plugins + module-uo and that was wrong.** web.rs routes every command explicitly, so `link` is in the phase. Corrected in both places. UOFIDDLER.md is DELETED, two phases earlier than §9.1 predicted -- creature art was the only thing still on it. SPAWN_ATLAS.md §Artwork is rewritten around the import, keeping the operator's own map as the thing that wins; the module's SCHEMA.md gains the three tables and API.md the two admin routes. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4 --- README.md | 1 - link/v8.md | 109 ++++++++++++++++++++++++++++++++++--- modules/uo/API.md | 4 +- modules/uo/README.md | 1 - modules/uo/SCHEMA.md | 50 +++++++++++++++-- website/SPAWN_ATLAS.md | 59 +++++++++++++++----- website/UOFIDDLER.md | 120 ----------------------------------------- 7 files changed, 195 insertions(+), 149 deletions(-) delete mode 100644 website/UOFIDDLER.md diff --git a/README.md b/README.md index c9e7723..6b65120 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,6 @@ sidecar as a service, and hands you the values the website needs. | [MODERATION_APPEALS.md](website/MODERATION_APPEALS.md) | Moderation actions, content reports and the appeals flow | | [SPAWN_ATLAS.md](website/SPAWN_ATLAS.md) | The bestiary / spawn atlas: what the shard contains, parsed from its own ServUO tree | | [CLILOCS.md](website/CLILOCS.md) | UO's id → name table: converting one from your client so items have names | -| [UOFIDDLER.md](website/UOFIDDLER.md) | **Operator runbook** — extracting creature art from your own UO client. Shrinking: the cliloc half moved to the [Asset Bridge](link/v8.md) in protocol 8 | | [MARKETPLACE.md](website/MARKETPLACE.md) | The player-vendor index: how it is gathered, what it costs, how to tune it | | [website-README.md](website/website-README.md) | Snapshot of the website repo's README (setup/run reference) | | [test-plan.md](website/test-plan.md) | The website's test strategy and harness | diff --git a/link/v8.md b/link/v8.md index d8dd5c1..72891f3 100644 --- a/link/v8.md +++ b/link/v8.md @@ -10,8 +10,8 @@ direction not five, ServUO's own decoders, the UOP reader for player bodies, the requirement). The phase 0 spike ran on 2026-09-10 and its findings are §4.5-§4.7 -- §4's decision stands, and the response to a malformed record is now measured rather than proposed. **Nothing in §17 is open.** -**Supersedes the manual half of:** [`../website/UOFIDDLER.md`](../website/UOFIDDLER.md), -[`../website/CLILOCS.md`](../website/CLILOCS.md) §Converting, +**Supersedes the manual half of:** `../website/UOFIDDLER.md` (deleted in phase 3, once creature +art stopped needing it), [`../website/CLILOCS.md`](../website/CLILOCS.md) §Converting, [`../website/SPAWN_ATLAS.md`](../website/SPAWN_ATLAS.md) §Artwork and §Configuring the tree. Two features on this platform read data that only exists inside a UO client, and today both reach @@ -695,6 +695,46 @@ Constructing arbitrary creature types has side effects: constructors pack items, timers. The mitigations are per-type `try`/`catch`, immediate `Delete()`, small batches, and the fact that the whole pass is admin-triggered rather than something that runs at boot. +### 8.1 What phase 3 measured + +**Built and walked 2026-09-10** against a live ServUO with a real world (43,000 mobiles, 210,000 +items) and the real sidecar. + +**The catalogue is 787, exactly as §4.8 predicted**, and it arrives in one page: the whole scan of +bodies 1–2047 — index validation, 787 decodes, 787 PNG encodes and 787 SHA-256s — took **734 ms** +cold. That is well inside the 3 s scan budget, so the wall-clock paging §11 was designed for never +fired on this client. It stays, because the budget is what keeps a slower host or a larger family +inside the 10 s reply timeout rather than producing replies that are always thrown away. + +Every prediction in §4.8 and §5.2 held when the bytes were actually rendered and looked at: + +| Asked for | Answer | Why it matters | +|---|---|---| +| `body/320/a0` (`length 0`) | **absent** | The 357-class bug. The library would have returned the previously-decoded creature | +| `body/607/a0`, `body/608/a0` (elf ghosts) | **absent** | §4.8's two rows that moved; the library returns the elf female here | +| `body/666/a0` (gargoyle → `anim5`) | **absent** | The spider trap. Nothing swept, nothing found | +| `body/400/a0/…` deeper key | **unsupported** | Well-formed under §5, not served until phase 6 | +| direction distribution across 787 | **783 at index 1, 4 at index 0** | Four player bodies, not six — §4.8 again, from `Race.AllRaces` rather than a table | + +**44 of the 787 hashes are shared by two or three bodies, and that is correct.** It is the exact +signature the wrong-picture bug produces, so it was chased rather than assumed: the client's own +`Body.def` says `83 {1}`, `84 {1}`, `138 {7}`, `139 {7}`, `106 {12, 59}`, and the sharing groups +match those lines rather than being runs of consecutive ids (which is what a reused stream buffer +produces). The distinguishing check is at the source: `Animations.Translate(ref body, ref hue)` +rewrites `body` **only when bit 31 of the table entry is set**, unlike the one-argument overload +which always does — and `BridgeAssetValidator.ResolveAnimation` calls that same two-argument +overload, matching `GetAnimation(…, preserveHue: false, …)`. Validator and decoder therefore +resolve the identical record, which is the property the whole §4.5 design rests on. + +**The Core-thread pass costs about 190 ms per 100 types.** All 455 stock `Scripts/Mobiles/Normal` +classes were constructed and deleted in five chunks, producing every status the protocol defines +(`ok` 436, `unknown` 11, `notCreature` 4, `failed` 4). The world's mobile count went from 43,000 at +boot to 42,924 afterwards and its item count fell too — so `Delete()` is reclaiming the packed +inventory as well as the mobile, and nothing leaked. + +The three refusals answer as designed end to end: an unknown `family` is **400**, a stale `catalog` +on a fetch is **422**, and 101 types in one `assets.bodies` is **400** naming the cap. + --- ## 9. The cliloc decompressor is ours now @@ -712,7 +752,8 @@ Protocol 8 writes rather than calls (§4) — and from that point: - No operator runs `dotnet build` on a converter. - No operator copies a 5 MB file to a server. - `website/server/tools/cliloc-export/` is retired, and `UOFIDDLER.md` is deleted rather than - rewritten. + rewritten. (Phase 2 deleted its Part 1; **phase 3 deleted the page**, a phase earlier than that + section predicted, because creature art was the only thing left on it.) ### 9.1 What phase 2 built, and what the port cost @@ -900,6 +941,46 @@ extracted on their own host, for their own shard. Nothing is committed, nothing and nothing is redistributed. What changes is only that the extraction stopped requiring a GUI on a desktop. +### 12.1 The one thing above that phase 3 had to build differently + +"`shard_spawn_creatures.art` … starts being filled by the import" is right about the outcome and +wrong about the mechanism, and the difference is not cosmetic. + +**That table is emptied and refilled by every atlas refresh** — `shardAtlas.db.js`'s `replaceAtlas` +DELETEs all six atlas tables inside one transaction — and a refresh runs on **every boot**. Before +protocol 8 that cost nothing, because `art` came from a file on disk and was simply re-read each +time. An imported sprite is the opposite: expensive to obtain, and gated on client-file hashes that +would say "unchanged" for weeks afterwards. Writing it onto the creature row would mean an ordinary +re-parse of the ServUO tree silently deleting every portrait, with the next Update reporting +*nothing to do* and never restoring them. Nothing would report a fault; the pictures would just be +gone. + +So phase 3 built three tables outside that blast radius, and the atlas import reads them on the way +past (org lead, 2026-09-10): + +| Table | Holds | Lifetime | +|---|---|---| +| `shard_assets` | `asset_key`, `sha256`, `bytes`, `width`, `height`, `body`, `direction`, `file` | Upserted per key; only an approved vanish deletes | +| `shard_creature_bodies` | `slug`, `type_name`, `body`, `status` | Replaced whole — it is derived from the atlas's creature list, so a slug that has left has no meaning | +| `shard_asset_meta` | The singleton an Update compares against | Replaced | + +`replaceAtlas` now takes `{ ...derived, ...operatorMap }`, which is where "the operator's map wins" +is actually enforced — one spread, in one place, applied on every rebuild rather than only at +import time. + +**Two details worth not rediscovering.** The derivation joins on the catalogue key +(`a.asset_key = CONCAT('body/', b.body, '/a0')`), not on `a.body = b.body`: today one body has +exactly one asset and the simpler join is correct, and it stops being correct the moment phase 6 +adds `body/400/a2/f0`, at which point one slug matches dozens of rows and whichever the engine +returned last becomes the portrait. And the stored filename is **content-addressed** +(`uo-body-34-a0-.png`), because a stable name overwritten in place leaves every browser and +CDN serving last month's client's sprite from cache with the database row perfectly correct. + +`shard_creature_bodies` also answers §8 without a schema change on the atlas side: +`shard_spawn_creatures.name` already holds the ServUO **class name** — the atlas build picks the +winning spelling of the spawn type token rather than inventing a display label — so the import has +something `ScriptCompiler.FindTypeByName` will resolve without storing it twice. + --- ## 13. Visibility @@ -921,9 +1002,9 @@ new pipe. | Command | Reply | Purpose | Built | |---|---|---|---| | `assets.sources` | `assets.sources.ok` | Stage 1: client file manifest + `EXTRACTOR_VERSION` | **phase 1** | -| `assets.manifest` | `assets.manifest.ok` | Stage 2: `[{key, sha256, bytes}]`, paged | phase 3 | -| `assets.fetch` | `assets.fetch.ok` | Content for an explicit key list, paged | phase 3 | -| `assets.bodies` | `assets.bodies.ok` | Slug → body id (§8, Core thread) | phase 3 | +| `assets.manifest` | `assets.manifest.ok` | Stage 2: `[{key, sha256, bytes, width, height, body, direction}]`, paged | **phase 3** | +| `assets.fetch` | `assets.fetch.ok` | Content for an explicit key list, paged; base64 PNG per row | **phase 3** | +| `assets.bodies` | `assets.bodies.ok` | Slug → body id (§8, Core thread) | **phase 3** | | `cliloc.table` | `cliloc.table.ok` | The decompressed table, paged (`?lang=`, `?cursor=`) | **phase 2** | | `tree.manifest` / `tree.fetch` | `.ok` | §10, the ServUO tree files | phase 7 | @@ -938,9 +1019,21 @@ ordinary answer mid-import rather than a rare one) and reads the refusal's `code **403** disabled, **404** no such file, **422** a file the shard cannot decode, **503** a shard that cannot do this now (§9.1). +Phase 3 added the other three: `GET /assets/manifest?family=&cursor=`, `POST /assets/fetch` and +`POST /assets/bodies`. **The two POSTs are reads**, and the method is the request body rather than a +side effect — a few hundred asset keys do not belong in a query string. They are the only reads on +this link that take one. `422` gains a second meaning on this plane alongside "cannot decode": the +mid-import guard, a `catalog` that no longer describes the files on disk. + +**§16 listed phase 3 as `servuo-plugins, module-uo` and that was wrong** — `web.rs` routes every +command explicitly and has no generic `/assets/*` forwarder, so `link` is in the phase too. The +table now says so. + **Website admin** (`Admin → Shard`, admin-only): status, **Import**, **Update**, approve/reject for a vanished key, and the existing path settings. Every action to the admin activity log, as -`shard.assets.*`. +`shard.assets.*`. Phase 3 shipped the two that make it reachable — `GET /admin/shard/assets` and +`POST /admin/shard/assets/import`, mirroring the cliloc pair — so the phase could be accepted on a +real rig; the full panel is phase 8 (org lead, 2026-09-10). --- @@ -970,7 +1063,7 @@ disagree, so a split bump means the next bundle silently fails to compose. | 0 | **DONE 2026-09-10.** Spike: the vendored decoders driven from inside a running ServUO, over a client broken 21 ways. §4 stands; the finding was 22,102 wrong pictures on a *stock* client, and the validator that answers them (§4.5-§4.7) | servuo-plugins | | 1 | **DONE 2026-09-10.** The transport: `assets.sources`, the single-slot gate (§3.2), the paging envelope (§3.4), the sidecar line cap (§3.3), `EXTRACTOR_VERSION`, `NO_IMAGING`, protocol 7→8. Plus §4.5's validator promoted into the overlay and extended to animations — which found **357 wrong pictures in the body catalogue on a stock client** and cut it from 1,144 to 787 (§4.8) | servuo-plugins, link | | 2 | **DONE 2026-09-10.** Clilocs end to end (§9.1, §9.2): the Mythic decompressor ported into the overlay, `cliloc.table` + `GET /cliloc`, the paging walk and the source switch on the website, module-uo's protocol pin 7→8. `cliloc-export/` deleted and `UOFIDDLER.md` §Part 1 with it. **67,496 rows, 290 ms, ~11 pages** — the same count UOFiddler's own DLL produced from this client | all | -| 3 | Body resolution (§8) + the **787**-body catalogue (§4.8); `shard_spawn_creatures.art` filled | servuo-plugins, module-uo | +| 3 | **DONE 2026-09-10.** Body resolution (§8) + the **787**-body catalogue (§4.8), `assets.manifest` / `assets.fetch` / `assets.bodies` and their REST mirrors, `shard_spawn_creatures.art` filled and rendered (§8.1, §12.1). **787 rows in one 734 ms page; 455 types resolved at ~190 ms per 100 on the Core thread; zero mobiles leaked.** `UOFIDDLER.md` deleted, two phases early | servuo-plugins, **link**, module-uo | | 4 | **The UOP animation decoder (§4.3)** — the **eight** player bodies the legacy path cannot reach (§4.8 moved the elf ghosts across), the never-sweep rule already enforced in code by phase 1's `ResolveAnimation` | servuo-plugins | | 5 | Item statics and land on demand, hued keys, the cache | servuo-plugins, module-uo | | 6 | Deep animation keys (`body//a/f`) for the future project, plus the bulk-fill switch | servuo-plugins, module-uo | diff --git a/modules/uo/API.md b/modules/uo/API.md index b67d98c..791c346 100644 --- a/modules/uo/API.md +++ b/modules/uo/API.md @@ -23,7 +23,7 @@ knowing a module answers now. |---|---|---| | `/api/v1/public/shard` | 19 | Anonymous. **Never site-mode gated** — the shard surface stays readable during maintenance, per feature audience. | | `/api/v1/public/atlas` | 6 | Anonymous, and unlike `/shard` it **is** site-mode gated: nothing here touches the sidecar, it is parsed shard content. | -| `/api/v1/admin/shard` | 26 | Behind core's `isLoggedIn + noindex + staffOnly` group gate, then **mixed per route** — see below. | +| `/api/v1/admin/shard` | 28 | Behind core's `isLoggedIn + noindex + staffOnly` group gate, then **mixed per route** — see below. | | `/api/v1/admin/uo-link` | 7 | `adminOnly`. The sidecar connection config, its live status, the admin SSE stream and the town crier. | | `/api/v1/player/shard` | 8 | `requireAuth`, **any role** — staff are a superset of players — and every handler is self-scoped to `req.user.id`. | | `/api/v1/admin/users/:id/shard/*` | 6 | `adminOnly`. The module's routes hanging off a **core** resource, through core's `admin.users.detail` extension slot: core owns the user, the module owns what it knows about their game accounts. | @@ -81,6 +81,8 @@ account-linking routes and the sidecar config under `/admin/uo-link` are in the | PUT | `/shard/atlas/path` | point the atlas at a different tree (persisted as `spawn_atlas_servuo_path`, which wins over `SERVUO_PATH`). Blank clears it. Deliberately **does not import** — moving the mount and reloading the world are separate decisions — and returns fresh status so the panel can offer the import next. | | GET | `/shard/clilocs` | cliloc-table status (`adminOnly`): every source found now (base first, then `custom/` overlays in merge order), what each contributed at the last import, readability, drift across the set, the entry count, and `missingSources`. `configured:false` is a supported state — item names then render as ids. No public counterpart: the table is never served *as* a table. | | POST | `/shard/clilocs/import` | reload after a client patch or an overlay edit; `{force}` ignores the hash gate, `{approve}` accepts a **vanished** source (refused by default — see the table notes above). **A missing path — or the likely mistake of pointing at the client's own COMPRESSED `Cliloc.enu` — answers 200 with `status:"unavailable"` and a `code`, not 500.** `COMPRESSED` is called out by name: a 500 would say only "something broke", and the operator needs to be told which file to convert. | +| GET | `/shard/assets` | client-asset import status (`adminOnly`, Protocol 8): the imported body catalogue, how many sprites are on disk, how many atlas creatures resolved to a body id, and the shard's own client files beside them. `drift:true` means the client was patched. `shard.imaging.ok:false` is the named `NO_IMAGING` state — a Linux shard host with no `libgdiplus` cannot decode a sprite at all, and the reason names the package. No public counterpart: the pictures are served as ordinary files under `/uploads`. | +| POST | `/shard/assets/import` | import creature artwork from the shard's UO client; `{force}` ignores the hash gate, `{approve}` accepts a catalogue that no longer offers assets this site holds (refused by default — an unmounted client volume and a deliberate downgrade are indistinguishable, and the wrong guess deletes artwork). **This is the ONLY thing that imports** — boot deliberately never calls the shard. An operator's `spawnAtlas.art.json` always wins over an imported sprite. A body this client has no art for is **not** a failure: two thirds of the playable ghost and gargoyle bodies are in that state on a stock client. | | PUT | `/shard/clilocs/path` | point the site at a different cliloc base file or directory (persisted as `cliloc_client_path`, which wins over `UO_CLIENT_PATH`). Overlays are read from `custom/` beside it either way. Blank clears it. Deliberately **does not import**, same reasoning as the atlas path. | ## 4. Shard visibility — the audience boundary (Protocol 3.0) diff --git a/modules/uo/README.md b/modules/uo/README.md index d7b410c..3ce5443 100644 --- a/modules/uo/README.md +++ b/modules/uo/README.md @@ -21,7 +21,6 @@ these routes *mean* are the ones that already existed and did not move: | [`SPAWN_ATLAS.md`](../../website/SPAWN_ATLAS.md) | The bestiary / spawn atlas, parsed from the shard's own ServUO tree | | [`MARKETPLACE.md`](../../website/MARKETPLACE.md) | The player-vendor index | | [`CLILOCS.md`](../../website/CLILOCS.md) | UO's id → name table | -| [`UOFIDDLER.md`](../../website/UOFIDDLER.md) | Operator runbook for extracting creature art. The cliloc table now arrives over the bridge ([`link/v8.md`](../../link/v8.md) §9) | | [`../../link/PLAN.md`](../../link/PLAN.md), [`../../link/INTEGRATION.md`](../../link/INTEGRATION.md) | The wire protocol this module speaks to the sidecar | --- diff --git a/modules/uo/SCHEMA.md b/modules/uo/SCHEMA.md index 2fa85ba..5b9955c 100644 --- a/modules/uo/SCHEMA.md +++ b/modules/uo/SCHEMA.md @@ -188,10 +188,52 @@ Four column choices worth stating, because each one is a trap: Deceit"). The live `champ.update` feed in `shard_champs` is the separate answer to "it is on level 3 right now". Both exist; they are not the same data. -**`shard_spawn_creatures.art` is always NULL on a fresh import.** The project ships no creature -artwork: sprites live in the operator's own client `.mul`/`.uop` files and are theirs, not ours to -redistribute. An operator supplies art via a gitignored map plus images under the (already -gitignored) `server/uploads/atlas/`. Text-only is the normal, supported state. +**`shard_spawn_creatures.art` is DERIVED, never written by the atlas import itself** — see +`shard_assets` below. The project still ships no creature artwork: sprites live in the operator's own +client `.mul`/`.uop` files and are theirs, not ours to redistribute. What changed in Protocol 8 is +who extracts them: the shard does, from its own client, over the bridge. An operator's hand-drawn map +plus images under the (already gitignored) `server/uploads/atlas/` still wins over anything imported. +Text-only is still the normal, supported state — an install with no shard link never imports one, and +even a complete import leaves two thirds of the playable ghost and gargoyle bodies without art. + +## shard_assets / shard_creature_bodies / shard_asset_meta — the Asset Bridge (Protocol 8) + +Creature artwork read from the shard's own UO client ([`link/v8.md`](../../link/v8.md) §6, §8, §12). + +| Table | Shape | +|---|---| +| `shard_assets` | `asset_key` VARCHAR PK (§5's key, e.g. `body/34/a0`), `family`, `sha256`, `bytes`, `width`, `height`, `body`, `direction`, `file`, `imported_at` | +| `shard_creature_bodies` | `slug` PK, `type_name` (the ServUO class name asked), `body` nullable, `status`, `resolved_at` | +| `shard_asset_meta` | Singleton (`id = 1`), `payload` JSON (catalogue id, extractor version, source fingerprint, counts), `imported_at` | + +**These are the one part of this schema that is deliberately NOT import-owned**, and the reason is +the atlas tables sitting directly above them. `replaceAtlas` empties and refills +`shard_spawn_creatures` on every refresh, and a refresh runs on every boot; an imported filename +stored on that row would be destroyed by an ordinary re-parse of the ServUO tree, with the next asset +Update finding the client files unchanged, reporting "nothing to do", and never restoring it. So the +assets live out here, upserted per key, and `replaceAtlas` re-derives `art` from them on the way +past — `{ ...derived, ...operatorMap }`, which is the one place "the operator's map wins" is +enforced. + +Four details that are load-bearing rather than incidental: + +- **`file` is a filename under the uploads directory, never a path**, and it is content-addressed + (`uo-body-34-a0-.png`). A stable name overwritten in place would leave every browser and CDN + serving the previous client's sprite from cache, with the row perfectly correct. +- **The `art` derivation joins on the catalogue key**, `a.asset_key = CONCAT('body/', b.body, '/a0')`, + not `a.body = b.body`. The simpler join is correct today and stops being correct the moment deeper + animation keys (`body/400/a2/f0`) arrive, at which point one slug matches dozens of rows. +- **`shard_creature_bodies` IS replaced whole**, unlike `shard_assets`: it is derived from the atlas's + creature list, so a slug that has left the atlas has no meaning, and the pass that rebuilds it is a + shard round trip rather than a file transfer. +- **`status` keeps the negative answers** — `unknown` (the spawn files name a type this shard's + scripts do not define, which is real drift), `notCreature` (a spawn entry for an item or + decoration, a permanent answer), `failed`. Without them the next pass asks again, and each name + costs a real constructor on the shard's Core thread. + +`shard_spawn_creatures.name` already holds the ServUO **class name** — the atlas build picks the +winning spelling of the spawn type token rather than inventing a display label — which is why the +body pass needs no extra column to ask its question. ## shard_clilocs / shard_cliloc_meta — UO's localization table (Protocol 3.0) diff --git a/website/SPAWN_ATLAS.md b/website/SPAWN_ATLAS.md index 4c51e87..ea7108a 100644 --- a/website/SPAWN_ATLAS.md +++ b/website/SPAWN_ATLAS.md @@ -213,28 +213,59 @@ implicitly commit, defeating the all-or-nothing guarantee. Point ids are assigne explicitly rather than left to `AUTO_INCREMENT`, because the join rows need them and `conn.batch()` reports no usable `insertId`. -## Artwork — operator-supplied, never shipped +## Artwork — the shard extracts it now (Protocol 8) **This project ships no creature art and no extraction tooling, and never will.** UO sprites live in the operator's own client `.mul`/`.uop` files. They are the operator's, not ours to redistribute. -The atlas is fully functional as text. `shard_spawn_creatures.art` is nullable -and is NULL on every fresh import; pages render without images, which is the -normal and supported state, not a degraded one. +What changed in protocol 8 is not that rule — it is who does the extracting. The +shard already has those files (a ServUO server cannot boot without a UO client), +so as of [`v8.md`](../link/v8.md) phase 3 it decodes them itself and hands the +pictures to the website over the bridge. Nobody installs UOFiddler and nobody +copies images to a web host. -An operator who wants art — step-by-step, with the UOFiddler side spelled out, in -[`UOFIDDLER.md`](UOFIDDLER.md) §Part 2: +**Admin → Shard → Import.** The import walks the shard's asset manifest, fetches +only the sprites whose hash changed, writes them under `uploads/atlas/`, asks the +shard for a body id per creature (§8 — the shard constructs the creature and +reads `Body.BodyID`, which is the only thing that is right for a shard's own +custom creatures) and points each `shard_spawn_creatures.art` at its picture. +Boot never calls the shard for this: the files change when an operator patches +their client, which is an event they know about and the site does not. -1. Extracts it from **their own** client files (UOFiddler, ClassicUO tooling, or - any art extractor). -2. Drops the images under `server/uploads/atlas/`. -3. Copies `server/db/data/spawnAtlas.art.example.json` to `spawnAtlas.art.json` - and maps creature slugs to file names. -4. Restarts, or runs `npm run atlas:import -- --force`. +On this machine's stock client that is **787 creature portraits**, about a +megabyte in total. -Both `spawnAtlas.art.json` and `server/uploads/` are gitignored, so neither the -map nor the images can be committed by accident. +**NULL stays a first-class state, and always will be.** An install with no shard +link has never imported one; a Linux shard host without `libgdiplus` cannot +render a sprite at all (a named `NO_IMAGING` status, not an error); and even +after a complete import, two thirds of the playable ghost and gargoyle bodies +have no art in the client. Pages render without images, which is normal and +supported, not degraded. + +### The operator's own artwork still wins + +An operator who has drawn their own portraits keeps them. The map is unchanged: + +1. Drop the images under `server/uploads/atlas/`. +2. Copy `server/db/data/spawnAtlas.art.example.json` to `spawnAtlas.art.json` and + map creature slugs to file names. +3. Restart, or run the import. + +`spawnAtlas.art.json` is applied **over** anything imported, per slug, so a sprite +rip never replaces a hand-drawn portrait on the next Update. Both it and +`server/uploads/` are gitignored, so neither the map nor the images can be +committed by accident. + +### Why the imported art is not stored on the creature row + +`shard_spawn_creatures` is emptied and refilled by every atlas refresh, and a +refresh happens on every boot. So the body ids and the imported files live in +`shard_creature_bodies` and `shard_assets`, outside that blast radius, and the +atlas import re-derives `art` from them on the way past. Storing it on the row +would mean an ordinary re-parse of the ServUO tree silently deleting every +portrait — with the next asset Update finding the client files unchanged, +reporting "nothing to do", and never putting them back. ## Code layout diff --git a/website/UOFIDDLER.md b/website/UOFIDDLER.md deleted file mode 100644 index f7bd5aa..0000000 --- a/website/UOFIDDLER.md +++ /dev/null @@ -1,120 +0,0 @@ -# Extracting from your own UO client (UOFiddler) - -**Audience:** the shard operator, once, at setup time. -**Related:** [`SPAWN_ATLAS.md`](SPAWN_ATLAS.md) (where creature art fits), -[`../link/v8.md`](../link/v8.md) (the Asset Bridge, which is replacing this page). - -> ### The cliloc conversion is gone — you no longer do this by hand -> -> **Protocol 8, phase 2.** The shard reads its own client's `Cliloc.enu` and -> serves the table over the bridge, so there is nothing to install, convert or -> copy. Press **Admin → Shard → Import** after you patch your client and that is -> the whole procedure; see [`CLILOCS.md`](CLILOCS.md). -> -> Part 1 of this guide has been deleted rather than rewritten. If you already -> have a converted file it keeps working on an install with no uo-link -> configured, but nobody should make a new one. -> -> **Creature art is next** (phase 5), after which this page goes away entirely. - -What remains here reads data that **only exists inside a UO client**, and a UO -client's files are EA's, not ours to redistribute. So neither this repo nor any -image we publish can ship them — the operator extracts from **their own** client -and points the site at the result. - -| Feature | What it needs | Required? | Without it | -|---|---|---|---| -| **Creature art** ([`SPAWN_ATLAS.md`](SPAWN_ATLAS.md)) | Sprites from `.mul`/`.uop` | No | Atlas pages render as text, which is the normal state | - -**It is optional and it is not load-bearing.** A shard that never does any of this -is fully supported. - -Everything you extract stays **outside the repository**: `spawnAtlas.art.json` and -`server/uploads/` are gitignored, so none of it can be committed by accident. - ---- - -## Part 0 — Get UOFiddler - -[UOFiddler](https://github.com/polserver/UOFiddler) is the community client-file -editor. What is left here uses its GUI to export sprites; the `Ultima.dll` this -guide used to reach for was the cliloc decompressor, and the shard has its own -now. - -1. Download the latest release zip from - — one asset, named - `UOFiddler-.zip` (4.22.2 is ~2 MB). -2. Extract it. The zip contains a single top-level folder, and the two files that - matter are at **its root**: - - ``` - UOFiddler-4.22.2/ - UoFiddler.exe ← the GUI (Part 2 needs this) - Ultima.dll - plugins/ - … - ``` - -3. **Runtime:** UOFiddler 4.22.2 is built for **.NET 10**, so running - `UoFiddler.exe` needs the .NET 10 **Desktop** Runtime (Windows only). Install - from . - -### Finding your client files - -The art files are in your **UO client installation directory**. Look for -`art.mul` / `artLegacyMUL.uop` and the `anim*.mul` set. The EA Classic Client's -default location is: - -``` -C:\Program Files (x86)\Electronic Arts\Ultima Online Classic\ -``` - -**If your shard distributes its own patched client to players, use that copy** — -it is what your players actually see. (It is also the copy the shard itself reads -from, since a ServUO server resolves `Config/DataPath.cfg` into its own client -path at boot, which is the premise the Asset Bridge is built on.) - ---- - -## Part 2 — Creature art for the spawn atlas (optional) - -**Goal:** put sprites on atlas pages. Purely cosmetic — the atlas is fully -functional as text, and `art` is NULL on every fresh import. - -**This project ships no art and no art-extraction tooling, and never will.** - -1. Open `UoFiddler.exe` and point it at your client directory - (**Settings → Paths**), then open the - **Animations** tab for creature sprites — or **Items** for object art — find - the creature, and export as PNG. Right-click an entry for its export options, - or use the tab's *Export All* action for a batch. (4.22.2 added an export - option to the Animation tab's thumbnail list, which is the convenient one - here.) -2. Put the images under `server/uploads/atlas/`. -3. Copy `server/db/data/spawnAtlas.art.example.json` to `spawnAtlas.art.json` in - the same directory and map creature slugs to file names: - - ```json - { - "lizardman": "lizardman.png", - "orc": "orc.png" - } - ``` - - **Keys are the slugs the atlas API reports**, derived from the type names in - your own shard's `Spawns/*.xml` — read them off the atlas rather than guessing. - A creature with no entry renders without art, which is the default. - -4. Restart, or `npm run atlas:import -- --force`. - -The art map is re-read on every atlas refresh, so adding one image is an edit plus -a refresh. Both `spawnAtlas.art.json` and `server/uploads/` are gitignored. - ---- - -## Licensing, briefly - -UO's strings and sprites are EA's. Extracting from **your own** client for -**your own** shard is the arrangement here; redistributing the extracted files is -not something this project does or can advise on. That is the whole reason this -page exists instead of a download link. -- 2.49.1 From d07772a3d85c1b6a0f5295629fc298c5696b8e6e Mon Sep 17 00:00:00 2001 From: wtclaude Date: Thu, 10 Sep 2026 18:50:45 -0500 Subject: [PATCH 2/2] docs(modules): the cliloc conversion step IS avoidable now (phase 2 debt) SCHEMA.md's cliloc section still described the pre-protocol-8 world, two phases after phase 2 changed it. Three sentences said the same false thing, so fixing only the flagged one would have left the section arguing with itself: - "Sourced from files the operator supplies" -- the BASE comes from the shard on any install with uo-link configured; only the overlays are the filesystem's, and that asymmetry has a reason worth stating (ServUO has no server-side notion of a custom cliloc, so there is nothing to ask for). - "A base (the converted client table)" -- not converted any more. - "The conversion step is not avoidable ... so the shard cannot supply names on our behalf" -- it does supply them. Phase 2 ported UOFiddler's Mythic decompressor into the overlay precisely so nobody converts anything. Rewritten to say what is true and why the file path still exists (deprecated, not removed, so an existing install keeps working), plus the two things a reader of this table actually needs: `shard_cliloc_meta.payload` keeps the base's fingerprint under `base` SEPARATELY from the overlay hashes -- because on the bridge the old `clilocs.plain` label is supposed to disappear and one flat hash map would read that upgrade as a vanished source -- and boot does not import on the bridge path at all. Each claim checked against the code rather than from memory: shardClilocs.model.js:307 (`base: fingerprint`), :438-440 (refreshOnBoot returns `skipped` on the bridge), clilocSource.js:333 (`missingOverlays`). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4 --- modules/uo/SCHEMA.md | 42 +++++++++++++++++++++++++++++++----------- 1 file changed, 31 insertions(+), 11 deletions(-) diff --git a/modules/uo/SCHEMA.md b/modules/uo/SCHEMA.md index 5b9955c..7007dae 100644 --- a/modules/uo/SCHEMA.md +++ b/modules/uo/SCHEMA.md @@ -245,29 +245,49 @@ marketplace listing — but with no table to resolve it against, the character s | Table | Shape | |---|---| | `shard_clilocs` | `number` INT PK, `flag`, `text` TEXT | -| `shard_cliloc_meta` | Singleton (`id = 1`), `payload` JSON (source file, sha256, count, `parserVersion`), `imported_at` | +| `shard_cliloc_meta` | Singleton (`id = 1`), `payload` JSON (`source` — `bridge` or `file` — the base's fingerprint under `base`, the overlay `hashes`, per-source counts, `parserVersion`), `imported_at` | Import-owned and all-or-nothing in one transaction, same contract as the atlas — including **`DELETE`, not `TRUNCATE`**, for the same reason. -**Sourced from files the operator supplies**, at a path from the `cliloc_client_path` setting falling -back to `UO_CLIENT_PATH`. Nothing client-derived is committed: UO's strings are EA's, exactly as the -creature sprites are. A shard with nothing configured is fully supported — names render as ids. Full -design and operator guide: [`CLILOCS.md`](../../website/CLILOCS.md). +**The base table comes from the SHARD** on any install with uo-link configured (Protocol 8, phase 2): +it reads its own client's compressed `Cliloc.enu` and serves the table paged over the bridge, so +nothing is converted and nothing is copied to the web host. Without a shard link it falls back to a +converted file on disk at a path from the `cliloc_client_path` setting, then `UO_CLIENT_PATH` — the +pre-protocol-8 pipeline, deprecated rather than removed so an existing install keeps working. + +**Overlays are always the filesystem's**, either way: ServUO has no server-side notion of a custom +cliloc, so `custom/` is the only place shard-added ids exist and there is nothing on the shard to ask +for. That gap is in the *game*, not in this pipeline. + +Nothing client-derived is committed: UO's strings are EA's, exactly as the creature sprites are. A +shard with nothing configured is fully supported — names render as ids. Full design and operator +guide: [`CLILOCS.md`](../../website/CLILOCS.md). **It reads a SET of sources, not one file**, because shards edit items and add new ones and those -carry cliloc ids no stock client table has. A base (the converted client table) plus every overlay -under `custom/` are re-read on every boot and hash-gated **together**, exactly as the atlas re-reads +carry cliloc ids no stock client table has. A base (from the shard, or a converted file) plus every +overlay under `custom/` are hash-gated **together**, exactly as the atlas re-reads `Regions.xml` + `Locations/*.xml` + `Spawns/*.xml` + `ChampionSpawns.xml`. Later sources win, so an overlay both adds ids and overrides stock ones, and adding one custom item never means re-exporting a 5 MB client file. Scale, measured on the live shard: its script tree references 16,434 cliloc ids and only 37 are absent from stock — tens of entries against a 67k base, which is why this is an overlay and not a second table. -The conversion step is not avoidable: **every current client ships its cliloc files compressed** -(first DWORD's high byte `0x8E`), and ServUO's own bundled `Ultima.StringList` cannot read that -either — so the shard cannot supply names on our behalf. The plain layout and a delimited text export -are both accepted, sniffed by header rather than extension. +**The conversion step used to be unavoidable, and is not any more.** Every current client ships its +cliloc files compressed (first DWORD's high byte `0x8E`) and ServUO's own bundled +`Ultima.StringList` cannot read that either — which is why, for two protocol versions, the operator +had to install UOFiddler, build a converter against its `Ultima.dll` and copy a 5 MB file to the web +host. Protocol 8 phase 2 ported the Mythic decompressor into the overlay, so **the shard reads its +own client and supplies the names**. The file half survives only as the fallback above, where the +plain layout and a delimited text export are both accepted, sniffed by header rather than extension. + +Two consequences for what this table holds. `shard_cliloc_meta.payload` carries the base's +fingerprint under `meta.base` (the shard's file size, mtime, hash and `EXTRACTOR_VERSION`) separately +from the overlay hashes, because on the bridge the old `clilocs.plain` label is *supposed* to +disappear and a single hash map would read that upgrade as a vanished source. And **boot does not +import on the bridge path**: a file could be re-hashed locally on every restart, but asking the shard +would put a sidecar round trip in the boot sequence for a table that changes only when an operator +patches their client. Importing is an admin action. Three decisions worth stating: -- 2.49.1