§11.1 is new and carries what phase 5 measured: 49,152 addressable static ids
(not the 81,884 `artidx.mul` declares -- `FileIndex` sizes its table from its
length ARGUMENT), 39,189 with art, 4,244 land tiles, 9,963 + 12,140 empty index
slots, and the whole set at 81 MB decoding in 34 s. That last number reopens the
bulk question and the answer is still no: 108 MB of base64 through a 512 KB
single-slot channel to store 43,433 pictures a shard displays a few hundred of.
Two traps, both §4.5's failure mode -- a confident, plausible, wrong picture:
- `Art.GetStatic` hands back the SAME cached Bitmap and `Hue.ApplyTo` repaints in
place, so hueing edits the library's own copy: the plain key comes back hued
from then on, and the next hue stacks. `Files.CacheData` off process-wide fixes
it and also stops a game server retaining 74 MB of Bitmap. Copying instead does
not solve the retention, and `new Bitmap(src)` throws on ARGB1555 anyway.
- `PartialHue` (13,259 of 65,536 ids) decides whether a hue repaints every pixel
or only the grey ones, from a file only the shard has. Item 597 is a wooden
screen with painted flowers; one mode reddens the flowers, the other the whole
screen, and both decode. Hence land takes no hue segment and `h0` is not a key.
Plus the namespace trap that compiled: unqualified `TileData` binds to ServUO's
own `Server.TileData`, because the enclosing namespace beats `using Ultima;`.
§14 records what the wire gained -- the `static` and `land` families, `families`
on `assets.sources`, and `assets.fetch` becoming shared plumbing whose family is
DERIVED from the keys (§5 made the key the address; a request naming its family
too would have two places to be wrong and one of them silent). Additive, so the
protocol stays 8 and EXTRACTOR_VERSION stays 2. §15 records that `link` needed
nothing in phases 4 or 5: it forwards verbatim in both directions.
§17.10 is the four org-lead decisions. §12 and modules/uo/SCHEMA.md carry the
website side: `uploads/items/`, per-row `catalog` staleness, and why a key with
no art writes no row at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
2026-09-11 06:16:15 -05:00
3 changed files with 172 additions and 8 deletions
@@ -1017,8 +1017,8 @@ project. §1.1's measurements make the sizing question concrete:
| Kind | Addressable | Bulk? |
|---|---|---|
| Item statics | **~49,150** | No — on demand, cached, keyed by `itemId` (+ hue) |
| Land tiles | **16,384** | No — on demand |
| Item statics | **49,152** addressable, **39,189** with art (§11.1, phase 5) | No — on demand, cached, keyed by `itemId` (+ hue) |
| Land tiles | **16,384** addressable, **4,244** with art | No — on demand |
| Creature/player bodies, first frame | **1,022** — 787 legacy (§4.8, *not* the 1,144 the library reports) + 235 UOP (§4.9) | **Yes** — this is the catalogue |
| One body, every action, one direction | **210 frames** (body 400); 96–210 measured across six bodies | No — on demand, per body |
| All bodies, every action, one direction | **~119,000 frames**, ~117 MB | No — but no longer unthinkable |
@@ -1054,9 +1054,94 @@ default and still not something to import before anything asks for it, but it ha
bulk-fill-everything switch rather than assuming on-demand is the only mode.
**Hued variants are on demand, always.**`static/3922/h33` is generated when something on the wire
actually carries hue 33. The cross product of 49,150 statics and ~3,000 hues is not a set anyone
actually carries hue 33. The cross product of 49,152 statics and 3,000 hues is not a set anyone
enumerates.
### 11.1 What phase 5 measured, and the two traps it found
The sizing above was an estimate taken off `art.mul`'s length. Measured through the reader itself,
against this machine's stock client:
| | Measured |
|---|---|
| Static ids addressable | **49,152** |
| ...with real art | **39,189** |
| ...empty index slots (§4.5's shape) | **9,963** |
| Land tiles addressable / with art | 16,384 / **4,244** |
| Whole static + land set, as PNG | 43,433 files, **81 MB**; mean 1.9 KB, max 30.9 KB (id 18213) |
| Time to decode and encode all of it | **34 s** |
| Hue slots in `hues.mul` | 3,000 (2,062 named) |
| Item ids flagged `PartialHue` | **13,259** of 65,536 |
Two of those need saying out loud.
**49,152, not the 81,884 entries `artidx.mul` declares.**`FileIndex` sizes its index table from the
**length argument it is constructed with** (`0x10000`), not from the idx file, so the addressable
static range is `0x10000 - 0x4000`. A ceiling read off the file instead would invent 16,348 ids and
answer every one of them out of an array nobody bounded. (The first probe of this phase made exactly
that mistake and reported 65,500 — PowerShell returns `$null` for an out-of-range array index rather
than throwing, so the over-run counted silently as "empty slots".)
**81 MB is small enough to reopen the bulk question, and the answer is still no.** Not on size — on
what the transfer buys. Base64 puts it at 108 MB through a 512 KB single-slot channel, roughly 210
round trips, to store 43,433 pictures of which a live shard displays a few hundred. On-demand stays
right; phase 6's bulk-fill switch is where an operator who wants the lot says so.
#### The library's cache poisons a hued sprite
`Art.GetStatic` and `Art.GetLand` memoise into a static `Bitmap[0xFFFF]` and hand back **the same
instance** on every call; `Hue.ApplyTo` repaints a bitmap **in place**. So the obvious implementation
— ask the library, apply the hue, encode — edits the library's own copy. Measured before the fix:
hue item 3922 once, and every later request for the **plain** 3922 comes back hued, with a second hue
stacking on the first.
This is §4.5's failure mode exactly — a confident, plausible, correctly-sized wrong picture that
every success count agrees with — reached through a door §4.5 never looked at, because phase 0 was
auditing *records* and this is the library's *cache*. Nothing downstream can see it: the key is
right, the dimensions are right, the hash is stable.
The fix is `Files.CacheData = false` for the life of the process, set once when the asset plane
initialises, and it pays twice: the same array is never trimmed, so decoding this client's 39,189
statics would otherwise leave **74 MB of `Bitmap` in a static field of a game server** to serve
pictures nobody asks for twice. `Animations` does not consult the flag at all, so the body catalogue
is untouched, and each reader keeps its own cache of *encoded PNG bytes* instead — a tenth of the
size, already hashed, released when it goes idle.
The obvious alternative, copying each bitmap before hueing, was rejected for the retention alone —
but also because `new Bitmap(src)`**throws** on the `Format16bppArgb1555` these decoders produce.
The copy has to name the source pixel format explicitly, which is a subtlety on the wrong side of a
correctness boundary. The invariant is instead re-checked before every hue: a hue is refused if the
cache is somehow on, because an invariant nothing verifies is a comment.
#### `PartialHue` decides the picture, and only the shard can read it
A hue is not a tint. It is a 32-entry colour ramp out of `hues.mul` indexed by each pixel's own red
channel — and whether it replaces **every** pixel or only the grey ones is a per-item-id flag in
`tiledata.mul`. On this client **13,259 of 65,536 item ids carry it**.
Item 597 is a wooden screen with painted flowers. Hued 33 the right way the flowers turn red; the
wrong way the whole screen turns red. Both decode, both are 44×112, both report success. This is why
§5 put hue in the key rather than leaving it to the website: shipping `Hues.mul` semantics and a
65,536-row flag table into Node, to answer a question the shard can answer for free, is the trade
§2.1 already refused.
Two consequences fall out of it. **Land takes no hue segment** — the mode is an *item* flag and land
has no equivalent, so `land/3/h33` is refused rather than guessed; nothing on the wire carries one
today, and if something ever does it will arrive with a reason to choose. And **`h0` is not a key**:
hue 0 on the wire means "not hued", so the plain key already names that picture, and accepting both
would store one PNG twice under two names and diff them separately forever.
#### The namespace trap that compiled
The first cut of the reader wrote `TileData.ItemTable` and `TileFlag.PartialHue` unqualified. ServUO
declares its **own**`Server.TileData`, `Server.ItemData` and `Server.TileFlag` — with a
`PartialHue` member — in `Server/TileData.cs`, and the reader lives in `Server.Custom.Bridge`, where
the enclosing namespace beats `using Ultima;`. It compiled. At runtime it read a file resolved
through `Core.DataDirectories` rather than through `Ultima.Files`, which is §4.6's rule broken in a
new place: deciding a picture with a file other than the one the pixels came out of. The live rig
caught it as a `TypeInitializationException` refusing every hued key, from a class the code never
meant to name.
**Gump art is out of scope for Protocol 8, and that is now a safety rule rather than a priority
call** — §4.1. It is the only decoder that reaches the `hasExtra: true` branch, and that branch
corrupts the process on the second id. Adding gump art later means fixing that path first,
@@ -1077,6 +1162,12 @@ so no `MODULE_API_VERSION` bump is needed to store them.
- **The operator-supplied `spawnAtlas.art.json` map stays supported** and continues to win over an
imported asset. An operator who has drawn their own creature portraits must not have them
overwritten by a sprite rip on the next Update.
- **Item and land pictures land in `uploads/items/`, beside the creature portraits and not among
them** (phase 5). Same content-addressed naming (`uo-static-3922-h33-<sha8>.png`), same "the API
returns a filename, the client builds the URL" contract, and the same rule that a missing picture
is a first-class state rather than an error. Separate directories because they have different
lifetimes: the catalogue is imported as a set and re-imported as a set, while these arrive one at
a time because something asked for them.
Licensing is unchanged and the reasoning is unchanged: these are the operator's own client files,
extracted on their own host, for their own shard. Nothing is committed, nothing ships in a repo,
| `tree.manifest` / `tree.fetch` | `.ok` | §10, the ServUO tree files | phase 7 |
@@ -1167,6 +1258,27 @@ side effect — a few hundred asset keys do not belong in a query string. They a
this link that take one. `422` gains a second meaning on this plane alongside "cannot decode": the
mid-import guard, a `catalog` that no longer describes the files on disk.
**Phase 5 added two families, one field, and no command.**`assets.fetch` grew `static` and `land`
(§5, §11.1) and `assets.sources` grew **`families`** — which key families this overlay serves. Both
are additive, so **the protocol stays 8**, and `EXTRACTOR_VERSION` stays **2**: no existing key's
bytes change, and a new key is not a re-derivation of an old one.
The command itself became shared plumbing. Phase 3 gave `assets.fetch` to the body catalogue
outright, which was right with one family and wrong with three: the command is the *transport* and
the family is a property of the key. So the correlation id, the operator's consent, the key-count
ceiling and the family decision now happen once, and a reader only ever sees keys it owns.
**The family is derived from the keys and is not a request field.** §5 made the key the address of an
asset; a request that also named its family would have two places to be wrong and one of them
silent. A batch must be of **one** family — mixing them is refused (400) rather than split — because
the reply carries a single `catalog` id, and two families have two fingerprints. A reply claiming one
of them would be lying about the other.
`families` matters more than it looks. Without it, a website talking to a phase-3 or phase-4 overlay
discovers the gap as a refusal *per key, per pass, forever*, with no picture ever appearing and a
warning in the log every few minutes. With it, that is one reported state carrying a sentence naming
the fix.
**Phase 4 added one field and no command.**`source` on a manifest or fetch row is `legacy` or
`uop` — which reader produced the bytes (§4.9). It is additive, so **the protocol stays 8**: a
consumer that does not read it is unaffected, and one that does can say which half of the extractor
@@ -1194,7 +1306,7 @@ disagree, so a split bump means the next bundle silently fails to compose.
| Repo | Work |
|---|---|
| `servuo-plugins/` | Extraction over ServUO's own `Ultima` (§4), the cliloc decompressor (§9), body resolution (§8), the request handlers, `overlay.toml` |
| `link/` | Six command families forwarded, the REST surface, **the inbound line cap (§3.3)**, `PROTOCOL_VERSION` |
| `link/` | Six command families forwarded, the REST surface, **the inbound line cap (§3.3)**, `PROTOCOL_VERSION`. **Nothing in phases 4 or 5** — `assets_call` forwards a request body verbatim and `respond_assets` returns the reply verbatim, so a new key family and a new reply field both pass through untouched |
| `website/` | Almost none — `ctx.uploads` already suffices (§12). Phase 2 deleted `server/tools/cliloc-export/`, the converter this protocol retires |
| `docs/` | This file; rewrite `CLILOCS.md` §Converting and `SPAWN_ATLAS.md` §Artwork + §Configuring; **delete `UOFIDDLER.md`**; add the libgdiplus prerequisite to `SHARD_PREREQS.md` (§4.4) |
@@ -1213,7 +1325,7 @@ disagree, so a split bump means the next bundle silently fails to compose.
| 2 | **DONE 2026-09-10.** Clilocs end to end (§9.1, §9.2): the Mythic decompressor ported into the overlay, `cliloc.table` + `GET /cliloc`, the paging walk and the source switch on the website, module-uo's protocol pin 7→8. `cliloc-export/` deleted and `UOFIDDLER.md` §Part 1 with it. **67,496 rows, 290 ms, ~11 pages** — the same count UOFiddler's own DLL produced from this client | all |
| 3 | **DONE 2026-09-10.** Body resolution (§8) + the **787**-body catalogue (§4.8), `assets.manifest` / `assets.fetch` / `assets.bodies` and their REST mirrors, `shard_spawn_creatures.art` filled and rendered (§8.1, §12.1). **787 rows in one 734 ms page; 455 types resolved at ~190 ms per 100 on the Core thread; zero mobiles leaked.**`UOFIDDLER.md` deleted, two phases early | servuo-plugins, **link**, module-uo |
| 4 | **DONE 2026-09-11.** The UOP animation decoder (§4.3, §4.9): `BridgeUop` + a PNG encoder that never touches `System.Drawing`, wired in beneath the legacy reader. **Two of the eight player bodies turned out to exist** (gargoyles 666/667); the other six are in no client file, and ghost ids left the player-body set (§5.2, §17.9). The same fallback added **233 other bodies**: the catalogue is **1,022 rows, 1,409 ms cold**, and all six player bodies have art for the first time. `EXTRACTOR_VERSION` 1 → 2 | servuo-plugins |
| 5 | Item statics and land on demand, hued keys, the cache | servuo-plugins, module-uo |
| 5 |**DONE 2026-09-11.** Item statics and land on demand (§11.1): the `static` and `land` families, hue applied on the shard from `tiledata.mul`, the byte-bounded art cache, `assets.fetch` made family-aware, `families` on `assets.sources`. Website side: the warm pass, per-row `catalog` staleness, and pictures on the marketplace and the character sheet. **39,189 statics and 4,244 land tiles served; the only refusals are the 9,963 + 12,140 empty index slots §4.5 predicted.** Two traps found — the library's bitmap cache poisons a hued sprite, and `PartialHue` decides the picture from a file only the shard has. Protocol stays 8; `EXTRACTOR_VERSION` stays 2 | servuo-plugins, module-uo |
| 6 | Deep animation keys (`body/<id>/a<n>/f<n>`) for the future project, plus the bulk-fill switch | servuo-plugins, module-uo |
| 7 | The atlas over the sidecar (§10); shared-filesystem requirement retired | module-uo |
@@ -56,7 +56,7 @@ feed) are described where their wire frames are, in
| GET | `/shard/ruleset` | the shard's own published ruleset (Protocol 3.0 `world.ruleset`): expansion, which optional systems are on, skill/stat caps, account and house limits, champion scroll rules, the save/restart schedule. Served from `shard_ruleset`, so it renders while the shard is down; live via `world.ruleset` on `/shard/stream`. Behind `requireFeature('ruleset')`. **`null`** means the shard has never published one — a real answer, distinct from a published ruleset. `caps.skill` / `caps.totalSkill` are in **tenths** (1000 = 100.0). |
| GET | `/shard/points` | every points/loyalty leaderboard the shard publishes (Protocol 3.0 `points.board`) — Queen's Loyalty, Void Pool, the nine city loyalties, Clean Up Britannia, … Served from `shard_points_boards`, so it renders while the shard is down; live via `points.board` on `/shard/stream`. Behind `requireFeature('leaderboards')`, ordered by display name. **`maxPoints: 0` means uncapped** (the common case), and `nameString` is usually `null` with `nameNumber` holding a cliloc — resolve client-side or humanise the `system` key. |
| GET | `/shard/points/:system` | one board by the shard's `PointsType` name (e.g. `QueensLoyalty`); `:system` must match `/^[A-Za-z][A-Za-z0-9_]{0,47}$/` or **400** before any query runs. **404** = the shard has never published that system, which is distinct from a published board nobody has scored in yet (**200** with an empty `top`). |
| GET | `/shard/market?q=&minPrice=&maxPrice=&itemId=&map=®ion=&sort=&limit=&offset=` | search the player-vendor marketplace (Protocol 3.0 `vendor.listing`). Returns **listings**, not vendors — "who sells X and for how much" is the question, and a vendor-shaped result would make every caller flatten the shops back out. Served from `shard_vendors` + `shard_vendor_items`, so it renders while the shard is down. Behind `requireFeature('market')`**and rate-limited** — the first genuinely expensive public read on the site (a `LIKE` scan plus a `COUNT` over what is typically the largest `shard_*` table, reachable with no session). `sort ∈ {price_asc, price_desc, recent}`. `q` matches the resolved display name **or** the item's literal name, with `%`/`_` escaped: they are `LIKE` metacharacters, not SQL ones, so parameterization alone would let `?q=%` match every listing on the shard. Every response repeats `staleAt` (the oldest vendor row) because the shard sweeps round-robin — a banner that ages with the results it labels, not one fetched once. |
| GET | `/shard/market?q=&minPrice=&maxPrice=&itemId=&map=®ion=&sort=&limit=&offset=` | search the player-vendor marketplace (Protocol 3.0 `vendor.listing`). Returns **listings**, not vendors — "who sells X and for how much" is the question, and a vendor-shaped result would make every caller flatten the shops back out. Served from `shard_vendors` + `shard_vendor_items`, so it renders while the shard is down. Behind `requireFeature('market')`**and rate-limited** — the first genuinely expensive public read on the site (a `LIKE` scan plus a `COUNT` over what is typically the largest `shard_*` table, reachable with no session). `sort ∈ {price_asc, price_desc, recent}`. `q` matches the resolved display name **or** the item's literal name, with `%`/`_` escaped: they are `LIKE` metacharacters, not SQL ones, so parameterization alone would let `?q=%` match every listing on the shard. Every response repeats `staleAt` (the oldest vendor row) because the shard sweeps round-robin — a banner that ages with the results it labels, not one fetched once. As of Protocol 8 phase 5 each listing also carries **`art`**: the FILENAME of the item's picture under `uploads/items/`, already hued, or `null` where this site does not hold one. `null` is ordinary — the picture is fetched behind the page and never by it, so a newly listed item shows text first and gains its icon a pass later, and some item ids have no art in any client. The same field appears on a character sheet's equipment entries. |
| GET | `/shard/market/meta` | index size, staleness (`staleAt`/`freshAt`) and which facets and regions actually hold vendors, so a client builds its filters without running a search it will discard. |
| GET | `/shard/market/vendors/:serial` | one shop and its listings; `:serial` must match `/^0x[0-9A-Fa-f]{1,16}$/` or **400** before any query runs. **404** = a serial the index has never seen, which also covers a vendor since dismissed or hidden — to an anonymous caller those are the same answer, and distinguishing them would leak that a hidden vendor exists. `truncated` (with `total` exceeding `count`) means the shop holds more than the shard publishes per frame. |
| GET | `/shard/features` | the shard features **this caller** may reach plus the audience rung they resolved to (§4 below), so a client hides nav it can't follow. Reports only what the caller can see — the list itself never discloses a gated feature. Consumed by the SPA header and (pending) the Android nav. |
@@ -83,6 +83,7 @@ account-linking routes and the sidecar config under `/admin/uo-link` are in the
| POST | `/shard/clilocs/import` | reload after a client patch or an overlay edit; `{force}` ignores the hash gate, `{approve}` accepts a **vanished** source (refused by default — see the table notes above). **A missing path — or the likely mistake of pointing at the client's own COMPRESSED `Cliloc.enu` — answers 200 with `status:"unavailable"` and a `code`, not 500.**`COMPRESSED` is called out by name: a 500 would say only "something broke", and the operator needs to be told which file to convert. |
| GET | `/shard/assets` | client-asset import status (`adminOnly`, Protocol 8): the imported body catalogue, how many sprites are on disk, how many atlas creatures resolved to a body id, and the shard's own client files beside them. `drift:true` means the client was patched. `shard.imaging.ok:false` is the named `NO_IMAGING` state — a Linux shard host with no `libgdiplus` cannot decode a sprite at all, and the reason names the package. No public counterpart: the pictures are served as ordinary files under `/uploads`. |
| POST | `/shard/assets/import` | import creature artwork from the shard's UO client; `{force}` ignores the hash gate, `{approve}` accepts a catalogue that no longer offers assets this site holds (refused by default — an unmounted client volume and a deliberate downgrade are indistinguishable, and the wrong guess deletes artwork). **This is the ONLY thing that imports** — boot deliberately never calls the shard. An operator's `spawnAtlas.art.json` always wins over an imported sprite. A body this client has no art for is **not** a failure: about half the addressable body range is in that state on a stock client. |
| POST | `/shard/assets/warm` | fetch item and land pictures this site is missing, now, instead of waiting for the warm timer (`adminOnly`, Protocol 8 phase 5). The pass works out which item pictures the site's own rows name — every distinct (ItemID, hue) on a player vendor, plus anything a character sheet has shown since the last pass — and fetches the ones it does not hold, **hued on the shard**, into `uploads/items/`. There is deliberately **no manifest and no bulk import**: the client addresses 49,152 item graphics times 3,000 hues, so the working set is what the site displays. `{force}` re-fetches pictures already held (how an operator recovers a wiped uploads volume); `{limit}` bounds one pass, default 400, because the shard serves one asset request at a time. A plugin overlay older than phase 5 answers `status:"unavailable"`, `code:"UNSUPPORTED"` with a sentence naming the fix. |
| PUT | `/shard/clilocs/path` | point the site at a different cliloc base file or directory (persisted as `cliloc_client_path`, which wins over `UO_CLIENT_PATH`). Overlays are read from `custom/` beside it either way. Blank clears it. Deliberately **does not import**, same reasoning as the atlas path. |
## 4. Shard visibility — the audience boundary (Protocol 3.0)
Items on the wire carry a `LabelNumber`, not a name. The bridge has always sent it —
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
