RunicGateway/docs
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(link): the hue belongs where the files are, and the cache poisons it (Phase 5)

Browse Source 
§11.1 is new and carries what phase 5 measured: 49,152 addressable static ids
(not the 81,884 `artidx.mul` declares -- `FileIndex` sizes its table from its
length ARGUMENT), 39,189 with art, 4,244 land tiles, 9,963 + 12,140 empty index
slots, and the whole set at 81 MB decoding in 34 s. That last number reopens the
bulk question and the answer is still no: 108 MB of base64 through a 512 KB
single-slot channel to store 43,433 pictures a shard displays a few hundred of.

Two traps, both §4.5's failure mode -- a confident, plausible, wrong picture:

- `Art.GetStatic` hands back the SAME cached Bitmap and `Hue.ApplyTo` repaints in
  place, so hueing edits the library's own copy: the plain key comes back hued
  from then on, and the next hue stacks. `Files.CacheData` off process-wide fixes
  it and also stops a game server retaining 74 MB of Bitmap. Copying instead does
  not solve the retention, and `new Bitmap(src)` throws on ARGB1555 anyway.

- `PartialHue` (13,259 of 65,536 ids) decides whether a hue repaints every pixel
  or only the grey ones, from a file only the shard has. Item 597 is a wooden
  screen with painted flowers; one mode reddens the flowers, the other the whole
  screen, and both decode. Hence land takes no hue segment and `h0` is not a key.

Plus the namespace trap that compiled: unqualified `TileData` binds to ServUO's
own `Server.TileData`, because the enclosing namespace beats `using Ultima;`.

§14 records what the wire gained -- the `static` and `land` families, `families`
on `assets.sources`, and `assets.fetch` becoming shared plumbing whose family is
DERIVED from the keys (§5 made the key the address; a request naming its family
too would have two places to be wrong and one of them silent). Additive, so the
protocol stays 8 and EXTRACTOR_VERSION stays 2. §15 records that `link` needed
nothing in phases 4 or 5: it forwards verbatim in both directions.

§17.10 is the four org-lead decisions. §12 and modules/uo/SCHEMA.md carry the
website side: `uploads/items/`, per-row `catalog` staleness, and why a key with
no art writes no row at all.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4
This commit is contained in:
claude
2026-09-11 06:16:15 -05:00
parent 2590d4cc58
commit a4b63d87d2
3 changed files with 172 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

147
link/v8.md
View File

@@ -1017,8 +1017,8 @@ project. §1.1's measurements make the sizing question concrete:
| Kind | Addressable | Bulk? |
|---|---|---|
| Item statics | **~49,150** | No — on demand, cached, keyed by `itemId` (+ hue) |
| Land tiles | **16,384** | No — on demand |
| Item statics | **49,152** addressable, **39,189** with art (§11.1, phase 5) | No — on demand, cached, keyed by `itemId` (+ hue) |
| Land tiles | **16,384** addressable, **4,244** with art | No — on demand |
| Creature/player bodies, first frame | **1,022** — 787 legacy (§4.8, *not* the 1,144 the library reports) + 235 UOP (§4.9) | **Yes** — this is the catalogue |
| One body, every action, one direction | **210 frames** (body 400); 96–210 measured across six bodies | No — on demand, per body |
| All bodies, every action, one direction | **~119,000 frames**, ~117 MB | No — but no longer unthinkable |
@@ -1054,9 +1054,94 @@ default and still not something to import before anything asks for it, but it ha
bulk-fill-everything switch rather than assuming on-demand is the only mode.
**Hued variants are on demand, always.** `static/3922/h33` is generated when something on the wire
actually carries hue 33. The cross product of 49,150 statics and ~3,000 hues is not a set anyone
actually carries hue 33. The cross product of 49,152 statics and 3,000 hues is not a set anyone
enumerates.
### 11.1 What phase 5 measured, and the two traps it found
The sizing above was an estimate taken off `art.mul`'s length. Measured through the reader itself,
against this machine's stock client:
| | Measured |
|---|---|
| Static ids addressable | **49,152** |
| ...with real art | **39,189** |
| ...empty index slots (§4.5's shape) | **9,963** |
| Land tiles addressable / with art | 16,384 / **4,244** |
| Whole static + land set, as PNG | 43,433 files, **81 MB**; mean 1.9 KB, max 30.9 KB (id 18213) |
| Time to decode and encode all of it | **34 s** |
| Hue slots in `hues.mul` | 3,000 (2,062 named) |
| Item ids flagged `PartialHue` | **13,259** of 65,536 |
Two of those need saying out loud.
**49,152, not the 81,884 entries `artidx.mul` declares.** `FileIndex` sizes its index table from the
**length argument it is constructed with** (`0x10000`), not from the idx file, so the addressable
static range is `0x10000 - 0x4000`. A ceiling read off the file instead would invent 16,348 ids and
answer every one of them out of an array nobody bounded. (The first probe of this phase made exactly
that mistake and reported 65,500 — PowerShell returns `$null` for an out-of-range array index rather
than throwing, so the over-run counted silently as "empty slots".)
**81 MB is small enough to reopen the bulk question, and the answer is still no.** Not on size — on
what the transfer buys. Base64 puts it at 108 MB through a 512 KB single-slot channel, roughly 210
round trips, to store 43,433 pictures of which a live shard displays a few hundred. On-demand stays
right; phase 6's bulk-fill switch is where an operator who wants the lot says so.
#### The library's cache poisons a hued sprite
`Art.GetStatic` and `Art.GetLand` memoise into a static `Bitmap[0xFFFF]` and hand back **the same
instance** on every call; `Hue.ApplyTo` repaints a bitmap **in place**. So the obvious implementation
— ask the library, apply the hue, encode — edits the library's own copy. Measured before the fix:
hue item 3922 once, and every later request for the **plain** 3922 comes back hued, with a second hue
stacking on the first.
This is §4.5's failure mode exactly — a confident, plausible, correctly-sized wrong picture that
every success count agrees with — reached through a door §4.5 never looked at, because phase 0 was
auditing *records* and this is the library's *cache*. Nothing downstream can see it: the key is
right, the dimensions are right, the hash is stable.
The fix is `Files.CacheData = false` for the life of the process, set once when the asset plane
initialises, and it pays twice: the same array is never trimmed, so decoding this client's 39,189
statics would otherwise leave **74 MB of `Bitmap` in a static field of a game server** to serve
pictures nobody asks for twice. `Animations` does not consult the flag at all, so the body catalogue
is untouched, and each reader keeps its own cache of *encoded PNG bytes* instead — a tenth of the
size, already hashed, released when it goes idle.
The obvious alternative, copying each bitmap before hueing, was rejected for the retention alone —
but also because `new Bitmap(src)` **throws** on the `Format16bppArgb1555` these decoders produce.
The copy has to name the source pixel format explicitly, which is a subtlety on the wrong side of a
correctness boundary. The invariant is instead re-checked before every hue: a hue is refused if the
cache is somehow on, because an invariant nothing verifies is a comment.
#### `PartialHue` decides the picture, and only the shard can read it
A hue is not a tint. It is a 32-entry colour ramp out of `hues.mul` indexed by each pixel's own red
channel — and whether it replaces **every** pixel or only the grey ones is a per-item-id flag in
`tiledata.mul`. On this client **13,259 of 65,536 item ids carry it**.
Item 597 is a wooden screen with painted flowers. Hued 33 the right way the flowers turn red; the
wrong way the whole screen turns red. Both decode, both are 44×112, both report success. This is why
§5 put hue in the key rather than leaving it to the website: shipping `Hues.mul` semantics and a
65,536-row flag table into Node, to answer a question the shard can answer for free, is the trade
§2.1 already refused.
Two consequences fall out of it. **Land takes no hue segment** — the mode is an *item* flag and land
has no equivalent, so `land/3/h33` is refused rather than guessed; nothing on the wire carries one
today, and if something ever does it will arrive with a reason to choose. And **`h0` is not a key**:
hue 0 on the wire means "not hued", so the plain key already names that picture, and accepting both
would store one PNG twice under two names and diff them separately forever.
#### The namespace trap that compiled
The first cut of the reader wrote `TileData.ItemTable` and `TileFlag.PartialHue` unqualified. ServUO
declares its **own** `Server.TileData`, `Server.ItemData` and `Server.TileFlag` — with a
`PartialHue` member — in `Server/TileData.cs`, and the reader lives in `Server.Custom.Bridge`, where
the enclosing namespace beats `using Ultima;`. It compiled. At runtime it read a file resolved
through `Core.DataDirectories` rather than through `Ultima.Files`, which is §4.6's rule broken in a
new place: deciding a picture with a file other than the one the pixels came out of. The live rig
caught it as a `TypeInitializationException` refusing every hued key, from a class the code never
meant to name.
**Gump art is out of scope for Protocol 8, and that is now a safety rule rather than a priority
call** — §4.1. It is the only decoder that reaches the `hasExtra: true` branch, and that branch
corrupts the process on the second id. Adding gump art later means fixing that path first,
@@ -1077,6 +1162,12 @@ so no `MODULE_API_VERSION` bump is needed to store them.
- **The operator-supplied `spawnAtlas.art.json` map stays supported** and continues to win over an
imported asset. An operator who has drawn their own creature portraits must not have them
overwritten by a sprite rip on the next Update.
- **Item and land pictures land in `uploads/items/`, beside the creature portraits and not among
them** (phase 5). Same content-addressed naming (`uo-static-3922-h33-<sha8>.png`), same "the API
returns a filename, the client builds the URL" contract, and the same rule that a missing picture
is a first-class state rather than an error. Separate directories because they have different
lifetimes: the catalogue is imported as a set and re-imported as a set, while these arrive one at
a time because something asked for them.
Licensing is unchanged and the reasoning is unchanged: these are the operator's own client files,
extracted on their own host, for their own shard. Nothing is committed, nothing ships in a repo,
@@ -1145,7 +1236,7 @@ new pipe.
|---|---|---|---|
| `assets.sources` | `assets.sources.ok` | Stage 1: client file manifest + `EXTRACTOR_VERSION` | **phase 1** |
| `assets.manifest` | `assets.manifest.ok` | Stage 2: `[{key, sha256, bytes, width, height, body, direction, source}]`, paged | **phase 3** (`source` phase 4) |
| `assets.fetch` | `assets.fetch.ok` | Content for an explicit key list, paged; base64 PNG per row | **phase 3** (`source` phase 4) |
| `assets.fetch` | `assets.fetch.ok` | Content for an explicit key list, paged; base64 PNG per row | **phase 3** (`source` phase 4; `static`/`land` families phase 5) |
| `assets.bodies` | `assets.bodies.ok` | Slug → body id (§8, Core thread) | **phase 3** |
| `cliloc.table` | `cliloc.table.ok` | The decompressed table, paged (`?lang=`, `?cursor=`) | **phase 2** |
| `tree.manifest` / `tree.fetch` | `.ok` | §10, the ServUO tree files | phase 7 |
@@ -1167,6 +1258,27 @@ side effect — a few hundred asset keys do not belong in a query string. They a
this link that take one. `422` gains a second meaning on this plane alongside "cannot decode": the
mid-import guard, a `catalog` that no longer describes the files on disk.
**Phase 5 added two families, one field, and no command.** `assets.fetch` grew `static` and `land`
(§5, §11.1) and `assets.sources` grew **`families`** — which key families this overlay serves. Both
are additive, so **the protocol stays 8**, and `EXTRACTOR_VERSION` stays **2**: no existing key's
bytes change, and a new key is not a re-derivation of an old one.
The command itself became shared plumbing. Phase 3 gave `assets.fetch` to the body catalogue
outright, which was right with one family and wrong with three: the command is the *transport* and
the family is a property of the key. So the correlation id, the operator's consent, the key-count
ceiling and the family decision now happen once, and a reader only ever sees keys it owns.
**The family is derived from the keys and is not a request field.** §5 made the key the address of an
asset; a request that also named its family would have two places to be wrong and one of them
silent. A batch must be of **one** family — mixing them is refused (400) rather than split — because
the reply carries a single `catalog` id, and two families have two fingerprints. A reply claiming one
of them would be lying about the other.
`families` matters more than it looks. Without it, a website talking to a phase-3 or phase-4 overlay
discovers the gap as a refusal *per key, per pass, forever*, with no picture ever appearing and a
warning in the log every few minutes. With it, that is one reported state carrying a sentence naming
the fix.
**Phase 4 added one field and no command.** `source` on a manifest or fetch row is `legacy` or
`uop` — which reader produced the bytes (§4.9). It is additive, so **the protocol stays 8**: a
consumer that does not read it is unaffected, and one that does can say which half of the extractor
@@ -1194,7 +1306,7 @@ disagree, so a split bump means the next bundle silently fails to compose.
| Repo | Work |
|---|---|
| `servuo-plugins/` | Extraction over ServUO's own `Ultima` (§4), the cliloc decompressor (§9), body resolution (§8), the request handlers, `overlay.toml` |
| `link/` | Six command families forwarded, the REST surface, **the inbound line cap (§3.3)**, `PROTOCOL_VERSION` |
| `link/` | Six command families forwarded, the REST surface, **the inbound line cap (§3.3)**, `PROTOCOL_VERSION`. **Nothing in phases 4 or 5** — `assets_call` forwards a request body verbatim and `respond_assets` returns the reply verbatim, so a new key family and a new reply field both pass through untouched |
| `module-uo/` | Client calls, asset store, the atlas source backend (§10), cliloc ingest, admin surface |
| `website/` | Almost none — `ctx.uploads` already suffices (§12). Phase 2 deleted `server/tools/cliloc-export/`, the converter this protocol retires |
| `docs/` | This file; rewrite `CLILOCS.md` §Converting and `SPAWN_ATLAS.md` §Artwork + §Configuring; **delete `UOFIDDLER.md`**; add the libgdiplus prerequisite to `SHARD_PREREQS.md` (§4.4) |
@@ -1213,7 +1325,7 @@ disagree, so a split bump means the next bundle silently fails to compose.
| 2 | **DONE 2026-09-10.** Clilocs end to end (§9.1, §9.2): the Mythic decompressor ported into the overlay, `cliloc.table` + `GET /cliloc`, the paging walk and the source switch on the website, module-uo's protocol pin 7→8. `cliloc-export/` deleted and `UOFIDDLER.md` §Part 1 with it. **67,496 rows, 290 ms, ~11 pages** — the same count UOFiddler's own DLL produced from this client | all |
| 3 | **DONE 2026-09-10.** Body resolution (§8) + the **787**-body catalogue (§4.8), `assets.manifest` / `assets.fetch` / `assets.bodies` and their REST mirrors, `shard_spawn_creatures.art` filled and rendered (§8.1, §12.1). **787 rows in one 734 ms page; 455 types resolved at ~190 ms per 100 on the Core thread; zero mobiles leaked.** `UOFIDDLER.md` deleted, two phases early | servuo-plugins, **link**, module-uo |
| 4 | **DONE 2026-09-11.** The UOP animation decoder (§4.3, §4.9): `BridgeUop` + a PNG encoder that never touches `System.Drawing`, wired in beneath the legacy reader. **Two of the eight player bodies turned out to exist** (gargoyles 666/667); the other six are in no client file, and ghost ids left the player-body set (§5.2, §17.9). The same fallback added **233 other bodies**: the catalogue is **1,022 rows, 1,409 ms cold**, and all six player bodies have art for the first time. `EXTRACTOR_VERSION` 1 → 2 | servuo-plugins |
| 5 | Item statics and land on demand, hued keys, the cache | servuo-plugins, module-uo |
| 5 | **DONE 2026-09-11.** Item statics and land on demand (§11.1): the `static` and `land` families, hue applied on the shard from `tiledata.mul`, the byte-bounded art cache, `assets.fetch` made family-aware, `families` on `assets.sources`. Website side: the warm pass, per-row `catalog` staleness, and pictures on the marketplace and the character sheet. **39,189 statics and 4,244 land tiles served; the only refusals are the 9,963 + 12,140 empty index slots §4.5 predicted.** Two traps found — the library's bitmap cache poisons a hued sprite, and `PartialHue` decides the picture from a file only the shard has. Protocol stays 8; `EXTRACTOR_VERSION` stays 2 | servuo-plugins, module-uo |
| 6 | Deep animation keys (`body/<id>/a<n>/f<n>`) for the future project, plus the bulk-fill switch | servuo-plugins, module-uo |
| 7 | The atlas over the sidecar (§10); shared-filesystem requirement retired | module-uo |
| 8 | Admin surface, Import/Update, approve/reject, activity log | module-uo |
@@ -1298,6 +1410,29 @@ in the document.
returning the previously-decoded body's bitmap. The elf ghosts moved from "decodes" to "no art"
in §5.2 for the same reason, taking phase 4's set from six player bodies to eight.
10. **§11.1: phase 5's four, settled 2026-09-11.** Put to the org lead after the client was measured
and before the reader was written, because the first measurement changed what the risk was:
- **Ingest warms; the route only serves.** A page renders the pictures already on disk and
leaves out the ones that are not; fetching happens behind it, on a timer, from the keys the
site's own rows name. Chosen over fetching on first request, on one number: the asset plane
serves **one request at a time** (§3.2), so a URL that fetched would let any anonymous visitor
walk 49,152 ids × 3,000 hues through that single slot and park an operator's own import behind
it. Warming from the site's own data has no such surface — the ceiling is the number of
distinct (item, hue) pairs the shard has already told the site about.
- **Staleness is a per-row catalogue id, not a manifest.** A client patch changes the shard's
`catalog` fingerprint and a restart does not, so "is this out of date?" is a column
comparison. The alternative — a `static` manifest family enumerating 39,189 rows with hashes —
would cost a 34-second scan of the whole art file per Update to answer a question about maybe
three hundred pictures, and would re-fetch art nobody looks at any more. Lazy costs nothing
for the ones nobody wants.
- **The pictures appear on the marketplace and the character sheet**, the two places the data
already existed and rendered as `id 1234, hue 33`. That gives the phase an acceptance test
checkable by eye, which §11.1 says is the only kind that catches this failure mode.
- **`Files.CacheData` goes off process-wide** at asset init, rather than copying each bitmap
before hueing. See §11.1: the copy does not solve the 74 MB retention, and the ordinary copy
constructor throws on ARGB1555 anyway.
9. **§4.3/§5.2: phase 4's four, settled 2026-09-11.** Put to the org lead after the packages were
opened and before the reader was written, because the first measurement changed what the phase
was worth: