diff --git a/README.md b/README.md index 6b65120..2830a51 100644 --- a/README.md +++ b/README.md @@ -35,8 +35,8 @@ sidecar as a service, and hands you the values the website needs. | [SHARD_VISIBILITY.md](website/SHARD_VISIBILITY.md) | Who sees which shard data — the admin-configurable audience framework | | [TRUSTED_DEVICES_MFA.md](website/TRUSTED_DEVICES_MFA.md) | TOTP two-factor, trusted devices and recovery codes | | [MODERATION_APPEALS.md](website/MODERATION_APPEALS.md) | Moderation actions, content reports and the appeals flow | -| [SPAWN_ATLAS.md](website/SPAWN_ATLAS.md) | The bestiary / spawn atlas: what the shard contains, parsed from its own ServUO tree | -| [CLILOCS.md](website/CLILOCS.md) | UO's id → name table: converting one from your client so items have names | +| [SPAWN_ATLAS.md](website/SPAWN_ATLAS.md) | The bestiary / spawn atlas: what the shard contains, parsed from the shard's own ServUO files — served over the bridge since Protocol 8, so no shared filesystem | +| [CLILOCS.md](website/CLILOCS.md) | UO's id → name table, so items have names. The shard decompresses and serves it over the bridge; the desktop conversion it replaced is gone | | [MARKETPLACE.md](website/MARKETPLACE.md) | The player-vendor index: how it is gathered, what it costs, how to tune it | | [website-README.md](website/website-README.md) | Snapshot of the website repo's README (setup/run reference) | | [test-plan.md](website/test-plan.md) | The website's test strategy and harness | @@ -64,9 +64,10 @@ particular game; a module is what makes it a site *for* one. | [PROTOCOL_2.md](link/PROTOCOL_2.md) | Protocol 2.0 / 2.1 design | | [v3.md](link/v3.md) | Protocol 3.0 design — shard content/standings streams + the visibility framework | | [v4.md](link/v4.md) | Protocol 4.0 — guild membership on the wire (`guild.roster`, `guild.leave`) | -| [v5.md](link/v5.md) | Protocol 5 — three enrichments in one bump: `house.decay`'s decay schedule, `vendor.listing`'s fee state, and `account.login.result`. **The current protocol**, shipped 2026-09-01 as bundle 2026.09.01 (sidecar v2.1.0 + overlay v1.1.0) | -| [v6.md](link/v6.md) | Protocol 6 — idempotent commands, config leases with a shard-side deadline, and the run-scoped participation ledger. On `edge`; reaches `main` at the events cutover | -| [v7.md](link/v7.md) | Protocol 7 — the world verbs an event OWNS: creatures, bosses, oracle NPCs, temporary gates, decoration, and the persisted ownership registry behind them. On `edge` | +| [v5.md](link/v5.md) | Protocol 5 — three enrichments in one bump: `house.decay`'s decay schedule, `vendor.listing`'s fee state, and `account.login.result`. Shipped 2026-09-01 as bundle 2026.09.01 (sidecar v2.1.0 + overlay v1.1.0) | +| [v6.md](link/v6.md) | Protocol 6 — idempotent commands, config leases with a shard-side deadline, and the run-scoped participation ledger | +| [v7.md](link/v7.md) | Protocol 7 — the world verbs an event OWNS: creatures, bosses, oracle NPCs, temporary gates, decoration, and the persisted ownership registry behind them. **The released protocol**, bundle 2026.09.10 (sidecar v2.2.0 + overlay v1.2.0) | +| [v8.md](link/v8.md) | Protocol 8 — the Asset Bridge: the shard reads its own UO client and serves creature art, item and land pictures, the cliloc table and its own spawn files, so nothing is converted on a desktop and the website needs no shared filesystem. On `edge` | | [ADMIN_CONTROLS.md](link/ADMIN_CONTROLS.md) | Staff write-plane (kick/ban/broadcast, page queue) | | [SHARD_PREREQS.md](link/SHARD_PREREQS.md) | Shard-side prerequisites for the bridge | | [PLAN.md](link/PLAN.md) | uo-link build plan | diff --git a/installer/INSTALL.md b/installer/INSTALL.md index 56e3689..d5e3877 100644 --- a/installer/INSTALL.md +++ b/installer/INSTALL.md @@ -57,6 +57,7 @@ Only the sidecar is exposed, and only to your website. | Outbound HTTPS | To `gitea.whitlocktech.com`, to fetch the bundle and the two artifacts. Nothing inbound is needed, and no Gitea account or git client is required. | | The sidecar on the **same host** as the shard | The shard connects to `127.0.0.1:7788`. Splitting them is not supported — the loopback socket *is* the trust boundary for inbound commands. | | Admin access to your Runic Gateway site | The last step is pasting four values into Admin → Shard (uo-link). | +| `libgdiplus` *(Linux hosts only)* | Only needed for **artwork**. ServUO runs under Mono on Linux, and Mono's `System.Drawing` — which the shard uses to decode sprites out of your UO client — is a thin layer over this library. Without it the site's creature portraits and item pictures stay empty; names and the spawn atlas are unaffected. `sudo apt-get install libgdiplus` (or `dnf install libgdiplus`); `doctor` checks for it. **Windows hosts need nothing.** See [`../link/SHARD_PREREQS.md`](../link/SHARD_PREREQS.md). | **Back up first.** The overlay overwrites `Scripts/Scripts.csproj` (a stock file), and the patch tier edits stock sources. A copy of `Scripts/` and `Config/` before you start costs nothing. @@ -527,10 +528,18 @@ first thing a maintainer will want. ✓ Protocol sidecar 4 = overlay manifest 4 ✗ Shard connected no — the shard is running (pid 8123) but has not dialed in ✓ Bundle 2026.08.19 — up to date +⚠ Imaging (libgdiplus) not found on this host + this shard cannot decode artwork out of its UO client — creature portraits and + item pictures will be absent; names and the spawn atlas are unaffected + install it: apt-get install libgdiplus / dnf install libgdiplus + see docs/link/SHARD_PREREQS.md — Windows hosts need nothing ✓ Backups 2026-08-19T09:12:44Z — 3 file(s) replaced by update to bundle 2026.08.19 3 kept in /etc/runicgateway/backups ``` +The imaging row appears **on Linux hosts only** — a Windows host ships `System.Drawing` with .NET +Framework and has nothing to check, so there is no row rather than a row saying "not applicable". + Rows come from asking the installed sidecar (`--version`, `--print-config`) rather than from reading `install.json`, so `doctor` reports what the binary would actually do — including which config and database file the *service* resolves — rather than what the installer believes it was told. The diff --git a/link/INTEGRATION.md b/link/INTEGRATION.md index 204c452..1a2cb9f 100644 --- a/link/INTEGRATION.md +++ b/link/INTEGRATION.md @@ -55,14 +55,14 @@ That is the same set of values Admin → Shard asks for — base URL and WS URL The wire protocol is versioned so a mismatch is caught immediately instead of failing weirdly. -The current version is **6**. It is not released yet — it lives on `edge` and ships with the event system's cutover; the last released pairing is protocol **5**, sidecar **v2.1.0** + overlay **v1.1.0**, resolved as bundle **2026.09.01**, never as "latest of each". +The current version is **8**. It is not released yet — it lives on `edge` and ships with the Asset Bridge's cutover; the last released pairing is protocol **7**, sidecar **v2.2.0** + overlay **v1.2.0**, resolved as bundle **2026.09.10**, never as "latest of each". - Every response carries an **`X-UOLink-Version: 8`** header. -- `GET /health` and the WebSocket `ws.hello` frame include `"protocol": 6`. +- `GET /health` and the WebSocket `ws.hello` frame include `"protocol": 8`. - **Optionally**, send `X-UOLink-Version: 8` on your requests. If it disagrees with the sidecar, the request is rejected **409 Conflict**: ```json - { "error": "protocol version mismatch", "sidecar_protocol": 6, "client_protocol": "5" } + { "error": "protocol version mismatch", "sidecar_protocol": 8, "client_protocol": "7" } ``` Pin the version you built against and compare it to the header (or `/health.protocol`) at startup. @@ -118,6 +118,41 @@ the work. Additive again: no existing field changed shape, **no new endpoint**, and a client that sends no key behaves exactly as it did under v5. +**v7 (Protocol 7)** adds the **event plane's command half** ([`v7.md`](v7.md)) — the verbs a +website-authored event needs in order to happen in the world: spawn a bounded, named, hued set of +creatures; place decoration from the shard's own `Data/Decoration` vocabulary; open a temporary +gate; run an oracle NPC's dialogue; grant an item to a run's participants; lease a property on an +existing spawner; toggle one of ServUO's seasonal events; start a world save. Every one of them is +capped on the shard, ledgered so it can be reverted, and gated by `Bridge.EventsEnabled` — an +operator who has not switched the plane on gets **403**, not a half-applied event. + +Additive on the feed, like every bump before it, and the new endpoints require a v7 sidecar. The +one thing to know as a consumer is that these are **world writes**, so they are exactly the calls +v6's `idempotencyKey` exists for: send one, and a retry after a lost acknowledgement is answered by +the original reply instead of spawning the creatures twice. + +**v8 (Protocol 8)** adds the **asset plane** ([`v8.md`](v8.md)) — the shard reads its own UO client +files and serves what is in them, so nothing has to be converted on somebody's desktop and no +component but the shard ever needs a copy of the client. It carries four things: creature and +player-body artwork, item and land art on demand, the decompressed cliloc table, and the shard's own +`Spawns/*.xml` tree. See §5's *Client assets* for the routes. + +Three properties of this plane are unlike the rest of this API, and a consumer that does not know +them will misread healthy behaviour as failure: + +- **The shard serves one asset request at a time**, and says so: `425 Too Early` is *flow control*, + the ordinary answer during an import rather than a rare collision. Back off and retry; do not + treat it as an error and do not abandon a transfer over it. +- **Every response is paged, and the caller drives the paging** — echo the previous reply's + `cursor` until one says `more: false`, then read `cut` to learn why that page was the last. Only + `cut: "end"` means you have the whole thing. +- **A key the shard cannot serve comes back as a row with a `status`, not as a failed request.** A + body this client has no art for is the expected answer for most ghost and gargoyle bodies on a + stock client; failing a page over one would make an import impossible. + +Additive again — no existing field changed shape, and a v7 client that never calls `/assets/*` or +`/cliloc` behaves exactly as it did. + **Upgrading a pinned client.** Every bump is an operator-visible hard break in one direction only: a client still declaring the old number gets a 409 on every protected route and, on the WebSocket, a closed connection on the `ws.hello` mismatch. So update the pinned version at the same time you @@ -743,7 +778,7 @@ Full character sheet: stats, all trained skills, worn equipment with flattened i Field notes: - `skills[].base` is trained value, `value` includes item/temp bonuses, `cap` is the cap. **Do not assume `base <= cap`** — GM characters can exceed it. - `equipment[].mods` is a flattened map of every non-zero AOS attribute on the item (weapon or armor). Empty `{}` for plain items. -- Item names are usually **clilocs**, not strings: use `name` when present, otherwise resolve `cliloc` against a UO cliloc table on the site. **Do not expect the shard to resolve them for you** — on any modern client ServUO's own `Ultima.StringList` cannot read the client's compressed cliloc files, so `VendorSearch.GetItemName` returns `item.Name` and the in-game Vendor Search gump has the same gap. Building that table is a consumer-side job; the website's is described in [`website/CLILOCS.md`](../website/CLILOCS.md). +- Item names are usually **clilocs**, not strings: use `name` when present, otherwise resolve `cliloc` against a UO cliloc table on the site. **Do not expect the shard to resolve them for you per item** — on any modern client ServUO's own `Ultima.StringList` cannot read the client's compressed cliloc files, so `VendorSearch.GetItemName` returns `item.Name` and the in-game Vendor Search gump has the same gap. As of Protocol 8 the shard will hand you the whole table instead: `GET /cliloc` (§5), which is where the website's own comes from ([`website/CLILOCS.md`](../website/CLILOCS.md)). - `titles` (Protocol 2.0): `selected` is the index into `reward` currently displayed (`-1` if none). `fameKarma`/`skill` are computed display titles, omitted when the character has none. `reward` entries may be a **cliloc number as a string** or a literal string — resolve numeric ones against your cliloc table, same as item names. - Errors: unknown account → **404** `{"kind":"bridge.error","reason":"unknown account"}`; bad slot → **404**/**400** similarly. @@ -778,6 +813,90 @@ Every player vendor owned by any character on the account, with held gold and cu ] } ] } ``` +### Client assets — artwork, names and the shard's own files (Protocol 8) + +Five routes, and they are unlike everything above them: they read the **UO client installed on the +shard host** rather than the live world. Nothing here changes unless an operator patches that +client, so these are the only reads on this API you should cache indefinitely and refresh on an +event you decide, not on a timer. The design is [`v8.md`](v8.md); what follows is what a consumer +needs. + +Everything on this plane obeys three rules stated in §2: **425 is flow control**, **the caller +drives the paging** (`more` / `cursor` / `cut`), and **an unserveable key is a row, not a failure**. + +``` +GET /assets/sources # what the shard's client files currently are +GET /assets/manifest?family=&cursor= # one row per asset: key, hash, size — no pixels +POST /assets/fetch # the bytes, for keys you name +POST /assets/bodies # creature class name → body id +GET /cliloc?lang=&cursor= # the decompressed cliloc table +``` + +**`GET /assets/sources` is the gate, and you call it first.** It is cheap, it touches no pixels, and +its answer decides whether there is anything to do at all: + +```json +{ "kind":"assets.sources.ok", "assetsEnabled":true, "treeEnabled":true, + "extractorVersion":3, "families":["body","land","static","tree"], + "hashing":false, "complete":true, "imaging":{"ok":true}, + "files":[ {"name":"anim.mul","size":194950053,"mtime":1778566017000,"sha256":"2d89…"} ], + "more":false, "cut":"end" } +``` + +- **`files[].sha256` is the whole update story.** Store the set; on the next run, compare. Equal + means the client has not been patched and there is nothing to transfer — which is the normal case + and must cost one round trip, not a re-download. +- **`hashing: true` means "not computed yet", never "changed".** The shard fingerprints a few + hundred megabytes in the background after a restart; a null hash during that window is an absence + of an answer, not an answer of absence. +- **`extractorVersion` is ours, not the client's.** It changes when the shard's *derivation* changes + — a better reader, a different frame — so a bump invalidates stored pictures even though the + client files are byte-identical. Treat a change in it exactly like a changed hash. +- **`families` says what this overlay can serve.** An older overlay answers `["body"]`; asking it + for `static` fails per key, per pass, forever. Read this and say "update your plugin" instead. +- **`imaging.ok: false` is `NO_IMAGING`** — a Linux shard host with no `libgdiplus` cannot decode a + sprite at all ([`SHARD_PREREQS.md`](SHARD_PREREQS.md)). Names and tree files are unaffected: they + have no pixels in them. + +**`GET /assets/manifest?family=body`** lists what exists, with a hash and a size and no pixels — so +you can diff it against what you hold and fetch only what moved. Rows carry +`{key, sha256, bytes, width, height, body, action, source}`; `source` is `legacy` or `uop` (which +reader produced the bytes) and `action` is which animation action the thumbnail came from, because a +body with nothing at action 0 is catalogued at the first action that has anything — `body/820/a23` +is a horse. This family pages on the shard's wall clock, so expect several pages of a few hundred +rows. + +**`POST /assets/fetch`** takes `{"keys":[…], "catalog":"…", "cursor":"…"}` and returns a row per key +carrying the sprite as **base64 PNG** (the `tree` family returns gzipped chunks instead). Two things +are load-bearing: + +- **Pass back the `catalog` id the manifest gave you.** It is derived from the client files + themselves, and it makes the shard refuse (**422**) if those files moved mid-import. Without it, + an operator who patches their client halfway through gets one asset set stitched out of two, with + no error anywhere. +- **A batch must be of one family**, derived from the keys rather than named as a field. Mixing is + refused (400), because a reply carries a single `catalog` and two families have two fingerprints. + +**`POST /assets/bodies`** takes `{"types":["GiantSpider", …]}` and answers the one question only code +running inside ServUO can: which body id a creature class actually uses. It runs on the Core thread, +so the shard **caps the batch and refuses rather than truncates** — a 400 here names the cap; chunk +your list. + +**`GET /cliloc?lang=enu`** is the client's id → text table, decompressed on the shard and delivered +as rows (`[{n, f, t}]`) inside the ordinary paging envelope. Blank entries are omitted — roughly +56,000 of them — because every consumer discards them anyway. This is the table you resolve +`cliloc` numbers against in `vendor.listing`, `char.profile` equipment and title rows; §6's note +that the shard cannot resolve them for you is **why this route exists**. + +Two absences worth stating, because both look like bugs and neither is: + +- **A body with no art is normal**, and on a stock client that is most ghost and gargoyle bodies. + The row says so; render text. +- **The shard never sweeps file types looking for a hit.** If the client's own mapping yields + nothing, the answer is nothing — asking the other animation files for the same index returns a + *different creature's* art that reports success, which is a wrong picture nothing downstream can + detect ([`v8.md`](v8.md) §4.6). + --- ## 6. REST — commands & history @@ -1254,13 +1373,13 @@ sidecar defines no audiences. Deciding who may see what is the consuming site's | 200 | OK | | 400 | Bad request (malformed body, invalid parameter, or a shard `*.error` that isn't a not-found) | | 401 | Missing or invalid auth token | -| 403 | Refused by the operator — the admin write plane, or the event plane (`Bridge.EventsEnabled`), is switched off on the shard | -| 404 | Not found (unknown account / character / id, a not-linked account, an unoffered lease key, or a run the shard is not counting) | +| 403 | Refused by the operator — the admin write plane, the event plane (`Bridge.EventsEnabled`), or the asset plane (`Bridge.AssetsEnabled` / `Bridge.TreeEnabled`), is switched off on the shard | +| 404 | Not found (unknown account / character / id, a not-linked account, an unoffered lease key, a run the shard is not counting, or a client file this install does not have) | | 409 | Conflict — protocol version mismatch, or an account name already taken on `POST /accounts/create` | -| 425 | Too Early — a command with this `idempotencyKey` is still in flight on the shard (Protocol 6). Nothing ran; retry | +| 425 | Too Early — a command with this `idempotencyKey` is still in flight (Protocol 6), or the shard's single asset slot is held by another `/assets/*` or `/cliloc` call (Protocol 8). Nothing ran; back off and retry | | 429 | Too many requests — the shard's per-IP account cap was hit on `POST /accounts/create` | | 500 | Internal error (e.g. database) | -| 503 | Shard not connected — the query needs the live game and it's down | +| 503 | Shard not connected — the query needs the live game and it's down. On the asset plane it also carries `NO_IMAGING`: the host cannot render images at all | | 504 | Shard connected but didn't reply within 10s | `503` vs `404`: a `503` is transient (shard restarting — retry), a `404` is a real "doesn't exist." @@ -1270,6 +1389,10 @@ deployment fault — your pinned protocol version disagrees with the sidecar's never help. A `425` is a retry that will succeed on its own. They are deliberately different codes so a retry loop cannot quietly swallow a mismatched deployment. +`422` exists only on the asset plane, and it means one of two things, both of which are the client +files moving under you: a file the shard cannot decode, or a `catalog` id that no longer describes +what is on disk — the mid-import guard. Start the import again rather than retrying the page. + --- ## 8. Putting it together @@ -1304,3 +1427,8 @@ const sales = await fetch(`${BASE}/history?kind=vendor.sale&limit=20`, { headers - **Cache freshness.** `GET /char/serial/...` may serve a stale cached profile when the shard is down; the account+slot form always goes live (503 if down). - **`bootId`** on `server.hello` is your signal to invalidate site-side caches: if it changed, the shard restarted. - **Protocol changes** bump `X-UOLink-Version`. Compare it on startup and fail fast rather than mis-parsing a newer shape. +- **The asset plane is a working set, not a stream.** `/assets/*` and `/cliloc` read files that only + change when an operator patches the shard's UO client, and the shard serves one such request at a + time. Import on an operator's action or a hash change — never on your own boot, and never on a + timer: a client-file pass costs hundreds of megabytes of hashing to discover that nothing moved, + and it holds the slot every other caller is waiting for. diff --git a/link/v8.md b/link/v8.md index aafcfbc..e108220 100644 --- a/link/v8.md +++ b/link/v8.md @@ -1678,10 +1678,10 @@ disagree, so a split bump means the next bundle silently fails to compose. | `link/` | Six command families forwarded, the REST surface, **the inbound line cap (§3.3)**, `PROTOCOL_VERSION`. **Nothing in phases 4 or 5** — `assets_call` forwards a request body verbatim and `respond_assets` returns the reply verbatim, so a new key family and a new reply field both pass through untouched | | `module-uo/` | Client calls, asset store, the atlas source backend (§10), cliloc ingest, admin surface. Phase 7 also bumped `PARSER_VERSION` 4 → 5 (§10.4) | | `website/` | Almost none — `ctx.uploads` already suffices (§12). Phase 2 deleted `server/tools/cliloc-export/`, the converter this protocol retires | -| `docs/` | This file; rewrite `CLILOCS.md` §Converting and `SPAWN_ATLAS.md` §Artwork + §Configuring; **delete `UOFIDDLER.md`**; add the libgdiplus prerequisite to `SHARD_PREREQS.md` (§4.4) | -| `installer/` | A `doctor` check for libgdiplus on Linux hosts (§4.4). Bundle pairing already enforces §15 | +| `docs/` | This file; rewrite `CLILOCS.md` §Converting and `SPAWN_ATLAS.md` §Artwork + §Configuring; **delete `UOFIDDLER.md`**; add the libgdiplus prerequisite to `SHARD_PREREQS.md` (§4.4). Phase 9a added the rest: `INTEGRATION.md` to protocol 8, an operator entry in `UPGRADE_NOTES.md`, the index rows, and the prerequisite + `doctor` row in `installer/INSTALL.md` | +| `installer/` | A `doctor` check for libgdiplus on Linux hosts (§4.4) — **built in phase 9a**, Linux-only (a Windows host has nothing to check, so there is no row rather than a row saying "not applicable") and a `⚠` rather than a `✗`, because names and the spawn atlas have no pixels in them. Bundle pairing already enforces §15 | | `android-app/` | Consumes images by URL; no parity gate expected until a screen shows one | -| `integration-kit/` | A chapter note only — this is UO-specific and teaches nothing about the module contract | +| `integration-kit/` | A chapter note only — this is UO-specific and teaches nothing about the module contract. **Built in phase 9a** as chapter 3 §2b: the *pattern* (the game host already has the files; route them over the channel you already have; request/reply, one at a time, two stages, version your derivation, never on boot) with no contract re-specified | --- @@ -1698,7 +1698,9 @@ disagree, so a split bump means the next bundle silently fails to compose. | 6 | **DONE 2026-09-11, and not what this row said.** The measurement came first and changed the phase: a complete one-direction animation set is **174,453 frames / 281.5 MB**, not the ~119,000 estimated, and **the site displays still pictures** — so the deep keys and the bulk-fill switch were **not built** (§11.2, org lead 2026-09-11). What shipped is what the still-picture site was missing: the **73 bodies with no art at action 0 and real art deeper** (a horse at `body/820/a23`), the catalogue key carrying its action, the atlas join that reads it, and §4.10's per-body **action ceiling** — without which the fallback walk itself would serve **452 validated pictures of the next body**. Catalogue **1,022 → 1,095**; `EXTRACTOR_VERSION` 2 → 3; protocol stays 8 | servuo-plugins, module-uo | | 7 | **DONE 2026-09-14.** The atlas over the sidecar (§10); shared-filesystem requirement retired. The measurement came first again and changed the shape: `tree/