From 6ce60a82c336db52f90a4ec9dc39b3d89343e31d Mon Sep 17 00:00:00 2001 From: wtclaude Date: Wed, 29 Jul 2026 09:52:06 -0500 Subject: [PATCH] =?UTF-8?q?docs(link):=20the=20player-vendor=20marketplace?= =?UTF-8?q?=20(Protocol=203.0=20=C2=A78)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documents order 5b across the four repos, and records what building it changed about §8 as designed. - NEW website/MARKETPLACE.md — the operator guide: what the pages must say out loud and why, the privacy contract (the player's in-game Vendor Search toggle wins, and no admin setting overrides it), the Bridge.cfg knobs and how they trade against each other, and the measured sweep costs. - INTEGRATION.md — catalog entry for vendor.listing / vendor.listing.remove with its six consumer gotchas, and the GET /market REST section (the sidecar's only paged read, and why it orders by serial rather than shop name). - BACKEND_DESIGN.md — shard_vendors / shard_vendor_items, the routes, and the marketplace search as the only rate-limited public read. - SHARD_VISIBILITY.md — why the market's fields default to Everyone (the in-game gump already shows exactly that set), why location is one setting covering four things, and why hiding the owner name without the owner id achieves nothing. - PLAN.md — the amortized round-robin as the one sweep pattern the bridge did not previously have, and an update to §7's cliloc note: pushing name resolution to the plugin was never an option, because ServUO cannot read a modern client's compressed cliloc files either. - v3.md §8.8 — the four things the build settled differently, chief among them that §8.1's FLAT location payload would have made Part A's pre-wired market.location rule inert, exactly like the characterName miss one part earlier. Co-Authored-By: Claude --- README.md | 3 + link/INTEGRATION.md | 101 ++++++++++++++++++++++ link/PLAN.md | 23 ++++- link/v3.md | 79 +++++++++++++++-- website/BACKEND_DESIGN.md | 53 +++++++++++- website/MARKETPLACE.md | 167 ++++++++++++++++++++++++++++++++++++ website/SHARD_VISIBILITY.md | 22 ++++- 7 files changed, 440 insertions(+), 8 deletions(-) create mode 100644 website/MARKETPLACE.md diff --git a/README.md b/README.md index 8dab127..fe4d540 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,9 @@ ci/ cross-cutting CI/quality notes | [HERO_EDITOR.md](website/HERO_EDITOR.md) | Hero canvas editor feature spec | | [WIKI_UPGRADE.md](website/WIKI_UPGRADE.md) | Wiki subsystem upgrade notes | | [SHARD_VISIBILITY.md](website/SHARD_VISIBILITY.md) | Who sees which shard data — the admin-configurable audience framework | +| [SPAWN_ATLAS.md](website/SPAWN_ATLAS.md) | The bestiary / spawn atlas: what the shard contains, parsed from its own ServUO tree | +| [CLILOCS.md](website/CLILOCS.md) | UO's id → name table: converting one from your client so items have names | +| [MARKETPLACE.md](website/MARKETPLACE.md) | The player-vendor index: how it is gathered, what it costs, how to tune it | | [website-README.md](website/website-README.md) | Snapshot of the website repo's README (setup/run reference) | | [PROJECT_TREE.md](website/PROJECT_TREE.md) | Auto-generated snapshot of the repo's tracked file layout | diff --git a/link/INTEGRATION.md b/link/INTEGRATION.md index 1b5ba20..fe3b9de 100644 --- a/link/INTEGRATION.md +++ b/link/INTEGRATION.md @@ -449,6 +449,78 @@ would carry ~25 zeroes. `maxPoints` follows the same `0 == uncapped` rule as the a points lookup stops at the character's own row, but a rank must count every row that beats them, in every system, on every profile build. Derive rank from `points.board` instead for anyone in the top N. +#### Player-vendor marketplace (Protocol 3.0) + +The shard-wide shop index: every player vendor's shop name, owner, location and priced inventory — +the same set the in-game **Vendor Search** gump reads, published so a site can offer the same search +from outside the game. + +An **amortized round-robin diff sweep**, not a snapshot RPC, and the distinction is load-bearing: +`rpc.rs::try_route` correlates a reply on the FIRST frame carrying a matching `reqId`, so a chunked +reply sharing one `reqId` would deliver chunk 1 to the HTTP caller and leak chunks 2..N onto the +broadcast feed. A whole-world snapshot could not fit in one frame inside the 10 s reply timeout +either. The per-account `vendor.snapshot` RPC (§5) is unaffected and still serves the player portal. + +Each tick inventories at most `Bridge.MarketSweepBatch` vendors (default 25) starting from a +persistent cursor, so **per-tick cost is bounded independently of world size**; full coverage takes +`ceil(vendors / batch) × MarketSweepSeconds`. A vendor is emitted only when its contents, prices, +shop name or location actually change. + +| kind | fields | notes | +|------|--------|-------| +| `vendor.listing` | `serial`, `shopName`, `ownerSerial`, `ownerName`, `location{}`, `count`, `total`, `truncated`, `items[]` | One vendor's complete shop — **never a delta**. The latest frame for a `serial` replaces the previous one outright. | +| `vendor.listing.remove` | `serial` | The shop is gone from the index: dismissed, expired, or its owner switched off the in-game Vendor Search flag. | + +```json +{"kind":"vendor.listing","serial":"0x40001234", + "shopName":"Darrow's Bargains","ownerSerial":"0x1A2B","ownerName":"Darrow", + "location":{"map":"Trammel","x":1421,"y":1699,"z":0, + "region":"Britain","house":"Darrow's Villa"}, + "count":2,"total":2,"truncated":false, + "items":[{"serial":"0x40012ABC","itemId":3922,"hue":0,"amount":1, + "price":25000,"name":null,"cliloc":1023721}, + {"serial":"0x40012ABD","itemId":7026,"hue":1157,"amount":3, + "price":500,"name":"a shard sigil","cliloc":1041243}], + "t":1752489280000} +``` + +**Six things consumers get wrong.** + +1. **`name` is `null` for nearly every item; `cliloc` is the real label.** Items carry a + `LabelNumber`, not a name. The plugin deliberately never calls `VendorSearch.GetItemName`, which + builds an `ObjectPropertyList`, serialises it and byte-parses the packet **per item** — a + multi-hundred-millisecond stall across a full pass. (It would not work anyway: every current + client ships its cliloc files compressed and ServUO's bundled `Ultima.StringList` cannot read + them, so the in-game gump has the same gap.) Resolve clilocs consumer-side; a non-null `name` is a + player-set literal and is strictly more specific, so **prefer it over the cliloc**. +2. **`location` is one nested object, and it may be absent entirely.** It is nested so that a + consumer gating vendor whereabouts gates one field rather than five that can drift apart — the + website's `market.location` rule removes the whole object. Treat a missing `location` as "not + published", not as an error. +3. **`truncated` means the shop holds more than the frame carries.** `count` is what was published, + `total` is what the shop actually holds, capped by `Bridge.MarketMaxListings` (default 250). A + commodity reseller with thousands of stacked resources is real and an uncapped frame for one is + measured in megabytes. Say "showing 250 of 3,104" rather than presenting a partial shop as + complete. +4. **`child: true` means the price buys the ENCLOSING CONTAINER.** ServUO prices a container as a + unit and everything inside inherits that price with no `VendorItem` of its own; `DoSearch` + surfaces the same flag. A UI that prints the container's price against each item inside it is + lying about the shard. +5. **Opted-out vendors are absent, and that is a privacy control.** `pv.VendorSearch` is the player's + own in-game toggle and the sweep honours it — hide your vendor in game and it is hidden here too. + The same goes for `Map.Internal` and a null backpack, matching `DoSearch`. Process + `vendor.listing.remove` promptly: it is how a player *revoking* that consent reaches you. +6. **Prices are inherently stale, by design.** The round-robin sweep means a shop can be a full cycle + behind. Any UI over this must say how old the data may be — the website derives it from the oldest + vendor row. + +Entries carry `ownerSerial`/`ownerName` and **never `acct` or `webId`**, the same rule `points.board` +follows. Absent entirely if the shard runs `Bridge.MarketEnabled=false` or an older plugin. Render +from `GET /market` (§6) on connect, then keep live with these events — though note that a live +firehose of whole vendor inventories is the largest stream the bridge produces, and a consumer that +only needs a browsable index (as the website does) is better served by the REST read plus the +periodic re-sweep. + --- ## 5. REST — read queries @@ -829,6 +901,35 @@ standings built over months and blanking them during a restart reads as data los one excluded by `Bridge.PointsSystems`). That is distinct from a published board nobody has scored in yet, which is **200** with an empty `top[]` — and the two are worth rendering differently. +### Player-vendor marketplace (Protocol 3.0) + +``` +GET /market?limit=200&offset=0 +→ { "vendors": [ {"kind":"vendor.listing","serial":"0x40001234", + "shopName":"Darrow's Bargains","ownerSerial":"0x1A2B","ownerName":"Darrow", + "location":{"map":"Trammel","x":1421,"y":1699,"z":0, + "region":"Britain","house":"Darrow's Villa"}, + "count":2,"total":2,"truncated":false,"items":[ ... ],"t":...}, ... ], + "total": 137, "limit": 200, "offset": 0 } +``` + +Every vendor's latest shop, exactly as `vendor.listing` published it (§4 for the frame and its six +gotchas). Served from the sidecar's projection, so it answers while the shard is down. + +**This is the only PAGED read the sidecar serves**, because it is the only board that can be a whole +world's inventory. `limit` is clamped to 1..1000 (default 200); `total` is returned so a caller knows +when to stop rather than paging until it sees a short page, which would race a concurrent sweep. +Ordering is by **serial**, not by shop name — a serial is stable while a shop name is renameable, so +a rename mid-walk cannot make a vendor skip or repeat a page. + +The route is `/market` and deliberately **not** `/vendors`: `/vendors/{account}` next door is the +per-account RPC (§5), and two routes a prefix apart meaning "this player's shops" and "every shop on +the shard" is a trap nobody wins. + +Frames are served **verbatim**, owner names and coordinates included. That is not an oversight: the +sidecar defines no audiences. Deciding who may see what is the consuming site's job — see +[`v3.md`](v3.md) §3 for how the website does it. + --- ## 7. Status codes diff --git a/link/PLAN.md b/link/PLAN.md index 709fead..b054421 100644 --- a/link/PLAN.md +++ b/link/PLAN.md @@ -284,6 +284,15 @@ Counts in `hello` are a live snapshot taken on the Core thread, not a cached val `Item.Name` is frequently `null`; the display name is `LabelNumber`, a cliloc id. **There is no `Data/Cliloc.enu` in this repo** — `BRIDGE_FINDINGS.md` §IV.4 is wrong about this. Cliloc data lives in the client install, which `DataPath` resolves to `D:\Games\Electronic Arts\Ultima Online Classic\`. Ship **both** `name` (when non-null) and `cliloc`, and resolve the number **on the website** against a cliloc map. That avoids a server-side dependency on the client directory. +**Update (3.0).** That recommendation held, and the reason it had to hold turned out to be stronger +than "avoids a dependency": **ServUO cannot resolve clilocs either.** Every current client ships its +`Cliloc.*` files compressed, and the bundled `Ultima.StringList` reads only the older plain layout — +so `VendorSearch.StringList` is null and `VendorSearch.GetItemName` returns `item.Name` on any modern +shard. The in-game Vendor Search gump has the same gap, which is why `vendor.listing` never calls it. +Pushing name resolution to the plugin was never an option. See [`v3.md`](v3.md) §8.6 and +`docs/website/CLILOCS.md` for how the site gets a table instead (the operator converts one from their +own client, once). + --- ## 8. Corrections to `BRIDGE_FINDINGS.md` @@ -327,6 +336,17 @@ leaderboards. `BridgePoints` is the widest read the bridge performs: ten of Serv keep a row for every character ever created, so it selects the top N in a single bounded pass rather than sorting, and runs on a deliberately slow 300 s interval. +Also shipped: **`vendor.listing`** ([`v3.md`](v3.md) §8), `BridgeMarket.cs`, the shard-wide +player-vendor index. It introduces the one sweep pattern the bridge did not previously have — an +**amortized round-robin**. Every other sweep walks its whole collection per tick, which is fine for +tens of houses or a fixed set of point systems and is not fine for a world of shops whose inventories +recurse into containers. `BridgeMarket` inventories at most `MarketSweepBatch` vendors per tick from +a persistent cursor, so the per-tick cost is bounded by the batch rather than by world size, and full +coverage takes `ceil(vendors / batch) x MarketSweepSeconds`. Measured at **15.4 ms** for a cold tick +of 25 vendors x 40 listings and **0.3 ms** in steady state (the per-vendor diff), on a shard of 209k +items / 43k mobiles. It is also the first stream to honour a per-player privacy toggle: ServUO's own +`PlayerVendor.VendorSearch` flag, so a shop hidden in game is hidden on the site. + ### Config keys (`Config/Bridge.cfg`) ```ini @@ -342,7 +362,8 @@ Read in `Configure()` via `Config.Get("Bridge.", default)`. Key scope is The set above is the 1.0 sample, not the current one — every later phase added keys (sweep intervals for each board, the town-crier/news caps, the admin write plane, account provisioning, and 3.0's -`RulesetEnabled` / `PublicConnectAddress` / `RulesetIncludeSchedule`, and the `Points*` block). +`RulesetEnabled` / `PublicConnectAddress` / `RulesetIncludeSchedule`, and the `Points*` and `Market*` +blocks). **`servuo-plugins/overlay/Config/Bridge.cfg` is the authoritative, commented list**; `BridgeConfig.cs` holds the defaults. diff --git a/link/v3.md b/link/v3.md index 362718d..b9d4abd 100644 --- a/link/v3.md +++ b/link/v3.md @@ -15,8 +15,8 @@ Each part is marked off here as it lands on `edge`. §9 carries the same state p | 2 | **B/1** — `world.ruleset` (§5) | ✅ **Done** | servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66) | | 3 | **C** — spawn atlas (§6) | ✅ **Done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables) + [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113) (API + pages + admin panel), docs [#67](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/67) + [#68](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/68) | | 4 | **B/2** — `points.board` (§7) | ✅ **Done** | servuo-plugins [#4](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/4), link [#18](https://gitea.whitlocktech.com/RunicGateway/link/pulls/18), website [#114](https://gitea.whitlocktech.com/RunicGateway/website/pulls/114), docs [#69](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/69) | -| 5a | **B/3 dependency** — cliloc table (§8.6) | 🟨 In review | website [#115](https://gitea.whitlocktech.com/RunicGateway/website/pulls/115), docs [#70](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/70) | -| 5b | **B/3** — `vendor.listing` (§8) | ⬜ Not started | — | +| 5a | **B/3 dependency** — cliloc table (§8.6) | ✅ **Done** | website [#115](https://gitea.whitlocktech.com/RunicGateway/website/pulls/115), docs [#70](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/70) | +| 5b | **B/3** — `vendor.listing` (§8) | 🟨 In review | servuo-plugins [#5](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/5), link [#19](https://gitea.whitlocktech.com/RunicGateway/link/pulls/19), website [#116](https://gitea.whitlocktech.com/RunicGateway/website/pulls/116), docs [#71](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/71) | | 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | ⬜ Not started | — | Order 5 split in two once §8.6's cliloc dependency turned out to be a client-format problem rather @@ -639,7 +639,7 @@ if it is renamed back. --- -## 8. Part B/3 — `vendor.listing` +## 8. Part B/3 — `vendor.listing` 🟨 In review ### 8.1 It cannot be an RPC, and this is load-bearing @@ -807,6 +807,75 @@ and text paths converge on identical content. driven by `staleAt` (the oldest `shard_vendors.updated_at`). The round-robin sweep means data is inherently up to one full cycle old, and the UI must say so. +Shipped with a second page, `routes/public/MarketVendor.jsx` at `/site/market/vendors/:serial` — +where a search result points. It is the only surface that can render the two states the result list +cannot: a `truncated` shop (*"showing 250 of 3,104 — this shop holds more than the shard +publishes"*) and a `location` an admin has gated away, which is a real answer rather than an empty +coordinate. + +### 8.8 What the build changed + +Four things the implementation settled differently from §8 as written, all of them found by building +against the live shard. + +**1. `location` is a nested object, not flat `map`/`x`/`y`/`region`.** §8.1's payload sketch had them +flat, and it would have made `market.location` — a rule Part A pre-wired — **inert**, exactly like +the `characterName` miss §7.5 records: `projectValue` matches literal JSON keys, so there is no +`location` key for the rule to match. Flat keys would have needed five rules that could drift apart. +Nesting makes one rule hide the facet, the coordinates, the region and the house together, on the +live frame and the stored read model alike, because both now spell it the same way. + +The other pre-wired rule, `market.ownerName`, checked out — it is a real key on the frame. Owner is +written as flat `ownerSerial`/`ownerName` rather than through `BridgeJson.Actor`, which would add +`acct` and `webId`; same argument `points.board` makes. `ownerSerial` was **added** to the +configurable fields alongside `ownerName`, because an admin who hides the owner's name and leaves a +serial every other board resolves back to that name has not hidden anything. + +**2. The per-vendor diff signature is the full listing set, not §8.3's `count | Σ(serial ^ price)`.** +That hash collides on the single most common change a shop makes: two items swapping prices, which +is what re-pricing looks like. The signature is built over the same buffer the frame is written +from, in the same order, so a match really does mean an identical frame. + +**3. There is no `payload` column on `shard_vendors`.** §8.5 implied the board pattern (whole frame +in JSON, columns hoisted for display). It does not apply here: the items ARE the searchable rows, so +they are normalized into `shard_vendor_items` and there is nothing left worth duplicating. The +sidecar keeps the whole blob, because outage resilience is its job and search is not. + +**4. Sweep cost is reported, and a slow tick warns.** The batch cap is a *claim* about per-tick cost, +and an operator tuning `MarketSweepBatch` was otherwise tuning blind. `[bridge status` now carries +`lastMs`/`maxMs`, and a tick over 50 ms prints a rate-limited warning naming the knob. + +Measured on the live shard (27 vendors × 40 listings, 209k items / 43k mobiles): + +| | | +|---|---| +| First tick — 25 vendors emitted cold | **15.4 ms** | +| Second tick — the remaining 2 | **3.4 ms** | +| Steady state — nothing changed | **0.3 ms** | +| Website `/market` search over 1,040 listings | 1,040 total, names resolved | +| Cliloc re-resolution pass over 1,040 rows | **50 ms** | + +The diff is what makes the steady state ~free; the batch cap is what bounds the cold case. Note the +arithmetic the warning exists for: at the default cap of 250 listings, a batch of 25 **full** shops +is 6,250 items ≈ 95 ms — over budget. Real shops hold tens, which is why 25 is the default, but a +shard of commodity resellers should lower the batch, and now it will be told to. + +Two smaller things worth not rediscovering: + +- **`BridgeJson.Escape` takes a NON-NULL string** — it dereferences `value.Length` immediately — and + `BridgeJson.Str` writes its own `,"key":` prefix, so neither serves a value inside a hand-built + object. Nearly everything this frame writes is legitimately null (an item's plain `Name` is null + for almost every item; a vendor in the street has no house), so that is the common path, not an + edge case. `BridgeMarket.Text()` is the two-line writer that was missing. +- **The ServUO console writes in the OS code page**, so an em dash in a `Console.WriteLine` renders + as `???` in the log an operator would paste into an issue. Bridge console output is ASCII. + +Search-side, one thing the site had to fix rather than inherit: `%` and `_` in a user's query are +**LIKE** metacharacters, not SQL ones, so parameterization does not neutralize them — a search for +`%` would otherwise match every listing on the shard. `shardMarket.db.js` escapes them. (The atlas's +`LIKE` searches predate this and have the same shape over a much smaller table; worth a follow-up, +not a blocker here.) + --- ## 9. Sequencing @@ -817,8 +886,8 @@ inherently up to one full cycle old, and the UI must say so. | 2 | **B/1** — `world.ruleset` (§5) | all four | new kind | ✅ Done | | 3 | **C** — spawn atlas (§6) | website, docs | none | ✅ Done | | 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | ✅ Done | -| 5a | **B/3 dependency** — cliloc table (§8.6) | website, docs | none | 🟨 In review | -| 5b | **B/3** — `vendor.listing` (§8) | all four | new kinds | ⬜ | +| 5a | **B/3 dependency** — cliloc table (§8.6) | website, docs | none | ✅ Done | +| 5b | **B/3** — `vendor.listing` (§8) | all four | new kinds | 🟨 In review | | 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | ⬜ | --- diff --git a/website/BACKEND_DESIGN.md b/website/BACKEND_DESIGN.md index 3172b92..409569c 100644 --- a/website/BACKEND_DESIGN.md +++ b/website/BACKEND_DESIGN.md @@ -407,6 +407,50 @@ Two values carry non-obvious meanings, both set by the plugin and both documente cliloc rather than a literal. Listing therefore orders by `COALESCE(name, system)`, so boards awaiting cliloc resolution sort by their own key rather than clumping together under NULL. +### shard_vendors / shard_vendor_items — the player-vendor marketplace (Protocol 3.0) + +The shard-wide shop index, fed by `vendor.listing` / `vendor.listing.remove`. One row per player +vendor and one per priced listing. Full operator detail in [`MARKETPLACE.md`](MARKETPLACE.md); the +design is `docs/link/v3.md` §8. + +| Table | Shape | +|---|---| +| `shard_vendors` | `serial` (PK), `shop_name`, `owner_serial`, `owner_name`, `map`/`x`/`y`/`z`, `region`, `house`, `item_count`, `item_total`, `truncated`, `t`, `updated_at`. Indexes on owner, map, region and `updated_at`. | +| `shard_vendor_items` | `id` (PK), `vendor_serial`, `serial`, `item_id`, `hue`, `amount`, `price`, `name`, `cliloc`, `display_name`, `child`. Indexes on `vendor_serial`, `price`, `item_id`, `display_name`, and `(display_name, price)`. | + +**Ingest is per-vendor and authoritative**: the frame is the whole shop, so ingest is +delete-then-insert of that vendor's listings inside one transaction. All-or-nothing matters +specifically because the two writes are "the shop" and "what is in it" — a failure between them +leaves a shop advertising an inventory it no longer has, which is visibly wrong and indistinguishable +from a genuinely empty shop. No foreign keys, consistent with every other `shard_*` table. + +**There is deliberately no `payload` column**, unlike `shard_points_boards` directly above. The +board's top-N is a fixed-size list read whole, so it lives in JSON; here the items *are* the +searchable rows, so they are normalized and nothing is left worth duplicating. The sidecar keeps the +whole blob — outage resilience is its job, search is ours. + +Market state, not events: neither kind is in `LOGGED_KINDS`, and this is the strongest case of the +three v3 kinds. One frame carries up to 250 listings and the sweep re-emits a shop on any price +change, so logging would turn `shard_events` into a price history nobody reads. + +Two columns carry non-obvious meanings: + +- **`item_count` vs `item_total`.** `item_count` is what the frame published; `item_total` is what + the shop actually holds. They differ when `truncated` — the shard caps listings per frame + (`Bridge.MarketMaxListings`, 250 by default), and a commodity reseller with thousands of stacks + genuinely exceeds it. Any UI must show both or it presents a partial shop as complete. +- **`display_name` is denormalized at ingest**, resolved from the item's literal `name` (preferred — + a player set it, so it is more specific) else its `cliloc` against `shard_clilocs`. Resolving at + query time would put the cliloc table on the hot path and make search-by-name impossible. Because + the shard's diff sweep will not re-send an unchanged shop just because the site learned what its + items are called, **a cliloc import triggers a bulk re-resolution** of this column (after a boot + import and after an admin import; ~50 ms per thousand rows, never throws). + +`updated_at` is written explicitly on every upsert rather than left to `ON UPDATE CURRENT_TIMESTAMP`, +which MariaDB does not fire when every column is written back unchanged. A shop re-published +identically is still *freshly confirmed*, and without this the staleness banner would age a perfectly +current shop forever. + ### shard_feature_visibility — per-feature audience config (Protocol 3.0) One row per shard feature: `feature` (PK), `enabled`, `audience` (a rung on the ladder in §6.5), @@ -745,6 +789,9 @@ from the per-route **siteMode** middleware (§5), never from an auth gate. | GET | `/shard/ruleset` | the shard's own published ruleset (Protocol 3.0 `world.ruleset`): expansion, which optional systems are on, skill/stat caps, account and house limits, champion scroll rules, the save/restart schedule. Served from `shard_ruleset`, so it renders while the shard is down; live via `world.ruleset` on `/shard/stream`. Behind `requireFeature('ruleset')`. **`null`** means the shard has never published one — a real answer, distinct from a published ruleset. `caps.skill` / `caps.totalSkill` are in **tenths** (1000 = 100.0). | | GET | `/shard/points` | every points/loyalty leaderboard the shard publishes (Protocol 3.0 `points.board`) — Queen's Loyalty, Void Pool, the nine city loyalties, Clean Up Britannia, … Served from `shard_points_boards`, so it renders while the shard is down; live via `points.board` on `/shard/stream`. Behind `requireFeature('leaderboards')`, ordered by display name. **`maxPoints: 0` means uncapped** (the common case), and `nameString` is usually `null` with `nameNumber` holding a cliloc — resolve client-side or humanise the `system` key. | | GET | `/shard/points/:system` | one board by the shard's `PointsType` name (e.g. `QueensLoyalty`); `:system` must match `/^[A-Za-z][A-Za-z0-9_]{0,47}$/` or **400** before any query runs. **404** = the shard has never published that system, which is distinct from a published board nobody has scored in yet (**200** with an empty `top`). | +| GET | `/shard/market?q=&minPrice=&maxPrice=&itemId=&map=®ion=&sort=&limit=&offset=` | search the player-vendor marketplace (Protocol 3.0 `vendor.listing`). Returns **listings**, not vendors — "who sells X and for how much" is the question, and a vendor-shaped result would make every caller flatten the shops back out. Served from `shard_vendors` + `shard_vendor_items`, so it renders while the shard is down. Behind `requireFeature('market')` **and rate-limited** — the first genuinely expensive public read on the site (a `LIKE` scan plus a `COUNT` over what is typically the largest `shard_*` table, reachable with no session). `sort ∈ {price_asc, price_desc, recent}`. `q` matches the resolved display name **or** the item's literal name, with `%`/`_` escaped: they are `LIKE` metacharacters, not SQL ones, so parameterization alone would let `?q=%` match every listing on the shard. Every response repeats `staleAt` (the oldest vendor row) because the shard sweeps round-robin — a banner that ages with the results it labels, not one fetched once. | +| GET | `/shard/market/meta` | index size, staleness (`staleAt`/`freshAt`) and which facets and regions actually hold vendors, so a client builds its filters without running a search it will discard. | +| GET | `/shard/market/vendors/:serial` | one shop and its listings; `:serial` must match `/^0x[0-9A-Fa-f]{1,16}$/` or **400** before any query runs. **404** = a serial the index has never seen, which also covers a vendor since dismissed or hidden — to an anonymous caller those are the same answer, and distinguishing them would leak that a hidden vendor exists. `truncated` (with `total` exceeding `count`) means the shop holds more than the shard publishes per frame. | | GET | `/shard/features` | the shard features **this caller** may reach plus the audience rung they resolved to (§6.5), so a client hides nav it can't follow. Reports only what the caller can see — the list itself never discloses a gated feature. Consumed by the SPA header and (pending) the Android nav. | | GET | `/atlas/creatures?q=&facet=&limit=&offset=` | the bestiary, most numerous first, with an unpaginated `total`. Static content parsed from the shard's ServUO tree — **not** sidecar-backed, which is why the atlas sits outside `/shard`, and unlike `/shard/*` it **is** site-mode gated. Behind `requireFeature('atlas')`. `?facet=` is matched exactly and never validated against a list (no facet name exists in the code); the filter is an `EXISTS` over the points rather than a JSON path or `JSON_SEARCH` built from caller input, whose `%`/`_` wildcards would make `?facet=%` match everything. | | GET | `/atlas/creatures/:slug` | one creature: `places` (the point-in-rect aggregate — "lizardman → Shrines, Isamu-Jima, Yew"), `spawners` (the bounded raw list, with `spawnersTruncated`), `alsoHere`. **`points` is a COUNT and `spawners` is the LIST** — named apart so one key never means a number on one route and an array on another. `minDelay`/`maxDelay` are in **seconds**, normalised at parse time from the source's per-record minutes-or-seconds. 404 = no such creature in this atlas. | @@ -835,7 +882,11 @@ who"; `activity_log` provides the history feed. - **Cookie**: `httpOnly`, `sameSite=Lax`, `path=/`, and **`secure` decided per-request** (`COOKIE_SECURE=auto` → `secure: req.secure`). - **Trusted-device MFA.** A second, separate httpOnly cookie (`rg_trust`, default 30d) — opaque, sha256-hashed server-side in `trusted_devices` — lets a browser/app **skip the TOTP step** (never the password) on future logins. It is a server-side, per-row-revocable record (never a JWT claim), so the stateless session JWT is unchanged and trust stays revocable. It only ever gates the **second factor**; it deliberately outlives logout, and is cleared on untrust / password change / password reset / TOTP disable. **Recovery codes** (bcrypt, single-use) are the 2FA-lockout fallback. All admin trusted-device/MFA actions and the self actions (`auth.login.trusted_device`, `account.trusted_device.*`, `account.recovery_code*`, `admin.trusted_device.*`, `admin.user.totp.reset`) are audit-logged. See `docs/website/TRUSTED_DEVICES_MFA.md`. This is the key to dual access: the cookie is `Secure` when reached through Pangolin (HTTPS, `X-Forwarded-Proto: https`) but **not** `Secure` when reached directly over the LAN IP on plain HTTP — so login works in both. `COOKIE_SECURE=true|false` can force it. Requires `trust proxy` (below). `localhost:5173` (Vite) and `localhost:3000` are same-site, so the cookie flows in dev too. - **bcrypt** hashing (cost 10+); plaintext passwords never stored, logged, or returned. -- **Rate limiting** (`express-rate-limit`) on `/auth/login` and `/public/contact`. +- **Rate limiting** (`express-rate-limit`) on `/auth/login`, `/public/contact`, and — the only limited + *read* — `/public/shard/market` and `/public/shard/market/vendors/:serial` (60/min/IP). Every other + public read is an indexed lookup of bounded size; the marketplace search is a `LIKE` scan plus a + `COUNT` over the largest `shard_*` table, anonymous by default, so it is the one public GET that is + worth money to serve. - **Validation** (`express-validator`) on all writes; centralized error handler. - **helmet** with a Content-Security-Policy tuned for the built React SPA. The policies now live in **`server/src/config/csp.js`** (`app.js` only wires them up): diff --git a/website/MARKETPLACE.md b/website/MARKETPLACE.md new file mode 100644 index 0000000..5212b9f --- /dev/null +++ b/website/MARKETPLACE.md @@ -0,0 +1,167 @@ +# Marketplace — the player-vendor index + +**Status:** On `edge` — servuo-plugins [#5](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/5), link [#19](https://gitea.whitlocktech.com/RunicGateway/link/pulls/19), website [#116](https://gitea.whitlocktech.com/RunicGateway/website/pulls/116). +**Design:** [`docs/link/v3.md` §8](../link/v3.md) — Protocol 3.0 Part B/3. +**Depends on:** [`CLILOCS.md`](CLILOCS.md) — without a cliloc table, listings render as item ids. + +The marketplace is a searchable index of every player vendor on the shard: what +each shop is selling, for how much, and where it is standing. It is the same set +the in-game **Vendor Search** gump reads, offered from outside the game — so a +player can find the vanquishing kryss they want before logging in, and someone +who does not play at all can see that the economy exists. + +Page: `/site/market`, plus `/site/market/vendors/:serial` for one shop. + +## Three things the pages must say out loud + +Everything below follows from how the data is gathered, and each has a visible +consequence the UI is required to surface. + +**1. The prices are not live.** The shard sweeps vendors **round-robin** — at +most `Bridge.MarketSweepBatch` shops per tick — so a given shop can be a full +cycle behind. The page carries a *"prices last refreshed N minutes ago"* banner +driven by the **oldest** vendor row, not the newest: the one stale shop is the +one that wastes somebody's trip. + +**2. A shop can be truncated.** `Bridge.MarketMaxListings` (250 by default) caps +how many listings one frame carries. A commodity reseller with thousands of +stacked resources is a real thing, and an uncapped frame for one is measured in +megabytes. Over the cap the shop reports `truncated`, and the vendor page says +*"showing 250 of 3,104 — this shop holds more than the shard publishes"* rather +than presenting a partial shop as complete. + +**3. An item may have no name.** Items on the wire carry a cliloc id, not a name. +On a shard whose operator has not converted a cliloc table +([`CLILOCS.md`](CLILOCS.md)) the honest render is the item id — never an invented +label, which would be indistinguishable from a real one. + +## Privacy: the player's own toggle wins + +Only vendors whose owner left the in-game **Vendor Search** flag ON are ever sent +to the site. A player who hides their shop in game is hidden here too, and no +admin setting overrides that. When they hide one that was already indexed, the +shard emits `vendor.listing.remove` and the row is deleted — so revoking consent +takes effect, it does not merely stop refreshing. + +Shop name, owner character name and location default to **Everyone**, because the +stock Vendor Search gump already shows exactly that set to any player in game. +They remain admin-configurable; see [`SHARD_VISIBILITY.md`](SHARD_VISIBILITY.md). +Account names and website user ids never cross the wire at all. + +## How it is put together + +``` +ServUO uo-link sidecar website +────── ─────────────── ─────── +BridgeMarket.cs vendors table shard_vendors + round-robin sweep ──────► (whole frame blob) ──────► shard_vendor_items + per-vendor diff GET /market (paged) + display_name + vendor.listing resolved at ingest + vendor.listing.remove +``` + +**The shard side** walks at most `MarketSweepBatch` vendors per tick from a +persistent cursor, diffs each against what it last published, and emits a whole +frame for any shop that moved. Per-tick cost is therefore bounded by the batch, +not by how many vendors the world holds — full coverage takes +`ceil(vendors / batch) × MarketSweepSeconds`. + +**The sidecar** stores each frame whole and serves `GET /market`, its only paged +read. It normalizes nothing and defines no audiences: it is a dumb forwarder, and +search is the website's job. + +**The website** splits each frame into a vendor row and its listings, replacing +that vendor's whole listing set inside one transaction (the frame is +authoritative for that vendor, never a delta). Item names are resolved against +the cliloc table **on the way in** and stored denormalized, which is what makes +search-by-name possible and keeps the cliloc table off the hot path. + +## Operating it + +Everything is in `Config/Bridge.cfg` on the shard. There is nothing to configure +on the website. + +| Setting | Default | What it does | +|---|---|---| +| `MarketEnabled` | `true` | Master switch. Off publishes nothing; the page shows an empty index. | +| `MarketSweepSeconds` | `60` | Tick interval. | +| `MarketSweepBatch` | `25` | Vendors inventoried per tick. Clamped 1..500. | +| `MarketMaxListings` | `250` | Per-shop listing cap, after which `truncated`. Clamped 1..5000. | + +**Faster coverage vs. per-tick cost.** Lowering `MarketSweepSeconds` or raising +`MarketSweepBatch` both refresh the index sooner and both cost more per tick. +The expensive part is the item walk, which recurses into every container a vendor +is selling — so a shard of big shops should raise the interval rather than the +batch. + +`[bridge status` reports the sweep, including `lastMs` and `maxMs`: + +``` +market(enabled=True sweeps=42 scanned=108 emitted=27 removed=0 skipped=0 + truncated=0 tracked=27 vendors=27 cursor=2 batch=25 lastMs=0.31 maxMs=15.40) +``` + +A tick over **50 ms** prints a rate-limited warning naming the knob: + +``` +[Bridge] market sweep took 82.4 ms (budget 50 ms) - lower Bridge.MarketSweepBatch (now 25) if this persists +``` + +Measured on a shard with 27 vendors × 40 listings (209k items, 43k mobiles): +**15.4 ms** for the first cold tick of 25 vendors, **0.3 ms** in steady state — +the diff is what makes an unchanged world nearly free. Note the arithmetic: 25 +*full* shops at the 250-listing cap is 6,250 items ≈ 95 ms, over budget. Real +shops hold tens, which is why 25 is the default and why the warning exists. + +`[bridge sweepnow` runs one tick immediately; `[bridge reload` re-reads the +settings above without a restart. + +## Names arriving late + +Item names come from the cliloc table, and the market sweep will **not** re-send +an unchanged shop just because the site learned what its items are called. So a +cliloc import triggers a bulk re-resolution of every stored listing — otherwise +an operator who configures clilocs after the first sweep would see item ids until +every shop happened to change on its own. It runs after a boot import and after +an admin import, takes ~50 ms per thousand listings, and never throws: a failure +leaves names exactly as they were. + +## API + +All under `/api/v1/public/shard`, gated by the `market` feature and +**rate-limited** — these are the first genuinely expensive public reads on the +site (a `LIKE` scan plus a `COUNT` over what is typically the largest `shard_*` +table, reachable with no session). + +| Route | What | +|---|---| +| `GET /market` | Search. Returns **listings**, not vendors — "who sells X and for how much" is the question. `?q=&minPrice=&maxPrice=&itemId=&map=®ion=&sort=&limit=&offset=`, `sort ∈ {price_asc, price_desc, recent}`. | +| `GET /market/meta` | Index size, staleness, and which facets and regions actually hold vendors — so a client builds its filters without running a search it will discard. | +| `GET /market/vendors/:serial` | One shop and its listings. **404** for a serial the index has never seen, which also covers a vendor since dismissed or hidden — to an anonymous caller those are the same answer. | + +`q` matches the resolved display name **or** the item's own literal name, because +an item with a player-set name (most of what is worth searching for on a +player-run shard) may carry a generic cliloc. `%` and `_` in a query are escaped: +they are `LIKE` metacharacters, not SQL ones, so parameterization alone would let +a search for `%` match every listing on the shard. + +Full schemas are in the OpenAPI spec (`ShardMarketPage`, `ShardMarketVendor`, +`ShardMarketMeta`, `ShardMarketListing`, `ShardMarketLocation`). + +## Tables + +`shard_vendors` (one row per shop) and `shard_vendor_items` (one row per priced +listing). Both are ingest-owned; nothing else writes to them. No foreign keys, +in keeping with every other `shard_*` table — the ingest transaction is what +keeps them consistent, and an FK would turn a malformed frame into a failed write +rather than a dropped row. + +There is deliberately **no `payload` column** on `shard_vendors`, unlike the +points board next door. The board's top-N is a fixed-size list read whole, so it +lives in JSON; here the items *are* the searchable rows, so they are normalized +and there is nothing left worth duplicating. The sidecar keeps the whole blob, +because outage resilience is its job. + +`shard_vendor_items.display_name` is denormalized and indexed (alone, and +composite with `price` for "cheapest matching X"). See "Names arriving late" +above for how it is kept current. diff --git a/website/SHARD_VISIBILITY.md b/website/SHARD_VISIBILITY.md index c8e301f..31254f9 100644 --- a/website/SHARD_VISIBILITY.md +++ b/website/SHARD_VISIBILITY.md @@ -60,12 +60,32 @@ board while holding back one column. See the table in §3. | **Shard rules** | Skill/stat caps, house limits, vet rewards, the ruleset | Everyone | Connect address → Everyone | | **Spawn atlas** | Bestiary and spawn locations (static content) | Everyone | — | | **Leaderboards** | Point and loyalty standings | Everyone | Character names → Everyone | -| **Marketplace** | The shard-wide player-vendor index | Everyone, **live updates off** | Vendor owner name → Everyone · Location → Everyone | +| **Marketplace** | The shard-wide player-vendor index | Everyone, **live updates off** | Vendor owner name → Everyone · Vendor owner character id → Everyone · In-game location → Everyone | **Why the marketplace ships with live updates off.** A live feed of every vendor's full inventory would be the single largest thing the site sends. No page needs it — the marketplace is a search over stored data with a “prices last refreshed N minutes ago” stamp. Turn it on only if you want it. +**Why the marketplace's fields default to Everyone.** A vendor's shop name, its owner's character +name and where it is standing are *already* visible to every player in game: the stock Vendor Search +gump surfaces exactly that set to anyone who opens it. Publishing them on the site is not a new +disclosure. They stay configurable because a shard may still prefer to keep its economy behind a +login — and because "already public in game" is a judgement about your shard, not ours. + +**Location is one setting covering four things.** Hiding it removes the facet, the coordinates, the +region *and* the house name together. That is deliberate: those are four ways of saying the same +thing, and a setting that hid the coordinates while publishing the house name would not have hidden +anything. + +**Hiding the owner name also hides the owner character id.** They are separate settings so you can +be explicit, but leaving the id published while hiding the name achieves nothing — the leaderboards +and guild boards resolve that same id back to a character name. Set both. + +**What hiding a vendor cannot do.** Only vendors whose owner left the in-game *Vendor Search* flag ON +are ever sent to the site, so a player who hides their shop in game is hidden here too — and no +setting on this page can override that. It works the other way as well: these settings control who +sees the index, not whether players can find each other's shops in game. + **Why house owner/price default to Staff.** The public Houses page has always been a "where are the falling houses" board — location only. Owner and price are the staff view. That split is preserved. -- 2.49.1