docs(link): the player-vendor marketplace (Protocol 3.0 §8) #71
@@ -20,6 +20,9 @@ ci/ cross-cutting CI/quality notes
|
||||
| [HERO_EDITOR.md](website/HERO_EDITOR.md) | Hero canvas editor feature spec |
|
||||
| [WIKI_UPGRADE.md](website/WIKI_UPGRADE.md) | Wiki subsystem upgrade notes |
|
||||
| [SHARD_VISIBILITY.md](website/SHARD_VISIBILITY.md) | Who sees which shard data — the admin-configurable audience framework |
|
||||
| [SPAWN_ATLAS.md](website/SPAWN_ATLAS.md) | The bestiary / spawn atlas: what the shard contains, parsed from its own ServUO tree |
|
||||
| [CLILOCS.md](website/CLILOCS.md) | UO's id → name table: converting one from your client so items have names |
|
||||
| [MARKETPLACE.md](website/MARKETPLACE.md) | The player-vendor index: how it is gathered, what it costs, how to tune it |
|
||||
| [website-README.md](website/website-README.md) | Snapshot of the website repo's README (setup/run reference) |
|
||||
| [PROJECT_TREE.md](website/PROJECT_TREE.md) | Auto-generated snapshot of the repo's tracked file layout |
|
||||
|
||||
|
||||
@@ -449,6 +449,78 @@ would carry ~25 zeroes. `maxPoints` follows the same `0 == uncapped` rule as the
|
||||
a points lookup stops at the character's own row, but a rank must count every row that beats them, in
|
||||
every system, on every profile build. Derive rank from `points.board` instead for anyone in the top N.
|
||||
|
||||
#### Player-vendor marketplace (Protocol 3.0)
|
||||
|
||||
The shard-wide shop index: every player vendor's shop name, owner, location and priced inventory —
|
||||
the same set the in-game **Vendor Search** gump reads, published so a site can offer the same search
|
||||
from outside the game.
|
||||
|
||||
An **amortized round-robin diff sweep**, not a snapshot RPC, and the distinction is load-bearing:
|
||||
`rpc.rs::try_route` correlates a reply on the FIRST frame carrying a matching `reqId`, so a chunked
|
||||
reply sharing one `reqId` would deliver chunk 1 to the HTTP caller and leak chunks 2..N onto the
|
||||
broadcast feed. A whole-world snapshot could not fit in one frame inside the 10 s reply timeout
|
||||
either. The per-account `vendor.snapshot` RPC (§5) is unaffected and still serves the player portal.
|
||||
|
||||
Each tick inventories at most `Bridge.MarketSweepBatch` vendors (default 25) starting from a
|
||||
persistent cursor, so **per-tick cost is bounded independently of world size**; full coverage takes
|
||||
`ceil(vendors / batch) × MarketSweepSeconds`. A vendor is emitted only when its contents, prices,
|
||||
shop name or location actually change.
|
||||
|
||||
| kind | fields | notes |
|
||||
|------|--------|-------|
|
||||
| `vendor.listing` | `serial`, `shopName`, `ownerSerial`, `ownerName`, `location{}`, `count`, `total`, `truncated`, `items[]` | One vendor's complete shop — **never a delta**. The latest frame for a `serial` replaces the previous one outright. |
|
||||
| `vendor.listing.remove` | `serial` | The shop is gone from the index: dismissed, expired, or its owner switched off the in-game Vendor Search flag. |
|
||||
|
||||
```json
|
||||
{"kind":"vendor.listing","serial":"0x40001234",
|
||||
"shopName":"Darrow's Bargains","ownerSerial":"0x1A2B","ownerName":"Darrow",
|
||||
"location":{"map":"Trammel","x":1421,"y":1699,"z":0,
|
||||
"region":"Britain","house":"Darrow's Villa"},
|
||||
"count":2,"total":2,"truncated":false,
|
||||
"items":[{"serial":"0x40012ABC","itemId":3922,"hue":0,"amount":1,
|
||||
"price":25000,"name":null,"cliloc":1023721},
|
||||
{"serial":"0x40012ABD","itemId":7026,"hue":1157,"amount":3,
|
||||
"price":500,"name":"a shard sigil","cliloc":1041243}],
|
||||
"t":1752489280000}
|
||||
```
|
||||
|
||||
**Six things consumers get wrong.**
|
||||
|
||||
1. **`name` is `null` for nearly every item; `cliloc` is the real label.** Items carry a
|
||||
`LabelNumber`, not a name. The plugin deliberately never calls `VendorSearch.GetItemName`, which
|
||||
builds an `ObjectPropertyList`, serialises it and byte-parses the packet **per item** — a
|
||||
multi-hundred-millisecond stall across a full pass. (It would not work anyway: every current
|
||||
client ships its cliloc files compressed and ServUO's bundled `Ultima.StringList` cannot read
|
||||
them, so the in-game gump has the same gap.) Resolve clilocs consumer-side; a non-null `name` is a
|
||||
player-set literal and is strictly more specific, so **prefer it over the cliloc**.
|
||||
2. **`location` is one nested object, and it may be absent entirely.** It is nested so that a
|
||||
consumer gating vendor whereabouts gates one field rather than five that can drift apart — the
|
||||
website's `market.location` rule removes the whole object. Treat a missing `location` as "not
|
||||
published", not as an error.
|
||||
3. **`truncated` means the shop holds more than the frame carries.** `count` is what was published,
|
||||
`total` is what the shop actually holds, capped by `Bridge.MarketMaxListings` (default 250). A
|
||||
commodity reseller with thousands of stacked resources is real and an uncapped frame for one is
|
||||
measured in megabytes. Say "showing 250 of 3,104" rather than presenting a partial shop as
|
||||
complete.
|
||||
4. **`child: true` means the price buys the ENCLOSING CONTAINER.** ServUO prices a container as a
|
||||
unit and everything inside inherits that price with no `VendorItem` of its own; `DoSearch`
|
||||
surfaces the same flag. A UI that prints the container's price against each item inside it is
|
||||
lying about the shard.
|
||||
5. **Opted-out vendors are absent, and that is a privacy control.** `pv.VendorSearch` is the player's
|
||||
own in-game toggle and the sweep honours it — hide your vendor in game and it is hidden here too.
|
||||
The same goes for `Map.Internal` and a null backpack, matching `DoSearch`. Process
|
||||
`vendor.listing.remove` promptly: it is how a player *revoking* that consent reaches you.
|
||||
6. **Prices are inherently stale, by design.** The round-robin sweep means a shop can be a full cycle
|
||||
behind. Any UI over this must say how old the data may be — the website derives it from the oldest
|
||||
vendor row.
|
||||
|
||||
Entries carry `ownerSerial`/`ownerName` and **never `acct` or `webId`**, the same rule `points.board`
|
||||
follows. Absent entirely if the shard runs `Bridge.MarketEnabled=false` or an older plugin. Render
|
||||
from `GET /market` (§6) on connect, then keep live with these events — though note that a live
|
||||
firehose of whole vendor inventories is the largest stream the bridge produces, and a consumer that
|
||||
only needs a browsable index (as the website does) is better served by the REST read plus the
|
||||
periodic re-sweep.
|
||||
|
||||
---
|
||||
|
||||
## 5. REST — read queries
|
||||
@@ -829,6 +901,35 @@ standings built over months and blanking them during a restart reads as data los
|
||||
one excluded by `Bridge.PointsSystems`). That is distinct from a published board nobody has scored in
|
||||
yet, which is **200** with an empty `top[]` — and the two are worth rendering differently.
|
||||
|
||||
### Player-vendor marketplace (Protocol 3.0)
|
||||
|
||||
```
|
||||
GET /market?limit=200&offset=0
|
||||
→ { "vendors": [ {"kind":"vendor.listing","serial":"0x40001234",
|
||||
"shopName":"Darrow's Bargains","ownerSerial":"0x1A2B","ownerName":"Darrow",
|
||||
"location":{"map":"Trammel","x":1421,"y":1699,"z":0,
|
||||
"region":"Britain","house":"Darrow's Villa"},
|
||||
"count":2,"total":2,"truncated":false,"items":[ ... ],"t":...}, ... ],
|
||||
"total": 137, "limit": 200, "offset": 0 }
|
||||
```
|
||||
|
||||
Every vendor's latest shop, exactly as `vendor.listing` published it (§4 for the frame and its six
|
||||
gotchas). Served from the sidecar's projection, so it answers while the shard is down.
|
||||
|
||||
**This is the only PAGED read the sidecar serves**, because it is the only board that can be a whole
|
||||
world's inventory. `limit` is clamped to 1..1000 (default 200); `total` is returned so a caller knows
|
||||
when to stop rather than paging until it sees a short page, which would race a concurrent sweep.
|
||||
Ordering is by **serial**, not by shop name — a serial is stable while a shop name is renameable, so
|
||||
a rename mid-walk cannot make a vendor skip or repeat a page.
|
||||
|
||||
The route is `/market` and deliberately **not** `/vendors`: `/vendors/{account}` next door is the
|
||||
per-account RPC (§5), and two routes a prefix apart meaning "this player's shops" and "every shop on
|
||||
the shard" is a trap nobody wins.
|
||||
|
||||
Frames are served **verbatim**, owner names and coordinates included. That is not an oversight: the
|
||||
sidecar defines no audiences. Deciding who may see what is the consuming site's job — see
|
||||
[`v3.md`](v3.md) §3 for how the website does it.
|
||||
|
||||
---
|
||||
|
||||
## 7. Status codes
|
||||
|
||||
23
link/PLAN.md
23
link/PLAN.md
@@ -284,6 +284,15 @@ Counts in `hello` are a live snapshot taken on the Core thread, not a cached val
|
||||
|
||||
`Item.Name` is frequently `null`; the display name is `LabelNumber`, a cliloc id. **There is no `Data/Cliloc.enu` in this repo** — `BRIDGE_FINDINGS.md` §IV.4 is wrong about this. Cliloc data lives in the client install, which `DataPath` resolves to `D:\Games\Electronic Arts\Ultima Online Classic\`. Ship **both** `name` (when non-null) and `cliloc`, and resolve the number **on the website** against a cliloc map. That avoids a server-side dependency on the client directory.
|
||||
|
||||
**Update (3.0).** That recommendation held, and the reason it had to hold turned out to be stronger
|
||||
than "avoids a dependency": **ServUO cannot resolve clilocs either.** Every current client ships its
|
||||
`Cliloc.*` files compressed, and the bundled `Ultima.StringList` reads only the older plain layout —
|
||||
so `VendorSearch.StringList` is null and `VendorSearch.GetItemName` returns `item.Name` on any modern
|
||||
shard. The in-game Vendor Search gump has the same gap, which is why `vendor.listing` never calls it.
|
||||
Pushing name resolution to the plugin was never an option. See [`v3.md`](v3.md) §8.6 and
|
||||
`docs/website/CLILOCS.md` for how the site gets a table instead (the operator converts one from their
|
||||
own client, once).
|
||||
|
||||
---
|
||||
|
||||
## 8. Corrections to `BRIDGE_FINDINGS.md`
|
||||
@@ -327,6 +336,17 @@ leaderboards. `BridgePoints` is the widest read the bridge performs: ten of Serv
|
||||
keep a row for every character ever created, so it selects the top N in a single bounded pass rather
|
||||
than sorting, and runs on a deliberately slow 300 s interval.
|
||||
|
||||
Also shipped: **`vendor.listing`** ([`v3.md`](v3.md) §8), `BridgeMarket.cs`, the shard-wide
|
||||
player-vendor index. It introduces the one sweep pattern the bridge did not previously have — an
|
||||
**amortized round-robin**. Every other sweep walks its whole collection per tick, which is fine for
|
||||
tens of houses or a fixed set of point systems and is not fine for a world of shops whose inventories
|
||||
recurse into containers. `BridgeMarket` inventories at most `MarketSweepBatch` vendors per tick from
|
||||
a persistent cursor, so the per-tick cost is bounded by the batch rather than by world size, and full
|
||||
coverage takes `ceil(vendors / batch) x MarketSweepSeconds`. Measured at **15.4 ms** for a cold tick
|
||||
of 25 vendors x 40 listings and **0.3 ms** in steady state (the per-vendor diff), on a shard of 209k
|
||||
items / 43k mobiles. It is also the first stream to honour a per-player privacy toggle: ServUO's own
|
||||
`PlayerVendor.VendorSearch` flag, so a shop hidden in game is hidden on the site.
|
||||
|
||||
### Config keys (`Config/Bridge.cfg`)
|
||||
|
||||
```ini
|
||||
@@ -342,7 +362,8 @@ Read in `Configure()` via `Config.Get<T>("Bridge.<Key>", default)`. Key scope is
|
||||
|
||||
The set above is the 1.0 sample, not the current one — every later phase added keys (sweep intervals
|
||||
for each board, the town-crier/news caps, the admin write plane, account provisioning, and 3.0's
|
||||
`RulesetEnabled` / `PublicConnectAddress` / `RulesetIncludeSchedule`, and the `Points*` block).
|
||||
`RulesetEnabled` / `PublicConnectAddress` / `RulesetIncludeSchedule`, and the `Points*` and `Market*`
|
||||
blocks).
|
||||
**`servuo-plugins/overlay/Config/Bridge.cfg`
|
||||
is the authoritative, commented list**; `BridgeConfig.cs` holds the defaults.
|
||||
|
||||
|
||||
79
link/v3.md
79
link/v3.md
@@ -15,8 +15,8 @@ Each part is marked off here as it lands on `edge`. §9 carries the same state p
|
||||
| 2 | **B/1** — `world.ruleset` (§5) | ✅ **Done** | servuo-plugins [#3](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/3), link [#17](https://gitea.whitlocktech.com/RunicGateway/link/pulls/17), website [#111](https://gitea.whitlocktech.com/RunicGateway/website/pulls/111), docs [#66](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/66) |
|
||||
| 3 | **C** — spawn atlas (§6) | ✅ **Done** | website [#112](https://gitea.whitlocktech.com/RunicGateway/website/pulls/112) (parsers + CLI + tables) + [#113](https://gitea.whitlocktech.com/RunicGateway/website/pulls/113) (API + pages + admin panel), docs [#67](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/67) + [#68](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/68) |
|
||||
| 4 | **B/2** — `points.board` (§7) | ✅ **Done** | servuo-plugins [#4](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/4), link [#18](https://gitea.whitlocktech.com/RunicGateway/link/pulls/18), website [#114](https://gitea.whitlocktech.com/RunicGateway/website/pulls/114), docs [#69](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/69) |
|
||||
| 5a | **B/3 dependency** — cliloc table (§8.6) | 🟨 In review | website [#115](https://gitea.whitlocktech.com/RunicGateway/website/pulls/115), docs [#70](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/70) |
|
||||
| 5b | **B/3** — `vendor.listing` (§8) | ⬜ Not started | — |
|
||||
| 5a | **B/3 dependency** — cliloc table (§8.6) | ✅ **Done** | website [#115](https://gitea.whitlocktech.com/RunicGateway/website/pulls/115), docs [#70](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/70) |
|
||||
| 5b | **B/3** — `vendor.listing` (§8) | 🟨 In review | servuo-plugins [#5](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/5), link [#19](https://gitea.whitlocktech.com/RunicGateway/link/pulls/19), website [#116](https://gitea.whitlocktech.com/RunicGateway/website/pulls/116), docs [#71](https://gitea.whitlocktech.com/RunicGateway/docs/pulls/71) |
|
||||
| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3 (§4) | ⬜ Not started | — |
|
||||
|
||||
Order 5 split in two once §8.6's cliloc dependency turned out to be a client-format problem rather
|
||||
@@ -639,7 +639,7 @@ if it is renamed back.
|
||||
|
||||
---
|
||||
|
||||
## 8. Part B/3 — `vendor.listing`
|
||||
## 8. Part B/3 — `vendor.listing` 🟨 In review
|
||||
|
||||
### 8.1 It cannot be an RPC, and this is load-bearing
|
||||
|
||||
@@ -807,6 +807,75 @@ and text paths converge on identical content.
|
||||
driven by `staleAt` (the oldest `shard_vendors.updated_at`). The round-robin sweep means data is
|
||||
inherently up to one full cycle old, and the UI must say so.
|
||||
|
||||
Shipped with a second page, `routes/public/MarketVendor.jsx` at `/site/market/vendors/:serial` —
|
||||
where a search result points. It is the only surface that can render the two states the result list
|
||||
cannot: a `truncated` shop (*"showing 250 of 3,104 — this shop holds more than the shard
|
||||
publishes"*) and a `location` an admin has gated away, which is a real answer rather than an empty
|
||||
coordinate.
|
||||
|
||||
### 8.8 What the build changed
|
||||
|
||||
Four things the implementation settled differently from §8 as written, all of them found by building
|
||||
against the live shard.
|
||||
|
||||
**1. `location` is a nested object, not flat `map`/`x`/`y`/`region`.** §8.1's payload sketch had them
|
||||
flat, and it would have made `market.location` — a rule Part A pre-wired — **inert**, exactly like
|
||||
the `characterName` miss §7.5 records: `projectValue` matches literal JSON keys, so there is no
|
||||
`location` key for the rule to match. Flat keys would have needed five rules that could drift apart.
|
||||
Nesting makes one rule hide the facet, the coordinates, the region and the house together, on the
|
||||
live frame and the stored read model alike, because both now spell it the same way.
|
||||
|
||||
The other pre-wired rule, `market.ownerName`, checked out — it is a real key on the frame. Owner is
|
||||
written as flat `ownerSerial`/`ownerName` rather than through `BridgeJson.Actor`, which would add
|
||||
`acct` and `webId`; same argument `points.board` makes. `ownerSerial` was **added** to the
|
||||
configurable fields alongside `ownerName`, because an admin who hides the owner's name and leaves a
|
||||
serial every other board resolves back to that name has not hidden anything.
|
||||
|
||||
**2. The per-vendor diff signature is the full listing set, not §8.3's `count | Σ(serial ^ price)`.**
|
||||
That hash collides on the single most common change a shop makes: two items swapping prices, which
|
||||
is what re-pricing looks like. The signature is built over the same buffer the frame is written
|
||||
from, in the same order, so a match really does mean an identical frame.
|
||||
|
||||
**3. There is no `payload` column on `shard_vendors`.** §8.5 implied the board pattern (whole frame
|
||||
in JSON, columns hoisted for display). It does not apply here: the items ARE the searchable rows, so
|
||||
they are normalized into `shard_vendor_items` and there is nothing left worth duplicating. The
|
||||
sidecar keeps the whole blob, because outage resilience is its job and search is not.
|
||||
|
||||
**4. Sweep cost is reported, and a slow tick warns.** The batch cap is a *claim* about per-tick cost,
|
||||
and an operator tuning `MarketSweepBatch` was otherwise tuning blind. `[bridge status` now carries
|
||||
`lastMs`/`maxMs`, and a tick over 50 ms prints a rate-limited warning naming the knob.
|
||||
|
||||
Measured on the live shard (27 vendors × 40 listings, 209k items / 43k mobiles):
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| First tick — 25 vendors emitted cold | **15.4 ms** |
|
||||
| Second tick — the remaining 2 | **3.4 ms** |
|
||||
| Steady state — nothing changed | **0.3 ms** |
|
||||
| Website `/market` search over 1,040 listings | 1,040 total, names resolved |
|
||||
| Cliloc re-resolution pass over 1,040 rows | **50 ms** |
|
||||
|
||||
The diff is what makes the steady state ~free; the batch cap is what bounds the cold case. Note the
|
||||
arithmetic the warning exists for: at the default cap of 250 listings, a batch of 25 **full** shops
|
||||
is 6,250 items ≈ 95 ms — over budget. Real shops hold tens, which is why 25 is the default, but a
|
||||
shard of commodity resellers should lower the batch, and now it will be told to.
|
||||
|
||||
Two smaller things worth not rediscovering:
|
||||
|
||||
- **`BridgeJson.Escape` takes a NON-NULL string** — it dereferences `value.Length` immediately — and
|
||||
`BridgeJson.Str` writes its own `,"key":` prefix, so neither serves a value inside a hand-built
|
||||
object. Nearly everything this frame writes is legitimately null (an item's plain `Name` is null
|
||||
for almost every item; a vendor in the street has no house), so that is the common path, not an
|
||||
edge case. `BridgeMarket.Text()` is the two-line writer that was missing.
|
||||
- **The ServUO console writes in the OS code page**, so an em dash in a `Console.WriteLine` renders
|
||||
as `???` in the log an operator would paste into an issue. Bridge console output is ASCII.
|
||||
|
||||
Search-side, one thing the site had to fix rather than inherit: `%` and `_` in a user's query are
|
||||
**LIKE** metacharacters, not SQL ones, so parameterization does not neutralize them — a search for
|
||||
`%` would otherwise match every listing on the shard. `shardMarket.db.js` escapes them. (The atlas's
|
||||
`LIKE` searches predate this and have the same shape over a much smaller table; worth a follow-up,
|
||||
not a blocker here.)
|
||||
|
||||
---
|
||||
|
||||
## 9. Sequencing
|
||||
@@ -817,8 +886,8 @@ inherently up to one full cycle old, and the UI must say so.
|
||||
| 2 | **B/1** — `world.ruleset` (§5) | all four | new kind | ✅ Done |
|
||||
| 3 | **C** — spawn atlas (§6) | website, docs | none | ✅ Done |
|
||||
| 4 | **B/2** — `points.board` (§7) | all four | new kind + `char.profile` field | ✅ Done |
|
||||
| 5a | **B/3 dependency** — cliloc table (§8.6) | website, docs | none | 🟨 In review |
|
||||
| 5b | **B/3** — `vendor.listing` (§8) | all four | new kinds | ⬜ |
|
||||
| 5a | **B/3 dependency** — cliloc table (§8.6) | website, docs | none | ✅ Done |
|
||||
| 5b | **B/3** — `vendor.listing` (§8) | all four | new kinds | 🟨 In review |
|
||||
| 6 | **Cutover** — `PROTOCOL_VERSION` 2→3, `edge` → `main` | all four | the bump | ⬜ |
|
||||
|
||||
---
|
||||
|
||||
@@ -407,6 +407,50 @@ Two values carry non-obvious meanings, both set by the plugin and both documente
|
||||
cliloc rather than a literal. Listing therefore orders by `COALESCE(name, system)`, so boards
|
||||
awaiting cliloc resolution sort by their own key rather than clumping together under NULL.
|
||||
|
||||
### shard_vendors / shard_vendor_items — the player-vendor marketplace (Protocol 3.0)
|
||||
|
||||
The shard-wide shop index, fed by `vendor.listing` / `vendor.listing.remove`. One row per player
|
||||
vendor and one per priced listing. Full operator detail in [`MARKETPLACE.md`](MARKETPLACE.md); the
|
||||
design is `docs/link/v3.md` §8.
|
||||
|
||||
| Table | Shape |
|
||||
|---|---|
|
||||
| `shard_vendors` | `serial` (PK), `shop_name`, `owner_serial`, `owner_name`, `map`/`x`/`y`/`z`, `region`, `house`, `item_count`, `item_total`, `truncated`, `t`, `updated_at`. Indexes on owner, map, region and `updated_at`. |
|
||||
| `shard_vendor_items` | `id` (PK), `vendor_serial`, `serial`, `item_id`, `hue`, `amount`, `price`, `name`, `cliloc`, `display_name`, `child`. Indexes on `vendor_serial`, `price`, `item_id`, `display_name`, and `(display_name, price)`. |
|
||||
|
||||
**Ingest is per-vendor and authoritative**: the frame is the whole shop, so ingest is
|
||||
delete-then-insert of that vendor's listings inside one transaction. All-or-nothing matters
|
||||
specifically because the two writes are "the shop" and "what is in it" — a failure between them
|
||||
leaves a shop advertising an inventory it no longer has, which is visibly wrong and indistinguishable
|
||||
from a genuinely empty shop. No foreign keys, consistent with every other `shard_*` table.
|
||||
|
||||
**There is deliberately no `payload` column**, unlike `shard_points_boards` directly above. The
|
||||
board's top-N is a fixed-size list read whole, so it lives in JSON; here the items *are* the
|
||||
searchable rows, so they are normalized and nothing is left worth duplicating. The sidecar keeps the
|
||||
whole blob — outage resilience is its job, search is ours.
|
||||
|
||||
Market state, not events: neither kind is in `LOGGED_KINDS`, and this is the strongest case of the
|
||||
three v3 kinds. One frame carries up to 250 listings and the sweep re-emits a shop on any price
|
||||
change, so logging would turn `shard_events` into a price history nobody reads.
|
||||
|
||||
Two columns carry non-obvious meanings:
|
||||
|
||||
- **`item_count` vs `item_total`.** `item_count` is what the frame published; `item_total` is what
|
||||
the shop actually holds. They differ when `truncated` — the shard caps listings per frame
|
||||
(`Bridge.MarketMaxListings`, 250 by default), and a commodity reseller with thousands of stacks
|
||||
genuinely exceeds it. Any UI must show both or it presents a partial shop as complete.
|
||||
- **`display_name` is denormalized at ingest**, resolved from the item's literal `name` (preferred —
|
||||
a player set it, so it is more specific) else its `cliloc` against `shard_clilocs`. Resolving at
|
||||
query time would put the cliloc table on the hot path and make search-by-name impossible. Because
|
||||
the shard's diff sweep will not re-send an unchanged shop just because the site learned what its
|
||||
items are called, **a cliloc import triggers a bulk re-resolution** of this column (after a boot
|
||||
import and after an admin import; ~50 ms per thousand rows, never throws).
|
||||
|
||||
`updated_at` is written explicitly on every upsert rather than left to `ON UPDATE CURRENT_TIMESTAMP`,
|
||||
which MariaDB does not fire when every column is written back unchanged. A shop re-published
|
||||
identically is still *freshly confirmed*, and without this the staleness banner would age a perfectly
|
||||
current shop forever.
|
||||
|
||||
### shard_feature_visibility — per-feature audience config (Protocol 3.0)
|
||||
|
||||
One row per shard feature: `feature` (PK), `enabled`, `audience` (a rung on the ladder in §6.5),
|
||||
@@ -745,6 +789,9 @@ from the per-route **siteMode** middleware (§5), never from an auth gate.
|
||||
| GET | `/shard/ruleset` | the shard's own published ruleset (Protocol 3.0 `world.ruleset`): expansion, which optional systems are on, skill/stat caps, account and house limits, champion scroll rules, the save/restart schedule. Served from `shard_ruleset`, so it renders while the shard is down; live via `world.ruleset` on `/shard/stream`. Behind `requireFeature('ruleset')`. **`null`** means the shard has never published one — a real answer, distinct from a published ruleset. `caps.skill` / `caps.totalSkill` are in **tenths** (1000 = 100.0). |
|
||||
| GET | `/shard/points` | every points/loyalty leaderboard the shard publishes (Protocol 3.0 `points.board`) — Queen's Loyalty, Void Pool, the nine city loyalties, Clean Up Britannia, … Served from `shard_points_boards`, so it renders while the shard is down; live via `points.board` on `/shard/stream`. Behind `requireFeature('leaderboards')`, ordered by display name. **`maxPoints: 0` means uncapped** (the common case), and `nameString` is usually `null` with `nameNumber` holding a cliloc — resolve client-side or humanise the `system` key. |
|
||||
| GET | `/shard/points/:system` | one board by the shard's `PointsType` name (e.g. `QueensLoyalty`); `:system` must match `/^[A-Za-z][A-Za-z0-9_]{0,47}$/` or **400** before any query runs. **404** = the shard has never published that system, which is distinct from a published board nobody has scored in yet (**200** with an empty `top`). |
|
||||
| GET | `/shard/market?q=&minPrice=&maxPrice=&itemId=&map=®ion=&sort=&limit=&offset=` | search the player-vendor marketplace (Protocol 3.0 `vendor.listing`). Returns **listings**, not vendors — "who sells X and for how much" is the question, and a vendor-shaped result would make every caller flatten the shops back out. Served from `shard_vendors` + `shard_vendor_items`, so it renders while the shard is down. Behind `requireFeature('market')` **and rate-limited** — the first genuinely expensive public read on the site (a `LIKE` scan plus a `COUNT` over what is typically the largest `shard_*` table, reachable with no session). `sort ∈ {price_asc, price_desc, recent}`. `q` matches the resolved display name **or** the item's literal name, with `%`/`_` escaped: they are `LIKE` metacharacters, not SQL ones, so parameterization alone would let `?q=%` match every listing on the shard. Every response repeats `staleAt` (the oldest vendor row) because the shard sweeps round-robin — a banner that ages with the results it labels, not one fetched once. |
|
||||
| GET | `/shard/market/meta` | index size, staleness (`staleAt`/`freshAt`) and which facets and regions actually hold vendors, so a client builds its filters without running a search it will discard. |
|
||||
| GET | `/shard/market/vendors/:serial` | one shop and its listings; `:serial` must match `/^0x[0-9A-Fa-f]{1,16}$/` or **400** before any query runs. **404** = a serial the index has never seen, which also covers a vendor since dismissed or hidden — to an anonymous caller those are the same answer, and distinguishing them would leak that a hidden vendor exists. `truncated` (with `total` exceeding `count`) means the shop holds more than the shard publishes per frame. |
|
||||
| GET | `/shard/features` | the shard features **this caller** may reach plus the audience rung they resolved to (§6.5), so a client hides nav it can't follow. Reports only what the caller can see — the list itself never discloses a gated feature. Consumed by the SPA header and (pending) the Android nav. |
|
||||
| GET | `/atlas/creatures?q=&facet=&limit=&offset=` | the bestiary, most numerous first, with an unpaginated `total`. Static content parsed from the shard's ServUO tree — **not** sidecar-backed, which is why the atlas sits outside `/shard`, and unlike `/shard/*` it **is** site-mode gated. Behind `requireFeature('atlas')`. `?facet=` is matched exactly and never validated against a list (no facet name exists in the code); the filter is an `EXISTS` over the points rather than a JSON path or `JSON_SEARCH` built from caller input, whose `%`/`_` wildcards would make `?facet=%` match everything. |
|
||||
| GET | `/atlas/creatures/:slug` | one creature: `places` (the point-in-rect aggregate — "lizardman → Shrines, Isamu-Jima, Yew"), `spawners` (the bounded raw list, with `spawnersTruncated`), `alsoHere`. **`points` is a COUNT and `spawners` is the LIST** — named apart so one key never means a number on one route and an array on another. `minDelay`/`maxDelay` are in **seconds**, normalised at parse time from the source's per-record minutes-or-seconds. 404 = no such creature in this atlas. |
|
||||
@@ -835,7 +882,11 @@ who"; `activity_log` provides the history feed.
|
||||
- **Cookie**: `httpOnly`, `sameSite=Lax`, `path=/`, and **`secure` decided per-request** (`COOKIE_SECURE=auto` → `secure: req.secure`).
|
||||
- **Trusted-device MFA.** A second, separate httpOnly cookie (`rg_trust`, default 30d) — opaque, sha256-hashed server-side in `trusted_devices` — lets a browser/app **skip the TOTP step** (never the password) on future logins. It is a server-side, per-row-revocable record (never a JWT claim), so the stateless session JWT is unchanged and trust stays revocable. It only ever gates the **second factor**; it deliberately outlives logout, and is cleared on untrust / password change / password reset / TOTP disable. **Recovery codes** (bcrypt, single-use) are the 2FA-lockout fallback. All admin trusted-device/MFA actions and the self actions (`auth.login.trusted_device`, `account.trusted_device.*`, `account.recovery_code*`, `admin.trusted_device.*`, `admin.user.totp.reset`) are audit-logged. See `docs/website/TRUSTED_DEVICES_MFA.md`. This is the key to dual access: the cookie is `Secure` when reached through Pangolin (HTTPS, `X-Forwarded-Proto: https`) but **not** `Secure` when reached directly over the LAN IP on plain HTTP — so login works in both. `COOKIE_SECURE=true|false` can force it. Requires `trust proxy` (below). `localhost:5173` (Vite) and `localhost:3000` are same-site, so the cookie flows in dev too.
|
||||
- **bcrypt** hashing (cost 10+); plaintext passwords never stored, logged, or returned.
|
||||
- **Rate limiting** (`express-rate-limit`) on `/auth/login` and `/public/contact`.
|
||||
- **Rate limiting** (`express-rate-limit`) on `/auth/login`, `/public/contact`, and — the only limited
|
||||
*read* — `/public/shard/market` and `/public/shard/market/vendors/:serial` (60/min/IP). Every other
|
||||
public read is an indexed lookup of bounded size; the marketplace search is a `LIKE` scan plus a
|
||||
`COUNT` over the largest `shard_*` table, anonymous by default, so it is the one public GET that is
|
||||
worth money to serve.
|
||||
- **Validation** (`express-validator`) on all writes; centralized error handler.
|
||||
- **helmet** with a Content-Security-Policy tuned for the built React SPA. The policies now live in
|
||||
**`server/src/config/csp.js`** (`app.js` only wires them up):
|
||||
|
||||
167
website/MARKETPLACE.md
Normal file
167
website/MARKETPLACE.md
Normal file
@@ -0,0 +1,167 @@
|
||||
# Marketplace — the player-vendor index
|
||||
|
||||
**Status:** On `edge` — servuo-plugins [#5](https://gitea.whitlocktech.com/RunicGateway/servuo-plugins/pulls/5), link [#19](https://gitea.whitlocktech.com/RunicGateway/link/pulls/19), website [#116](https://gitea.whitlocktech.com/RunicGateway/website/pulls/116).
|
||||
**Design:** [`docs/link/v3.md` §8](../link/v3.md) — Protocol 3.0 Part B/3.
|
||||
**Depends on:** [`CLILOCS.md`](CLILOCS.md) — without a cliloc table, listings render as item ids.
|
||||
|
||||
The marketplace is a searchable index of every player vendor on the shard: what
|
||||
each shop is selling, for how much, and where it is standing. It is the same set
|
||||
the in-game **Vendor Search** gump reads, offered from outside the game — so a
|
||||
player can find the vanquishing kryss they want before logging in, and someone
|
||||
who does not play at all can see that the economy exists.
|
||||
|
||||
Page: `/site/market`, plus `/site/market/vendors/:serial` for one shop.
|
||||
|
||||
## Three things the pages must say out loud
|
||||
|
||||
Everything below follows from how the data is gathered, and each has a visible
|
||||
consequence the UI is required to surface.
|
||||
|
||||
**1. The prices are not live.** The shard sweeps vendors **round-robin** — at
|
||||
most `Bridge.MarketSweepBatch` shops per tick — so a given shop can be a full
|
||||
cycle behind. The page carries a *"prices last refreshed N minutes ago"* banner
|
||||
driven by the **oldest** vendor row, not the newest: the one stale shop is the
|
||||
one that wastes somebody's trip.
|
||||
|
||||
**2. A shop can be truncated.** `Bridge.MarketMaxListings` (250 by default) caps
|
||||
how many listings one frame carries. A commodity reseller with thousands of
|
||||
stacked resources is a real thing, and an uncapped frame for one is measured in
|
||||
megabytes. Over the cap the shop reports `truncated`, and the vendor page says
|
||||
*"showing 250 of 3,104 — this shop holds more than the shard publishes"* rather
|
||||
than presenting a partial shop as complete.
|
||||
|
||||
**3. An item may have no name.** Items on the wire carry a cliloc id, not a name.
|
||||
On a shard whose operator has not converted a cliloc table
|
||||
([`CLILOCS.md`](CLILOCS.md)) the honest render is the item id — never an invented
|
||||
label, which would be indistinguishable from a real one.
|
||||
|
||||
## Privacy: the player's own toggle wins
|
||||
|
||||
Only vendors whose owner left the in-game **Vendor Search** flag ON are ever sent
|
||||
to the site. A player who hides their shop in game is hidden here too, and no
|
||||
admin setting overrides that. When they hide one that was already indexed, the
|
||||
shard emits `vendor.listing.remove` and the row is deleted — so revoking consent
|
||||
takes effect, it does not merely stop refreshing.
|
||||
|
||||
Shop name, owner character name and location default to **Everyone**, because the
|
||||
stock Vendor Search gump already shows exactly that set to any player in game.
|
||||
They remain admin-configurable; see [`SHARD_VISIBILITY.md`](SHARD_VISIBILITY.md).
|
||||
Account names and website user ids never cross the wire at all.
|
||||
|
||||
## How it is put together
|
||||
|
||||
```
|
||||
ServUO uo-link sidecar website
|
||||
────── ─────────────── ───────
|
||||
BridgeMarket.cs vendors table shard_vendors
|
||||
round-robin sweep ──────► (whole frame blob) ──────► shard_vendor_items
|
||||
per-vendor diff GET /market (paged) + display_name
|
||||
vendor.listing resolved at ingest
|
||||
vendor.listing.remove
|
||||
```
|
||||
|
||||
**The shard side** walks at most `MarketSweepBatch` vendors per tick from a
|
||||
persistent cursor, diffs each against what it last published, and emits a whole
|
||||
frame for any shop that moved. Per-tick cost is therefore bounded by the batch,
|
||||
not by how many vendors the world holds — full coverage takes
|
||||
`ceil(vendors / batch) × MarketSweepSeconds`.
|
||||
|
||||
**The sidecar** stores each frame whole and serves `GET /market`, its only paged
|
||||
read. It normalizes nothing and defines no audiences: it is a dumb forwarder, and
|
||||
search is the website's job.
|
||||
|
||||
**The website** splits each frame into a vendor row and its listings, replacing
|
||||
that vendor's whole listing set inside one transaction (the frame is
|
||||
authoritative for that vendor, never a delta). Item names are resolved against
|
||||
the cliloc table **on the way in** and stored denormalized, which is what makes
|
||||
search-by-name possible and keeps the cliloc table off the hot path.
|
||||
|
||||
## Operating it
|
||||
|
||||
Everything is in `Config/Bridge.cfg` on the shard. There is nothing to configure
|
||||
on the website.
|
||||
|
||||
| Setting | Default | What it does |
|
||||
|---|---|---|
|
||||
| `MarketEnabled` | `true` | Master switch. Off publishes nothing; the page shows an empty index. |
|
||||
| `MarketSweepSeconds` | `60` | Tick interval. |
|
||||
| `MarketSweepBatch` | `25` | Vendors inventoried per tick. Clamped 1..500. |
|
||||
| `MarketMaxListings` | `250` | Per-shop listing cap, after which `truncated`. Clamped 1..5000. |
|
||||
|
||||
**Faster coverage vs. per-tick cost.** Lowering `MarketSweepSeconds` or raising
|
||||
`MarketSweepBatch` both refresh the index sooner and both cost more per tick.
|
||||
The expensive part is the item walk, which recurses into every container a vendor
|
||||
is selling — so a shard of big shops should raise the interval rather than the
|
||||
batch.
|
||||
|
||||
`[bridge status` reports the sweep, including `lastMs` and `maxMs`:
|
||||
|
||||
```
|
||||
market(enabled=True sweeps=42 scanned=108 emitted=27 removed=0 skipped=0
|
||||
truncated=0 tracked=27 vendors=27 cursor=2 batch=25 lastMs=0.31 maxMs=15.40)
|
||||
```
|
||||
|
||||
A tick over **50 ms** prints a rate-limited warning naming the knob:
|
||||
|
||||
```
|
||||
[Bridge] market sweep took 82.4 ms (budget 50 ms) - lower Bridge.MarketSweepBatch (now 25) if this persists
|
||||
```
|
||||
|
||||
Measured on a shard with 27 vendors × 40 listings (209k items, 43k mobiles):
|
||||
**15.4 ms** for the first cold tick of 25 vendors, **0.3 ms** in steady state —
|
||||
the diff is what makes an unchanged world nearly free. Note the arithmetic: 25
|
||||
*full* shops at the 250-listing cap is 6,250 items ≈ 95 ms, over budget. Real
|
||||
shops hold tens, which is why 25 is the default and why the warning exists.
|
||||
|
||||
`[bridge sweepnow` runs one tick immediately; `[bridge reload` re-reads the
|
||||
settings above without a restart.
|
||||
|
||||
## Names arriving late
|
||||
|
||||
Item names come from the cliloc table, and the market sweep will **not** re-send
|
||||
an unchanged shop just because the site learned what its items are called. So a
|
||||
cliloc import triggers a bulk re-resolution of every stored listing — otherwise
|
||||
an operator who configures clilocs after the first sweep would see item ids until
|
||||
every shop happened to change on its own. It runs after a boot import and after
|
||||
an admin import, takes ~50 ms per thousand listings, and never throws: a failure
|
||||
leaves names exactly as they were.
|
||||
|
||||
## API
|
||||
|
||||
All under `/api/v1/public/shard`, gated by the `market` feature and
|
||||
**rate-limited** — these are the first genuinely expensive public reads on the
|
||||
site (a `LIKE` scan plus a `COUNT` over what is typically the largest `shard_*`
|
||||
table, reachable with no session).
|
||||
|
||||
| Route | What |
|
||||
|---|---|
|
||||
| `GET /market` | Search. Returns **listings**, not vendors — "who sells X and for how much" is the question. `?q=&minPrice=&maxPrice=&itemId=&map=®ion=&sort=&limit=&offset=`, `sort ∈ {price_asc, price_desc, recent}`. |
|
||||
| `GET /market/meta` | Index size, staleness, and which facets and regions actually hold vendors — so a client builds its filters without running a search it will discard. |
|
||||
| `GET /market/vendors/:serial` | One shop and its listings. **404** for a serial the index has never seen, which also covers a vendor since dismissed or hidden — to an anonymous caller those are the same answer. |
|
||||
|
||||
`q` matches the resolved display name **or** the item's own literal name, because
|
||||
an item with a player-set name (most of what is worth searching for on a
|
||||
player-run shard) may carry a generic cliloc. `%` and `_` in a query are escaped:
|
||||
they are `LIKE` metacharacters, not SQL ones, so parameterization alone would let
|
||||
a search for `%` match every listing on the shard.
|
||||
|
||||
Full schemas are in the OpenAPI spec (`ShardMarketPage`, `ShardMarketVendor`,
|
||||
`ShardMarketMeta`, `ShardMarketListing`, `ShardMarketLocation`).
|
||||
|
||||
## Tables
|
||||
|
||||
`shard_vendors` (one row per shop) and `shard_vendor_items` (one row per priced
|
||||
listing). Both are ingest-owned; nothing else writes to them. No foreign keys,
|
||||
in keeping with every other `shard_*` table — the ingest transaction is what
|
||||
keeps them consistent, and an FK would turn a malformed frame into a failed write
|
||||
rather than a dropped row.
|
||||
|
||||
There is deliberately **no `payload` column** on `shard_vendors`, unlike the
|
||||
points board next door. The board's top-N is a fixed-size list read whole, so it
|
||||
lives in JSON; here the items *are* the searchable rows, so they are normalized
|
||||
and there is nothing left worth duplicating. The sidecar keeps the whole blob,
|
||||
because outage resilience is its job.
|
||||
|
||||
`shard_vendor_items.display_name` is denormalized and indexed (alone, and
|
||||
composite with `price` for "cheapest matching X"). See "Names arriving late"
|
||||
above for how it is kept current.
|
||||
@@ -60,12 +60,32 @@ board while holding back one column. See the table in §3.
|
||||
| **Shard rules** | Skill/stat caps, house limits, vet rewards, the ruleset | Everyone | Connect address → Everyone |
|
||||
| **Spawn atlas** | Bestiary and spawn locations (static content) | Everyone | — |
|
||||
| **Leaderboards** | Point and loyalty standings | Everyone | Character names → Everyone |
|
||||
| **Marketplace** | The shard-wide player-vendor index | Everyone, **live updates off** | Vendor owner name → Everyone · Location → Everyone |
|
||||
| **Marketplace** | The shard-wide player-vendor index | Everyone, **live updates off** | Vendor owner name → Everyone · Vendor owner character id → Everyone · In-game location → Everyone |
|
||||
|
||||
**Why the marketplace ships with live updates off.** A live feed of every vendor's full inventory
|
||||
would be the single largest thing the site sends. No page needs it — the marketplace is a search over
|
||||
stored data with a “prices last refreshed N minutes ago” stamp. Turn it on only if you want it.
|
||||
|
||||
**Why the marketplace's fields default to Everyone.** A vendor's shop name, its owner's character
|
||||
name and where it is standing are *already* visible to every player in game: the stock Vendor Search
|
||||
gump surfaces exactly that set to anyone who opens it. Publishing them on the site is not a new
|
||||
disclosure. They stay configurable because a shard may still prefer to keep its economy behind a
|
||||
login — and because "already public in game" is a judgement about your shard, not ours.
|
||||
|
||||
**Location is one setting covering four things.** Hiding it removes the facet, the coordinates, the
|
||||
region *and* the house name together. That is deliberate: those are four ways of saying the same
|
||||
thing, and a setting that hid the coordinates while publishing the house name would not have hidden
|
||||
anything.
|
||||
|
||||
**Hiding the owner name also hides the owner character id.** They are separate settings so you can
|
||||
be explicit, but leaving the id published while hiding the name achieves nothing — the leaderboards
|
||||
and guild boards resolve that same id back to a character name. Set both.
|
||||
|
||||
**What hiding a vendor cannot do.** Only vendors whose owner left the in-game *Vendor Search* flag ON
|
||||
are ever sent to the site, so a player who hides their shop in game is hidden here too — and no
|
||||
setting on this page can override that. It works the other way as well: these settings control who
|
||||
sees the index, not whether players can find each other's shops in game.
|
||||
|
||||
**Why house owner/price default to Staff.** The public Houses page has always been a "where are the
|
||||
falling houses" board — location only. Owner and price are the staff view. That split is preserved.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user