feat(shard): the player-vendor marketplace #116

wtclaude · 2026-07-29T14:56:13Z

wtclaude commented

2026-07-29 14:56:13 +00:00

What & why

Protocol 3.0 §8 (docs/link/v3.md) — order 5b, the website side, and the last part before the cutover. Plugin in servuo-plugins #5; sidecar in link #19; docs in docs #71.

The shard-wide shop index: what every player vendor is selling, for how much, and where it is standing — the same set the in-game Vendor Search gump reads, offered from outside the game. /site/market and /site/market/vendors/:serial.

This is also the payoff for #115: item names come from the cliloc table, so listings read "longsword" rather than "id 3937".

⚠️ Part A's pre-wired visibility rules, re-checked against the real frame

Part A pre-wired market.ownerName and market.location before the frame existed. The sibling rule it pre-wired for leaderboards — characterName — turned out to be inert, because projectValue matches literal JSON keys and the wire key is name. So both market rules were checked this time rather than trusted:

ownerName is a real key on the frame. Kept.
location is a real key only because the frame nests it. §8.1's payload sketch had flat map/x/y/region, which would have made the rule match nothing — the same failure, one part later. It is nested on the wire and on the read model so one rule hides the facet, the coordinates, the region and the house together; five flat keys would be five rules that drift apart.
ownerSerial was ADDED. An admin who hides the owner's name and leaves a serial that the leaderboards and guild boards resolve straight back to that name has not hidden anything.

Tests assert all three bite on the stored read model and on the raw frame. The market's SSE stream is off by default, but an admin can turn it on, and a field rule that worked on only one of the two paths is exactly the leak §3.6.1 records.

Three things the pages must say out loud

All consequences of how the data is gathered, and each has a visible UI obligation:

The prices are not live. The shard sweeps round-robin, so a shop can be a full cycle behind. The banner is driven by the oldest vendor row, not the newest — the one stale shop is the one that wastes somebody's trip. staleAt rides on every search response rather than sitting behind a separate call, so the stamp ages with the results it labels.
A shop can be truncated. total exceeding count means the shop holds more than the shard publishes per frame. The vendor page says "showing 250 of 3,104"; a commodity reseller with thousands of stacks is real.
An item may have no name. With no cliloc table configured, displayName is null and the honest render is the item id — never a fabricated "Item 3922", which would be indistinguishable from a real name.

Notable

No payload column on shard_vendors, unlike shard_points_boards next door. The board's top-N is a fixed-size list read whole, so it lives in JSON; here the items are the searchable rows, so they are normalized and nothing is left worth duplicating. The sidecar keeps the whole blob — outage resilience is its job, search is ours.
display_name is denormalized at ingest, resolved from the item's literal name (preferred — a player set it, so it is more specific) else its cliloc. Resolving at query time would put the cliloc table on the hot path and make search-by-name impossible.
A cliloc import triggers a bulk re-resolution, and it has to: the shard's diff sweep will not re-send an unchanged shop just because the site learned what its items are called. Without this, an operator who configures clilocs after the first sweep sees item ids until every shop happens to change. 50 ms per thousand rows, never throws.
updated_at is written explicitly on every upsert. MariaDB does not fire ON UPDATE CURRENT_TIMESTAMP when every column is written back unchanged — and a shop re-published identically is still freshly confirmed. Without this the staleness banner would age a perfectly current shop forever.
LIKE wildcards in q are escaped. % and _ are LIKE metacharacters, not SQL ones, so parameterization does not neutralize them: ?q=% would otherwise match every listing on the shard. (The atlas's LIKE searches predate this and have the same shape over a much smaller table — worth a follow-up, not a blocker here.)
Rate-limited, 60/min/IP — the only limited read on the site. Every other public GET is an indexed lookup of bounded size; this is a LIKE scan plus a COUNT over what is typically the largest shard_* table, anonymous by default.
Reconnect backfill pages /market, bounded by MARKET_SNAPSHOT_MAX = 5000 and stopping on a short page as well as on total, so a concurrent sweep shrinking the index cannot spin the walk. Backfill runs before the site serves the live feed, so an unbounded loop there is downtime, not slowness.
Ingest is delete-then-insert per vendor in one transaction. All-or-nothing matters for a specific reason: the two writes are "the shop" and "what is in it", and a failure between them leaves a shop advertising an inventory it no longer has — visibly wrong, and indistinguishable from a genuinely empty shop.

How it was tested

673 server tests pass (27 new: shardIngest.market.test.js for routing, shardMarket.model.test.js for the flatten/shape rules and the visibility projection). Client builds clean. swagger-output.json, routes.manifest.json and routes.guards.json regenerated and committed.

Verified full-stack against the live MariaDB and a real shard, not only in units — plugin → Rust sidecar → website → API:

27 real vendors / 1,040 listings swept off the ServUO tree and into the site, with names resolving through the cliloc table ("longsword", "katana", "broadsword") and real facets and regions populating the filters.
Search: ?q=sword → 682, ?q=long → 352, ?q=% → 0 and ?q=_ → 0 (the escape). Price range, facet, region, all three sorts, paging, and the vendor detail route.
Visibility live, all four gates: fields tightened to staff vanish for an anonymous caller while shopName and price survive; audience=player → 403; enabled=0 → 404; and /shard/features correctly drops market, so the nav hides the link rather than rendering one that 403s.
Validation: ?sort=bogus → 400, ?limit=999 → 400, /market/vendors/0xNOPE! → 400, unknown serial → 404.
Re-publishing a shop smaller leaves no orphan items; an identical re-publish still moves updated_at; removing a vendor leaves zero orphan rows.
The limiter fires: 38×200 then 32×429 on a 70-request burst.

Not covered by an automated test: the two React pages are presentational and this repo's client suite covers pure-logic modules only (there is no DOM harness). They were driven against the live API above, but not rendered in one.

CI note: pr-checks.yml only triggers on PRs into main, so this PR (targeting edge) runs no CI — local verification above is the gate.

Checklist

I have read CONTRIBUTING.md.
The change builds and existing tests/checks pass locally.
I have added or updated tests/docs where it makes sense.
My commits are reasonably scoped with clear messages.

AI-assisted contributions (required)

No AI tools were used to produce this contribution.
AI tools were used. Tool(s): Claude Code (Opus 5). I have reviewed and understand every change, and take responsibility for it. AI-authored commits are marked with a Co-Authored-By trailer.

License

I agree that my contribution is licensed under this project's license (GNU GPL v3.0 or later), and I have the right to contribute it.

## What & why Protocol 3.0 **§8** ([`docs/link/v3.md`](https://gitea.whitlocktech.com/RunicGateway/docs/src/branch/edge/link/v3.md)) — order 5b, the website side, and the last part before the cutover. Plugin in servuo-plugins #5; sidecar in link #19; docs in docs #71. The shard-wide shop index: what every player vendor is selling, for how much, and where it is standing — the same set the in-game **Vendor Search** gump reads, offered from outside the game. `/site/market` and `/site/market/vendors/:serial`. This is also the payoff for #115: item names come from the cliloc table, so listings read *"longsword"* rather than *"id 3937"*. ### ⚠️ Part A's pre-wired visibility rules, re-checked against the real frame Part A pre-wired `market.ownerName` and `market.location` before the frame existed. The sibling rule it pre-wired for leaderboards — `characterName` — turned out to be **inert**, because `projectValue` matches literal JSON keys and the wire key is `name`. So both market rules were checked this time rather than trusted: - **`ownerName`** is a real key on the frame. Kept. - **`location`** is a real key **only because the frame nests it.** §8.1's payload sketch had flat `map`/`x`/`y`/`region`, which would have made the rule match nothing — the same failure, one part later. It is nested on the wire *and* on the read model so one rule hides the facet, the coordinates, the region and the house together; five flat keys would be five rules that drift apart. - **`ownerSerial` was ADDED.** An admin who hides the owner's name and leaves a serial that the leaderboards and guild boards resolve straight back to that name has not hidden anything. Tests assert all three bite **on the stored read model and on the raw frame**. The market's SSE stream is off by default, but an admin can turn it on, and a field rule that worked on only one of the two paths is exactly the leak §3.6.1 records. ### Three things the pages must say out loud All consequences of how the data is gathered, and each has a visible UI obligation: 1. **The prices are not live.** The shard sweeps round-robin, so a shop can be a full cycle behind. The banner is driven by the **oldest** vendor row, not the newest — the one stale shop is the one that wastes somebody's trip. `staleAt` rides on every search response rather than sitting behind a separate call, so the stamp ages with the results it labels. 2. **A shop can be truncated.** `total` exceeding `count` means the shop holds more than the shard publishes per frame. The vendor page says *"showing 250 of 3,104"*; a commodity reseller with thousands of stacks is real. 3. **An item may have no name.** With no cliloc table configured, `displayName` is null and the honest render is the item id — never a fabricated *"Item 3922"*, which would be indistinguishable from a real name. ### Notable - **No `payload` column on `shard_vendors`**, unlike `shard_points_boards` next door. The board's top-N is a fixed-size list read whole, so it lives in JSON; here the items **are** the searchable rows, so they are normalized and nothing is left worth duplicating. The sidecar keeps the whole blob — outage resilience is its job, search is ours. - **`display_name` is denormalized at ingest**, resolved from the item's literal `name` (preferred — a player set it, so it is more specific) else its cliloc. Resolving at query time would put the cliloc table on the hot path *and* make search-by-name impossible. - **A cliloc import triggers a bulk re-resolution**, and it has to: the shard's diff sweep will not re-send an unchanged shop just because the site learned what its items are called. Without this, an operator who configures clilocs after the first sweep sees item ids until every shop happens to change. 50 ms per thousand rows, never throws. - **`updated_at` is written explicitly** on every upsert. MariaDB does **not** fire `ON UPDATE CURRENT_TIMESTAMP` when every column is written back unchanged — and a shop re-published identically is still *freshly confirmed*. Without this the staleness banner would age a perfectly current shop forever. - **LIKE wildcards in `q` are escaped.** `%` and `_` are `LIKE` metacharacters, not SQL ones, so parameterization does not neutralize them: `?q=%` would otherwise match every listing on the shard. (The atlas's `LIKE` searches predate this and have the same shape over a much smaller table — worth a follow-up, not a blocker here.) - **Rate-limited, 60/min/IP** — the only limited *read* on the site. Every other public GET is an indexed lookup of bounded size; this is a `LIKE` scan plus a `COUNT` over what is typically the largest `shard_*` table, anonymous by default. - **Reconnect backfill pages `/market`**, bounded by `MARKET_SNAPSHOT_MAX = 5000` and stopping on a short page as well as on `total`, so a concurrent sweep shrinking the index cannot spin the walk. Backfill runs before the site serves the live feed, so an unbounded loop there is downtime, not slowness. - Ingest is **delete-then-insert per vendor in one transaction**. All-or-nothing matters for a specific reason: the two writes are "the shop" and "what is in it", and a failure between them leaves a shop advertising an inventory it no longer has — visibly wrong, and indistinguishable from a genuinely empty shop. ## How it was tested **673 server tests pass** (27 new: `shardIngest.market.test.js` for routing, `shardMarket.model.test.js` for the flatten/shape rules and the visibility projection). Client builds clean. `swagger-output.json`, `routes.manifest.json` and `routes.guards.json` regenerated and committed. Verified **full-stack against the live MariaDB and a real shard**, not only in units — plugin → Rust sidecar → website → API: - **27 real vendors / 1,040 listings** swept off the ServUO tree and into the site, with names resolving through the cliloc table (*"longsword"*, *"katana"*, *"broadsword"*) and real facets and regions populating the filters. - Search: `?q=sword` → 682, `?q=long` → 352, **`?q=%` → 0 and `?q=_` → 0** (the escape). Price range, facet, region, all three sorts, paging, and the vendor detail route. - **Visibility live, all four gates:** fields tightened to `staff` vanish for an anonymous caller while `shopName` and `price` survive; `audience=player` → **403**; `enabled=0` → **404**; and `/shard/features` correctly drops `market`, so the nav hides the link rather than rendering one that 403s. - Validation: `?sort=bogus` → 400, `?limit=999` → 400, `/market/vendors/0xNOPE!` → 400, unknown serial → 404. - Re-publishing a shop **smaller** leaves no orphan items; an **identical** re-publish still moves `updated_at`; removing a vendor leaves zero orphan rows. - The limiter fires: 38×200 then **32×429** on a 70-request burst. **Not covered by an automated test:** the two React pages are presentational and this repo's client suite covers pure-logic modules only (there is no DOM harness). They were driven against the live API above, but not rendered in one. **CI note:** `pr-checks.yml` only triggers on PRs into `main`, so this PR (targeting `edge`) runs no CI — local verification above is the gate. ## Checklist - [x] I have read [CONTRIBUTING.md](CONTRIBUTING.md). - [x] The change builds and existing tests/checks pass locally. - [x] I have added or updated tests/docs where it makes sense. - [x] My commits are reasonably scoped with clear messages. ## AI-assisted contributions (required) - [ ] No AI tools were used to produce this contribution. - [x] AI tools were used. Tool(s): `Claude Code (Opus 5)`. I have reviewed and understand every change, and take responsibility for it. AI-authored commits are marked with a `Co-Authored-By` trailer. ## License - [x] I agree that my contribution is licensed under this project's license (**GNU GPL v3.0 or later**), and I have the right to contribute it.

wtclaude added 1 commit 2026-07-29 14:56:14 +00:00

feat(shard): the player-vendor marketplace 8771a1cf6c

Protocol 3.0 §8, the website half. Ingests vendor.listing / vendor.listing.remove
into shard_vendors + shard_vendor_items, serves a searchable public API over
them, and ships /site/market and /site/market/vendors/:serial.

Three things the pages have to say out loud, all consequences of how the data is
gathered:

- The prices are NOT live. The shard sweeps vendors round-robin, so a shop can be
  a full cycle behind. The banner is driven by the OLDEST vendor row, not the
  newest — the one stale shop is the one that wastes somebody's trip.
- A shop can be truncated. `total` exceeding `count` means the shop holds more
  than the shard publishes per frame; the vendor page says "showing 250 of 3,104"
  rather than presenting a partial shop as complete.
- An item may have no name. On a shard with no cliloc table the honest render is
  the item id, never an invented label.

## The pre-wired visibility rules, re-checked

Part A pre-wired market.ownerName and market.location before the frame existed,
and the sibling rule it pre-wired for leaderboards (`characterName`) turned out
to be INERT because projectValue matches literal JSON keys. Both market rules
were checked against the real frame this time:

- `ownerName` is a real key. Kept.
- `location` is a real key ONLY because the frame nests it. Flat map/x/y/region
  would have made the rule match nothing — the same failure, one part later. It
  is nested on the wire and on the read model so one rule hides the facet, the
  coordinates, the region and the house together; five flat keys would be five
  rules that drift apart.
- `ownerSerial` was ADDED. An admin who hides the owner's name and leaves a
  serial that the leaderboards and guild boards resolve back to that same name
  has not hidden anything.

Tests assert all three bite, on the stored read model AND on the raw frame —
the market's SSE stream is off by default but an admin can turn it on, and a rule
that worked on only one path is exactly the leak §3.6.1 records.

## Notable

- **No payload column on shard_vendors**, unlike shard_points_boards next door.
  The board's top-N is a fixed-size list read whole; here the items ARE the
  searchable rows, so they are normalized and nothing is left worth duplicating.
- **display_name is denormalized at ingest** (literal name preferred over the
  cliloc — a player set it, so it is more specific). Resolving at query time
  would put the cliloc table on the hot path and make search-by-name impossible.
  Because the shard's diff sweep will not re-send an unchanged shop just because
  the site learned what its items are called, a cliloc import now triggers a bulk
  re-resolution — 50 ms per thousand rows, never throws.
- **updated_at is written explicitly** on every upsert. MariaDB does not fire ON
  UPDATE CURRENT_TIMESTAMP when every column is written back unchanged, and a
  shop re-published identically is still freshly confirmed — without this the
  staleness banner would age a perfectly current shop forever.
- **LIKE wildcards in `q` are escaped.** `%` and `_` are LIKE metacharacters, not
  SQL ones, so parameterization does not neutralize them: `?q=%` would otherwise
  match every listing on the shard.
- **Rate-limited** (60/min/IP), the only limited public read. Every other public
  GET is an indexed lookup of bounded size; this is a LIKE scan plus a COUNT over
  the largest shard_* table, anonymous by default.
- Reconnect backfill pages /market, bounded by MARKET_SNAPSHOT_MAX = 5000 and
  stopping on a short page as well as on `total`, so a concurrent sweep shrinking
  the index cannot spin the walk.

## How it was tested

673 server tests pass (27 new). Client builds clean; swagger-output.json,
routes.manifest.json and routes.guards.json regenerated.

Verified full-stack against the live MariaDB and a real shard, not only units:

- 27 real vendors / 1,040 listings swept off the ServUO tree, through the Rust
  sidecar, into the site — names resolving through the cliloc table ("longsword",
  "katana"), real facets and regions in the filters.
- `?q=sword` 682, `?q=%` and `?q=_` **0** (the escape), map/region/price/sort
  filters, paging, and the vendor detail route.
- Visibility live: fields gated to staff vanish for an anonymous caller while
  shopName and price survive; audience=player 403s; enabled=0 404s; and
  /shard/features correctly drops `market` so the nav hides it.
- Re-publishing a shop smaller leaves no orphan items; an identical re-publish
  moves updated_at.
- The limiter fires (38x200 then 32x429 on a 70-request burst).

Not covered by an automated test: the two React pages are presentational and this
repo's client suite covers pure-logic modules only. They were driven against the
live API above, but not rendered in a DOM harness.

Co-Authored-By: Claude <noreply@anthropic.com>

whitlocktech merged commit c6c0c257dd into edge

2026-07-29 20:03:24 +00:00

whitlocktech deleted branch feat/vendor-listing

2026-07-29 20:03:25 +00:00

whitlocktech referenced this issue from a commit

2026-07-29 20:03:25 +00:00

Merge pull request 'feat(shard): the player-vendor marketplace' (#116) from feat/vendor-listing into edge

wtclaude referenced this pull request

2026-07-29 23:07:57 +00:00

feat(shard)!: Protocol 3.0 cutover — visibility framework, spawn atlas, marketplace #118

Sign in to join this conversation.