RunicGateway/website
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Activity

feat(shard): the player-vendor marketplace

Browse Source 
Protocol 3.0 §8, the website half. Ingests vendor.listing / vendor.listing.remove
into shard_vendors + shard_vendor_items, serves a searchable public API over
them, and ships /site/market and /site/market/vendors/:serial.

Three things the pages have to say out loud, all consequences of how the data is
gathered:

- The prices are NOT live. The shard sweeps vendors round-robin, so a shop can be
  a full cycle behind. The banner is driven by the OLDEST vendor row, not the
  newest — the one stale shop is the one that wastes somebody's trip.
- A shop can be truncated. `total` exceeding `count` means the shop holds more
  than the shard publishes per frame; the vendor page says "showing 250 of 3,104"
  rather than presenting a partial shop as complete.
- An item may have no name. On a shard with no cliloc table the honest render is
  the item id, never an invented label.

## The pre-wired visibility rules, re-checked

Part A pre-wired market.ownerName and market.location before the frame existed,
and the sibling rule it pre-wired for leaderboards (`characterName`) turned out
to be INERT because projectValue matches literal JSON keys. Both market rules
were checked against the real frame this time:

- `ownerName` is a real key. Kept.
- `location` is a real key ONLY because the frame nests it. Flat map/x/y/region
  would have made the rule match nothing — the same failure, one part later. It
  is nested on the wire and on the read model so one rule hides the facet, the
  coordinates, the region and the house together; five flat keys would be five
  rules that drift apart.
- `ownerSerial` was ADDED. An admin who hides the owner's name and leaves a
  serial that the leaderboards and guild boards resolve back to that same name
  has not hidden anything.

Tests assert all three bite, on the stored read model AND on the raw frame —
the market's SSE stream is off by default but an admin can turn it on, and a rule
that worked on only one path is exactly the leak §3.6.1 records.

## Notable

- **No payload column on shard_vendors**, unlike shard_points_boards next door.
  The board's top-N is a fixed-size list read whole; here the items ARE the
  searchable rows, so they are normalized and nothing is left worth duplicating.
- **display_name is denormalized at ingest** (literal name preferred over the
  cliloc — a player set it, so it is more specific). Resolving at query time
  would put the cliloc table on the hot path and make search-by-name impossible.
  Because the shard's diff sweep will not re-send an unchanged shop just because
  the site learned what its items are called, a cliloc import now triggers a bulk
  re-resolution — 50 ms per thousand rows, never throws.
- **updated_at is written explicitly** on every upsert. MariaDB does not fire ON
  UPDATE CURRENT_TIMESTAMP when every column is written back unchanged, and a
  shop re-published identically is still freshly confirmed — without this the
  staleness banner would age a perfectly current shop forever.
- **LIKE wildcards in `q` are escaped.** `%` and `_` are LIKE metacharacters, not
  SQL ones, so parameterization does not neutralize them: `?q=%` would otherwise
  match every listing on the shard.
- **Rate-limited** (60/min/IP), the only limited public read. Every other public
  GET is an indexed lookup of bounded size; this is a LIKE scan plus a COUNT over
  the largest shard_* table, anonymous by default.
- Reconnect backfill pages /market, bounded by MARKET_SNAPSHOT_MAX = 5000 and
  stopping on a short page as well as on `total`, so a concurrent sweep shrinking
  the index cannot spin the walk.

## How it was tested

673 server tests pass (27 new). Client builds clean; swagger-output.json,
routes.manifest.json and routes.guards.json regenerated.

Verified full-stack against the live MariaDB and a real shard, not only units:

- 27 real vendors / 1,040 listings swept off the ServUO tree, through the Rust
  sidecar, into the site — names resolving through the cliloc table ("longsword",
  "katana"), real facets and regions in the filters.
- `?q=sword` 682, `?q=%` and `?q=_` **0** (the escape), map/region/price/sort
  filters, paging, and the vendor detail route.
- Visibility live: fields gated to staff vanish for an anonymous caller while
  shopName and price survive; audience=player 403s; enabled=0 404s; and
  /shard/features correctly drops `market` so the nav hides it.
- Re-publishing a shop smaller leaves no orphan items; an identical re-publish
  moves updated_at.
- The limiter fires (38x200 then 32x429 on a 70-request burst).

Not covered by an automated test: the two React pages are presentational and this
repo's client suite covers pure-logic modules only. They were driven against the
live API above, but not rendered in a DOM harness.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
claude
2026-07-29 09:51:50 -05:00
parent 8da658f223
commit 8771a1cf6c
24 changed files with 2951 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
server/src/router/v1/admin/shardClilocs.controller.js
View File

@@ -17,6 +17,7 @@
// to be told which file to convert.
const clilocs = require('../../../model/shardClilocs/shardClilocs.model')
const market = require('../../../model/shardMarket/shardMarket.model')
const activity = require('../../../model/activity/activity.model')
const log = require('../../../utils/logger')('admin-shard-clilocs')
@@ -50,6 +51,17 @@ async function importClilocs(req, res) {
const force = !!req.body?.force
const approve = !!req.body?.approve
const result = await clilocs.refresh({ force, approve })
// The marketplace denormalizes resolved item names into
// shard_vendor_items.display_name, and the shard's market sweep will NOT
// re-send an unchanged shop just because the site learned what its items are
// called — so without this pass, an operator who imports clilocs after the
// first sweep keeps seeing item ids until every shop happens to change.
// Awaited (rather than fired and forgotten) so the panel's "imported" is
// honest about the names being live; the pass is a bounded walk of one table
// and never throws.
if (result.status === 'imported') await market.refreshDisplayNames()
await activity.log({
req,
action: 'shard.clilocs.import',

82
server/src/router/v1/public/shard.controller.js
View File

@@ -11,6 +11,7 @@
const shardEvents = require('../../../model/shardEvents/shardEvents.model')
const shardState = require('../../../model/shardState/shardState.model')
const shardMarket = require('../../../model/shardMarket/shardMarket.model')
const uoLinkConfig = require('../../../model/uoLinkConfig/uoLinkConfig.model')
const broadcast = require('../../../utils/shardBroadcast')
const visibility = require('../../../utils/shardVisibility')
@@ -299,6 +300,84 @@ async function getPointsBoard(req, res) {
}
}
// ── Marketplace (Protocol 3.0 vendor.listing) ──────────────────────────────
//
// The shard-wide player-vendor index. Served entirely from our own tables — the
// sidecar is never touched on this path — so shops stay browsable while the shard
// is down, labelled with how stale they may be.
//
// The staleness label is not decoration. The shard sweeps vendors round-robin, so
// a shop can legitimately be a full cycle behind; a page that implied live prices
// would send people to a vendor whose item sold twenty minutes ago.
// The serial spelling the bridge uses everywhere: "0x" and hex. Constrained
// before it reaches the model, like SYSTEM_RE above.
const SERIAL_RE = /^0x[0-9A-Fa-f]{1,16}$/
const intParam = (value) => {
const n = Number.parseInt(value, 10)
return Number.isFinite(n) ? n : undefined
}
// GET /public/shard/market — search the index.
//
// Returns LISTINGS, not vendors: "who sells a vanquishing kryss and for how much"
// is the question, and a vendor-shaped result would make every caller flatten the
// shops back out.
async function getMarket(req, res) {
try {
const page = await shardMarket.search({
q: typeof req.query.q === 'string' ? req.query.q : '',
minPrice: intParam(req.query.minPrice),
maxPrice: intParam(req.query.maxPrice),
itemId: intParam(req.query.itemId),
map: typeof req.query.map === 'string' ? req.query.map : '',
region: typeof req.query.region === 'string' ? req.query.region : '',
sort: typeof req.query.sort === 'string' ? req.query.sort : 'price_asc',
limit: intParam(req.query.limit) ?? 50,
offset: intParam(req.query.offset) ?? 0,
})
return res.json(await visibility.project('market', page, req))
} catch (err) {
log.error('shard.getMarket', err)
return res.status(500).json({ message: 'Internal Server Error' })
}
}
// GET /public/shard/market/meta — index size, staleness, and the filter options
// (which facets and regions actually hold vendors). Separate from the search so
// the page can build its filters without running a query it will throw away.
async function getMarketMeta(req, res) {
try {
return res.json(await visibility.project('market', await shardMarket.meta(), req))
} catch (err) {
log.error('shard.getMarketMeta', err)
return res.status(500).json({ message: 'Internal Server Error' })
}
}
// GET /public/shard/market/vendors/:serial — one shop and its listings.
//
// 404 for a serial the index has never seen, which also covers a vendor that has
// since been dismissed or hidden: to an anonymous caller "no such shop" is the
// only honest answer, and distinguishing the two would leak that a vendor exists
// but was hidden.
async function getMarketVendor(req, res) {
const { serial } = req.params
if (!SERIAL_RE.test(serial)) return res.status(400).json({ message: 'Invalid vendor serial.' })
try {
const vendor = await shardMarket.getVendor(serial, {
limit: intParam(req.query.limit) ?? 250,
offset: intParam(req.query.offset) ?? 0,
})
if (!vendor) return res.status(404).json({ message: 'Unknown vendor.' })
return res.json(await visibility.project('market', vendor, req))
} catch (err) {
log.error('shard.getMarketVendor', err)
return res.status(500).json({ message: 'Internal Server Error' })
}
}
// GET /public/shard/features — the shard features THIS caller can actually see,
// so the SPA (and the Android client) can hide nav entries instead of rendering
// links that 403. Deliberately reports only what the viewer may reach: the list
@@ -335,6 +414,9 @@ module.exports = {
getRuleset,
getPointsBoards,
getPointsBoard,
getMarket,
getMarketMeta,
getMarketVendor,
getFeatures,
stream,
}

66
server/src/router/v1/public/shard.router.js
View File

@@ -25,6 +25,7 @@ const { param, query } = require('express-validator')
const shard = require('./shard.controller')
const validate = require('../../../middleware/validate')
const { marketLimiter } = require('../../../middleware/rateLimit')
const { requireFeature } = require('../../../utils/shardVisibility')
const shardRouter = express.Router()
@@ -167,6 +168,71 @@ shardRouter.get(
/* #swagger.responses[404] = { description: 'The shard has never published that system' } */
shard.getPointsBoard,
)
// ── Marketplace ────────────────────────────────────────────────────────────
//
// Rate-limited, unlike every other route in this file. These are the first
// genuinely expensive PUBLIC reads on the site — a LIKE scan plus a COUNT over
// what is typically the largest shard_* table, reachable with no session.
shardRouter.get(
'/market',
requireFeature('market'),
marketLimiter,
// #swagger.tags = ['Public · Shard']
// #swagger.summary = 'Search the player-vendor marketplace'
// #swagger.description = 'Every priced listing on every player vendor the shard publishes — the same index the in-game Vendor Search gump reads, and it honours the same per-vendor opt-out, so a player who hid their shop in game is hidden here too. Results are LISTINGS, each carrying enough of its shop to be actionable. Served from the site\'s own tables (the sidecar is not touched), so it renders while the shard is down; `staleAt` is the oldest vendor row and the page must say how far behind the index can be — the shard sweeps vendors round-robin, so prices are inherently up to one full cycle old. Item names are resolved server-side against the cliloc table (docs/website/CLILOCS.md); on a shard that has not configured one, `displayName` is null and clients render the item id.'
// #swagger.parameters['q'] = { in: 'query', required: false, schema: { type: 'string' }, description: 'Substring match on the resolved item name or the item\'s own literal name (max 60 chars).' }
// #swagger.parameters['minPrice'] = { in: 'query', required: false, schema: { type: 'integer' }, description: 'Lowest price to include.' }
// #swagger.parameters['maxPrice'] = { in: 'query', required: false, schema: { type: 'integer' }, description: 'Highest price to include.' }
// #swagger.parameters['itemId'] = { in: 'query', required: false, schema: { type: 'integer' }, description: 'Exact ItemID (art id) match, for "more like this".' }
// #swagger.parameters['map'] = { in: 'query', required: false, schema: { type: 'string' }, description: 'Limit to one facet. Facet names come from the shard\'s own data; an unknown one returns an empty page.' }
// #swagger.parameters['region'] = { in: 'query', required: false, schema: { type: 'string' }, description: 'Limit to one named region.' }
// #swagger.parameters['sort'] = { in: 'query', required: false, schema: { type: 'string', enum: ['price_asc','price_desc','recent'] }, description: 'Default price_asc. `recent` orders by when the shop was last seen.' }
// #swagger.parameters['limit'] = { in: 'query', required: false, schema: { type: 'integer' }, description: 'Page size, 1..100 (default 50).' }
// #swagger.parameters['offset'] = { in: 'query', required: false, schema: { type: 'integer' }, description: 'Rows to skip (default 0).' }
/* #swagger.responses[200] = { description: 'A page of listings plus the unpaginated total and the staleness stamp', content: { "application/json": { schema: { $ref: "#/components/schemas/ShardMarketPage" } } } } */
/* #swagger.responses[403] = { description: 'The market feature is gated above this caller', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
/* #swagger.responses[404] = { description: 'The market feature is disabled', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
/* #swagger.responses[429] = { description: 'Rate limited', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
query('q').optional({ values: 'falsy' }).isString().isLength({ max: 60 }),
query('minPrice').optional({ values: 'falsy' }).isInt({ min: 0, max: 999999999 }),
query('maxPrice').optional({ values: 'falsy' }).isInt({ min: 0, max: 999999999 }),
query('itemId').optional({ values: 'falsy' }).isInt({ min: 0, max: 65535 }),
query('map').optional({ values: 'falsy' }).isString().isLength({ max: 40 }),
query('region').optional({ values: 'falsy' }).isString().isLength({ max: 80 }),
query('sort').optional({ values: 'falsy' }).isIn(['price_asc', 'price_desc', 'recent']),
query('limit').optional().isInt({ min: 1, max: 100 }),
query('offset').optional().isInt({ min: 0, max: 100000 }),
validate,
shard.getMarket,
)
shardRouter.get(
'/market/meta',
requireFeature('market'),
// #swagger.tags = ['Public · Shard']
// #swagger.summary = 'Marketplace size, staleness and filter options'
// #swagger.description = 'How many vendors and listings the index holds, how stale it may be (`staleAt` = the oldest vendor row, `freshAt` = the newest), and which facets and regions actually hold vendors — so a client can build its filters without running a search it will discard.'
/* #swagger.responses[200] = { description: 'Marketplace metadata', content: { "application/json": { schema: { $ref: "#/components/schemas/ShardMarketMeta" } } } } */
shard.getMarketMeta,
)
shardRouter.get(
'/market/vendors/:serial',
requireFeature('market'),
marketLimiter,
// #swagger.tags = ['Public · Shard']
// #swagger.summary = 'One player vendor and everything it is selling'
// #swagger.description = 'A single shop by its vendor serial, with its listings. `truncated` (and `total` exceeding `count`) means the shop holds more than the shard publishes per frame — a commodity reseller with thousands of stacks is a real thing, and the page says so rather than presenting a partial shop as complete. Returns 404 for a serial the index has never seen, which also covers a vendor since dismissed or hidden.'
/* #swagger.parameters['serial'] = { in: 'path', required: true, description: 'Vendor serial, e.g. 0x40001234', schema: { type: 'string' } } */
// #swagger.parameters['limit'] = { in: 'query', required: false, schema: { type: 'integer' }, description: 'Listings to return, 1..500 (default 250).' }
// #swagger.parameters['offset'] = { in: 'query', required: false, schema: { type: 'integer' }, description: 'Listings to skip (default 0).' }
/* #swagger.responses[200] = { description: 'The vendor', content: { "application/json": { schema: { $ref: "#/components/schemas/ShardMarketVendor" } } } } */
/* #swagger.responses[400] = { description: 'Malformed vendor serial' } */
/* #swagger.responses[404] = { description: 'No such vendor in the index' } */
param('serial').isString().isLength({ max: 20 }),
query('limit').optional().isInt({ min: 1, max: 500 }),
query('offset').optional().isInt({ min: 0, max: 100000 }),
validate,
shard.getMarketVendor,
)
shardRouter.get(
'/features',
// #swagger.tags = ['Public · Shard']