Files
website/server/src/model/shardMarket/shardMarket.db.js
wtclaude 8771a1cf6c feat(shard): the player-vendor marketplace
Protocol 3.0 §8, the website half. Ingests vendor.listing / vendor.listing.remove
into shard_vendors + shard_vendor_items, serves a searchable public API over
them, and ships /site/market and /site/market/vendors/:serial.

Three things the pages have to say out loud, all consequences of how the data is
gathered:

- The prices are NOT live. The shard sweeps vendors round-robin, so a shop can be
  a full cycle behind. The banner is driven by the OLDEST vendor row, not the
  newest — the one stale shop is the one that wastes somebody's trip.
- A shop can be truncated. `total` exceeding `count` means the shop holds more
  than the shard publishes per frame; the vendor page says "showing 250 of 3,104"
  rather than presenting a partial shop as complete.
- An item may have no name. On a shard with no cliloc table the honest render is
  the item id, never an invented label.

## The pre-wired visibility rules, re-checked

Part A pre-wired market.ownerName and market.location before the frame existed,
and the sibling rule it pre-wired for leaderboards (`characterName`) turned out
to be INERT because projectValue matches literal JSON keys. Both market rules
were checked against the real frame this time:

- `ownerName` is a real key. Kept.
- `location` is a real key ONLY because the frame nests it. Flat map/x/y/region
  would have made the rule match nothing — the same failure, one part later. It
  is nested on the wire and on the read model so one rule hides the facet, the
  coordinates, the region and the house together; five flat keys would be five
  rules that drift apart.
- `ownerSerial` was ADDED. An admin who hides the owner's name and leaves a
  serial that the leaderboards and guild boards resolve back to that same name
  has not hidden anything.

Tests assert all three bite, on the stored read model AND on the raw frame —
the market's SSE stream is off by default but an admin can turn it on, and a rule
that worked on only one path is exactly the leak §3.6.1 records.

## Notable

- **No payload column on shard_vendors**, unlike shard_points_boards next door.
  The board's top-N is a fixed-size list read whole; here the items ARE the
  searchable rows, so they are normalized and nothing is left worth duplicating.
- **display_name is denormalized at ingest** (literal name preferred over the
  cliloc — a player set it, so it is more specific). Resolving at query time
  would put the cliloc table on the hot path and make search-by-name impossible.
  Because the shard's diff sweep will not re-send an unchanged shop just because
  the site learned what its items are called, a cliloc import now triggers a bulk
  re-resolution — 50 ms per thousand rows, never throws.
- **updated_at is written explicitly** on every upsert. MariaDB does not fire ON
  UPDATE CURRENT_TIMESTAMP when every column is written back unchanged, and a
  shop re-published identically is still freshly confirmed — without this the
  staleness banner would age a perfectly current shop forever.
- **LIKE wildcards in `q` are escaped.** `%` and `_` are LIKE metacharacters, not
  SQL ones, so parameterization does not neutralize them: `?q=%` would otherwise
  match every listing on the shard.
- **Rate-limited** (60/min/IP), the only limited public read. Every other public
  GET is an indexed lookup of bounded size; this is a LIKE scan plus a COUNT over
  the largest shard_* table, anonymous by default.
- Reconnect backfill pages /market, bounded by MARKET_SNAPSHOT_MAX = 5000 and
  stopping on a short page as well as on `total`, so a concurrent sweep shrinking
  the index cannot spin the walk.

## How it was tested

673 server tests pass (27 new). Client builds clean; swagger-output.json,
routes.manifest.json and routes.guards.json regenerated.

Verified full-stack against the live MariaDB and a real shard, not only units:

- 27 real vendors / 1,040 listings swept off the ServUO tree, through the Rust
  sidecar, into the site — names resolving through the cliloc table ("longsword",
  "katana"), real facets and regions in the filters.
- `?q=sword` 682, `?q=%` and `?q=_` **0** (the escape), map/region/price/sort
  filters, paging, and the vendor detail route.
- Visibility live: fields gated to staff vanish for an anonymous caller while
  shopName and price survive; audience=player 403s; enabled=0 404s; and
  /shard/features correctly drops `market` so the nav hides it.
- Re-publishing a shop smaller leaves no orphan items; an identical re-publish
  moves updated_at.
- The limiter fires (38x200 then 32x429 on a 70-request burst).

Not covered by an automated test: the two React pages are presentational and this
repo's client suite covers pure-logic modules only. They were driven against the
live API above, but not rendered in a DOM harness.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 09:51:50 -05:00

300 lines
10 KiB
JavaScript

const { pool, query } = require('../../utils/db')
// Raw SQL for the player-vendor market index (Protocol 3.0 vendor.listing).
//
// Two tables, both INGEST-OWNED: `shard_vendors` (one row per shop) and
// `shard_vendor_items` (one row per priced listing). Nothing else in the codebase
// writes to either. No foreign keys, consistent with every other shard_* table.
// Insert batch size for one vendor's listings. A shop is capped at
// MarketMaxListings (250 by default) on the shard side, so in practice this is
// one batch — it exists for the operator who raised that cap.
const BATCH = 500
// LIKE wildcards in user input. `%` and `_` are not special to the parameterized
// query — they are special to LIKE itself — so a search for "50% off" would
// otherwise match everything containing "50" and a search for "_" would match
// every single-character name. Escaped with a backslash, which is MariaDB's
// default LIKE escape (no ESCAPE clause needed).
const likeTerm = (q) => `%${String(q).replace(/[\\%_]/g, (c) => `\\${c}`)}%`
/**
* Replace one vendor's whole row and listing set, in one transaction.
*
* Delete-then-insert rather than a diff, because the frame is AUTHORITATIVE for
* that vendor: the shard's sweep only emits a shop whose contents, prices or
* location moved, and when it does it sends the whole shop. Reconciling it item
* by item would be more code for the same result and would leave sold items
* behind on any path the reconciliation missed.
*
* All-or-nothing matters here for a specific reason: the two writes are "the
* shop" and "what is in it", and a failure between them leaves a shop advertising
* an inventory it no longer has (or none at all) — visibly wrong on the page, and
* indistinguishable from a genuinely empty shop.
*/
async function replaceVendor(vendor, items) {
const conn = await pool.getConnection()
try {
await conn.beginTransaction()
await conn.query(
`INSERT INTO shard_vendors
(serial, shop_name, owner_serial, owner_name, map, x, y, z, region, house,
item_count, item_total, truncated, t)
VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?)
ON DUPLICATE KEY UPDATE shop_name = VALUES(shop_name), owner_serial = VALUES(owner_serial),
owner_name = VALUES(owner_name), map = VALUES(map), x = VALUES(x), y = VALUES(y),
z = VALUES(z), region = VALUES(region), house = VALUES(house),
item_count = VALUES(item_count), item_total = VALUES(item_total),
truncated = VALUES(truncated), t = VALUES(t),
-- Touched explicitly rather than left to ON UPDATE CURRENT_TIMESTAMP:
-- MariaDB does not fire that when every column is written back
-- unchanged, and a shop that is re-published identically is still
-- FRESHLY CONFIRMED. Without this the staleness banner would age a
-- perfectly current shop forever.
updated_at = CURRENT_TIMESTAMP`,
[
vendor.serial,
vendor.shopName ?? null,
vendor.ownerSerial ?? null,
vendor.ownerName ?? null,
vendor.map ?? null,
Number.isFinite(vendor.x) ? vendor.x : null,
Number.isFinite(vendor.y) ? vendor.y : null,
Number.isFinite(vendor.z) ? vendor.z : null,
vendor.region ?? null,
vendor.house ?? null,
items.length,
Number.isFinite(vendor.itemTotal) ? vendor.itemTotal : items.length,
vendor.truncated ? 1 : 0,
Number.isFinite(vendor.t) ? vendor.t : null,
],
)
await conn.query('DELETE FROM shard_vendor_items WHERE vendor_serial = ?', [vendor.serial])
const rows = items.map((i) => [
vendor.serial,
i.serial,
i.itemId,
i.hue,
i.amount,
i.price,
i.name,
i.cliloc,
i.displayName,
i.child ? 1 : 0,
])
for (let i = 0; i < rows.length; i += BATCH) {
await conn.batch(
`INSERT INTO shard_vendor_items
(vendor_serial, serial, item_id, hue, amount, price, name, cliloc, display_name, child)
VALUES (?,?,?,?,?,?,?,?,?,?)`,
rows.slice(i, i + BATCH),
)
}
await conn.commit()
return { items: rows.length }
} catch (err) {
await conn.rollback().catch(() => {})
throw err
} finally {
conn.release()
}
}
/** Drop one vendor and its listings (vendor.listing.remove). */
async function removeVendor(serial) {
const conn = await pool.getConnection()
try {
await conn.beginTransaction()
await conn.query('DELETE FROM shard_vendor_items WHERE vendor_serial = ?', [serial])
await conn.query('DELETE FROM shard_vendors WHERE serial = ?', [serial])
await conn.commit()
} catch (err) {
await conn.rollback().catch(() => {})
throw err
} finally {
conn.release()
}
}
// ── Search ─────────────────────────────────────────────────────────────────
//
// The unit of a search RESULT is a listing, not a vendor: "who sells a vanquishing
// kryss and for how much" is the question, and answering it per vendor would make
// the caller flatten the shops back out. The vendor's columns ride along on the
// join so a result row is self-contained.
function searchWhere({ q, minPrice, maxPrice, itemId, map, region }) {
const where = ['i.price > 0']
const params = []
if (q) {
// Both the resolved display name and the item's own literal, because an item
// with a player-set name (most of what is actually worth searching for on a
// player-run shard) may have a generic cliloc.
where.push('(i.display_name LIKE ? OR i.name LIKE ?)')
params.push(likeTerm(q), likeTerm(q))
}
if (Number.isFinite(minPrice)) {
where.push('i.price >= ?')
params.push(minPrice)
}
if (Number.isFinite(maxPrice)) {
where.push('i.price <= ?')
params.push(maxPrice)
}
if (Number.isFinite(itemId)) {
where.push('i.item_id = ?')
params.push(itemId)
}
if (map) {
where.push('v.map = ?')
params.push(map)
}
if (region) {
where.push('v.region = ?')
params.push(region)
}
return { sql: `WHERE ${where.join(' AND ')}`, params }
}
// Whitelisted, because this interpolates into the statement. `recent` sorts by
// the vendor's freshness, which is the only way to see what has just been listed
// on a shard whose sweep is minutes wide.
const SORTS = {
price_asc: 'i.price ASC, i.id ASC',
price_desc: 'i.price DESC, i.id ASC',
recent: 'v.updated_at DESC, i.id ASC',
}
async function searchListings({ q, minPrice, maxPrice, itemId, map, region, sort, limit, offset }) {
const { sql, params } = searchWhere({ q, minPrice, maxPrice, itemId, map, region })
const order = SORTS[sort] || SORTS.price_asc
const rows = await query(
`SELECT i.serial, i.item_id, i.hue, i.amount, i.price, i.name, i.cliloc, i.display_name, i.child,
v.serial AS vendor_serial, v.shop_name, v.owner_serial, v.owner_name,
v.map, v.x, v.y, v.z, v.region, v.house, v.updated_at
FROM shard_vendor_items i
JOIN shard_vendors v ON v.serial = i.vendor_serial
${sql}
ORDER BY ${order}
LIMIT ? OFFSET ?`,
[...params, limit, offset],
)
const counted = await query(
`SELECT COUNT(*) AS n
FROM shard_vendor_items i
JOIN shard_vendors v ON v.serial = i.vendor_serial
${sql}`,
params,
)
return { rows, total: Number(counted[0]?.n) || 0 }
}
async function getVendor(serial) {
const rows = await query(
`SELECT serial, shop_name, owner_serial, owner_name, map, x, y, z, region, house,
item_count, item_total, truncated, t, updated_at
FROM shard_vendors WHERE serial = ?`,
[serial],
)
return rows[0] || null
}
async function listVendorItems(serial, { limit, offset }) {
return query(
`SELECT serial, item_id, hue, amount, price, name, cliloc, display_name, child
FROM shard_vendor_items
WHERE vendor_serial = ?
ORDER BY price ASC, id ASC
LIMIT ? OFFSET ?`,
[serial, limit, offset],
)
}
/**
* What the market page's header needs: how big the index is, and how stale it may
* be. `staleAt` is the OLDEST vendor row — the round-robin sweep means a shop can
* be a full cycle behind, and the page says so rather than implying live prices.
*/
async function meta() {
const rows = await query(
`SELECT COUNT(*) AS vendors, MIN(updated_at) AS stale_at, MAX(updated_at) AS fresh_at
FROM shard_vendors`,
)
const items = await query('SELECT COUNT(*) AS n FROM shard_vendor_items')
return {
vendors: Number(rows[0]?.vendors) || 0,
items: Number(items[0]?.n) || 0,
staleAt: rows[0]?.stale_at || null,
freshAt: rows[0]?.fresh_at || null,
}
}
/** The distinct facets and regions holding vendors — drives the page's filters. */
async function listPlaces() {
const maps = await query(
'SELECT DISTINCT map FROM shard_vendors WHERE map IS NOT NULL ORDER BY map',
)
const regions = await query(
'SELECT DISTINCT region FROM shard_vendors WHERE region IS NOT NULL ORDER BY region',
)
return { maps: maps.map((r) => r.map), regions: regions.map((r) => r.region) }
}
// ── Cliloc re-resolution ───────────────────────────────────────────────────
/**
* One page of listings whose name still needs resolving, for the bulk pass that
* runs after a cliloc import.
*
* Keyed on `id > after` rather than OFFSET: the pass updates the very rows it is
* scanning, and an OFFSET walk over a table being rewritten skips rows. Every
* row with a cliloc is re-read, not just the unresolved ones, because an import
* can also CHANGE a name — a shard overlay relabelling a stock item is the whole
* reason overlays exist.
*/
async function listResolvableItems(after, limit) {
return query(
`SELECT id, cliloc, name, display_name
FROM shard_vendor_items
WHERE cliloc IS NOT NULL AND cliloc > 0 AND id > ?
ORDER BY id
LIMIT ?`,
[after, limit],
)
}
/** Write back a batch of re-resolved display names. */
async function updateDisplayNames(pairs) {
if (pairs.length === 0) return 0
const conn = await pool.getConnection()
try {
await conn.batch('UPDATE shard_vendor_items SET display_name = ? WHERE id = ?', pairs)
return pairs.length
} finally {
conn.release()
}
}
module.exports = {
replaceVendor,
removeVendor,
searchListings,
getVendor,
listVendorItems,
meta,
listPlaces,
listResolvableItems,
updateDisplayNames,
likeTerm,
}