feat(shard): the player-vendor marketplace

Protocol 3.0 §8, the website half. Ingests vendor.listing / vendor.listing.remove into shard_vendors + shard_vendor_items, serves a searchable public API over them, and ships /site/market and /site/market/vendors/:serial. Three things the pages have to say out loud, all consequences of how the data is gathered: - The prices are NOT live. The shard sweeps vendors round-robin, so a shop can be a full cycle behind. The banner is driven by the OLDEST vendor row, not the newest — the one stale shop is the one that wastes somebody's trip. - A shop can be truncated. `total` exceeding `count` means the shop holds more than the shard publishes per frame; the vendor page says "showing 250 of 3,104" rather than presenting a partial shop as complete. - An item may have no name. On a shard with no cliloc table the honest render is the item id, never an invented label. ## The pre-wired visibility rules, re-checked Part A pre-wired market.ownerName and market.location before the frame existed, and the sibling rule it pre-wired for leaderboards (`characterName`) turned out to be INERT because projectValue matches literal JSON keys. Both market rules were checked against the real frame this time: - `ownerName` is a real key. Kept. - `location` is a real key ONLY because the frame nests it. Flat map/x/y/region would have made the rule match nothing — the same failure, one part later. It is nested on the wire and on the read model so one rule hides the facet, the coordinates, the region and the house together; five flat keys would be five rules that drift apart. - `ownerSerial` was ADDED. An admin who hides the owner's name and leaves a serial that the leaderboards and guild boards resolve back to that same name has not hidden anything. Tests assert all three bite, on the stored read model AND on the raw frame — the market's SSE stream is off by default but an admin can turn it on, and a rule that worked on only one path is exactly the leak §3.6.1 records. ## Notable - **No payload column on shard_vendors**, unlike shard_points_boards next door. The board's top-N is a fixed-size list read whole; here the items ARE the searchable rows, so they are normalized and nothing is left worth duplicating. - **display_name is denormalized at ingest** (literal name preferred over the cliloc — a player set it, so it is more specific). Resolving at query time would put the cliloc table on the hot path and make search-by-name impossible. Because the shard's diff sweep will not re-send an unchanged shop just because the site learned what its items are called, a cliloc import now triggers a bulk re-resolution — 50 ms per thousand rows, never throws. - **updated_at is written explicitly** on every upsert. MariaDB does not fire ON UPDATE CURRENT_TIMESTAMP when every column is written back unchanged, and a shop re-published identically is still freshly confirmed — without this the staleness banner would age a perfectly current shop forever. - **LIKE wildcards in `q` are escaped.** `%` and `_` are LIKE metacharacters, not SQL ones, so parameterization does not neutralize them: `?q=%` would otherwise match every listing on the shard. - **Rate-limited** (60/min/IP), the only limited public read. Every other public GET is an indexed lookup of bounded size; this is a LIKE scan plus a COUNT over the largest shard_* table, anonymous by default. - Reconnect backfill pages /market, bounded by MARKET_SNAPSHOT_MAX = 5000 and stopping on a short page as well as on `total`, so a concurrent sweep shrinking the index cannot spin the walk. ## How it was tested 673 server tests pass (27 new). Client builds clean; swagger-output.json, routes.manifest.json and routes.guards.json regenerated. Verified full-stack against the live MariaDB and a real shard, not only units: - 27 real vendors / 1,040 listings swept off the ServUO tree, through the Rust sidecar, into the site — names resolving through the cliloc table ("longsword", "katana"), real facets and regions in the filters. - `?q=sword` 682, `?q=%` and `?q=_` **0** (the escape), map/region/price/sort filters, paging, and the vendor detail route. - Visibility live: fields gated to staff vanish for an anonymous caller while shopName and price survive; audience=player 403s; enabled=0 404s; and /shard/features correctly drops `market` so the nav hides it. - Re-publishing a shop smaller leaves no orphan items; an identical re-publish moves updated_at. - The limiter fires (38x200 then 32x429 on a 70-request burst). Not covered by an automated test: the two React pages are presentational and this repo's client suite covers pure-logic modules only. They were driven against the live API above, but not rendered in a DOM harness. Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 09:51:50 -05:00
parent 8da658f223
commit 8771a1cf6c
24 changed files with 2951 additions and 2 deletions
--- a/server/test/shardIngest.market.test.js
+++ b/server/test/shardIngest.market.test.js
@@ -0,0 +1,114 @@
+const { test, beforeEach } = require('node:test')
+const assert = require('node:assert/strict')
+
+const shardIngest = require('../src/utils/shardIngest')
+
+// Protocol 3.0 vendor.listing / vendor.listing.remove routing. Same shape as
+// shardIngest.points.test.js: stubbed deps, asserting where the dispatcher sends
+// the frame and whether it is appended to the event log.
+function makeDeps() {
+  const calls = { upserts: [], removes: [], appended: [], broadcast: [] }
+  const noop = async () => {}
+  return {
+    calls,
+    shardEvents: { append: async (row) => { calls.appended.push(row); return true } },
+    shardState: {
+      // Present so any stray routing is a harmless no-op rather than a crash.
+      clearOnline: noop, upsertOnline: noop, setOffline: noop, upsertHouse: noop,
+      addEconomySample: noop, setRuleset: noop, upsertPointsBoard: noop,
+    },
+    shardMarket: {
+      upsertVendor: async (ev) => { calls.upserts.push(ev) },
+      removeVendor: async (serial) => { calls.removes.push(serial) },
+    },
+    shardLinks: { removeByAccount: noop },
+    uoLinkConfig: { recordStatus: noop },
+    broadcast: (ev) => { calls.broadcast.push(ev) },
+    pushDispatch: async () => {},
+    log: { warn() {}, info() {}, error() {} },
+  }
+}
+
+const FRAME = {
+  kind: 'vendor.listing',
+  t: 1000,
+  serial: '0x40001234',
+  shopName: "Darrow's Bargains",
+  ownerSerial: '0x1A2B',
+  ownerName: 'Darrow',
+  location: { map: 'Trammel', x: 1421, y: 1699, z: 0, region: 'Britain', house: "Darrow's Villa" },
+  count: 2,
+  total: 2,
+  truncated: false,
+  items: [
+    { serial: '0x40012ABC', itemId: 3922, hue: 0, amount: 1, price: 25000, name: null, cliloc: 1023721 },
+    { serial: '0x40012ABD', itemId: 7026, hue: 1157, amount: 3, price: 500, name: 'a shard sigil', cliloc: 1041243 },
+  ],
+}
+
+beforeEach(() => shardIngest.reset())
+
+test('vendor.listing routes to the market model with the whole frame', async () => {
+  const deps = makeDeps()
+  await shardIngest.ingest(FRAME, deps)
+  assert.equal(deps.calls.upserts.length, 1)
+  const stored = deps.calls.upserts[0]
+  assert.equal(stored.serial, '0x40001234')
+  assert.equal(stored.location.region, 'Britain')
+  assert.equal(stored.items.length, 2)
+})
+
+test('vendor.listing.remove routes to removeVendor with the serial', async () => {
+  const deps = makeDeps()
+  await shardIngest.ingest({ kind: 'vendor.listing.remove', t: 2000, serial: '0x40001234' }, deps)
+  assert.deepEqual(deps.calls.removes, ['0x40001234'])
+  assert.equal(deps.calls.upserts.length, 0)
+})
+
+// The market IS the state. One frame carries up to 250 listings and the sweep
+// re-emits a shop on any price change, so logging would turn shard_events into a
+// price history nobody reads — the strongest case of the three v3 kinds.
+test('neither market kind is appended to the event log', async () => {
+  const deps = makeDeps()
+  const a = await shardIngest.ingest(FRAME, deps)
+  const b = await shardIngest.ingest({ kind: 'vendor.listing.remove', serial: '0x40001234' }, deps)
+  assert.equal(a.logged, false)
+  assert.equal(b.logged, false)
+  assert.equal(deps.calls.appended.length, 0)
+  assert.equal(shardIngest.LOGGED_KINDS.has('vendor.listing'), false)
+  assert.equal(shardIngest.LOGGED_KINDS.has('vendor.listing.remove'), false)
+})
+
+// Broadcast is unconditional at this layer — whether it actually reaches anyone
+// is shardBroadcast's call, and the market feature ships with its stream off.
+test('vendor.listing is handed to the broadcaster', async () => {
+  const deps = makeDeps()
+  await shardIngest.ingest(FRAME, deps)
+  assert.equal(deps.calls.broadcast.length, 1)
+  assert.equal(deps.calls.broadcast[0].kind, 'vendor.listing')
+})
+
+test('a backfilled vendor.listing still stores but does not broadcast', async () => {
+  const deps = makeDeps()
+  await shardIngest.ingest(FRAME, { ...deps, fromBackfill: true })
+  assert.equal(deps.calls.upserts.length, 1)
+  assert.equal(deps.calls.broadcast.length, 0)
+})
+
+// The reconnect backfill replays the whole index through this path, so a single
+// bad vendor must not abort it.
+test('an upsertVendor failure does not throw or stop the broadcast', async () => {
+  const deps = makeDeps()
+  deps.shardMarket.upsertVendor = async () => { throw new Error('db down') }
+  const r = await shardIngest.ingest(FRAME, deps)
+  assert.equal(r.logged, false)
+  assert.equal(deps.calls.broadcast.length, 1)
+})
+
+// Each vendor is its own row; the frame is authoritative for that vendor only.
+test('two vendors are stored independently', async () => {
+  const deps = makeDeps()
+  await shardIngest.ingest(FRAME, deps)
+  await shardIngest.ingest({ ...FRAME, serial: '0x40009999', shopName: 'Second Shop' }, deps)
+  assert.deepEqual(deps.calls.upserts.map((v) => v.serial), ['0x40001234', '0x40009999'])
+})
--- a/server/test/shardMarket.model.test.js
+++ b/server/test/shardMarket.model.test.js
@@ -0,0 +1,260 @@
+// Point the DB at a closed port BEFORE requiring anything that builds a pool.
+// Nothing here reaches the database: these are the model's PURE parts — the
+// flatten/shape rules the frame passes through on the way in and out — plus the
+// visibility projection over the shapes they produce.
+process.env.DB_HOST = '127.0.0.1'
+process.env.DB_PORT = '59999'
+
+const { test, after } = require('node:test')
+const assert = require('node:assert/strict')
+
+const market = require('../src/model/shardMarket/shardMarket.model')
+const clilocs = require('../src/model/shardClilocs/shardClilocs.model')
+const clilocDb = require('../src/model/shardClilocs/shardClilocs.db')
+const visibility = require('../src/utils/shardVisibility')
+const db = require('../src/utils/db')
+
+after(() => db.close())
+
+// Stand in for the cliloc table. Without this each unresolved lookup waits out
+// the pool's 10s acquire timeout against the dead port — the model swallows the
+// failure exactly as it would in production (an operator who never converted a
+// cliloc file is in a supported state), so the RESULT is the same either way;
+// this only stops the suite spending half a minute proving it.
+const TABLE = new Map([[1023721, 'quarter staff']])
+clilocDb.lookup = async (numbers) =>
+  numbers.filter((n) => TABLE.has(n)).map((n) => ({ number: n, text: TABLE.get(n) }))
+
+const FRAME = {
+  kind: 'vendor.listing',
+  t: 1000,
+  serial: '0x40001234',
+  shopName: "Darrow's Bargains",
+  ownerSerial: '0x1A2B',
+  ownerName: 'Darrow',
+  location: { map: 'Trammel', x: 1421, y: 1699, z: 0, region: 'Britain', house: "Darrow's Villa" },
+  count: 2,
+  total: 2,
+  truncated: false,
+  items: [],
+}
+
+// ── flattenFrame ───────────────────────────────────────────────────────────
+
+test('flattenFrame lifts the nested location into columns', () => {
+  const v = market.flattenFrame(FRAME)
+  assert.equal(v.serial, '0x40001234')
+  assert.equal(v.map, 'Trammel')
+  assert.equal(v.x, 1421)
+  assert.equal(v.region, 'Britain')
+  assert.equal(v.house, "Darrow's Villa")
+})
+
+// A vendor standing in the street has no house, and a frame from an older plugin
+// may have no location at all. Neither is an error.
+test('flattenFrame tolerates a missing location entirely', () => {
+  const v = market.flattenFrame({ serial: '0x1', shopName: null })
+  assert.equal(v.map, null)
+  assert.equal(v.x, null)
+  assert.equal(v.region, null)
+  assert.equal(v.house, null)
+})
+
+// `total` is what the SHOP holds; `count` is what the frame carried. A truncated
+// shop must not report its published slice as its size, or the page says
+// "showing 250 of 250" for a vendor holding three thousand stacks.
+test('flattenFrame keeps the shop total separate from the published count', () => {
+  const v = market.flattenFrame({ ...FRAME, count: 250, total: 3104, truncated: true })
+  assert.equal(v.itemTotal, 3104)
+  assert.equal(v.truncated, true)
+})
+
+// An older plugin sends no `total`. Falling back to `count` is right — it is the
+// only number available and it is correct whenever nothing was truncated.
+test('flattenFrame falls back to count when total is absent', () => {
+  const v = market.flattenFrame({ ...FRAME, count: 7, total: undefined })
+  assert.equal(v.itemTotal, 7)
+})
+
+test('flattenFrame clips over-length strings rather than letting the insert fail', () => {
+  const v = market.flattenFrame({ ...FRAME, ownerName: 'x'.repeat(200) })
+  assert.equal(v.ownerName.length, 64)
+})
+
+// ── shapeItems ─────────────────────────────────────────────────────────────
+//
+// resolveMany never throws and, with no cliloc table reachable, resolves nothing
+// — which is exactly the state of a shard whose operator never converted one, so
+// these run against the real function rather than a stub.
+
+test('shapeItems prefers the item\'s literal name over its cliloc', async () => {
+  const items = await market.shapeItems({
+    items: [{ serial: '0x1', itemId: 3922, price: 100, name: 'a shard sigil', cliloc: 1023721 }],
+  })
+  assert.equal(items[0].displayName, 'a shard sigil')
+  // The cliloc is kept regardless, so a later import can still re-resolve it.
+  assert.equal(items[0].cliloc, 1023721)
+})
+
+test('shapeItems resolves the cliloc when the item has no literal name', async () => {
+  const items = await market.shapeItems({
+    items: [{ serial: '0x1', itemId: 3922, price: 100, name: null, cliloc: 1023721 }],
+  })
+  assert.equal(items[0].displayName, 'quarter staff')
+})
+
+// The supported state for a shard whose operator never converted a cliloc file:
+// no name, not a fabricated one. Clients render the item id, exactly as they did
+// before the table existed.
+test('shapeItems leaves displayName null for an unknown cliloc', async () => {
+  const items = await market.shapeItems({
+    items: [{ serial: '0x1', itemId: 3922, price: 100, name: null, cliloc: 9999999 }],
+  })
+  assert.equal(items[0].displayName, null)
+})
+
+// Unpriced rows are inventory, not listings. The shard drops them too; enforcing
+// it here as well means a plugin that stops doing so cannot put un-buyable rows
+// on the market page.
+test('shapeItems drops unpriced listings', async () => {
+  const items = await market.shapeItems({
+    items: [
+      { serial: '0x1', itemId: 1, price: 0 },
+      { serial: '0x2', itemId: 2, price: -1 },
+      { serial: '0x3', itemId: 3, price: 5 },
+    ],
+  })
+  assert.deepEqual(items.map((i) => i.serial), ['0x3'])
+})
+
+test('shapeItems caps a pathological frame', async () => {
+  const many = Array.from({ length: market.MAX_ITEMS_PER_VENDOR + 50 }, (_, i) => ({
+    serial: `0x${i}`,
+    itemId: 1,
+    price: 1,
+  }))
+  const items = await market.shapeItems({ items: many })
+  assert.equal(items.length, market.MAX_ITEMS_PER_VENDOR)
+})
+
+test('shapeItems tolerates a frame with no items array', async () => {
+  assert.deepEqual(await market.shapeItems({}), [])
+})
+
+// ── Visibility projection ──────────────────────────────────────────────────
+//
+// The regression that matters. Part A pre-wired `market.ownerName` and
+// `market.location` before the frame existed, and the sibling rule it pre-wired
+// for leaderboards (`characterName`) turned out to be INERT because projectValue
+// matches literal JSON keys. These assert the market rules actually bite — on the
+// read model AND on the wire frame, which is why both carry the same key names.
+
+const config = visibility.compileDefaults()
+
+const listing = market.shapeListing({
+  serial: '0x40012ABC',
+  item_id: 3922,
+  hue: 0,
+  amount: 1,
+  price: 25000,
+  name: null,
+  cliloc: 1023721,
+  display_name: 'quarter staff',
+  child: 0,
+  vendor_serial: '0x40001234',
+  shop_name: "Darrow's Bargains",
+  owner_serial: '0x1A2B',
+  owner_name: 'Darrow',
+  map: 'Trammel',
+  x: 1421,
+  y: 1699,
+  z: 0,
+  region: 'Britain',
+  house: "Darrow's Villa",
+  updated_at: new Date(0),
+})
+
+test('market defaults expose owner and location (they are already public in game)', () => {
+  const out = visibility.projectFeature('market', listing, 'anonymous', config)
+  assert.equal(out.vendor.ownerName, 'Darrow')
+  assert.equal(out.vendor.location.region, 'Britain')
+})
+
+test('tightening market.ownerName hides it from below that rung', () => {
+  const tightened = { ...config, market: { ...config.market, fields: { ...config.market.fields, ownerName: 'staff' } } }
+  const anon = visibility.projectFeature('market', listing, 'anonymous', tightened)
+  const staff = visibility.projectFeature('market', listing, 'staff', tightened)
+  assert.equal('ownerName' in anon.vendor, false)
+  assert.equal(staff.vendor.ownerName, 'Darrow')
+  // The shop name is a separate field and must survive — hiding the owner is not
+  // the same as hiding the shop.
+  assert.equal(anon.vendor.shopName, "Darrow's Bargains")
+})
+
+// The whole reason `location` is one nested object: a single rule has to take the
+// facet, the coordinates, the region and the house together. Five flat keys would
+// be five rules that drift apart.
+test('tightening market.location hides the whole location object at once', () => {
+  const tightened = { ...config, market: { ...config.market, fields: { ...config.market.fields, location: 'player' } } }
+  const anon = visibility.projectFeature('market', listing, 'anonymous', tightened)
+  const player = visibility.projectFeature('market', listing, 'player', tightened)
+  assert.equal('location' in anon.vendor, false)
+  assert.equal(player.vendor.location.map, 'Trammel')
+})
+
+// The same rules must bite on the LIVE frame, not just the stored read model —
+// the market's SSE stream is off by default but an admin can turn it on, and a
+// field rule that only worked on one of the two paths is exactly the leak §3.6.1
+// records.
+test('the same rules apply to the raw vendor.listing frame', () => {
+  const tightened = { ...config, market: { ...config.market, fields: { ...config.market.fields, ownerName: 'admin', location: 'admin' } } }
+  const out = visibility.projectFeature('market', FRAME, 'anonymous', tightened)
+  assert.equal('ownerName' in out, false)
+  assert.equal('location' in out, false)
+  assert.equal(out.shopName, "Darrow's Bargains")
+})
+
+// Rule 1 is not configurable and does not depend on the market rules at all: a
+// frame that somehow carried an account name must never publish it.
+test('acct and webId are stripped from a market payload regardless of config', () => {
+  const out = visibility.projectFeature(
+    'market',
+    { serial: '0x1', ownerAcct: 'darrow', webId: '42', shopName: 'Shop' },
+    'staff',
+    config,
+  )
+  assert.equal('ownerAcct' in out, false)
+  assert.equal('webId' in out, false)
+  assert.equal(out.shopName, 'Shop')
+})
+
+// Both kinds must be attributed to a feature, or rule 2 makes them admin-only by
+// omission — which would be a silent failure rather than a loud one.
+test('both market kinds are mapped to the market feature', () => {
+  assert.equal(visibility.KIND_FEATURE.get('vendor.listing'), 'market')
+  assert.equal(visibility.KIND_FEATURE.get('vendor.listing.remove'), 'market')
+})
+
+// The market's live firehose is off by default (a page of whole vendor
+// inventories is the site's biggest bandwidth item and no page needs it live),
+// but the REST reads are unaffected — which is what `visibleKinds` ignoring the
+// stream flag encodes.
+test('market kinds are stream-suppressed by default but still readable', () => {
+  assert.equal(visibility.DEFAULT_STREAM_OFF.has('market'), true)
+  assert.equal(visibility.kindVisibleTo('vendor.listing', 'anonymous', config), false)
+  assert.equal(visibility.PUBLIC_KINDS.has('vendor.listing'), false)
+  assert.ok(visibility.visibleKinds('anonymous', config).includes('vendor.listing'))
+})
+
+test('an admin who enables the stream gets the frames', () => {
+  const on = { ...config, market: { ...config.market, stream: true } }
+  assert.equal(visibility.kindVisibleTo('vendor.listing', 'anonymous', on), true)
+})
+
+// Guards the stub above against silently doing nothing: if the model stopped
+// going through db.lookup, every shapeItems assertion would still "pass" by
+// resolving nothing, which is also what a real miss looks like.
+test('the cliloc resolver is the path shapeItems resolves through', async () => {
+  const found = await clilocs.resolveMany([1023721])
+  assert.equal(found.get(1023721), 'quarter staff')
+})