From 8771a1cf6c892f3d3da051ec2c9bf39c5f09e7c5 Mon Sep 17 00:00:00 2001 From: wtclaude Date: Wed, 29 Jul 2026 09:51:50 -0500 Subject: [PATCH] feat(shard): the player-vendor marketplace MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Protocol 3.0 §8, the website half. Ingests vendor.listing / vendor.listing.remove into shard_vendors + shard_vendor_items, serves a searchable public API over them, and ships /site/market and /site/market/vendors/:serial. Three things the pages have to say out loud, all consequences of how the data is gathered: - The prices are NOT live. The shard sweeps vendors round-robin, so a shop can be a full cycle behind. The banner is driven by the OLDEST vendor row, not the newest — the one stale shop is the one that wastes somebody's trip. - A shop can be truncated. `total` exceeding `count` means the shop holds more than the shard publishes per frame; the vendor page says "showing 250 of 3,104" rather than presenting a partial shop as complete. - An item may have no name. On a shard with no cliloc table the honest render is the item id, never an invented label. ## The pre-wired visibility rules, re-checked Part A pre-wired market.ownerName and market.location before the frame existed, and the sibling rule it pre-wired for leaderboards (`characterName`) turned out to be INERT because projectValue matches literal JSON keys. Both market rules were checked against the real frame this time: - `ownerName` is a real key. Kept. - `location` is a real key ONLY because the frame nests it. Flat map/x/y/region would have made the rule match nothing — the same failure, one part later. It is nested on the wire and on the read model so one rule hides the facet, the coordinates, the region and the house together; five flat keys would be five rules that drift apart. - `ownerSerial` was ADDED. An admin who hides the owner's name and leaves a serial that the leaderboards and guild boards resolve back to that same name has not hidden anything. Tests assert all three bite, on the stored read model AND on the raw frame — the market's SSE stream is off by default but an admin can turn it on, and a rule that worked on only one path is exactly the leak §3.6.1 records. ## Notable - **No payload column on shard_vendors**, unlike shard_points_boards next door. The board's top-N is a fixed-size list read whole; here the items ARE the searchable rows, so they are normalized and nothing is left worth duplicating. - **display_name is denormalized at ingest** (literal name preferred over the cliloc — a player set it, so it is more specific). Resolving at query time would put the cliloc table on the hot path and make search-by-name impossible. Because the shard's diff sweep will not re-send an unchanged shop just because the site learned what its items are called, a cliloc import now triggers a bulk re-resolution — 50 ms per thousand rows, never throws. - **updated_at is written explicitly** on every upsert. MariaDB does not fire ON UPDATE CURRENT_TIMESTAMP when every column is written back unchanged, and a shop re-published identically is still freshly confirmed — without this the staleness banner would age a perfectly current shop forever. - **LIKE wildcards in `q` are escaped.** `%` and `_` are LIKE metacharacters, not SQL ones, so parameterization does not neutralize them: `?q=%` would otherwise match every listing on the shard. - **Rate-limited** (60/min/IP), the only limited public read. Every other public GET is an indexed lookup of bounded size; this is a LIKE scan plus a COUNT over the largest shard_* table, anonymous by default. - Reconnect backfill pages /market, bounded by MARKET_SNAPSHOT_MAX = 5000 and stopping on a short page as well as on `total`, so a concurrent sweep shrinking the index cannot spin the walk. ## How it was tested 673 server tests pass (27 new). Client builds clean; swagger-output.json, routes.manifest.json and routes.guards.json regenerated. Verified full-stack against the live MariaDB and a real shard, not only units: - 27 real vendors / 1,040 listings swept off the ServUO tree, through the Rust sidecar, into the site — names resolving through the cliloc table ("longsword", "katana"), real facets and regions in the filters. - `?q=sword` 682, `?q=%` and `?q=_` **0** (the escape), map/region/price/sort filters, paging, and the vendor detail route. - Visibility live: fields gated to staff vanish for an anonymous caller while shopName and price survive; audience=player 403s; enabled=0 404s; and /shard/features correctly drops `market` so the nav hides it. - Re-publishing a shop smaller leaves no orphan items; an identical re-publish moves updated_at. - The limiter fires (38x200 then 32x429 on a 70-request burst). Not covered by an automated test: the two React pages are presentational and this repo's client suite covers pure-logic modules only. They were driven against the live API above, but not rendered in a DOM harness. Co-Authored-By: Claude --- client/src/App.jsx | 4 + client/src/api/client.js | 22 + client/src/components/SiteHeader.jsx | 1 + .../routes/admin/views/ShardVisibility.jsx | 4 + client/src/routes/public/Market.jsx | 325 ++++++ client/src/routes/public/MarketVendor.jsx | 102 ++ server/db/schema.sql | 70 ++ server/routes.guards.json | 24 + server/routes.manifest.json | 12 + server/src/middleware/rateLimit.js | 14 + .../src/model/shardMarket/shardMarket.db.js | 299 +++++ .../model/shardMarket/shardMarket.model.js | 329 ++++++ .../v1/admin/shardClilocs.controller.js | 12 + .../src/router/v1/public/shard.controller.js | 82 ++ server/src/router/v1/public/shard.router.js | 66 ++ server/src/server.js | 11 +- server/src/utils/shardIngest.js | 15 + server/src/utils/shardVisibility.js | 16 +- server/src/utils/uoLinkClient.js | 6 + server/src/utils/uoLinkSocket.js | 51 + server/swagger/swagger-output.json | 1023 +++++++++++++++++ server/swagger/swagger.js | 91 ++ server/test/shardIngest.market.test.js | 114 ++ server/test/shardMarket.model.test.js | 260 +++++ 24 files changed, 2951 insertions(+), 2 deletions(-) create mode 100644 client/src/routes/public/Market.jsx create mode 100644 client/src/routes/public/MarketVendor.jsx create mode 100644 server/src/model/shardMarket/shardMarket.db.js create mode 100644 server/src/model/shardMarket/shardMarket.model.js create mode 100644 server/test/shardIngest.market.test.js create mode 100644 server/test/shardMarket.model.test.js diff --git a/client/src/App.jsx b/client/src/App.jsx index 7b2d994..3d46f16 100644 --- a/client/src/App.jsx +++ b/client/src/App.jsx @@ -26,6 +26,8 @@ import Rules from './routes/public/Rules.jsx' import Atlas from './routes/public/Atlas.jsx' import AtlasCreature from './routes/public/AtlasCreature.jsx' import Leaderboards from './routes/public/Leaderboards.jsx' +import Market from './routes/public/Market.jsx' +import MarketVendor from './routes/public/MarketVendor.jsx' import Wiki from './routes/wiki/Wiki.jsx' import WikiArticle from './routes/wiki/WikiArticle.jsx' import CmsPage from './routes/public/CmsPage.jsx' @@ -107,6 +109,8 @@ export default function App() { } /> } /> } /> + } /> + } /> } /> } /> {/* CMS pages: top-level /:slug, matched only after the named routes diff --git a/client/src/api/client.js b/client/src/api/client.js index 24a39d4..07f1917 100644 --- a/client/src/api/client.js +++ b/client/src/api/client.js @@ -154,6 +154,28 @@ export const api = { // `board` 404s for a system the shard has never published. points: () => req('/public/shard/points'), pointsBoard: (system) => req(`/public/shard/points/${encodeURIComponent(system)}`), + // Protocol 3.0: the player-vendor marketplace. Rate-limited server-side, so + // the page debounces its search box rather than firing per keystroke. + market: (opts = {}) => { + const qs = new URLSearchParams() + if (opts.q) qs.set('q', opts.q) + if (opts.minPrice != null && opts.minPrice !== '') qs.set('minPrice', opts.minPrice) + if (opts.maxPrice != null && opts.maxPrice !== '') qs.set('maxPrice', opts.maxPrice) + if (opts.itemId != null && opts.itemId !== '') qs.set('itemId', opts.itemId) + if (opts.map) qs.set('map', opts.map) + if (opts.region) qs.set('region', opts.region) + if (opts.sort) qs.set('sort', opts.sort) + if (opts.limit) qs.set('limit', opts.limit) + if (opts.offset) qs.set('offset', opts.offset) + return req(`/public/shard/market${withQs(qs.toString())}`) + }, + marketMeta: () => req('/public/shard/market/meta'), + marketVendor: (serial, opts = {}) => { + const qs = new URLSearchParams() + if (opts.limit) qs.set('limit', opts.limit) + if (opts.offset) qs.set('offset', opts.offset) + return req(`/public/shard/market/vendors/${encodeURIComponent(serial)}${withQs(qs.toString())}`) + }, // Which shard surfaces this caller may reach, plus the audience rung they // resolved to. Drives nav so we never render a link that would 403. features: () => req('/public/shard/features'), diff --git a/client/src/components/SiteHeader.jsx b/client/src/components/SiteHeader.jsx index a110381..a3c0341 100644 --- a/client/src/components/SiteHeader.jsx +++ b/client/src/components/SiteHeader.jsx @@ -26,6 +26,7 @@ const NAV = [ { label: 'Rules', to: '/site/rules', feature: 'ruleset' }, { label: 'Atlas', to: '/site/atlas', feature: 'atlas' }, { label: 'Leaderboards', to: '/site/leaderboards', feature: 'leaderboards' }, + { label: 'Market', to: '/site/market', feature: 'market' }, { label: 'About', to: '/site/about' }, ] diff --git a/client/src/routes/admin/views/ShardVisibility.jsx b/client/src/routes/admin/views/ShardVisibility.jsx index f102e70..e2de447 100644 --- a/client/src/routes/admin/views/ShardVisibility.jsx +++ b/client/src/routes/admin/views/ShardVisibility.jsx @@ -70,6 +70,10 @@ const FIELD_LABEL = { // field's meaning. The label is what carries the meaning to the admin. name: 'Character names on leaderboards', ownerName: 'Vendor owner name', + // One rule, one key — `location` is a nested object on both the wire frame and + // the stored read model precisely so that hiding it takes the facet, the + // coordinates, the region and the house together. + ownerSerial: 'Vendor owner character id', } function RungSelect({ value, onChange, ladder, disabled }) { diff --git a/client/src/routes/public/Market.jsx b/client/src/routes/public/Market.jsx new file mode 100644 index 0000000..612cf17 --- /dev/null +++ b/client/src/routes/public/Market.jsx @@ -0,0 +1,325 @@ +import { useCallback, useEffect, useState } from 'react' +import { Link } from 'react-router-dom' +import PublicLayout from '../../components/PublicLayout.jsx' +import PageHeader from '../../components/PageHeader.jsx' +import { Loading, ErrorState, EmptyState } from '../../components/PageState.jsx' +import { useAsync } from '../../lib/useAsync.js' +import { api } from '../../api/client.js' + +// ── The player-vendor marketplace ─────────────────────────────────────────── +// +// What every player vendor on the shard is selling, for how much, and where it +// is standing — the same index the in-game Vendor Search gump reads, honouring +// the same per-vendor opt-out, reachable without logging in to the game. +// +// Three things this page must be honest about, all of them consequences of how +// the data is gathered (docs/link/v3.md §8): +// +// • **The prices are not live.** The shard sweeps vendors round-robin, so a +// shop can be a full cycle behind. The banner says how far, from `staleAt`. +// A page that implied live prices would send people across the world to a +// vendor whose item sold twenty minutes ago. +// • **A shop can be truncated.** A commodity reseller with thousands of stacks +// publishes only the first N, and saying so beats presenting a partial shop +// as complete. +// • **An item may have no name.** On a shard whose operator has not converted +// a cliloc table, `displayName` is null and the honest render is the item id +// — not an invented name. +// +// There is deliberately no live feed here. The market feature's SSE stream ships +// disabled: a firehose of whole vendor inventories would be the site's single +// biggest bandwidth consumer, and nothing on this page needs it. + +const PAGE = 50 + +const num = (v) => (Number.isFinite(Number(v)) ? Number(v).toLocaleString() : '—') + +const SORTS = [ + { key: 'price_asc', label: 'Cheapest' }, + { key: 'price_desc', label: 'Priciest' }, + { key: 'recent', label: 'Recently seen' }, +] + +// How old the index may be, in words. `staleAt` is the OLDEST vendor row, so +// this is a worst case rather than an average — which is the number worth +// showing, because the one stale shop is the one that wastes a trip. +function staleness(staleAt) { + if (!staleAt) return null + const ms = Date.now() - new Date(staleAt).getTime() + if (!Number.isFinite(ms) || ms < 0) return null + const mins = Math.round(ms / 60000) + if (mins < 1) return 'just now' + if (mins < 60) return `${mins} minute${mins === 1 ? '' : 's'} ago` + const hours = Math.round(mins / 60) + if (hours < 48) return `${hours} hour${hours === 1 ? '' : 's'} ago` + return `${Math.round(hours / 24)} days ago` +} + +// The item's name, or an honest statement that we do not have one. Never a +// fabricated label — "Item 3922" would be indistinguishable from a real name. +const itemLabel = (l) => l.displayName || l.name || `id ${l.itemId}` + +function Chip({ active, onClick, children }) { + return ( + + ) +} + +function ListingRow({ listing }) { + const v = listing.vendor || {} + // `location` is one field the admin can gate away wholesale, so everything + // that reads from it has to tolerate its absence rather than assuming a map. + const loc = v.location || null + const where = loc ? [loc.region, loc.map].filter(Boolean).join(', ') : null + + return ( +
+
+
+ {listing.amount > 1 ? `${num(listing.amount)} × ` : ''} + {itemLabel(listing)} +
+
+ {v.serial ? ( + + {v.shopName || 'an unnamed shop'} + + ) : ( + v.shopName || 'an unnamed shop' + )} + {v.ownerName ? ` · ${v.ownerName}` : ''} + {where ? ` · ${where}` : ''} + {/* Priced by the container it sits in, exactly as the in-game search + reports it — the price buys the whole container, not this item. */} + {listing.child ? ' · sold with its container' : ''} +
+
+
+
{num(listing.price)}
+
gold
+
+
+ ) +} + +export default function Market() { + const [input, setInput] = useState('') + const [q, setQ] = useState('') + const [map, setMap] = useState('') + const [region, setRegion] = useState('') + const [sort, setSort] = useState('price_asc') + const [minPrice, setMinPrice] = useState('') + const [maxPrice, setMaxPrice] = useState('') + // Applied prices are separate from the typed ones so the search fires when the + // user is done, not on every digit of "250000". + const [prices, setPrices] = useState({ min: '', max: '' }) + + const [state, setState] = useState({ loading: true, error: null, listings: [], total: 0, staleAt: null }) + const [more, setMore] = useState(false) + + const meta = useAsync(() => api.shard.marketMeta()) + + // Debounced: typing "vanquishing" should be one request, not eleven — and the + // endpoint is rate-limited, so an undebounced box would 429 a fast typist. + useEffect(() => { + const timer = setTimeout(() => setQ(input.trim()), 300) + return () => clearTimeout(timer) + }, [input]) + + useEffect(() => { + const timer = setTimeout(() => setPrices({ min: minPrice, max: maxPrice }), 500) + return () => clearTimeout(timer) + }, [minPrice, maxPrice]) + + const load = useCallback( + (offset) => + api.shard.market({ + q, + map, + region, + sort, + minPrice: prices.min, + maxPrice: prices.max, + limit: PAGE, + offset, + }), + [q, map, region, sort, prices], + ) + + useEffect(() => { + let alive = true + setState({ loading: true, error: null, listings: [], total: 0, staleAt: null }) + load(0) + .then((page) => { + if (!alive) return + setState({ + loading: false, + error: null, + listings: page.listings || [], + total: page.total || 0, + staleAt: page.staleAt || null, + }) + }) + .catch((error) => alive && setState({ loading: false, error, listings: [], total: 0, staleAt: null })) + return () => { + alive = false + } + }, [load]) + + const loadMore = async () => { + setMore(true) + try { + const page = await load(state.listings.length) + setState((s) => ({ + ...s, + listings: [...s.listings, ...(page.listings || [])], + total: page.total ?? s.total, + staleAt: page.staleAt ?? s.staleAt, + })) + } catch { + // A failed "load more" leaves what is on screen alone; the button stays + // available to retry. + } finally { + setMore(false) + } + } + + const maps = meta.data?.maps || [] + const regions = meta.data?.regions || [] + const age = staleness(state.staleAt) + + return ( + +
+ + + {/* Not decoration. The sweep is round-robin, so the index is inherently + up to one full cycle old and the page has to say so. */} + {age && ( +

+ Prices last refreshed {age} + {meta.data?.vendors ? ` · ${num(meta.data.vendors)} shops` : ''} + {meta.data?.items ? ` · ${num(meta.data.items)} listings` : ''} +

+ )} + + setInput(e.target.value)} + placeholder="Search listings…" + style={{ width: '100%', marginBottom: 10 }} + /> + +
+ setMinPrice(e.target.value)} + placeholder="Min price" + style={{ maxWidth: 140 }} + /> + setMaxPrice(e.target.value)} + placeholder="Max price" + style={{ maxWidth: 140 }} + /> +
+ +
+ {SORTS.map((s) => ( + setSort(s.key)}> + {s.label} + + ))} +
+ + {/* Facet and region names come from the shard's own data, never a list in + this file — a shard running custom maps gets its own names here with + no code change (docs/link/v3.md §6.1 R2). */} + {maps.length > 0 && ( +
+ setMap('')}>All facets + {maps.map((m) => ( + setMap(m)}>{m} + ))} +
+ )} + + {regions.length > 0 && ( + + )} + + {state.loading && } + {state.error && } + + {!state.loading && !state.error && state.listings.length === 0 && ( + + {meta.data?.vendors + ? 'Nothing on the shard matches that.' + : 'No player vendors have been indexed yet.'} + + )} + + {!state.loading && !state.error && state.listings.length > 0 && ( + <> +

+ Showing {num(state.listings.length)} of {num(state.total)} +

+
+ {state.listings.map((l) => ( + + ))} +
+ {state.listings.length < state.total && ( +
+ +
+ )} + + )} +
+ + ) +} diff --git a/client/src/routes/public/MarketVendor.jsx b/client/src/routes/public/MarketVendor.jsx new file mode 100644 index 0000000..7359b9e --- /dev/null +++ b/client/src/routes/public/MarketVendor.jsx @@ -0,0 +1,102 @@ +import { Link, useParams } from 'react-router-dom' +import PublicLayout from '../../components/PublicLayout.jsx' +import PageHeader from '../../components/PageHeader.jsx' +import { Loading, ErrorState, EmptyState } from '../../components/PageState.jsx' +import { useAsync } from '../../lib/useAsync.js' +import { api } from '../../api/client.js' + +// One player vendor: where to find it and everything it is selling. +// +// The page a search result points at. Two states it has to render honestly and +// which the search list cannot (docs/link/v3.md §8): +// +// • `truncated` — the shop holds more than the shard publishes per frame. A +// commodity reseller with thousands of stacks is a real thing, and showing +// 250 of 3,104 as if it were the whole shop would be a lie about the shard. +// • a gated `location` — an admin may put vendor whereabouts behind a rung, in +// which case there is nothing to render and the page says so rather than +// showing an empty coordinate. + +const num = (v) => (Number.isFinite(Number(v)) ? Number(v).toLocaleString() : '—') + +const itemLabel = (i) => i.displayName || i.name || `id ${i.itemId}` + +export default function MarketVendor() { + const { serial } = useParams() + const { loading, error, data } = useAsync(() => api.shard.marketVendor(serial), [serial]) + + if (loading) { + return ( + +
+ + ) + } + + if (error || !data) { + return ( + +
+ +

+ ← Back to the marketplace +

+
+ + ) + } + + const loc = data.location || null + const items = data.items || [] + + return ( + +
+ + +

+ {data.truncated + ? `Showing ${num(data.count)} of ${num(data.total)} listings — this shop holds more than the shard publishes.` + : `${num(data.total)} listing${data.total === 1 ? '' : 's'}`} + {data.updatedAt ? ` · last seen ${new Date(data.updatedAt).toLocaleString()}` : ''} +

+ + {items.length === 0 ? ( + This shop has nothing priced for sale. + ) : ( +
+ {items.map((i) => ( +
+ + {i.amount > 1 ? `${num(i.amount)} × ` : ''} + {itemLabel(i)} + {i.child ? · sold with its container : null} + + + {num(i.price)} + +
+ ))} +
+ )} + +

+ ← Back to the marketplace +

+
+ + ) +} diff --git a/server/db/schema.sql b/server/db/schema.sql index 1c5df9f..95532e0 100644 --- a/server/db/schema.sql +++ b/server/db/schema.sql @@ -640,6 +640,76 @@ CREATE TABLE IF NOT EXISTS shard_points_boards ( updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4; +-- Player-vendor market index (Protocol 3.0 vendor.listing). One row per player +-- vendor and one per priced listing, so the site can offer the search the in-game +-- Vendor Search gump offers — from outside the game. +-- +-- The shard sweeps vendors round-robin and emits one AUTHORITATIVE frame per +-- vendor, so ingest is delete-then-insert of that vendor's items inside one +-- transaction (see shardMarket.db.js). No foreign key from items to vendors, in +-- keeping with every other shard_* table: the ingest transaction is what keeps +-- them consistent, and an FK would turn a malformed frame into a failed write +-- rather than a dropped row. +-- +-- Only vendors whose owner left the in-game Vendor Search flag ON are ever sent, +-- so a player who hid their shop in game is hidden here too — see BridgeMarket.cs. +CREATE TABLE IF NOT EXISTS shard_vendors ( + serial VARCHAR(20) NOT NULL PRIMARY KEY, -- "0x40001234" + shop_name VARCHAR(160) NULL, + owner_serial VARCHAR(20) NULL, + owner_name VARCHAR(64) NULL, + map VARCHAR(40) NULL, + x INT NULL, + y INT NULL, + z INT NULL, + region VARCHAR(80) NULL, + house VARCHAR(160) NULL, -- the house SIGN's name, not the house type + item_count INT NOT NULL DEFAULT 0, -- listings published in the frame + item_total INT NOT NULL DEFAULT 0, -- listings the shop actually holds + truncated TINYINT(1) NOT NULL DEFAULT 0, -- item_total > item_count + t BIGINT NULL, -- frame time, epoch ms + updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, + INDEX idx_shard_vendors_owner (owner_name), + INDEX idx_shard_vendors_map (map), + INDEX idx_shard_vendors_region (region), + -- The market page's staleness banner is MIN(updated_at) over this column: the + -- round-robin sweep means the oldest row is how far behind the index can be. + INDEX idx_shard_vendors_updated (updated_at) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4; + +-- One priced listing. Unlike the points board's top-N — a fixed-size list read +-- whole — these are the searchable rows the whole feature exists for, so they are +-- normalized rather than left inside a payload column, and there is no payload +-- column on shard_vendors at all. +-- +-- `display_name` is DENORMALIZED at ingest: the shard sends `cliloc` (the item's +-- LabelNumber) and, rarely, a literal `name`, and resolving 50 clilocs per page +-- at query time would make the cliloc table a join on the hot path AND make +-- search-by-name impossible. Resolving once on write buys the index. It is +-- re-resolved in bulk after a cliloc import, because the diff sweep will not +-- re-send an unchanged shop just because the site learned what its items are +-- called. +CREATE TABLE IF NOT EXISTS shard_vendor_items ( + id BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY, + vendor_serial VARCHAR(20) NOT NULL, + serial VARCHAR(20) NOT NULL, + item_id INT NOT NULL DEFAULT 0, -- ItemID (the art/graphic id) + hue INT NOT NULL DEFAULT 0, + amount INT NOT NULL DEFAULT 1, + price BIGINT NOT NULL DEFAULT 0, + name VARCHAR(160) NULL, -- the item's literal Name, null for most + cliloc INT NULL, -- LabelNumber, resolved against shard_clilocs + display_name VARCHAR(160) NULL, -- resolved at ingest; what search matches + child TINYINT(1) NOT NULL DEFAULT 0, -- priced by an enclosing container, not itself + INDEX idx_shard_vendor_items_vendor (vendor_serial), + INDEX idx_shard_vendor_items_price (price), + INDEX idx_shard_vendor_items_item (item_id), + INDEX idx_shard_vendor_items_name (display_name), + -- Search filters on name and sorts on price; the composite covers the common + -- "cheapest matching X" without a filesort over the whole table. + INDEX idx_shard_vendor_items_name_price (display_name, price) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4; + -- Per-feature visibility for every shard-derived surface (Protocol 3.0). One row -- per feature; an absent row means "use the compiled default", and the compiled -- defaults reproduce the behavior that shipped before v3 — so an empty table is diff --git a/server/routes.guards.json b/server/routes.guards.json index c42ddc2..c7c6c5e 100644 --- a/server/routes.guards.json +++ b/server/routes.guards.json @@ -2027,6 +2027,30 @@ "handlers": 2, "gates": [] }, + { + "method": "GET", + "path": "/api/v1/public/shard/market", + "handlers": 13, + "gates": [ + "middleware", + "validate" + ] + }, + { + "method": "GET", + "path": "/api/v1/public/shard/market/meta", + "handlers": 2, + "gates": [] + }, + { + "method": "GET", + "path": "/api/v1/public/shard/market/vendors/:serial", + "handlers": 7, + "gates": [ + "middleware", + "validate" + ] + }, { "method": "GET", "path": "/api/v1/public/shard/online", diff --git a/server/routes.manifest.json b/server/routes.manifest.json index 6c6544b..1692271 100644 --- a/server/routes.manifest.json +++ b/server/routes.manifest.json @@ -829,6 +829,18 @@ "method": "GET", "path": "/api/v1/public/shard/idoc" }, + { + "method": "GET", + "path": "/api/v1/public/shard/market" + }, + { + "method": "GET", + "path": "/api/v1/public/shard/market/meta" + }, + { + "method": "GET", + "path": "/api/v1/public/shard/market/vendors/:serial" + }, { "method": "GET", "path": "/api/v1/public/shard/online" diff --git a/server/src/middleware/rateLimit.js b/server/src/middleware/rateLimit.js index 33ab13f..7821694 100644 --- a/server/src/middleware/rateLimit.js +++ b/server/src/middleware/rateLimit.js @@ -118,6 +118,19 @@ const passwordResetConfirmLimiter = makeLimiter({ message: 'Too many attempts. Please try again later.', }) +// The player-vendor market search. The first genuinely expensive PUBLIC endpoint +// on the site: every call is a LIKE scan plus a COUNT over the listings table, +// which on a large shard is the biggest table there is, and it is anonymous by +// default. Generous for a human browsing shops (a typed search is debounced to +// one request, and paging is a click), tight enough that it cannot be used as a +// cheap way to load the database. +const marketLimiter = makeLimiter({ + windowMs: 60 * 1000, + max: 60, + label: 'market', + message: 'Too many searches. Please slow down.', +}) + // CSP violation reports. Unauthenticated by necessity (browsers send them with no // session), and every accepted report writes a log line — so an attacker who can get // a victim to load a page could otherwise use it as a log-flood amplifier. Generous @@ -141,5 +154,6 @@ module.exports = { mobileSsoExchangeLimiter, passwordResetRequestLimiter, passwordResetConfirmLimiter, + marketLimiter, cspReportLimiter, } diff --git a/server/src/model/shardMarket/shardMarket.db.js b/server/src/model/shardMarket/shardMarket.db.js new file mode 100644 index 0000000..44db273 --- /dev/null +++ b/server/src/model/shardMarket/shardMarket.db.js @@ -0,0 +1,299 @@ +const { pool, query } = require('../../utils/db') + +// Raw SQL for the player-vendor market index (Protocol 3.0 vendor.listing). +// +// Two tables, both INGEST-OWNED: `shard_vendors` (one row per shop) and +// `shard_vendor_items` (one row per priced listing). Nothing else in the codebase +// writes to either. No foreign keys, consistent with every other shard_* table. + +// Insert batch size for one vendor's listings. A shop is capped at +// MarketMaxListings (250 by default) on the shard side, so in practice this is +// one batch — it exists for the operator who raised that cap. +const BATCH = 500 + +// LIKE wildcards in user input. `%` and `_` are not special to the parameterized +// query — they are special to LIKE itself — so a search for "50% off" would +// otherwise match everything containing "50" and a search for "_" would match +// every single-character name. Escaped with a backslash, which is MariaDB's +// default LIKE escape (no ESCAPE clause needed). +const likeTerm = (q) => `%${String(q).replace(/[\\%_]/g, (c) => `\\${c}`)}%` + +/** + * Replace one vendor's whole row and listing set, in one transaction. + * + * Delete-then-insert rather than a diff, because the frame is AUTHORITATIVE for + * that vendor: the shard's sweep only emits a shop whose contents, prices or + * location moved, and when it does it sends the whole shop. Reconciling it item + * by item would be more code for the same result and would leave sold items + * behind on any path the reconciliation missed. + * + * All-or-nothing matters here for a specific reason: the two writes are "the + * shop" and "what is in it", and a failure between them leaves a shop advertising + * an inventory it no longer has (or none at all) — visibly wrong on the page, and + * indistinguishable from a genuinely empty shop. + */ +async function replaceVendor(vendor, items) { + const conn = await pool.getConnection() + try { + await conn.beginTransaction() + + await conn.query( + `INSERT INTO shard_vendors + (serial, shop_name, owner_serial, owner_name, map, x, y, z, region, house, + item_count, item_total, truncated, t) + VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?) + ON DUPLICATE KEY UPDATE shop_name = VALUES(shop_name), owner_serial = VALUES(owner_serial), + owner_name = VALUES(owner_name), map = VALUES(map), x = VALUES(x), y = VALUES(y), + z = VALUES(z), region = VALUES(region), house = VALUES(house), + item_count = VALUES(item_count), item_total = VALUES(item_total), + truncated = VALUES(truncated), t = VALUES(t), + -- Touched explicitly rather than left to ON UPDATE CURRENT_TIMESTAMP: + -- MariaDB does not fire that when every column is written back + -- unchanged, and a shop that is re-published identically is still + -- FRESHLY CONFIRMED. Without this the staleness banner would age a + -- perfectly current shop forever. + updated_at = CURRENT_TIMESTAMP`, + [ + vendor.serial, + vendor.shopName ?? null, + vendor.ownerSerial ?? null, + vendor.ownerName ?? null, + vendor.map ?? null, + Number.isFinite(vendor.x) ? vendor.x : null, + Number.isFinite(vendor.y) ? vendor.y : null, + Number.isFinite(vendor.z) ? vendor.z : null, + vendor.region ?? null, + vendor.house ?? null, + items.length, + Number.isFinite(vendor.itemTotal) ? vendor.itemTotal : items.length, + vendor.truncated ? 1 : 0, + Number.isFinite(vendor.t) ? vendor.t : null, + ], + ) + + await conn.query('DELETE FROM shard_vendor_items WHERE vendor_serial = ?', [vendor.serial]) + + const rows = items.map((i) => [ + vendor.serial, + i.serial, + i.itemId, + i.hue, + i.amount, + i.price, + i.name, + i.cliloc, + i.displayName, + i.child ? 1 : 0, + ]) + + for (let i = 0; i < rows.length; i += BATCH) { + await conn.batch( + `INSERT INTO shard_vendor_items + (vendor_serial, serial, item_id, hue, amount, price, name, cliloc, display_name, child) + VALUES (?,?,?,?,?,?,?,?,?,?)`, + rows.slice(i, i + BATCH), + ) + } + + await conn.commit() + return { items: rows.length } + } catch (err) { + await conn.rollback().catch(() => {}) + throw err + } finally { + conn.release() + } +} + +/** Drop one vendor and its listings (vendor.listing.remove). */ +async function removeVendor(serial) { + const conn = await pool.getConnection() + try { + await conn.beginTransaction() + await conn.query('DELETE FROM shard_vendor_items WHERE vendor_serial = ?', [serial]) + await conn.query('DELETE FROM shard_vendors WHERE serial = ?', [serial]) + await conn.commit() + } catch (err) { + await conn.rollback().catch(() => {}) + throw err + } finally { + conn.release() + } +} + +// ── Search ───────────────────────────────────────────────────────────────── +// +// The unit of a search RESULT is a listing, not a vendor: "who sells a vanquishing +// kryss and for how much" is the question, and answering it per vendor would make +// the caller flatten the shops back out. The vendor's columns ride along on the +// join so a result row is self-contained. + +function searchWhere({ q, minPrice, maxPrice, itemId, map, region }) { + const where = ['i.price > 0'] + const params = [] + + if (q) { + // Both the resolved display name and the item's own literal, because an item + // with a player-set name (most of what is actually worth searching for on a + // player-run shard) may have a generic cliloc. + where.push('(i.display_name LIKE ? OR i.name LIKE ?)') + params.push(likeTerm(q), likeTerm(q)) + } + if (Number.isFinite(minPrice)) { + where.push('i.price >= ?') + params.push(minPrice) + } + if (Number.isFinite(maxPrice)) { + where.push('i.price <= ?') + params.push(maxPrice) + } + if (Number.isFinite(itemId)) { + where.push('i.item_id = ?') + params.push(itemId) + } + if (map) { + where.push('v.map = ?') + params.push(map) + } + if (region) { + where.push('v.region = ?') + params.push(region) + } + + return { sql: `WHERE ${where.join(' AND ')}`, params } +} + +// Whitelisted, because this interpolates into the statement. `recent` sorts by +// the vendor's freshness, which is the only way to see what has just been listed +// on a shard whose sweep is minutes wide. +const SORTS = { + price_asc: 'i.price ASC, i.id ASC', + price_desc: 'i.price DESC, i.id ASC', + recent: 'v.updated_at DESC, i.id ASC', +} + +async function searchListings({ q, minPrice, maxPrice, itemId, map, region, sort, limit, offset }) { + const { sql, params } = searchWhere({ q, minPrice, maxPrice, itemId, map, region }) + const order = SORTS[sort] || SORTS.price_asc + + const rows = await query( + `SELECT i.serial, i.item_id, i.hue, i.amount, i.price, i.name, i.cliloc, i.display_name, i.child, + v.serial AS vendor_serial, v.shop_name, v.owner_serial, v.owner_name, + v.map, v.x, v.y, v.z, v.region, v.house, v.updated_at + FROM shard_vendor_items i + JOIN shard_vendors v ON v.serial = i.vendor_serial + ${sql} + ORDER BY ${order} + LIMIT ? OFFSET ?`, + [...params, limit, offset], + ) + + const counted = await query( + `SELECT COUNT(*) AS n + FROM shard_vendor_items i + JOIN shard_vendors v ON v.serial = i.vendor_serial + ${sql}`, + params, + ) + + return { rows, total: Number(counted[0]?.n) || 0 } +} + +async function getVendor(serial) { + const rows = await query( + `SELECT serial, shop_name, owner_serial, owner_name, map, x, y, z, region, house, + item_count, item_total, truncated, t, updated_at + FROM shard_vendors WHERE serial = ?`, + [serial], + ) + return rows[0] || null +} + +async function listVendorItems(serial, { limit, offset }) { + return query( + `SELECT serial, item_id, hue, amount, price, name, cliloc, display_name, child + FROM shard_vendor_items + WHERE vendor_serial = ? + ORDER BY price ASC, id ASC + LIMIT ? OFFSET ?`, + [serial, limit, offset], + ) +} + +/** + * What the market page's header needs: how big the index is, and how stale it may + * be. `staleAt` is the OLDEST vendor row — the round-robin sweep means a shop can + * be a full cycle behind, and the page says so rather than implying live prices. + */ +async function meta() { + const rows = await query( + `SELECT COUNT(*) AS vendors, MIN(updated_at) AS stale_at, MAX(updated_at) AS fresh_at + FROM shard_vendors`, + ) + const items = await query('SELECT COUNT(*) AS n FROM shard_vendor_items') + return { + vendors: Number(rows[0]?.vendors) || 0, + items: Number(items[0]?.n) || 0, + staleAt: rows[0]?.stale_at || null, + freshAt: rows[0]?.fresh_at || null, + } +} + +/** The distinct facets and regions holding vendors — drives the page's filters. */ +async function listPlaces() { + const maps = await query( + 'SELECT DISTINCT map FROM shard_vendors WHERE map IS NOT NULL ORDER BY map', + ) + const regions = await query( + 'SELECT DISTINCT region FROM shard_vendors WHERE region IS NOT NULL ORDER BY region', + ) + return { maps: maps.map((r) => r.map), regions: regions.map((r) => r.region) } +} + +// ── Cliloc re-resolution ─────────────────────────────────────────────────── + +/** + * One page of listings whose name still needs resolving, for the bulk pass that + * runs after a cliloc import. + * + * Keyed on `id > after` rather than OFFSET: the pass updates the very rows it is + * scanning, and an OFFSET walk over a table being rewritten skips rows. Every + * row with a cliloc is re-read, not just the unresolved ones, because an import + * can also CHANGE a name — a shard overlay relabelling a stock item is the whole + * reason overlays exist. + */ +async function listResolvableItems(after, limit) { + return query( + `SELECT id, cliloc, name, display_name + FROM shard_vendor_items + WHERE cliloc IS NOT NULL AND cliloc > 0 AND id > ? + ORDER BY id + LIMIT ?`, + [after, limit], + ) +} + +/** Write back a batch of re-resolved display names. */ +async function updateDisplayNames(pairs) { + if (pairs.length === 0) return 0 + const conn = await pool.getConnection() + try { + await conn.batch('UPDATE shard_vendor_items SET display_name = ? WHERE id = ?', pairs) + return pairs.length + } finally { + conn.release() + } +} + +module.exports = { + replaceVendor, + removeVendor, + searchListings, + getVendor, + listVendorItems, + meta, + listPlaces, + listResolvableItems, + updateDisplayNames, + likeTerm, +} diff --git a/server/src/model/shardMarket/shardMarket.model.js b/server/src/model/shardMarket/shardMarket.model.js new file mode 100644 index 0000000..a0c781d --- /dev/null +++ b/server/src/model/shardMarket/shardMarket.model.js @@ -0,0 +1,329 @@ +// ── Player-vendor market index (Protocol 3.0 vendor.listing) ─────────────── +// +// The shard-wide shop index: what every player vendor is selling, for how much, +// and where it is standing. This is the website's half of the search the in-game +// Vendor Search gump offers — the same data, the same opt-out, reachable without +// logging in to the game. +// +// Ingest is per-vendor and authoritative: the shard's round-robin sweep emits one +// `vendor.listing` frame per shop whose contents, prices or location moved, and +// the frame is the whole shop (see docs/link/v3.md §8 and BridgeMarket.cs). This +// module normalizes it into shard_vendors + shard_vendor_items and, crucially, +// resolves each listing's cliloc to a DISPLAY NAME on the way in — a search for +// "kryss" is a search over names, and the shard only ever sends numbers. + +const db = require('./shardMarket.db') +const clilocs = require('../shardClilocs/shardClilocs.model') +const log = require('../../utils/logger')('shard-market') + +// Defense in depth on top of the shard's own MarketMaxListings cap. The shard is +// trusted, but it is a separately-versioned component: a frame from a plugin +// whose cap was raised (or a shard running modified scripts) must not be able to +// turn one ingest into an unbounded transaction. +const MAX_ITEMS_PER_VENDOR = 5000 + +// Column widths in schema.sql. Truncating here rather than letting MariaDB do it +// keeps the behavior the same in strict mode, where an over-length value is an +// ERROR and would fail the whole vendor rather than shortening one name. +const MAX_NAME = 160 +const MAX_SHOP = 160 +const MAX_OWNER = 64 +const MAX_MAP = 40 +const MAX_REGION = 80 +const MAX_SERIAL = 20 + +const clip = (value, max) => { + if (value == null) return null + const s = String(value) + return s.length > max ? s.slice(0, max) : s +} + +const int = (value, fallback = 0) => { + const n = Number(value) + return Number.isFinite(n) ? Math.trunc(n) : fallback +} + +// ── Ingest ───────────────────────────────────────────────────────────────── + +/** + * Flatten one `vendor.listing` frame into the row shapes the DB layer wants. + * + * `location` arrives as a nested object rather than flat map/x/y/region, and that + * shape is load-bearing rather than cosmetic: the visibility projection matches + * literal JSON keys, so ONE `market.location` rule can hide a vendor's + * whereabouts only if `location` is a single key on both the live frame and the + * stored read model. Flattening it here for storage and re-nesting it on read is + * what keeps that true on both paths. + * + * Exported for tests — it is the part with rules in it, and it is pure. + */ +function flattenFrame(ev) { + const loc = (ev && ev.location) || {} + return { + serial: clip(ev.serial, MAX_SERIAL), + shopName: clip(ev.shopName, MAX_SHOP), + ownerSerial: clip(ev.ownerSerial, MAX_SERIAL), + ownerName: clip(ev.ownerName, MAX_OWNER), + map: clip(loc.map, MAX_MAP), + x: Number.isFinite(loc.x) ? Math.trunc(loc.x) : null, + y: Number.isFinite(loc.y) ? Math.trunc(loc.y) : null, + z: Number.isFinite(loc.z) ? Math.trunc(loc.z) : null, + region: clip(loc.region, MAX_REGION), + house: clip(loc.house, MAX_SHOP), + // What the SHOP holds, which is not what the frame carries when it was + // truncated. Kept apart so the page can say "showing 250 of 3,104" rather + // than presenting a partial shop as a complete one. + itemTotal: int(ev.total, int(ev.count, 0)), + truncated: ev.truncated === true, + t: Number.isFinite(ev.t) ? ev.t : null, + } +} + +/** + * Resolve each listing's display name. + * + * Order of preference is the item's own literal `name` first, then the cliloc. + * That is the opposite of what "resolve the id" suggests and it is right: a + * literal name only exists because a player set one ("Bob's vanquishing kryss"), + * and it is strictly more specific than the generic cliloc the item still + * carries. + * + * One batched lookup per frame rather than per item; `resolveMany` is cached and + * never throws, so a cliloc table that is missing entirely just leaves + * `displayName` null and the page renders item ids, exactly as it did before the + * table existed. + */ +async function shapeItems(ev) { + const raw = Array.isArray(ev.items) ? ev.items.slice(0, MAX_ITEMS_PER_VENDOR) : [] + + const wanted = raw + .map((i) => int(i && i.cliloc, 0)) + .filter((n) => n > 0) + + const names = await clilocs.resolveMany(wanted) + + return raw + .filter((i) => i && i.serial) + .map((i) => { + const literal = clip(i.name, MAX_NAME) + const cliloc = int(i.cliloc, 0) || null + return { + serial: clip(i.serial, MAX_SERIAL), + itemId: int(i.itemId, 0), + hue: int(i.hue, 0), + amount: int(i.amount, 1), + price: int(i.price, 0), + name: literal, + cliloc, + displayName: literal || (cliloc ? clip(names.get(cliloc) ?? null, MAX_NAME) : null), + child: i.child === true, + } + }) + // Unpriced rows are inventory, not listings. The shard already drops them; + // this is the same rule enforced where the table is written, so a plugin that + // stops enforcing it cannot put un-buyable rows on the market page. + .filter((i) => i.price > 0) +} + +/** Ingest one `vendor.listing` frame. */ +async function upsertVendor(ev) { + if (!ev || !ev.serial) return + const vendor = flattenFrame(ev) + const items = await shapeItems(ev) + await db.replaceVendor(vendor, items) +} + +/** Ingest one `vendor.listing.remove` frame. */ +async function removeVendor(serial) { + if (!serial) return + await db.removeVendor(String(serial).slice(0, MAX_SERIAL)) +} + +// ── Read models ──────────────────────────────────────────────────────────── +// +// `location` is re-nested (see flattenFrame) so the stored read model and the +// live wire frame present the same keys to the visibility projection. + +const place = (r) => ({ + map: r.map, + x: r.x, + y: r.y, + z: r.z, + region: r.region, + house: r.house, +}) + +// A listing as the search returns it: the item, plus enough of its shop to be +// actionable without a second request. `displayName` falls back to nothing rather +// than to a fabricated "Item 3922" — the client decides how to render an +// unresolved id, and inventing a name here would make it indistinguishable from +// a real one. +const shapeListing = (r) => ({ + serial: r.serial, + itemId: r.item_id, + hue: r.hue, + amount: r.amount, + price: Number(r.price), + name: r.name, + cliloc: r.cliloc, + displayName: r.display_name, + child: !!r.child, + vendor: { + serial: r.vendor_serial, + shopName: r.shop_name, + ownerSerial: r.owner_serial, + ownerName: r.owner_name, + location: place(r), + updatedAt: r.updated_at, + }, +}) + +const shapeVendor = (r) => ({ + serial: r.serial, + shopName: r.shop_name, + ownerSerial: r.owner_serial, + ownerName: r.owner_name, + location: place(r), + count: r.item_count, + total: r.item_total, + truncated: !!r.truncated, + updatedAt: r.updated_at, +}) + +const shapeItem = (r) => ({ + serial: r.serial, + itemId: r.item_id, + hue: r.hue, + amount: r.amount, + price: Number(r.price), + name: r.name, + cliloc: r.cliloc, + displayName: r.display_name, + child: !!r.child, +}) + +/** + * Search the index. Returns a page of LISTINGS (not vendors) plus the + * unpaginated total and the staleness stamp the page's banner needs. + */ +async function search({ + q = '', + minPrice, + maxPrice, + itemId, + map = '', + region = '', + sort = 'price_asc', + limit = 50, + offset = 0, +} = {}) { + const { rows, total } = await db.searchListings({ + q: q.trim(), + minPrice: Number.isFinite(minPrice) ? minPrice : undefined, + maxPrice: Number.isFinite(maxPrice) ? maxPrice : undefined, + itemId: Number.isFinite(itemId) ? itemId : undefined, + map: map.trim(), + region: region.trim(), + sort, + limit, + offset, + }) + + const info = await db.meta() + + return { + listings: rows.map(shapeListing), + total, + limit, + offset, + // Repeated on every search response rather than left to a separate /meta + // call: the banner that says how old these prices are must age with the + // results it labels, and a client that fetched it once would keep showing a + // stamp from before the page it is looking at. + staleAt: info.staleAt, + vendors: info.vendors, + } +} + +/** One shop and its listings. `null` when the index has never seen that serial. */ +async function getVendor(serial, { limit = 250, offset = 0 } = {}) { + const row = await db.getVendor(serial) + if (!row) return null + const items = await db.listVendorItems(serial, { limit, offset }) + return { ...shapeVendor(row), items: items.map(shapeItem) } +} + +/** Index size, staleness, and the facet/region filter options. */ +async function meta() { + const [info, places] = await Promise.all([db.meta(), db.listPlaces()]) + return { ...info, ...places } +} + +// ── Cliloc re-resolution ─────────────────────────────────────────────────── + +// Batch size for the post-import pass. Big enough that a 40k-row table is ~40 +// round trips, small enough that a single batch is not a long-held connection. +const RESOLVE_BATCH = 1000 + +/** + * Re-resolve every listing's display name against the current cliloc table. + * + * Called after a cliloc import, and it has to be: the market's diff sweep will + * NOT re-send an unchanged shop just because the site learned what its items are + * called, so without this an operator who configures clilocs after the first + * market sweep sees item ids until every shop happens to change. That is the same + * class of staleness the spawn atlas avoids by re-parsing on boot — here the + * source of truth for names moved, not the data. + * + * Never throws. It is a cosmetic backfill on a table that is already serving; a + * failure means names stay as they were, which is exactly the pre-import state. + */ +async function refreshDisplayNames() { + let after = 0 + let scanned = 0 + let changed = 0 + + try { + for (;;) { + const rows = await db.listResolvableItems(after, RESOLVE_BATCH) + if (rows.length === 0) break + + after = rows[rows.length - 1].id + scanned += rows.length + + const names = await clilocs.resolveMany(rows.map((r) => Number(r.cliloc))) + + const pairs = [] + for (const row of rows) { + // The literal name still wins, so a re-resolution never overwrites a + // player-set name with the generic cliloc behind it. + const next = row.name + ? clip(row.name, MAX_NAME) + : clip(names.get(Number(row.cliloc)) ?? null, MAX_NAME) + if (next !== row.display_name) pairs.push([next, row.id]) + } + + changed += await db.updateDisplayNames(pairs) + } + + if (changed > 0) log.info('market display names refreshed', { scanned, changed }) + return { scanned, changed } + } catch (err) { + log.warn('market display-name refresh failed', { message: err.message, scanned, changed }) + return { scanned, changed, error: err.message } + } +} + +module.exports = { + upsertVendor, + removeVendor, + search, + getVendor, + meta, + refreshDisplayNames, + flattenFrame, + shapeItems, + shapeListing, + shapeVendor, + MAX_ITEMS_PER_VENDOR, +} diff --git a/server/src/router/v1/admin/shardClilocs.controller.js b/server/src/router/v1/admin/shardClilocs.controller.js index dc3eb55..2ecf4a3 100644 --- a/server/src/router/v1/admin/shardClilocs.controller.js +++ b/server/src/router/v1/admin/shardClilocs.controller.js @@ -17,6 +17,7 @@ // to be told which file to convert. const clilocs = require('../../../model/shardClilocs/shardClilocs.model') +const market = require('../../../model/shardMarket/shardMarket.model') const activity = require('../../../model/activity/activity.model') const log = require('../../../utils/logger')('admin-shard-clilocs') @@ -50,6 +51,17 @@ async function importClilocs(req, res) { const force = !!req.body?.force const approve = !!req.body?.approve const result = await clilocs.refresh({ force, approve }) + + // The marketplace denormalizes resolved item names into + // shard_vendor_items.display_name, and the shard's market sweep will NOT + // re-send an unchanged shop just because the site learned what its items are + // called — so without this pass, an operator who imports clilocs after the + // first sweep keeps seeing item ids until every shop happens to change. + // Awaited (rather than fired and forgotten) so the panel's "imported" is + // honest about the names being live; the pass is a bounded walk of one table + // and never throws. + if (result.status === 'imported') await market.refreshDisplayNames() + await activity.log({ req, action: 'shard.clilocs.import', diff --git a/server/src/router/v1/public/shard.controller.js b/server/src/router/v1/public/shard.controller.js index a2ef47c..e94fc3f 100644 --- a/server/src/router/v1/public/shard.controller.js +++ b/server/src/router/v1/public/shard.controller.js @@ -11,6 +11,7 @@ const shardEvents = require('../../../model/shardEvents/shardEvents.model') const shardState = require('../../../model/shardState/shardState.model') +const shardMarket = require('../../../model/shardMarket/shardMarket.model') const uoLinkConfig = require('../../../model/uoLinkConfig/uoLinkConfig.model') const broadcast = require('../../../utils/shardBroadcast') const visibility = require('../../../utils/shardVisibility') @@ -299,6 +300,84 @@ async function getPointsBoard(req, res) { } } +// ── Marketplace (Protocol 3.0 vendor.listing) ────────────────────────────── +// +// The shard-wide player-vendor index. Served entirely from our own tables — the +// sidecar is never touched on this path — so shops stay browsable while the shard +// is down, labelled with how stale they may be. +// +// The staleness label is not decoration. The shard sweeps vendors round-robin, so +// a shop can legitimately be a full cycle behind; a page that implied live prices +// would send people to a vendor whose item sold twenty minutes ago. + +// The serial spelling the bridge uses everywhere: "0x" and hex. Constrained +// before it reaches the model, like SYSTEM_RE above. +const SERIAL_RE = /^0x[0-9A-Fa-f]{1,16}$/ + +const intParam = (value) => { + const n = Number.parseInt(value, 10) + return Number.isFinite(n) ? n : undefined +} + +// GET /public/shard/market — search the index. +// +// Returns LISTINGS, not vendors: "who sells a vanquishing kryss and for how much" +// is the question, and a vendor-shaped result would make every caller flatten the +// shops back out. +async function getMarket(req, res) { + try { + const page = await shardMarket.search({ + q: typeof req.query.q === 'string' ? req.query.q : '', + minPrice: intParam(req.query.minPrice), + maxPrice: intParam(req.query.maxPrice), + itemId: intParam(req.query.itemId), + map: typeof req.query.map === 'string' ? req.query.map : '', + region: typeof req.query.region === 'string' ? req.query.region : '', + sort: typeof req.query.sort === 'string' ? req.query.sort : 'price_asc', + limit: intParam(req.query.limit) ?? 50, + offset: intParam(req.query.offset) ?? 0, + }) + return res.json(await visibility.project('market', page, req)) + } catch (err) { + log.error('shard.getMarket', err) + return res.status(500).json({ message: 'Internal Server Error' }) + } +} + +// GET /public/shard/market/meta — index size, staleness, and the filter options +// (which facets and regions actually hold vendors). Separate from the search so +// the page can build its filters without running a query it will throw away. +async function getMarketMeta(req, res) { + try { + return res.json(await visibility.project('market', await shardMarket.meta(), req)) + } catch (err) { + log.error('shard.getMarketMeta', err) + return res.status(500).json({ message: 'Internal Server Error' }) + } +} + +// GET /public/shard/market/vendors/:serial — one shop and its listings. +// +// 404 for a serial the index has never seen, which also covers a vendor that has +// since been dismissed or hidden: to an anonymous caller "no such shop" is the +// only honest answer, and distinguishing the two would leak that a vendor exists +// but was hidden. +async function getMarketVendor(req, res) { + const { serial } = req.params + if (!SERIAL_RE.test(serial)) return res.status(400).json({ message: 'Invalid vendor serial.' }) + try { + const vendor = await shardMarket.getVendor(serial, { + limit: intParam(req.query.limit) ?? 250, + offset: intParam(req.query.offset) ?? 0, + }) + if (!vendor) return res.status(404).json({ message: 'Unknown vendor.' }) + return res.json(await visibility.project('market', vendor, req)) + } catch (err) { + log.error('shard.getMarketVendor', err) + return res.status(500).json({ message: 'Internal Server Error' }) + } +} + // GET /public/shard/features — the shard features THIS caller can actually see, // so the SPA (and the Android client) can hide nav entries instead of rendering // links that 403. Deliberately reports only what the viewer may reach: the list @@ -335,6 +414,9 @@ module.exports = { getRuleset, getPointsBoards, getPointsBoard, + getMarket, + getMarketMeta, + getMarketVendor, getFeatures, stream, } diff --git a/server/src/router/v1/public/shard.router.js b/server/src/router/v1/public/shard.router.js index fae040b..38fb66b 100644 --- a/server/src/router/v1/public/shard.router.js +++ b/server/src/router/v1/public/shard.router.js @@ -25,6 +25,7 @@ const { param, query } = require('express-validator') const shard = require('./shard.controller') const validate = require('../../../middleware/validate') +const { marketLimiter } = require('../../../middleware/rateLimit') const { requireFeature } = require('../../../utils/shardVisibility') const shardRouter = express.Router() @@ -167,6 +168,71 @@ shardRouter.get( /* #swagger.responses[404] = { description: 'The shard has never published that system' } */ shard.getPointsBoard, ) +// ── Marketplace ──────────────────────────────────────────────────────────── +// +// Rate-limited, unlike every other route in this file. These are the first +// genuinely expensive PUBLIC reads on the site — a LIKE scan plus a COUNT over +// what is typically the largest shard_* table, reachable with no session. +shardRouter.get( + '/market', + requireFeature('market'), + marketLimiter, + // #swagger.tags = ['Public · Shard'] + // #swagger.summary = 'Search the player-vendor marketplace' + // #swagger.description = 'Every priced listing on every player vendor the shard publishes — the same index the in-game Vendor Search gump reads, and it honours the same per-vendor opt-out, so a player who hid their shop in game is hidden here too. Results are LISTINGS, each carrying enough of its shop to be actionable. Served from the site\'s own tables (the sidecar is not touched), so it renders while the shard is down; `staleAt` is the oldest vendor row and the page must say how far behind the index can be — the shard sweeps vendors round-robin, so prices are inherently up to one full cycle old. Item names are resolved server-side against the cliloc table (docs/website/CLILOCS.md); on a shard that has not configured one, `displayName` is null and clients render the item id.' + // #swagger.parameters['q'] = { in: 'query', required: false, schema: { type: 'string' }, description: 'Substring match on the resolved item name or the item\'s own literal name (max 60 chars).' } + // #swagger.parameters['minPrice'] = { in: 'query', required: false, schema: { type: 'integer' }, description: 'Lowest price to include.' } + // #swagger.parameters['maxPrice'] = { in: 'query', required: false, schema: { type: 'integer' }, description: 'Highest price to include.' } + // #swagger.parameters['itemId'] = { in: 'query', required: false, schema: { type: 'integer' }, description: 'Exact ItemID (art id) match, for "more like this".' } + // #swagger.parameters['map'] = { in: 'query', required: false, schema: { type: 'string' }, description: 'Limit to one facet. Facet names come from the shard\'s own data; an unknown one returns an empty page.' } + // #swagger.parameters['region'] = { in: 'query', required: false, schema: { type: 'string' }, description: 'Limit to one named region.' } + // #swagger.parameters['sort'] = { in: 'query', required: false, schema: { type: 'string', enum: ['price_asc','price_desc','recent'] }, description: 'Default price_asc. `recent` orders by when the shop was last seen.' } + // #swagger.parameters['limit'] = { in: 'query', required: false, schema: { type: 'integer' }, description: 'Page size, 1..100 (default 50).' } + // #swagger.parameters['offset'] = { in: 'query', required: false, schema: { type: 'integer' }, description: 'Rows to skip (default 0).' } + /* #swagger.responses[200] = { description: 'A page of listings plus the unpaginated total and the staleness stamp', content: { "application/json": { schema: { $ref: "#/components/schemas/ShardMarketPage" } } } } */ + /* #swagger.responses[403] = { description: 'The market feature is gated above this caller', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */ + /* #swagger.responses[404] = { description: 'The market feature is disabled', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */ + /* #swagger.responses[429] = { description: 'Rate limited', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */ + query('q').optional({ values: 'falsy' }).isString().isLength({ max: 60 }), + query('minPrice').optional({ values: 'falsy' }).isInt({ min: 0, max: 999999999 }), + query('maxPrice').optional({ values: 'falsy' }).isInt({ min: 0, max: 999999999 }), + query('itemId').optional({ values: 'falsy' }).isInt({ min: 0, max: 65535 }), + query('map').optional({ values: 'falsy' }).isString().isLength({ max: 40 }), + query('region').optional({ values: 'falsy' }).isString().isLength({ max: 80 }), + query('sort').optional({ values: 'falsy' }).isIn(['price_asc', 'price_desc', 'recent']), + query('limit').optional().isInt({ min: 1, max: 100 }), + query('offset').optional().isInt({ min: 0, max: 100000 }), + validate, + shard.getMarket, +) +shardRouter.get( + '/market/meta', + requireFeature('market'), + // #swagger.tags = ['Public · Shard'] + // #swagger.summary = 'Marketplace size, staleness and filter options' + // #swagger.description = 'How many vendors and listings the index holds, how stale it may be (`staleAt` = the oldest vendor row, `freshAt` = the newest), and which facets and regions actually hold vendors — so a client can build its filters without running a search it will discard.' + /* #swagger.responses[200] = { description: 'Marketplace metadata', content: { "application/json": { schema: { $ref: "#/components/schemas/ShardMarketMeta" } } } } */ + shard.getMarketMeta, +) +shardRouter.get( + '/market/vendors/:serial', + requireFeature('market'), + marketLimiter, + // #swagger.tags = ['Public · Shard'] + // #swagger.summary = 'One player vendor and everything it is selling' + // #swagger.description = 'A single shop by its vendor serial, with its listings. `truncated` (and `total` exceeding `count`) means the shop holds more than the shard publishes per frame — a commodity reseller with thousands of stacks is a real thing, and the page says so rather than presenting a partial shop as complete. Returns 404 for a serial the index has never seen, which also covers a vendor since dismissed or hidden.' + /* #swagger.parameters['serial'] = { in: 'path', required: true, description: 'Vendor serial, e.g. 0x40001234', schema: { type: 'string' } } */ + // #swagger.parameters['limit'] = { in: 'query', required: false, schema: { type: 'integer' }, description: 'Listings to return, 1..500 (default 250).' } + // #swagger.parameters['offset'] = { in: 'query', required: false, schema: { type: 'integer' }, description: 'Listings to skip (default 0).' } + /* #swagger.responses[200] = { description: 'The vendor', content: { "application/json": { schema: { $ref: "#/components/schemas/ShardMarketVendor" } } } } */ + /* #swagger.responses[400] = { description: 'Malformed vendor serial' } */ + /* #swagger.responses[404] = { description: 'No such vendor in the index' } */ + param('serial').isString().isLength({ max: 20 }), + query('limit').optional().isInt({ min: 1, max: 500 }), + query('offset').optional().isInt({ min: 0, max: 100000 }), + validate, + shard.getMarketVendor, +) shardRouter.get( '/features', // #swagger.tags = ['Public · Shard'] diff --git a/server/src/server.js b/server/src/server.js index e2ecadd..e6e7afa 100644 --- a/server/src/server.js +++ b/server/src/server.js @@ -16,6 +16,7 @@ const revokedSessions = require('./model/revokedSessions/revokedSessions.model') const mobileAuthBridge = require('./model/mobileAuthBridge/mobileAuthBridge.model') const shardAtlas = require('./model/shardAtlas/shardAtlas.model') const shardClilocs = require('./model/shardClilocs/shardClilocs.model') +const shardMarket = require('./model/shardMarket/shardMarket.model') const createLogger = require('./utils/logger') const { evaluateBotInternalKey } = require('./utils/botInternalKey') const brand = require('./config/brand') @@ -95,7 +96,15 @@ async function start() { // hash-gated so an unchanged file costs one read, and best-effort so a missing // or wrong-format file never stops the site coming up — it just means item // names render as ids, which is what they did before the table existed. - await shardClilocs.refreshOnBoot() + const clilocResult = await shardClilocs.refreshOnBoot() + + // A cliloc import changes what item names RESOLVE to, and the marketplace + // stores those names denormalized (shard_vendor_items.display_name) so it can + // index and search them. The shard's market sweep will not re-send an unchanged + // shop just because the site learned what its items are called, so the backfill + // has to be pulled rather than waited for. Only after an actual import — the + // common boot is hash-gated to a no-op and must stay one. + if (clilocResult && clilocResult.status === 'imported') await shardMarket.refreshDisplayNames() const mode = await settings.get('site_mode') log.info(`site mode: ${String(mode || 'live').toUpperCase()}`) diff --git a/server/src/utils/shardIngest.js b/server/src/utils/shardIngest.js index 3b47329..a614b1f 100644 --- a/server/src/utils/shardIngest.js +++ b/server/src/utils/shardIngest.js @@ -15,6 +15,7 @@ const shardEventsModel = require('../model/shardEvents/shardEvents.model') const shardStateModel = require('../model/shardState/shardState.model') const shardLinksModel = require('../model/shardLinks/shardLinks.model') +const shardMarketModel = require('../model/shardMarket/shardMarket.model') const uoLinkConfigModel = require('../model/uoLinkConfig/uoLinkConfig.model') const broadcaster = require('./shardBroadcast') const pushDispatch = require('./pushDispatch') @@ -187,6 +188,19 @@ async function applyStateChange(event, deps) { case 'points.board': await shardState.upsertPointsBoard(event) return + // Player-vendor market index. Each frame is authoritative for one shop, so + // the model replaces that vendor's whole listing set rather than merging. + // + // NOT in LOGGED_KINDS, and this is the strongest case of the three v3 kinds: + // one frame carries up to 250 listings, the sweep re-emits a shop on any + // price change, and appending each of those to the event log would make + // shard_events mostly a price history nobody reads. The market IS the state. + case 'vendor.listing': + await deps.shardMarket.upsertVendor(event) + return + case 'vendor.listing.remove': + await deps.shardMarket.removeVendor(event.serial) + return case 'account.unlinked': // A player ran [unlink in game (or a site-side unlink echoed back) — drop // our local link mirror so attribution stops immediately. @@ -209,6 +223,7 @@ function resolveDeps(deps) { shardEvents: deps.shardEvents || shardEventsModel, shardState: deps.shardState || shardStateModel, shardLinks: deps.shardLinks || shardLinksModel, + shardMarket: deps.shardMarket || shardMarketModel, uoLinkConfig: deps.uoLinkConfig || uoLinkConfigModel, broadcast: deps.broadcast || broadcaster.broadcast, pushDispatch: deps.pushDispatch || pushDispatch.fromShardEvent, diff --git a/server/src/utils/shardVisibility.js b/server/src/utils/shardVisibility.js index edd04e7..4d65605 100644 --- a/server/src/utils/shardVisibility.js +++ b/server/src/utils/shardVisibility.js @@ -113,7 +113,21 @@ const FEATURES = { // Shop name, owner character name and vendor location are already globally // visible in-game via the stock Vendor Search gump, so publishing them is not // a new disclosure — but they stay configurable so an admin can tighten them. - market: { audience: 'anonymous', fields: { ownerName: 'anonymous', location: 'anonymous' } }, + // + // `ownerName` and `location` were pre-wired here by Part A, before the frame + // existed; both were re-checked against the real `vendor.listing` and both are + // genuine keys on it (unlike leaderboards' `characterName`, which was inert). + // `location` is a NESTED object on the wire and on the read model precisely so + // that one rule hides map, coordinates, region and house together — five flat + // keys would be five rules that drift apart. + // + // `ownerSerial` is listed alongside `ownerName` for the same reason `houses` + // lists both: an admin who hides the owner's name and is left with a serial + // that every other board resolves back to that name has not hidden anything. + market: { + audience: 'anonymous', + fields: { ownerName: 'anonymous', ownerSerial: 'anonymous', location: 'anonymous' }, + }, } const FEATURE_NAMES = Object.keys(FEATURES) diff --git a/server/src/utils/uoLinkClient.js b/server/src/utils/uoLinkClient.js index a00f232..1d503a6 100644 --- a/server/src/utils/uoLinkClient.js +++ b/server/src/utils/uoLinkClient.js @@ -135,6 +135,11 @@ const getRuleset = () => call('/ruleset') // under `boards`); the per-system read 404s for a system the shard never published. const getPoints = () => call('/points') const getPointsBoard = (system) => call(`/points/${encodeURIComponent(system)}`) +// Protocol 3.0: the player-vendor market index. The one PAGED sidecar read — a +// whole-world market does not fit in a response — so it answers with +// `{ vendors, total, limit, offset }` and the caller walks it (see uoLinkSocket). +const getMarket = ({ limit = 200, offset = 0 } = {}) => + call(`/market?limit=${encodeURIComponent(limit)}&offset=${encodeURIComponent(offset)}`) // ── Commands ────────────────────────────────────────────────────────────── const confirmLink = (code, websiteUserId) => @@ -201,6 +206,7 @@ module.exports = { getRuleset, getPoints, getPointsBoard, + getMarket, confirmLink, linkLookup, createAccount, diff --git a/server/src/utils/uoLinkSocket.js b/server/src/utils/uoLinkSocket.js index b934d8c..86c9ca9 100644 --- a/server/src/utils/uoLinkSocket.js +++ b/server/src/utils/uoLinkSocket.js @@ -63,6 +63,55 @@ async function ingestEach(events) { for (const ev of events) await shardIngest.ingest(ev, { fromBackfill: true }) } +// ── Market backfill ──────────────────────────────────────────────────────── +// +// The market is the only board that does not fit in one response, so /market is +// paged and this walks it. Two bounds, both deliberate: +// +// • MARKET_SNAPSHOT_MAX caps the walk. A pathological world (or a sidecar whose +// store was never pruned) must not be able to hang startup — backfill runs +// before the site is serving the live feed, so an unbounded loop here is +// downtime, not slowness. +// • The loop stops on a SHORT page as well as on `total`, because a concurrent +// sweep can shrink the index underneath the walk and paging to a stale total +// would spin. +// +// Vendors are upserted, never reconciled-by-replacement. A vendor absent from the +// snapshot is absent because the sidecar dropped it on vendor.listing.remove — +// which our own ingest already processed — so clearing the table first would only +// create a window where the market page is empty. +const MARKET_SNAPSHOT_MAX = 5000 +const MARKET_PAGE = 200 + +async function backfillMarket() { + let offset = 0 + let seen = 0 + + for (;;) { + const res = await uoLinkClient.getMarket({ limit: MARKET_PAGE, offset }) + if (!res.ok || !res.data || !Array.isArray(res.data.vendors)) return + + const page = res.data.vendors + if (page.length === 0) break + + await ingestEach(page) + seen += page.length + offset += page.length + + if (page.length < MARKET_PAGE) break + if (seen >= MARKET_SNAPSHOT_MAX) { + log.warn('market snapshot truncated at the safety cap', { + cap: MARKET_SNAPSHOT_MAX, + total: res.data.total, + }) + break + } + if (Number.isFinite(res.data.total) && offset >= res.data.total) break + } + + if (seen > 0) log.info('snapshotted player-vendor market from /market', { count: seen }) +} + // Pull recent events from the sidecar's own store and replay them through the // dispatcher (fromBackfill = no SSE re-broadcast). dedupe_key + INSERT IGNORE // make this idempotent, so overlap with what we already stored is harmless. @@ -106,6 +155,8 @@ async function backfill() { // which is the right answer for a month-scale standing. await snapshot(() => uoLinkClient.getPoints(), 'boards', ingestEach, 'snapshotted points boards from /points') + await backfillMarket() + const presence = await uoLinkClient.getPresence() if (presence.ok && presence.data && typeof presence.data.count === 'number') { await shardState.setPresence(presence.data) diff --git a/server/swagger/swagger-output.json b/server/swagger/swagger-output.json index 84f021b..766d02e 100644 --- a/server/swagger/swagger-output.json +++ b/server/swagger/swagger-output.json @@ -12008,6 +12008,226 @@ } } }, + "/api/v1/public/shard/market": { + "get": { + "tags": [ + "Public · Shard" + ], + "summary": "Search the player-vendor marketplace", + "description": "Every priced listing on every player vendor the shard publishes — the same index the in-game Vendor Search gump reads, and it honours the same per-vendor opt-out, so a player who hid their shop in game is hidden here too. Results are LISTINGS, each carrying enough of its shop to be actionable. Served from the site\\'s own tables (the sidecar is not touched), so it renders while the shard is down; `staleAt` is the oldest vendor row and the page must say how far behind the index can be — the shard sweeps vendors round-robin, so prices are inherently up to one full cycle old. Item names are resolved server-side against the cliloc table (docs/website/CLILOCS.md); on a shard that has not configured one, `displayName` is null and clients render the item id.", + "parameters": [ + { + "name": "itemId", + "in": "query", + "schema": { + "type": "string" + } + }, + { + "name": "map", + "in": "query", + "schema": { + "type": "string" + } + }, + { + "name": "region", + "in": "query", + "schema": { + "type": "string" + } + }, + { + "name": "sort", + "in": "query", + "schema": { + "type": "string" + } + }, + { + "name": "limit", + "in": "query", + "schema": { + "type": "string" + } + }, + { + "name": "offset", + "in": "query", + "schema": { + "type": "string" + } + }, + { + "name": "q", + "in": "query", + "required": false, + "schema": { + "type": "string" + }, + "description": "Substring match on the resolved item name or the item's own literal name (max 60 chars)." + }, + { + "name": "minPrice", + "in": "query", + "required": false, + "schema": { + "type": "integer" + }, + "description": "Lowest price to include." + }, + { + "name": "maxPrice", + "in": "query", + "required": false, + "schema": { + "type": "integer" + }, + "description": "Highest price to include." + } + ], + "responses": { + "200": { + "description": "A page of listings plus the unpaginated total and the staleness stamp", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ShardMarketPage" + } + } + } + }, + "400": { + "description": "Bad Request" + }, + "403": { + "description": "The market feature is gated above this caller", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/Error" + } + } + } + }, + "404": { + "description": "The market feature is disabled", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/Error" + } + } + } + }, + "429": { + "description": "Rate limited", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/Error" + } + } + } + }, + "500": { + "description": "Internal Server Error" + } + } + } + }, + "/api/v1/public/shard/market/meta": { + "get": { + "tags": [ + "Public · Shard" + ], + "summary": "Marketplace size, staleness and filter options", + "description": "How many vendors and listings the index holds, how stale it may be (`staleAt` = the oldest vendor row, `freshAt` = the newest), and which facets and regions actually hold vendors — so a client can build its filters without running a search it will discard.", + "responses": { + "200": { + "description": "Marketplace metadata", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ShardMarketMeta" + } + } + } + }, + "403": { + "description": "Forbidden" + }, + "404": { + "description": "Not Found" + }, + "500": { + "description": "Internal Server Error" + } + } + } + }, + "/api/v1/public/shard/market/vendors/{serial}": { + "get": { + "tags": [ + "Public · Shard" + ], + "summary": "One player vendor and everything it is selling", + "description": "A single shop by its vendor serial, with its listings. `truncated` (and `total` exceeding `count`) means the shop holds more than the shard publishes per frame — a commodity reseller with thousands of stacks is a real thing, and the page says so rather than presenting a partial shop as complete. Returns 404 for a serial the index has never seen, which also covers a vendor since dismissed or hidden.", + "parameters": [ + { + "name": "serial", + "in": "path", + "required": true, + "schema": { + "type": "string" + }, + "description": "Vendor serial, e.g. 0x40001234" + }, + { + "name": "limit", + "in": "query", + "required": false, + "schema": { + "type": "integer" + }, + "description": "Listings to return, 1..500 (default 250)." + }, + { + "name": "offset", + "in": "query", + "required": false, + "schema": { + "type": "integer" + }, + "description": "Listings to skip (default 0)." + } + ], + "responses": { + "200": { + "description": "The vendor", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ShardMarketVendor" + } + } + } + }, + "400": { + "description": "Malformed vendor serial" + }, + "403": { + "description": "Forbidden" + }, + "404": { + "description": "No such vendor in the index" + }, + "500": { + "description": "Internal Server Error" + } + } + } + }, "/api/v1/public/shard/online": { "get": { "tags": [ @@ -18685,6 +18905,809 @@ } } }, + "ShardMarketLocation": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "object" + }, + "nullable": { + "type": "boolean", + "example": true + }, + "description": { + "type": "string", + "example": "Where a vendor is standing. ONE nested object rather than flat map/x/y/region because it is one admin-configurable field (`market.location`) — the whole object is omitted when that field is gated above the caller." + }, + "properties": { + "type": "object", + "properties": { + "map": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "string" + }, + "nullable": { + "type": "boolean", + "example": true + }, + "example": { + "type": "string", + "example": "Trammel" + } + } + }, + "x": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "integer" + }, + "nullable": { + "type": "boolean", + "example": true + }, + "example": { + "type": "number", + "example": 1421 + } + } + }, + "y": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "integer" + }, + "nullable": { + "type": "boolean", + "example": true + }, + "example": { + "type": "number", + "example": 1699 + } + } + }, + "z": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "integer" + }, + "nullable": { + "type": "boolean", + "example": true + }, + "example": { + "type": "number", + "example": 0 + } + } + }, + "region": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "string" + }, + "nullable": { + "type": "boolean", + "example": true + }, + "example": { + "type": "string", + "example": "Britain" + } + } + }, + "house": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "string" + }, + "nullable": { + "type": "boolean", + "example": true + }, + "example": { + "type": "string", + "example": "Darrow's Villa" + }, + "description": { + "type": "string", + "example": "The house SIGN's name, not the house type. Null for a vendor standing outside one." + } + } + } + } + } + } + }, + "ShardMarketListing": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "object" + }, + "description": { + "type": "string", + "example": "One priced listing on a player vendor, carrying enough of its shop to be actionable without a second request." + }, + "properties": { + "type": "object", + "properties": { + "serial": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "string" + }, + "example": { + "type": "string", + "example": "0x40012ABC" + } + } + }, + "itemId": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "integer" + }, + "example": { + "type": "number", + "example": 3922 + }, + "description": { + "type": "string", + "example": "ItemID (the art/graphic id)." + } + } + }, + "hue": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "integer" + }, + "example": { + "type": "number", + "example": 0 + } + } + }, + "amount": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "integer" + }, + "example": { + "type": "number", + "example": 1 + } + } + }, + "price": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "integer" + }, + "example": { + "type": "number", + "example": 25000 + } + } + }, + "name": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "string" + }, + "nullable": { + "type": "boolean", + "example": true + }, + "description": { + "type": "string", + "example": "The item's own literal name, set by a player. Null for most items." + } + } + }, + "cliloc": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "integer" + }, + "nullable": { + "type": "boolean", + "example": true + }, + "example": { + "type": "number", + "example": 1023721 + }, + "description": { + "type": "string", + "example": "The item's LabelNumber." + } + } + }, + "displayName": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "string" + }, + "nullable": { + "type": "boolean", + "example": true + }, + "example": { + "type": "string", + "example": "quarter staff" + }, + "description": { + "type": "string", + "example": "Resolved server-side from `name` (preferred, being player-set and more specific) else `cliloc`. Null on a shard with no cliloc table configured — render the item id." + } + } + }, + "child": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "boolean" + }, + "example": { + "type": "boolean", + "example": false + }, + "description": { + "type": "string", + "example": "Priced by an enclosing container rather than itself, exactly as the in-game Vendor Search reports it." + } + } + }, + "vendor": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "object" + }, + "properties": { + "type": "object", + "properties": { + "serial": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "string" + }, + "example": { + "type": "string", + "example": "0x40001234" + } + } + }, + "shopName": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "string" + }, + "nullable": { + "type": "boolean", + "example": true + }, + "example": { + "type": "string", + "example": "Darrow's Bargains" + } + } + }, + "ownerSerial": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "string" + }, + "nullable": { + "type": "boolean", + "example": true + }, + "example": { + "type": "string", + "example": "0x1A2B" + }, + "description": { + "type": "string", + "example": "Omitted when the market `ownerSerial` field is gated above the caller." + } + } + }, + "ownerName": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "string" + }, + "nullable": { + "type": "boolean", + "example": true + }, + "example": { + "type": "string", + "example": "Darrow" + }, + "description": { + "type": "string", + "example": "Omitted when the market `ownerName` field is gated above the caller." + } + } + }, + "location": { + "$ref": "#/components/schemas/ShardMarketLocation" + }, + "updatedAt": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "string" + }, + "format": { + "type": "string", + "example": "date-time" + }, + "description": { + "type": "string", + "example": "When the shard last published this shop." + } + } + } + } + } + } + } + } + } + } + }, + "ShardMarketPage": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "object" + }, + "description": { + "type": "string", + "example": "A page of marketplace listings plus the unpaginated total and the staleness stamp." + }, + "properties": { + "type": "object", + "properties": { + "listings": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "array" + }, + "items": { + "$ref": "#/components/schemas/ShardMarketListing" + } + } + }, + "total": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "integer" + }, + "example": { + "type": "number", + "example": 1284 + }, + "description": { + "type": "string", + "example": "Matching listings, ignoring paging." + } + } + }, + "limit": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "integer" + }, + "example": { + "type": "number", + "example": 50 + } + } + }, + "offset": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "integer" + }, + "example": { + "type": "number", + "example": 0 + } + } + }, + "vendors": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "integer" + }, + "example": { + "type": "number", + "example": 137 + }, + "description": { + "type": "string", + "example": "Vendors in the whole index." + } + } + }, + "staleAt": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "string" + }, + "format": { + "type": "string", + "example": "date-time" + }, + "nullable": { + "type": "boolean", + "example": true + }, + "description": { + "type": "string", + "example": "The OLDEST vendor row. The shard sweeps vendors round-robin, so the index can be a full cycle behind and a client must say so rather than implying live prices." + } + } + } + } + } + } + }, + "ShardMarketVendor": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "object" + }, + "description": { + "type": "string", + "example": "One player vendor and its listings." + }, + "properties": { + "type": "object", + "properties": { + "serial": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "string" + }, + "example": { + "type": "string", + "example": "0x40001234" + } + } + }, + "shopName": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "string" + }, + "nullable": { + "type": "boolean", + "example": true + }, + "example": { + "type": "string", + "example": "Darrow's Bargains" + } + } + }, + "ownerSerial": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "string" + }, + "nullable": { + "type": "boolean", + "example": true + } + } + }, + "ownerName": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "string" + }, + "nullable": { + "type": "boolean", + "example": true + }, + "example": { + "type": "string", + "example": "Darrow" + } + } + }, + "location": { + "$ref": "#/components/schemas/ShardMarketLocation" + }, + "count": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "integer" + }, + "example": { + "type": "number", + "example": 250 + }, + "description": { + "type": "string", + "example": "Listings the shard published for this shop." + } + } + }, + "total": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "integer" + }, + "example": { + "type": "number", + "example": 3104 + }, + "description": { + "type": "string", + "example": "Listings the shop actually holds." + } + } + }, + "truncated": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "boolean" + }, + "example": { + "type": "boolean", + "example": true + }, + "description": { + "type": "string", + "example": "`total` exceeds `count` — the shop holds more than the shard publishes per frame." + } + } + }, + "updatedAt": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "string" + }, + "format": { + "type": "string", + "example": "date-time" + } + } + }, + "items": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "array" + }, + "items": { + "$ref": "#/components/schemas/ShardMarketListing" + } + } + } + } + } + } + }, + "ShardMarketMeta": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "object" + }, + "description": { + "type": "string", + "example": "Marketplace size, staleness and the filter options a client needs to build its UI." + }, + "properties": { + "type": "object", + "properties": { + "vendors": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "integer" + }, + "example": { + "type": "number", + "example": 137 + } + } + }, + "items": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "integer" + }, + "example": { + "type": "number", + "example": 18422 + } + } + }, + "staleAt": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "string" + }, + "format": { + "type": "string", + "example": "date-time" + }, + "nullable": { + "type": "boolean", + "example": true + } + } + }, + "freshAt": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "string" + }, + "format": { + "type": "string", + "example": "date-time" + }, + "nullable": { + "type": "boolean", + "example": true + } + } + }, + "maps": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "array" + }, + "items": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "string" + } + } + }, + "example": { + "type": "array", + "example": [ + "Felucca", + "Trammel" + ], + "items": { + "type": "string" + } + }, + "description": { + "type": "string", + "example": "Facets that actually hold vendors. From the shard's own data — never a hardcoded list." + } + } + }, + "regions": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "array" + }, + "items": { + "type": "object", + "properties": { + "type": { + "type": "string", + "example": "string" + } + } + }, + "example": { + "type": "array", + "example": [ + "Britain", + "Luna" + ], + "items": { + "type": "string" + } + } + } + } + } + } + } + }, "ShardFeatures": { "type": "object", "properties": { diff --git a/server/swagger/swagger.js b/server/swagger/swagger.js index 4f56d65..d1d6ba7 100644 --- a/server/swagger/swagger.js +++ b/server/swagger/swagger.js @@ -910,6 +910,97 @@ const doc = { updatedAt: { type: 'string', format: 'date-time' }, }, }, + ShardMarketLocation: { + type: 'object', + nullable: true, + description: + "Where a vendor is standing. ONE nested object rather than flat map/x/y/region because it is one admin-configurable field (`market.location`) — the whole object is omitted when that field is gated above the caller.", + properties: { + map: { type: 'string', nullable: true, example: 'Trammel' }, + x: { type: 'integer', nullable: true, example: 1421 }, + y: { type: 'integer', nullable: true, example: 1699 }, + z: { type: 'integer', nullable: true, example: 0 }, + region: { type: 'string', nullable: true, example: 'Britain' }, + house: { type: 'string', nullable: true, example: "Darrow's Villa", description: "The house SIGN's name, not the house type. Null for a vendor standing outside one." }, + }, + }, + ShardMarketListing: { + type: 'object', + description: + 'One priced listing on a player vendor, carrying enough of its shop to be actionable without a second request.', + properties: { + serial: { type: 'string', example: '0x40012ABC' }, + itemId: { type: 'integer', example: 3922, description: 'ItemID (the art/graphic id).' }, + hue: { type: 'integer', example: 0 }, + amount: { type: 'integer', example: 1 }, + price: { type: 'integer', example: 25000 }, + name: { type: 'string', nullable: true, description: "The item's own literal name, set by a player. Null for most items." }, + cliloc: { type: 'integer', nullable: true, example: 1023721, description: "The item's LabelNumber." }, + displayName: { + type: 'string', + nullable: true, + example: 'quarter staff', + description: 'Resolved server-side from `name` (preferred, being player-set and more specific) else `cliloc`. Null on a shard with no cliloc table configured — render the item id.', + }, + child: { type: 'boolean', example: false, description: 'Priced by an enclosing container rather than itself, exactly as the in-game Vendor Search reports it.' }, + vendor: { + type: 'object', + properties: { + serial: { type: 'string', example: '0x40001234' }, + shopName: { type: 'string', nullable: true, example: "Darrow's Bargains" }, + ownerSerial: { type: 'string', nullable: true, example: '0x1A2B', description: 'Omitted when the market `ownerSerial` field is gated above the caller.' }, + ownerName: { type: 'string', nullable: true, example: 'Darrow', description: 'Omitted when the market `ownerName` field is gated above the caller.' }, + location: { $ref: '#/components/schemas/ShardMarketLocation' }, + updatedAt: { type: 'string', format: 'date-time', description: 'When the shard last published this shop.' }, + }, + }, + }, + }, + ShardMarketPage: { + type: 'object', + description: 'A page of marketplace listings plus the unpaginated total and the staleness stamp.', + properties: { + listings: { type: 'array', items: { $ref: '#/components/schemas/ShardMarketListing' } }, + total: { type: 'integer', example: 1284, description: 'Matching listings, ignoring paging.' }, + limit: { type: 'integer', example: 50 }, + offset: { type: 'integer', example: 0 }, + vendors: { type: 'integer', example: 137, description: 'Vendors in the whole index.' }, + staleAt: { + type: 'string', + format: 'date-time', + nullable: true, + description: 'The OLDEST vendor row. The shard sweeps vendors round-robin, so the index can be a full cycle behind and a client must say so rather than implying live prices.', + }, + }, + }, + ShardMarketVendor: { + type: 'object', + description: 'One player vendor and its listings.', + properties: { + serial: { type: 'string', example: '0x40001234' }, + shopName: { type: 'string', nullable: true, example: "Darrow's Bargains" }, + ownerSerial: { type: 'string', nullable: true }, + ownerName: { type: 'string', nullable: true, example: 'Darrow' }, + location: { $ref: '#/components/schemas/ShardMarketLocation' }, + count: { type: 'integer', example: 250, description: 'Listings the shard published for this shop.' }, + total: { type: 'integer', example: 3104, description: 'Listings the shop actually holds.' }, + truncated: { type: 'boolean', example: true, description: '`total` exceeds `count` — the shop holds more than the shard publishes per frame.' }, + updatedAt: { type: 'string', format: 'date-time' }, + items: { type: 'array', items: { $ref: '#/components/schemas/ShardMarketListing' } }, + }, + }, + ShardMarketMeta: { + type: 'object', + description: 'Marketplace size, staleness and the filter options a client needs to build its UI.', + properties: { + vendors: { type: 'integer', example: 137 }, + items: { type: 'integer', example: 18422 }, + staleAt: { type: 'string', format: 'date-time', nullable: true }, + freshAt: { type: 'string', format: 'date-time', nullable: true }, + maps: { type: 'array', items: { type: 'string' }, example: ['Felucca', 'Trammel'], description: "Facets that actually hold vendors. From the shard's own data — never a hardcoded list." }, + regions: { type: 'array', items: { type: 'string' }, example: ['Britain', 'Luna'] }, + }, + }, ShardFeatures: { type: 'object', description: diff --git a/server/test/shardIngest.market.test.js b/server/test/shardIngest.market.test.js new file mode 100644 index 0000000..82d0bd0 --- /dev/null +++ b/server/test/shardIngest.market.test.js @@ -0,0 +1,114 @@ +const { test, beforeEach } = require('node:test') +const assert = require('node:assert/strict') + +const shardIngest = require('../src/utils/shardIngest') + +// Protocol 3.0 vendor.listing / vendor.listing.remove routing. Same shape as +// shardIngest.points.test.js: stubbed deps, asserting where the dispatcher sends +// the frame and whether it is appended to the event log. +function makeDeps() { + const calls = { upserts: [], removes: [], appended: [], broadcast: [] } + const noop = async () => {} + return { + calls, + shardEvents: { append: async (row) => { calls.appended.push(row); return true } }, + shardState: { + // Present so any stray routing is a harmless no-op rather than a crash. + clearOnline: noop, upsertOnline: noop, setOffline: noop, upsertHouse: noop, + addEconomySample: noop, setRuleset: noop, upsertPointsBoard: noop, + }, + shardMarket: { + upsertVendor: async (ev) => { calls.upserts.push(ev) }, + removeVendor: async (serial) => { calls.removes.push(serial) }, + }, + shardLinks: { removeByAccount: noop }, + uoLinkConfig: { recordStatus: noop }, + broadcast: (ev) => { calls.broadcast.push(ev) }, + pushDispatch: async () => {}, + log: { warn() {}, info() {}, error() {} }, + } +} + +const FRAME = { + kind: 'vendor.listing', + t: 1000, + serial: '0x40001234', + shopName: "Darrow's Bargains", + ownerSerial: '0x1A2B', + ownerName: 'Darrow', + location: { map: 'Trammel', x: 1421, y: 1699, z: 0, region: 'Britain', house: "Darrow's Villa" }, + count: 2, + total: 2, + truncated: false, + items: [ + { serial: '0x40012ABC', itemId: 3922, hue: 0, amount: 1, price: 25000, name: null, cliloc: 1023721 }, + { serial: '0x40012ABD', itemId: 7026, hue: 1157, amount: 3, price: 500, name: 'a shard sigil', cliloc: 1041243 }, + ], +} + +beforeEach(() => shardIngest.reset()) + +test('vendor.listing routes to the market model with the whole frame', async () => { + const deps = makeDeps() + await shardIngest.ingest(FRAME, deps) + assert.equal(deps.calls.upserts.length, 1) + const stored = deps.calls.upserts[0] + assert.equal(stored.serial, '0x40001234') + assert.equal(stored.location.region, 'Britain') + assert.equal(stored.items.length, 2) +}) + +test('vendor.listing.remove routes to removeVendor with the serial', async () => { + const deps = makeDeps() + await shardIngest.ingest({ kind: 'vendor.listing.remove', t: 2000, serial: '0x40001234' }, deps) + assert.deepEqual(deps.calls.removes, ['0x40001234']) + assert.equal(deps.calls.upserts.length, 0) +}) + +// The market IS the state. One frame carries up to 250 listings and the sweep +// re-emits a shop on any price change, so logging would turn shard_events into a +// price history nobody reads — the strongest case of the three v3 kinds. +test('neither market kind is appended to the event log', async () => { + const deps = makeDeps() + const a = await shardIngest.ingest(FRAME, deps) + const b = await shardIngest.ingest({ kind: 'vendor.listing.remove', serial: '0x40001234' }, deps) + assert.equal(a.logged, false) + assert.equal(b.logged, false) + assert.equal(deps.calls.appended.length, 0) + assert.equal(shardIngest.LOGGED_KINDS.has('vendor.listing'), false) + assert.equal(shardIngest.LOGGED_KINDS.has('vendor.listing.remove'), false) +}) + +// Broadcast is unconditional at this layer — whether it actually reaches anyone +// is shardBroadcast's call, and the market feature ships with its stream off. +test('vendor.listing is handed to the broadcaster', async () => { + const deps = makeDeps() + await shardIngest.ingest(FRAME, deps) + assert.equal(deps.calls.broadcast.length, 1) + assert.equal(deps.calls.broadcast[0].kind, 'vendor.listing') +}) + +test('a backfilled vendor.listing still stores but does not broadcast', async () => { + const deps = makeDeps() + await shardIngest.ingest(FRAME, { ...deps, fromBackfill: true }) + assert.equal(deps.calls.upserts.length, 1) + assert.equal(deps.calls.broadcast.length, 0) +}) + +// The reconnect backfill replays the whole index through this path, so a single +// bad vendor must not abort it. +test('an upsertVendor failure does not throw or stop the broadcast', async () => { + const deps = makeDeps() + deps.shardMarket.upsertVendor = async () => { throw new Error('db down') } + const r = await shardIngest.ingest(FRAME, deps) + assert.equal(r.logged, false) + assert.equal(deps.calls.broadcast.length, 1) +}) + +// Each vendor is its own row; the frame is authoritative for that vendor only. +test('two vendors are stored independently', async () => { + const deps = makeDeps() + await shardIngest.ingest(FRAME, deps) + await shardIngest.ingest({ ...FRAME, serial: '0x40009999', shopName: 'Second Shop' }, deps) + assert.deepEqual(deps.calls.upserts.map((v) => v.serial), ['0x40001234', '0x40009999']) +}) diff --git a/server/test/shardMarket.model.test.js b/server/test/shardMarket.model.test.js new file mode 100644 index 0000000..0c5760b --- /dev/null +++ b/server/test/shardMarket.model.test.js @@ -0,0 +1,260 @@ +// Point the DB at a closed port BEFORE requiring anything that builds a pool. +// Nothing here reaches the database: these are the model's PURE parts — the +// flatten/shape rules the frame passes through on the way in and out — plus the +// visibility projection over the shapes they produce. +process.env.DB_HOST = '127.0.0.1' +process.env.DB_PORT = '59999' + +const { test, after } = require('node:test') +const assert = require('node:assert/strict') + +const market = require('../src/model/shardMarket/shardMarket.model') +const clilocs = require('../src/model/shardClilocs/shardClilocs.model') +const clilocDb = require('../src/model/shardClilocs/shardClilocs.db') +const visibility = require('../src/utils/shardVisibility') +const db = require('../src/utils/db') + +after(() => db.close()) + +// Stand in for the cliloc table. Without this each unresolved lookup waits out +// the pool's 10s acquire timeout against the dead port — the model swallows the +// failure exactly as it would in production (an operator who never converted a +// cliloc file is in a supported state), so the RESULT is the same either way; +// this only stops the suite spending half a minute proving it. +const TABLE = new Map([[1023721, 'quarter staff']]) +clilocDb.lookup = async (numbers) => + numbers.filter((n) => TABLE.has(n)).map((n) => ({ number: n, text: TABLE.get(n) })) + +const FRAME = { + kind: 'vendor.listing', + t: 1000, + serial: '0x40001234', + shopName: "Darrow's Bargains", + ownerSerial: '0x1A2B', + ownerName: 'Darrow', + location: { map: 'Trammel', x: 1421, y: 1699, z: 0, region: 'Britain', house: "Darrow's Villa" }, + count: 2, + total: 2, + truncated: false, + items: [], +} + +// ── flattenFrame ─────────────────────────────────────────────────────────── + +test('flattenFrame lifts the nested location into columns', () => { + const v = market.flattenFrame(FRAME) + assert.equal(v.serial, '0x40001234') + assert.equal(v.map, 'Trammel') + assert.equal(v.x, 1421) + assert.equal(v.region, 'Britain') + assert.equal(v.house, "Darrow's Villa") +}) + +// A vendor standing in the street has no house, and a frame from an older plugin +// may have no location at all. Neither is an error. +test('flattenFrame tolerates a missing location entirely', () => { + const v = market.flattenFrame({ serial: '0x1', shopName: null }) + assert.equal(v.map, null) + assert.equal(v.x, null) + assert.equal(v.region, null) + assert.equal(v.house, null) +}) + +// `total` is what the SHOP holds; `count` is what the frame carried. A truncated +// shop must not report its published slice as its size, or the page says +// "showing 250 of 250" for a vendor holding three thousand stacks. +test('flattenFrame keeps the shop total separate from the published count', () => { + const v = market.flattenFrame({ ...FRAME, count: 250, total: 3104, truncated: true }) + assert.equal(v.itemTotal, 3104) + assert.equal(v.truncated, true) +}) + +// An older plugin sends no `total`. Falling back to `count` is right — it is the +// only number available and it is correct whenever nothing was truncated. +test('flattenFrame falls back to count when total is absent', () => { + const v = market.flattenFrame({ ...FRAME, count: 7, total: undefined }) + assert.equal(v.itemTotal, 7) +}) + +test('flattenFrame clips over-length strings rather than letting the insert fail', () => { + const v = market.flattenFrame({ ...FRAME, ownerName: 'x'.repeat(200) }) + assert.equal(v.ownerName.length, 64) +}) + +// ── shapeItems ───────────────────────────────────────────────────────────── +// +// resolveMany never throws and, with no cliloc table reachable, resolves nothing +// — which is exactly the state of a shard whose operator never converted one, so +// these run against the real function rather than a stub. + +test('shapeItems prefers the item\'s literal name over its cliloc', async () => { + const items = await market.shapeItems({ + items: [{ serial: '0x1', itemId: 3922, price: 100, name: 'a shard sigil', cliloc: 1023721 }], + }) + assert.equal(items[0].displayName, 'a shard sigil') + // The cliloc is kept regardless, so a later import can still re-resolve it. + assert.equal(items[0].cliloc, 1023721) +}) + +test('shapeItems resolves the cliloc when the item has no literal name', async () => { + const items = await market.shapeItems({ + items: [{ serial: '0x1', itemId: 3922, price: 100, name: null, cliloc: 1023721 }], + }) + assert.equal(items[0].displayName, 'quarter staff') +}) + +// The supported state for a shard whose operator never converted a cliloc file: +// no name, not a fabricated one. Clients render the item id, exactly as they did +// before the table existed. +test('shapeItems leaves displayName null for an unknown cliloc', async () => { + const items = await market.shapeItems({ + items: [{ serial: '0x1', itemId: 3922, price: 100, name: null, cliloc: 9999999 }], + }) + assert.equal(items[0].displayName, null) +}) + +// Unpriced rows are inventory, not listings. The shard drops them too; enforcing +// it here as well means a plugin that stops doing so cannot put un-buyable rows +// on the market page. +test('shapeItems drops unpriced listings', async () => { + const items = await market.shapeItems({ + items: [ + { serial: '0x1', itemId: 1, price: 0 }, + { serial: '0x2', itemId: 2, price: -1 }, + { serial: '0x3', itemId: 3, price: 5 }, + ], + }) + assert.deepEqual(items.map((i) => i.serial), ['0x3']) +}) + +test('shapeItems caps a pathological frame', async () => { + const many = Array.from({ length: market.MAX_ITEMS_PER_VENDOR + 50 }, (_, i) => ({ + serial: `0x${i}`, + itemId: 1, + price: 1, + })) + const items = await market.shapeItems({ items: many }) + assert.equal(items.length, market.MAX_ITEMS_PER_VENDOR) +}) + +test('shapeItems tolerates a frame with no items array', async () => { + assert.deepEqual(await market.shapeItems({}), []) +}) + +// ── Visibility projection ────────────────────────────────────────────────── +// +// The regression that matters. Part A pre-wired `market.ownerName` and +// `market.location` before the frame existed, and the sibling rule it pre-wired +// for leaderboards (`characterName`) turned out to be INERT because projectValue +// matches literal JSON keys. These assert the market rules actually bite — on the +// read model AND on the wire frame, which is why both carry the same key names. + +const config = visibility.compileDefaults() + +const listing = market.shapeListing({ + serial: '0x40012ABC', + item_id: 3922, + hue: 0, + amount: 1, + price: 25000, + name: null, + cliloc: 1023721, + display_name: 'quarter staff', + child: 0, + vendor_serial: '0x40001234', + shop_name: "Darrow's Bargains", + owner_serial: '0x1A2B', + owner_name: 'Darrow', + map: 'Trammel', + x: 1421, + y: 1699, + z: 0, + region: 'Britain', + house: "Darrow's Villa", + updated_at: new Date(0), +}) + +test('market defaults expose owner and location (they are already public in game)', () => { + const out = visibility.projectFeature('market', listing, 'anonymous', config) + assert.equal(out.vendor.ownerName, 'Darrow') + assert.equal(out.vendor.location.region, 'Britain') +}) + +test('tightening market.ownerName hides it from below that rung', () => { + const tightened = { ...config, market: { ...config.market, fields: { ...config.market.fields, ownerName: 'staff' } } } + const anon = visibility.projectFeature('market', listing, 'anonymous', tightened) + const staff = visibility.projectFeature('market', listing, 'staff', tightened) + assert.equal('ownerName' in anon.vendor, false) + assert.equal(staff.vendor.ownerName, 'Darrow') + // The shop name is a separate field and must survive — hiding the owner is not + // the same as hiding the shop. + assert.equal(anon.vendor.shopName, "Darrow's Bargains") +}) + +// The whole reason `location` is one nested object: a single rule has to take the +// facet, the coordinates, the region and the house together. Five flat keys would +// be five rules that drift apart. +test('tightening market.location hides the whole location object at once', () => { + const tightened = { ...config, market: { ...config.market, fields: { ...config.market.fields, location: 'player' } } } + const anon = visibility.projectFeature('market', listing, 'anonymous', tightened) + const player = visibility.projectFeature('market', listing, 'player', tightened) + assert.equal('location' in anon.vendor, false) + assert.equal(player.vendor.location.map, 'Trammel') +}) + +// The same rules must bite on the LIVE frame, not just the stored read model — +// the market's SSE stream is off by default but an admin can turn it on, and a +// field rule that only worked on one of the two paths is exactly the leak §3.6.1 +// records. +test('the same rules apply to the raw vendor.listing frame', () => { + const tightened = { ...config, market: { ...config.market, fields: { ...config.market.fields, ownerName: 'admin', location: 'admin' } } } + const out = visibility.projectFeature('market', FRAME, 'anonymous', tightened) + assert.equal('ownerName' in out, false) + assert.equal('location' in out, false) + assert.equal(out.shopName, "Darrow's Bargains") +}) + +// Rule 1 is not configurable and does not depend on the market rules at all: a +// frame that somehow carried an account name must never publish it. +test('acct and webId are stripped from a market payload regardless of config', () => { + const out = visibility.projectFeature( + 'market', + { serial: '0x1', ownerAcct: 'darrow', webId: '42', shopName: 'Shop' }, + 'staff', + config, + ) + assert.equal('ownerAcct' in out, false) + assert.equal('webId' in out, false) + assert.equal(out.shopName, 'Shop') +}) + +// Both kinds must be attributed to a feature, or rule 2 makes them admin-only by +// omission — which would be a silent failure rather than a loud one. +test('both market kinds are mapped to the market feature', () => { + assert.equal(visibility.KIND_FEATURE.get('vendor.listing'), 'market') + assert.equal(visibility.KIND_FEATURE.get('vendor.listing.remove'), 'market') +}) + +// The market's live firehose is off by default (a page of whole vendor +// inventories is the site's biggest bandwidth item and no page needs it live), +// but the REST reads are unaffected — which is what `visibleKinds` ignoring the +// stream flag encodes. +test('market kinds are stream-suppressed by default but still readable', () => { + assert.equal(visibility.DEFAULT_STREAM_OFF.has('market'), true) + assert.equal(visibility.kindVisibleTo('vendor.listing', 'anonymous', config), false) + assert.equal(visibility.PUBLIC_KINDS.has('vendor.listing'), false) + assert.ok(visibility.visibleKinds('anonymous', config).includes('vendor.listing')) +}) + +test('an admin who enables the stream gets the frames', () => { + const on = { ...config, market: { ...config.market, stream: true } } + assert.equal(visibility.kindVisibleTo('vendor.listing', 'anonymous', on), true) +}) + +// Guards the stub above against silently doing nothing: if the model stopped +// going through db.lookup, every shapeItems assertion would still "pass" by +// resolving nothing, which is also what a real miss looks like. +test('the cliloc resolver is the path shapeItems resolves through', async () => { + const found = await clilocs.resolveMany([1023721]) + assert.equal(found.get(1023721), 'quarter staff') +}) -- 2.49.1