feat(shard): the player-vendor marketplace
Protocol 3.0 §8, the website half. Ingests vendor.listing / vendor.listing.remove
into shard_vendors + shard_vendor_items, serves a searchable public API over
them, and ships /site/market and /site/market/vendors/:serial.
Three things the pages have to say out loud, all consequences of how the data is
gathered:
- The prices are NOT live. The shard sweeps vendors round-robin, so a shop can be
a full cycle behind. The banner is driven by the OLDEST vendor row, not the
newest — the one stale shop is the one that wastes somebody's trip.
- A shop can be truncated. `total` exceeding `count` means the shop holds more
than the shard publishes per frame; the vendor page says "showing 250 of 3,104"
rather than presenting a partial shop as complete.
- An item may have no name. On a shard with no cliloc table the honest render is
the item id, never an invented label.
## The pre-wired visibility rules, re-checked
Part A pre-wired market.ownerName and market.location before the frame existed,
and the sibling rule it pre-wired for leaderboards (`characterName`) turned out
to be INERT because projectValue matches literal JSON keys. Both market rules
were checked against the real frame this time:
- `ownerName` is a real key. Kept.
- `location` is a real key ONLY because the frame nests it. Flat map/x/y/region
would have made the rule match nothing — the same failure, one part later. It
is nested on the wire and on the read model so one rule hides the facet, the
coordinates, the region and the house together; five flat keys would be five
rules that drift apart.
- `ownerSerial` was ADDED. An admin who hides the owner's name and leaves a
serial that the leaderboards and guild boards resolve back to that same name
has not hidden anything.
Tests assert all three bite, on the stored read model AND on the raw frame —
the market's SSE stream is off by default but an admin can turn it on, and a rule
that worked on only one path is exactly the leak §3.6.1 records.
## Notable
- **No payload column on shard_vendors**, unlike shard_points_boards next door.
The board's top-N is a fixed-size list read whole; here the items ARE the
searchable rows, so they are normalized and nothing is left worth duplicating.
- **display_name is denormalized at ingest** (literal name preferred over the
cliloc — a player set it, so it is more specific). Resolving at query time
would put the cliloc table on the hot path and make search-by-name impossible.
Because the shard's diff sweep will not re-send an unchanged shop just because
the site learned what its items are called, a cliloc import now triggers a bulk
re-resolution — 50 ms per thousand rows, never throws.
- **updated_at is written explicitly** on every upsert. MariaDB does not fire ON
UPDATE CURRENT_TIMESTAMP when every column is written back unchanged, and a
shop re-published identically is still freshly confirmed — without this the
staleness banner would age a perfectly current shop forever.
- **LIKE wildcards in `q` are escaped.** `%` and `_` are LIKE metacharacters, not
SQL ones, so parameterization does not neutralize them: `?q=%` would otherwise
match every listing on the shard.
- **Rate-limited** (60/min/IP), the only limited public read. Every other public
GET is an indexed lookup of bounded size; this is a LIKE scan plus a COUNT over
the largest shard_* table, anonymous by default.
- Reconnect backfill pages /market, bounded by MARKET_SNAPSHOT_MAX = 5000 and
stopping on a short page as well as on `total`, so a concurrent sweep shrinking
the index cannot spin the walk.
## How it was tested
673 server tests pass (27 new). Client builds clean; swagger-output.json,
routes.manifest.json and routes.guards.json regenerated.
Verified full-stack against the live MariaDB and a real shard, not only units:
- 27 real vendors / 1,040 listings swept off the ServUO tree, through the Rust
sidecar, into the site — names resolving through the cliloc table ("longsword",
"katana"), real facets and regions in the filters.
- `?q=sword` 682, `?q=%` and `?q=_` **0** (the escape), map/region/price/sort
filters, paging, and the vendor detail route.
- Visibility live: fields gated to staff vanish for an anonymous caller while
shopName and price survive; audience=player 403s; enabled=0 404s; and
/shard/features correctly drops `market` so the nav hides it.
- Re-publishing a shop smaller leaves no orphan items; an identical re-publish
moves updated_at.
- The limiter fires (38x200 then 32x429 on a 70-request burst).
Not covered by an automated test: the two React pages are presentational and this
repo's client suite covers pure-logic modules only. They were driven against the
live API above, but not rendered in a DOM harness.
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -118,6 +118,19 @@ const passwordResetConfirmLimiter = makeLimiter({
|
||||
message: 'Too many attempts. Please try again later.',
|
||||
})
|
||||
|
||||
// The player-vendor market search. The first genuinely expensive PUBLIC endpoint
|
||||
// on the site: every call is a LIKE scan plus a COUNT over the listings table,
|
||||
// which on a large shard is the biggest table there is, and it is anonymous by
|
||||
// default. Generous for a human browsing shops (a typed search is debounced to
|
||||
// one request, and paging is a click), tight enough that it cannot be used as a
|
||||
// cheap way to load the database.
|
||||
const marketLimiter = makeLimiter({
|
||||
windowMs: 60 * 1000,
|
||||
max: 60,
|
||||
label: 'market',
|
||||
message: 'Too many searches. Please slow down.',
|
||||
})
|
||||
|
||||
// CSP violation reports. Unauthenticated by necessity (browsers send them with no
|
||||
// session), and every accepted report writes a log line — so an attacker who can get
|
||||
// a victim to load a page could otherwise use it as a log-flood amplifier. Generous
|
||||
@@ -141,5 +154,6 @@ module.exports = {
|
||||
mobileSsoExchangeLimiter,
|
||||
passwordResetRequestLimiter,
|
||||
passwordResetConfirmLimiter,
|
||||
marketLimiter,
|
||||
cspReportLimiter,
|
||||
}
|
||||
|
||||
299
server/src/model/shardMarket/shardMarket.db.js
Normal file
299
server/src/model/shardMarket/shardMarket.db.js
Normal file
@@ -0,0 +1,299 @@
|
||||
const { pool, query } = require('../../utils/db')
|
||||
|
||||
// Raw SQL for the player-vendor market index (Protocol 3.0 vendor.listing).
|
||||
//
|
||||
// Two tables, both INGEST-OWNED: `shard_vendors` (one row per shop) and
|
||||
// `shard_vendor_items` (one row per priced listing). Nothing else in the codebase
|
||||
// writes to either. No foreign keys, consistent with every other shard_* table.
|
||||
|
||||
// Insert batch size for one vendor's listings. A shop is capped at
|
||||
// MarketMaxListings (250 by default) on the shard side, so in practice this is
|
||||
// one batch — it exists for the operator who raised that cap.
|
||||
const BATCH = 500
|
||||
|
||||
// LIKE wildcards in user input. `%` and `_` are not special to the parameterized
|
||||
// query — they are special to LIKE itself — so a search for "50% off" would
|
||||
// otherwise match everything containing "50" and a search for "_" would match
|
||||
// every single-character name. Escaped with a backslash, which is MariaDB's
|
||||
// default LIKE escape (no ESCAPE clause needed).
|
||||
const likeTerm = (q) => `%${String(q).replace(/[\\%_]/g, (c) => `\\${c}`)}%`
|
||||
|
||||
/**
|
||||
* Replace one vendor's whole row and listing set, in one transaction.
|
||||
*
|
||||
* Delete-then-insert rather than a diff, because the frame is AUTHORITATIVE for
|
||||
* that vendor: the shard's sweep only emits a shop whose contents, prices or
|
||||
* location moved, and when it does it sends the whole shop. Reconciling it item
|
||||
* by item would be more code for the same result and would leave sold items
|
||||
* behind on any path the reconciliation missed.
|
||||
*
|
||||
* All-or-nothing matters here for a specific reason: the two writes are "the
|
||||
* shop" and "what is in it", and a failure between them leaves a shop advertising
|
||||
* an inventory it no longer has (or none at all) — visibly wrong on the page, and
|
||||
* indistinguishable from a genuinely empty shop.
|
||||
*/
|
||||
async function replaceVendor(vendor, items) {
|
||||
const conn = await pool.getConnection()
|
||||
try {
|
||||
await conn.beginTransaction()
|
||||
|
||||
await conn.query(
|
||||
`INSERT INTO shard_vendors
|
||||
(serial, shop_name, owner_serial, owner_name, map, x, y, z, region, house,
|
||||
item_count, item_total, truncated, t)
|
||||
VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?)
|
||||
ON DUPLICATE KEY UPDATE shop_name = VALUES(shop_name), owner_serial = VALUES(owner_serial),
|
||||
owner_name = VALUES(owner_name), map = VALUES(map), x = VALUES(x), y = VALUES(y),
|
||||
z = VALUES(z), region = VALUES(region), house = VALUES(house),
|
||||
item_count = VALUES(item_count), item_total = VALUES(item_total),
|
||||
truncated = VALUES(truncated), t = VALUES(t),
|
||||
-- Touched explicitly rather than left to ON UPDATE CURRENT_TIMESTAMP:
|
||||
-- MariaDB does not fire that when every column is written back
|
||||
-- unchanged, and a shop that is re-published identically is still
|
||||
-- FRESHLY CONFIRMED. Without this the staleness banner would age a
|
||||
-- perfectly current shop forever.
|
||||
updated_at = CURRENT_TIMESTAMP`,
|
||||
[
|
||||
vendor.serial,
|
||||
vendor.shopName ?? null,
|
||||
vendor.ownerSerial ?? null,
|
||||
vendor.ownerName ?? null,
|
||||
vendor.map ?? null,
|
||||
Number.isFinite(vendor.x) ? vendor.x : null,
|
||||
Number.isFinite(vendor.y) ? vendor.y : null,
|
||||
Number.isFinite(vendor.z) ? vendor.z : null,
|
||||
vendor.region ?? null,
|
||||
vendor.house ?? null,
|
||||
items.length,
|
||||
Number.isFinite(vendor.itemTotal) ? vendor.itemTotal : items.length,
|
||||
vendor.truncated ? 1 : 0,
|
||||
Number.isFinite(vendor.t) ? vendor.t : null,
|
||||
],
|
||||
)
|
||||
|
||||
await conn.query('DELETE FROM shard_vendor_items WHERE vendor_serial = ?', [vendor.serial])
|
||||
|
||||
const rows = items.map((i) => [
|
||||
vendor.serial,
|
||||
i.serial,
|
||||
i.itemId,
|
||||
i.hue,
|
||||
i.amount,
|
||||
i.price,
|
||||
i.name,
|
||||
i.cliloc,
|
||||
i.displayName,
|
||||
i.child ? 1 : 0,
|
||||
])
|
||||
|
||||
for (let i = 0; i < rows.length; i += BATCH) {
|
||||
await conn.batch(
|
||||
`INSERT INTO shard_vendor_items
|
||||
(vendor_serial, serial, item_id, hue, amount, price, name, cliloc, display_name, child)
|
||||
VALUES (?,?,?,?,?,?,?,?,?,?)`,
|
||||
rows.slice(i, i + BATCH),
|
||||
)
|
||||
}
|
||||
|
||||
await conn.commit()
|
||||
return { items: rows.length }
|
||||
} catch (err) {
|
||||
await conn.rollback().catch(() => {})
|
||||
throw err
|
||||
} finally {
|
||||
conn.release()
|
||||
}
|
||||
}
|
||||
|
||||
/** Drop one vendor and its listings (vendor.listing.remove). */
|
||||
async function removeVendor(serial) {
|
||||
const conn = await pool.getConnection()
|
||||
try {
|
||||
await conn.beginTransaction()
|
||||
await conn.query('DELETE FROM shard_vendor_items WHERE vendor_serial = ?', [serial])
|
||||
await conn.query('DELETE FROM shard_vendors WHERE serial = ?', [serial])
|
||||
await conn.commit()
|
||||
} catch (err) {
|
||||
await conn.rollback().catch(() => {})
|
||||
throw err
|
||||
} finally {
|
||||
conn.release()
|
||||
}
|
||||
}
|
||||
|
||||
// ── Search ─────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// The unit of a search RESULT is a listing, not a vendor: "who sells a vanquishing
|
||||
// kryss and for how much" is the question, and answering it per vendor would make
|
||||
// the caller flatten the shops back out. The vendor's columns ride along on the
|
||||
// join so a result row is self-contained.
|
||||
|
||||
function searchWhere({ q, minPrice, maxPrice, itemId, map, region }) {
|
||||
const where = ['i.price > 0']
|
||||
const params = []
|
||||
|
||||
if (q) {
|
||||
// Both the resolved display name and the item's own literal, because an item
|
||||
// with a player-set name (most of what is actually worth searching for on a
|
||||
// player-run shard) may have a generic cliloc.
|
||||
where.push('(i.display_name LIKE ? OR i.name LIKE ?)')
|
||||
params.push(likeTerm(q), likeTerm(q))
|
||||
}
|
||||
if (Number.isFinite(minPrice)) {
|
||||
where.push('i.price >= ?')
|
||||
params.push(minPrice)
|
||||
}
|
||||
if (Number.isFinite(maxPrice)) {
|
||||
where.push('i.price <= ?')
|
||||
params.push(maxPrice)
|
||||
}
|
||||
if (Number.isFinite(itemId)) {
|
||||
where.push('i.item_id = ?')
|
||||
params.push(itemId)
|
||||
}
|
||||
if (map) {
|
||||
where.push('v.map = ?')
|
||||
params.push(map)
|
||||
}
|
||||
if (region) {
|
||||
where.push('v.region = ?')
|
||||
params.push(region)
|
||||
}
|
||||
|
||||
return { sql: `WHERE ${where.join(' AND ')}`, params }
|
||||
}
|
||||
|
||||
// Whitelisted, because this interpolates into the statement. `recent` sorts by
|
||||
// the vendor's freshness, which is the only way to see what has just been listed
|
||||
// on a shard whose sweep is minutes wide.
|
||||
const SORTS = {
|
||||
price_asc: 'i.price ASC, i.id ASC',
|
||||
price_desc: 'i.price DESC, i.id ASC',
|
||||
recent: 'v.updated_at DESC, i.id ASC',
|
||||
}
|
||||
|
||||
async function searchListings({ q, minPrice, maxPrice, itemId, map, region, sort, limit, offset }) {
|
||||
const { sql, params } = searchWhere({ q, minPrice, maxPrice, itemId, map, region })
|
||||
const order = SORTS[sort] || SORTS.price_asc
|
||||
|
||||
const rows = await query(
|
||||
`SELECT i.serial, i.item_id, i.hue, i.amount, i.price, i.name, i.cliloc, i.display_name, i.child,
|
||||
v.serial AS vendor_serial, v.shop_name, v.owner_serial, v.owner_name,
|
||||
v.map, v.x, v.y, v.z, v.region, v.house, v.updated_at
|
||||
FROM shard_vendor_items i
|
||||
JOIN shard_vendors v ON v.serial = i.vendor_serial
|
||||
${sql}
|
||||
ORDER BY ${order}
|
||||
LIMIT ? OFFSET ?`,
|
||||
[...params, limit, offset],
|
||||
)
|
||||
|
||||
const counted = await query(
|
||||
`SELECT COUNT(*) AS n
|
||||
FROM shard_vendor_items i
|
||||
JOIN shard_vendors v ON v.serial = i.vendor_serial
|
||||
${sql}`,
|
||||
params,
|
||||
)
|
||||
|
||||
return { rows, total: Number(counted[0]?.n) || 0 }
|
||||
}
|
||||
|
||||
async function getVendor(serial) {
|
||||
const rows = await query(
|
||||
`SELECT serial, shop_name, owner_serial, owner_name, map, x, y, z, region, house,
|
||||
item_count, item_total, truncated, t, updated_at
|
||||
FROM shard_vendors WHERE serial = ?`,
|
||||
[serial],
|
||||
)
|
||||
return rows[0] || null
|
||||
}
|
||||
|
||||
async function listVendorItems(serial, { limit, offset }) {
|
||||
return query(
|
||||
`SELECT serial, item_id, hue, amount, price, name, cliloc, display_name, child
|
||||
FROM shard_vendor_items
|
||||
WHERE vendor_serial = ?
|
||||
ORDER BY price ASC, id ASC
|
||||
LIMIT ? OFFSET ?`,
|
||||
[serial, limit, offset],
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* What the market page's header needs: how big the index is, and how stale it may
|
||||
* be. `staleAt` is the OLDEST vendor row — the round-robin sweep means a shop can
|
||||
* be a full cycle behind, and the page says so rather than implying live prices.
|
||||
*/
|
||||
async function meta() {
|
||||
const rows = await query(
|
||||
`SELECT COUNT(*) AS vendors, MIN(updated_at) AS stale_at, MAX(updated_at) AS fresh_at
|
||||
FROM shard_vendors`,
|
||||
)
|
||||
const items = await query('SELECT COUNT(*) AS n FROM shard_vendor_items')
|
||||
return {
|
||||
vendors: Number(rows[0]?.vendors) || 0,
|
||||
items: Number(items[0]?.n) || 0,
|
||||
staleAt: rows[0]?.stale_at || null,
|
||||
freshAt: rows[0]?.fresh_at || null,
|
||||
}
|
||||
}
|
||||
|
||||
/** The distinct facets and regions holding vendors — drives the page's filters. */
|
||||
async function listPlaces() {
|
||||
const maps = await query(
|
||||
'SELECT DISTINCT map FROM shard_vendors WHERE map IS NOT NULL ORDER BY map',
|
||||
)
|
||||
const regions = await query(
|
||||
'SELECT DISTINCT region FROM shard_vendors WHERE region IS NOT NULL ORDER BY region',
|
||||
)
|
||||
return { maps: maps.map((r) => r.map), regions: regions.map((r) => r.region) }
|
||||
}
|
||||
|
||||
// ── Cliloc re-resolution ───────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* One page of listings whose name still needs resolving, for the bulk pass that
|
||||
* runs after a cliloc import.
|
||||
*
|
||||
* Keyed on `id > after` rather than OFFSET: the pass updates the very rows it is
|
||||
* scanning, and an OFFSET walk over a table being rewritten skips rows. Every
|
||||
* row with a cliloc is re-read, not just the unresolved ones, because an import
|
||||
* can also CHANGE a name — a shard overlay relabelling a stock item is the whole
|
||||
* reason overlays exist.
|
||||
*/
|
||||
async function listResolvableItems(after, limit) {
|
||||
return query(
|
||||
`SELECT id, cliloc, name, display_name
|
||||
FROM shard_vendor_items
|
||||
WHERE cliloc IS NOT NULL AND cliloc > 0 AND id > ?
|
||||
ORDER BY id
|
||||
LIMIT ?`,
|
||||
[after, limit],
|
||||
)
|
||||
}
|
||||
|
||||
/** Write back a batch of re-resolved display names. */
|
||||
async function updateDisplayNames(pairs) {
|
||||
if (pairs.length === 0) return 0
|
||||
const conn = await pool.getConnection()
|
||||
try {
|
||||
await conn.batch('UPDATE shard_vendor_items SET display_name = ? WHERE id = ?', pairs)
|
||||
return pairs.length
|
||||
} finally {
|
||||
conn.release()
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
replaceVendor,
|
||||
removeVendor,
|
||||
searchListings,
|
||||
getVendor,
|
||||
listVendorItems,
|
||||
meta,
|
||||
listPlaces,
|
||||
listResolvableItems,
|
||||
updateDisplayNames,
|
||||
likeTerm,
|
||||
}
|
||||
329
server/src/model/shardMarket/shardMarket.model.js
Normal file
329
server/src/model/shardMarket/shardMarket.model.js
Normal file
@@ -0,0 +1,329 @@
|
||||
// ── Player-vendor market index (Protocol 3.0 vendor.listing) ───────────────
|
||||
//
|
||||
// The shard-wide shop index: what every player vendor is selling, for how much,
|
||||
// and where it is standing. This is the website's half of the search the in-game
|
||||
// Vendor Search gump offers — the same data, the same opt-out, reachable without
|
||||
// logging in to the game.
|
||||
//
|
||||
// Ingest is per-vendor and authoritative: the shard's round-robin sweep emits one
|
||||
// `vendor.listing` frame per shop whose contents, prices or location moved, and
|
||||
// the frame is the whole shop (see docs/link/v3.md §8 and BridgeMarket.cs). This
|
||||
// module normalizes it into shard_vendors + shard_vendor_items and, crucially,
|
||||
// resolves each listing's cliloc to a DISPLAY NAME on the way in — a search for
|
||||
// "kryss" is a search over names, and the shard only ever sends numbers.
|
||||
|
||||
const db = require('./shardMarket.db')
|
||||
const clilocs = require('../shardClilocs/shardClilocs.model')
|
||||
const log = require('../../utils/logger')('shard-market')
|
||||
|
||||
// Defense in depth on top of the shard's own MarketMaxListings cap. The shard is
|
||||
// trusted, but it is a separately-versioned component: a frame from a plugin
|
||||
// whose cap was raised (or a shard running modified scripts) must not be able to
|
||||
// turn one ingest into an unbounded transaction.
|
||||
const MAX_ITEMS_PER_VENDOR = 5000
|
||||
|
||||
// Column widths in schema.sql. Truncating here rather than letting MariaDB do it
|
||||
// keeps the behavior the same in strict mode, where an over-length value is an
|
||||
// ERROR and would fail the whole vendor rather than shortening one name.
|
||||
const MAX_NAME = 160
|
||||
const MAX_SHOP = 160
|
||||
const MAX_OWNER = 64
|
||||
const MAX_MAP = 40
|
||||
const MAX_REGION = 80
|
||||
const MAX_SERIAL = 20
|
||||
|
||||
const clip = (value, max) => {
|
||||
if (value == null) return null
|
||||
const s = String(value)
|
||||
return s.length > max ? s.slice(0, max) : s
|
||||
}
|
||||
|
||||
const int = (value, fallback = 0) => {
|
||||
const n = Number(value)
|
||||
return Number.isFinite(n) ? Math.trunc(n) : fallback
|
||||
}
|
||||
|
||||
// ── Ingest ─────────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Flatten one `vendor.listing` frame into the row shapes the DB layer wants.
|
||||
*
|
||||
* `location` arrives as a nested object rather than flat map/x/y/region, and that
|
||||
* shape is load-bearing rather than cosmetic: the visibility projection matches
|
||||
* literal JSON keys, so ONE `market.location` rule can hide a vendor's
|
||||
* whereabouts only if `location` is a single key on both the live frame and the
|
||||
* stored read model. Flattening it here for storage and re-nesting it on read is
|
||||
* what keeps that true on both paths.
|
||||
*
|
||||
* Exported for tests — it is the part with rules in it, and it is pure.
|
||||
*/
|
||||
function flattenFrame(ev) {
|
||||
const loc = (ev && ev.location) || {}
|
||||
return {
|
||||
serial: clip(ev.serial, MAX_SERIAL),
|
||||
shopName: clip(ev.shopName, MAX_SHOP),
|
||||
ownerSerial: clip(ev.ownerSerial, MAX_SERIAL),
|
||||
ownerName: clip(ev.ownerName, MAX_OWNER),
|
||||
map: clip(loc.map, MAX_MAP),
|
||||
x: Number.isFinite(loc.x) ? Math.trunc(loc.x) : null,
|
||||
y: Number.isFinite(loc.y) ? Math.trunc(loc.y) : null,
|
||||
z: Number.isFinite(loc.z) ? Math.trunc(loc.z) : null,
|
||||
region: clip(loc.region, MAX_REGION),
|
||||
house: clip(loc.house, MAX_SHOP),
|
||||
// What the SHOP holds, which is not what the frame carries when it was
|
||||
// truncated. Kept apart so the page can say "showing 250 of 3,104" rather
|
||||
// than presenting a partial shop as a complete one.
|
||||
itemTotal: int(ev.total, int(ev.count, 0)),
|
||||
truncated: ev.truncated === true,
|
||||
t: Number.isFinite(ev.t) ? ev.t : null,
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve each listing's display name.
|
||||
*
|
||||
* Order of preference is the item's own literal `name` first, then the cliloc.
|
||||
* That is the opposite of what "resolve the id" suggests and it is right: a
|
||||
* literal name only exists because a player set one ("Bob's vanquishing kryss"),
|
||||
* and it is strictly more specific than the generic cliloc the item still
|
||||
* carries.
|
||||
*
|
||||
* One batched lookup per frame rather than per item; `resolveMany` is cached and
|
||||
* never throws, so a cliloc table that is missing entirely just leaves
|
||||
* `displayName` null and the page renders item ids, exactly as it did before the
|
||||
* table existed.
|
||||
*/
|
||||
async function shapeItems(ev) {
|
||||
const raw = Array.isArray(ev.items) ? ev.items.slice(0, MAX_ITEMS_PER_VENDOR) : []
|
||||
|
||||
const wanted = raw
|
||||
.map((i) => int(i && i.cliloc, 0))
|
||||
.filter((n) => n > 0)
|
||||
|
||||
const names = await clilocs.resolveMany(wanted)
|
||||
|
||||
return raw
|
||||
.filter((i) => i && i.serial)
|
||||
.map((i) => {
|
||||
const literal = clip(i.name, MAX_NAME)
|
||||
const cliloc = int(i.cliloc, 0) || null
|
||||
return {
|
||||
serial: clip(i.serial, MAX_SERIAL),
|
||||
itemId: int(i.itemId, 0),
|
||||
hue: int(i.hue, 0),
|
||||
amount: int(i.amount, 1),
|
||||
price: int(i.price, 0),
|
||||
name: literal,
|
||||
cliloc,
|
||||
displayName: literal || (cliloc ? clip(names.get(cliloc) ?? null, MAX_NAME) : null),
|
||||
child: i.child === true,
|
||||
}
|
||||
})
|
||||
// Unpriced rows are inventory, not listings. The shard already drops them;
|
||||
// this is the same rule enforced where the table is written, so a plugin that
|
||||
// stops enforcing it cannot put un-buyable rows on the market page.
|
||||
.filter((i) => i.price > 0)
|
||||
}
|
||||
|
||||
/** Ingest one `vendor.listing` frame. */
|
||||
async function upsertVendor(ev) {
|
||||
if (!ev || !ev.serial) return
|
||||
const vendor = flattenFrame(ev)
|
||||
const items = await shapeItems(ev)
|
||||
await db.replaceVendor(vendor, items)
|
||||
}
|
||||
|
||||
/** Ingest one `vendor.listing.remove` frame. */
|
||||
async function removeVendor(serial) {
|
||||
if (!serial) return
|
||||
await db.removeVendor(String(serial).slice(0, MAX_SERIAL))
|
||||
}
|
||||
|
||||
// ── Read models ────────────────────────────────────────────────────────────
|
||||
//
|
||||
// `location` is re-nested (see flattenFrame) so the stored read model and the
|
||||
// live wire frame present the same keys to the visibility projection.
|
||||
|
||||
const place = (r) => ({
|
||||
map: r.map,
|
||||
x: r.x,
|
||||
y: r.y,
|
||||
z: r.z,
|
||||
region: r.region,
|
||||
house: r.house,
|
||||
})
|
||||
|
||||
// A listing as the search returns it: the item, plus enough of its shop to be
|
||||
// actionable without a second request. `displayName` falls back to nothing rather
|
||||
// than to a fabricated "Item 3922" — the client decides how to render an
|
||||
// unresolved id, and inventing a name here would make it indistinguishable from
|
||||
// a real one.
|
||||
const shapeListing = (r) => ({
|
||||
serial: r.serial,
|
||||
itemId: r.item_id,
|
||||
hue: r.hue,
|
||||
amount: r.amount,
|
||||
price: Number(r.price),
|
||||
name: r.name,
|
||||
cliloc: r.cliloc,
|
||||
displayName: r.display_name,
|
||||
child: !!r.child,
|
||||
vendor: {
|
||||
serial: r.vendor_serial,
|
||||
shopName: r.shop_name,
|
||||
ownerSerial: r.owner_serial,
|
||||
ownerName: r.owner_name,
|
||||
location: place(r),
|
||||
updatedAt: r.updated_at,
|
||||
},
|
||||
})
|
||||
|
||||
const shapeVendor = (r) => ({
|
||||
serial: r.serial,
|
||||
shopName: r.shop_name,
|
||||
ownerSerial: r.owner_serial,
|
||||
ownerName: r.owner_name,
|
||||
location: place(r),
|
||||
count: r.item_count,
|
||||
total: r.item_total,
|
||||
truncated: !!r.truncated,
|
||||
updatedAt: r.updated_at,
|
||||
})
|
||||
|
||||
const shapeItem = (r) => ({
|
||||
serial: r.serial,
|
||||
itemId: r.item_id,
|
||||
hue: r.hue,
|
||||
amount: r.amount,
|
||||
price: Number(r.price),
|
||||
name: r.name,
|
||||
cliloc: r.cliloc,
|
||||
displayName: r.display_name,
|
||||
child: !!r.child,
|
||||
})
|
||||
|
||||
/**
|
||||
* Search the index. Returns a page of LISTINGS (not vendors) plus the
|
||||
* unpaginated total and the staleness stamp the page's banner needs.
|
||||
*/
|
||||
async function search({
|
||||
q = '',
|
||||
minPrice,
|
||||
maxPrice,
|
||||
itemId,
|
||||
map = '',
|
||||
region = '',
|
||||
sort = 'price_asc',
|
||||
limit = 50,
|
||||
offset = 0,
|
||||
} = {}) {
|
||||
const { rows, total } = await db.searchListings({
|
||||
q: q.trim(),
|
||||
minPrice: Number.isFinite(minPrice) ? minPrice : undefined,
|
||||
maxPrice: Number.isFinite(maxPrice) ? maxPrice : undefined,
|
||||
itemId: Number.isFinite(itemId) ? itemId : undefined,
|
||||
map: map.trim(),
|
||||
region: region.trim(),
|
||||
sort,
|
||||
limit,
|
||||
offset,
|
||||
})
|
||||
|
||||
const info = await db.meta()
|
||||
|
||||
return {
|
||||
listings: rows.map(shapeListing),
|
||||
total,
|
||||
limit,
|
||||
offset,
|
||||
// Repeated on every search response rather than left to a separate /meta
|
||||
// call: the banner that says how old these prices are must age with the
|
||||
// results it labels, and a client that fetched it once would keep showing a
|
||||
// stamp from before the page it is looking at.
|
||||
staleAt: info.staleAt,
|
||||
vendors: info.vendors,
|
||||
}
|
||||
}
|
||||
|
||||
/** One shop and its listings. `null` when the index has never seen that serial. */
|
||||
async function getVendor(serial, { limit = 250, offset = 0 } = {}) {
|
||||
const row = await db.getVendor(serial)
|
||||
if (!row) return null
|
||||
const items = await db.listVendorItems(serial, { limit, offset })
|
||||
return { ...shapeVendor(row), items: items.map(shapeItem) }
|
||||
}
|
||||
|
||||
/** Index size, staleness, and the facet/region filter options. */
|
||||
async function meta() {
|
||||
const [info, places] = await Promise.all([db.meta(), db.listPlaces()])
|
||||
return { ...info, ...places }
|
||||
}
|
||||
|
||||
// ── Cliloc re-resolution ───────────────────────────────────────────────────
|
||||
|
||||
// Batch size for the post-import pass. Big enough that a 40k-row table is ~40
|
||||
// round trips, small enough that a single batch is not a long-held connection.
|
||||
const RESOLVE_BATCH = 1000
|
||||
|
||||
/**
|
||||
* Re-resolve every listing's display name against the current cliloc table.
|
||||
*
|
||||
* Called after a cliloc import, and it has to be: the market's diff sweep will
|
||||
* NOT re-send an unchanged shop just because the site learned what its items are
|
||||
* called, so without this an operator who configures clilocs after the first
|
||||
* market sweep sees item ids until every shop happens to change. That is the same
|
||||
* class of staleness the spawn atlas avoids by re-parsing on boot — here the
|
||||
* source of truth for names moved, not the data.
|
||||
*
|
||||
* Never throws. It is a cosmetic backfill on a table that is already serving; a
|
||||
* failure means names stay as they were, which is exactly the pre-import state.
|
||||
*/
|
||||
async function refreshDisplayNames() {
|
||||
let after = 0
|
||||
let scanned = 0
|
||||
let changed = 0
|
||||
|
||||
try {
|
||||
for (;;) {
|
||||
const rows = await db.listResolvableItems(after, RESOLVE_BATCH)
|
||||
if (rows.length === 0) break
|
||||
|
||||
after = rows[rows.length - 1].id
|
||||
scanned += rows.length
|
||||
|
||||
const names = await clilocs.resolveMany(rows.map((r) => Number(r.cliloc)))
|
||||
|
||||
const pairs = []
|
||||
for (const row of rows) {
|
||||
// The literal name still wins, so a re-resolution never overwrites a
|
||||
// player-set name with the generic cliloc behind it.
|
||||
const next = row.name
|
||||
? clip(row.name, MAX_NAME)
|
||||
: clip(names.get(Number(row.cliloc)) ?? null, MAX_NAME)
|
||||
if (next !== row.display_name) pairs.push([next, row.id])
|
||||
}
|
||||
|
||||
changed += await db.updateDisplayNames(pairs)
|
||||
}
|
||||
|
||||
if (changed > 0) log.info('market display names refreshed', { scanned, changed })
|
||||
return { scanned, changed }
|
||||
} catch (err) {
|
||||
log.warn('market display-name refresh failed', { message: err.message, scanned, changed })
|
||||
return { scanned, changed, error: err.message }
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
upsertVendor,
|
||||
removeVendor,
|
||||
search,
|
||||
getVendor,
|
||||
meta,
|
||||
refreshDisplayNames,
|
||||
flattenFrame,
|
||||
shapeItems,
|
||||
shapeListing,
|
||||
shapeVendor,
|
||||
MAX_ITEMS_PER_VENDOR,
|
||||
}
|
||||
@@ -17,6 +17,7 @@
|
||||
// to be told which file to convert.
|
||||
|
||||
const clilocs = require('../../../model/shardClilocs/shardClilocs.model')
|
||||
const market = require('../../../model/shardMarket/shardMarket.model')
|
||||
const activity = require('../../../model/activity/activity.model')
|
||||
|
||||
const log = require('../../../utils/logger')('admin-shard-clilocs')
|
||||
@@ -50,6 +51,17 @@ async function importClilocs(req, res) {
|
||||
const force = !!req.body?.force
|
||||
const approve = !!req.body?.approve
|
||||
const result = await clilocs.refresh({ force, approve })
|
||||
|
||||
// The marketplace denormalizes resolved item names into
|
||||
// shard_vendor_items.display_name, and the shard's market sweep will NOT
|
||||
// re-send an unchanged shop just because the site learned what its items are
|
||||
// called — so without this pass, an operator who imports clilocs after the
|
||||
// first sweep keeps seeing item ids until every shop happens to change.
|
||||
// Awaited (rather than fired and forgotten) so the panel's "imported" is
|
||||
// honest about the names being live; the pass is a bounded walk of one table
|
||||
// and never throws.
|
||||
if (result.status === 'imported') await market.refreshDisplayNames()
|
||||
|
||||
await activity.log({
|
||||
req,
|
||||
action: 'shard.clilocs.import',
|
||||
|
||||
@@ -11,6 +11,7 @@
|
||||
|
||||
const shardEvents = require('../../../model/shardEvents/shardEvents.model')
|
||||
const shardState = require('../../../model/shardState/shardState.model')
|
||||
const shardMarket = require('../../../model/shardMarket/shardMarket.model')
|
||||
const uoLinkConfig = require('../../../model/uoLinkConfig/uoLinkConfig.model')
|
||||
const broadcast = require('../../../utils/shardBroadcast')
|
||||
const visibility = require('../../../utils/shardVisibility')
|
||||
@@ -299,6 +300,84 @@ async function getPointsBoard(req, res) {
|
||||
}
|
||||
}
|
||||
|
||||
// ── Marketplace (Protocol 3.0 vendor.listing) ──────────────────────────────
|
||||
//
|
||||
// The shard-wide player-vendor index. Served entirely from our own tables — the
|
||||
// sidecar is never touched on this path — so shops stay browsable while the shard
|
||||
// is down, labelled with how stale they may be.
|
||||
//
|
||||
// The staleness label is not decoration. The shard sweeps vendors round-robin, so
|
||||
// a shop can legitimately be a full cycle behind; a page that implied live prices
|
||||
// would send people to a vendor whose item sold twenty minutes ago.
|
||||
|
||||
// The serial spelling the bridge uses everywhere: "0x" and hex. Constrained
|
||||
// before it reaches the model, like SYSTEM_RE above.
|
||||
const SERIAL_RE = /^0x[0-9A-Fa-f]{1,16}$/
|
||||
|
||||
const intParam = (value) => {
|
||||
const n = Number.parseInt(value, 10)
|
||||
return Number.isFinite(n) ? n : undefined
|
||||
}
|
||||
|
||||
// GET /public/shard/market — search the index.
|
||||
//
|
||||
// Returns LISTINGS, not vendors: "who sells a vanquishing kryss and for how much"
|
||||
// is the question, and a vendor-shaped result would make every caller flatten the
|
||||
// shops back out.
|
||||
async function getMarket(req, res) {
|
||||
try {
|
||||
const page = await shardMarket.search({
|
||||
q: typeof req.query.q === 'string' ? req.query.q : '',
|
||||
minPrice: intParam(req.query.minPrice),
|
||||
maxPrice: intParam(req.query.maxPrice),
|
||||
itemId: intParam(req.query.itemId),
|
||||
map: typeof req.query.map === 'string' ? req.query.map : '',
|
||||
region: typeof req.query.region === 'string' ? req.query.region : '',
|
||||
sort: typeof req.query.sort === 'string' ? req.query.sort : 'price_asc',
|
||||
limit: intParam(req.query.limit) ?? 50,
|
||||
offset: intParam(req.query.offset) ?? 0,
|
||||
})
|
||||
return res.json(await visibility.project('market', page, req))
|
||||
} catch (err) {
|
||||
log.error('shard.getMarket', err)
|
||||
return res.status(500).json({ message: 'Internal Server Error' })
|
||||
}
|
||||
}
|
||||
|
||||
// GET /public/shard/market/meta — index size, staleness, and the filter options
|
||||
// (which facets and regions actually hold vendors). Separate from the search so
|
||||
// the page can build its filters without running a query it will throw away.
|
||||
async function getMarketMeta(req, res) {
|
||||
try {
|
||||
return res.json(await visibility.project('market', await shardMarket.meta(), req))
|
||||
} catch (err) {
|
||||
log.error('shard.getMarketMeta', err)
|
||||
return res.status(500).json({ message: 'Internal Server Error' })
|
||||
}
|
||||
}
|
||||
|
||||
// GET /public/shard/market/vendors/:serial — one shop and its listings.
|
||||
//
|
||||
// 404 for a serial the index has never seen, which also covers a vendor that has
|
||||
// since been dismissed or hidden: to an anonymous caller "no such shop" is the
|
||||
// only honest answer, and distinguishing the two would leak that a vendor exists
|
||||
// but was hidden.
|
||||
async function getMarketVendor(req, res) {
|
||||
const { serial } = req.params
|
||||
if (!SERIAL_RE.test(serial)) return res.status(400).json({ message: 'Invalid vendor serial.' })
|
||||
try {
|
||||
const vendor = await shardMarket.getVendor(serial, {
|
||||
limit: intParam(req.query.limit) ?? 250,
|
||||
offset: intParam(req.query.offset) ?? 0,
|
||||
})
|
||||
if (!vendor) return res.status(404).json({ message: 'Unknown vendor.' })
|
||||
return res.json(await visibility.project('market', vendor, req))
|
||||
} catch (err) {
|
||||
log.error('shard.getMarketVendor', err)
|
||||
return res.status(500).json({ message: 'Internal Server Error' })
|
||||
}
|
||||
}
|
||||
|
||||
// GET /public/shard/features — the shard features THIS caller can actually see,
|
||||
// so the SPA (and the Android client) can hide nav entries instead of rendering
|
||||
// links that 403. Deliberately reports only what the viewer may reach: the list
|
||||
@@ -335,6 +414,9 @@ module.exports = {
|
||||
getRuleset,
|
||||
getPointsBoards,
|
||||
getPointsBoard,
|
||||
getMarket,
|
||||
getMarketMeta,
|
||||
getMarketVendor,
|
||||
getFeatures,
|
||||
stream,
|
||||
}
|
||||
|
||||
@@ -25,6 +25,7 @@ const { param, query } = require('express-validator')
|
||||
|
||||
const shard = require('./shard.controller')
|
||||
const validate = require('../../../middleware/validate')
|
||||
const { marketLimiter } = require('../../../middleware/rateLimit')
|
||||
const { requireFeature } = require('../../../utils/shardVisibility')
|
||||
|
||||
const shardRouter = express.Router()
|
||||
@@ -167,6 +168,71 @@ shardRouter.get(
|
||||
/* #swagger.responses[404] = { description: 'The shard has never published that system' } */
|
||||
shard.getPointsBoard,
|
||||
)
|
||||
// ── Marketplace ────────────────────────────────────────────────────────────
|
||||
//
|
||||
// Rate-limited, unlike every other route in this file. These are the first
|
||||
// genuinely expensive PUBLIC reads on the site — a LIKE scan plus a COUNT over
|
||||
// what is typically the largest shard_* table, reachable with no session.
|
||||
shardRouter.get(
|
||||
'/market',
|
||||
requireFeature('market'),
|
||||
marketLimiter,
|
||||
// #swagger.tags = ['Public · Shard']
|
||||
// #swagger.summary = 'Search the player-vendor marketplace'
|
||||
// #swagger.description = 'Every priced listing on every player vendor the shard publishes — the same index the in-game Vendor Search gump reads, and it honours the same per-vendor opt-out, so a player who hid their shop in game is hidden here too. Results are LISTINGS, each carrying enough of its shop to be actionable. Served from the site\'s own tables (the sidecar is not touched), so it renders while the shard is down; `staleAt` is the oldest vendor row and the page must say how far behind the index can be — the shard sweeps vendors round-robin, so prices are inherently up to one full cycle old. Item names are resolved server-side against the cliloc table (docs/website/CLILOCS.md); on a shard that has not configured one, `displayName` is null and clients render the item id.'
|
||||
// #swagger.parameters['q'] = { in: 'query', required: false, schema: { type: 'string' }, description: 'Substring match on the resolved item name or the item\'s own literal name (max 60 chars).' }
|
||||
// #swagger.parameters['minPrice'] = { in: 'query', required: false, schema: { type: 'integer' }, description: 'Lowest price to include.' }
|
||||
// #swagger.parameters['maxPrice'] = { in: 'query', required: false, schema: { type: 'integer' }, description: 'Highest price to include.' }
|
||||
// #swagger.parameters['itemId'] = { in: 'query', required: false, schema: { type: 'integer' }, description: 'Exact ItemID (art id) match, for "more like this".' }
|
||||
// #swagger.parameters['map'] = { in: 'query', required: false, schema: { type: 'string' }, description: 'Limit to one facet. Facet names come from the shard\'s own data; an unknown one returns an empty page.' }
|
||||
// #swagger.parameters['region'] = { in: 'query', required: false, schema: { type: 'string' }, description: 'Limit to one named region.' }
|
||||
// #swagger.parameters['sort'] = { in: 'query', required: false, schema: { type: 'string', enum: ['price_asc','price_desc','recent'] }, description: 'Default price_asc. `recent` orders by when the shop was last seen.' }
|
||||
// #swagger.parameters['limit'] = { in: 'query', required: false, schema: { type: 'integer' }, description: 'Page size, 1..100 (default 50).' }
|
||||
// #swagger.parameters['offset'] = { in: 'query', required: false, schema: { type: 'integer' }, description: 'Rows to skip (default 0).' }
|
||||
/* #swagger.responses[200] = { description: 'A page of listings plus the unpaginated total and the staleness stamp', content: { "application/json": { schema: { $ref: "#/components/schemas/ShardMarketPage" } } } } */
|
||||
/* #swagger.responses[403] = { description: 'The market feature is gated above this caller', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
|
||||
/* #swagger.responses[404] = { description: 'The market feature is disabled', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
|
||||
/* #swagger.responses[429] = { description: 'Rate limited', content: { "application/json": { schema: { $ref: "#/components/schemas/Error" } } } } */
|
||||
query('q').optional({ values: 'falsy' }).isString().isLength({ max: 60 }),
|
||||
query('minPrice').optional({ values: 'falsy' }).isInt({ min: 0, max: 999999999 }),
|
||||
query('maxPrice').optional({ values: 'falsy' }).isInt({ min: 0, max: 999999999 }),
|
||||
query('itemId').optional({ values: 'falsy' }).isInt({ min: 0, max: 65535 }),
|
||||
query('map').optional({ values: 'falsy' }).isString().isLength({ max: 40 }),
|
||||
query('region').optional({ values: 'falsy' }).isString().isLength({ max: 80 }),
|
||||
query('sort').optional({ values: 'falsy' }).isIn(['price_asc', 'price_desc', 'recent']),
|
||||
query('limit').optional().isInt({ min: 1, max: 100 }),
|
||||
query('offset').optional().isInt({ min: 0, max: 100000 }),
|
||||
validate,
|
||||
shard.getMarket,
|
||||
)
|
||||
shardRouter.get(
|
||||
'/market/meta',
|
||||
requireFeature('market'),
|
||||
// #swagger.tags = ['Public · Shard']
|
||||
// #swagger.summary = 'Marketplace size, staleness and filter options'
|
||||
// #swagger.description = 'How many vendors and listings the index holds, how stale it may be (`staleAt` = the oldest vendor row, `freshAt` = the newest), and which facets and regions actually hold vendors — so a client can build its filters without running a search it will discard.'
|
||||
/* #swagger.responses[200] = { description: 'Marketplace metadata', content: { "application/json": { schema: { $ref: "#/components/schemas/ShardMarketMeta" } } } } */
|
||||
shard.getMarketMeta,
|
||||
)
|
||||
shardRouter.get(
|
||||
'/market/vendors/:serial',
|
||||
requireFeature('market'),
|
||||
marketLimiter,
|
||||
// #swagger.tags = ['Public · Shard']
|
||||
// #swagger.summary = 'One player vendor and everything it is selling'
|
||||
// #swagger.description = 'A single shop by its vendor serial, with its listings. `truncated` (and `total` exceeding `count`) means the shop holds more than the shard publishes per frame — a commodity reseller with thousands of stacks is a real thing, and the page says so rather than presenting a partial shop as complete. Returns 404 for a serial the index has never seen, which also covers a vendor since dismissed or hidden.'
|
||||
/* #swagger.parameters['serial'] = { in: 'path', required: true, description: 'Vendor serial, e.g. 0x40001234', schema: { type: 'string' } } */
|
||||
// #swagger.parameters['limit'] = { in: 'query', required: false, schema: { type: 'integer' }, description: 'Listings to return, 1..500 (default 250).' }
|
||||
// #swagger.parameters['offset'] = { in: 'query', required: false, schema: { type: 'integer' }, description: 'Listings to skip (default 0).' }
|
||||
/* #swagger.responses[200] = { description: 'The vendor', content: { "application/json": { schema: { $ref: "#/components/schemas/ShardMarketVendor" } } } } */
|
||||
/* #swagger.responses[400] = { description: 'Malformed vendor serial' } */
|
||||
/* #swagger.responses[404] = { description: 'No such vendor in the index' } */
|
||||
param('serial').isString().isLength({ max: 20 }),
|
||||
query('limit').optional().isInt({ min: 1, max: 500 }),
|
||||
query('offset').optional().isInt({ min: 0, max: 100000 }),
|
||||
validate,
|
||||
shard.getMarketVendor,
|
||||
)
|
||||
shardRouter.get(
|
||||
'/features',
|
||||
// #swagger.tags = ['Public · Shard']
|
||||
|
||||
@@ -16,6 +16,7 @@ const revokedSessions = require('./model/revokedSessions/revokedSessions.model')
|
||||
const mobileAuthBridge = require('./model/mobileAuthBridge/mobileAuthBridge.model')
|
||||
const shardAtlas = require('./model/shardAtlas/shardAtlas.model')
|
||||
const shardClilocs = require('./model/shardClilocs/shardClilocs.model')
|
||||
const shardMarket = require('./model/shardMarket/shardMarket.model')
|
||||
const createLogger = require('./utils/logger')
|
||||
const { evaluateBotInternalKey } = require('./utils/botInternalKey')
|
||||
const brand = require('./config/brand')
|
||||
@@ -95,7 +96,15 @@ async function start() {
|
||||
// hash-gated so an unchanged file costs one read, and best-effort so a missing
|
||||
// or wrong-format file never stops the site coming up — it just means item
|
||||
// names render as ids, which is what they did before the table existed.
|
||||
await shardClilocs.refreshOnBoot()
|
||||
const clilocResult = await shardClilocs.refreshOnBoot()
|
||||
|
||||
// A cliloc import changes what item names RESOLVE to, and the marketplace
|
||||
// stores those names denormalized (shard_vendor_items.display_name) so it can
|
||||
// index and search them. The shard's market sweep will not re-send an unchanged
|
||||
// shop just because the site learned what its items are called, so the backfill
|
||||
// has to be pulled rather than waited for. Only after an actual import — the
|
||||
// common boot is hash-gated to a no-op and must stay one.
|
||||
if (clilocResult && clilocResult.status === 'imported') await shardMarket.refreshDisplayNames()
|
||||
|
||||
const mode = await settings.get('site_mode')
|
||||
log.info(`site mode: ${String(mode || 'live').toUpperCase()}`)
|
||||
|
||||
@@ -15,6 +15,7 @@
|
||||
const shardEventsModel = require('../model/shardEvents/shardEvents.model')
|
||||
const shardStateModel = require('../model/shardState/shardState.model')
|
||||
const shardLinksModel = require('../model/shardLinks/shardLinks.model')
|
||||
const shardMarketModel = require('../model/shardMarket/shardMarket.model')
|
||||
const uoLinkConfigModel = require('../model/uoLinkConfig/uoLinkConfig.model')
|
||||
const broadcaster = require('./shardBroadcast')
|
||||
const pushDispatch = require('./pushDispatch')
|
||||
@@ -187,6 +188,19 @@ async function applyStateChange(event, deps) {
|
||||
case 'points.board':
|
||||
await shardState.upsertPointsBoard(event)
|
||||
return
|
||||
// Player-vendor market index. Each frame is authoritative for one shop, so
|
||||
// the model replaces that vendor's whole listing set rather than merging.
|
||||
//
|
||||
// NOT in LOGGED_KINDS, and this is the strongest case of the three v3 kinds:
|
||||
// one frame carries up to 250 listings, the sweep re-emits a shop on any
|
||||
// price change, and appending each of those to the event log would make
|
||||
// shard_events mostly a price history nobody reads. The market IS the state.
|
||||
case 'vendor.listing':
|
||||
await deps.shardMarket.upsertVendor(event)
|
||||
return
|
||||
case 'vendor.listing.remove':
|
||||
await deps.shardMarket.removeVendor(event.serial)
|
||||
return
|
||||
case 'account.unlinked':
|
||||
// A player ran [unlink in game (or a site-side unlink echoed back) — drop
|
||||
// our local link mirror so attribution stops immediately.
|
||||
@@ -209,6 +223,7 @@ function resolveDeps(deps) {
|
||||
shardEvents: deps.shardEvents || shardEventsModel,
|
||||
shardState: deps.shardState || shardStateModel,
|
||||
shardLinks: deps.shardLinks || shardLinksModel,
|
||||
shardMarket: deps.shardMarket || shardMarketModel,
|
||||
uoLinkConfig: deps.uoLinkConfig || uoLinkConfigModel,
|
||||
broadcast: deps.broadcast || broadcaster.broadcast,
|
||||
pushDispatch: deps.pushDispatch || pushDispatch.fromShardEvent,
|
||||
|
||||
@@ -113,7 +113,21 @@ const FEATURES = {
|
||||
// Shop name, owner character name and vendor location are already globally
|
||||
// visible in-game via the stock Vendor Search gump, so publishing them is not
|
||||
// a new disclosure — but they stay configurable so an admin can tighten them.
|
||||
market: { audience: 'anonymous', fields: { ownerName: 'anonymous', location: 'anonymous' } },
|
||||
//
|
||||
// `ownerName` and `location` were pre-wired here by Part A, before the frame
|
||||
// existed; both were re-checked against the real `vendor.listing` and both are
|
||||
// genuine keys on it (unlike leaderboards' `characterName`, which was inert).
|
||||
// `location` is a NESTED object on the wire and on the read model precisely so
|
||||
// that one rule hides map, coordinates, region and house together — five flat
|
||||
// keys would be five rules that drift apart.
|
||||
//
|
||||
// `ownerSerial` is listed alongside `ownerName` for the same reason `houses`
|
||||
// lists both: an admin who hides the owner's name and is left with a serial
|
||||
// that every other board resolves back to that name has not hidden anything.
|
||||
market: {
|
||||
audience: 'anonymous',
|
||||
fields: { ownerName: 'anonymous', ownerSerial: 'anonymous', location: 'anonymous' },
|
||||
},
|
||||
}
|
||||
|
||||
const FEATURE_NAMES = Object.keys(FEATURES)
|
||||
|
||||
@@ -135,6 +135,11 @@ const getRuleset = () => call('/ruleset')
|
||||
// under `boards`); the per-system read 404s for a system the shard never published.
|
||||
const getPoints = () => call('/points')
|
||||
const getPointsBoard = (system) => call(`/points/${encodeURIComponent(system)}`)
|
||||
// Protocol 3.0: the player-vendor market index. The one PAGED sidecar read — a
|
||||
// whole-world market does not fit in a response — so it answers with
|
||||
// `{ vendors, total, limit, offset }` and the caller walks it (see uoLinkSocket).
|
||||
const getMarket = ({ limit = 200, offset = 0 } = {}) =>
|
||||
call(`/market?limit=${encodeURIComponent(limit)}&offset=${encodeURIComponent(offset)}`)
|
||||
|
||||
// ── Commands ──────────────────────────────────────────────────────────────
|
||||
const confirmLink = (code, websiteUserId) =>
|
||||
@@ -201,6 +206,7 @@ module.exports = {
|
||||
getRuleset,
|
||||
getPoints,
|
||||
getPointsBoard,
|
||||
getMarket,
|
||||
confirmLink,
|
||||
linkLookup,
|
||||
createAccount,
|
||||
|
||||
@@ -63,6 +63,55 @@ async function ingestEach(events) {
|
||||
for (const ev of events) await shardIngest.ingest(ev, { fromBackfill: true })
|
||||
}
|
||||
|
||||
// ── Market backfill ────────────────────────────────────────────────────────
|
||||
//
|
||||
// The market is the only board that does not fit in one response, so /market is
|
||||
// paged and this walks it. Two bounds, both deliberate:
|
||||
//
|
||||
// • MARKET_SNAPSHOT_MAX caps the walk. A pathological world (or a sidecar whose
|
||||
// store was never pruned) must not be able to hang startup — backfill runs
|
||||
// before the site is serving the live feed, so an unbounded loop here is
|
||||
// downtime, not slowness.
|
||||
// • The loop stops on a SHORT page as well as on `total`, because a concurrent
|
||||
// sweep can shrink the index underneath the walk and paging to a stale total
|
||||
// would spin.
|
||||
//
|
||||
// Vendors are upserted, never reconciled-by-replacement. A vendor absent from the
|
||||
// snapshot is absent because the sidecar dropped it on vendor.listing.remove —
|
||||
// which our own ingest already processed — so clearing the table first would only
|
||||
// create a window where the market page is empty.
|
||||
const MARKET_SNAPSHOT_MAX = 5000
|
||||
const MARKET_PAGE = 200
|
||||
|
||||
async function backfillMarket() {
|
||||
let offset = 0
|
||||
let seen = 0
|
||||
|
||||
for (;;) {
|
||||
const res = await uoLinkClient.getMarket({ limit: MARKET_PAGE, offset })
|
||||
if (!res.ok || !res.data || !Array.isArray(res.data.vendors)) return
|
||||
|
||||
const page = res.data.vendors
|
||||
if (page.length === 0) break
|
||||
|
||||
await ingestEach(page)
|
||||
seen += page.length
|
||||
offset += page.length
|
||||
|
||||
if (page.length < MARKET_PAGE) break
|
||||
if (seen >= MARKET_SNAPSHOT_MAX) {
|
||||
log.warn('market snapshot truncated at the safety cap', {
|
||||
cap: MARKET_SNAPSHOT_MAX,
|
||||
total: res.data.total,
|
||||
})
|
||||
break
|
||||
}
|
||||
if (Number.isFinite(res.data.total) && offset >= res.data.total) break
|
||||
}
|
||||
|
||||
if (seen > 0) log.info('snapshotted player-vendor market from /market', { count: seen })
|
||||
}
|
||||
|
||||
// Pull recent events from the sidecar's own store and replay them through the
|
||||
// dispatcher (fromBackfill = no SSE re-broadcast). dedupe_key + INSERT IGNORE
|
||||
// make this idempotent, so overlap with what we already stored is harmless.
|
||||
@@ -106,6 +155,8 @@ async function backfill() {
|
||||
// which is the right answer for a month-scale standing.
|
||||
await snapshot(() => uoLinkClient.getPoints(), 'boards', ingestEach, 'snapshotted points boards from /points')
|
||||
|
||||
await backfillMarket()
|
||||
|
||||
const presence = await uoLinkClient.getPresence()
|
||||
if (presence.ok && presence.data && typeof presence.data.count === 'number') {
|
||||
await shardState.setPresence(presence.data)
|
||||
|
||||
Reference in New Issue
Block a user