Files
website/server/src/server.js
wtclaude 8771a1cf6c feat(shard): the player-vendor marketplace
Protocol 3.0 §8, the website half. Ingests vendor.listing / vendor.listing.remove
into shard_vendors + shard_vendor_items, serves a searchable public API over
them, and ships /site/market and /site/market/vendors/:serial.

Three things the pages have to say out loud, all consequences of how the data is
gathered:

- The prices are NOT live. The shard sweeps vendors round-robin, so a shop can be
  a full cycle behind. The banner is driven by the OLDEST vendor row, not the
  newest — the one stale shop is the one that wastes somebody's trip.
- A shop can be truncated. `total` exceeding `count` means the shop holds more
  than the shard publishes per frame; the vendor page says "showing 250 of 3,104"
  rather than presenting a partial shop as complete.
- An item may have no name. On a shard with no cliloc table the honest render is
  the item id, never an invented label.

## The pre-wired visibility rules, re-checked

Part A pre-wired market.ownerName and market.location before the frame existed,
and the sibling rule it pre-wired for leaderboards (`characterName`) turned out
to be INERT because projectValue matches literal JSON keys. Both market rules
were checked against the real frame this time:

- `ownerName` is a real key. Kept.
- `location` is a real key ONLY because the frame nests it. Flat map/x/y/region
  would have made the rule match nothing — the same failure, one part later. It
  is nested on the wire and on the read model so one rule hides the facet, the
  coordinates, the region and the house together; five flat keys would be five
  rules that drift apart.
- `ownerSerial` was ADDED. An admin who hides the owner's name and leaves a
  serial that the leaderboards and guild boards resolve back to that same name
  has not hidden anything.

Tests assert all three bite, on the stored read model AND on the raw frame —
the market's SSE stream is off by default but an admin can turn it on, and a rule
that worked on only one path is exactly the leak §3.6.1 records.

## Notable

- **No payload column on shard_vendors**, unlike shard_points_boards next door.
  The board's top-N is a fixed-size list read whole; here the items ARE the
  searchable rows, so they are normalized and nothing is left worth duplicating.
- **display_name is denormalized at ingest** (literal name preferred over the
  cliloc — a player set it, so it is more specific). Resolving at query time
  would put the cliloc table on the hot path and make search-by-name impossible.
  Because the shard's diff sweep will not re-send an unchanged shop just because
  the site learned what its items are called, a cliloc import now triggers a bulk
  re-resolution — 50 ms per thousand rows, never throws.
- **updated_at is written explicitly** on every upsert. MariaDB does not fire ON
  UPDATE CURRENT_TIMESTAMP when every column is written back unchanged, and a
  shop re-published identically is still freshly confirmed — without this the
  staleness banner would age a perfectly current shop forever.
- **LIKE wildcards in `q` are escaped.** `%` and `_` are LIKE metacharacters, not
  SQL ones, so parameterization does not neutralize them: `?q=%` would otherwise
  match every listing on the shard.
- **Rate-limited** (60/min/IP), the only limited public read. Every other public
  GET is an indexed lookup of bounded size; this is a LIKE scan plus a COUNT over
  the largest shard_* table, anonymous by default.
- Reconnect backfill pages /market, bounded by MARKET_SNAPSHOT_MAX = 5000 and
  stopping on a short page as well as on `total`, so a concurrent sweep shrinking
  the index cannot spin the walk.

## How it was tested

673 server tests pass (27 new). Client builds clean; swagger-output.json,
routes.manifest.json and routes.guards.json regenerated.

Verified full-stack against the live MariaDB and a real shard, not only units:

- 27 real vendors / 1,040 listings swept off the ServUO tree, through the Rust
  sidecar, into the site — names resolving through the cliloc table ("longsword",
  "katana"), real facets and regions in the filters.
- `?q=sword` 682, `?q=%` and `?q=_` **0** (the escape), map/region/price/sort
  filters, paging, and the vendor detail route.
- Visibility live: fields gated to staff vanish for an anonymous caller while
  shopName and price survive; audience=player 403s; enabled=0 404s; and
  /shard/features correctly drops `market` so the nav hides it.
- Re-publishing a shop smaller leaves no orphan items; an identical re-publish
  moves updated_at.
- The limiter fires (38x200 then 32x429 on a 70-request burst).

Not covered by an automated test: the two React pages are presentational and this
repo's client suite covers pure-logic modules only. They were driven against the
live API above, but not rendered in a DOM harness.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 09:51:50 -05:00

206 lines
8.9 KiB
JavaScript

require('dotenv').config()
const http = require('http')
const app = require('./app')
const internalApp = require('./internalApp')
const botScore = require('./middleware/botScore')
const uoLinkSocket = require('./utils/uoLinkSocket')
const uoLinkClient = require('./utils/uoLinkClient')
const uoLinkConfig = require('./model/uoLinkConfig/uoLinkConfig.model')
const shardBroadcast = require('./utils/shardBroadcast')
const announceWorker = require('./utils/announceWorker')
const { ensureSchema, close } = require('./utils/db')
const { seedDefaults, createInitialAdminFromEnv } = require('../db/seed')
const settings = require('./model/settings/settings.model')
const revokedSessions = require('./model/revokedSessions/revokedSessions.model')
const mobileAuthBridge = require('./model/mobileAuthBridge/mobileAuthBridge.model')
const shardAtlas = require('./model/shardAtlas/shardAtlas.model')
const shardClilocs = require('./model/shardClilocs/shardClilocs.model')
const shardMarket = require('./model/shardMarket/shardMarket.model')
const createLogger = require('./utils/logger')
const { evaluateBotInternalKey } = require('./utils/botInternalKey')
const brand = require('./config/brand')
const pkg = require('../package.json')
const log = createLogger('server')
const PORT = Number(process.env.PORT) || 3000
// Separate, UNPUBLISHED listener for server<->bot /internal/* traffic. Kept off
// the public PORT so the decrypted-token route can't ride the listener Pangolin
// proxies to the world (issue #33). Must match the port in the bot's
// SITE_INTERNAL_URL (docker-compose.yml).
const INTERNAL_PORT = Number(process.env.INTERNAL_PORT) || 3001
const HOST = '0.0.0.0' // bind all interfaces so Pangolin / the LAN can reach it
async function start() {
log.info(`starting ${brand.name} server v${pkg.version}`, {
node: process.version,
env: process.env.NODE_ENV || 'development',
logLevel: process.env.LOG_LEVEL || 'info',
logFile: createLogger.logFilePath || 'disabled (console only)',
db: `${process.env.DB_HOST || '127.0.0.1'}:${process.env.DB_PORT || 3306}/${process.env.DB_NAME || 'runic_gateway'}`,
cookieSecure: process.env.COOKIE_SECURE || 'auto',
email: 'gmail-oauth2 (configured in admin → settings)',
})
// Fail fast if the server<->bot shared secret is weak/placeholder. Fatal in
// production (the /internal/bot-config route hands back the decrypted Discord
// token and this key is its only guard); a warning otherwise.
const keyCheck = evaluateBotInternalKey({
key: process.env.BOT_INTERNAL_KEY,
nodeEnv: process.env.NODE_ENV,
})
if (keyCheck.fatal) {
log.error(keyCheck.message)
process.exit(1)
} else if (!keyCheck.ok) {
log.warn(keyCheck.message)
}
log.info('ensuring database schema...')
await ensureSchema()
log.info('seeding defaults...')
await seedDefaults()
await createInitialAdminFromEnv()
// Clear out session-denylist rows whose token has already expired (dead weight).
// Best-effort — a prune failure must never block startup.
try {
const pruned = await revokedSessions.pruneExpired()
if (pruned) log.info(`pruned ${pruned} expired revoked-session row(s)`)
} catch (err) {
log.warn('revoked-session prune failed', { error: err.message })
}
// Same treatment for the mobile SSO bridge tables (also pruned opportunistically
// on each bridge write). Boot-time sweep catches rows orphaned by a crash.
try {
const pruned = await mobileAuthBridge.pruneExpired()
if (pruned) log.info(`pruned ${pruned} expired mobile-auth-bridge row(s)`)
} catch (err) {
log.warn('mobile-auth-bridge prune failed', { error: err.message })
}
// Re-derive the spawn atlas from the shard's own ServUO tree. The shard's maps
// change over its lifetime — facets get added, replaced or renamed — so the
// atlas is rebuilt on every boot rather than shipped as a snapshot that would
// silently go stale. Hash-gated, so an unchanged tree costs one read pass and
// no database write.
//
// Best-effort by contract: no configured path, an unreadable mount or a
// malformed file must never stop the site coming up. A refresh that would
// REMOVE a facet is staged for admin approval instead of being applied.
await shardAtlas.refreshOnBoot()
// Refresh the cliloc table (UO's id → display-string map) from the file the
// operator converted out of their own client. Same contract as the atlas:
// hash-gated so an unchanged file costs one read, and best-effort so a missing
// or wrong-format file never stops the site coming up — it just means item
// names render as ids, which is what they did before the table existed.
const clilocResult = await shardClilocs.refreshOnBoot()
// A cliloc import changes what item names RESOLVE to, and the marketplace
// stores those names denormalized (shard_vendor_items.display_name) so it can
// index and search them. The shard's market sweep will not re-send an unchanged
// shop just because the site learned what its items are called, so the backfill
// has to be pulled rather than waited for. Only after an actual import — the
// common boot is hash-gated to a no-op and must stay one.
if (clilocResult && clilocResult.status === 'imported') await shardMarket.refreshDisplayNames()
const mode = await settings.get('site_mode')
log.info(`site mode: ${String(mode || 'live').toUpperCase()}`)
const server = http.createServer(app)
server.listen(PORT, HOST, () => {
log.info(`listening on http://${HOST}:${PORT} (API at /api/v1, health at /api/health)`)
})
// Internal server<->bot API on a separate, unpublished port. NEVER expose this
// through Pangolin/the public reverse proxy — it serves the decrypted Discord
// bot token to the bot process over the private compose network only (#33).
const internalServer = http.createServer(internalApp)
internalServer.listen(INTERNAL_PORT, HOST, () => {
log.info(`internal API listening on http://${HOST}:${INTERNAL_PORT} (server<->bot only — do NOT proxy)`)
})
// Start the uo-link WebSocket ingest client. Self-guards: it only actually
// connects when the admin has enabled the integration and saved a token, so
// this is a no-op on shards that haven't configured the sidecar. Never let a
// sidecar problem block server startup.
try {
await uoLinkSocket.start()
await checkUoLink()
} catch (err) {
log.warn('uo-link socket failed to start (continuing)', { error: err.message })
}
// Start the news-announcement dispatcher: a light in-process poller that pushes
// published news posts to the in-game town crier + Discord with independent
// retry per leg. No-op until a news post is actually published.
announceWorker.start()
setupShutdown(server, internalServer)
}
// Best-effort startup probe of the uo-link sidecar: if the integration is
// enabled, log whether it is reachable and warn loudly on a protocol mismatch
// (fail-fast visibility rather than silently mis-parsing a newer wire format).
async function checkUoLink() {
const config = await uoLinkConfig.getSafe()
if (!config.enabled) return
const health = await uoLinkClient.health()
if (!health.ok) {
log.warn('uo-link is enabled but the sidecar is unreachable at startup', {
baseUrl: config.baseUrl,
error: health.error || `status ${health.status}`,
})
return
}
if (health.data && health.data.protocol && health.data.protocol !== config.protocol) {
log.error('uo-link PROTOCOL MISMATCH — pinned vs sidecar', {
pinned: config.protocol,
sidecar: health.data.protocol,
})
} else {
log.info('uo-link sidecar reachable', {
pluginConnected: health.data && health.data.plugin_connected,
protocol: health.data && health.data.protocol,
})
}
}
function setupShutdown(server, internalServer) {
let closing = false
const shutdown = async (signal) => {
if (closing) return
closing = true
log.warn(`${signal} received — shutting down gracefully`)
botScore.stopSweeper() // stop the bot-store cleanup interval
announceWorker.stop() // stop the news-announcement dispatcher poller
uoLinkSocket.stop() // close the uo-link WS ingest client
shardBroadcast.closeAll() // end any open shard live-feed SSE streams
server.close(() => log.info('http server closed'))
if (internalServer) internalServer.close(() => log.info('internal http server closed'))
try {
await close()
log.info('database pool closed')
} catch (err) {
log.error('error closing database pool', err)
}
await createLogger.close() // flush the log file
process.exit(0)
}
process.on('SIGINT', () => shutdown('SIGINT'))
process.on('SIGTERM', () => shutdown('SIGTERM'))
process.on('unhandledRejection', (reason) => log.error('unhandledRejection', { reason: String(reason) }))
process.on('uncaughtException', (err) => {
log.error('uncaughtException', err)
process.exit(1)
})
}
start().catch((err) => {
log.error('failed to start server', err)
process.exit(1)
})