Protocol 3.0 §8, the website half. Ingests vendor.listing / vendor.listing.remove
into shard_vendors + shard_vendor_items, serves a searchable public API over
them, and ships /site/market and /site/market/vendors/:serial.
Three things the pages have to say out loud, all consequences of how the data is
gathered:
- The prices are NOT live. The shard sweeps vendors round-robin, so a shop can be
a full cycle behind. The banner is driven by the OLDEST vendor row, not the
newest — the one stale shop is the one that wastes somebody's trip.
- A shop can be truncated. `total` exceeding `count` means the shop holds more
than the shard publishes per frame; the vendor page says "showing 250 of 3,104"
rather than presenting a partial shop as complete.
- An item may have no name. On a shard with no cliloc table the honest render is
the item id, never an invented label.
## The pre-wired visibility rules, re-checked
Part A pre-wired market.ownerName and market.location before the frame existed,
and the sibling rule it pre-wired for leaderboards (`characterName`) turned out
to be INERT because projectValue matches literal JSON keys. Both market rules
were checked against the real frame this time:
- `ownerName` is a real key. Kept.
- `location` is a real key ONLY because the frame nests it. Flat map/x/y/region
would have made the rule match nothing — the same failure, one part later. It
is nested on the wire and on the read model so one rule hides the facet, the
coordinates, the region and the house together; five flat keys would be five
rules that drift apart.
- `ownerSerial` was ADDED. An admin who hides the owner's name and leaves a
serial that the leaderboards and guild boards resolve back to that same name
has not hidden anything.
Tests assert all three bite, on the stored read model AND on the raw frame —
the market's SSE stream is off by default but an admin can turn it on, and a rule
that worked on only one path is exactly the leak §3.6.1 records.
## Notable
- **No payload column on shard_vendors**, unlike shard_points_boards next door.
The board's top-N is a fixed-size list read whole; here the items ARE the
searchable rows, so they are normalized and nothing is left worth duplicating.
- **display_name is denormalized at ingest** (literal name preferred over the
cliloc — a player set it, so it is more specific). Resolving at query time
would put the cliloc table on the hot path and make search-by-name impossible.
Because the shard's diff sweep will not re-send an unchanged shop just because
the site learned what its items are called, a cliloc import now triggers a bulk
re-resolution — 50 ms per thousand rows, never throws.
- **updated_at is written explicitly** on every upsert. MariaDB does not fire ON
UPDATE CURRENT_TIMESTAMP when every column is written back unchanged, and a
shop re-published identically is still freshly confirmed — without this the
staleness banner would age a perfectly current shop forever.
- **LIKE wildcards in `q` are escaped.** `%` and `_` are LIKE metacharacters, not
SQL ones, so parameterization does not neutralize them: `?q=%` would otherwise
match every listing on the shard.
- **Rate-limited** (60/min/IP), the only limited public read. Every other public
GET is an indexed lookup of bounded size; this is a LIKE scan plus a COUNT over
the largest shard_* table, anonymous by default.
- Reconnect backfill pages /market, bounded by MARKET_SNAPSHOT_MAX = 5000 and
stopping on a short page as well as on `total`, so a concurrent sweep shrinking
the index cannot spin the walk.
## How it was tested
673 server tests pass (27 new). Client builds clean; swagger-output.json,
routes.manifest.json and routes.guards.json regenerated.
Verified full-stack against the live MariaDB and a real shard, not only units:
- 27 real vendors / 1,040 listings swept off the ServUO tree, through the Rust
sidecar, into the site — names resolving through the cliloc table ("longsword",
"katana"), real facets and regions in the filters.
- `?q=sword` 682, `?q=%` and `?q=_` **0** (the escape), map/region/price/sort
filters, paging, and the vendor detail route.
- Visibility live: fields gated to staff vanish for an anonymous caller while
shopName and price survive; audience=player 403s; enabled=0 404s; and
/shard/features correctly drops `market` so the nav hides it.
- Re-publishing a shop smaller leaves no orphan items; an identical re-publish
moves updated_at.
- The limiter fires (38x200 then 32x429 on a 70-request burst).
Not covered by an automated test: the two React pages are presentational and this
repo's client suite covers pure-logic modules only. They were driven against the
live API above, but not rendered in a DOM harness.
Co-Authored-By: Claude <noreply@anthropic.com>
206 lines
8.9 KiB
JavaScript
206 lines
8.9 KiB
JavaScript
require('dotenv').config()
|
|
const http = require('http')
|
|
|
|
const app = require('./app')
|
|
const internalApp = require('./internalApp')
|
|
const botScore = require('./middleware/botScore')
|
|
const uoLinkSocket = require('./utils/uoLinkSocket')
|
|
const uoLinkClient = require('./utils/uoLinkClient')
|
|
const uoLinkConfig = require('./model/uoLinkConfig/uoLinkConfig.model')
|
|
const shardBroadcast = require('./utils/shardBroadcast')
|
|
const announceWorker = require('./utils/announceWorker')
|
|
const { ensureSchema, close } = require('./utils/db')
|
|
const { seedDefaults, createInitialAdminFromEnv } = require('../db/seed')
|
|
const settings = require('./model/settings/settings.model')
|
|
const revokedSessions = require('./model/revokedSessions/revokedSessions.model')
|
|
const mobileAuthBridge = require('./model/mobileAuthBridge/mobileAuthBridge.model')
|
|
const shardAtlas = require('./model/shardAtlas/shardAtlas.model')
|
|
const shardClilocs = require('./model/shardClilocs/shardClilocs.model')
|
|
const shardMarket = require('./model/shardMarket/shardMarket.model')
|
|
const createLogger = require('./utils/logger')
|
|
const { evaluateBotInternalKey } = require('./utils/botInternalKey')
|
|
const brand = require('./config/brand')
|
|
const pkg = require('../package.json')
|
|
|
|
const log = createLogger('server')
|
|
const PORT = Number(process.env.PORT) || 3000
|
|
// Separate, UNPUBLISHED listener for server<->bot /internal/* traffic. Kept off
|
|
// the public PORT so the decrypted-token route can't ride the listener Pangolin
|
|
// proxies to the world (issue #33). Must match the port in the bot's
|
|
// SITE_INTERNAL_URL (docker-compose.yml).
|
|
const INTERNAL_PORT = Number(process.env.INTERNAL_PORT) || 3001
|
|
const HOST = '0.0.0.0' // bind all interfaces so Pangolin / the LAN can reach it
|
|
|
|
async function start() {
|
|
log.info(`starting ${brand.name} server v${pkg.version}`, {
|
|
node: process.version,
|
|
env: process.env.NODE_ENV || 'development',
|
|
logLevel: process.env.LOG_LEVEL || 'info',
|
|
logFile: createLogger.logFilePath || 'disabled (console only)',
|
|
db: `${process.env.DB_HOST || '127.0.0.1'}:${process.env.DB_PORT || 3306}/${process.env.DB_NAME || 'runic_gateway'}`,
|
|
cookieSecure: process.env.COOKIE_SECURE || 'auto',
|
|
email: 'gmail-oauth2 (configured in admin → settings)',
|
|
})
|
|
|
|
// Fail fast if the server<->bot shared secret is weak/placeholder. Fatal in
|
|
// production (the /internal/bot-config route hands back the decrypted Discord
|
|
// token and this key is its only guard); a warning otherwise.
|
|
const keyCheck = evaluateBotInternalKey({
|
|
key: process.env.BOT_INTERNAL_KEY,
|
|
nodeEnv: process.env.NODE_ENV,
|
|
})
|
|
if (keyCheck.fatal) {
|
|
log.error(keyCheck.message)
|
|
process.exit(1)
|
|
} else if (!keyCheck.ok) {
|
|
log.warn(keyCheck.message)
|
|
}
|
|
|
|
log.info('ensuring database schema...')
|
|
await ensureSchema()
|
|
log.info('seeding defaults...')
|
|
await seedDefaults()
|
|
await createInitialAdminFromEnv()
|
|
|
|
// Clear out session-denylist rows whose token has already expired (dead weight).
|
|
// Best-effort — a prune failure must never block startup.
|
|
try {
|
|
const pruned = await revokedSessions.pruneExpired()
|
|
if (pruned) log.info(`pruned ${pruned} expired revoked-session row(s)`)
|
|
} catch (err) {
|
|
log.warn('revoked-session prune failed', { error: err.message })
|
|
}
|
|
|
|
// Same treatment for the mobile SSO bridge tables (also pruned opportunistically
|
|
// on each bridge write). Boot-time sweep catches rows orphaned by a crash.
|
|
try {
|
|
const pruned = await mobileAuthBridge.pruneExpired()
|
|
if (pruned) log.info(`pruned ${pruned} expired mobile-auth-bridge row(s)`)
|
|
} catch (err) {
|
|
log.warn('mobile-auth-bridge prune failed', { error: err.message })
|
|
}
|
|
|
|
// Re-derive the spawn atlas from the shard's own ServUO tree. The shard's maps
|
|
// change over its lifetime — facets get added, replaced or renamed — so the
|
|
// atlas is rebuilt on every boot rather than shipped as a snapshot that would
|
|
// silently go stale. Hash-gated, so an unchanged tree costs one read pass and
|
|
// no database write.
|
|
//
|
|
// Best-effort by contract: no configured path, an unreadable mount or a
|
|
// malformed file must never stop the site coming up. A refresh that would
|
|
// REMOVE a facet is staged for admin approval instead of being applied.
|
|
await shardAtlas.refreshOnBoot()
|
|
|
|
// Refresh the cliloc table (UO's id → display-string map) from the file the
|
|
// operator converted out of their own client. Same contract as the atlas:
|
|
// hash-gated so an unchanged file costs one read, and best-effort so a missing
|
|
// or wrong-format file never stops the site coming up — it just means item
|
|
// names render as ids, which is what they did before the table existed.
|
|
const clilocResult = await shardClilocs.refreshOnBoot()
|
|
|
|
// A cliloc import changes what item names RESOLVE to, and the marketplace
|
|
// stores those names denormalized (shard_vendor_items.display_name) so it can
|
|
// index and search them. The shard's market sweep will not re-send an unchanged
|
|
// shop just because the site learned what its items are called, so the backfill
|
|
// has to be pulled rather than waited for. Only after an actual import — the
|
|
// common boot is hash-gated to a no-op and must stay one.
|
|
if (clilocResult && clilocResult.status === 'imported') await shardMarket.refreshDisplayNames()
|
|
|
|
const mode = await settings.get('site_mode')
|
|
log.info(`site mode: ${String(mode || 'live').toUpperCase()}`)
|
|
|
|
const server = http.createServer(app)
|
|
server.listen(PORT, HOST, () => {
|
|
log.info(`listening on http://${HOST}:${PORT} (API at /api/v1, health at /api/health)`)
|
|
})
|
|
|
|
// Internal server<->bot API on a separate, unpublished port. NEVER expose this
|
|
// through Pangolin/the public reverse proxy — it serves the decrypted Discord
|
|
// bot token to the bot process over the private compose network only (#33).
|
|
const internalServer = http.createServer(internalApp)
|
|
internalServer.listen(INTERNAL_PORT, HOST, () => {
|
|
log.info(`internal API listening on http://${HOST}:${INTERNAL_PORT} (server<->bot only — do NOT proxy)`)
|
|
})
|
|
|
|
// Start the uo-link WebSocket ingest client. Self-guards: it only actually
|
|
// connects when the admin has enabled the integration and saved a token, so
|
|
// this is a no-op on shards that haven't configured the sidecar. Never let a
|
|
// sidecar problem block server startup.
|
|
try {
|
|
await uoLinkSocket.start()
|
|
await checkUoLink()
|
|
} catch (err) {
|
|
log.warn('uo-link socket failed to start (continuing)', { error: err.message })
|
|
}
|
|
|
|
// Start the news-announcement dispatcher: a light in-process poller that pushes
|
|
// published news posts to the in-game town crier + Discord with independent
|
|
// retry per leg. No-op until a news post is actually published.
|
|
announceWorker.start()
|
|
|
|
setupShutdown(server, internalServer)
|
|
}
|
|
|
|
// Best-effort startup probe of the uo-link sidecar: if the integration is
|
|
// enabled, log whether it is reachable and warn loudly on a protocol mismatch
|
|
// (fail-fast visibility rather than silently mis-parsing a newer wire format).
|
|
async function checkUoLink() {
|
|
const config = await uoLinkConfig.getSafe()
|
|
if (!config.enabled) return
|
|
const health = await uoLinkClient.health()
|
|
if (!health.ok) {
|
|
log.warn('uo-link is enabled but the sidecar is unreachable at startup', {
|
|
baseUrl: config.baseUrl,
|
|
error: health.error || `status ${health.status}`,
|
|
})
|
|
return
|
|
}
|
|
if (health.data && health.data.protocol && health.data.protocol !== config.protocol) {
|
|
log.error('uo-link PROTOCOL MISMATCH — pinned vs sidecar', {
|
|
pinned: config.protocol,
|
|
sidecar: health.data.protocol,
|
|
})
|
|
} else {
|
|
log.info('uo-link sidecar reachable', {
|
|
pluginConnected: health.data && health.data.plugin_connected,
|
|
protocol: health.data && health.data.protocol,
|
|
})
|
|
}
|
|
}
|
|
|
|
function setupShutdown(server, internalServer) {
|
|
let closing = false
|
|
const shutdown = async (signal) => {
|
|
if (closing) return
|
|
closing = true
|
|
log.warn(`${signal} received — shutting down gracefully`)
|
|
botScore.stopSweeper() // stop the bot-store cleanup interval
|
|
announceWorker.stop() // stop the news-announcement dispatcher poller
|
|
uoLinkSocket.stop() // close the uo-link WS ingest client
|
|
shardBroadcast.closeAll() // end any open shard live-feed SSE streams
|
|
server.close(() => log.info('http server closed'))
|
|
if (internalServer) internalServer.close(() => log.info('internal http server closed'))
|
|
try {
|
|
await close()
|
|
log.info('database pool closed')
|
|
} catch (err) {
|
|
log.error('error closing database pool', err)
|
|
}
|
|
await createLogger.close() // flush the log file
|
|
process.exit(0)
|
|
}
|
|
|
|
process.on('SIGINT', () => shutdown('SIGINT'))
|
|
process.on('SIGTERM', () => shutdown('SIGTERM'))
|
|
process.on('unhandledRejection', (reason) => log.error('unhandledRejection', { reason: String(reason) }))
|
|
process.on('uncaughtException', (err) => {
|
|
log.error('uncaughtException', err)
|
|
process.exit(1)
|
|
})
|
|
}
|
|
|
|
start().catch((err) => {
|
|
log.error('failed to start server', err)
|
|
process.exit(1)
|
|
})
|