feat: ingest protocol 2, and keep the record a wipe cannot erase

The module half of the read path. Seven tables, an ingest cursor, four public
routes, and one file whose only job is deciding who may see what.

**The record and the window are different things.** `rust_player_wipe_stats` and
`rust_gather_totals` are permanent and per-wipe, so all-time is those rows SUMmed
rather than a second set of counters that can disagree with them — that is R12's
"per-wipe detail plus all-time rollups" in one table instead of two.
`rust_events` is a bounded 30-day window of raw frames for the killfeed, and
`rust_presence` is a board: replaced wholesale, never appended.

**The feed is a cursor, not a socket, and the header says why.** Core runs Node
20, where a global WebSocket is still behind a flag, so a socket means taking
`ws` — against a release that asserts it has no runtime dependencies (D5). The
deciding argument is the other one though: a socket needs a cursor anyway, for
whatever it missed while the module was restarting, and the catch-up path is the
one that has to be right. A cursor alone is one mechanism exercised every five
seconds rather than two where the second only runs after an outage.

**The cursor advances after the batch, never before.** A crash between the two
re-reads events already counted, which inflates a total; the other order loses
them silently and for ever. One is visible and bounded, the other is invisible
and permanent, so the code fails in the visible direction. A server with no
cursor starts at the sidecar's current END rather than at zero — replaying a
fortnight of deaths into stats for wipes the site never saw is not a catch-up.

**`catalogue.js` is a security boundary, default-deny.** Protocol 2 carries IP
addresses (login attempts, approvals, bans), one player's report about another,
and the grid reference of somebody's base. They are stored, because an operator
chasing ban evasion needs them; they are not served below the admin tier. The
allowlist lives here rather than as a field on the wire, because a boundary
declared by the sender is one a compromised or merely out-of-date game host can
widen — the same reason core's own shard fan-out filters on the serving side. A
kind this build has never heard of is not public, and a test holds the list
against PROTOCOL.md §8.4 so that adding a kind to the protocol without
classifying it fails a build.

`PROTOCOL_VERSION` goes to 2 here in the same change as the emitters, though this
module consumes none of the new frames yet: the sidecar refuses a mismatched
client with a 409, so a module left on 1 would stop being able to read the board
it has been reading all along. A constant that lags the deployment is an outage
with a version number on it.

95 server tests, 20 client tests, every guard green, and `routes.manifest.json`
regenerated against a real core at the pinned ref: 10 routes, all documented,
none of core's moved.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016wDDVXWMDz82WqE1i969r4

server/test/catalogue.test.js

@@ -0,0 +1,110 @@
+// ── The boundary, asserted ────────────────────────────────────────────────
+//
+// `catalogue.js` is the only thing standing between a frame carrying an IP
+// address and a public page, so it gets a suite of its own rather than being
+// covered incidentally by a route test.
+//
+// The most valuable test here is the last one: it holds the classification
+// against the specification in `docs/rust-link/PROTOCOL.md` §8.4. Without it the
+// two drift the first time somebody adds a kind to the protocol, and the drift
+// is silent in the direction that matters — a new kind is simply never served,
+// until the day somebody "fixes" that by adding it to the wrong list.
+
+const test = require('node:test')
+const assert = require('node:assert')
+
+const catalogue = require('../catalogue')
+
+test('an unknown kind is not public — the default is deny', () => {
+  assert.equal(catalogue.isPublic('player.death'), true)
+  assert.equal(catalogue.isPublic('something.new'), false)
+  assert.equal(catalogue.isPublic(''), false)
+  assert.equal(catalogue.isPublic(undefined), false)
+
+  // The shape of the mistake this prevents: a kind a LATER protocol adds, which
+  // this build ingests happily and would publish on the day it first arrived if
+  // the filter were a deny list.
+  assert.equal(catalogue.isKnown('player.location'), false)
+  assert.equal(catalogue.isPublic('player.location'), false)
+})
+
+test('nothing carrying an IP address or a report is public', () => {
+  for (const kind of [
+    'player.login.attempt',
+    'player.approved',
+    'player.banned',
+    'player.unbanned',
+    'player.reported',
+    'entity.destroyed',
+  ]) {
+    assert.equal(catalogue.isPublic(kind), false, `${kind} must not be public`)
+    assert.ok(catalogue.STAFF_KINDS.includes(kind), `${kind} must be classified, not merely absent`)
+  }
+})
+
+test('a viewer with no kinds asked for gets the allowlist, never everything', () => {
+  const asPublic = catalogue.kindsFor({})
+  const asAdmin = catalogue.kindsFor({ admin: true })
+
+  assert.deepEqual(asPublic, [...catalogue.PUBLIC_KINDS])
+  assert.equal(asAdmin.length, catalogue.ALL_KINDS.length)
+
+  // The property that makes the route safe by construction: there is no argument
+  // a caller can omit that turns the filter off.
+  assert.ok(asPublic.length > 0)
+  assert.ok(!asPublic.includes('player.banned'))
+})
+
+test('a kind a viewer may not see is dropped, not refused', () => {
+  const asked = catalogue.kindsFor({ requested: ['player.death', 'player.banned'] })
+
+  assert.deepEqual(asked, ['player.death'])
+
+  // Asking for only forbidden kinds answers with nothing to select, which the
+  // model turns into an empty list — the events are, as far as this viewer is
+  // concerned, not there.
+  assert.deepEqual(catalogue.kindsFor({ requested: ['player.banned'] }), [])
+
+  // And an admin gets what they asked for.
+  assert.deepEqual(catalogue.kindsFor({ admin: true, requested: ['player.banned'] }), [
+    'player.banned',
+  ])
+})
+
+test('every kind is classified exactly once', () => {
+  const seen = new Set()
+
+  for (const kind of catalogue.ALL_KINDS) {
+    assert.ok(!seen.has(kind), `${kind} appears in both lists`)
+    seen.add(kind)
+  }
+
+  assert.equal(seen.size, catalogue.PUBLIC_KINDS.length + catalogue.STAFF_KINDS.length)
+})
+
+test('the classification covers exactly the kinds protocol 2 defines', () => {
+  // The spec lives in another repository, so the list is restated here rather
+  // than parsed — and restating it is the point: adding a kind to the protocol
+  // without deciding who may see it has to fail somewhere, and this is where.
+  //
+  // Sourced from docs/rust-link/PROTOCOL.md §8.4.
+  const PROTOCOL_2 = [
+    'player.connected',
+    'player.disconnected',
+    'player.respawned',
+    'player.death',
+    'player.chat',
+    'player.tally',
+    'entity.destroyed',
+    'player.reported',
+    'player.banned',
+    'player.unbanned',
+    'player.login.attempt',
+    'player.approved',
+    'server.wipe',
+    'server.initialized',
+    'server.shutdown',
+  ]
+
+  assert.deepEqual([...catalogue.ALL_KINDS].sort(), [...PROTOCOL_2].sort())
+})

server/test/events.test.js

@@ -0,0 +1,144 @@
+// ── The read path's logic ─────────────────────────────────────────────────
+//
+// The model decides what a caller gets. Two properties are worth more than the
+// rest, and both are about a caller who did something slightly wrong:
+//
+//   • a route that forgets to say who is asking gets the PUBLIC view;
+//   • a caller asking for a million rows gets two hundred.
+
+const test = require('node:test')
+const assert = require('node:assert')
+
+const { fakeCtx } = require('./_fakes')
+
+function withCore() {
+  require('../core')._reset()
+  require('../core').init(fakeCtx())
+}
+
+test('the limit is bounded, whatever was asked for', () => {
+  withCore()
+  const model = require('../model/events/events.model')
+
+  assert.equal(model.boundedLimit(10), 10)
+  assert.equal(model.boundedLimit(undefined), 50)
+  assert.equal(model.boundedLimit('nonsense'), 50)
+  assert.equal(model.boundedLimit(-5), 50)
+  assert.equal(model.boundedLimit(0), 50)
+  assert.equal(model.boundedLimit(1e9), model.MAX_LIMIT)
+  assert.equal(model.boundedLimit(12.9), 12)
+})
+
+test('kinds parse from one name or a list, and nothing means "not specified"', () => {
+  withCore()
+  const model = require('../model/events/events.model')
+
+  assert.deepEqual(model.parseKinds('player.death'), ['player.death'])
+  assert.deepEqual(model.parseKinds('player.death, player.chat'), ['player.death', 'player.chat'])
+
+  // Null rather than an empty list: "I did not ask" and "I asked for nothing"
+  // are different, and only the first means "whatever I am allowed".
+  assert.equal(model.parseKinds(''), null)
+  assert.equal(model.parseKinds(undefined), null)
+  assert.equal(model.parseKinds(' , , '), null)
+})
+
+test('a reader who does not say who they are gets the public view', async () => {
+  withCore()
+
+  const db = require('../model/events/events.db')
+  const model = require('../model/events/events.model')
+  const original = db.recentEvents
+  let asked = null
+
+  db.recentEvents = async (args) => {
+    asked = args
+    return []
+  }
+
+  try {
+    await model.recent({ serverId: 'main' })
+
+    assert.ok(!asked.kinds.includes('player.banned'), 'no IP-carrying kind by default')
+    assert.ok(asked.kinds.includes('player.death'))
+
+    await model.recent({ serverId: 'main', admin: true })
+    assert.ok(asked.kinds.includes('player.banned'), 'an admin who says so gets them')
+  } finally {
+    db.recentEvents = original
+  }
+})
+
+test('asking only for kinds you may not see answers with nothing, and queries nothing', async () => {
+  withCore()
+
+  const db = require('../model/events/events.db')
+  const model = require('../model/events/events.model')
+  const original = db.recentEvents
+  let called = false
+
+  db.recentEvents = async () => {
+    called = true
+    return []
+  }
+
+  try {
+    const rows = await model.recent({ serverId: 'main', kind: 'player.banned,player.approved' })
+
+    assert.deepEqual(rows, [])
+    assert.equal(called, false, 'a query with no permitted kinds must not reach the database')
+  } finally {
+    db.recentEvents = original
+  }
+})
+
+test('a row whose stored frame will not parse still answers with its envelope', async () => {
+  withCore()
+
+  const db = require('../model/events/events.db')
+  const model = require('../model/events/events.model')
+  const original = db.recentEvents
+
+  db.recentEvents = async () => [
+    { id: 7, kind: 'player.death', t: 12, wipeId: 'w-1', steamId: 'p1', raw: '{not json' },
+  ]
+
+  try {
+    const [row] = await model.recent({ serverId: 'main' })
+
+    // One unreadable row must not fail a whole page. What is known is still
+    // reported; the body is empty rather than absent.
+    assert.equal(row.id, 7)
+    assert.equal(row.kind, 'player.death')
+    assert.deepEqual(row.frame, {})
+  } finally {
+    db.recentEvents = original
+  }
+})
+
+test('the leaderboard answers numbers, never nulls', async () => {
+  withCore()
+
+  const db = require('../model/events/events.db')
+  const model = require('../model/events/events.model')
+  const original = db.leaderboard
+
+  // SUM() over no rows is NULL in SQL, and a JOIN with no player row gives a
+  // null name. A page that has to defend against both is a page with the
+  // defence in three places.
+  db.leaderboard = async () => [
+    { steamId: 'p1', name: null, kills: null, deaths: '3', npcKills: null, playtimeSec: null },
+  ]
+
+  try {
+    const [row] = await model.leaderboard({ serverId: 'main' })
+
+    assert.equal(row.kills, 0)
+    assert.equal(row.deaths, 3)
+    assert.equal(row.npcKills, 0)
+    assert.equal(row.playtimeSec, 0)
+    assert.equal(row.name, null)
+  } finally {
+    db.leaderboard = original
+  }
+})

server/test/ingest.test.js

@@ -0,0 +1,329 @@
+// ── The ingest ────────────────────────────────────────────────────────────
+//
+// Every test here is about one of three things, and all three are mistakes that
+// look correct in review:
+//
+//   • **who gets credited.** A suicide must not credit the victim with a kill.
+//     That single line would produce a leaderboard topped by whoever died most,
+//     and it would look plausible for a whole wipe.
+//   • **the cursor's ordering.** It advances AFTER the batch, never before, so a
+//     crash re-reads rather than skips. Skipping is silent and permanent.
+//   • **absent is not zero.** A session whose start was never seen contributes
+//     no playtime rather than zero playtime.
+//
+// The database is a recorder. Asserting the SQL exactly would be a test of the
+// SQL's punctuation, so each case asserts the *statement shape* and the values —
+// which table was written, and with what.
+
+const test = require('node:test')
+const assert = require('node:assert')
+
+const { fakeCtx } = require('./_fakes')
+
+/** Installs a core whose `db.query` records every statement. */
+function withRecorder() {
+  const statements = []
+
+  const ctx = fakeCtx({
+    db: {
+      query: (sql, params = []) => {
+        statements.push({ sql, params })
+        return Promise.resolve([])
+      },
+      pool: {},
+    },
+  })
+
+  require('../core')._reset()
+  require('../core').init(ctx)
+
+  return {
+    statements,
+    /** Every statement that touched a table, with its parameters. */
+    touching(table) {
+      return statements.filter((s) => s.sql.includes(table))
+    },
+  }
+}
+
+const frame = (over = {}) => ({
+  type: 'event',
+  t: 1789560564452,
+  serverId: 'main',
+  wipeId: 'w-20260915T195817Z',
+  ...over,
+})
+
+const item = (kind, over = {}) => ({ id: 1, t: 1, kind, frame: frame({ kind, ...over }) })
+
+test('every frame is stored, whether or not this build understands it', async () => {
+  const rec = withRecorder()
+  const { apply } = require('../ingest')
+
+  await apply('main', item('player.death', { steamId: '76561198000000001' }))
+  await apply('main', item('something.from.protocol.9'))
+
+  const stored = rec.touching('rust_events')
+  assert.equal(stored.length, 2, 'an unrecognised kind must still be stored')
+
+  // The one copy of an event a later version will know how to read is the one
+  // this version chose not to throw away.
+  assert.ok(stored[1].params.includes('something.from.protocol.9'))
+})
+
+test('a wipe exists because a frame mentioned it', async () => {
+  const rec = withRecorder()
+  const { apply } = require('../ingest')
+
+  await apply('main', item('player.chat', { steamId: '1', message: 'hello' }))
+
+  const wipes = rec.touching('rust_wipes')
+  assert.equal(wipes.length, 1)
+  assert.deepEqual(wipes[0].params.slice(0, 2), ['main', 'w-20260915T195817Z'])
+})
+
+test('a kill credits the attacker and a death the victim', async () => {
+  const rec = withRecorder()
+  const { apply } = require('../ingest')
+
+  await apply(
+    'main',
+    item('player.death', {
+      steamId: 'victim',
+      attackerType: 'player',
+      attackerId: 'killer',
+      attackerName: 'Killer',
+    }),
+  )
+
+  const stats = rec.touching('rust_player_wipe_stats')
+  assert.equal(stats.length, 2, 'one row for the victim, one for the attacker')
+
+  // The parameter order is (server, wipe, steam, kills, deaths, suicides, ...).
+  const victim = stats.find((s) => s.params[2] === 'victim')
+  const killer = stats.find((s) => s.params[2] === 'killer')
+
+  assert.ok(victim && killer)
+  assert.equal(victim.params[3], 0, 'the victim scored no kill')
+  assert.equal(victim.params[4], 1, 'the victim died once')
+  assert.equal(killer.params[3], 1, 'the attacker scored one kill')
+  assert.equal(killer.params[4], 0, 'the attacker did not die')
+})
+
+test('a suicide is a death and a suicide, and credits nobody with a kill', async () => {
+  const rec = withRecorder()
+  const { apply } = require('../ingest')
+
+  await apply('main', item('player.death', { steamId: 'victim', attackerType: 'self' }))
+
+  const stats = rec.touching('rust_player_wipe_stats')
+  assert.equal(stats.length, 1, 'nobody is credited with the kill')
+  assert.equal(stats[0].params[4], 1, 'it is still a death')
+  assert.equal(stats[0].params[5], 1, 'and a suicide')
+  assert.equal(stats[0].params[3], 0)
+})
+
+test('an environment or NPC death credits no attacker', async () => {
+  for (const attackerType of ['environment', 'npc']) {
+    const rec = withRecorder()
+    const { apply } = require('../ingest')
+
+    await apply('main', item('player.death', { steamId: 'victim', attackerType }))
+
+    const stats = rec.touching('rust_player_wipe_stats')
+    assert.equal(stats.length, 1, `${attackerType} must credit nobody`)
+    assert.equal(stats[0].params[4], 1)
+  }
+})
+
+test('an absent session length adds no playtime and no session', async () => {
+  const rec = withRecorder()
+  const { apply } = require('../ingest')
+
+  // A player who was already on the server when the plugin loaded: the plugin
+  // omits `sessionSec` rather than sending 0, and the difference has to survive
+  // all the way to the column. Adding a zero would record a session of no
+  // length, which is a different claim from recording no session.
+  await apply('main', item('player.disconnected', { steamId: 'p1', reason: 'quit' }))
+
+  const stats = rec.touching('rust_player_wipe_stats')
+  assert.equal(stats[0].params[8], 0, 'no session counted')
+  assert.equal(stats[0].params[9], 0, 'no playtime added')
+
+  const rec2 = withRecorder()
+  await require('../ingest').apply(
+    'main',
+    item('player.disconnected', { steamId: 'p1', sessionSec: 600 }),
+  )
+
+  const counted = rec2.touching('rust_player_wipe_stats')
+  assert.equal(counted[0].params[8], 1)
+  assert.equal(counted[0].params[9], 600)
+})
+
+test('a tally is added per resource, as a delta', async () => {
+  const rec = withRecorder()
+  const { apply } = require('../ingest')
+
+  await apply(
+    'main',
+    item('player.tally', {
+      steamId: 'p1',
+      gathered: { wood: 1200, stones: 300 },
+      npcKills: 3,
+      structures: 2,
+    }),
+  )
+
+  const gathered = rec.touching('rust_gather_totals')
+  assert.equal(gathered.length, 2)
+  assert.deepEqual(
+    gathered.map((g) => [g.params[3], g.params[4]]),
+    [
+      ['wood', 1200],
+      ['stones', 300],
+    ],
+  )
+
+  const stats = rec.touching('rust_player_wipe_stats')
+  assert.equal(stats[0].params[6], 3, 'npc kills')
+  assert.equal(stats[0].params[7], 2, 'structures')
+
+  // `amount = amount + VALUES(amount)` is what makes a delta correct. A running
+  // total on the wire would double every number here, slowly, looking right.
+  assert.match(gathered[0].sql, /amount = amount \+ VALUES\(amount\)/)
+})
+
+test('a new server starts at the feed tail, not at the beginning of history', async () => {
+  withRecorder()
+
+  const sidecar = require('../sidecarClient')
+  const db = require('../model/events/events.db')
+  const ingest = require('../ingest')
+
+  const originalTail = sidecar.feedTail
+  const originalCursor = db.getCursor
+  const originalSet = db.setCursor
+  const written = []
+
+  db.getCursor = async () => null
+  db.setCursor = async (...args) => written.push(args)
+  sidecar.feedTail = async () => ({ ok: true, status: 'ok', data: { lastId: 4021, items: [] } })
+
+  try {
+    const applied = await ingest.ingestServer({ id: 'main' })
+
+    assert.equal(applied, 0, 'nothing is replayed')
+    assert.deepEqual(written, [['main', 4021, 0]], 'the cursor starts at the end')
+  } finally {
+    sidecar.feedTail = originalTail
+    db.getCursor = originalCursor
+    db.setCursor = originalSet
+  }
+})
+
+test('an unreachable sidecar writes no cursor at all', async () => {
+  withRecorder()
+
+  const sidecar = require('../sidecarClient')
+  const db = require('../model/events/events.db')
+  const ingest = require('../ingest')
+
+  const originalTail = sidecar.feedTail
+  const originalCursor = db.getCursor
+  const originalSet = db.setCursor
+  const written = []
+
+  db.getCursor = async () => null
+  db.setCursor = async (...args) => written.push(args)
+  sidecar.feedTail = async () => ({ ok: false, status: 'transport-error', data: null })
+
+  try {
+    await ingest.ingestServer({ id: 'main' })
+
+    // A cursor of 0 written here would replay the sidecar's whole retained
+    // history the moment it came back — which is the failure that looks like a
+    // working catch-up until somebody reads the leaderboard.
+    assert.deepEqual(written, [])
+  } finally {
+    sidecar.feedTail = originalTail
+    db.getCursor = originalCursor
+    db.setCursor = originalSet
+  }
+})
+
+test('the cursor advances after the batch, and one bad event does not wedge it', async () => {
+  withRecorder()
+
+  const sidecar = require('../sidecarClient')
+  const db = require('../model/events/events.db')
+  const ingest = require('../ingest')
+
+  const originals = {
+    feed: sidecar.feed,
+    getCursor: db.getCursor,
+    setCursor: db.setCursor,
+    insertEvent: db.insertEvent,
+  }
+
+  const order = []
+
+  db.getCursor = async () => ({ lastEventId: 10 })
+  db.setCursor = async (_id, last) => order.push(`cursor:${last}`)
+  db.insertEvent = async (row) => {
+    order.push(`event:${row.kind}`)
+    if (row.kind === 'player.chat') throw new Error('malformed')
+  }
+
+  sidecar.feed = async (_server, since) =>
+    since === 10
+      ? {
+          ok: true,
+          status: 'ok',
+          data: {
+            items: [item('player.chat'), item('player.connected', { steamId: 'p1' })],
+            lastId: 12,
+            more: false,
+          },
+        }
+      : { ok: true, status: 'ok', data: { items: [], lastId: since, more: false } }
+
+  try {
+    const applied = await ingest.ingestServer({ id: 'main' })
+
+    // The bad row is logged and skipped; the good one still counts.
+    assert.equal(applied, 1)
+
+    // And the ordering the whole design rests on: every event is written before
+    // the cursor moves past it.
+    assert.deepEqual(order, ['event:player.chat', 'event:player.connected', 'cursor:12'])
+  } finally {
+    Object.assign(db, {
+      getCursor: originals.getCursor,
+      setCursor: originals.setCursor,
+      insertEvent: originals.insertEvent,
+    })
+    sidecar.feed = originals.feed
+  }
+})
+
+test('a board replaces presence rather than appending to it', async () => {
+  const rec = withRecorder()
+  const ingest = require('../ingest')
+
+  await ingest.applyBoards('main', {
+    'players.online': {
+      kind: 'players.online',
+      type: 'snapshot',
+      count: 1,
+      players: [{ steamId: 'p1', name: 'One', sleeping: false }],
+    },
+  })
+
+  const presence = rec.touching('rust_presence')
+
+  // The DELETE is what makes it a board. Without it a player who left stays
+  // online for ever, which is the exact drift the board exists to correct.
+  assert.match(presence[0].sql, /^DELETE FROM rust_presence/)
+  assert.match(presence[1].sql, /INSERT INTO rust_presence/)
+})