feat(events): conditions, phase advancement and the diagnosis panel (Phase 5)
All checks were successful
PR Checks / bot-tests (pull_request) Successful in 31s
PR Checks / server-tests (pull_request) Successful in 5m28s
PR Checks / client-build (pull_request) Successful in 8m47s

A phase used to advance on one fact - every step terminal. It can now also carry
an advance CONDITION: `{ after: '30m' }` or `{ on: '<triggerId>', where:
<conditions>, count: n }`, reusing `engagement/conditions.js` unchanged. The
phase's real deliverable is the diagnosis panel: "why didn't phase 3 start?"
answered in the condition builder's own words, with the tally, the elapsed time
and the last related firing whether or not it counted.

`POST /admin/events/runs/:runId/advance` arrives beside it. It has been absent
since Phase 3 for want of a meaning; a phase with a gate can wait on a boss that
will never spawn, and that is the one state "force it anyway" names.

One new table, `event_run_phase_gates`. The emit path writes the tally at the
moment a firing happens - a gate waiting on three spawns counts things that
occur between two ticks, and a tally held in a process's memory is one a restart
silently zeroes - and the runner's tick reads it.

A gate that never opens is HELD, with no automatic advance and no authored
timeout (org lead, 2026-09-02). What the engine owes instead is visibility:
`EVENT_PHASE_STALL_MS` takes the run's health to `stalled`, and `setHealth` is
now escalation-only so a later retry cannot demote it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T6t8mrAWhZU5vnyYgZTMtL
This commit is contained in:
2026-09-02 22:11:20 -05:00
parent 9c23c5fd0e
commit 9bc0bf5a3d
26 changed files with 2646 additions and 51 deletions

View File

@@ -54,6 +54,32 @@
// for `waiting_steps` and a LEFT JOIN that must not drop a definition with no
// series.
//
// **Phase 5 added the gate statements**, and the increment is the one on this
// list with the closest precedent: engagement's cooldown claim was green against
// its stub and always allowed the send against a real server, because the
// connector defaults `foundRows: true`. A gate's tally is the same shape of
// claim — a conditional UPDATE whose answer decides something — so it is proved
// here rather than believed.
//
// * **`GATE_COUNT`** - `tally = tally + 1` with `CASE WHEN tally + 1 >=
// needed` inside the same statement, guarded on `satisfied_at IS NULL`. Two
// emits arriving together must each add one and exactly ONE of them must
// cross the threshold; a read-then-write would let both see 2 of 3. **This
// is the statement that failed here first**: MariaDB evaluates SET
// assignments left to right with the values already assigned, so an
// increment written before the CASE made the CASE read the new tally and
// close a two-firing gate on the first firing. The ORDER of the SET list is
// load-bearing, and only a real server says so.
// * **`GATE_SATISFY`** - the same guard for the two reasons that are not a
// firing: an `after` deadline passing, and a human forcing it. A force that
// races the tick must lose harmlessly.
// * **`GATE_OPEN_FOR_TRIGGER`** - the emit path's only query, and the one
// index in this feature on a hot path. Its JOIN is what stops a gate
// belonging to a cancelled run counting for ever.
// * **`uq_evgate_phase`** - what makes `open` an INSERT IGNORE, so a process
// that died between entering a phase and opening its gate opens no second
// one on the next tick.
//
// Plus the two unique indexes that are load-bearing rather than tidy:
// `uq_evrun_occurrence` (which, not the claim, is what stops two runs of one
// occurrence existing) and `uq_evstep_slot` (which is what makes re-materialising
@@ -147,6 +173,29 @@ CREATE TABLE event_run_steps (
UNIQUE KEY uq_evstep_slot (run_id, phase, seq),
INDEX idx_evstep_due (status, due_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
CREATE TABLE event_run_phase_gates (
id BIGINT AUTO_INCREMENT PRIMARY KEY,
run_id BIGINT NOT NULL,
phase VARCHAR(64) NOT NULL,
kind ENUM('after','on') NOT NULL,
after_seconds INT NULL,
trigger_id VARCHAR(96) NULL,
conditions JSON NULL,
needed INT NOT NULL DEFAULT 1,
tally INT NOT NULL DEFAULT 0,
entered_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
due_at DATETIME NULL,
last_event JSON NULL,
last_event_at DATETIME NULL,
satisfied_at DATETIME NULL,
satisfied_by VARCHAR(16) NULL,
forced_by INT NULL,
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
CONSTRAINT fk_evgate_run FOREIGN KEY (run_id) REFERENCES event_runs(id) ON DELETE CASCADE,
UNIQUE KEY uq_evgate_phase (run_id, phase),
INDEX idx_evgate_open (trigger_id, satisfied_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
`
// The statements under test, verbatim from `eventRuns.db.js` and
@@ -228,6 +277,34 @@ const LAST_STARTED_SEQ = `
SELECT MAX(seq) AS seq FROM event_run_steps
WHERE run_id = ? AND phase = ? AND status <> 'pending'`
// Phase 5's four, verbatim from `eventPhaseGates.db.js`.
const GATE_OPEN = `
INSERT IGNORE INTO event_run_phase_gates
(run_id, phase, kind, after_seconds, trigger_id, conditions, needed, entered_at, due_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`
const GATE_COUNT = `
UPDATE event_run_phase_gates
SET satisfied_at = CASE WHEN tally + 1 >= needed THEN ? ELSE NULL END,
satisfied_by = CASE WHEN tally + 1 >= needed THEN 'condition' ELSE NULL END,
last_event = ?,
last_event_at = ?,
tally = tally + 1
WHERE id = ? AND satisfied_at IS NULL`
const GATE_SATISFY = `
UPDATE event_run_phase_gates
SET satisfied_at = ?, satisfied_by = ?, forced_by = ?
WHERE id = ? AND satisfied_at IS NULL`
const GATE_OPEN_FOR_TRIGGER = `
SELECT g.* FROM event_run_phase_gates g
JOIN event_runs r ON r.id = g.run_id
WHERE g.trigger_id = ? AND g.satisfied_at IS NULL
AND r.status IN ('running','paused')
AND r.current_phase = g.phase
LIMIT 200`
const MATERIALISE_RUN = `
INSERT IGNORE INTO event_runs (definition_id, version_id, scope, scheduled_for, concurrency_key)
VALUES (?, ?, ?, ?, ?)`
@@ -341,6 +418,7 @@ const rows = (r) => Number(r.affectedRows)
beforeEach(async () => {
if (!available) return
await pool.query('DELETE FROM event_run_phase_gates')
await pool.query('DELETE FROM event_run_steps')
await pool.query('DELETE FROM event_runs')
await pool.query('DELETE FROM event_definitions')
@@ -891,3 +969,131 @@ test('a scheduled run whose started_at is somehow set is left alone', async (t)
const after = (await pool.query('SELECT version_id FROM event_runs WHERE id = ?', [r.insertId]))[0]
assert.equal(Number(after.version_id), 3)
})
// ── Phase 5: the gate statements ───────────────────────────────────────────
/** A run in `running` on one phase, plus an open gate for it. */
async function seedGate({ kind = 'on', needed = 1, trigger = 'uo.champ.boss_up', status = 'running', phase = 'boss' } = {}) {
const { runId } = await seedRun({ status })
await pool.query('UPDATE event_runs SET current_phase = ? WHERE id = ?', [phase, runId])
const g = await pool.query(GATE_OPEN, [
runId,
phase,
kind,
kind === 'after' ? 1800 : null,
kind === 'on' ? trigger : null,
kind === 'on' ? JSON.stringify({ variable: 'region', cmp: 'eq', value: 'Yew' }) : null,
needed,
T0,
kind === 'after' ? later(1800_000) : null,
])
return { runId, gateId: Number(g.insertId) }
}
const gateById = async (id) => (await pool.query('SELECT * FROM event_run_phase_gates WHERE id = ?', [id]))[0]
test('two firings arriving together each count, and exactly one crosses the threshold', async (t) => {
if (needDb(t)) return
const { gateId } = await seedGate({ needed: 2 })
// The whole reason the CASE is inside the UPDATE: a read-then-write would let
// both of these see "1 of 2" and neither satisfy, or both satisfy and advance
// one phase twice.
const [a, b] = await Promise.all([
pool.query(GATE_COUNT, [T0, null, T0, gateId]),
pool.query(GATE_COUNT, [T0, null, T0, gateId]),
])
assert.equal(rows(a) + rows(b), 2, 'both increments land')
const gate = await gateById(gateId)
assert.equal(Number(gate.tally), 2)
assert.ok(gate.satisfied_at, 'and the second one closed it')
assert.equal(gate.satisfied_by, 'condition')
})
test('the threshold is read BEFORE the increment, not after it', async (t) => {
if (needDb(t)) return
// The defect this file was written to catch, and did: MariaDB evaluates SET
// assignments left to right with the values already assigned, so an increment
// placed FIRST makes the CASE after it read `new + 1 >= needed` and close a
// two-firing gate on the first firing. Every stub of this agrees with the
// intent rather than with the server, which is exactly why it survived one.
const { gateId } = await seedGate({ needed: 2 })
await pool.query(GATE_COUNT, [T0, null, T0, gateId])
const half = await gateById(gateId)
assert.equal(Number(half.tally), 1)
assert.equal(half.satisfied_at, null, 'one of two is not two')
await pool.query(GATE_COUNT, [T0, null, T0, gateId])
assert.ok((await gateById(gateId)).satisfied_at, 'and the second one is')
})
test('an increment against a closed gate is refused, not applied', async (t) => {
if (needDb(t)) return
const { gateId } = await seedGate({ needed: 1 })
assert.equal(rows(await pool.query(GATE_COUNT, [T0, null, T0, gateId])), 1)
// `WHERE satisfied_at IS NULL` — and 0 rather than 1, which is the whole
// reason this file exists: `foundRows: true` would answer 1 for a no-op.
assert.equal(rows(await pool.query(GATE_COUNT, [T0, null, T0, gateId])), 0)
assert.equal(Number((await gateById(gateId)).tally), 1, 'the tally does not keep climbing after the phase moved on')
})
test('a force and a deadline race the same guard, and only one of them wins', async (t) => {
if (needDb(t)) return
const { gateId } = await seedGate({ kind: 'after' })
const [forced, elapsed] = await Promise.all([
pool.query(GATE_SATISFY, [T0, 'forced', 7, gateId]),
pool.query(GATE_SATISFY, [T0, 'elapsed', null, gateId]),
])
assert.equal(rows(forced) + rows(elapsed), 1, 'exactly one of the two writes')
const gate = await gateById(gateId)
assert.ok(['forced', 'elapsed'].includes(gate.satisfied_by))
// Whichever won is the one in the row, and the log records that one — never
// both.
assert.equal(gate.satisfied_by === 'forced' ? Number(gate.forced_by) : gate.forced_by, gate.satisfied_by === 'forced' ? 7 : null)
})
test('opening a gate twice writes one row', async (t) => {
if (needDb(t)) return
const { runId } = await seedGate()
// The INSERT IGNORE that makes recovery from a process which died between
// entering a phase and opening its gate uneventful.
const again = await pool.query(GATE_OPEN, [runId, 'boss', 'on', null, 'uo.champ.boss_up', null, 1, later(60_000), null])
assert.equal(rows(again), 0)
const all = await pool.query('SELECT * FROM event_run_phase_gates WHERE run_id = ?', [runId])
assert.equal(all.length, 1)
assert.equal(new Date(all[0].entered_at).getTime(), T0.getTime(), 'and the first entry time is the one that stands')
})
test('the emit path sees only gates whose run is live and on that phase', async (t) => {
if (needDb(t)) return
const live = await seedGate()
const paused = await seedGate({ status: 'paused' })
const cancelled = await seedGate({ status: 'scheduled' })
await pool.query('UPDATE event_runs SET status = ? WHERE id = ?', ['cancelled', cancelled.runId])
const moved = await seedGate()
await pool.query('UPDATE event_runs SET current_phase = ? WHERE id = ?', ['cleanup', moved.runId])
const closed = await seedGate()
await pool.query(GATE_SATISFY, [T0, 'forced', null, closed.gateId])
const found = (await pool.query(GATE_OPEN_FOR_TRIGGER, ['uo.champ.boss_up'])).map((g) => Number(g.id))
assert.deepEqual(found.sort((a, b) => a - b), [live.gateId, paused.gateId].sort((a, b) => a - b))
// `paused` counts: the world does not stop because an operator paused the
// console, and discarding firings during a pause would make pause destructive.
assert.ok(found.includes(paused.gateId))
})
test('a gate goes with its run', async (t) => {
if (needDb(t)) return
// ON DELETE CASCADE, which is what keeps an orphaned gate from outliving the
// run whose phase it was about.
const { runId, gateId } = await seedGate()
await pool.query('DELETE FROM event_run_steps WHERE run_id = ?', [runId])
await pool.query('DELETE FROM event_runs WHERE id = ?', [runId])
assert.equal((await pool.query('SELECT * FROM event_run_phase_gates WHERE id = ?', [gateId])).length, 0)
})