fix(events): three defects the live rig found, two of them data loss
All checks were successful
PR Checks / client-build (pull_request) Successful in 20s
PR Checks / frozen-manifest (pull_request) Successful in 41s
PR Checks / server-tests (pull_request) Successful in 8m33s

The whole-rig walk (ServUO + sidecar + website) against a real two-phase event.

- **A WS reconnect would have orphaned every live resource.** The backfill
  replays the last several `server.hello` frames in order — this rig saw three,
  each with a different `bootId` — so every replayed frame reads as a restart,
  and the intermediate ones compare a resource stamped with the CURRENT boot
  against a boot that ended hours ago. The row is then `orphaned`: a live crier
  line core will never take down again, lost to nothing worse than the website
  reconnecting. Gated on `!fromBackfill`, the rule the engagement fan-out and
  the SSE broadcast beside it already state. The website-was-down case is not
  missed — core asks every module at its own boot.
- **The shard explains its refusals and the run log dropped the explanation.**
  A 403 body reads `{"reason":"admin write plane disabled"}`; `legError` looks
  for `data.message`, finds nothing, and reports "sidecar responded 403". For a
  staff member clicking a button that is survivable. For an event that ran at
  four in the morning the run log is the only place anyone will learn why.
- **The "not retried" clause explained the wrong thing on a permanent status.**
  A 403 will not succeed on any attempt, so telling an operator it was not
  retried "because a repeat would announce twice" points them at a policy
  decision instead of at the switch they have to flip. The clause is now added
  only where a retry was genuinely given up, and 403/404 join the statuses the
  keyed verbs treat as terminal.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2026-09-04 07:36:03 -05:00
parent 57419111e6
commit 021f191f65
4 changed files with 132 additions and 11 deletions

View File

@@ -106,6 +106,31 @@ async function currentBootId() {
}
}
// Statuses that will never succeed however many times they are tried: a data
// refusal, a bad token, a switched-off write plane, a protocol mismatch. Named
// here rather than folded into `shardAnnounce.classify` because 403 is reachable
// only from the `/admin/*` verbs — the announce leg posts to the town crier,
// which the admin write plane does not gate — and widening a shared classifier
// for a case its own caller cannot produce is how a shared rule stops being one.
const PERMANENT_STATUSES = new Set([400, 401, 403, 404, 409])
/**
* What the shard actually said, in its own words.
*
* **The sidecar explains its refusals and `legError` drops the explanation**, and
* this was worth its own helper the moment an event started making these calls
* unattended. A `403` body reads `{"reason":"admin write plane disabled"}`;
* `legError` looks for `data.message`, finds nothing, and falls back to "sidecar
* responded 403". For a staff member clicking a button that is survivable — they
* know what they just switched off. For an event that ran at four in the morning,
* the run log is the only place anyone will ever learn why, and "403" is not an
* answer an operator can act on.
*/
function sidecarReason(result, what) {
const data = (result && result.data) || {}
return data.reason || data.message || (result && result.error) || `the shard refused the ${what}`
}
/**
* The sidecar's answer, as an event outcome.
*
@@ -117,8 +142,13 @@ async function currentBootId() {
* a leg's own `classify()`.
*/
function sidecarFailure(result, what) {
const { outcome, error } = classifySidecarWrite(result)
return { ok: false, retry: outcome === 'retry', error: error || `the shard refused the ${what}` }
const { outcome } = classifySidecarWrite(result)
const permanent = PERMANENT_STATUSES.has(result && result.status)
return {
ok: false,
retry: outcome === 'retry' && !permanent,
error: sidecarReason(result, what),
}
}
/** Split an authored text block into crier lines, and say why it is not one. */
@@ -228,11 +258,20 @@ const ACTIONS = [
// that retry away; that is the trade, taken knowingly, because the failure
// this refuses to risk is announcing twice to everyone online. Phase 11
// puts an idempotency key on the wire and this line is what changes.
const { error } = classifySidecarWrite(result)
//
// **The clause is only added where a retry was genuinely given up**, and
// the rig is what made that distinction matter. A 403 — the shard's admin
// write plane switched off — will not succeed on any attempt, so telling an
// operator it was "not retried because a repeat would announce twice" points
// them at a policy decision when what they need is the sentence the shard
// already wrote: "admin write plane disabled". A reason that explains the
// wrong thing is worse than a bare status code.
const reason = sidecarReason(result, 'broadcast')
if (PERMANENT_STATUSES.has(result.status)) return { ok: false, retry: false, error: reason }
return {
ok: false,
retry: false,
error: `${error || 'the shard refused the broadcast'} (not retried: a repeat would announce twice)`,
error: `${reason} (not retried: a repeat would announce twice)`,
}
},
},
@@ -527,6 +566,8 @@ module.exports = {
MAX_NEWS_TITLE,
MAX_NEWS_BODY,
MAX_OPTIONS,
PERMANENT_STATUSES,
sidecarReason,
resourceId,
crierLines,
reconcileByBootId,