fix(events): three defects the live rig found, two of them data loss
The whole-rig walk (ServUO + sidecar + website) against a real two-phase event.
- **A WS reconnect would have orphaned every live resource.** The backfill
replays the last several `server.hello` frames in order — this rig saw three,
each with a different `bootId` — so every replayed frame reads as a restart,
and the intermediate ones compare a resource stamped with the CURRENT boot
against a boot that ended hours ago. The row is then `orphaned`: a live crier
line core will never take down again, lost to nothing worse than the website
reconnecting. Gated on `!fromBackfill`, the rule the engagement fan-out and
the SSE broadcast beside it already state. The website-was-down case is not
missed — core asks every module at its own boot.
- **The shard explains its refusals and the run log dropped the explanation.**
A 403 body reads `{"reason":"admin write plane disabled"}`; `legError` looks
for `data.message`, finds nothing, and reports "sidecar responded 403". For a
staff member clicking a button that is survivable. For an event that ran at
four in the morning the run log is the only place anyone will learn why.
- **The "not retried" clause explained the wrong thing on a permanent status.**
A 403 will not succeed on any attempt, so telling an operator it was not
retried "because a repeat would announce twice" points them at a policy
decision instead of at the switch they have to flip. The clause is now added
only where a retry was genuinely given up, and 403/404 join the statuses the
keyed verbs treat as terminal.
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -106,6 +106,31 @@ async function currentBootId() {
|
||||
}
|
||||
}
|
||||
|
||||
// Statuses that will never succeed however many times they are tried: a data
|
||||
// refusal, a bad token, a switched-off write plane, a protocol mismatch. Named
|
||||
// here rather than folded into `shardAnnounce.classify` because 403 is reachable
|
||||
// only from the `/admin/*` verbs — the announce leg posts to the town crier,
|
||||
// which the admin write plane does not gate — and widening a shared classifier
|
||||
// for a case its own caller cannot produce is how a shared rule stops being one.
|
||||
const PERMANENT_STATUSES = new Set([400, 401, 403, 404, 409])
|
||||
|
||||
/**
|
||||
* What the shard actually said, in its own words.
|
||||
*
|
||||
* **The sidecar explains its refusals and `legError` drops the explanation**, and
|
||||
* this was worth its own helper the moment an event started making these calls
|
||||
* unattended. A `403` body reads `{"reason":"admin write plane disabled"}`;
|
||||
* `legError` looks for `data.message`, finds nothing, and falls back to "sidecar
|
||||
* responded 403". For a staff member clicking a button that is survivable — they
|
||||
* know what they just switched off. For an event that ran at four in the morning,
|
||||
* the run log is the only place anyone will ever learn why, and "403" is not an
|
||||
* answer an operator can act on.
|
||||
*/
|
||||
function sidecarReason(result, what) {
|
||||
const data = (result && result.data) || {}
|
||||
return data.reason || data.message || (result && result.error) || `the shard refused the ${what}`
|
||||
}
|
||||
|
||||
/**
|
||||
* The sidecar's answer, as an event outcome.
|
||||
*
|
||||
@@ -117,8 +142,13 @@ async function currentBootId() {
|
||||
* a leg's own `classify()`.
|
||||
*/
|
||||
function sidecarFailure(result, what) {
|
||||
const { outcome, error } = classifySidecarWrite(result)
|
||||
return { ok: false, retry: outcome === 'retry', error: error || `the shard refused the ${what}` }
|
||||
const { outcome } = classifySidecarWrite(result)
|
||||
const permanent = PERMANENT_STATUSES.has(result && result.status)
|
||||
return {
|
||||
ok: false,
|
||||
retry: outcome === 'retry' && !permanent,
|
||||
error: sidecarReason(result, what),
|
||||
}
|
||||
}
|
||||
|
||||
/** Split an authored text block into crier lines, and say why it is not one. */
|
||||
@@ -228,11 +258,20 @@ const ACTIONS = [
|
||||
// that retry away; that is the trade, taken knowingly, because the failure
|
||||
// this refuses to risk is announcing twice to everyone online. Phase 11
|
||||
// puts an idempotency key on the wire and this line is what changes.
|
||||
const { error } = classifySidecarWrite(result)
|
||||
//
|
||||
// **The clause is only added where a retry was genuinely given up**, and
|
||||
// the rig is what made that distinction matter. A 403 — the shard's admin
|
||||
// write plane switched off — will not succeed on any attempt, so telling an
|
||||
// operator it was "not retried because a repeat would announce twice" points
|
||||
// them at a policy decision when what they need is the sentence the shard
|
||||
// already wrote: "admin write plane disabled". A reason that explains the
|
||||
// wrong thing is worse than a bare status code.
|
||||
const reason = sidecarReason(result, 'broadcast')
|
||||
if (PERMANENT_STATUSES.has(result.status)) return { ok: false, retry: false, error: reason }
|
||||
return {
|
||||
ok: false,
|
||||
retry: false,
|
||||
error: `${error || 'the shard refused the broadcast'} (not retried: a repeat would announce twice)`,
|
||||
error: `${reason} (not retried: a repeat would announce twice)`,
|
||||
}
|
||||
},
|
||||
},
|
||||
@@ -527,6 +566,8 @@ module.exports = {
|
||||
MAX_NEWS_TITLE,
|
||||
MAX_NEWS_BODY,
|
||||
MAX_OPTIONS,
|
||||
PERMANENT_STATUSES,
|
||||
sidecarReason,
|
||||
resourceId,
|
||||
crierLines,
|
||||
reconcileByBootId,
|
||||
|
||||
Reference in New Issue
Block a user