From 8195454201f15a2a8e4254fa05457c8207e7e8db Mon Sep 17 00:00:00 2001 From: wtclaude Date: Mon, 24 Aug 2026 12:43:24 -0500 Subject: [PATCH] ci(release): show the error body, retry the POST, and sweep for orphan tags installer#22's release run built every artifact, pushed its tag, then took a 500 from POST /releases one second later and exited 22 -- leaving the tag orphaned with no binaries. Re-running published it unchanged, so the 500 was a race with the tag push rather than a bad request. This repo's release step has the same two gaps verbatim, and it is where the whole problem was first seen. `curl -sSf` prints no response body on an error status, so such a failure leaves only "curl: (22) ... error: 500" in the log and the cause has to be inferred from timestamps. Every call now captures the body and prints it on failure. Nothing retried, so a transient 5xx became a permanent orphan. The POST now retries five times with a 5/10/15/20s backoff. 4xx is deliberately not retried: a bad token or a malformed body will not improve by being sent again. The asset upload gets the same treatment. That matters more here than anywhere else: this release ships an overlay tarball and a SHA256SUMS, and a release whose checksums do not cover the tarball they advertise is worse than no release, because that file is the trust anchor and the installer verifies against it. The third gap is the one this repo proves. The orphan-tag recovery in the plan step is VERSION-SCOPED -- it computes VERSION from the newest tag plus the bump, then only checks refs/tags/v${VERSION}. That recovers an orphan on the very next run and is useless afterwards, because once any releasable commit lands the next run computes a NEW version and never looks at the old tag again. v0.1.0 was the proof, and the proof is pointed: the commit that ADDED that recovery was itself typed "fix(release): preflight credentials and recover the orphaned v0.1.0 tag", so it bumped the version to v0.1.1 -- and the run that introduced the recovery stepped straight past the tag it was written to rescue. The tag stayed orphaned from 2026-08-04 until today. So the plan step now sweeps every v* tag and warns about any without a release. It warns rather than recovers, deliberately: publishing an old version would mean building today's tree and shipping it under a tag whose tree it is not, which is worse than the inconsistency it fixes. It never fails the run either -- a sweep that can break a good release is a sweep someone will delete. v0.1.0 itself is deleted, on the org lead's decision. Nothing referenced it: it is three releases behind, and no published bundle names it -- not even bundle-2026.08.04, because the tag never had a release for a bundle to point at. It was 724262548b8193ff7d31ed92136ad449464d66e6, the merge of #7, recorded here so the tag can be recreated if that turns out to be wrong. Verified by extracting every run block from the YAML: bash -n clean across all of them, the YAML parses, no empty template token, the asset loop still the tarball-and-checksums pair rather than link's three binaries, the retry loop exercised against a stubbed curl across seven cases, and the sweep run against the real repositories -- reporting v0.1.0 before the deletion and clean after. Typed ci(...) rather than fix(...) on purpose: the plan step bumps on feat/fix, and this changes no artifact, so a release here would be an empty one. That is the same rule the fix commit above tripped over. Co-Authored-By: Claude --- .gitea/workflows/release.yml | 104 ++++++++++++++++++++++++++++++++--- 1 file changed, 96 insertions(+), 8 deletions(-) diff --git a/.gitea/workflows/release.yml b/.gitea/workflows/release.yml index 821ed8f..f59fbac 100644 --- a/.gitea/workflows/release.yml +++ b/.gitea/workflows/release.yml @@ -176,6 +176,39 @@ jobs: fi fi + # ── Orphan sweep ──────────────────────────────────────────────── + # + # The check above is VERSION-SCOPED: it only ever asks about the one + # version this run computed. That is enough to recover an orphan on + # the very next run, and useless afterwards — once any releasable + # commit lands, the next run computes a NEW version, never looks at + # the old tag again, and the orphan becomes permanent and silent. + # + # servuo-plugins v0.1.0 is the proof, and the proof is pointed: the + # commit that ADDED the recovery above was itself typed + # `fix(release): ... recover the orphaned v0.1.0 tag`, so it bumped to + # v0.1.1 — and the run that introduced the recovery stepped straight + # past the tag it was written to rescue. That tag is still orphaned. + # + # So every v* tag is checked, and anything missing a release is + # WARNED about. Deliberately not recovered: publishing an old version + # would mean building today's tree and shipping it under a tag whose + # tree it is not, which is worse than the inconsistency it fixes. + # A human decides whether to recover or drop it. + # + # Never fails the run. A sweep that can break a good release is a + # sweep someone will delete. + ORPHANS="" + for T in $(git tag -l 'v*' --sort=-v:refname); do + T_HTTP="$(curl -s -o /dev/null -w '%{http_code}' \ + -H "Authorization: token $(printf '%s' "${REGISTRY_TOKEN:-}" | tr -d '\r\n')" \ + "https://${GITEA_HOST}/api/v1/repos/${REPO}/releases/tags/${T}" || echo 000)" + [ "$T_HTTP" = "404" ] && ORPHANS="${ORPHANS} ${T}" + done + if [ -n "${ORPHANS}" ]; then + echo "::warning::Tags with no release:${ORPHANS} — a run failed after tagging. Publish or delete them; this job will not do either." + fi + # Changelog range. A recovery run has nothing after the tag, so # summarize what the tag itself contains rather than emitting an empty # list: the range that produced it, i.e. previous-tag..this-tag. @@ -477,18 +510,73 @@ jobs: # corrupt the Authorization header. CI_TOKEN="$(printf '%s' "${REGISTRY_TOKEN}" | tr -d '\r\n')" - REL_ID="$(curl -sSf -X POST "${API}/releases" \ - -H "Authorization: token ${CI_TOKEN}" \ - -H "Content-Type: application/json" \ - -d "$(jq -n --arg tag "$TAG" --arg body "$BODY" \ - '{tag_name:$tag, name:$tag, body:$body, draft:false, prerelease:false}')" \ - | jq -r '.id')" + PAYLOAD="$(jq -n --arg tag "$TAG" --arg body "$BODY" \ + '{tag_name:$tag, name:$tag, body:$body, draft:false, prerelease:false}')" + + # installer#22's release run failed exactly here: it landed one second + # after the tag push and Gitea answered 500, having not finished + # processing the pushed tag. Re-running published the same artifacts + # untouched, so it was a race, not a bad request — but the tag sat + # orphaned until a human noticed. + # + # Two things made that worse than it needed to be. + # + # 1. `curl -sSf` prints NO response body on an error status, so all the + # log carried was "curl: (22) ... error: 500" and the cause had to be + # inferred from timestamps. Capture the body and print it. + # 2. Nothing retried, so a transient 5xx became a permanent orphan. + # + # 4xx is deliberately NOT retried: a bad token or a malformed body does + # not improve by being sent again, and retrying only turns a clear + # failure into a slow one. + REL_ID="" + for attempt in 1 2 3 4 5; do + HTTP="$(curl -s -o /tmp/rel.json -w '%{http_code}' -X POST "${API}/releases" \ + -H "Authorization: token ${CI_TOKEN}" \ + -H "Content-Type: application/json" \ + -d "${PAYLOAD}" || echo 000)" + + if [ "$HTTP" = "201" ] || [ "$HTTP" = "200" ]; then + REL_ID="$(jq -r '.id' /tmp/rel.json)" + break + fi + + echo "::warning::POST /releases attempt ${attempt} returned HTTP ${HTTP}" + echo "--- response body ---" + cat /tmp/rel.json || true + echo + echo "---------------------" + + case "$HTTP" in + 4*) echo "::error::HTTP ${HTTP} is a client error - not retrying."; exit 1 ;; + esac + + if [ "$attempt" = 5 ]; then + echo "::error::POST /releases still failing after 5 attempts. Tag ${TAG} is pushed but has no release." + echo "::error::Re-run this workflow - the plan step detects the orphan tag and republishes it." + exit 1 + fi + sleep $(( attempt * 5 )) + done + + if [ -z "$REL_ID" ] || [ "$REL_ID" = "null" ]; then + echo "::error::Release created but no id came back; refusing to upload assets blind." + exit 1 + fi echo "Created release ${TAG} (id=${REL_ID})" for f in "${TARBALL}" SHA256SUMS; do - curl -sSf -X POST "${API}/releases/${REL_ID}/assets?name=${f}" \ + # Same treatment. An upload that fails quietly leaves a release whose + # SHA256SUMS does not cover every artifact it advertises, which is + # worse than no release at all -- that file is the trust anchor. + HTTP="$(curl -s -o /tmp/asset.json -w '%{http_code}' -X POST "${API}/releases/${REL_ID}/assets?name=${f}" \ -H "Authorization: token ${CI_TOKEN}" \ - -F "attachment=@dist/${f}" >/dev/null + -F "attachment=@dist/${f}" || echo 000)" + if [ "$HTTP" != "201" ] && [ "$HTTP" != "200" ]; then + echo "::error::uploading ${f} returned HTTP ${HTTP}" + cat /tmp/asset.json || true + exit 1 + fi echo " uploaded ${f}" done