review agent lost its drafted comments when the runner was preempted: persistence: how to fix

TL;DR

Persist every drafted finding twice: append it to a local checkpoint file the moment it is finalized, and post it incrementally to a PENDING GitHub review. When the runner dies, the next run resumes from the checkpoint instead of starting over, and nothing the agent already wrote is lost.

The problem

review agent lost its drafted comments when the runner was preempted: persistence: how to fix

When to use

• Reviews run on spot, preemptible, or otherwise killable runners
• A preemption wipes out all drafted comments and the rerun starts from zero
• Partial reviews never reach GitHub because the agent only posts at the very end
• You see "runner lost communication" in the job log with no review output

When not to use

• Drafts were lost to a 422 or auth error at submit time (a posting bug, not a persistence bug)
• The runner is stable and the loss came from the agent crashing on bad input
• Reviews are short enough that rerunning from scratch is cheap

Steps

1. Append each finalized finding to a checkpoint file immediately

After each file's findings are finalized, append one JSON line per finding to a checkpoint file. Never batch this to the end of the run.

printf '%s
' "$FINDING_JSON" | tee -a /tmp/review-checkpoint.jsonl

Expected output: a growing JSONL file where every completed file's work survives even if the process is killed a second later.

2. Create a PENDING review first, then add comments incrementally

Open the review with a PENDING event via the reviews API, keep the returned review id, and post each finding as a comment under that review as it is drafted. Pending comments are stored server-side and invisible until submitted.

Expected output: GitHub itself holds a copy of everything drafted; a dead runner loses nothing that was already posted there.

3. Upload the checkpoint as a workflow artifact

At job end (including failure), upload the checkpoint file as an artifact. On retry, download it first and skip every file it already covers.

Expected output: retried runs resume mid-review instead of redoing finished files.

4. On restart, reconcile checkpoint against the PENDING review

Load the checkpoint, list the pending review's comments, and treat the union as the covered set. Skip those files; review only the remainder.

Expected output: no duplicated work and no duplicated comments after a resume.

5. Submit or dismiss the pending review at the end

A finished run submits the pending review; an abandoned run should dismiss it so stale PENDING reviews do not pile up on the PR. Expected output: no orphaned pending reviews cluttering the PR.

Tool compatibility

• GitHub REST reviews API: the PENDING event and per-review comment posting work on github.com and GHES.
• GitHub Actions artifacts: upload on failure via if: always() so the checkpoint survives the preemption.
• gh CLI 2.x: can create the pending review and post comments from shell steps.

Variant phrasings

preemptible runner killed the review agent mid-run

Same fix. If preemptions are frequent, also shrink the unit of work: checkpoint per finding, not per file, so less is lost between writes.

drafted review comments disappeared with the spot VM

The server-side PENDING review (step 2) is the main defense here; the local checkpoint only helps if the disk survives, which on spot VMs it often does not.

review rerun started from zero after preemption

Add the step-4 resume logic. A checkpoint without resume logic is just a log file.

Why it happens

Agents naturally accumulate drafts in memory and post once at the end, which assumes the runner lives for the whole review. Preemptible infrastructure breaks that assumption: the process can die at any minute, and anything not yet flushed to disk or to the API dies with it.

Edge cases

• Two runners racing after a retry: namespace the checkpoint and the pending review body by run id so they do not interleave.
• Artifact retention limits: old checkpoints expire; the PENDING review on GitHub does not, so prefer it as the source of truth.
• PENDING review limits: GitHub allows one pending review per user per PR; reuse it across resumes instead of opening a new one.
• Runner disk is ephemeral: steps 2 and 3 matter more than step 1 on fully ephemeral runners.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_w7H1PXOew3fhpOhC4e1r6w