agent's bulk ticket update failed halfway: partial-failure resume

TL;DR

A bulk update dying halfway leaves the fleet half-updated: rerunning from the start wastes work and double-applies side effects. Checkpoint every update durably, make each update idempotent so replays are safe, and resume from the last checkpoint instead of restarting. Bulk work must be resumable by design, not by luck.

The query

agent's bulk ticket update failed halfway: partial-failure resume

Use this when

• Bulk updates die mid-run
• Half the tickets got updated
• You are building bulk operations
• Auditing bulk job reliability

Not for

• Bulk update UI design
• Selecting tickets for bulk actions
• Learning the ticket API
• Webhook design

Steps

1. Checkpoint each completed update durably

Write the completed ticket id to durable storage after every successful update, not at the end. The checkpoint is the resume point; without per-item checkpoints, a crash loses all progress information.

Expected output: per-item durable checkpoints.

2. Make every update idempotent

Each update must be safe to replay: setting a field to a value (not toggling), adding a tag only if absent, using idempotency keys on side-effecting calls. Idempotency makes resume safe even when the checkpoint is slightly behind reality.

Expected output: idempotent update operations.

3. Resume from the checkpoint, skip completed

On restart, read the checkpoint and skip already-completed tickets. The resume path is the normal path for long jobs; design it first, not as an afterthought. Restart-from-zero is the failure mode.

Expected output: checkpoint-driven resume skipping completed work.

4. Quarantine per-item failures, do not abort the batch

A ticket that fails validation or errors should go to a failure list while the batch continues. One bad ticket must not kill a thousand good updates. Abort only on systemic failures (auth, API down), not item failures.

Expected output: per-item failure quarantine with batch continuation.

5. Report the three lists at the end

Every run reports: completed, failed (with reasons), and skipped-as-already-done. The three lists make partial failure legible: operators see exactly what happened and what needs attention. Silent partial success is how half-updated fleets hide.

Expected output: completed/failed/skipped reporting per run.

Template: the checklist

Bulk update resume:
[ ] Durable per-item checkpoints
[ ] Idempotent update operations
[ ] Checkpoint-driven resume
[ ] Per-item failure quarantine
[ ] Three-list reporting (completed/failed/skipped)

Variant phrasings

bulk update died halfway

Steps 1 and 3. Checkpoint, resume.

half tickets updated bulk job failed

Steps 3 and 5. Resume, then report the three lists.

resume failed bulk operation

Steps 1 through 3 as the resume design.

Why it happens

Bulk jobs get written as simple loops: for each ticket, update. The loop works on the happy path and has no vocabulary for interruption: no checkpoints, no resume, no failure quarantine. Production interrupts loops constantly (rate limits, deploys, crashes), and each interruption meets a design with no answer. Resumability is not a feature of bulk work; it is the precondition.

Edge cases

• Checkpoint write failing: if the checkpoint itself cannot be written, stop rather than proceed blind. Uncheckpointed progress is lost progress.
• Updates with external side effects: notifications sent cannot be unsent. Idempotency must cover side effects, or quarantine them separately.
• The ticket set changing mid-run: new tickets matching the query after the run started. Decide: fixed snapshot at start, or live query with dedup.
• Very large batches: checkpoint storage and the skip-scan must scale. Paginate the resume efficiently.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Sq5W2Zlbq-iFSW11WVIM-Q