agent loop detected: partition backfill never converges, backfill failed

TL;DR

A backfill that never converges has partitions that fail and retry forever: poisoned data, a broken task for certain keys, or a range that grows as fast as it fills. Measure per-partition progress, quarantine the failing partitions, and complete the rest. An 80 percent backfill with a known bad-partition list beats a 0 percent backfill still looping.

agent loop detected: partition backfill never converges, backfill failed

Use this when

• A partition backfill runs without converging.
• The same partitions fail and retry repeatedly.
• Completed partition count stalls while attempts climb.

Not for this skill when

• The backfill is slow but the completed count grows. That is patience, not a loop.
• The backfill failed fast entirely. That is a different failure.

Steps

1. Measure per-partition state: completed, failed, and attempt counts. Note the distribution. Verify: the stall is localized
2. Identify the partitions that fail repeatedly and read one failure in full. Note the cause. Verify: the poison is named
3. Quarantine the bad partitions: exclude them and let the rest complete. Note the quarantine list. Verify: the good work finishes
4. Fix the quarantined partitions deliberately: data repair, code fix, or manual load. Note the fix. Verify: the bad partitions get real attention
5. Record the convergence: what completed, what was quarantined, what remains. Note the record. Verify: the outcome is explicit, not infinite

Variant phrasings

backfill stuck retrying partitions

The symptom phrasing.

partition backfill poison partitions

The cause-shaped search.

Compatibility: Partitioned backfills on any orchestrator. The pattern is orchestrator-independent.

Why it happens

Backfills are loops over partitions, and a poisoned partition turns the loop into an infinite one: fail, retry, fail, retry, while the completed count never moves. The aggregate view (backfill running) hides the per-partition truth (seven partitions failing forever). Quarantine is the circuit breaker: it lets the healthy majority finish and converts the poisoned minority from an infinite loop into a finite, named work list. Convergence is a decision, not just a property.

Edge cases / pitfalls

• Quarantined partitions are debt. Track them or they become permanent gaps.
• A backfill range defined as up-to-now grows while it runs and never converges by construction. Freeze the end date.
• Retrying the whole backfill after quarantine re-runs completed partitions wastefully. Resume from the quarantine list.
• Some partitions fail from transient downstream issues and succeed later. Distinguish poisoned from unlucky before quarantining.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst__v5HOXU8T60uz70Dhwz6uQ