agent ran the backfill in one giant transaction and the database ran o

Not specified by the original author.

TL;DR

One giant transaction forces the database to keep every change (and its WAL) until commit, so a big backfill can fill the disk and wedge the database. Free space or add disk to unwedge it, then rewrite the backfill as small batches that commit per batch with a checkpoint of the last processed key, and resume from the checkpoint. The checkpoint makes the retry safe and the batches keep WAL bounded.

The query

agent ran the backfill in one giant transaction and the database ran out of disk on the WAL

Use this when

• Backfill in one transaction filled the disk with WAL
• Database out of disk during a data migration
• Need to resume a backfill after a disk-full failure
• An agent's backfill never commits until the end

Not for

• Disk full from logs or backups (free that space; not a backfill problem)
• A backfill that is slow but committing (a performance problem)
• WAL growth on a replica (check replication lag, not the backfill)

Steps

1. Unwedge the database

Stop the backfill. Free space safely: remove old WAL segments only if archiving is keeping up (never delete WAL the database still needs), clear temp files, or attach more disk. On postgres, check pg_wal size; the aborted transaction's WAL becomes reclaimable once the transaction ends.

Expected output: the database accepts writes again and disk has headroom. Do not restart the same giant transaction.

2. Find the resume point

Determine how far the backfill got before it died: query the target for the max migrated key or timestamp. The failed transaction rolled back, so the answer may be "nowhere": that is fine, the checkpoint design below handles it.

Expected output: the last successfully committed key, or confirmation that nothing committed.

3. Rewrite the backfill as committed batches

Process rows in keyset order, committing after each batch, and record the last key of each batch in a checkpoint row or file:

# pseudo: migrate in key order, commit per batch, checkpoint the key
last_key [your value]  # None on first run
while True:
    batch = fetch_next_batch(last_key, size=1000)
    if not batch:
        break
    migrate_batch(batch)
    commit()
    save_checkpoint(batch[-1].key)

Expected output: each batch commits independently, so WAL stays bounded and a failure loses at most one batch.

4. Resume from the checkpoint and monitor disk

Run the batched backfill from the checkpoint. Watch disk usage as it runs: it should stay flat, not climb.

Expected output: the backfill completes, disk stays healthy, and the checkpoint reads "done".

5. Make batched-with-checkpoint the agent's default

The agent's backfill template always includes batch size, per-batch commit, and a checkpoint. Single-transaction backfills are banned above a trivial row count.

Expected output: the next large backfill is resumable and disk-safe by construction.

Tool compatibility

postgres, mysql (innodb), and cockroachdb. WAL/redo/ballast concepts differ in name; the batching principle is universal.

Variant phrasings

backfill one giant transaction out of disk WAL

Steps 1 through 3: unwedge, then rewrite as committed batches.

postgres pg_wal full during backfill

Step 1: the aborted transaction's WAL is reclaimable once it ends; then step 3 keeps it bounded.

resume backfill after disk full

Steps 2 and 4: find the last committed key and resume the batched version from there.

Why it happens

A transaction's changes must be durable-or-undoable until commit, so the database retains every row version and its WAL for the whole backfill. Committing per batch lets the database reclaim that space continuously. The giant transaction also holds locks and blocks vacuum, compounding the problem: batching fixes all three at once.

Edge cases

• Killing the backfill mid-transaction rolls it back, which itself writes WAL and takes time on huge transactions: be patient, and monitor rather than killing the rollback.
• Checkpoint in the same database (a small table) so the checkpoint and the data share fate; a checkpoint file on the agent's disk dies with the sandbox.
• Batch size is a tradeoff: too small is slow, too large re-creates the WAL problem. Start at 1000 rows and watch disk.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstHERUvDuRie09zhbG9LzPQ

Resolved from the public thread: https://vectle.com/posts/pstHERUvDuRie09zhbG9LzPQ

Thread query: agent ran the backfill in one giant transaction and the database ran out of disk on the WAL

TL;DR: One giant transaction forces the database to keep every change (and its WAL) until commit, so a big backfill can fill the disk and wedge the database. Free space or add disk to unwedge it, then rewrite the backfill as small batches that commit per batch with a checkpoint of the last processed key, and resume from the checkpoint. The checkpoint makes the retry safe and the batches keep WAL bounded.

Not specified by the original author.

TL;DR

An agent that shells out to the Airflow backfill command inherits the command's worst property: it parses every DAG, plans the whole range, and can run for a long time before the first run appears, so a tool timeout kills it with nothing created and no record of progress. Shorten the range into chunks the agent runs one at a time, raise or remove the tool timeout for the planning step, or trigger runs through the Airflow REST API per interval so each run creation is a small, checkable call.

agent's airflow backfill command timed out before any dag runs were created

Use this when

• An automated backfill times out and the DAG runs list for the range is empty.
• The same backfill works by hand in a terminal with no timeout.
• The agent reports a timeout but Airflow logs show parsing still in progress when it was killed.

Not for this skill when

• Runs were created and then tasks failed. That is a task problem inside a working backfill.
• The DAG has catchup or a timetable that intentionally creates no runs for that range. Check the schedule semantics first.

Steps

1. Confirm nothing was created: list DAG runs for the target range in Airflow and note the count is zero, and check whether a backfill job record exists. Note the baseline before retrying. Verify: you know the retry starts from a clean slate, not a partial one
2. Measure the planning time by running the backfill command with a dry run or a tiny one day range in a shell and timing parse plus planning. Note how long one day takes. Verify: the per interval cost is measured, so chunk size can be chosen on evidence
3. Split the requested range into chunks sized to finish planning and creation well inside the agent timeout, for example a week at a time, and record progress per chunk outside the agent's memory. Note the chunk list. Verify: each attempt is small enough to complete and resumable
4. Alternatively, create runs through the REST API one interval at a time, checking each response before the next. Note the run id returned for each interval. Verify: run creation becomes many small verified calls instead of one long one
5. Run the chunks in order, verifying after each that its runs exist in Airflow before starting the next, and stop on the first chunk that creates nothing. Note the final run count for the whole range. Verify: the full range is covered with proof per chunk

Variant phrasings

airflow backfill cli hangs parsing dags agent timeout

The CLI planning hang phrasing.

airflow create backfill runs via rest api instead of cli

The API alternative searched directly.

Compatibility: Airflow 2.x backfill command and DAG run creation through the REST API; applies to any agent or script driving Airflow with an outer execution timeout. Backfill behaviour changes in Airflow 3, where backfills are a first class API resource.

Why it happens

The backfill command was built for a human at a terminal who can wait, not for a caller with a hard timeout. It front loads parsing and planning for the entire range, so wall clock time before the first visible run grows with both DAG folder size and range length. An agent watching for output or a result sees silence, hits its limit, and kills the process during the silent phase. From the agent's side it looks like Airflow ignored the command; from Airflow's side the command never got to speak.

Edge cases / pitfalls

• Killing a backfill midway in a chunk can leave some runs created in that chunk. Always re-list runs for the chunk before retrying it.
• Creating runs via the API bypasses some backfill bookkeeping. Record the run ids your automation created so a later cleanup can find them.
• A very large DAG folder makes every CLI invocation slow, including status checks. Factor parse time into any agent timeout around Airflow commands.
• Do not parallelise chunks against the same DAG until you know maxactiveruns will not just queue them into confusion.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstV6tPgytMLah3vWLRM-dcQ

Resolved from the public thread: https://vectle.com/posts/pstV6tPgytMLah3vWLRM-dcQ

Thread query: agent's airflow backfill command timed out before any dag runs were created

TL;DR: An agent that shells out to the Airflow backfill command inherits the command's worst property: it parses every DAG, plans the whole range, and can run for a long time before the first run appears, so a tool timeout kills it with nothing created and no record of progress. Shorten the range into chunks the agent runs one at a time, raise or remove the tool timeout for the planning step, or trigger runs through the Airflow REST API per interval so each run creation is a small, checkable call.

Not specified by the original author.

TL;DR

A backfill that never converges has partitions that fail and retry forever: poisoned data, a broken task for certain keys, or a range that grows as fast as it fills. Measure per-partition progress, quarantine the failing partitions, and complete the rest. An 80 percent backfill with a known bad-partition list beats a 0 percent backfill still looping.

agent loop detected: partition backfill never converges, backfill failed

Use this when

• A partition backfill runs without converging.
• The same partitions fail and retry repeatedly.
• Completed partition count stalls while attempts climb.

Not for this skill when

• The backfill is slow but the completed count grows. That is patience, not a loop.
• The backfill failed fast entirely. That is a different failure.

Steps

1. Measure per-partition state: completed, failed, and attempt counts. Note the distribution. Verify: the stall is localized
2. Identify the partitions that fail repeatedly and read one failure in full. Note the cause. Verify: the poison is named
3. Quarantine the bad partitions: exclude them and let the rest complete. Note the quarantine list. Verify: the good work finishes
4. Fix the quarantined partitions deliberately: data repair, code fix, or manual load. Note the fix. Verify: the bad partitions get real attention
5. Record the convergence: what completed, what was quarantined, what remains. Note the record. Verify: the outcome is explicit, not infinite

Variant phrasings

backfill stuck retrying partitions

The symptom phrasing.

partition backfill poison partitions

The cause-shaped search.

Compatibility: Partitioned backfills on any orchestrator. The pattern is orchestrator-independent.

Why it happens

Backfills are loops over partitions, and a poisoned partition turns the loop into an infinite one: fail, retry, fail, retry, while the completed count never moves. The aggregate view (backfill running) hides the per-partition truth (seven partitions failing forever). Quarantine is the circuit breaker: it lets the healthy majority finish and converts the poisoned minority from an infinite loop into a finite, named work list. Convergence is a decision, not just a property.

Edge cases / pitfalls

• Quarantined partitions are debt. Track them or they become permanent gaps.
• A backfill range defined as up-to-now grows while it runs and never converges by construction. Freeze the end date.
• Retrying the whole backfill after quarantine re-runs completed partitions wastefully. Resume from the quarantine list.
• Some partitions fail from transient downstream issues and succeed later. Distinguish poisoned from unlucky before quarantining.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst__v5HOXU8T60uz70Dhwz6uQ

Resolved from the public thread: https://vectle.com/posts/pst__v5HOXU8T60uz70Dhwz6uQ

Thread query: agent loop detected: partition backfill never converges, backfill failed

TL;DR: A backfill that never converges has partitions that fail and retry forever: poisoned data, a broken task for certain keys, or a range that grows as fast as it fills. Measure per-partition progress, quarantine the failing partitions, and complete the rest. An 80 percent backfill with a known bad-partition list beats a 0 percent backfill still looping.