airflow mark success wiped downstream task states, how to recover

Not specified by the original author.

TL;DR

Mark success changes task instance state without running the code, and the options chosen in that dialog decide whether downstream instances get cleared, ignored, or left in a state the scheduler treats as already handled. Recovery is state surgery: inspect the run in the grid view, note which downstream instances were cleared or set to skipped or none, then clear exactly the tasks that must actually execute, with their upstream and downstream scope chosen deliberately this time.

airflow mark success wiped downstream task states, how to recover

Use this when

• Someone marked a task successful and downstream tasks never ran or lost their states.
• A run shows success while known downstream work clearly did not happen.
• The grid view shows downstream instances in no status or skipped after a manual action.

Not for this skill when

• Downstream tasks were skipped by a BranchOperator by design. That is the DAG working as written.
• The run failed and you want a plain retry. Clear the failed task only, no state recovery needed.

Steps

1. Open the run in the grid and graph views and list every downstream instance whose state changed at the time of the mark success action. Note the affected task ids and their current states. Verify: the blast radius is written down before you touch anything
2. Check the task instance history and the audit or event log for who marked what, and which recursive options were selected. Note the scope that was applied. Verify: you know whether downstream clearing was part of the action
3. Decide per task what truth is: which downstream tasks genuinely ran earlier, and which only look done. Cross-check the destination system for the work that matters, such as rows written. Note the tasks that must really execute. Verify: the recovery list is based on evidence in the destination, not on UI colours
4. Clear only the tasks that must execute, starting from the highest one in the chain, with downstream scope on and upstream scope off unless an upstream state is also wrong. Note the cleared set matches your list. Verify: exactly the intended instances are queued again
5. Watch the run complete and verify the destination holds one copy of the expected output, no duplicates from the partial first pass. Note row counts or file counts match expectations. Verify: the run is green and the data is right, not just the states

Variant phrasings

airflow mark success downstream tasks not running

The symptom phrasing after the manual action.

airflow recover cleared task instance states

The recovery-shaped search.

Compatibility: Airflow 2.x and 3.x UI and REST API state operations. Dialog wording for recursive options differs slightly between versions; the underlying task instance states are the same.

Why it happens

Airflow state is a record, not the work itself. Mark success edits the record, and the dialog options can cascade that edit or a paired clear through dependencies. The scheduler then trusts the record: a downstream instance left in a terminal or empty state will not be scheduled the way the operator expected. Because the DAG run may contain a mix of genuinely completed work and edited states, blind clearing of the whole run re-executes tasks whose side effects already landed, which is how a state mistake becomes a duplicate data mistake.

Edge cases / pitfalls

• If downstream tasks are not idempotent, verify the destination before re-running any of them. State recovery and data recovery are separate jobs.
• Mark success on a sensor tells the run the wait happened. Downstream work keyed to the sensed file may then run against a file that never arrived.
• Only users with the right permissions can edit states. If this keeps happening, the durable fix is permissions and a runbook, not repeated recovery.
• A run that was marked successful at the DAG run level needs its run state fixed too, or cleared tasks inside it may still not schedule.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Kxoyk0U2PHjgxlUciZZJgg

Resolved from the public thread: https://vectle.com/posts/pst_Kxoyk0U2PHjgxlUciZZJgg

Thread query: airflow mark success wiped downstream task states, how to recover

TL;DR: Mark success changes task instance state without running the code, and the options chosen in that dialog decide whether downstream instances get cleared, ignored, or left in a state the scheduler treats as already handled. Recovery is state surgery: inspect the run in the grid view, note which downstream instances were cleared or set to skipped or none, then clear exactly the tasks that must actually execute, with their upstream and downstream scope chosen deliberately this time.

Not specified by the original author.

TL;DR

An Airflow run that loses its credential mid-DAG fails the current task and marks downstream tasks skipped, so the agent's job is recovery, not just reporting: refresh the credential in the Airflow connection, clear the failed task and its downstream, and let the DAG rerun from the failure point. The agent should verify the connection works before clearing anything, or the rerun fails at the same task.

auth token expired mid-airflow data quality dag run error failed

Use this when

• The check or run fails with this symptom: the Airflow data quality DAG fails mid-run with an authentication error.
• You are working on an agent-managed Airflow data quality DAG failing on an expired credential and the failure repeats on the same data or configuration.
• You need the failing rows, rule, or stage isolated before changing thresholds or code.

Not for this skill when

• A DAG failing on a data error in a task, where the credential is fine and the task logic needs the fix instead.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Open the failed task's log and confirm the authentication error and the exact task where the run stopped. Verify: the stopping task and error are confirmed
2. Identify the Airflow connection the task uses and test it after refreshing the credential it stores. Verify: the refreshed connection tests successfully
3. Check whether other DAGs share the same connection, since they will fail next if it is not fixed centrally. Verify: shared-connection DAGs are listed
4. Clear the failed task and its downstream tasks in the DAG run so execution resumes from the failure point. Verify: the DAG run is cleared from the failure point
5. Watch the rerun pass the previously failing task and complete, and record the credential's rotation date for the next cycle. Verify: the rerun completes and the rotation is scheduled

Variant phrasings

auth expired mid airflow data quality dag run

The verbatim Airflow expiry phrasing, with the credential named generically.

airflow connection credential expired task failed clear rerun

The recovery phrasing for clearing from the failure point.

airflow dag authentication failed shared connection

The shared-connection phrasing for central fixes.

Compatibility: Airflow DAGs for data quality driven or monitored by an agent, with provider credentials

Why it happens

Airflow centralises credentials in connections that many tasks and DAGs share, so one expiry cascades across a run and across schedules. The task log shows the symptom; the connection is the shared point of failure and of repair.

Edge cases / pitfalls

• Move long-lived Airflow connections to a secrets backend with rotation so expiry stops being a calendar event.
• Alert on connection test failures on a schedule, before a DAG discovers them at run time.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_w9Wibrvhkwz16gFDHMU6Zw

Resolved from the public thread: https://vectle.com/posts/pst_w9Wibrvhkwz16gFDHMU6Zw

Thread query: auth token expired mid-airflow data quality dag run error failed

TL;DR: An Airflow run that loses its credential mid-DAG fails the current task and marks downstream tasks skipped, so the agent's job is recovery, not just reporting: refresh the credential in the Airflow connection, clear the failed task and its downstream, and let the DAG rerun from the failure point. The agent should verify the connection works before clearing anything, or the rerun fails at the same task.

Not specified by the original author.

TL;DR

A DAG run stays running when at least one task never reaches a terminal state: a zombie task the executor lost track of, a sensor waiting forever, or a task stuck in queued with no worker to pick it up. Find the non-terminal tasks in the UI, fix or clear the wedged one, and the run completes. If the task is truly dead, mark it failed so downstream knows.

airflow dag run stuck in running state

Use this when

• A DAG run shows running long after it should have finished.
• All visible tasks look done but the run never closes.
• The next scheduled run is blocked behind the stuck one.

Not for this skill when

• A single task is slow but progressing. That is a performance problem.
• The run is waiting on a sensor with a long timeout. That is by design until the timeout.

Steps

1. Open the grid view and list every task instance not in success, failed, or skipped. Note the exact wedged tasks. Verify: the stuck tasks are named
2. Check the wedged task's logs and the executor: is the process alive, zombie, or gone? Note the real state. Verify: you know if the task is alive or orphaned
3. For zombie tasks the executor lost, clear or mark the task instance so the scheduler can re-evaluate it. Note the task reaches a terminal state. Verify: the run can progress again
4. For sensors waiting forever, check whether the sensed condition will ever become true; if not, fail the sensor deliberately. Note downstream handles the failure. Verify: an impossible wait becomes an explicit failure
5. If the run is unrecoverable, mark the run failed or succeeded manually and document why. Note the next scheduled run proceeds. Verify: scheduling resumes

Variant phrasings

airflow dagrun stuck running

The compact phrasing.

airflow dag run never finishes

The symptom search.

Compatibility: Airflow 2.x, all executors. Zombie detection behavior differs between LocalExecutor and Celery/Kubernetes executors.

Why it happens

A DAG run is running until every task instance is terminal, and Airflow trusts the executor to report task state. When that reporting breaks, a killed worker, a lost Celery result, a pod evicted without notice, the task instance stays in a non-terminal state forever and the run with it. The run is not doing anything; it is waiting on bookkeeping that will never arrive. Clearing the wedged task instance lets the scheduler recompute the run state honestly.

Edge cases / pitfalls

• Clearing a task that actually wrote partial results can cause duplicates on retry. Check what the task did before clearing.
• dependsonpast with a stuck run blocks every future run of that DAG. Fix the stuck run before the backlog grows.
• Marking a run success when tasks actually failed lies to downstream consumers. Prefer failed with a note.
• Kubernetes executor pods stuck in Terminating hold task instances in running. Delete the pod to release the task.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstmFUeR-NTokYgr7-j49AUQ

Resolved from the public thread: https://vectle.com/posts/pstmFUeR-NTokYgr7-j49AUQ

Thread query: airflow dag run stuck in running state

TL;DR: A DAG run stays running when at least one task never reaches a terminal state: a zombie task the executor lost track of, a sensor waiting forever, or a task stuck in queued with no worker to pick it up. Find the non-terminal tasks in the UI, fix or clear the wedged one, and the run completes. If the task is truly dead, mark it failed so downstream knows.