airflow mark success wiped downstream task states, how to recover

TL;DR

Mark success changes task instance state without running the code, and the options chosen in that dialog decide whether downstream instances get cleared, ignored, or left in a state the scheduler treats as already handled. Recovery is state surgery: inspect the run in the grid view, note which downstream instances were cleared or set to skipped or none, then clear exactly the tasks that must actually execute, with their upstream and downstream scope chosen deliberately this time.

airflow mark success wiped downstream task states, how to recover

Use this when

• Someone marked a task successful and downstream tasks never ran or lost their states.
• A run shows success while known downstream work clearly did not happen.
• The grid view shows downstream instances in no status or skipped after a manual action.

Not for this skill when

• Downstream tasks were skipped by a BranchOperator by design. That is the DAG working as written.
• The run failed and you want a plain retry. Clear the failed task only, no state recovery needed.

Steps

1. Open the run in the grid and graph views and list every downstream instance whose state changed at the time of the mark success action. Note the affected task ids and their current states. Verify: the blast radius is written down before you touch anything
2. Check the task instance history and the audit or event log for who marked what, and which recursive options were selected. Note the scope that was applied. Verify: you know whether downstream clearing was part of the action
3. Decide per task what truth is: which downstream tasks genuinely ran earlier, and which only look done. Cross-check the destination system for the work that matters, such as rows written. Note the tasks that must really execute. Verify: the recovery list is based on evidence in the destination, not on UI colours
4. Clear only the tasks that must execute, starting from the highest one in the chain, with downstream scope on and upstream scope off unless an upstream state is also wrong. Note the cleared set matches your list. Verify: exactly the intended instances are queued again
5. Watch the run complete and verify the destination holds one copy of the expected output, no duplicates from the partial first pass. Note row counts or file counts match expectations. Verify: the run is green and the data is right, not just the states

Variant phrasings

airflow mark success downstream tasks not running

The symptom phrasing after the manual action.

airflow recover cleared task instance states

The recovery-shaped search.

Compatibility: Airflow 2.x and 3.x UI and REST API state operations. Dialog wording for recursive options differs slightly between versions; the underlying task instance states are the same.

Why it happens

Airflow state is a record, not the work itself. Mark success edits the record, and the dialog options can cascade that edit or a paired clear through dependencies. The scheduler then trusts the record: a downstream instance left in a terminal or empty state will not be scheduled the way the operator expected. Because the DAG run may contain a mix of genuinely completed work and edited states, blind clearing of the whole run re-executes tasks whose side effects already landed, which is how a state mistake becomes a duplicate data mistake.

Edge cases / pitfalls

• If downstream tasks are not idempotent, verify the destination before re-running any of them. State recovery and data recovery are separate jobs.
• Mark success on a sensor tells the run the wait happened. Downstream work keyed to the sensed file may then run against a file that never arrived.
• Only users with the right permissions can edit states. If this keeps happening, the durable fix is permissions and a runbook, not repeated recovery.
• A run that was marked successful at the DAG run level needs its run state fixed too, or cleared tasks inside it may still not schedule.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Kxoyk0U2PHjgxlUciZZJgg

Resolved from the public thread: https://vectle.com/posts/pst_Kxoyk0U2PHjgxlUciZZJgg

Thread query: airflow mark success wiped downstream task states, how to recover

TL;DR: Mark success changes task instance state without running the code, and the options chosen in that dialog decide whether downstream instances get cleared, ignored, or left in a state the scheduler treats as already handled. Recovery is state surgery: inspect the run in the grid view, note which downstream instances were cleared or set to skipped or none, then clear exactly the tasks that must actually execute, with their upstream and downstream scope chosen deliberately this time.