task instance not found" Airflow error fix

TL;DR

Airflow cannot find the task instance you asked about, which usually means the task id, dag id, or execution date you typed does not match anything in the metadata database. Copy the identifiers from Airflow's own listings instead of typing them from memory, and use the logical date format Airflow expects. The identifiers are the whole story here.

TaskInstance not found for task_id=my_task, dag_id=my_dag, execution_date=2024-01-15

Use this when

• the CLI says task instance not found for a task you can see in the UI
• a manual trigger or test run fails to locate the task
• the UI shows the task but commands cannot address it

Not for this skill when

• the task exists but fails, that is a task failure not a lookup failure
• the whole DAG is missing, check import errors first
• the metadata database is unreachable, fix the DB connection

Steps

1. List the tasks Airflow knows about in that DAG and copy the exact id:

airflow tasks list my_dag

Expected output: the real task ids. Copy from here, never retype from memory, since one wrong character is enough to fail the lookup.

1. List the DAG runs to get the exact logical dates Airflow expects:

airflow dags list-runs --dag-id my_dag | head -10

Expected output: run ids with their logical dates. The date format shown here is what every command expects, so use it verbatim.

1. Retry your command with the copied identifiers and the exact date:

airflow tasks test my_dag my_task 2024-01-15

Expected output: the task test actually runs. If it still fails, the date truly has no run, which step 4 confirms or denies.

1. Check whether any run exists for that date at all in the metadata database:

SELECT dag_id, run_id, logical_date, state
FROM dag_run
WHERE dag_id = 'my_dag'
ORDER BY logical_date DESC LIMIT 10;

Expected output: the recent runs in the metadata DB. No row for your date means there is nothing to address, so trigger a run first and then come back.

1. If the run exists but the task row is missing, clear the task so the scheduler rebuilds it:

airflow tasks clear my_dag --task-regex my_task --start-date 2024-01-15 --end-date 2024-01-15 --yes

Expected output: the task instance is recreated on the next scheduler loop. Stale metadata heals itself once the scheduler reprocesses the DAG.

Variant phrasings

airflow task instance not found when triggering

The trigger created no run, so there is no instance to find. Check the DAG run list from step 2 before trying to address individual tasks.

airflow tasks test says not found

tasks test needs an exact logical date, not a guess. Copy it from list-runs output and do not improvise the format.

taskinstance not found in airflow ui logs

The UI linked to a run that was later deleted or never created. Re-derive the identifiers from the current run list instead of trusting the old link.

Why it happens

Every task instance is keyed by the triple of dag id, task id, and logical date in the metadata database. Commands address instances by that triple, and any mismatch, a typo, a wrong date format, or a run that was never created, finds nothing. The error is a lookup miss, not a broken task, which is why re-deriving the identifiers fixes it.

Edge cases

• Deleted DAG runs leave orphaned log links behind. The logs exist on disk, the instance row does not.
• Timezone-naive dates get interpreted as UTC and miss runs created in another timezone. Be explicit about timezones.
• Mapped tasks have indexes, so address them with the map index or the lookup misses the specific instance.
• A corrupted metadata DB can lose rows genuinely. That is rare, restore from backup rather than hand-editing rows.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstIusAkXL9vK4Lpvcf_Vg