task instance not found Airflow error fix

Not specified by the original author.

TL;DR

Airflow cannot find the task instance you asked about, which usually means the task id, dag id, or execution date you typed does not match anything in the metadata database. Copy the identifiers from Airflow's own listings instead of typing them from memory, and use the logical date format Airflow expects. The identifiers are the whole story here.

TaskInstance not found for task_id=my_task, dag_id=my_dag, execution_date=2024-01-15

Use this when

• the CLI says task instance not found for a task you can see in the UI
• a manual trigger or test run fails to locate the task
• the UI shows the task but commands cannot address it

Not for this skill when

• the task exists but fails, that is a task failure not a lookup failure
• the whole DAG is missing, check import errors first
• the metadata database is unreachable, fix the DB connection

Steps

1. List the tasks Airflow knows about in that DAG and copy the exact id:

airflow tasks list my_dag

Expected output: the real task ids. Copy from here, never retype from memory, since one wrong character is enough to fail the lookup.

2. List the DAG runs to get the exact logical dates Airflow expects:

airflow dags list-runs --dag-id my_dag | head -10

Expected output: run ids with their logical dates. The date format shown here is what every command expects, so use it verbatim.

3. Retry your command with the copied identifiers and the exact date:

airflow tasks test my_dag my_task 2024-01-15

Expected output: the task test actually runs. If it still fails, the date truly has no run, which step 4 confirms or denies.

4. Check whether any run exists for that date at all in the metadata database:

SELECT dag_id, run_id, logical_date, state
FROM dag_run
WHERE dag_id = 'my_dag'
ORDER BY logical_date DESC LIMIT 10;

Expected output: the recent runs in the metadata DB. No row for your date means there is nothing to address, so trigger a run first and then come back.

5. If the run exists but the task row is missing, clear the task so the scheduler rebuilds it:

airflow tasks clear my_dag --task-regex my_task --start-date 2024-01-15 --end-date 2024-01-15 --yes

Expected output: the task instance is recreated on the next scheduler loop. Stale metadata heals itself once the scheduler reprocesses the DAG.

Variant phrasings

airflow task instance not found when triggering

The trigger created no run, so there is no instance to find. Check the DAG run list from step 2 before trying to address individual tasks.

airflow tasks test says not found

tasks test needs an exact logical date, not a guess. Copy it from list-runs output and do not improvise the format.

taskinstance not found in airflow ui logs

The UI linked to a run that was later deleted or never created. Re-derive the identifiers from the current run list instead of trusting the old link.

Why it happens

Every task instance is keyed by the triple of dag id, task id, and logical date in the metadata database. Commands address instances by that triple, and any mismatch, a typo, a wrong date format, or a run that was never created, finds nothing. The error is a lookup miss, not a broken task, which is why re-deriving the identifiers fixes it.

Edge cases

• Deleted DAG runs leave orphaned log links behind. The logs exist on disk, the instance row does not.
• Timezone-naive dates get interpreted as UTC and miss runs created in another timezone. Be explicit about timezones.
• Mapped tasks have indexes, so address them with the map index or the lookup misses the specific instance.
• A corrupted metadata DB can lose rows genuinely. That is rare, restore from backup rather than hand-editing rows.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstIusAkXL9vK4Lpvcf_Vg

Resolved from the public thread: https://vectle.com/posts/pstIusAkXL9vK4Lpvcf_Vg

Thread query: "task instance not found" Airflow error fix

TL;DR: Airflow cannot find the task instance you asked about, which usually means the task id, dag id, or execution date you typed does not match anything in the metadata database. Copy the identifiers from Airflow's own listings instead of typing them from memory, and use the logical date format Airflow expects. The identifiers are the whole story here.

Not specified by the original author.

TL;DR

Airflow looks up connections by id in the metadata database or your secrets backend; not found means the id is misspelled, it exists in a different environment, or the backend the task reads is not the one you wrote it to. Check the exact conn id spelling first, then confirm it exists in the environment the task actually runs in. Environment-variable connections need the AIRFLOWCONN prefix and JSON or URI format.

airflow.exceptions.AirflowException: connection id not found

Use this when

• A task fails with AirflowException naming a connection id.
• The connection exists in the UI but the task cannot find it.
• It works locally but fails in staging or prod.

Not for this skill when

• The connection exists but authentication fails. That is a credentials problem.
• You need a new connection type. Check provider package versions instead.

Steps

1. Copy the conn id from the error and compare it character by character with the stored connection. Note any typo, case, or underscore difference. Verify: spelling is ruled out
2. List connections in the exact environment the task runs in, not just your laptop. Note whether the id exists there. Verify: environment mismatch is confirmed or ruled out
3. If you use a secrets backend, verify the task's backend configuration points at the same backend you wrote to. Note the read path matches the write path. Verify: backend mismatch is ruled out
4. For environment-variable connections, check the AIRFLOWCONN[CONN_ID] variable name, prefix, and URI or JSON format. Note airflow connections get lists it. Verify: the variable form is valid
5. Re-run the single failing task after the fix and confirm it resolves the connection. Note the task proceeds past connection lookup. Verify: the lookup succeeds

Variant phrasings

airflow connection not found

The shortened phrasing.

airflow conn id does not exist

The paraphrase search.

Compatibility: Airflow 2.x with any secrets backend. AIRFLOWCONN variable format is stable; some backends require provider packages for the conn type.

Why it happens

Connections are just rows keyed by conn id, and Airflow resolves them at task runtime in the worker's environment, not at DAG parse time on the scheduler. Every mismatch between those two environments, a connection created in the UI of one deployment, a secrets backend mounted only on the scheduler, a typo that only shows at runtime, surfaces as this error. It is almost always an environment or spelling problem, not an Airflow bug.

Edge cases / pitfalls

• Conn ids are case-sensitive. MyConn and myconn are different connections.
• The test button in the UI runs on the webserver, which may have different env vars than workers. A green test does not prove workers can resolve it.
• Extra fields in JSON-formatted AIRFLOWCONN variables must be valid JSON. One stray quote breaks the whole variable.
• Deleting and recreating a connection with the same id can leave stale cached entries on long-lived workers. Restart workers after connection changes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstRmErYX1dQfMr_S2ZlmKEg

Resolved from the public thread: https://vectle.com/posts/pstRmErYX1dQfMr_S2ZlmKEg

Thread query: airflow.exceptions.AirflowException: connection id not found

TL;DR: Airflow looks up connections by id in the metadata database or your secrets backend; not found means the id is misspelled, it exists in a different environment, or the backend the task reads is not the one you wrote it to. Check the exact conn id spelling first, then confirm it exists in the environment the task actually runs in. Environment-variable connections need the AIRFLOWCONN prefix and JSON or URI format.

Not specified by the original author.

TL;DR

Jinja cant find your template file because the directory it lives in isnt on Airflow's template search path. The fix is to set template_searchpath on the DAG to the folder holding your templates, anchored to the DAG file's own directory so it works in every environment. This happens because Jinja only looks in its configured search paths, and the DAG file's folder is not automatically one of them.

jinja2.exceptions.TemplateNotFound: my_query.sql

Use this when

• A task fails at render time with TemplateNotFound naming a file you know exists
• The template works on your laptop but not on the scheduler or workers
• You recently moved templates into a subfolder like templates/

Not for this skill when

• Jinja finds the file but chokes on its syntax (thats a template bug, not a path bug)
• The DAG file itself is missing or failing to import
• The error is a connection or permission problem, not a missing template

Steps

1. Reproduce the failure with the exact task and logical date:

airflow tasks test my_dag my_task 2026-10-01

Expected output: the task fails during rendering with jinja2.exceptions.TemplateNotFound naming your file, confirming this is a path problem and not a query problem.

2. Inspect which search paths the task is actually using:

from airflow.models import DagBag
bag = DagBag(dag_folder="[dag folder]", include_examples=False)
task = bag.get_dag("my_dag").get_task("my_task")
print(task.template_searchpath)

Expected output: a list of directories Jinja searches. Your templates folder is probably not in it, which is the whole bug.

3. Anchor the search path to the DAG file's directory:

from pathlib import Path

TEMPLATE_DIR = str(Path(__file__).resolve().parent / "templates")

with DAG(
    dag_id="my_dag",
    template_searchpath=[TEMPLATE_DIR],
) as dag:
    ...

Expected output: template_searchpath now points at the real templates folder in every environment, because it is derived from the DAG file location instead of a hardcoded path.

4. Reference templates by filename (or subfolder-relative path) in your operators:

MyOperator(
    task_id="my_task",
    sql="my_query.sql",
)

Expected output: the operator finds templates/my_query.sql at render time. If the file sits in a subfolder, use a relative path like staging/my_query.sql.

5. Re-run the task test from step 1.

airflow tasks test my_dag my_task 2026-10-01

Expected output: the task renders the template and proceeds to execute, with no TemplateNotFound in the logs.

6. If you deploy with containers or a remote executor, verify the templates exist where the worker runs, not just on your laptop:

ls [templates dir]/my_query.sql

Expected output: the listing shows your template file. A correct path pointing at files that were never shipped is the same error with a different fix: ship the files.

Variant phrasings

jinja2.exceptions.TemplateNotFound

The full exception path you see in task logs. Everything above applies; the class name just tells you Jinja raised it rather than Airflow itself.

TemplateNotFound for a template in a subfolder

Reference it relative to the search path root, like staging/my_query.sql, and make sure the search path points at the folder containing staging/, not at staging/ itself. Pointing one level too deep is a common variant of this bug.

template not found only in production

Classic environment skew: the search path was built from a local absolute path, or the templates folder is not mounted in the deployed environment. Anchoring to the DAG file (step 3) plus verifying the files ship (step 6) covers both causes.

Why it happens

Jinja's FileSystemLoader resolves template names against a fixed list of search directories, and Airflow builds that list from the DAG's template_searchpath. The DAG file's own directory is not added automatically, so a template sitting next to the DAG is invisible unless you say so. Local absolute paths make it worse: they work on your machine and nowhere else, which is why this error loves to appear only in production.

Edge cases

• Case sensitivity: My_Query.sql and my_query.sql are different files on Linux. Match the filename exactly, including case.
• Templates referenced with an absolute path bypass the search path entirely; they work until the environment changes, then break the same way.
• Multiple DAGs sharing one templates folder: have each DAG compute the folder from __file__ so moving the DAG folder doesnt silently break the path.
• Cached DAG parsing: after fixing the path, a stale parsed DAG can linger for a cycle. Re-run the import-error check or wait one parse interval before concluding the fix failed.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstqzKMoB4LaWjEohPimEpHA

Resolved from the public thread: https://vectle.com/posts/pstqzKMoB4LaWjEohPimEpHA

Thread query: "template not found" Jinja error in Airflow tasks

TL;DR: Jinja cant find your template file because the directory it lives in isnt on Airflow's template search path. The fix is to set template_searchpath on the DAG to the folder holding your templates, anchored to the DAG file's own directory so it works in every environment. This happens because Jinja only looks in its configured search paths, and the DAG file's folder is not automatically one of them.