connection not found" Airflow error fix

TL;DR

The task references a conn_id that doesnt exist in the environment where it is running. The fix is to create the connection in that environment, via the UI, the CLI, or an environment variable, with the exact same conn_id spelling. This happens because connections live in each environment's metadata DB or config, not in the DAG file, so they dont travel with your code.

airflow.exceptions.AirflowException: The conn_id `my_postgres` isn't defined

Use this when

• A task fails with "conn_id isn't defined" or "connection not found"
• The connection works locally or in staging but not in production
• You just renamed a connection or copied a DAG from another project

Not for this skill when

• The connection exists but authentication fails (wrong password, expired token)
• The host is unreachable (network, firewall, DNS)
• The missing thing is a Variable, not a Connection

Steps

1. List the connections in the failing environment and confirm yours is missing:

airflow connections list

Expected output: the full connection list. If your conn_id isnt there, that is the entire bug.

1. Compare spelling and case character by character between the DAG and the stored connection:

airflow connections get my_postgres

Expected output: either the connection details or a not-found error. my_postgres and my-Postgres are different conn_ids; typos here are embarrassingly common.

1. Create the connection with the CLI in the environment where the task runs:

airflow connections add my_postgres \
  --conn-type postgres \
  --conn-host [db host] \
  --conn-login [db user] \
  --conn-port 5432 \
  --conn-schema analytics

Expected output: confirmation that the connection was added. Fill credentials from your secrets manager, not from chat history.

1. Or define it as an environment variable, which is the cleanest route for containerized deploys:

export AIRFLOW_CONN_MY_POSTGRES='{"conn_type": "postgres", "host": "[db host]", "login": "[db user]", "port": 5432, "schema": "analytics"}'

Expected output: Airflow picks up any AIRFLOW_CONN_[CONN_ID] variable automatically, uppercase with the conn_id uppercased. No DB write needed, and it works the same on every replica.

1. Test the connection before re-running the DAG:

airflow connections test my_postgres

Expected output: a successful connection test. Then re-run the failed task and watch it pick up the connection.

Variant phrasings

airflow conn_id isn't defined

The canonical error text. It always means the metadata DB (or env config) of the executing environment has no such connection, regardless of what your laptop has.

connection works in UI test but task still fails

The UI test ran against one environment's config while the task executed in another (common with remote executors). Create the connection where the worker runs, or use the env-var form which applies everywhere.

connection not found after upgrading airflow

Upgrades dont usually delete connections, but a fresh metadata DB does. If you rebuilt the DB, re-create the connections; better yet, keep them as code via env vars or a secrets backend so rebuilds are painless.

Why it happens

Connections are runtime configuration, stored per environment in the metadata DB or in process env. DAG files reference them by name only. So the DAG is portable and the connections are not, which means every new environment (local, staging, prod, a fresh container) starts with zero connections until someone defines them there.

Edge cases

• Extra fields must be valid JSON; a trailing comma in the extras blob fails silently at read time with a confusing error.
• Secrets backends override the metadata DB: if one is configured, the DB connection you keep editing may not be the one tasks read.
• Default connids like `postgresdefault` only exist if someone created them; dont assume they are there.
• The env-var form uppercases the connid and replaces dashes, so my-postgres becomes `AIRFLOWCONNMYPOSTGRES`. Get the transform right.
• Connection edits in the UI require the webserver to write to the same metadata DB the workers read; split-brain setups can show a connection that workers cant see.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Ed-3jwFeM755Y8Ls0QA2vw