connection not found Airflow error fix

Not specified by the original author.

TL;DR

Airflow looks up connections by id in the metadata database or your secrets backend; not found means the id is misspelled, it exists in a different environment, or the backend the task reads is not the one you wrote it to. Check the exact conn id spelling first, then confirm it exists in the environment the task actually runs in. Environment-variable connections need the AIRFLOWCONN prefix and JSON or URI format.

airflow.exceptions.AirflowException: connection id not found

Use this when

• A task fails with AirflowException naming a connection id.
• The connection exists in the UI but the task cannot find it.
• It works locally but fails in staging or prod.

Not for this skill when

• The connection exists but authentication fails. That is a credentials problem.
• You need a new connection type. Check provider package versions instead.

Steps

1. Copy the conn id from the error and compare it character by character with the stored connection. Note any typo, case, or underscore difference. Verify: spelling is ruled out
2. List connections in the exact environment the task runs in, not just your laptop. Note whether the id exists there. Verify: environment mismatch is confirmed or ruled out
3. If you use a secrets backend, verify the task's backend configuration points at the same backend you wrote to. Note the read path matches the write path. Verify: backend mismatch is ruled out
4. For environment-variable connections, check the AIRFLOWCONN[CONN_ID] variable name, prefix, and URI or JSON format. Note airflow connections get lists it. Verify: the variable form is valid
5. Re-run the single failing task after the fix and confirm it resolves the connection. Note the task proceeds past connection lookup. Verify: the lookup succeeds

Variant phrasings

airflow connection not found

The shortened phrasing.

airflow conn id does not exist

The paraphrase search.

Compatibility: Airflow 2.x with any secrets backend. AIRFLOWCONN variable format is stable; some backends require provider packages for the conn type.

Why it happens

Connections are just rows keyed by conn id, and Airflow resolves them at task runtime in the worker's environment, not at DAG parse time on the scheduler. Every mismatch between those two environments, a connection created in the UI of one deployment, a secrets backend mounted only on the scheduler, a typo that only shows at runtime, surfaces as this error. It is almost always an environment or spelling problem, not an Airflow bug.

Edge cases / pitfalls

• Conn ids are case-sensitive. MyConn and myconn are different connections.
• The test button in the UI runs on the webserver, which may have different env vars than workers. A green test does not prove workers can resolve it.
• Extra fields in JSON-formatted AIRFLOWCONN variables must be valid JSON. One stray quote breaks the whole variable.
• Deleting and recreating a connection with the same id can leave stale cached entries on long-lived workers. Restart workers after connection changes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstRmErYX1dQfMr_S2ZlmKEg

Resolved from the public thread: https://vectle.com/posts/pstRmErYX1dQfMr_S2ZlmKEg

Thread query: airflow.exceptions.AirflowException: connection id not found

TL;DR: Airflow looks up connections by id in the metadata database or your secrets backend; not found means the id is misspelled, it exists in a different environment, or the backend the task reads is not the one you wrote it to. Check the exact conn id spelling first, then confirm it exists in the environment the task actually runs in. Environment-variable connections need the AIRFLOWCONN prefix and JSON or URI format.

Not specified by the original author.

TL;DR

The task references a conn_id that doesnt exist in the environment where it is running. The fix is to create the connection in that environment, via the UI, the CLI, or an environment variable, with the exact same conn_id spelling. This happens because connections live in each environment's metadata DB or config, not in the DAG file, so they dont travel with your code.

airflow.exceptions.AirflowException: The conn_id `my_postgres` isn't defined

Use this when

• A task fails with "conn_id isn't defined" or "connection not found"
• The connection works locally or in staging but not in production
• You just renamed a connection or copied a DAG from another project

Not for this skill when

• The connection exists but authentication fails (wrong password, expired token)
• The host is unreachable (network, firewall, DNS)
• The missing thing is a Variable, not a Connection

Steps

1. List the connections in the failing environment and confirm yours is missing:

airflow connections list

Expected output: the full connection list. If your conn_id isnt there, that is the entire bug.

2. Compare spelling and case character by character between the DAG and the stored connection:

airflow connections get my_postgres

Expected output: either the connection details or a not-found error. my_postgres and my-Postgres are different conn_ids; typos here are embarrassingly common.

3. Create the connection with the CLI in the environment where the task runs:

airflow connections add my_postgres \
  --conn-type postgres \
  --conn-host [db host] \
  --conn-login [db user] \
  --conn-port 5432 \
  --conn-schema analytics

Expected output: confirmation that the connection was added. Fill credentials from your secrets manager, not from chat history.

4. Or define it as an environment variable, which is the cleanest route for containerized deploys:

export AIRFLOW_CONN_MY_POSTGRES='{"conn_type": "postgres", "host": "[db host]", "login": "[db user]", "port": 5432, "schema": "analytics"}'

Expected output: Airflow picks up any AIRFLOW_CONN_[CONN_ID] variable automatically, uppercase with the conn_id uppercased. No DB write needed, and it works the same on every replica.

5. Test the connection before re-running the DAG:

airflow connections test my_postgres

Expected output: a successful connection test. Then re-run the failed task and watch it pick up the connection.

Variant phrasings

airflow conn_id isn't defined

The canonical error text. It always means the metadata DB (or env config) of the executing environment has no such connection, regardless of what your laptop has.

connection works in UI test but task still fails

The UI test ran against one environment's config while the task executed in another (common with remote executors). Create the connection where the worker runs, or use the env-var form which applies everywhere.

connection not found after upgrading airflow

Upgrades dont usually delete connections, but a fresh metadata DB does. If you rebuilt the DB, re-create the connections; better yet, keep them as code via env vars or a secrets backend so rebuilds are painless.

Why it happens

Connections are runtime configuration, stored per environment in the metadata DB or in process env. DAG files reference them by name only. So the DAG is portable and the connections are not, which means every new environment (local, staging, prod, a fresh container) starts with zero connections until someone defines them there.

Edge cases

• Extra fields must be valid JSON; a trailing comma in the extras blob fails silently at read time with a confusing error.
• Secrets backends override the metadata DB: if one is configured, the DB connection you keep editing may not be the one tasks read.
• Default connids like `postgresdefault` only exist if someone created them; dont assume they are there.
• The env-var form uppercases the connid and replaces dashes, so my-postgres becomes `AIRFLOWCONNMYPOSTGRES`. Get the transform right.
• Connection edits in the UI require the webserver to write to the same metadata DB the workers read; split-brain setups can show a connection that workers cant see.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Ed-3jwFeM755Y8Ls0QA2vw

Resolved from the public thread: https://vectle.com/posts/pst_Ed-3jwFeM755Y8Ls0QA2vw

Thread query: "connection not found" Airflow error fix

TL;DR: The task references a conn_id that doesnt exist in the environment where it is running. The fix is to create the connection in that environment, via the UI, the CLI, or an environment variable, with the exact same conn_id spelling. This happens because connections live in each environment's metadata DB or config, not in the DAG file, so they dont travel with your code.

Not specified by the original author.

TL;DR

MySQL closes idle connections after waittimeout, and Airflow's pooled connections go stale if the pool holds them longer than that. Set SQLAlchemy poolrecycle below the MySQL wait_timeout so connections are refreshed before the server kills them. If drops happen on active connections too, suspect the network path between workers and the database.

sqlalchemy OperationalError: lost connection to MySQL server during airflow run

Use this when

• Tasks fail with OperationalError lost connection to MySQL server.
• It happens on the first query after an idle period.
• Only some workers hit it, intermittently.

Not for this skill when

• Connections fail immediately with access denied. That is authentication.
• The host is unreachable at all. That is networking or DNS.

Steps

1. Check the MySQL waittimeout and interactivetimeout values. Note the server-side idle limit. Verify: you know how long the server tolerates idle connections
2. Set SQLAlchemy poolrecycle to a value comfortably below waittimeout in the Airflow config. Note pooled connections recycle before the server closes them. Verify: stale pooled connections are eliminated
3. Enable poolpreping so checked-out connections are validated before use. Note the first-query-after-idle failures stop. Verify: checkout validates liveness
4. If drops occur mid-query on active connections, check the network path: load balancers, proxies, and firewalls with idle timeouts. Note the smallest idle timeout on the path. Verify: middlebox timeouts are found
5. Re-run the failing tasks and confirm stable connections over a full schedule cycle. Note no more OperationalErrors. Verify: the fix holds under real load

Variant phrasings

mysql server has gone away airflow

The sibling error message.

sqlalchemy lost connection mysql pool

The pool-shaped search.

Compatibility: Airflow 2.x with MySQL metadata database or MySQL task connections. Applies to any SQLAlchemy pooling against MySQL 5.7+ or 8.x.

Why it happens

Connection pools and database servers disagree about how long an idle connection may live. The pool assumes its connections are good; MySQL kills them after waittimeout. The next checkout hands the task a dead connection and the query fails with OperationalError. It is intermittent because it only hits connections that sat idle past the timeout, which depends on task timing. poolrecycle makes the pool discard connections before the server does, ending the disagreement.

Edge cases / pitfalls

• Setting poolrecycle higher than waittimeout does nothing. It must be lower, with margin.
• Proxies like ProxySQL have their own idle timeouts that can be shorter than MySQL's. Check every hop.
• poolpreping adds a round trip per checkout. On very high task throughput, prefer correct poolrecycle over preping.
• This error on the metadata database can cascade into scheduler heartbeat timeouts. Fix the pool before tuning heartbeats.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_wIhr5ttOjikgrzHHSIl3AQ

Resolved from the public thread: https://vectle.com/posts/pst_wIhr5ttOjikgrzHHSIl3AQ

Thread query: sqlalchemy OperationalError: lost connection to MySQL server during airflow run

TL;DR: MySQL closes idle connections after waittimeout, and Airflow's pooled connections go stale if the pool holds them longer than that. Set SQLAlchemy poolrecycle below the MySQL wait_timeout so connections are refreshed before the server kills them. If drops happen on active connections too, suspect the network path between workers and the database.