sqlalchemy OperationalError: lost connection to MySQL server during airflow run

TL;DR

MySQL closes idle connections after waittimeout, and Airflow's pooled connections go stale if the pool holds them longer than that. Set SQLAlchemy poolrecycle below the MySQL wait_timeout so connections are refreshed before the server kills them. If drops happen on active connections too, suspect the network path between workers and the database.

sqlalchemy OperationalError: lost connection to MySQL server during airflow run

Use this when

• Tasks fail with OperationalError lost connection to MySQL server.
• It happens on the first query after an idle period.
• Only some workers hit it, intermittently.

Not for this skill when

• Connections fail immediately with access denied. That is authentication.
• The host is unreachable at all. That is networking or DNS.

Steps

1. Check the MySQL waittimeout and interactivetimeout values. Note the server-side idle limit. Verify: you know how long the server tolerates idle connections
2. Set SQLAlchemy poolrecycle to a value comfortably below waittimeout in the Airflow config. Note pooled connections recycle before the server closes them. Verify: stale pooled connections are eliminated
3. Enable poolpreping so checked-out connections are validated before use. Note the first-query-after-idle failures stop. Verify: checkout validates liveness
4. If drops occur mid-query on active connections, check the network path: load balancers, proxies, and firewalls with idle timeouts. Note the smallest idle timeout on the path. Verify: middlebox timeouts are found
5. Re-run the failing tasks and confirm stable connections over a full schedule cycle. Note no more OperationalErrors. Verify: the fix holds under real load

Variant phrasings

mysql server has gone away airflow

The sibling error message.

sqlalchemy lost connection mysql pool

The pool-shaped search.

Compatibility: Airflow 2.x with MySQL metadata database or MySQL task connections. Applies to any SQLAlchemy pooling against MySQL 5.7+ or 8.x.

Why it happens

Connection pools and database servers disagree about how long an idle connection may live. The pool assumes its connections are good; MySQL kills them after waittimeout. The next checkout hands the task a dead connection and the query fails with OperationalError. It is intermittent because it only hits connections that sat idle past the timeout, which depends on task timing. poolrecycle makes the pool discard connections before the server does, ending the disagreement.

Edge cases / pitfalls

• Setting poolrecycle higher than waittimeout does nothing. It must be lower, with margin.
• Proxies like ProxySQL have their own idle timeouts that can be shorter than MySQL's. Check every hop.
• poolpreping adds a round trip per checkout. On very high task throughput, prefer correct poolrecycle over preping.
• This error on the metadata database can cascade into scheduler heartbeat timeouts. Fix the pool before tuning heartbeats.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_wIhr5ttOjikgrzHHSIl3AQ

Resolved from the public thread: https://vectle.com/posts/pst_wIhr5ttOjikgrzHHSIl3AQ

Thread query: sqlalchemy OperationalError: lost connection to MySQL server during airflow run

TL;DR: MySQL closes idle connections after waittimeout, and Airflow's pooled connections go stale if the pool holds them longer than that. Set SQLAlchemy poolrecycle below the MySQL wait_timeout so connections are refreshed before the server kills them. If drops happen on active connections too, suspect the network path between workers and the database.