airflow dagbag import timeout on startup

TL;DR

Dagbag import timeouts mean your DAG files do too much work at import time: API calls, heavy imports, or huge generated DAG counts. Profile which files take longest, move the heavy work out of module scope, and only then raise the parse timeout. Raising the timeout on slow files just makes every restart slower.

airflow dagbag import timeout on startup

Use this when

• Scheduler or webserver logs show dagbag import timed out on startup.
• Airflow restarts take tens of minutes before DAGs appear.
• Adding one more DAG file pushes parsing over the timeout.

Not for this skill when

• Tasks run slowly. Parse time and run time are different problems.
• A single DAG file fails to import. That is an import error, not a timeout.

Steps

1. Time the parse per file with airflow dags list-import-errors or a manual timing loop. Note the two or three slowest files. Verify: you know exactly which files are slow
2. Inspect the slow files for top-level API calls, database queries, or heavy library imports. Note the specific lines doing the work. Verify: the parse-time work is identified
3. Move that work into task callables or cache it so imports stay light. Note the slow files now parse in seconds. Verify: parsing is fast in isolation
4. If parsing is legitimately heavy (hundreds of generated DAGs), raise dagbagimporttimeout to fit the measured time plus headroom. Note startup completes without timeouts. Verify: the timeout matches measured reality
5. Add a parse-time budget to CI so new DAG files cannot regress startup. Note the next slow file is caught pre-deploy. Verify: startup stays fast

Variant phrasings

airflow dagbag timeout

The shortened search.

airflow slow dag parsing startup

The symptom phrasing.

Compatibility: Airflow 2.x. dagbagimporttimeout lives in airflow.cfg under [core]; per-file timing via the CLI works on all 2.x.

Why it happens

Airflow imports every DAG file on startup to build the dagbag, and it does that work serially per file with a global timeout. Anything at module scope, a client constructor that phones home, a query that lists tables, runs during that import. A handful of such files is enough to blow the timeout on large deployments. The timeout exists to catch hangs, but slow-by-design files trip it too.

Edge cases / pitfalls

• Dynamic DAG generation that loops over thousands of configs multiplies parse time linearly. Generate lazily or split across files.
• Importing pandas or similar heavy libraries at module scope in every file adds seconds per file. Import inside the task function instead.
• The webserver parses too, so a timeout there shows as a slow UI even when the scheduler is fine. Check both logs.
• Partial parsing caches help, but a changed file invalidates the cache. Do not rely on the cache to hide slow files.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_FQvaN7BNK7ZmedgIEaI5Aw