agent's airflow dag generation failing lint: validate before deploy

TL;DR

The agent writes DAG files straight into the DAG folder, and broken ones take down the scheduler's parsing. Validate before deploy: syntax-check the generated file, run Airflow's DAG test on it, and lint it. Only copy it into the DAG folder after all three pass.

Symptom

An agent generating Airflow DAGs produces files that fail lint, fail to import, or break the scheduler's DAG parsing. Each bad deploy pollutes the DAG folder and the failure surfaces far from the generation step that caused it.

When to use

• Agent-written DAGs break scheduler parsing
• Generated DAG Python has syntax or import errors
• DAGs import fine standalone but fail inside Airflow

When not to use

• A valid DAG's tasks fail at runtime (thats the task logic, not generation)
• The scheduler is slow but DAGs parse (thats performance)
• A human wrote the DAG by hand (review it normally, this is an agent guardrail)

Steps

1. Syntax-check the generated file before anything else:

python -m py_compile /tmp/staging/my_dag.py && echo OK

Expected output: OK with no traceback. This catches the trivial breakage (bad indentation, unclosed brackets) in milliseconds.

2. Test the DAG the way Airflow will actually load it:

airflow dags test my_dag 2026-01-01 --subdir /tmp/staging

Expected output: the DAG parses and its tasks execute in test mode. This catches Airflow-specific problems: bad default_args, unknown operators, top-level code that should be inside tasks.

3. Lint the generated code for the mistakes agents repeat:

ruff check /tmp/staging/my_dag.py

Expected output: no findings, or a short list the agent can fix. Agents commonly generate unused imports, undefined names from hallucinated helpers, and top-level DB calls.

4. Only then promote the file into the real DAG folder, and keep generation staging separate from the live folder:

cp /tmp/staging/my_dag.py /opt/airflow/dags/

Expected output: the scheduler picks up a known-good DAG. The live folder never sees an unvalidated file, so one bad generation cannot break parsing for every other DAG.

Variant phrasings

generated dag fails with broken dag import error

The file has a top-level exception. The py_compile plus dags test pair catches these before deploy.

agent dag works once then breaks on regeneration

Regeneration overwrote a hand-fixed file. Keep generated DAGs clearly marked and re-validate on every regeneration, not just the first.

Why it happens

Agents generate code probabilistically, so the output is sometimes subtly broken. Deploying straight to the DAG folder skips the feedback loop that human authors get from their editor and CI. Airflow then parses the broken file on its schedule scan, and the error shows up in the scheduler logs instead of at generation time.

Edge cases

• airflow dags test needs the same Airflow version and provider packages as production. Testing against a different version validates nothing.
• DAGs that read Variables or Connections at import time fail the test outside production. Generate DAGs that read those inside task callables, not at module top level.
• Very large generated DAGs can exceed the scheduler's parse timeout even when valid. Keep generated DAGs small or raise the parse timeout deliberately.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstBicWN0st1ttTINEmlLS6g