airflow.exceptions.AirflowDagDuplicatedIdException: duplicate dag id

TL;DR

Two DAG objects with the same dag_id reached the DagBag in one parsing pass. The usual causes are a copied DAG file left in the dags folder, a backup file the parser still scans, or a loop that builds DAGs from a list containing a duplicate entry. Find every place the id is defined, keep one, and confirm with airflow dags list that the id appears once.

airflow.exceptions.AirflowDagDuplicatedIdException: duplicate dag id

Use this when

• A DAG fails to load and the import error names AirflowDagDuplicatedIdException.
• You copied or renamed a DAG file and the original is still in the folder.
• Dynamically generated DAGs collide after a config list changed.

Not for this skill when

• Two different environments show the same dag_id. Duplication only matters inside one Airflow instance.
• Task ids repeat inside one DAG. That is a different exception and a different fix.

Steps

1. Search the whole dags folder for the dag_id string, including subfolders and files your editor hides. Note every file that defines it. Verify: all definitions of the id are listed
2. Check for backup or generated copies such as a file ending in a backup suffix, a copy made by a sync tool, or a rendered file next to its template. Note which files the parser actually scans. Verify: you know which copies Airflow reads, not just which exist
3. If DAGs are generated in a loop, print or inspect the source list for duplicate entries and dedupe it at the source, for example a config table with the same name twice. Note the duplicate source row. Verify: generated ids are unique before the loop runs
4. Keep one definition: delete or rename the stray file, or give the genuinely different DAG a distinct dag_id. Note the surviving file. Verify: exactly one object carries the id
5. Run airflow dags list-import-errors and airflow dags list, and confirm the exception is gone and the id appears once. Note the clean parse. Verify: the DagBag loads with no duplicate id error

Variant phrasings

airflow duplicate dag id exception fix

The exception name searched in plain words.

airflow dag id already exists in dagbag

The older wording of the same collision.

Compatibility: Airflow 2.x and 3.x, all executors. The exception is raised by the DagBag during parsing, so it reproduces with the CLI as well as in the scheduler.

Why it happens

The dag_id is the primary key Airflow uses everywhere: the metadata tables, the UI, the API, and run history. Two live objects claiming one id would make every one of those ambiguous, so the parser refuses both rather than picking a winner. Because parsing walks the whole folder tree, anything file-like left in that tree counts, which is why editor backups, git conflict copies, and code generator output are the classic triggers.

Edge cases / pitfalls

• Renaming the file does not help. The collision is on the dag_id inside the file, not the filename.
• A renamed DAG keeps its old run history under the old id. Expect a fresh history for the new id and plan backfills accordingly.
• Dynamic DAG generators that read a database at parse time can collide only when the source data has a duplicate. Fix the source, not just the symptom.
• Git sync sidecars can resurrect a deleted duplicate on the next sync. Remove it from the repo, not only from the running pod.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_iKojOsnlmMgCNNkN0YxcEw

Resolved from the public thread: https://vectle.com/posts/pst_iKojOsnlmMgCNNkN0YxcEw

Thread query: airflow.exceptions.AirflowDagDuplicatedIdException: duplicate dag id

TL;DR: Two DAG objects with the same dag_id reached the DagBag in one parsing pass. The usual causes are a copied DAG file left in the dags folder, a backup file the parser still scans, or a loop that builds DAGs from a list containing a duplicate entry. Find every place the id is defined, keep one, and confirm with airflow dags list that the id appears once.