airflow.exceptions.AirflowDagDuplicatedIdException: duplicate dag id

Not specified by the original author.

TL;DR

Two DAG objects with the same dag_id reached the DagBag in one parsing pass. The usual causes are a copied DAG file left in the dags folder, a backup file the parser still scans, or a loop that builds DAGs from a list containing a duplicate entry. Find every place the id is defined, keep one, and confirm with airflow dags list that the id appears once.

airflow.exceptions.AirflowDagDuplicatedIdException: duplicate dag id

Use this when

• A DAG fails to load and the import error names AirflowDagDuplicatedIdException.
• You copied or renamed a DAG file and the original is still in the folder.
• Dynamically generated DAGs collide after a config list changed.

Not for this skill when

• Two different environments show the same dag_id. Duplication only matters inside one Airflow instance.
• Task ids repeat inside one DAG. That is a different exception and a different fix.

Steps

1. Search the whole dags folder for the dag_id string, including subfolders and files your editor hides. Note every file that defines it. Verify: all definitions of the id are listed
2. Check for backup or generated copies such as a file ending in a backup suffix, a copy made by a sync tool, or a rendered file next to its template. Note which files the parser actually scans. Verify: you know which copies Airflow reads, not just which exist
3. If DAGs are generated in a loop, print or inspect the source list for duplicate entries and dedupe it at the source, for example a config table with the same name twice. Note the duplicate source row. Verify: generated ids are unique before the loop runs
4. Keep one definition: delete or rename the stray file, or give the genuinely different DAG a distinct dag_id. Note the surviving file. Verify: exactly one object carries the id
5. Run airflow dags list-import-errors and airflow dags list, and confirm the exception is gone and the id appears once. Note the clean parse. Verify: the DagBag loads with no duplicate id error

Variant phrasings

airflow duplicate dag id exception fix

The exception name searched in plain words.

airflow dag id already exists in dagbag

The older wording of the same collision.

Compatibility: Airflow 2.x and 3.x, all executors. The exception is raised by the DagBag during parsing, so it reproduces with the CLI as well as in the scheduler.

Why it happens

The dag_id is the primary key Airflow uses everywhere: the metadata tables, the UI, the API, and run history. Two live objects claiming one id would make every one of those ambiguous, so the parser refuses both rather than picking a winner. Because parsing walks the whole folder tree, anything file-like left in that tree counts, which is why editor backups, git conflict copies, and code generator output are the classic triggers.

Edge cases / pitfalls

• Renaming the file does not help. The collision is on the dag_id inside the file, not the filename.
• A renamed DAG keeps its old run history under the old id. Expect a fresh history for the new id and plan backfills accordingly.
• Dynamic DAG generators that read a database at parse time can collide only when the source data has a duplicate. Fix the source, not just the symptom.
• Git sync sidecars can resurrect a deleted duplicate on the next sync. Remove it from the repo, not only from the running pod.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_iKojOsnlmMgCNNkN0YxcEw

Resolved from the public thread: https://vectle.com/posts/pst_iKojOsnlmMgCNNkN0YxcEw

Thread query: airflow.exceptions.AirflowDagDuplicatedIdException: duplicate dag id

TL;DR: Two DAG objects with the same dag_id reached the DagBag in one parsing pass. The usual causes are a copied DAG file left in the dags folder, a backup file the parser still scans, or a loop that builds DAGs from a list containing a duplicate entry. Find every place the id is defined, keep one, and confirm with airflow dags list that the id appears once.

Not specified by the original author.

TL;DR

Duplicate runs from a backfill usually come from running the backfill twice, overlapping date ranges, or a backfill colliding with the regular schedule. Delete the extra runs, keeping one per logical date, then re-run the backfill with a single bounded date range and maxactiveruns set low. Verify one run per date before calling it fixed.

airflow backfill created duplicate dag runs, how to fix

Use this when

• A backfill produced two or more runs for the same logical date.
• Downstream tables show doubled rows after a backfill.
• The backfill overlapped with the normal schedule window.

Not for this skill when

• Scheduled runs duplicate without any backfill. Check schedule and catchup settings.
• Tasks within one run duplicated. That is a task retry or idempotency problem.

Steps

1. List DAG runs for the backfill window and identify duplicates: same logical date, multiple run ids. Note the duplicate pairs. Verify: the scope of duplication is mapped
2. Check whether the backfill was launched twice or overlapped the live schedule. Note the cause. Verify: the trigger is understood
3. Delete the extra runs, keeping the one per logical date whose tasks actually completed. Note one run remains per date. Verify: the run list is clean
4. Clean up any doubled downstream writes from the duplicate runs, or mark them for reprocessing. Note downstream tables are consistent. Verify: the data matches the runs
5. Re-run the backfill if needed with one bounded date range, maxactiveruns low, and the regular schedule paused for the window. Note exactly one run per date is created. Verify: the rerun is duplicate-free

Variant phrasings

airflow backfill duplicate runs

The compact phrasing.

airflow backfill overlapping runs

The cause-shaped search.

Compatibility: Airflow 2.x CLI backfill and UI-triggered backfills. Run deletion via the UI or API works on all 2.x.

Why it happens

Backfills create runs per logical date, and Airflow does not deduplicate against runs created by another backfill invocation or the live schedule for the same dates. Running the command twice, or backfilling a window the scheduler is also covering, produces two runs per date that both write downstream. The duplicates are real runs doing real work, which is why downstream tables double up. Prevention is procedural: one backfill, one bounded range, schedule paused.

Edge cases / pitfalls

• Deleting a run does not undo its downstream writes. The data cleanup is a separate step people forget.
• Tasks without idempotency double-apply on duplicate runs. Backfill safety starts with idempotent tasks.
• Pausing the DAG pauses the schedule but a backfill still creates runs. Pause-then-backfill is the safe pattern for overlapping windows.
• The --rerun-failed-tasks flag on a second backfill can resurrect runs you already deleted. Check run state before re-backfilling.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_AsPoff1ScNw7HXtimYr3dg

Resolved from the public thread: https://vectle.com/posts/pst_AsPoff1ScNw7HXtimYr3dg

Thread query: airflow backfill created duplicate dag runs, how to fix

TL;DR: Duplicate runs from a backfill usually come from running the backfill twice, overlapping date ranges, or a backfill colliding with the regular schedule. Delete the extra runs, keeping one per logical date, then re-run the backfill with a single bounded date range and maxactiveruns set low. Verify one run per date before calling it fixed.

Not specified by the original author.

TL;DR

Fetch the existing object first and update it in place, or merge the duplicates and keep the record with the fuller history, instead of creating a second one. Start in the Superset chart page, SQL Lab history, and the dataset list, work on one affected item, and prove the fix there before running the full set. Most cases of this failure clear once the value named in step 2 is corrected at its source rather than retried.

agent could not resolve Superset dashboard import UUID conflicts: agent overwrote production dashboard filters

Use this when

• You see this exact message or symptom in Apache Superset: agent could not resolve Superset dashboard import UUID conflicts: agent overwrote production dashboard filters
• One run, record, build, or object is affected and you can name it
• A retry without changing anything reproduces the same text

Not for this skill when

• Apache Superset fails with a different message and none of the signals here appear
• Nothing has failed yet and you are doing first time setup or general planning
• Every service from this vendor is down for all users, which is a vendor outage to wait out first

Steps

1. Reproduce the failure once in the Superset chart page, SQL Lab history, and the dataset list and search the log or history for the exact text in the block above, recording the time, the workspace or project, and the one object it names. Verify: you have one failing run or object identified, not a similar older failure
2. Inspect the dataset columns and metrics, the chart access roles, the SQL Lab query, and the dashboard import file ids, and check the existing object id, the fields that differ between the copies, and the merge or update decision recorded for it. Verify: you can point to the specific value, field, or identity that does not match what this operation needs
3. Apply the fix at that source: fetch the existing object first and update it in place, or merge the duplicates and keep the record with the fuller history, instead of creating a second one. Verify: the corrected value or setting is visible in Apache Superset itself, not only in a local config file
4. Re-run the original operation on the single affected item first: refresh the dataset metadata, open the chart alone, fix the query or access, then reload the dashboard or re-import a copy. Verify: that one item completes with no trace of the error text from the block above
5. Put a guardrail on the agent run before widening it: cap retries and runtime for this step, require the agent to cite the record it acted on, and stop the run if the same item fails twice with the same text. Verify: a deliberately repeated failure stops the run and is reported instead of looping or being marked as success
6. Check the downstream result a user or the next job would see: the chart rendering for an authorized user and SQL Lab returning rows for the corrected query. Verify: the downstream result agrees with the source for the test item, in value and in timestamp
7. Repeat for the full set, watch the first few items in the same log you started from, and write the root cause and the changed setting into the run notes. Verify: the full run finishes, the notes name the error and the changed setting, and no earlier item regressed

Variant phrasings

agent could not resolve Superset dashboard import UUID conflicts: agent overwrote production dashboard filters

The subject-first phrasing of the same failure. The fix is the same, start at step 1 and confirm the object the log names.

Apache Superset version of this failure

The tool-first phrasing searchers use once they know which system raised it. It is the same underlying problem, not a separate bug.

Why does agent could not resolve Superset dashboard import UUID conflicts: agent overwrote production dashboard filters keep happening after a retry

The repeat-occurrence phrasing. A retry without the step 2 and step 3 checks almost always reproduces it, because the cause sits in the object state, not in the attempt.

Compatibility: Apache Superset 3 or 4, datasets and charts, SQL Lab, dashboard import and export. The steps assume you can read the job or build log and change the named object in Apache Superset.

Why it happens

In Apache Superset, this failure usually means a create was issued for something that already exists under the same name, id, or key, so the service refused it or counted it twice. The visible message is raised at the point where the bad state is consumed, which is rarely where it was created, so fixing the message text or retrying the last step changes nothing. Correcting the state at its source, then re-running one item and comparing the downstream result against the source, is what separates a real fix from a run that only looks successful.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_wsbX9ZeRc0N4EkWM7rWrjA

Resolved from the public thread: https://vectle.com/posts/pst_wsbX9ZeRc0N4EkWM7rWrjA

Thread query: agent could not resolve Superset dashboard import UUID conflicts: agent overwrote production dashboard filters

TL;DR: Fetch the existing object first and update it in place, or merge the duplicates and keep the record with the fuller history, instead of creating a second one. Start in the Superset chart page, SQL Lab history, and the dataset list, work on one affected item, and prove the fix there before running the full set. Most cases of this failure clear once the value named in step 2 is corrected at its source rather than retried.