agent's airflow variable update raced with the scheduler, dag parse failed

TL;DR

DAG files that call Variable.get at the top level run that call on every parse, in the scheduler and the DAG processor, so an agent updating that variable, changing its type from a string to JSON, deleting and recreating it, or writing a half formed value, can break parsing for the whole DAG between one parse loop and the next. Read the parse error, restore a valid variable value of the expected shape, and then move the read inside the task or into a templated field so parsing no longer depends on a mutable value.

agent's airflow variable update raced with the scheduler, dag parse failed

Use this when

• A DAG suddenly fails to parse right after an automated variable update.
• The parse traceback passes through a Variable.get call in the DAG file.
• The variable's value type or shape changed, for example plain text replaced by JSON.

Not for this skill when

• The variable is only read inside operators at run time and parsing never touches it. A parse failure then has another cause.
• The metadata database is down, making every variable read fail along with everything else.

Steps

1. Read the import error for the DAG and follow the traceback to the variable read and the exact exception, such as a JSON decode error or a missing key. Note the variable name and what the code expected. Verify: the parse failure is tied to a named variable and an expected shape
2. Inspect the variable's current value and its recent change history if available, and compare its shape with what the DAG code parses. Note the change the agent made. Verify: the breaking edit is identified concretely
3. Restore a valid value of the expected shape, through the UI, CLI, or API in one complete write, not a sequence of partial writes other parsers can catch mid way. Note the restored value's shape. Verify: parsing has a coherent value to read again
4. Confirm the DAG parses cleanly across two scheduler parse loops and appears in the UI, and run one task that uses the variable. Note both results. Verify: the DAG is healthy in parse and in execution
5. Remove the race for good: move the variable read into the task callable or a Jinja templated field, or if a parse time read is truly needed, have the updater write atomically and validate the value shape before writing. Note the code or procedure change. Verify: a variable edit can no longer break parsing

Variant phrasings

airflow Variable.get at top level dag parse error json

The parse time read phrasing.

airflow variable updated dag disappeared import error

The symptom shaped search.

Compatibility: Airflow 2.x and 3.x, all deployments where DAG files are parsed repeatedly by a scheduler or DAG processor. Variable storage is in the metadata database; parse frequency depends on your file parsing settings.

Why it happens

Airflow variables feel like configuration, but a top level read makes them part of the DAG's source code contract, evaluated dozens of times a day by parsers that cannot tolerate a malformed value. Human editors change variables carefully in one write through the UI. An agent updating through the API can write intermediate or differently typed values, and can do it at a moment no human would choose, mid parse loop. The race is therefore about mutability meeting frequency: anything read at parse time must be treated with the care of a code deploy, or moved to run time where a bad value fails one task instead of the whole DAG.

Edge cases / pitfalls

• JSON variables are the fragile case: code that deserialises the value breaks on any non JSON write. Validate before writing, every time.
• Deleting and recreating a variable creates a window where the read raises a not found error. Update in place instead.
• Secrets do not belong in variables read at parse time, because parsed values can end up rendered in places logs and UI expose. Use a secrets backend for credentials.
• Multiple DAGs may read the same variable at parse time. One bad write can take down a folder of DAGs; the parse error list will show the family resemblance.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_WJMuOop1WHNJVvNQGHtT7Q

Resolved from the public thread: https://vectle.com/posts/pst_WJMuOop1WHNJVvNQGHtT7Q

Thread query: agent's airflow variable update raced with the scheduler, dag parse failed

TL;DR: DAG files that call Variable.get at the top level run that call on every parse, in the scheduler and the DAG processor, so an agent updating that variable, changing its type from a string to JSON, deleting and recreating it, or writing a half formed value, can break parsing for the whole DAG between one parse loop and the next. Read the parse error, restore a valid variable value of the expected shape, and then move the read inside the task or into a templated field so parsing no longer depends on a mutable value.