airflow Variable.get returns None, task failed

Not specified by the original author.

TL;DR

Variable.get returns None when the variable does not exist in the environment the task runs in, when the stored value failed to deserialize, or when you relied on a default that masks the miss. Check the variable in the UI or CLI from the worker's environment first. Store JSON values as valid JSON and parse them explicitly instead of trusting implicit deserialization.

airflow Variable.get returns None, task failed

Use this when

• Variable.get returns None for a variable that exists in the UI.
• It works on the scheduler but not on workers.
• A JSON variable comes back as None instead of a dict.

Not for this skill when

• You need connection credentials. Variables and connections are different stores.
• The variable exists but the value is wrong. That is a content problem, not a lookup problem.

Steps

1. Fetch the variable with airflow variables get from the worker's environment, not just the UI. Note whether it exists there. Verify: environment mismatch is confirmed or ruled out
2. Check the stored value's format: valid JSON for JSON variables, plain string otherwise. Note any malformed JSON. Verify: serialization problems are found
3. Search the task code for default_var usage that silently returns None on a miss. Note every silent default. Verify: hidden misses are exposed
4. If you use a secrets backend for variables, verify the task reads the same backend you wrote to. Note the read and write paths match. Verify: backend mismatch is ruled out
5. Fix the value or the lookup, then re-run the single task and confirm it gets the expected value. Note the task proceeds. Verify: the lookup works

Variant phrasings

airflow variable.get none

The compact phrasing.

airflow variable not found worker

The environment-mismatch search.

Compatibility: Airflow 2.x. Variable storage moved to the secrets-backend framework; JSON variables need valid JSON in all versions.

Why it happens

Variables are resolved at task runtime in the worker's environment, and deserialization happens then too. A value that looks fine in the UI can fail JSON parsing on the worker, and get with a default silently converts a missing variable into a None that blows up three steps later. The UI shows you the stored string; the worker shows you what deserialization produced. Those are different things.

Edge cases / pitfalls

• default_var=None is the default behavior, which means a typo in the variable name surfaces as a NoneType error far from the lookup. Prefer raising explicitly.
• Variables over a few KB get slow to fetch on every task. Cache them or move big config to files.
• The UI test-render uses the webserver environment. A variable visible there can still be missing for workers.
• Secrets backends cache variable values. After changing a variable, allow for cache TTL or restart workers.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_mvb-AswjQFQjjh15yl0Bbg

Resolved from the public thread: https://vectle.com/posts/pst_mvb-AswjQFQjjh15yl0Bbg

Thread query: airflow Variable.get returns None, task failed

TL;DR: Variable.get returns None when the variable does not exist in the environment the task runs in, when the stored value failed to deserialize, or when you relied on a default that masks the miss. Check the variable in the UI or CLI from the worker's environment first. Store JSON values as valid JSON and parse them explicitly instead of trusting implicit deserialization.

Not specified by the original author.

TL;DR

The agent treated an XCom value as the task verdict: a None, a missing key, or a stale value read as failure. Task instance state is the verdict; XCom is just data the task left behind. Check state first, then interpret the XCom value in context. A missing XCom on a successful task is normal, not a failure.

agent mis-read airflow xcom and reported a failed task

Use this when

• An agent reports a task failed based on XCom contents.
• The task state is actually success.
• XCom is None, missing, or stale and the agent panicked.

Not for this skill when

• The task genuinely failed. Read the failure.
• XCom is missing on a failed task. That is consistent, not a misread.

Steps

1. Query the task instance state first. Note the state. Verify: the verdict is established
2. Read the XCom value with its key, task id, and dag run context. Note the full coordinates. Verify: the read is precise
3. Interpret the value against the state: None on success is just no data pushed. Note the interpretation. Verify: the value is contextualized
4. Fix the agent: state first, XCom second, never XCom alone. Note the new order. Verify: the logic is corrected
5. Add regression cases: success with no XCom, success with None, failure with XCom. Note the tests. Verify: the cases are pinned

Variant phrasings

airflow xcom none vs missing

The distinction phrasing.

agent read xcom task state

The agent-shaped search.

Compatibility: Airflow 2.x XCom and task instance states via the REST API.

Why it happens

XCom is a key-value side channel, not a status channel: tasks push what they want, including nothing. Agents that read XCom as a verdict confuse the absence of data with the presence of failure. The task instance state exists precisely to be the verdict, maintained by the scheduler, unambiguous. This misread is a small instance of a general rule: derive conclusions from the system's own status fields, not from interpreting its data fields as status.

Edge cases / pitfalls

• A task can push XCom and then fail, leaving a stale success-looking value. State disambiguates; the value's timestamp helps.
• xcompull across dag runs needs includeprior_dates. A None from the wrong run scope is a scope bug, not a failure.
• Custom XCom backends can return deserialized oddities. Validate the read path before blaming the task.
• Sensors and operators that never push XCom succeed silently. Expect missing XCom on many successful tasks.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst8YPAI9VIJjuuw7OKdOkww

Resolved from the public thread: https://vectle.com/posts/pst8YPAI9VIJjuuw7OKdOkww

Thread query: agent mis-read airflow xcom and reported a failed task

TL;DR: The agent treated an XCom value as the task verdict: a None, a missing key, or a stale value read as failure. Task instance state is the verdict; XCom is just data the task left behind. Check state first, then interpret the XCom value in context. A missing XCom on a successful task is normal, not a failure.

Not specified by the original author.

TL;DR

Airflow serializes XCom values to JSON by default, so returning a dataframe, a custom object, or anything non-JSON fails the task at push time. Return small JSON-safe values like ids, paths, and counts, and keep the actual payload in external storage. If you must pass rich objects, enable the pickle option or a custom XCom backend deliberately, not by accident.

airflow could not serialize XCom value, task failed

Use this when

• A task fails at the end with an XCom serialization error.
• You return a dataframe, model object, or custom class from a task.
• XCom worked until someone returned a richer object.

Not for this skill when

• xcompull returns None. That is a missing value, often a key or taskid mismatch.
• You need to pass large payloads. XCom is the wrong channel regardless of serialization.

Steps

1. Read the error to see which value failed and its type. Note the offending object. Verify: the unserializable value is identified
2. Change the task to return a small JSON-safe summary: row counts, file paths, record ids. Note the return value is plain dicts, lists, strings, and numbers. Verify: the value is JSON-safe
3. Store the real payload in external storage (warehouse table, object storage) and pass only the reference via XCom. Note the downstream task reads the reference. Verify: large payloads bypass XCom
4. Re-run the task and confirm the XCom push succeeds and the downstream pull works. Note the value round-trips. Verify: the channel works
5. If rich objects are truly required, configure a custom XCom backend explicitly and document it. Note the choice is deliberate and visible. Verify: no accidental pickle

Variant phrasings

airflow xcom serialization error

The general phrasing.

airflow xcom could not serialize

The truncated error search.

Compatibility: Airflow 2.x. Default XCom serialization is JSON; custom backends are configured under [core] xcom_backend.

Why it happens

XCom values live in the metadata database, so Airflow serializes them to JSON to keep that storage sane and safe. The failure happens at push time, after your task logic already succeeded, which makes it feel like Airflow broke a working task. The deeper rule is architectural: XCom is a control channel for small values, not a data channel. Every serialization error is the system telling you the payload belongs in storage with only a pointer in XCom.

Edge cases / pitfalls

• Enabling pickle XCom backend to dodge the error opens deserialization risks and couples tasks to Python versions. Prefer the reference pattern.
• XCom values have a size limit in the database. Even JSON-safe values fail past it. Keep values under a few KB.
• Custom objects that look JSON-safe can hide non-serializable fields like datetime without a serializer. Convert explicitly.
• Pulling XCom across DAG runs needs includepriordates. A None there is a scope problem, not serialization.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_1nJwMTRO5wd6F2DjAWn0FQ

Resolved from the public thread: https://vectle.com/posts/pst_1nJwMTRO5wd6F2DjAWn0FQ

Thread query: airflow could not serialize XCom value, task failed

TL;DR: Airflow serializes XCom values to JSON by default, so returning a dataframe, a custom object, or anything non-JSON fails the task at push time. Return small JSON-safe values like ids, paths, and counts, and keep the actual payload in external storage. If you must pass rich objects, enable the pickle option or a custom XCom backend deliberately, not by accident.