airflow could not serialize XCom value, task failed

TL;DR

Airflow serializes XCom values to JSON by default, so returning a dataframe, a custom object, or anything non-JSON fails the task at push time. Return small JSON-safe values like ids, paths, and counts, and keep the actual payload in external storage. If you must pass rich objects, enable the pickle option or a custom XCom backend deliberately, not by accident.

airflow could not serialize XCom value, task failed

Use this when

• A task fails at the end with an XCom serialization error.
• You return a dataframe, model object, or custom class from a task.
• XCom worked until someone returned a richer object.

Not for this skill when

• xcompull returns None. That is a missing value, often a key or taskid mismatch.
• You need to pass large payloads. XCom is the wrong channel regardless of serialization.

Steps

1. Read the error to see which value failed and its type. Note the offending object. Verify: the unserializable value is identified
2. Change the task to return a small JSON-safe summary: row counts, file paths, record ids. Note the return value is plain dicts, lists, strings, and numbers. Verify: the value is JSON-safe
3. Store the real payload in external storage (warehouse table, object storage) and pass only the reference via XCom. Note the downstream task reads the reference. Verify: large payloads bypass XCom
4. Re-run the task and confirm the XCom push succeeds and the downstream pull works. Note the value round-trips. Verify: the channel works
5. If rich objects are truly required, configure a custom XCom backend explicitly and document it. Note the choice is deliberate and visible. Verify: no accidental pickle

Variant phrasings

airflow xcom serialization error

The general phrasing.

airflow xcom could not serialize

The truncated error search.

Compatibility: Airflow 2.x. Default XCom serialization is JSON; custom backends are configured under [core] xcom_backend.

Why it happens

XCom values live in the metadata database, so Airflow serializes them to JSON to keep that storage sane and safe. The failure happens at push time, after your task logic already succeeded, which makes it feel like Airflow broke a working task. The deeper rule is architectural: XCom is a control channel for small values, not a data channel. Every serialization error is the system telling you the payload belongs in storage with only a pointer in XCom.

Edge cases / pitfalls

• Enabling pickle XCom backend to dodge the error opens deserialization risks and couples tasks to Python versions. Prefer the reference pattern.
• XCom values have a size limit in the database. Even JSON-safe values fail past it. Keep values under a few KB.
• Custom objects that look JSON-safe can hide non-serializable fields like datetime without a serializer. Convert explicitly.
• Pulling XCom across DAG runs needs includepriordates. A None there is a scope problem, not serialization.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_1nJwMTRO5wd6F2DjAWn0FQ