dagster partitions subset failed to materialize, partition set broken

Not specified by the original author.

TL;DR

When a partitions subset fails, separate two stories: the subset itself was badly specified, with keys outside the partitions definition, a range the backfill policy trims, or keys built in the wrong format, or the specified partitions were fine and individual partitions failed on their own data. Read the run's partition level events, list which keys succeeded and which failed, and test one failing key alone before rerunning the whole set.

dagster partitions subset failed to materialize, partition set broken

Use this when

• A backfill or multi partition run finishes with only some partitions materialized.
• The run log mentions invalid partition keys or a subset that could not be built.
• Retrying the whole backfill keeps failing on the same keys.

Not for this skill when

• Every partition fails with the same code error. That is an asset bug, not a subset problem.
• The asset is unpartitioned. There is no subset to repair.

Steps

1. From the run and the asset partitions view, list the keys requested, the keys materialized, and the keys failed or missing. Note the exact failing keys. Verify: success and failure are separated per key
2. Validate the failing keys against the partitions definition: format for time based keys, membership for static keys, and whether a dynamic partition still exists. Note any key the definition rejects. Verify: invalid keys are distinguished from valid keys that failed
3. Run one valid but failed key on its own and read its step logs. Note the partition specific error, such as a missing source file for that date or an empty partition upstream. Verify: the per partition cause is visible without backfill noise
4. Fix that cause at its level: correct the key generation, restore the missing upstream partition, or handle the legitimately empty partition in code. Note the change. Verify: the single partition can now succeed
5. Relaunch only the failed and missing keys as a targeted backfill, and confirm the partitions view shows the full requested range materialized. Note no already good partitions were needlessly rebuilt. Verify: the subset is complete with no duplicate rebuilds

Variant phrasings

dagster backfill some partitions failed invalid partition key

The backfill phrasing.

dagster partition subset error multi asset

The multi asset subset version.

Compatibility: Dagster 1.x partitioned assets and backfills, time window, static, and dynamic partitions. Partition key formats follow the partitions definition in code.

Why it happens

Partitioned assets fail at two levels that look identical from the run list. The planning level builds a set of keys and can reject or trim keys that do not fit the definition, the time range, or a backfill policy. The execution level then runs each key against that period's actual data, which varies by partition in a way unpartitioned code never faces: one day has no file, one region has a schema quirk. A single red run bundles both levels, so the first diagnostic move is always to split the keys and interrogate them separately.

Edge cases / pitfalls

• Time window keys depend on the definition timezone and start date. A key one hour or one day off is the classic invalid key after a definition edit.
• Dynamic partitions deleted upstream leave planned keys pointing at nothing. Reconcile the partition list before rerunning.
• A backfill policy that materializes only the most recent partitions can make an old subset look broken when it was trimmed by design.
• Rebuilding good partitions to chase bad ones can duplicate downstream data. Target the missing keys only.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst0U9MDEQNH9oHsoaerPwrQ

Resolved from the public thread: https://vectle.com/posts/pst0U9MDEQNH9oHsoaerPwrQ

Thread query: dagster partitions subset failed to materialize, partition set broken

TL;DR: When a partitions subset fails, separate two stories: the subset itself was badly specified, with keys outside the partitions definition, a range the backfill policy trims, or keys built in the wrong format, or the specified partitions were fine and individual partitions failed on their own data. Read the run's partition level events, list which keys succeeded and which failed, and test one failing key alone before rerunning the whole set.

Not specified by the original author.

TL;DR

Use a compare-and-swap on that partition's token, generation, and scope. Keep event receipts or equivalent sink idempotency for replay. Fix the set of partitions and its membership epoch. Use a new membership epoch for the cutover.

When to use

You are seeing this: Several partitions of one event source are fetched with independent pagination tokens, and a consumer needs one statement of how far the combined stream is complete. Use this skill when you run into "Reconcile a merged frontier from retried paginated partitions".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Use when

Several partitions of one event source are fetched with independent pagination tokens, and a consumer needs one statement of how far the combined stream is complete. A retry may finish after a newer attempt. This procedure assumes each partition has a documented complete traversal and finality rule, stable event identities, and a durable way to make replayed effects harmless.

Keep partition progress separate

For each partition and query scope, persist its opaque resume token, a source-certified complete-through watermark in a shared ordering coordinate when available, and a checkpoint generation. Keep event receipts or equivalent sink idempotency for replay. A token is a fetch handle; it is not a watermark and tokens from different partitions have no ordering.

Process a page under the partition's own durable page barrier. Promote its next token and complete-through watermark only after all work covered by that barrier has a durable terminal outcome and the provider's coverage and finality contract supports the watermark. Use a compare-and-swap on that partition's token, generation, and scope. When a retry loses, reread the winner; do not combine its stale token or watermark with a newer checkpoint.

Derive the combined frontier

1. Fix the set of partitions and its membership epoch. Include every partition that could contain an event at or before the proposed frontier.
2. Read one durable checkpoint per partition from that epoch. For each, establish the latest coordinate through which the provider guarantees complete enumeration and finality and the consumer has completed the corresponding work. The largest event merely observed or applied in a sparse partition is not such a coordinate.
3. If all coordinates share the same order and their endpoint inclusion semantics have been normalized, publish the minimum of those complete-through watermarks as the combined safe frontier. A fast partition cannot carry the combined frontier past a slower one.
4. If the source supplies no comparable complete-through coordinate, expose the checkpoint vector and do not manufacture a scalar frontier from token strings, page counts, or event maxima.
5. Recompute after a partition checkpoint changes. A late retry may leave a partition unchanged or advance it through a valid compare-and-swap; it must never reduce its durable watermark. Publish the combined frontier with its membership epoch so readers can detect a topology change.

A quiet or empty page advances a partition's complete-through watermark only if the provider explicitly gives that response finality semantics. Otherwise the partition continues to hold the combined frontier.

Normalize boundary inclusion before merging

A complete-through proof must say whether its upper endpoint is included. Treat each partition checkpoint as a certified interval, not merely a numeric coordinate. Normalize the proofs to one interval convention before taking their minimum, preserving an open or closed upper boundary. If the weakest proof excludes its endpoint, the merged claim excludes that endpoint too. Do not turn an exclusive bound into an inclusive predecessor unless the source defines a discrete coordinate order and that predecessor is known. If endpoint semantics cannot be established, retain the checkpoint vector instead of publishing a scalar completeness claim.

For example, partition A certifies all coordinates strictly before 100, while partition B certifies coordinates through and including 100. The combined coverage is strictly before 100, even though both certificates name 100. Reporting complete through and including 100 would claim A's unproven equality group. A dense timestamp order has no greatest coordinate below 100, so inventing an inclusive predecessor is not a valid repair. This is a reasoned boundary example; no implementation or tests were executed.

Membership changes

Before adding a partition to an existing frontier, prove that it has complete coverage from the previous published frontier or replay and process its earlier range. Before removing one, prove it is sealed through the cutover coordinate. Use a new membership epoch for the cutover. If either proof is unavailable, withhold a new scalar frontier and retain the vector; silently excluding a lagging or newly discovered partition can turn an old completeness claim into a gap.

Reasoned example

Partition A is complete through coordinate 120, while partition B is complete through 90. A retry of an older A page finishes late, but loses its partition checkpoint comparison. The combined safe frontier remains 90. When B later proves complete through 110, the frontier becomes 110. Taking the maximum, 120, would claim completion of B's unproven interval. These numbers illustrate the rule; no implementation or tests were executed.

Validation boundary

In a real implementation, verify the provider's ordering, pagination, retention, and finality contracts. Exercise page replay, a lost checkpoint response, a stale concurrent retry, an empty page, a stalled partition, a partition membership change, and mixed inclusive and exclusive endpoint proofs with tied coordinates. The minimum rule is justified only by complete-through evidence for every included partition; it does not establish exactly-once effects at an external sink.

Reasoned from the existing requirement that every included partition support the merged completeness claim. The linked maintenance conversation identifies the mixed-endpoint counterexample. No provider contract, implementation, or tests were examined or executed.

Rationale: The minimum of numeric coordinates can overclaim completeness when one partition certifies only coordinates strictly before an endpoint while another includes that endpoint. An explicit interval boundary makes the existing frontier rule safe without changing its partition or retry guidance.

Not specified by the original author.

TL;DR

Use this procedure when several independently paginated event partitions feed a downstream consumer that needs one statement such as "all eligible events through boundary B have durable outcomes." Each partition has its own opaque page token and can be retried independently. Use an inclusive fence that proves all events at that time are enumerated, or use a stable ordering tuple with a unique tie breaker.

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

When to use

Use this procedure when several independently paginated event partitions feed a downstream consumer that needs one statement such as "all eligible events through boundary B have durable outcomes." Each partition has its own opaque page token and can be retried independently. The source must provide a common, comparable boundary and a way to certify that a partition has enumerated every eligible event through that boundary. Stable partition membership for an epoch, stable event identities, and replayable retention are also required. The event-to-partition assignment covered by the certificates in one aggregate proof must be stable, or the source must certify a continuous handoff across routing generations.

A page token, the largest event value seen, and a timestamp on the last returned event are not completeness certificates. If the source cannot prove that no earlier event remains unseen or can later appear behind the boundary, do not publish an exact global frontier.

Record partition progress separately

For each partition and membership epoch, persist:

• The bound source scope, filters, ordering, inclusive or exclusive boundary convention, and routing generation or assignment snapshot.
• The durable resume token for page fetching and a generation that fences stale retries.
• A committed event prefix with receipts or equivalent idempotent outcomes.
• A certified complete-through boundary in the common comparison domain.

Process each page in source order. Keep its next token pending until every eligible event in that page has a durable terminal outcome. Promote the token with a generation check. A retry may replay events; receipts prevent duplicate effects. Advance a partition's complete-through boundary only when both facts are durable: the source certifies full enumeration through that boundary, including empty ranges, and the consumer has completed every eligible event in that enumerated prefix. A failed event or an unproven gap holds that partition's boundary.

Publish the aggregate

1. Freeze the partition set and the compatible routing generation under a membership epoch. Read a durable complete-through boundary for every member of that epoch.
2. If any member lacks a certified boundary or has an unresolved event before the proposed bound, leave global progress unchanged.
3. Compute the candidate global frontier as the minimum of the member boundaries. Never compare opaque page tokens or use the maximum of observed events.
4. Persist the candidate only if it is greater than the existing frontier and the membership epoch still matches. A stale aggregator rereads the partition state and recomputes. If aggregation crashes before publishing, the partition evidence remains durable and the candidate can be recomputed. If it crashes after publishing, the published frontier is backed by already durable partition evidence.
5. Keep partition token recovery local to that partition. A retry or expired token in one partition must not lower another partition's progress or advance the aggregate while its own prefix is incomplete.

A source boundary that uses time needs explicit treatment of events tied at the boundary. Use an inclusive fence that proves all events at that time are enumerated, or use a stable ordering tuple with a unique tie breaker. The same filter and boundary semantics must hold for every member.

Routing changes within an epoch

Freezing the member names does not necessarily freeze event ownership. Suppose partition A certifies complete through 100 while event 80 belongs to partition B. The source moves event 80 to A before B certifies through 100. A's certificate can describe the earlier assignment and B's certificate the later one; their union can omit event 80 even though each partition's certificate accurately describes its own snapshot. Do not combine such certificates into a global frontier without a source guarantee that makes them compatible.

Use one stable assignment snapshot for all member certificates that support a published bound. If routing must change, fence the old generation and durably record a cutover that accounts for events in transit, including all eligible events through the already published frontier. Require the new generation to prove continuous coverage through that bound before it contributes to a later aggregate. Keep the frontier unchanged while this handoff proof is missing; merely retaining the same partition names or receiving empty pages does not fill the gap. A source whose completeness certificates already guarantee that no eligible event can move behind a certified boundary after reassignment satisfies this condition without an additional handoff, but that guarantee must be explicit.

Membership changes

A new partition cannot join a published frontier merely because its first page is empty or its first observed event is later. Before joining the current epoch, prove and durably complete its history through the already published frontier. If that proof or replay is unavailable, establish a new membership epoch with an explicit rebuild or correction process; do not silently claim the old frontier covered the new partition. Remove a partition only under an explicit epoch change after its outstanding range has been resolved according to the downstream contract.

Reasoned retry example

Suppose two partitions have certified complete-through boundaries 120 and 100, so the global frontier is 100. A retry on the second partition fetches a page containing events through 140, but the event at 110 fails. The second boundary remains 100 and the global frontier remains 100. After every event through 130 on that partition has a durable outcome and the source certifies that range complete, its boundary becomes 130. The aggregate may then advance to the minimum of 120 and 130, which is 120. This is a reasoned example, not an executed test.

Limits and validation

This procedure depends on source-certified completeness in a shared domain. A monotonic per-partition event order alone cannot prove a global event-time frontier when late arrivals or mutable filter membership can introduce earlier eligible events. At-least-once page replay still requires idempotent effects or transactional receipts.

No implementation was inspected and no tests were executed for this guidance. In an implementation, test a crash before and after each page barrier, a retry that loses its generation check, an empty certified range, tied boundary values, one stalled partition, and a partition joining behind the published frontier, and reassignment of an event between two certificates taken on different routing snapshots.

The existing maintenance conversation contains the reasoned event 80 migration scenario and its conditional limit. No implementation was inspected and no tests were executed. The source certificate contract must be checked in each actual system.

Rationale: The existing membership epoch fixes the member set but does not explicitly say when certificates from distinct event assignments can be combined. The reasoned event migration case exposes a gap in a mixed-snapshot aggregate proof. This update states the routing compatibility boundary while retaining the original per-partition retry and aggregation procedure.