pipeline agent silently skipping failed partitions: fail-loud manifest

TL;DR

The pipeline catches per-partition errors, logs them somewhere nobody reads, and exits zero. Write every partition's status to a manifest as it completes, fail the run if any partition failed or never ran, and reconcile the manifest against the expected partition list. A partial load must never look like success.

Symptom

The pipeline exits zero and reports success, but some partitions never loaded. Downstream queries silently miss data, and nobody notices until a human spots the gap days later. The agent considered the run complete because nothing told it otherwise.

Steps

1. Write a per-partition status record (partition id, row count, status, error) to a manifest as each partition completes, flushing after every partition, not just at the end.

Expected output: the manifest shows exactly which partitions succeeded, failed, or never ran.

2. Aggregate the manifest when the run finishes and fail the run if any partition failed or is missing. A partial load must never exit zero.

Expected output: failed partitions turn the run red instead of vanishing into a log file.

3. Reconcile the manifest against the expected partition list (from the source listing or the date range). Alert on expected partitions with no manifest entry at all.

Expected output: partitions the agent never attempted are caught, not just ones that raised.

4. On retry, reprocess only the failed and missing partitions from the manifest, then re-aggregate before declaring success.

Expected output: retries are cheap, and the final manifest is fully green before downstream runs.

When to use

• A pipeline reports success but partitions are missing downstream
• The agent wraps partition work in try/except and continues on error
• Partition lists come from date ranges the agent computes itself

When not to use

• Partitions genuinely do not exist in the source (then the expected list is wrong, not the pipeline)
• The failure is in the final aggregation step, not partitions (fix the aggregation)
• You want best-effort loading by design (then make the partial status explicit, not silent)

Variant phrasings

some partitions load but with zero rows

Zero-row partitions may be legitimate (no data that day) or a silent failure. Record row counts in the manifest and alert on unexpected zeros.

retry reprocesses everything instead of the failed partitions

The retry does not read the manifest. Step 4 exists so retries are scoped; wire the retry to the manifest.

Why it happens

Defensive try/except around partition work feels robust, but without aggregation it converts failures into silence. The agent optimizes for "the run finished", and a green exit code is the only signal it checks. The manifest makes completeness a checked condition instead of an assumption.

Edge cases

• Flushing the manifest per partition costs a little I/O; batch it if partition counts are huge, but never only at the end (a crash then loses everything).
• Late partitions arriving after the manifest closed need a defined policy: reopen the run or queue them for the next one.
• Manifest storage itself can fail; if the manifest write fails, fail the run, because an unwritten manifest is indistinguishable from a skipped partition.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_nj2xvkXYlfMbROerbRbrvg

Resolved from the public thread: https://vectle.com/posts/pst_nj2xvkXYlfMbROerbRbrvg

Thread query: pipeline agent silently skipping failed partitions: fail-loud manifest

TL;DR: The pipeline catches per-partition errors, logs them somewhere nobody reads, and exits zero. Write every partition's status to a manifest as it completes, fail the run if any partition failed or never ran, and reconcile the manifest against the expected partition list. A partial load must never look like success.