recovering from API outages mid-pipeline

TL;DR

Treat API outages as expected weather: retry retryable errors with backoff, fail fast on fatal ones, and resume from the last persisted watermark instead of restarting. The pipeline that resumes cleanly after an outage is the pipeline that was designed for outages. Log what happened, note the gap in coverage, and move on.

recovering from API outages mid-pipeline

Use this when

• A pipeline run died because an API returned 500s, timed out, or rate-limited
• You are designing retry behavior for a new pipeline
• A rerun after an outage produced duplicates or skipped items
• You need the postmortem format for a dependency failure
• You are deciding how much outage-proofing a pipeline deserves

Not for this skill when

• You run the API that went down (that is incident response, different skill)
• You need to communicate the outage to your users (status page discipline)
• The failure is your code's bug, not the dependency's outage
• You are choosing a more reliable vendor (different decision)

Steps

1. Classify the error before retrying anything. Retryable: 500s, 502/503/504, timeouts, 429s, connection resets. Fatal: 400s, 401/403, 404s, and any error the docs say is permanent. Retrying fatal errors burns quota and time; giving up on retryable ones wastes a run.

   retryable: [5xx, timeouts, 429, connection errors]
   fatal: [4xx except 429, auth failures, permanent rejections]

   Expected output: a classification list in the code or runbook. Success check: the last outage's errors map cleanly onto it.

2. Retry with backoff and a cap, then stop and save state. A few quick retries for blips, longer waits for sustained outages, and a hard stop that persists the watermark instead of spinning forever. A pipeline that retries forever is a pipeline that never reports it is stuck.

   retries: [n] quick, then backoff to [max wait], then stop and persist

   Expected output: bounded retry behavior. Success check: a sustained outage ends in a clean stopped state, not a hung process.

3. Resume from the watermark; never restart from zero. Load the last persisted position, re-verify in-progress items, and continue. Combined with idempotent workers, resume-after-outage is just a normal restart.

   [ ] watermark loaded on boot
   [ ] in-progress items re-verified before continuing
   [ ] zero duplicates after resume (spot-checked)

   Expected output: a clean resume. Success check: the resumed run's results contain no duplicates and no gaps.

4. Distinguish outage damage from real failures in the results. Items that failed because the API was down get marked as such and retried; items that failed on their own merits stay failed. Mixing the two corrupts your quality metrics.

   [ ] outage-caused failures tagged separately from genuine failures
   [ ] genuine failures investigated on their own

   Expected output: tagged results. Success check: the failure rate excluding outage damage matches the historical baseline.

5. Write the postmortem note: what went down, how long, what the pipeline did, what you will change. One paragraph plus one action item. Outages are free load tests of your recovery design; the note is how you collect the value.

   outage: [what/when/duration], pipeline behavior: [summary], change: [one action]

   Expected output: a dated note. Success check: the action item has an owner and a date.

Variant phrasings

API outage retry best practices

Retry phrasing. Steps 1 and 2: classify first, then bounded backoff; never retry blindly.

How to resume a pipeline after a failure

Resume phrasing. Step 3: watermark, re-verify, continue; idempotent workers make it safe.

503 errors mid-run, what to do

Error-specific phrasing. 503 is retryable (step 1); the playbook is backoff, persist, resume.

Postmortem template for dependency outages

Postmortem phrasing. Step 5's one-paragraph-plus-action format; keep it short or nobody writes it.

Why it happens

Distributed systems fail partially and temporarily; a pipeline that assumes its dependencies are always up assumes wrong. The recovery design (watermarks, idempotency, bounded retries) looks like extra work until the first outage, when it becomes the difference between "resumed in an hour" and "reran everything and deduplicated by hand". Outages are not exceptional; they are the operating environment.

Edge cases / pitfalls

• A 429 during an outage is the API asking you to back off; honor it aggressively. Retry storms during partial outages turn your pipeline into part of the problem.
• Partial outages produce partial results that look complete. Verify result counts against expectations after any outage-touched run.
• Do not auto-resume into a still-degraded API without a health check. Resume should verify the dependency is healthy first, or it will just fail again louder.
• Cascading retries across a fleet multiply load. If many workers share one dependency, coordinate the backoff (add jitter, or serialize) instead of letting them thundering-herd the recovery.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_s1GgqTBrgXKa0DA2KeEFhA