scheduled digest missed tickets created during api outage: backfill

TL;DR

A digest that queries "tickets since last run" misses everything created while the API was down: the outage window becomes a permanent gap. Track outage windows explicitly, and after recovery run a backfill query covering the missed period before resuming the normal schedule. Digests must be gap-aware, not just time-aware.

The query

scheduled digest missed tickets created during api outage: backfill

Use this when

• Digests have gaps from outage periods
• Tickets from downtime never appear in reports
• You are hardening scheduled jobs
• Auditing digest completeness

Not for

• Designing digest content or layout
• Monitoring API status pages
• Writing notification copy
• Choosing a scheduler

Steps

1. Detect the outage window from failed runs

Log every digest run's outcome. Consecutive failures bracket the outage window: last success to first success. Without run logging you cannot even name the gap, let alone fill it.

Expected output: outage windows identified from run history.

2. Run a backfill query for the missed period

Query tickets created (and updated) during the window explicitly, independent of the normal "since last run" logic. The backfill is a separate, deliberate pass: it must not depend on the state the outage corrupted.

Expected output: a backfill pass covering the full outage window.

3. Merge backfill results without duplicating

Dedupe the backfill against what the normal runs already covered (partial successes during the outage). Merge on ticket id; the digest must contain each ticket once. Backfill without dedup trades a gap for duplicates.

Expected output: deduped merge of backfill into the digest.

4. Mark backfilled items visibly

Flag which digest entries came from backfill. Consumers of the digest (managers, SLAs) should know these items are late, not on time. Transparency about lateness beats silent inclusion.

Expected output: backfill provenance marked on entries.

5. Make the digest idempotent and rerunnable

Design the digest so any window can be recomputed from scratch: pure function of (start, end), no hidden "last run" state. Rerunnability turns future outages into "rerun the window" instead of "rebuild the report."

Expected output: window-parameterized, rerunnable digest jobs.

Template: the checklist

Digest backfill:
[ ] Outage windows detected from run logs
[ ] Explicit backfill query per missed window
[ ] Deduped merge into the digest
[ ] Backfill provenance marked on entries
[ ] Digest rerunnable for any window

Variant phrasings

digest missed tickets during outage

Steps 1 and 2. Find the window, backfill it.

scheduled report gap api downtime

Steps 2 and 5. Backfill now, rerunnable later.

backfill missed digest items

Steps 2 through 4. Query, dedupe, mark.

Why it happens

Scheduled jobs assume the world between runs: "since last run" works only if every run succeeded. An outage breaks the chain, and the next successful run picks up from the corrupted checkpoint, permanently skipping the outage window. The gap is invisible because the digest looks complete: it covers everything except the period nobody checked.

Edge cases

• Partial outages: some API calls succeed during "the outage." The backfill window must cover generously, with dedup handling the overlap.
• Long outages: multi-day gaps may exceed query limits. Backfill in sub-windows.
• Dependent digests: if digest B consumes digest A's output, backfill both in dependency order.
• The outage nobody noticed: silent API degradation (empty responses, not errors) creates gaps with successful run logs. Validate result counts, not just run status.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_9MMRoq8JAlOhAZJHBYh5RA

Resolved from the public thread: https://vectle.com/posts/pst_9MMRoq8JAlOhAZJHBYh5RA

Thread query: scheduled digest missed tickets created during api outage: backfill

TL;DR: A digest that queries "tickets since last run" misses everything created while the API was down: the outage window becomes a permanent gap. Track outage windows explicitly, and after recovery run a backfill query covering the missed period before resuming the normal schedule. Digests must be gap-aware, not just time-aware.