Crash-safe cursor handoff for retried event pages

When to use

Use this procedure when an event API returns an ordered page plus an opaque continuation token, a consumer performs durable effects, and a timeout or crash can cause the same page or an overlapping page to be fetched again.

Assume event positions form a stable total order. Treat continuation tokens as opaque transport state: they may encode server state and need not be numerically comparable to event positions.

Keep two kinds of progress

Persist a checkpoint with separate fields:

• resume_cursor: the provider token to use for the next fetch.
• applied_position: the greatest position in the source-ordered prefix whose events have durable outcomes.
• checkpoint_version: a compare-and-swap generation for fencing concurrent or late retries.
• An optional event receipt ledger keyed by stable event identity when positions can be backfilled, reused, or delivered out of order.

The cursor says where to fetch. The applied position says what work is durably complete. Never substitute one for the other.

Required invariants

1. A returned continuation token remains pending until every event in that response is durably classified as applied, intentionally ignored, or permanently quarantined by policy. A worker's in-memory quarantine decision is not a durable terminal classification.
2. applied_position advances only after the corresponding effect and deduplication receipt are durable. It is never the maximum position merely observed.
3. A page barrier may replace resume_cursor only with a compare-and-swap that still matches the request cursor and checkpoint version used to fetch the page.
4. Replaying an event cannot repeat an unsafe effect. Enforce a unique event receipt, use a sink idempotency key, or couple the effect and receipt in one transaction.
5. A failed item, retryable dead-letter outcome, operator-review hold, or reversible quarantine blocks the page barrier even when later items were visible in the response.

Procedure

1. Load the checkpoint and remember its cursor and version as the page attempt identity.
2. Fetch with that cursor. Hold the returned next cursor in memory or as non-authoritative attempt metadata; do not publish it as the resume cursor yet.
3. Validate that new events respect the source order. Overlap at or below the applied prefix is acceptable. An unexpected regression, conflicting identity at the same position, or a gap that violates the provider contract stops the attempt for investigation.
4. Process events in source order.

• If a durable receipt proves the event already has its intended outcome, treat it as replayed.
• A high-water mark alone may prove replay only when positions are immutable and the stored prefix is known complete. If the source permits late backfills, use per-event receipts instead.
• Commit each effect together with its receipt and the resulting applied_position, or use a transactional inbox or outbox that gives the same durable ordering.
• Treat quarantine as terminal only when a durable receipt binds the stable event identity to a final policy disposition. A transient dead-letter entry, a pending operator decision, or a reversible hold does not close the prefix.
• Stop at the first outcome that is not durable and terminal.

1. After every event in the page has a durable outcome, compare-and-swap the checkpoint from the attempt cursor and version to the returned next cursor and a greater version. A stale retry that loses this comparison must reread state rather than overwrite newer progress.
2. On a crash before the page barrier, retry from the unchanged resume cursor. Receipts or the complete-prefix watermark make already applied events harmless replays.
3. Treat a successful empty page with a new continuation token as a zero-item page: it may pass the barrier without changing applied_position if the provider documents that behavior. Do not equate emptiness with end of stream. If the returned token equals the request token, stop or back off instead of spinning.

Reasoned retry walkthrough

Suppose the durable checkpoint contains cursor C7 and applied position 105. Fetching C7 returns positions 104, 106, and 107, followed by cursor C8.

Position 104 is an overlap. The consumer confirms its durable receipt, applies 106, records position 106, and then crashes before 107. The resume cursor remains C7. On retry, 104 and 106 are recognized as complete, 107 is applied, and only then does the page barrier replace C7 with C8. If 107 never obtains a durable outcome, C8 is never promoted.

This walkthrough is a reasoned example, not an executed test.

Quarantine boundary example

Suppose a page contains positions 210, 211, and 212. The outcomes for 210 and 212 are durable, but the worker has only decided in memory that 211 should be quarantined. The complete durable prefix stops at 210, so applied_position cannot advance past 210 and the page's next cursor remains pending.

If the system later stores a durable quarantine receipt for 211 that binds its stable event identity to a final policy disposition, position 211 has a terminal outcome. Because 212 is already durable, the complete prefix may then extend through 212 and the page barrier may proceed. By contrast, a receipt that means retry later, await operator review, or hold reversibly is not terminal and continues to block the prefix.

This example defines checkpoint eligibility; it does not require quarantined work to be represented as successful work, and it does not prescribe a universal receipt schema or retention period.

Common unsafe shortcuts

• Persisting the returned token immediately after a successful fetch.
• Setting the applied watermark to the largest position present in the page.
• Comparing or incrementing opaque provider tokens as if they were event positions.
• Letting a late retry overwrite a newer checkpoint without a version check.
• Relying on a watermark when the provider permits backfills behind it.
• Recording success before a non-idempotent external effect is durably known.
• Treating an in-memory, review-pending, or reversible quarantine as a durable terminal outcome.

Limits and validation

This procedure gives at-least-once page replay with no skipped page prefix under the stated ordering assumptions. It does not create exactly-once behavior for an external sink that lacks transactions or idempotency. Validate the implementation with crash injection before each event commit, after each event commit, before and after a durable quarantine receipt, before the page barrier, and after the barrier; also test overlapping pages, stale concurrent retries, reversible quarantine holds, empty pages with advancing tokens, and non-advancing tokens.

The procedure and walkthroughs above are based on invariant reasoning. No tests were executed while creating this guidance.