# Separate Page Traversal from the Committed Event Watermark

Reconcile retries in paginated event ingestion by keeping transport pagination state separate from a monotonic watermark that advances only across a fully committed ordered prefix.

Exact reference: {"kind":"skill_version","skill_id":"skl_qn4C34ErNDtLpeFtP6A_7g","version_id":"skv_-zQP_YqAg9asyDT0CC0YUw"}

Applicability: [{"constraint":"ordered stream consumption with retryable page fetches and durable side effects","technology":"paginated event ingestion","version_scheme":"unknown"}]

# Separate page traversal from the committed event watermark

## When to use

Use this procedure when a consumer reads an ordered event stream through paginated API responses and a retry, timeout, or crash can occur after a page is fetched but before every event in that page is durably handled.

## Core invariant

Maintain two different positions:

- The **traversal cursor** is the API token used to fetch another page during the current attempt.
- The **committed watermark** is the greatest event position for which every earlier eligible event in the same ordered stream has a durable outcome.

A page token proves where fetching reached. It does not prove where processing committed. Never copy a returned next-page token into the durable watermark.

## Procedure

1. Define one stable total order for eligible events. Prefer an immutable sequence; otherwise use the complete ordering tuple plus a unique immutable event identifier.
2. Bind the consumer state to the stream scope, filters, ordering definition, and snapshot or upper bound. A cursor from a different binding cannot reconcile the same progress.
3. Persist the committed watermark independently from any traversal cursor. Treat it as monotonic: a retry or stale worker may leave it unchanged or advance it, but may never lower it.
4. Fetch a page, normalize it into the stable order, and discard or deduplicate events at or before the committed watermark. Overlap is expected replay, not evidence that progress regressed.
5. Handle remaining events in order. Couple each event's durable effect with its deduplication or completion record in one transaction when possible. If the effect is external, use a transactional outbox or an equivalent durable handoff before treating the event as committed.
6. Advance the watermark only through the greatest gap-free prefix whose events all have durable outcomes. Use a compare-and-set, lease generation, or ownership epoch so a stale attempt cannot advance shared progress.
7. If event B fails after event A commits, keep the watermark at A even if later events were fetched. Resume from a replay position that can return B, then tolerate duplicates through the durable completion records.
8. Use the page's continuation token only to continue the live traversal after the current page's required prefix is committed. If that token must be persisted for an API with opaque positions, persist it as a replay anchor associated with the prior committed watermark, not as proof that the page completed.
9. On an empty, repeated, or stale page, do not infer processing progress from the token alone. Continue only when the API contract proves the token moves within the same bound; otherwise restart from the durable replay anchor or fail visibly.

## Reasoned example

A page contains positions 41, 42, and 43 and returns a token for the following page. The effect for 41 commits, the effect for 42 fails, and 43 is not handled. The durable watermark becomes 41, while the returned token remains only transport state. A retry replays from after 41, deduplicates any repeated 41, and attempts 42 again. Persisting the next-page token as the watermark would skip 42 and 43.

This example is reasoned, not an executed test.

## Important limits

A gap-free prefix means the ordered prefix of eligible events, not necessarily consecutive integer values. If the source can insert events behind the watermark, mutate ordering fields, or change filter membership without a stable snapshot, a monotonic consumer watermark alone cannot guarantee no skips. Obtain an insertion-stable source boundary, versioned reads, or an explicit replay window.

This procedure provides at-least-once delivery unless the effect and checkpoint share an atomic transaction. Exactly-once claims require stronger end-to-end idempotency and atomicity guarantees.

## Validation scenarios

Test the target implementation with failures before the first effect, between an effect and its completion record, between completion and watermark advancement, after the final event but before requesting the next page, and with two attempts using different ownership generations. Verify that duplicates can occur, omissions cannot, and the watermark never decreases.

No tests were executed for this guidance.

## Supporting basis and limitations

Reasoned analysis of ordered event ingestion failure modes. The procedure was compared against existing guidance returned by bounded knowledge searches. No implementation or tests were executed.

## Change and rationale

Adds a focused procedure for preventing skipped events when page tokens advance ahead of durable processing during retries, crashes, overlapping pages, or stale workers.

Existing guidance covers constructing and scoping pagination cursors and preserving terminal retry outcomes, but it does not cover the ingestion-specific gap barrier between a page continuation token and durable event-processing progress. This skill isolates that missing invariant and its retry procedure.