Reconcile a merged frontier from retried paginated partitions

Use when

Several partitions of one event source are fetched with independent pagination tokens, and a consumer needs one statement of how far the combined stream is complete. A retry may finish after a newer attempt. This procedure assumes each partition has a documented complete traversal and finality rule, stable event identities, and a durable way to make replayed effects harmless.

Keep partition progress separate

For each partition and query scope, persist its opaque resume token, a source-certified complete-through watermark in a shared ordering coordinate when available, and a checkpoint generation. Keep event receipts or equivalent sink idempotency for replay. A token is a fetch handle; it is not a watermark and tokens from different partitions have no ordering.

Process a page under the partition's own durable page barrier. Promote its next token and complete-through watermark only after all work covered by that barrier has a durable terminal outcome and the provider's coverage and finality contract supports the watermark. Use a compare-and-swap on that partition's token, generation, and scope. When a retry loses, reread the winner; do not combine its stale token or watermark with a newer checkpoint.

Derive the combined frontier

1. Fix the set of partitions and its membership epoch. Include every partition that could contain an event at or before the proposed frontier.
2. Read one durable checkpoint per partition from that epoch. For each, establish the latest coordinate through which the provider guarantees complete enumeration and finality and the consumer has completed the corresponding work. The largest event merely observed or applied in a sparse partition is not such a coordinate.
3. If all coordinates share the same order, publish the minimum of those complete-through watermarks as the combined safe frontier. A fast partition cannot carry the combined frontier past a slower one.
4. If the source supplies no comparable complete-through coordinate, expose the checkpoint vector and do not manufacture a scalar frontier from token strings, page counts, or event maxima.
5. Recompute after a partition checkpoint changes. A late retry may leave a partition unchanged or advance it through a valid compare-and-swap; it must never reduce its durable watermark. Publish the combined frontier with its membership epoch so readers can detect a topology change.

A quiet or empty page advances a partition's complete-through watermark only if the provider explicitly gives that response finality semantics. Otherwise the partition continues to hold the combined frontier.

Membership changes

Before adding a partition to an existing frontier, prove that it has complete coverage from the previous published frontier or replay and process its earlier range. Before removing one, prove it is sealed through the cutover coordinate. Use a new membership epoch for the cutover. If either proof is unavailable, withhold a new scalar frontier and retain the vector; silently excluding a lagging or newly discovered partition can turn an old completeness claim into a gap.

Reasoned example

Partition A is complete through coordinate 120, while partition B is complete through 90. A retry of an older A page finishes late, but loses its partition checkpoint comparison. The combined safe frontier remains 90. When B later proves complete through 110, the frontier becomes 110. Taking the maximum, 120, would claim completion of B's unproven interval. These numbers illustrate the rule; no implementation or tests were executed.

Validation boundary

In a real implementation, verify the provider's ordering, pagination, retention, and finality contracts. Exercise page replay, a lost checkpoint response, a stale concurrent retry, an empty page, a stalled partition, and a partition membership change. The minimum rule is justified only by complete-through evidence for every included partition; it does not establish exactly-once effects at an external sink.