Derive a global frontier from retried paginated partitions

When to use

Use this procedure when several independently paginated event partitions feed a downstream consumer that needs one statement such as "all eligible events through boundary B have durable outcomes." Each partition has its own opaque page token and can be retried independently. The source must provide a common, comparable boundary and a way to certify that a partition has enumerated every eligible event through that boundary. Stable partition membership for an epoch, stable event identities, and replayable retention are also required.

A page token, the largest event value seen, and a timestamp on the last returned event are not completeness certificates. If the source cannot prove that no earlier event remains unseen or can later appear behind the boundary, do not publish an exact global frontier.

Record partition progress separately

For each partition and membership epoch, persist:

• The bound source scope, filters, ordering, and inclusive or exclusive boundary convention.
• The durable resume token for page fetching and a generation that fences stale retries.
• A committed event prefix with receipts or equivalent idempotent outcomes.
• A certified complete-through boundary in the common comparison domain.

Process each page in source order. Keep its next token pending until every eligible event in that page has a durable terminal outcome. Promote the token with a generation check. A retry may replay events; receipts prevent duplicate effects. Advance a partition's complete-through boundary only when both facts are durable: the source certifies full enumeration through that boundary, including empty ranges, and the consumer has completed every eligible event in that enumerated prefix. A failed event or an unproven gap holds that partition's boundary.

Publish the aggregate

1. Freeze the partition set under a membership epoch. Read a durable complete-through boundary for every member of that epoch.
2. If any member lacks a certified boundary or has an unresolved event before the proposed bound, leave global progress unchanged.
3. Compute the candidate global frontier as the minimum of the member boundaries. Never compare opaque page tokens or use the maximum of observed events.
4. Persist the candidate only if it is greater than the existing frontier and the membership epoch still matches. A stale aggregator rereads the partition state and recomputes. If aggregation crashes before publishing, the partition evidence remains durable and the candidate can be recomputed. If it crashes after publishing, the published frontier is backed by already durable partition evidence.
5. Keep partition token recovery local to that partition. A retry or expired token in one partition must not lower another partition's progress or advance the aggregate while its own prefix is incomplete.

A source boundary that uses time needs explicit treatment of events tied at the boundary. Use an inclusive fence that proves all events at that time are enumerated, or use a stable ordering tuple with a unique tie breaker. The same filter and boundary semantics must hold for every member.

Membership changes

A new partition cannot join a published frontier merely because its first page is empty or its first observed event is later. Before joining the current epoch, prove and durably complete its history through the already published frontier. If that proof or replay is unavailable, establish a new membership epoch with an explicit rebuild or correction process; do not silently claim the old frontier covered the new partition. Remove a partition only under an explicit epoch change after its outstanding range has been resolved according to the downstream contract.

Reasoned retry example

Suppose two partitions have certified complete-through boundaries 120 and 100, so the global frontier is 100. A retry on the second partition fetches a page containing events through 140, but the event at 110 fails. The second boundary remains 100 and the global frontier remains 100. After every event through 130 on that partition has a durable outcome and the source certifies that range complete, its boundary becomes 130. The aggregate may then advance to the minimum of 120 and 130, which is 120. This is a reasoned example, not an executed test.

Limits and validation

This procedure depends on source-certified completeness in a shared domain. A monotonic per-partition event order alone cannot prove a global event-time frontier when late arrivals or mutable filter membership can introduce earlier eligible events. At-least-once page replay still requires idempotent effects or transactional receipts.

No implementation was inspected and no tests were executed for this guidance. In an implementation, test a crash before and after each page barrier, a retry that loses its generation check, an empty certified range, tied boundary values, one stalled partition, and a partition joining behind the published frontier.