Monotonic settlement for shared durable operation records

When to use

Multiple client retries share one durable operation record keyed by an idempotency key. Attempts can finish in either order: one may observe committed success while another later times out. Naive last-write-wins settlement can overwrite completed with unknown and discard the stored result.

A further failure appears when a late success carries the correct operation identifier but a different request fingerprint or authorization binding generation. Monotonic merge must therefore be locked: success is absorbing only for the exact reserved request and authorization binding.

Core rule

Separate canonical operation outcome from per-attempt observations.

• The canonical record holds the authoritative status and immutable committed result.
• Each attempt may log its own terminal observation, but attempt logs must not blindly overwrite the canonical record.

Settlement is locked monotonic merge, not last-write-wins and not success-at-any-cost.

Locked reservation (claim phase)

On first claim of the operation identifier, atomically record a reservation lock:

• operation_id — idempotency key scoped to tenant or actor
• request_fingerprint — hash of effect-defining fields (amount, destination, mutation kind, etc.)
• authorization_binding — principal, tenant, scopes, and a binding generation (token epoch, session version, or delegation revision)

Rules at claim time:

• Same operation id plus matching fingerprint and binding: join the in-flight or settled operation (retry path).
• Same operation id plus mismatch: reject with conflict; never execute or settle under that reservation.
• The lock is immutable for the life of the record.

Without this lock, monotonic success allows cross-intent or cross-authorization poisoning under one key.

Status lattice

Assign each status an ordinal rank (higher wins on conflict within the same lock):

1. pending / running — in flight
2. unknown — caller lost the response; effect may or may not have committed
3. conflict — incompatible fingerprint or binding observed (terminal for that mismatch)
4. failed — committed failure for the reserved request (terminal)
5. completed — committed success for the reserved request with stored result (terminal, absorbing for that lock)

Rules:

• Never downgrade from completed or failed because of a weaker observation on the same lock.
• Upgrade to completed only if the success observation matches request_fingerprint and authorization_binding on the record.
• Store the result atomically with the transition to completed; treat it as immutable afterward.
• A late success with correct operation id but wrong fingerprint or binding must not upgrade canonical state; record conflict or reject the write.

Why blindly preferring every success is unsafe

Operation identifiers are reused by clients and do not alone define intent. Preferring any late success lets:

• Cross-intent overwrite: retry after a client bug sends a different payload under the same key; late success for the wrong payload settles as canonical.
• Cross-authorization bleed: success under a stale token epoch or changed delegation settles after reservation under a new binding generation.
• False reconciliation: a timed-out caller believes operation A is unknown while canonical record shows success for operation B under the same id.
• Replay confusion: duplicate external effects may exist, but only the reserved fingerprint's outcome may be replayed to callers.

Success is strong evidence only when tied to the reserved lock, not globally.

Settlement writes (binding-gated)

Before rank-based merge, verify the lock match:

if observation.fingerprint != record.reserved_fingerprint: reject or set conflict
if observation.binding != record.reserved_binding: reject or set conflict

Then apply monotonic merge:

• Write unknown only if current rank is below unknown and the attempt matched the lock at claim.
• Write completed plus result only if lock matches and current status is not already completed.
• If already completed, ignore duplicate matching success or late timeout writes.
• Ignore or conflict on success observations that fail the lock check, regardless of timestamp.

Order A: success first, timeout second (matching lock)

1. Attempt A commits reserved effect; writes completed plus result.
2. Attempt B times out; tries to write unknown.
3. Guard rejects downgrade; record stays completed.
4. Attempt B reads canonical record and replays the stored result.

Order B: timeout first, success second (matching lock)

1. Attempt B times out; writes unknown while not yet completed.
2. Attempt A commits; writes completed plus result (lock matched).
3. Upgrade succeeds; callers see success for the reserved request.

Order C: late success with mismatched fingerprint or binding

1. Record reserved for fingerprint F1 and binding B1.
2. Attempt times out; canonical may be unknown.
3. Late success arrives for same operation id but fingerprint F2 or binding B2.
4. Settlement must not upgrade to completed; set conflict or reject. Canonical stays unknown or moves to conflict.
5. Callers with F1 must not receive F2's result. Operators investigate duplicate effect risk separately.

Client contract

• Timeout means response unknown, not failure.
• Retry with the same operation id, fingerprint, and authorization binding.
• A new logical operation or changed intent requires a new operation id.
• After re-authentication that bumps binding generation, treat prior in-flight attempts as stale; do not expect their success to settle the new binding.

What to verify in tests (not design reasoning)

• Matching lock: success then timeout leaves completed unchanged.
• Matching lock: timeout then success ends at completed with result present.
• Mismatched fingerprint: late success does not upgrade canonical state or replace result.
• Mismatched binding generation: late success rejected; no cross-authorization replay.
• Same operation id with different fingerprint at claim: second claimant gets conflict without executing.
• Result field never cleared after first successful commit for the reserved lock.