agent's okta system log api returned events out of order: dedupe by event id

TL;DR

Never trust the order Okta returns log events in; sort and dedupe by the event's unique id first. Collect each page of results, drop events whose id you have already processed, sort the rest by published timestamp, and only then trigger ticket actions.

Steps

1. Add a processed-event store keyed by the Okta event id, persisted across runs. Expected: Re-polled pages return the same ids and are recognized as already handled.

2. Sort each batch by the published timestamp before acting on any event. Expected: Actions fire in true chronological order.

3. Process the sorted, deduped batch and record the newest id as the watermark. Expected: No event is acted on twice and none is skipped.

4. Replay a known out-of-order window in a test to confirm ordering holds. Expected: The test shows each event handled exactly once, in order.

When to use

• The agent polls the Okta system log on a schedule
• Duplicate tickets appear from the same Okta event
• Actions fire in the wrong order after a busy period

When not to use

• The agent uses Okta event hooks with delivery guarantees (different path)
• Events are only read for dashboards, never acted on
• The log volume is tiny and ordering never mattered

Why this happens

The system log API paginates and backfills, so a poll can return the same event twice or deliver newer events before older ones. Agents that act on arrival order therefore double-fire or mis-order, and only id-keyed dedupe plus timestamp sorting fixes it.

Edge cases

• Clock skew between Okta and the agent makes published timestamps slightly unreliable; the event id is the only safe dedupe key.
• Very high-volume tenants need the watermark persisted, or a restart replays the whole window.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_8hG2B2SaiSeHmw4BTHK70Q