agent hit context overflow on Okta System Log bulk import, dropped a week
Fixes license agents that drop log data when the Okta System Log bulk import overflows context. Use when imports silently drop time ranges. Not for pagination errors.
TL;DR
Import in small time-bounded chunks with per-chunk checkpoints. One giant import overflows the context; chunked imports with resume do not.
Error
(import covered 3 of 4 weeks; one week silently dropped after context overflow; no error text)Steps
- Confirm the gap: compare the imported time range against the requested range. Expected: the dropped window identified.
- Split the import into small windows (hours or a day each, tuned to fit context) and process them sequentially. Expected: each chunk fits.
- Checkpoint after each chunk: record the completed watermark durably. Expected: resume from the last completed chunk.
- Re-run the import. Expected: full coverage with no drops.
When to use
- Bulk log imports silently drop time ranges
- The agent processes the whole range in one shot
- Context limits are hit mid-import
When not to use
- The API paginates with errors (a pagination problem)
- The log genuinely lacks events for the window (a source gap)
- The import is small (chunking still harmless)
Tool compatibility
Okta System Log API with any agent runtime.
Variant phrasings
overflow on a single huge day
Split hot days into hourly chunks.
checkpoint lost on agent restart
Write checkpoints to disk, not memory.
Why it happens
Context is finite and log volume is not. Agents that pull "everything" in one request hit the limit partway, and without chunking the remainder is silently lost.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_m80R3jwtHlJt20gTogiH8A