agent's log parser failed on rotated backup logs and missed the real failure

TL;DR

Rotated logs break agents that read only the live file: the failure lines sit in a compressed archive while the agent reads a fresh, clean log and reports success. Point collection at the whole log set for the run window, decompress archives in the pipeline, order records by timestamp across files, and alert when expected log volume is suspiciously low for the window.

agent's log parser failed on rotated backup logs and missed the real failure

Use this when

• Agent reports success for nights when the backup console shows failures.
• The missing failures are found manually in rotated or compressed logs.
• Failures near midnight, the common rotation boundary, are missed most.

Not for this skill when

• The parser breaks inside one file on multiline errors. Record parsing problem.
• Backups genuinely wrote no logs because the job never ran. Scheduling problem the parser should flag differently.

Steps

1. Map the log directory rotation scheme: naming, compression, retention and rotation triggers. Verify: the full log set layout is documented
2. Change collection to gather every file covering the run window including compressed archives, in timestamp order. Verify: a test failure seeded in an archive is collected
3. Add a completeness check comparing collected job identifiers with the product job list for the window. Verify: missing jobs trigger a collection alert
4. Update the parser pipeline to run on the merged stream. Verify: triage results match a manual review of the full set
5. Re-run triage for the past week and file any newly found failures. Verify: previously missed failures are now reported

Variant phrasings

rotated logs agent missed failure compressed

The rotation phrasing.

log rotation boundary backup triage incomplete

The boundary variant.

agent read wrong log file after rotation

The plain wording.

Compatibility: Agents triaging Linux logrotate and Windows rollover log sets for backup products, with gzip and zip archive handling.

Why it happens

Rotation is invisible to a reader who opens the canonical filename, because the name now points at new content while history moved sideways into archives. Agents built and tested during the day against a single file inherit this blind spot exactly where overnight failures live. Window-based set collection is the only version that matches how logs actually exist.

Edge cases / pitfalls

• Retention shorter than your triage lag deletes evidence. Align log retention with the slowest review cycle.
• A suddenly tiny current log at triage time is itself a signal worth alerting on.
• Compressed archives need disk headroom to process. Stream decompression in the pipeline instead of extracting whole sets.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ytZObbEpIiiqqKwkcs8NEQ