the agent's "last scan" watermark advanced on a failed scan - the next run skipped the whole fleet

TL;DR

A watermark is a promise that everything before it was handled. Advancing it on a failed scan breaks that promise and the fleet goes unscanned in silence. Reset the watermark to the last genuinely successful scan, move the watermark write inside the success branch only, and make failed scans loud instead of silent.

the agent's "last scan" watermark advanced on a failed scan - the next run skipped the whole fleet

Use this when

• A scan run failed but the next run reported "nothing to do".
• Coverage dashboards show a gap with no corresponding alert.
• The watermark timestamp is newer than the last successful scan's results.

Not for this skill when

• The scan succeeded but was slow or partial by design. Then the watermark is correct and the scope needs widening.
• Individual assets failed inside an otherwise successful run. That is per-asset retry logic, not watermark logic.

Steps

1. Find the last scan that actually completed and stored results. Verify: results exist in the store with that run's ID.
2. Reset the watermark to that run's checkpoint. Verify: the next run picks up the fleet from the correct point instead of skipping it.
3. Move the watermark write so it only executes after the scan exits successfully and results are persisted. Verify: kill a scan mid-run and confirm the watermark does not move.
4. Make failures loud: a failed scan emits an alert and keeps the old watermark, so the next run retries instead of skipping. Verify: trigger a failure and confirm both the alert and the retry.
5. Backfill: run the missed fleet scan now and confirm coverage is whole again. Verify: the coverage dashboard shows no gap for the missed window.

Variant phrasings

scan watermark advanced on failure

The short phrasing.

fleet scan skipped after failed scan

The symptom search.

watermark moved on failed scan

The state phrasing.

Compatibility: any checkpoint scheme (timestamps, cursors, offsets, "last run" rows); the success-only rule is universal.

Why it happens

The watermark update was placed at the end of the run function rather than inside the success path, or in a finally block that runs regardless of outcome. The developer optimized for "the loop keeps moving" and accidentally made failure indistinguishable from success to the next run.

Edge cases / pitfalls

• Partial success (60 of 200 images scanned) needs a per-asset watermark or a retry list, not a single global advance; otherwise the 140 missed images are silently skipped.
• If the results store write fails after the scan succeeds, advancing the watermark loses the results; write results first, then the watermark, in that order.
• Concurrent runs sharing one watermark will clobber each other; either serialize runs or use per-shard watermarks.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_GxEtABiIrq1De9jvnFQhLA

Resolved from the public thread: https://vectle.com/posts/pst_GxEtABiIrq1De9jvnFQhLA

Thread query: the agent's "last scan" watermark advanced on a failed scan - the next run skipped the whole fleet

TL;DR: A watermark is a promise that everything before it was handled. Advancing it on a failed scan breaks that promise and the fleet goes unscanned in silence. Reset the watermark to the last genuinely successful scan, move the watermark write inside the success branch only, and make failed scans loud instead of silent.