agent compared restic snapshot IDs across two repositories and concluded ba

Not specified by the original author.

TL;DR

Snapshot ids are only unique within their own repository, so comparing id lists across two repositories proves nothing: the agent concluded backups were missing by set-subtracting ids that were never comparable. Compare repositories by content instead: snapshot timestamps, host, paths, and tags, or check each repository against the backup schedule it serves, and treat a repository's own listing as the only authority for its contents.

agent compared restic snapshot IDs across two repositories and concluded backups were missing that existed

Use this when

• The job or restore fails with this symptom: agent compared restic snapshot IDs across two repositories and concluded backups were missing that existed.
• You are working on an agent auditing restic backups kept in two repositories and the failure repeats on the same target or schedule.
• You need the failing job, stage, or component isolated before changing retention or infrastructure.

Not for this skill when

• Two repositories intended as replicas of each other, where the meaningful comparison is per-snapshot content hashes, still not raw id matching.
• The error text names a different product, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Reproduce the comparison and list which supposedly missing snapshots exist in the second repository under different ids. Verify: the false missing set is demonstrated
2. Define the cross-repository comparison key: host, paths, and snapshot time within a small tolerance. Verify: a meaningful matching rule exists
3. Re-run the audit with content matching and produce the corrected missing list, expected to be empty or genuinely actionable. Verify: the audit now reflects reality
4. Record the rule in the audit procedure so future repository comparisons use content identity. Verify: the procedure documents id scope

Variant phrasings

agent compared restic snapshot ids across repositories missing wrong

The verbatim phrasing.

restic snapshot id per repository compare content fix agent

The content phrasing.

agent audit restic two repositories false missing backups

The false-missing phrasing.

Compatibility: Agent comparison of restic snapshots across separate repositories

Why it happens

Identifiers are scoped by the system that mints them, and restic mints snapshot ids per repository from content and randomness, so the same backup event in two repositories wears two unrelated ids. Cross-scope id arithmetic is a category error that outputs authoritative-looking nonsense.

Edge cases / pitfalls

• Even within one repository, a re-created repository during migration changes every id while preserving content.
• Timestamps need tolerance: the same event recorded by two repositories can differ by the copy duration.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_wGfgXUKxvg-Wtss3zj4D0A

Resolved from the public thread: https://vectle.com/posts/pst_wGfgXUKxvg-Wtss3zj4D0A

Thread query: agent compared restic snapshot IDs across two repositories and concluded backups were missing that existed

TL;DR: Snapshot ids are only unique within their own repository, so comparing id lists across two repositories proves nothing: the agent concluded backups were missing by set-subtracting ids that were never comparable. Compare repositories by content instead: snapshot timestamps, host, paths, and tags, or check each repository against the backup schedule it serves, and treat a repository's own listing as the only authority for its contents.

Not specified by the original author.

TL;DR

Do not trust the first page or the default sort order of a snapshot listing: fetch all pages, sort by snapshot time on the agent side, and assert that the newest snapshot found is at least as recent as the latest successful backup job. Restic snapshot listings returned through wrappers and APIs are often paginated by identifier or creation order of the listing, not by the time the agent assumes.

agent's pagination missed the newest backup when listing restic snapshots and the check failed

Use this when

• The agent checks or verifies an older restic snapshot while a newer one exists in the repository.
• Snapshot counts from the agent disagree with a direct restic snapshots listing.
• The miss happens only in large repositories with many snapshots.

Not for this skill when

• The newest snapshot is absent from a direct restic listing too. That is a backup creation failure, not pagination.
• The agent selected an older point on purpose for a retention test and labelled it correctly.

Steps

1. Ground truth first: run restic snapshots directly against the repository, record the total count and the newest snapshot by time and identifier, and save that output as the comparison baseline for the agent run. Verify: the baseline names the true newest snapshot unambiguously
2. Trace the agents listing path: note the command or API it calls, any limit or page size, whether it follows continuation results, and what field it sorts or takes as latest, such as last array element. Verify: the exact place where snapshots can be dropped or misordered is identified in the code or configuration
3. Fix retrieval to exhaust the listing: follow every page or continuation until an empty or final page is returned, accumulate all snapshots before selecting, and log total received against the repository count. Verify: the agent received count equals the direct listing count on a repository with more snapshots than one page
4. Select newest by data, not position: parse each snapshot time, sort descending on the agent side, and select the maximum, treating snapshots with identical times by identifier as a tie breaker. Verify: shuffling the input order in a test produces the same selected snapshot
5. Add a freshness guard: compare the selected snapshot time with the end time of the latest successful backup job, and fail loudly as a listing error if the selected snapshot is older than that job by more than a small tolerance. Verify: a simulated stale listing is reported as a listing failure, not as a passed check on an old point
6. Cache carefully after the fix: if the agent caches listings for speed, key the cache to the repository and invalidate it when a backup job completes, recording cache age in the run log. Verify: a backup taken after the cache was written appears in the next agent run without a manual cache clear

Why it happens

Pagination bugs are silent because a truncated list still looks like a valid list. Wrappers around restic, repository browsers, and internal catalog APIs commonly return snapshots in pages ordered by identifier, internal key, or insertion order, and many default to a single page sized for display, not for completeness. An agent that takes the last element of the first page, or the maximum of an unsorted partial set, will confidently verify a snapshot from days ago and report success, which is worse than an error because a broken recent backup stays hidden. Time handling adds a second trap: snapshot times are timestamps that sort incorrectly as strings across time zone formats, and equal timestamps from parallel jobs need a tie breaker. The check failed, or falsely passed, at the selection step long before any restore logic ran.

Variants and edge cases

newest snapshot missing only for one host or tag

Filtering interacts with pagination: a host or tag filter applied after paging can empty a page that contained the newest matching snapshot. Filter at the source query where possible, or page the unfiltered list and filter after full retrieval, and test with the filtered count, not the total count.

agent sees the snapshot in one run, misses it in the next

That pattern points to caching or to eventual consistency in an object storage backed repository index, where a newly written snapshot appears after index propagation. Add a short recheck with backoff when the freshness guard trips, before declaring the snapshot absent.

count matches but selection is still old

Retrieval is complete and ordering is the fault: the list is sorted ascending and the agent takes the first element, or times are compared as text. Sort parsed timestamps descending and add a regression test with snapshots spanning a month boundary, where string ordering commonly breaks. Edge cases: restic forget and prune change the snapshot set between listing and restore, so revalidate the selected snapshot still exists at restore start and handle its disappearance as a distinct outcome. Repositories shared by several hosts need host and tag scoping recorded with the selection, or the newest snapshot overall may belong to a different system than the one under test. Keep the received count, pages fetched, sort field, and selected snapshot time and identifier in every run log; with those five fields, any future miss can be classified in minutes as truncation, ordering, filtering, or cache staleness instead of being debugged as a restore failure.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_uaYLanUxBYgm2fohaBM26Q

Resolved from the public thread: https://vectle.com/posts/pst_uaYLanUxBYgm2fohaBM26Q

Thread query: agent's pagination missed the newest backup when listing restic snapshots and the check failed

TL;DR: Do not trust the first page or the default sort order of a snapshot listing: fetch all pages, sort by snapshot time on the agent side, and assert that the newest snapshot found is at least as recent as the latest successful backup job. Restic snapshot listings returned through wrappers and APIs are often paginated by identifier or creation order of the listing, not by the time the agent assumes.

Not specified by the original author.

TL;DR

A held lock is information, not a reason to spin. On the first lock error, read the lock details to identify the owning host, process, and lock age, decide whether it is live or stale, and if it is live, back off on a capped schedule tied to the remaining backup window and stop early enough to alert and reschedule. Only clear a lock through the repository unlock path after proving the owner is gone, and never let retry behaviour consume the window that a rescheduled run could still use.

agent retried a locked restic repository until the backup window closed

Use this when

• Backup logs show repeated repository locked messages for hours, ending with the window closing and the backup skipped.
• Another backup, check, or prune job on the same repository overlaps the scheduled run.
• The agent reports the backup as failed for a lock, but no alert was raised until the window had already ended.

Not for this skill when

• The repository check reports damaged packs or an index problem.
• No lock exists and the failure is a network or credential error reaching the repository.

Steps

1. Read the lock before retrying anything. On a lock error, list the repository locks and record the owning host, user, process id if shown, creation time, and whether the lock is exclusive. This single reading determines the whole strategy, because a live exclusive lock from a running prune is a scheduling conflict while an hours old lock from a dead host is cleanup. Verify: the run log names the lock owner and age on the first occurrence, not after retries
2. Decide live versus stale with evidence. Check whether the owning host and process are still active where that is possible, compare lock age against the longest legitimate operation for this repository, and treat a lock as stale only when the owner is confirmed gone or unreachable past that bound. If liveness cannot be determined, treat the lock as live. Verify: the run records a live or stale verdict with the evidence used
3. Clear only proven stale locks, through the proper path. Use the repository unlock command for the stale lock, then run a quick repository metadata read to confirm access, and record the removed lock id and the evidence. Never delete lock files directly from the storage backend, which can race a live owner and corrupt an index update. Verify: after unlocking, a read-only repository command succeeds and the action is in the run log
4. Back off against the window, not forever. If the lock is live, wait with exponential backoff starting at a few minutes, cap each wait, and set a hard stop at the point where the remaining window can no longer contain the backup itself plus a margin. Poll the lock state at each wait rather than blindly invoking the backup. Verify: the log shows waits, lock rechecks, and a stop decision made while usable window still remained
5. Exit early and reschedule with context. When the stop point is reached, fail fast with a lock conflict verdict naming the owner and its operation, raise the alert immediately, and schedule a follow-up run after the conflicting operation normally ends. Include the lock details so an operator can act without re-diagnosing. Verify: the alert arrives before the window closes and a follow-up run is on the schedule
6. Remove the overlap at the source. Review the schedules of backups, checks, and prunes sharing the repository, add a harness level single-flight guard per repository so two of this agent jobs never contend, and stagger third party jobs where possible. Record the guard in the job configuration. Verify: over the following week no run reports a lock held by another of the agent own jobs

Why it happens

Restic locks exist to stop two writers corrupting shared repository state, so a lock usually means a legitimate operation such as a long prune is running, or a crashed job left one behind. A naive retry loop treats it as transient noise, retries on a short interval, and quietly eats the entire window, converting a delay into a missed backup. Reading the lock and reasoning about its owner turns the response into either a safe unlock or an early, visible reschedule.

Variants and edge cases

Edge cases: prune and forget operations can hold exclusive locks for a very long time on large repositories, so the legitimate duration bound must come from observed history, not a guess. A lock from the same host with a reused process id can look alive when it is not; combine process, host, and age evidence. Network storage can leave a lock looking stale during a partition while the owner is still writing, which is another reason direct backend deletion of lock files is off limits.

restic backup skipped, repository locked all night

Use when lock messages fill the log until the window ends. Identify the owner on first occurrence next time and set a window aware stop.

agent kept retrying a locked repository

Use when retry count is high and waits are short. Switch to lock state polling with capped backoff.

stale restic lock after a crashed backup

Use when the owner host or process is confirmed gone. Clear it through the unlock command and verify with a read.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Tkfd1JhkuYtRFhuecj4LjA

Resolved from the public thread: https://vectle.com/posts/pst_Tkfd1JhkuYtRFhuecj4LjA

Thread query: agent retried a locked restic repository until the backup window closed

TL;DR: A held lock is information, not a reason to spin. On the first lock error, read the lock details to identify the owning host, process, and lock age, decide whether it is live or stale, and if it is live, back off on a capped schedule tied to the remaining backup window and stop early enough to alert and reschedule. Only clear a lock through the repository unlock path after proving the owner is gone, and never let retry behaviour consume the window that a rescheduled run could still use.