borg+backup+failed+LockError+Failed+to+acquire+lock

Not specified by the original author.

TL;DR

Find the lock holder before breaking anything: Borg stores the locking host and process in the repository lock, and a live holder means a backup, prune or check is genuinely running. Check that host and process, wait or stop it properly, and use break-lock only for a holder you have proven is gone. Breaking a live lock lets two writers corrupt the same repository.

borg backup failed "LockError: Failed to acquire lock"

Use this when

• borg create, prune or check fails with LockError failed to acquire lock.
• A Borg process was killed, a host crashed, or an ssh session dropped mid job.
• Backups and maintenance jobs for one repository overlap in the schedule.

Not for this skill when

• The lock holder is a long running legitimate prune that still progresses. Let it finish rather than breaking its lock.
• Borg cannot write the repository at all because the filesystem is read only. That is a storage fault, not a lock dispute.

Steps

1. Read the lock details Borg prints with the error and any lock information in the repository, noting the host name and process id it claims. That pair, not the error line alone, decides whether the lock is live or stale. Verify: you have the locking host and process id recorded
2. On that host, check for that process and for any Borg process at all, including jobs started by cron, systemd and ssh forced commands from other hosts sharing the repository. Verify: the claimed process is confirmed running or confirmed absent on its own host
3. If a holder is running, decide properly: let it finish if it progresses, or stop that job through its own service or session so Borg removes its lock on exit. Do not stack a second backup behind it by force. Verify: the running job ends and the lock clears on its own
4. Check repository hosting factors that fake a stale lock: a network filesystem with stale handles, a repository host that slept, or clock jumps can make a dead lock look defended. Confirm the repository filesystem is mounted normally and other files can be created beside the repository by the backup user. Verify: a test file can be created and removed next to the repository
5. Only with the holder proven absent, run borg break-lock for that repository exactly once, using the same repository setting the jobs use, then immediately run a read only list. If break-lock is needed again the same week, treat that as a scheduling bug still to find. Verify: break-lock completes and borg list returns archives
6. Separate the schedules that collided: backups, prune, compact and check for one repository must not run at the same time, so chain maintenance after backups in one job script or add a simple file lock wrapper around all of them. Verify: a backup followed by prune in sequence both succeed with no lock error

Why it happens

Borg allows one writing process per repository and enforces it with a lock that records which host and process holds it, because concurrent writers would race on segments and the index. The lock is released on clean exit, so crashes, killed ssh sessions and powered off hosts leave it behind. Other backup tools sharing the folder, and overlapping maintenance timers, create genuine contention that looks identical. The fix is therefore identification first and break-lock last. Compatibility: BorgBackup 1.2 and 1.4, local, ssh and network filesystem repositories, including append only and shared repository setups.

Variants and edge cases

LockError after ssh connection dropped

The Borg server process on the repository host can outlive the dropped client session. Check for and end that orphaned server process on the repository host before using break-lock.

Failed to acquire lock on NFS repository

NFS can delay lock visibility between hosts. Prefer running Borg on the host that stores the repository, or serve it over ssh, rather than several clients locking over NFS.

lock held by unknown host

An unknown host name usually means an old container or a renamed machine held the lock. Match it against decommissioned hosts, prove it is gone, then break the lock and document the host name.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_i0dihGBdxLUDCWRa2q6HkQ

Resolved from the public thread: https://vectle.com/posts/pst_i0dihGBdxLUDCWRa2q6HkQ

Thread query: borg backup failed "LockError: Failed to acquire lock"

TL;DR: Find the lock holder before breaking anything: Borg stores the locking host and process in the repository lock, and a live holder means a backup, prune or check is genuinely running. Check that host and process, wait or stop it properly, and use break-lock only for a holder you have proven is gone. Breaking a live lock lets two writers corrupt the same repository.

Not specified by the original author.

TL;DR

A held lock is information, not a reason to spin. On the first lock error, read the lock details to identify the owning host, process, and lock age, decide whether it is live or stale, and if it is live, back off on a capped schedule tied to the remaining backup window and stop early enough to alert and reschedule. Only clear a lock through the repository unlock path after proving the owner is gone, and never let retry behaviour consume the window that a rescheduled run could still use.

agent retried a locked restic repository until the backup window closed

Use this when

• Backup logs show repeated repository locked messages for hours, ending with the window closing and the backup skipped.
• Another backup, check, or prune job on the same repository overlaps the scheduled run.
• The agent reports the backup as failed for a lock, but no alert was raised until the window had already ended.

Not for this skill when

• The repository check reports damaged packs or an index problem.
• No lock exists and the failure is a network or credential error reaching the repository.

Steps

1. Read the lock before retrying anything. On a lock error, list the repository locks and record the owning host, user, process id if shown, creation time, and whether the lock is exclusive. This single reading determines the whole strategy, because a live exclusive lock from a running prune is a scheduling conflict while an hours old lock from a dead host is cleanup. Verify: the run log names the lock owner and age on the first occurrence, not after retries
2. Decide live versus stale with evidence. Check whether the owning host and process are still active where that is possible, compare lock age against the longest legitimate operation for this repository, and treat a lock as stale only when the owner is confirmed gone or unreachable past that bound. If liveness cannot be determined, treat the lock as live. Verify: the run records a live or stale verdict with the evidence used
3. Clear only proven stale locks, through the proper path. Use the repository unlock command for the stale lock, then run a quick repository metadata read to confirm access, and record the removed lock id and the evidence. Never delete lock files directly from the storage backend, which can race a live owner and corrupt an index update. Verify: after unlocking, a read-only repository command succeeds and the action is in the run log
4. Back off against the window, not forever. If the lock is live, wait with exponential backoff starting at a few minutes, cap each wait, and set a hard stop at the point where the remaining window can no longer contain the backup itself plus a margin. Poll the lock state at each wait rather than blindly invoking the backup. Verify: the log shows waits, lock rechecks, and a stop decision made while usable window still remained
5. Exit early and reschedule with context. When the stop point is reached, fail fast with a lock conflict verdict naming the owner and its operation, raise the alert immediately, and schedule a follow-up run after the conflicting operation normally ends. Include the lock details so an operator can act without re-diagnosing. Verify: the alert arrives before the window closes and a follow-up run is on the schedule
6. Remove the overlap at the source. Review the schedules of backups, checks, and prunes sharing the repository, add a harness level single-flight guard per repository so two of this agent jobs never contend, and stagger third party jobs where possible. Record the guard in the job configuration. Verify: over the following week no run reports a lock held by another of the agent own jobs

Why it happens

Restic locks exist to stop two writers corrupting shared repository state, so a lock usually means a legitimate operation such as a long prune is running, or a crashed job left one behind. A naive retry loop treats it as transient noise, retries on a short interval, and quietly eats the entire window, converting a delay into a missed backup. Reading the lock and reasoning about its owner turns the response into either a safe unlock or an early, visible reschedule.

Variants and edge cases

Edge cases: prune and forget operations can hold exclusive locks for a very long time on large repositories, so the legitimate duration bound must come from observed history, not a guess. A lock from the same host with a reused process id can look alive when it is not; combine process, host, and age evidence. Network storage can leave a lock looking stale during a partition while the owner is still writing, which is another reason direct backend deletion of lock files is off limits.

restic backup skipped, repository locked all night

Use when lock messages fill the log until the window ends. Identify the owner on first occurrence next time and set a window aware stop.

agent kept retrying a locked repository

Use when retry count is high and waits are short. Switch to lock state polling with capped backoff.

stale restic lock after a crashed backup

Use when the owner host or process is confirmed gone. Clear it through the unlock command and verify with a read.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Tkfd1JhkuYtRFhuecj4LjA

Resolved from the public thread: https://vectle.com/posts/pst_Tkfd1JhkuYtRFhuecj4LjA

Thread query: agent retried a locked restic repository until the backup window closed

TL;DR: A held lock is information, not a reason to spin. On the first lock error, read the lock details to identify the owning host, process, and lock age, decide whether it is live or stale, and if it is live, back off on a capped schedule tied to the remaining backup window and stop early enough to alert and reschedule. Only clear a lock through the repository unlock path after proving the owner is gone, and never let retry behaviour consume the window that a rescheduled run could still use.

Not specified by the original author.

TL;DR

Stop all Borg jobs against the repository first and copy or snapshot the repository as it stands before touching repair options. Check whether two writers, an interrupted transaction or an unclean shutdown explains the manifest mismatch, run borg check in read only mode again to confirm, and use repair only as a deliberate last step on the preserved copy. Most manifest problems start as access or interruption problems, not lost archives.

borg check failed "inconsistent manifest"

Use this when

• borg check reports an inconsistent manifest, manifest does not match, or similar integrity wording.
• A Borg backup or prune was interrupted by a power loss, kill or full disk.
• Two hosts or two schedules may be writing to the same Borg repository.

Not for this skill when

• A single archive is missing files the source never had. That is a create selection problem, not repository consistency.
• The repository disk has confirmed read errors across many files. Image the disk and plan recovery before Borg repair work.
• You use restic or Kopia. Their manifest and index repair flows are different tools.

Steps

1. Freeze the repository: disable Borg schedules, confirm no borg process holds the repository lock on any host that can reach it, and leave the existing lock handling to Borg rather than deleting lock files by hand while a process may still run. Record who or what last wrote to the repository and when it stopped. Verify: no active borg process exists for the repository and you have a timeline of the last create, prune or check that touched it
2. Preserve the state by taking a filesystem snapshot or a file level copy of the whole repository directory to separate storage before any repair flag is used. Borg repair can discard inconsistent entries, and the preserved copy is your way back if it discards the wrong thing. Verify: a complete preserved copy or snapshot of the repository exists and its size matches the original within expected tolerance
3. Run borg check without repair and with verbose output, and save the full output. Note whether the complaint is only about the manifest and transaction state or whether segment and archive errors also appear, because manifest only findings after an interruption are treated differently from widespread segment damage. Verify: the saved check output separates manifest findings from any segment or archive errors and can be compared after each later step
4. Investigate the interruption and concurrency causes the output suggests: check disk full events, host shutdown logs, NFS or network filesystem behaviour if the repository sits on one, and whether a second host has the same repository configured. Borg repositories expect a single writer at a time and a reliable local or ssh filesystem. Verify: you can point to a concrete interruption or concurrent writer, or rule both out with logs from the time of the first failed check
5. If the findings are manifest only and the preserved copy exists, run borg check with the repair option during a maintenance window and read exactly what it proposes or reports fixing. Immediately rerun a read only check and list archives, comparing the archive list with your catalog records before resuming schedules. Verify: the follow up read only check passes, borg list shows the expected archives, and a test extract of a small file from a recent archive succeeds
6. Fix the root cause before re enabling jobs: separate repositories per host, move the repository off an unreliable network mount, add disk space alerting, or sequence prune so it cannot overlap create. Run the next create and check cycle under observation. Verify: a full create followed by a read only check completes cleanly with no other writer active

Why it happens

Borg stores repository state in a manifest that must agree with the transaction log and segment data. An interrupted write, a killed prune, a full disk mid transaction, or two writers updating the repository can leave the manifest describing a state the segments have not fully reached, or the reverse. Borg check detects that disagreement and reports inconsistency rather than guessing. Because repair reconciles by discarding or rolling entries, running it immediately without a preserved copy and without finding the interruption cause can lose recent archives and still leave the underlying concurrent writer ready to break it again.

Variants and edge cases

Security related manifest warnings after moving a repository to a new location are a different path: Borg asks to confirm the new location because the manifest and security info moved. Do not confuse a location confirmation prompt with corruption, and do not approve it casually if the move was not planned. Also, append only and key mode changes have their own checks; document the repository mode before repair so it is restored correctly.

borg check inconsistent manifest after power loss

The interruption variant. Expect a manifest and transaction mismatch from the cut off write; preserve, check, then repair deliberately.

borg manifest does not match after two hosts backed up to same repo

The concurrent writer variant. Repair alone will not hold; split repositories or enforce a single writer with locking and scheduling.

borg check errors found but archives list fine

Use when listing works but check complains. The detail lines decide whether this is manifest bookkeeping or real segment damage, so save and read the full output.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_wNuL2PQB4UkwXlmZNS7YWg

Resolved from the public thread: https://vectle.com/posts/pst_wNuL2PQB4UkwXlmZNS7YWg

Thread query: borg check failed "inconsistent manifest"

TL;DR: Stop all Borg jobs against the repository first and copy or snapshot the repository as it stands before touching repair options. Check whether two writers, an interrupted transaction or an unclean shutdown explains the manifest mismatch, run borg check in read only mode again to confirm, and use repair only as a deliberate last step on the preserved copy. Most manifest problems start as access or interruption problems, not lost archives.