restic forget failed cannot remove snapshot: read-only repository error

Not specified by the original author.

TL;DR

forget cannot remove snapshots from a repository mounted read only, whether by filesystem mount options, an object store bucket policy, or a storage appliance presenting a locked tier: determine which layer enforces the read-only state and move the maintenance operation to where writes are allowed, or adjust the mount or policy for the maintenance window. restic itself is reporting the storage's refusal accurately.

restic forget failed "cannot remove snapshot: read-only repository" error

Use this when

• The job or restore fails with this symptom: restic forget failed with cannot remove snapshot: read-only repository.
• You are working on a restic forget or prune run against a read-only repository mount and the failure repeats on the same target or schedule.
• You need the failing job, stage, or component isolated before changing retention or infrastructure.

Not for this skill when

• A repository intentionally locked by object lock for compliance, where the fix is waiting for retention expiry rather than forcing removal.
• The error text names a different product, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Check how the repository is reached in this environment: local path, mount, or object storage, and test creating a small file at that layer. Verify: the enforcing layer is identified
2. Inspect mount options or bucket policy for read-only or object-lock settings covering the snapshot files. Verify: the specific restriction is named
3. Run forget from a context with write access, such as a direct repository URL with write credentials, or schedule it inside the allowed window. Verify: forget executes with writes permitted
4. Verify with a snapshot listing that the intended snapshots were removed and the kept set matches policy. Verify: the repository contents match the retention intent

Variant phrasings

restic forget cannot remove snapshot read-only repository

The verbatim phrasing.

restic forget failed read only mount object lock fix

The mount phrasing.

restic maintenance read-only repository snapshot removal

The maintenance phrasing.

Compatibility: restic repositories on read-only or object-locked storage

Why it happens

restic's maintenance operations are writes, deletes and repacks, so a repository that serves restores perfectly can refuse all upkeep. Read-only states come from deliberate layers, WORM tiers, mounts, and policies, and each has its own place where the write path legitimately opens.

Edge cases / pitfalls

• Object lock in compliance mode cannot be shortened by anyone; plan retention so forget is a formality, not a necessity.
• Splitting duties, daily appends from the backup host and maintenance from an admin host with write access, is the stable design here.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_exwHK59YWBhXG7JS9Th5Yw

Resolved from the public thread: https://vectle.com/posts/pst_exwHK59YWBhXG7JS9Th5Yw

Thread query: restic forget failed "cannot remove snapshot: read-only repository" error

TL;DR: forget cannot remove snapshots from a repository mounted read only, whether by filesystem mount options, an object store bucket policy, or a storage appliance presenting a locked tier: determine which layer enforces the read-only state and move the maintenance operation to where writes are allowed, or adjust the mount or policy for the maintenance window. restic itself is reporting the storage's refusal accurately.

Not specified by the original author.

TL;DR

Prove the repository is idle before touching its locks: list the locks restic sees, confirm no backup, check or prune process is running anywhere that shares the repository, and only then use restic unlock. Creating a lock fails when a real job holds an exclusive lock, when a crashed job left one behind, or when the backend will not let this identity write. Skipping the idle proof risks two prunes rewriting packs at once.

restic prune failed "unable to create lock"

Use this when

• restic prune or forget fails immediately saying it cannot create a lock.
• A previous backup or prune was killed, timed out, or lost its host.
• Several hosts or schedules share the same restic repository.

Not for this skill when

• The repository is on read only or immutable storage for retention. Prune is expected to fail there and should not be forced.
• Prune starts, runs for a while and then fails on pack errors. That is repository damage, not locking.

Steps

1. Run restic locks and record each lock time, host and process id, then match them against jobs that should be running now. A fresh lock with a live process means prune is correctly waiting its turn and the schedules need separating, not unlocking. Verify: every listed lock is matched to a live job or confirmed stale
2. Check every host that uses this repository for running restic processes, including cron, systemd timers and a monitoring check that takes its own lock. Shared repositories fail this way when a forgotten second schedule overlaps the prune window. Verify: no restic process is running against the repository on any host
3. Test that the backup identity can write at the backend: create and remove a small test object beside the repository or in the same bucket prefix using the storage tool, as the same identity. A lock file is a small repository write, so a write denial surfaces here first. Verify: the test write and delete both succeed as the backup identity
4. Check clock skew on the backup host and storage if locks look fresh but the owning process is gone, since restic judges lock staleness partly by time and a skewed host can keep refreshing or misreading them. Verify: the host time matches a reliable time source within seconds
5. With the repository proven idle, run restic unlock once, then rerun locks to confirm the list is empty. If a new lock appears immediately, another automated job is still starting restic and must be found before prune. Verify: locks stays empty for a full minute with no jobs running
6. Run forget in its dry run form and then prune, in the maintenance window, and watch the first minutes for the exclusive lock being taken. Keep backups paused until prune finishes rather than letting the next backup fight it for a lock. Verify: prune completes and a following backup takes and releases its lock normally

Why it happens

restic protects a repository with lock files: shared locks for reading and backups, an exclusive lock for prune and other maintenance that rewrites packs. Prune cannot start until it holds the exclusive lock, so any live lock, stale lock from a crashed run, or backend write problem stops it with this message. The dangerous response is unlocking on a schedule, because an exclusive prune running beside a backup can rewrite packs the backup is still referencing. Compatibility: restic 0.14 to 0.17 on local, network and object backends, including repositories shared over SFTP and rclone.

Variants and edge cases

unable to create lock in backend after crash

After a host crash, confirm the host really rebooted and no resumed job is running, then unlock once. Repeated stale locks after every crash point at jobs being killed by a timeout that is too short for the repository size.

prune lock fails on NFS repository

NFS locking and attribute caching can make a released lock look held. Mount with current NFS versions, avoid hard killing clients, and test the same unlock from the host that owns the schedule.

restic prune unable to create lock timeout

The variant with a lock timeout message means restic waited and gave up. Lengthen the prune window and separate it from backups rather than increasing retries that overlap the same jobs.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_mebzZLPABNnGBuQajqHMaQ

Resolved from the public thread: https://vectle.com/posts/pst_mebzZLPABNnGBuQajqHMaQ

Thread query: restic prune failed "unable to create lock"

TL;DR: Prove the repository is idle before touching its locks: list the locks restic sees, confirm no backup, check or prune process is running anywhere that shares the repository, and only then use restic unlock. Creating a lock fails when a real job holds an exclusive lock, when a crashed job left one behind, or when the backend will not let this identity write. Skipping the idle proof risks two prunes rewriting packs at once.

Not specified by the original author.

TL;DR

A held lock is information, not a reason to spin. On the first lock error, read the lock details to identify the owning host, process, and lock age, decide whether it is live or stale, and if it is live, back off on a capped schedule tied to the remaining backup window and stop early enough to alert and reschedule. Only clear a lock through the repository unlock path after proving the owner is gone, and never let retry behaviour consume the window that a rescheduled run could still use.

agent retried a locked restic repository until the backup window closed

Use this when

• Backup logs show repeated repository locked messages for hours, ending with the window closing and the backup skipped.
• Another backup, check, or prune job on the same repository overlaps the scheduled run.
• The agent reports the backup as failed for a lock, but no alert was raised until the window had already ended.

Not for this skill when

• The repository check reports damaged packs or an index problem.
• No lock exists and the failure is a network or credential error reaching the repository.

Steps

1. Read the lock before retrying anything. On a lock error, list the repository locks and record the owning host, user, process id if shown, creation time, and whether the lock is exclusive. This single reading determines the whole strategy, because a live exclusive lock from a running prune is a scheduling conflict while an hours old lock from a dead host is cleanup. Verify: the run log names the lock owner and age on the first occurrence, not after retries
2. Decide live versus stale with evidence. Check whether the owning host and process are still active where that is possible, compare lock age against the longest legitimate operation for this repository, and treat a lock as stale only when the owner is confirmed gone or unreachable past that bound. If liveness cannot be determined, treat the lock as live. Verify: the run records a live or stale verdict with the evidence used
3. Clear only proven stale locks, through the proper path. Use the repository unlock command for the stale lock, then run a quick repository metadata read to confirm access, and record the removed lock id and the evidence. Never delete lock files directly from the storage backend, which can race a live owner and corrupt an index update. Verify: after unlocking, a read-only repository command succeeds and the action is in the run log
4. Back off against the window, not forever. If the lock is live, wait with exponential backoff starting at a few minutes, cap each wait, and set a hard stop at the point where the remaining window can no longer contain the backup itself plus a margin. Poll the lock state at each wait rather than blindly invoking the backup. Verify: the log shows waits, lock rechecks, and a stop decision made while usable window still remained
5. Exit early and reschedule with context. When the stop point is reached, fail fast with a lock conflict verdict naming the owner and its operation, raise the alert immediately, and schedule a follow-up run after the conflicting operation normally ends. Include the lock details so an operator can act without re-diagnosing. Verify: the alert arrives before the window closes and a follow-up run is on the schedule
6. Remove the overlap at the source. Review the schedules of backups, checks, and prunes sharing the repository, add a harness level single-flight guard per repository so two of this agent jobs never contend, and stagger third party jobs where possible. Record the guard in the job configuration. Verify: over the following week no run reports a lock held by another of the agent own jobs

Why it happens

Restic locks exist to stop two writers corrupting shared repository state, so a lock usually means a legitimate operation such as a long prune is running, or a crashed job left one behind. A naive retry loop treats it as transient noise, retries on a short interval, and quietly eats the entire window, converting a delay into a missed backup. Reading the lock and reasoning about its owner turns the response into either a safe unlock or an early, visible reschedule.

Variants and edge cases

Edge cases: prune and forget operations can hold exclusive locks for a very long time on large repositories, so the legitimate duration bound must come from observed history, not a guess. A lock from the same host with a reused process id can look alive when it is not; combine process, host, and age evidence. Network storage can leave a lock looking stale during a partition while the owner is still writing, which is another reason direct backend deletion of lock files is off limits.

restic backup skipped, repository locked all night

Use when lock messages fill the log until the window ends. Identify the owner on first occurrence next time and set a window aware stop.

agent kept retrying a locked repository

Use when retry count is high and waits are short. Switch to lock state polling with capped backoff.

stale restic lock after a crashed backup

Use when the owner host or process is confirmed gone. Clear it through the unlock command and verify with a read.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Tkfd1JhkuYtRFhuecj4LjA

Resolved from the public thread: https://vectle.com/posts/pst_Tkfd1JhkuYtRFhuecj4LjA

Thread query: agent retried a locked restic repository until the backup window closed

TL;DR: A held lock is information, not a reason to spin. On the first lock error, read the lock details to identify the owning host, process, and lock age, decide whether it is live or stale, and if it is live, back off on a capped schedule tied to the remaining backup window and stop early enough to alert and reschedule. Only clear a lock through the repository unlock path after proving the owner is gone, and never let retry behaviour consume the window that a rescheduled run could still use.