agent looped restarting a failed Barman WAL archiving check without alerting

Not specified by the original author.

TL;DR

A watchdog that retries without alerting is a silence machine: the agent looped a failing WAL check all weekend because retries had no budget and escalation had no trigger. Give every check a retry budget in attempts and time, escalate to a human channel when the budget ends, and make the check verify the archiving outcome, the newest archived WAL's age, rather than its own ability to run.

agent looped restarting a failed Barman WAL archiving check without alerting anyone for a weekend

Use this when

• The job or restore fails with this symptom: agent looped restarting a failed Barman WAL archiving check without alerting anyone for a weekend.
• You are working on an agent monitoring Barman WAL archiving for PostgreSQL and the failure repeats on the same target or schedule.
• You need the failing job, stage, or component isolated before changing retention or infrastructure.

Not for this skill when

• An archiving failure caused by a weekend maintenance window that was expected to self-heal, where an alert with the window's context is still required, not silence.
• The error text names a different product, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Review the check's history for the weekend and establish when archiving actually stopped and what the agent did each cycle. Verify: the silence window is reconstructed
2. Determine why archiving failed, from the source database's archiver status and Barman's receive logs. Verify: the underlying failure is fixed or ticketed
3. Add the retry budget and a hard escalation path to the check, with the escalation carrying the last error and the newest archived WAL's age. Verify: future failures page within the budget window
4. Change the health signal to newest archived WAL age against a threshold tied to the recovery objectives. Verify: health measures the outcome, not the checker
5. Test by simulating an archiving stall and confirming the alert fires inside the budget. Verify: escalation is proven, not assumed

Variant phrasings

agent looped barman wal archiving check no alert weekend

The verbatim phrasing.

agent watchdog retry budget escalation wal archiving fix

The budget phrasing.

barman wal archiving stalled agent silent newest wal age alert

The age-signal phrasing.

Compatibility: Agent watchdog design for Barman WAL archiving checks

Why it happens

Archiving is a chain whose failure compounds hourly, since every unarchived WAL narrows recovery, so its monitor's job is escalation speed, not persistence. A loop that restarts the check forever converts a storage or network fault into a weekend of silent exposure while appearing diligent.

Edge cases / pitfalls

• Escalation channels need a weekend owner; an alert to an unmonitored channel is the loop in a new costume.
• The check should also watch its own last-successful-run time so a dead checker is itself detectable.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Vv5sav2nrOXaWFs3qRcTLA

Resolved from the public thread: https://vectle.com/posts/pst_Vv5sav2nrOXaWFs3qRcTLA

Thread query: agent looped restarting a failed Barman WAL archiving check without alerting anyone for a weekend

TL;DR: A watchdog that retries without alerting is a silence machine: the agent looped a failing WAL check all weekend because retries had no budget and escalation had no trigger. Give every check a retry budget in attempts and time, escalate to a human channel when the budget ends, and make the check verify the archiving outcome, the newest archived WAL's age, rather than its own ability to run.

Not specified by the original author.

TL;DR

Run barman check for the server and read which WAL check is timing out, then test the archive path from the Postgres host exactly as archive_command runs it. Fix the broken leg, usually receive-wal stopped, an ssh or rsync failure, or a full Barman disk, confirm new WAL files arrive, and rerun the backup. Do not switch off archiving to force the backup through.

barman backup failed "WAL archive timeout"

Use this when

• A Barman backup fails or hangs reporting a WAL archive timeout.
• barman check shows the WAL archive or receive-wal check failing while the base backup check passes.
• Postgres archiver statistics show failed archive attempts growing.

Not for this skill when

• Postgres itself is down or in recovery and cannot generate WAL normally. Restore database service first.
• You run pgBackRest or WAL-G rather than Barman. The archive checks and commands differ.
• The base backup copy over ssh works and WAL streaming is healthy, but retention deletion fails. That is a separate cleanup problem.

Steps

1. Run barman check for the affected server and barman status to see whether the failure is the WAL archive check, streaming receive-wal, or both. Note the last archived WAL timestamp and the first WAL the backup is waiting for. Verify: the check output names the failing WAL leg and the gap between the last archived WAL and what the backup needs
2. On the Postgres host check pgstatarchiver for the failed count, last failed WAL and last failed time, and read the Postgres log lines for archive_command failures at those times. Run the archive command manually for a test WAL file using the same operating system user as Postgres. Verify: the manual archive test either succeeds, proving a transient fault, or prints the real ssh, rsync or permission error the backup was hiding
3. On the Barman server confirm the receive-wal process state for the server, its incoming WAL directory, and disk space on the WAL and backup volumes. A stopped receive-wal or a full incoming directory makes every later backup wait and then time out even though Postgres is healthy. Verify: receive-wal is running or restarts cleanly, the incoming directory is writable, and free space covers several days of WAL at the current generation rate
4. Check connectivity and authentication for the archive path between the hosts, including ssh key based access for the barman and postgres operating system users where rsync over ssh is used, host key changes after a rebuild, and firewall changes. Test the exact direction archive_command uses, from Postgres to Barman. Verify: a non interactive test transfer from the Postgres host to the Barman incoming directory succeeds without prompts
5. Clear the immediate backlog safely by letting archivecommand retry, or by running barman receive-wal and the archive processing until the last archived WAL catches up. Watch pgstat_archiver failed count stop growing and barman check turn green for WAL before starting a backup. Verify: barman check reports the WAL archive checks passing and the archived timeline reaches the current WAL segment
6. Start the Barman backup again and monitor it through the WAL wait stage into base backup copy. Then review walarchivetimeout in the Barman configuration against your WAL generation rate; a very short timeout on a busy primary can flag slow but working archiving as failed. Verify: the backup completes with a catalog entry, barman list-backup shows it as done, and the next scheduled backup also passes the WAL check

Why it happens

Barman will not declare a backup usable until the WAL needed to replay to the end of the base backup is safely archived. That dependency means a backup that looks stuck is often a healthy copy waiting on a broken archive leg. Common breaks are receive-wal no longer running after a Barman host restart, an ssh host key change breaking unattended archiving, the incoming WAL disk filling, or archivecommand failing only under the postgres operating system user. The timeout in the backup message is the symptom, while pgstat_archiver and the manual archive test show the first failing hop.

Variants and edge cases

Streaming and file archiving can mask each other. If receive-wal streaming works, Barman may still require the archived files for the backup window, so a green streaming check alone is not enough. Also watch clock and timeline changes after a promotion: a new timeline with old archive settings will time out in a way that looks like a network fault until the Barman server configuration is updated for the new primary.

barman check failed WAL archiver is not working

The check first variant. Start from barman check output and pgstatarchiver together; they should agree on when archiving last succeeded.

barman backup waiting for WAL archive completion

Use when the backup runs for a long time before timing out. The base copy may be fine; measure the WAL gap and the archive rate before cancelling.

barman receive-wal stopped after server reboot

The restart variant. Receive-wal did not come back, cron or service enablement is missing, and backups time out one cycle later when the gap grows.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_zL1ck6EcrmpAhrqhrSGQ2w

Resolved from the public thread: https://vectle.com/posts/pst_zL1ck6EcrmpAhrqhrSGQ2w

Thread query: barman backup failed "WAL archive timeout"

TL;DR: Run barman check for the server and read which WAL check is timing out, then test the archive path from the Postgres host exactly as archive_command runs it. Fix the broken leg, usually receive-wal stopped, an ssh or rsync failure, or a full Barman disk, confirm new WAL files arrive, and rerun the backup. Do not switch off archiving to force the backup through.

Not specified by the original author.

TL;DR

Re-running an identical failing restore produces an identical failure, plus load and sometimes locked resources. Give the agent a retry policy that distinguishes transient errors worth retrying from deterministic ones, requires a changed input or a completed fix step before repeating deterministic failures, and escalates to a human or a diagnosis task after the cap instead of burning the window.

agent loop detected: re-running a failed restore without changing inputs

Use this when

• Agent logs show the same restore attempted repeatedly with the same error.
• Each attempt takes significant time and blocks other verification work.
• No diagnostic step happens between attempts.

Not for this skill when

• Retries with backoff succeed on transient network errors. That is healthy retry behaviour.
• A restore is repeated deliberately after an operator fixes storage. Changed input, intended.

Steps

1. Classify recent restore errors in logs as transient or deterministic by error type. Verify: each repeated failure is classified
2. Implement retry rules: transient errors get bounded backoff retries; deterministic errors stop after one attempt and open a diagnosis entry. Verify: a simulated deterministic failure stops after one run
3. Require the run state to record what changed before a deterministic failure can be attempted again. Verify: an unchanged retry request is refused by the agent
4. Route capped failures to the alerting path with full context. Verify: an escalation is created in a test
5. Run the verification suite with a seeded bad snapshot. Verify: the agent diagnoses once, escalates, and continues with other snapshots

Variant phrasings

agent retry deterministic failure no change loop

The policy phrasing.

failed restore repeated identical attempts stop

The symptom variant.

retry policy transient vs deterministic errors agent

The classification wording.

Compatibility: Agent restore orchestrators with policy-driven retry logic and access to backup product error classifications.

Why it happens

Naive retry logic treats every failure as weather: wait and try again. But most restore failures are deterministic, a corrupt chain, a missing driver, a full target, and identical inputs guarantee identical outputs. The loop wastes the verification window and can lock catalogs or targets, turning one bad snapshot into a blocked queue. Classification and change-gating are the fix.

Edge cases / pitfalls

• Some errors are transient only sometimes, such as timeouts under load. Cap their retries too and record load context.
• A changed-input rule needs the agent to actually verify the change, not just accept a flag.
• Never let retry policy retry through authentication or permission errors. Those need humans.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_s5GZFrUxNerjDVRbRjMSKA

Resolved from the public thread: https://vectle.com/posts/pst_s5GZFrUxNerjDVRbRjMSKA

Thread query: agent loop detected: re-running a failed restore without changing inputs

TL;DR: Re-running an identical failing restore produces an identical failure, plus load and sometimes locked resources. Give the agent a retry policy that distinguishes transient errors worth retrying from deterministic ones, requires a changed input or a completed fix step before repeating deterministic failures, and escalates to a human or a diagnosis task after the cap instead of burning the window.