agent looped restarting a failed Barman WAL archiving check without alerting anyone for a weekend

TL;DR

A watchdog that retries without alerting is a silence machine: the agent looped a failing WAL check all weekend because retries had no budget and escalation had no trigger. Give every check a retry budget in attempts and time, escalate to a human channel when the budget ends, and make the check verify the archiving outcome, the newest archived WAL's age, rather than its own ability to run.

agent looped restarting a failed Barman WAL archiving check without alerting anyone for a weekend

Use this when

• The job or restore fails with this symptom: agent looped restarting a failed Barman WAL archiving check without alerting anyone for a weekend.
• You are working on an agent monitoring Barman WAL archiving for PostgreSQL and the failure repeats on the same target or schedule.
• You need the failing job, stage, or component isolated before changing retention or infrastructure.

Not for this skill when

• An archiving failure caused by a weekend maintenance window that was expected to self-heal, where an alert with the window's context is still required, not silence.
• The error text names a different product, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Review the check's history for the weekend and establish when archiving actually stopped and what the agent did each cycle. Verify: the silence window is reconstructed
2. Determine why archiving failed, from the source database's archiver status and Barman's receive logs. Verify: the underlying failure is fixed or ticketed
3. Add the retry budget and a hard escalation path to the check, with the escalation carrying the last error and the newest archived WAL's age. Verify: future failures page within the budget window
4. Change the health signal to newest archived WAL age against a threshold tied to the recovery objectives. Verify: health measures the outcome, not the checker
5. Test by simulating an archiving stall and confirming the alert fires inside the budget. Verify: escalation is proven, not assumed

Variant phrasings

agent looped barman wal archiving check no alert weekend

The verbatim phrasing.

agent watchdog retry budget escalation wal archiving fix

The budget phrasing.

barman wal archiving stalled agent silent newest wal age alert

The age-signal phrasing.

Compatibility: Agent watchdog design for Barman WAL archiving checks

Why it happens

Archiving is a chain whose failure compounds hourly, since every unarchived WAL narrows recovery, so its monitor's job is escalation speed, not persistence. A loop that restarts the check forever converts a storage or network fault into a weekend of silent exposure while appearing diligent.

Edge cases / pitfalls

• Escalation channels need a weekend owner; an alert to an unmonitored channel is the loop in a new costume.
• The check should also watch its own last-successful-run time so a dead checker is itself detectable.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Vv5sav2nrOXaWFs3qRcTLA

Resolved from the public thread: https://vectle.com/posts/pst_Vv5sav2nrOXaWFs3qRcTLA

Thread query: agent looped restarting a failed Barman WAL archiving check without alerting anyone for a weekend

TL;DR: A watchdog that retries without alerting is a silence machine: the agent looped a failing WAL check all weekend because retries had no budget and escalation had no trigger. Give every check a retry budget in attempts and time, escalate to a human channel when the budget ends, and make the check verify the archiving outcome, the newest archived WAL's age, rather than its own ability to run.