## TL;DR
Monitor the fleet's heartbeat, not just its errors: every worker reports "alive and on item N" on a schedule, a watcher alerts when a heartbeat goes stale, and a central log shows progress per worker. Silent workers are the failure mode that error alerting misses. Fifteen minutes of heartbeat plumbing saves the morning you discover a fleet that died on Friday.

```text
how to monitor a fleet of background workers
```

## Use this when
- Background workers run for hours or days unattended
- You have discovered a dead fleet long after it died
- Workers stall (alive but not progressing) rather than crash
- You need one view of what every worker is doing
- You are handing a fleet to someone else to operate

## Not for this skill when
- You need sub-second alerting (that is real-time monitoring, different tooling)
- There is only one worker (check its logs; you do not need fleet monitoring)
- The question is APM or request tracing for a service
- You need to monitor the work's quality, not the workers' liveness

## Steps
1. Give every worker a heartbeat: a timestamped "alive, working on X" written on a fixed interval. A file per worker, a row in a status table, or a log line; the mechanism matters less than the regularity. No heartbeat means no visibility.
   ```text
   heartbeat: every [n] minutes -> [location], content: worker id, item id, timestamp
   ```
   Expected output: heartbeats flowing from all workers. Success check: you can list every live worker and its current item from the heartbeats alone.

2. Run a watcher that alerts on stale heartbeats, not just on errors. If a worker's heartbeat is older than twice the interval, something is wrong: crashed, wedged, or starved. Alert on the silence; errors will announce themselves.
   ```text
   stale threshold: [2x heartbeat interval]
   alert goes to: [channel], with worker id and last item
   ```
   Expected output: a watcher with alerts. Success check: killing a worker produces an alert within the threshold, tested deliberately.

3. Keep one central append-only log for the fleet. Claims, completions, errors, and heartbeats in one place with worker ids and timestamps. Per-worker logs are for debugging; the central log is for "what is the fleet doing right now".
   ```text
   [timestamp] worker-A claimed 42
   [timestamp] worker-B completed 57
   [timestamp] worker-A heartbeat, on item 43
   ```
   Expected output: a central log. Success check: "fleet status" is answerable from the log in under a minute.

4. Track progress rate per worker, and alert on stalls. A worker with a fresh heartbeat but no completions for an hour is wedged on something; completions-per-hour per worker catches what liveness checks miss. Stalls are the quiet killer of long runs.
   ```text
   expected rate: [completions/hour/worker], stall alert after: [n] hours flat
   ```
   Expected output: rate tracking with stall alerts. Success check: a deliberately wedged worker triggers the stall alert.

5. Build the morning fleet summary into the routine. Workers alive, items done overnight, errors seen, stalls handled. Five lines in the morning briefing (or its own message) keeps the fleet a managed operation instead of a background hope.
   ```text
   fleet summary: [n] alive, [n] items done, [n] errors, [n] stalls
   ```
   Expected output: a daily summary. Success check: three consecutive days summarized without manual log-diving.

## Variant phrasings
### Background job monitoring best practices
Jobs phrasing. Heartbeats (step 1), stale alerts (step 2), and the central log (step 3) are the practice.

### How to tell if a cron fleet is healthy
Cron phrasing. Same monitoring; note that cron-shaped fleets also need "did the job even start" checks, since schedulers fail silently too.

### Detecting stalled workers
Stall phrasing. Step 4: fresh heartbeat plus flat completions equals wedged; that combination is the signal.

### Agent fleet observability
Observability phrasing. The whole skill: heartbeat, central log, rates, and a daily summary are the observability stack for agent fleets.

## Why it happens
Workers fail quietly: infinite loops, deadlocked locks, exhausted API quotas that return empty successes, and hosts that kill processes without telling anyone. Error alerting catches the loud failures; the quiet ones look like a fleet that is "still running" while producing nothing. Heartbeats plus completion rates catch both, because they measure aliveness and progress independently.

## Edge cases / pitfalls
- Heartbeat writes can contend. Keep them tiny and infrequent relative to the work; a heartbeat per second per worker becomes its own load problem.
- Alert fatigue kills monitoring. Tune thresholds so alerts mean action; a watcher that pages nightly for nothing gets muted, and then the real failure goes unnoticed.
- Do not monitor the monitor with itself. The watcher needs its own liveness check (a cron that checks the watcher, or an external ping); watchers die too.
- Log volume grows fast. Rotate the central log and aggregate old entries; a ten-gigabyte log nobody can open is not observability.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_QtCIjyMojwXI4LLEn7D3jA