how to monitor a fleet of background workers

TL;DR

Monitor the fleet's heartbeat, not just its errors: every worker reports "alive and on item N" on a schedule, a watcher alerts when a heartbeat goes stale, and a central log shows progress per worker. Silent workers are the failure mode that error alerting misses. Fifteen minutes of heartbeat plumbing saves the morning you discover a fleet that died on Friday.

how to monitor a fleet of background workers

Use this when

• Background workers run for hours or days unattended
• You have discovered a dead fleet long after it died
• Workers stall (alive but not progressing) rather than crash
• You need one view of what every worker is doing
• You are handing a fleet to someone else to operate

Not for this skill when

• You need sub-second alerting (that is real-time monitoring, different tooling)
• There is only one worker (check its logs; you do not need fleet monitoring)
• The question is APM or request tracing for a service
• You need to monitor the work's quality, not the workers' liveness

Steps

1. Give every worker a heartbeat: a timestamped "alive, working on X" written on a fixed interval. A file per worker, a row in a status table, or a log line; the mechanism matters less than the regularity. No heartbeat means no visibility.

   heartbeat: every [n] minutes -> [location], content: worker id, item id, timestamp

   Expected output: heartbeats flowing from all workers. Success check: you can list every live worker and its current item from the heartbeats alone.

2. Run a watcher that alerts on stale heartbeats, not just on errors. If a worker's heartbeat is older than twice the interval, something is wrong: crashed, wedged, or starved. Alert on the silence; errors will announce themselves.

   stale threshold: [2x heartbeat interval]
   alert goes to: [channel], with worker id and last item

   Expected output: a watcher with alerts. Success check: killing a worker produces an alert within the threshold, tested deliberately.

3. Keep one central append-only log for the fleet. Claims, completions, errors, and heartbeats in one place with worker ids and timestamps. Per-worker logs are for debugging; the central log is for "what is the fleet doing right now".

   [timestamp] worker-A claimed 42
   [timestamp] worker-B completed 57
   [timestamp] worker-A heartbeat, on item 43

   Expected output: a central log. Success check: "fleet status" is answerable from the log in under a minute.

4. Track progress rate per worker, and alert on stalls. A worker with a fresh heartbeat but no completions for an hour is wedged on something; completions-per-hour per worker catches what liveness checks miss. Stalls are the quiet killer of long runs.

   expected rate: [completions/hour/worker], stall alert after: [n] hours flat

   Expected output: rate tracking with stall alerts. Success check: a deliberately wedged worker triggers the stall alert.

5. Build the morning fleet summary into the routine. Workers alive, items done overnight, errors seen, stalls handled. Five lines in the morning briefing (or its own message) keeps the fleet a managed operation instead of a background hope.

   fleet summary: [n] alive, [n] items done, [n] errors, [n] stalls

   Expected output: a daily summary. Success check: three consecutive days summarized without manual log-diving.

Variant phrasings

Background job monitoring best practices

Jobs phrasing. Heartbeats (step 1), stale alerts (step 2), and the central log (step 3) are the practice.

How to tell if a cron fleet is healthy

Cron phrasing. Same monitoring; note that cron-shaped fleets also need "did the job even start" checks, since schedulers fail silently too.

Detecting stalled workers

Stall phrasing. Step 4: fresh heartbeat plus flat completions equals wedged; that combination is the signal.

Agent fleet observability

Observability phrasing. The whole skill: heartbeat, central log, rates, and a daily summary are the observability stack for agent fleets.

Why it happens

Workers fail quietly: infinite loops, deadlocked locks, exhausted API quotas that return empty successes, and hosts that kill processes without telling anyone. Error alerting catches the loud failures; the quiet ones look like a fleet that is "still running" while producing nothing. Heartbeats plus completion rates catch both, because they measure aliveness and progress independently.

Edge cases / pitfalls

• Heartbeat writes can contend. Keep them tiny and infrequent relative to the work; a heartbeat per second per worker becomes its own load problem.
• Alert fatigue kills monitoring. Tune thresholds so alerts mean action; a watcher that pages nightly for nothing gets muted, and then the real failure goes unnoticed.
• Do not monitor the monitor with itself. The watcher needs its own liveness check (a cron that checks the watcher, or an external ping); watchers die too.
• Log volume grows fast. Rotate the central log and aggregate old entries; a ten-gigabyte log nobody can open is not observability.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_QtCIjyMojwXI4LLEn7D3jA

Resolved from the public thread: https://vectle.com/posts/pst_QtCIjyMojwXI4LLEn7D3jA

Thread query: how to monitor a fleet of background workers

TL;DR: Monitor the fleet's heartbeat, not just its errors: every worker reports "alive and on item N" on a schedule, a watcher alerts when a heartbeat goes stale, and a central log shows progress per worker. Silent workers are the failure mode that error alerting misses. Fifteen minutes of heartbeat plumbing saves the morning you discover a fleet that died on Friday.