watchdog killed run after intel agent timed out and briefing failed

Watchdog killed the run after the intel agent timed out

TL;DR

A watchdog kill means the agent hung past its deadline, and the watchdog did its job. The fix is making timeouts survivable: checkpoint often, so the kill loses minutes not hours, and make the next run resume from the checkpoint. Then fix the hang itself, usually an unbounded wait or retry loop, so the watchdog stays a safety net, not a routine event.

The error

(watchdog kill)
watchdog killed run after intel agent timed out and briefing failed; SIGTERM after 6h, no output

When this helps

• a watchdog kills agent runs
• long runs die without output
• building resumable agents
• calibrating watchdog deadlines

When it doesn't

• the watchdog deadline is too tight; measure first, then set it
• the hang is a deadlock bug; fix the bug
• runs are short; watchdogs add little

Works with

python 3.8+ with json. Any scheduler with watchdog support.

Steps

1. Checkpoint every unit of work

import json
open("run_state.json", "w").write(json.dumps({"done": 12, "total": 40}))
print("checkpoint after each filing; the kill loses one unit at most")

Expected: A checkpoint file. Watchdog kills become resume events, not disasters.

2. Resume from the checkpoint on restart

import json
state = json.load(open("run_state.json"))
print("resuming at unit", state["done"] + 1, "of", state["total"])

Expected: A resume. The rerun skips completed work automatically.

3. Find the hang: audit for unbounded waits

import re
code = open("agent.py").read()
print("while True occurrences:", code.count("while True"))
print("each needs a bound or a break condition")

Expected: A hang audit. Unbounded loops and waits are the usual cause of watchdog kills.

4. Set the watchdog deadline from measured runtimes

import json
typical = 240
open("watchdog.json", "w").write(json.dumps({"deadline_minutes": int(typical * 1.5)}))
print("deadline at 1.5x typical; kills stay exceptional")

Expected: A calibrated deadline. The watchdog fires on real hangs, not on slow normal runs.

Other ways people phrase this

watchdog killed agent run timeout

Checkpoint often, resume on restart, fix the hang.

agent timed out briefing failed watchdog

The kill is the safety net working. Make it survivable.

sigterm long running agent

Handle SIGTERM by checkpointing; die gracefully.

Why it happens

Watchdogs kill runs that exceed their deadline, which is correct behavior for hung agents. The damage comes from lost work, not the kill. Frequent checkpoints bound the loss, resume logic recovers it, and hang audits address the cause so kills stay rare.

Edge cases

• Checkpoint writes must be atomic; a torn checkpoint is worse than none.
• Handle SIGTERM explicitly to checkpoint on the way out.
• A watchdog that fires routinely is miscalibrated or the agent is too slow.
• Log the kill event with the last checkpoint for post-mortems.
• A run killed twice in a row needs its deadline or workload re-examined; repeated kills are a planning failure, not bad luck.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_CCFqvh1hGapfxmH-Db3SRg