prefect agent crashed and scheduled flow runs failed

TL;DR

Prefect agents poll for scheduled work, so a dead agent means scheduled runs never get infrastructure and fail or never start. Restart the agent under a supervisor so it cannot stay dead, then triage the missed window: rerun what matters, skip what does not. An unsupervised agent is a single point of failure for every schedule it serves.

prefect agent crashed and scheduled flow runs failed

Use this when

• A Prefect agent crashed and scheduled runs failed around the same time.
• Schedules silently stop producing runs.
• It happens after deploys or host restarts.

Not for this skill when

• The agent is running but idle while runs queue. That is a matching or capacity problem.
• Runs start but fail. That is a flow or infrastructure problem.

Steps

1. Confirm the agent process is dead and find why: logs, OOM, host restart. Note the cause of death. Verify: the crash is explained
2. Restart the agent under a process supervisor or as a managed service. Note it is supervised now. Verify: it cannot stay dead silently
3. List the scheduled runs missed during the outage window. Note the missed set. Verify: the blast radius is mapped
4. Rerun the missed runs that matter; explicitly skip the ones that do not. Note each missed run has a decision. Verify: the window is triaged
5. Add alerting on agent health so the next crash pages someone. Note the alert is tested. Verify: the next crash is visible

Variant phrasings

prefect agent keeps crashing

The recurrence phrasing.

prefect scheduled runs not starting

The symptom search.

Compatibility: Prefect 2.x agents. Supervision approach depends on your platform: systemd, Kubernetes, or a process manager.

Why it happens

The agent is the bridge between Prefect's schedule and your infrastructure: it watches for scheduled runs and submits them. Nothing else performs that polling, so a dead agent silently stops all its schedules. Prefect 2 moved toward workers with similar semantics, but the failure mode is identical: the poller dies, the schedules starve. Supervision is not optional for the component that starts all your work.

Edge cases / pitfalls

• Multiple agents polling the same work queue can double-submit runs. One supervised agent per queue, or use work pools correctly.
• An agent that crashes on a poison-pill run will crash again on restart when it picks up the same run. Quarantine the run first.
• Missed runs that wrote partial results need cleanup before rerun. Triage the data, not just the schedule.
• Agent upgrades that change the work-queue matching can strand schedules that matched the old config. Verify matching after upgrades.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstKTwhrOhUOTAGJaqqKWTMw