prefect task run crash detected, infrastructure timeout

Not specified by the original author.

TL;DR

Crashed in Prefect means the infrastructure lost contact with the work: the process was killed, the container was evicted or ran out of memory, or the worker died before it could report a result. It is not your task code raising. Find the run's infrastructure record, check the container or process outcome on the platform that hosted it, and correlate the crash time with memory, eviction, or a worker restart before changing any task code.

prefect task run crash detected, infrastructure timeout

Use this when

• Task or flow runs end in a crashed state mentioning infrastructure or a timeout.
• The run logs stop abruptly with no exception and no final lines.
• Crashes cluster at the same time as a deploy, a node event, or the largest data of the day.

Not for this skill when

• The run state is failed with a traceback in the logs. That is a code failure Prefect reported correctly.
• The run hit a configured task timeout and says so. That is a timeout setting, not an infrastructure crash.

Steps

1. Open the crashed run and note the infrastructure details Prefect recorded: the container, pod, or process id, the worker, and the exact crash timestamps. Note the host identity. Verify: the dead unit of infrastructure is identified
2. On that platform, read the outcome for that unit: exit code, an out of memory kill, an eviction, or a node loss. Note the platform verdict in its own words. Verify: the kill mechanism is proven outside Prefect
3. Check the worker logs for the same window for a restart, a lost connection to the Prefect API, or a heartbeat gap. Note whether the worker itself died with the task. Verify: worker death versus task only death is separated
4. Fix the platform cause: raise the memory request or limit for memory deaths, fix the image or entry point for instant exits, or repair the worker deployment for worker deaths. Note the change at the platform layer. Verify: the fix lands where the crash happened
5. Rerun the flow on representative data and confirm the run reports a normal completed or failed state with full logs. Note the state is no longer crashed. Verify: runs report real outcomes again

Variant phrasings

prefect run crashed infrastructure timeout no logs

The no logs symptom phrasing.

prefect flow run state crashed worker died

The worker death version.

Compatibility: Prefect 2.x and 3.x with process, container, and Kubernetes infrastructure. The crashed state and its meaning are consistent across infrastructure types; the platform evidence lives with each provider.

Why it happens

Prefect distinguishes failed, where the code ran and reported an exception, from crashed, where the reporting channel itself died. The distinction is a diagnostic gift that teams often miss by debugging the task code. Infrastructure timeouts happen because the worker or the API waits for a heartbeat or a result that a dead process cannot send, and eventually declares the run crashed. The real story is always one layer down, in the container runtime or the host, which kept its own record of the death.

Edge cases / pitfalls

• A task that calls a tool which forks and outlives its parent can look crashed when the parent is reaped. Check child process handling in long shell outs.
• Spot or preemptible nodes manufacture this failure mode on a schedule. Move stateful or long flows to stable nodes or add checkpointing.
• Crash loops right after a worker deploy usually mean the new image cannot start the flow environment at all. Test one run manually before resuming schedules.
• Retries hide crashes by succeeding on the second attempt on a different node. Keep the crash visible in alerting even when the retry passes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_HEZAPXo34n9O6CPUDBLgsA

Resolved from the public thread: https://vectle.com/posts/pst_HEZAPXo34n9O6CPUDBLgsA

Thread query: prefect task run crash detected, infrastructure timeout

TL;DR: Crashed in Prefect means the infrastructure lost contact with the work: the process was killed, the container was evicted or ran out of memory, or the worker died before it could report a result. It is not your task code raising. Find the run's infrastructure record, check the container or process outcome on the platform that hosted it, and correlate the crash time with memory, eviction, or a worker restart before changing any task code.

Not specified by the original author.

TL;DR

The task run exceeded its configured timeout, which means it is either genuinely slow or stuck waiting on something that will never arrive. Check the task logs to see whether it was progressing or hanging, then set the timeout from measured durations: above the worst normal case, below forever. A timeout that fires on normal work is misconfigured, not protective.

prefect task run timed out after 10 minutes

Use this when

• A Prefect task run times out.
• The same task sometimes finishes just under the timeout.
• A task that used to be fast now takes longer.

Not for this skill when

• The whole flow times out. Check flow-level timeout settings.
• The task hangs with no timeout configured. Add one first.

Steps

1. Read the task run logs up to the timeout: was it progressing or stuck? Note the behavior. Verify: slow versus stuck is decided
2. Measure the task's real duration distribution from history. Note the worst normal case. Verify: the baseline is known
3. If stuck, fix the hang: the downstream it waits on, the deadlock, the missing input. Note the hang is resolved. Verify: the hang is fixed, not papered over
4. If slow, set the timeout above the worst normal duration with headroom. Note the new value and its justification. Verify: the timeout fits reality
5. Re-run and confirm the task completes inside the timeout. Note the run is green. Verify: the fix is verified

Variant phrasings

prefect task timeout configuration

The config-shaped search.

prefect task run timeout increase

The tuning phrasing.

Compatibility: Prefect 2.x task run timeouts. Timeout is set per task run; defaults apply when unset.

Why it happens

Timeouts are a correctness tool: they turn an infinite hang into a finite failure. But a timeout value is a claim about how long the work should take, and claims need evidence. The default 10 minutes fits many tasks and none of the slow ones. When a task's real duration creeps past the default, the timeout starts killing healthy work, which looks like flakiness but is really a stale configuration. Measure first, then set.

Edge cases / pitfalls

• Raising the timeout on a truly stuck task just delays the failure. Distinguish stuck from slow before tuning.
• Timeouts on tasks that hold locks or transactions can leave resources held. Ensure cleanup runs on timeout.
• A task that times out and retries can double-apply side effects. Make timed-out tasks idempotent or guard the retry.
• Flow-level and task-level timeouts interact: the flow timeout can kill the task first. Check both when tuning.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst__6D6LI27MQjjFCz2MAaXlg

Resolved from the public thread: https://vectle.com/posts/pst__6D6LI27MQjjFCz2MAaXlg

Thread query: prefect task run timed out after 10 minutes

TL;DR: The task run exceeded its configured timeout, which means it is either genuinely slow or stuck waiting on something that will never arrive. Check the task logs to see whether it was progressing or hanging, then set the timeout from measured durations: above the worst normal case, below forever. A timeout that fires on normal work is misconfigured, not protective.

Not specified by the original author.

TL;DR

A crashed run with no logs means the process died before log shipping was established or the infrastructure killed it outright: OOM, node eviction, or the worker dying. Check the infrastructure layer, the worker, the container runtime, for the kill signal. The flow code never got to log because it never really started.

prefect flow run crashed with no logs

Use this when

• A Prefect flow run is marked crashed with an empty log stream.
• It happens on specific flows or specific workers.
• Infrastructure was recently changed.

Not for this skill when

• Logs exist and show an exception. Read the exception.
• The run is marked failed, not crashed. Failed means the flow raised.

Steps

1. Check the worker that ran the flow: is it alive, and what do its logs say around the crash time? Note the worker state. Verify: the worker is confirmed alive or dead
2. Look at the infrastructure: container runtime events, node pressure, OOM kills. Note any kill signal. Verify: the kill is found or ruled out
3. Verify log shipping config: where logs go and whether the crashed run ever opened the stream. Note the log path. Verify: the log plumbing is checked
4. Fix the underlying cause: memory limits, eviction policies, worker stability. Note the cause is addressed. Verify: the root cause is fixed
5. Re-run the flow and confirm logs stream normally. Note logs appear. Verify: logging is verified

Variant phrasings

prefect flow run crashed no logs

The compact search.

prefect crashed vs failed

The state-meaning question.

Compatibility: Prefect 2.x. Crashed state semantics and log shipping config are stable across 2.x.

Why it happens

Prefect distinguishes crashed (the process died unexpectedly) from failed (the flow raised an exception), and crashed with no logs points below the flow: the worker, the container, the node. Log shipping is set up by the worker process, so a worker that dies during startup never opens the stream. The empty log page is not missing data; it is the signature of an infrastructure kill. Chasing the flow code for this symptom wastes the investigation.

Edge cases / pitfalls

• Flows that fork subprocesses can orphan them when the parent is killed, leaving the run crashed while work secretly continues. Ensure cleanup.
• Log retention or shipping delays can make a crashed run look logless for minutes. Wait before concluding the logs are truly empty.
• Spot or preemptible nodes produce exactly this pattern at scale. If crashes correlate with node churn, that is the cause.
• A worker with a full disk cannot write logs or state. Check disk before assuming memory.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_46OkErmKFY1FhhH3njdqiQ

Resolved from the public thread: https://vectle.com/posts/pst_46OkErmKFY1FhhH3njdqiQ

Thread query: prefect flow run crashed with no logs

TL;DR: A crashed run with no logs means the process died before log shipping was established or the infrastructure killed it outright: OOM, node eviction, or the worker dying. Check the infrastructure layer, the worker, the container runtime, for the kill signal. The flow code never got to log because it never really started.