Concurrency lease renewal failed - slots are no longer reserved. Terminating execution to prevent over-allocation

TL;DR

Issue PrefectHQ/prefect#19068 (closed, 30 comments): ### Bug summary Prefect 3.4.20 All Tasks are completed.[br]But the Flow that has worked correctly terminates with an error:[br]Finished in state Crashed('Execution was cancelled by the runtime environment.') ``` Flow run exceeded timeout of 300.0 second(s) 11:01:36 AM prefect.flow_runs Concurrency lease renewal failed - slots are no longer reserved.

Steps

1. Issue PrefectHQ/prefect#19068 (closed, 30 comments): ### Bug summary Prefect 3.4.20 All Tasks are completed.[br]But the Flow that has worked correctly terminates with an error:[br]Finished in state Crashed('Execution was cancelled by the runtime environment.') ``` Flow run exceeded timeout of 300.0 second(s) 11:01:36 AM prefect.flowruns Concurrency lease renewal failed - slots are no longer reserved. Terminating execution to prevent over-allocation. 11:01:37 AM prefect.flowruns Crash detected! Execution was cancelled by the runtime environment. 11:01:37 AM prefect.

When to use

You are seeing this: Terminating execution to prevent over-allocation. Use this skill when you run into "Concurrency lease renewal failed - slots are no longer reserved. Terminating execution to prevent over-allocation".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: Prefect 3.4.20, Prefect 3.4.24. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Sources

• https://github.com/PrefectHQ/prefect/issues/19068

In gh:PrefectHQ/prefect#19068, the reporter ran into: Issue PrefectHQ/prefect#19068 (closed, 30 comments): ### Bug summary Prefect 3.4.20 All Tasks are completed.[br]But the Flow that has worked correctly terminates with an error:[br]Finished in state Crashed('Execution was cancelled by the runtime environment.') ``` Flow run exceeded timeout of 300.0 second(s) 11:01:36 AM prefect.flowruns Concurrency lease renewal failed - slots are no longer reserved. Terminating execution to prevent over-allocation. 11:01:37 AM prefect.flowruns Crash detected! Execution was cancelled by the runtime environment. 11:01:37 AM prefect. The verified answer was: 'Concurrency lease renewal failed - slots are no longer reserved' with the flow crashing after all tasks completed is usually a blocked event loop, not a real timeout. Lease renewal runs on the event loop of the main thread, so if one of your async coroutines blocks it, renewal stops and the runtime kills the flow. If you call synchronous code from an async task or flow, wrap it in asyncio.to_thread so it doesn't block. This was also fixed upstream in Prefect 3.4.24 (#19181), so upgrade too. Source issue: https://github.com/PrefectHQ/prefect/issues/19068