Multiprocessing deadlock
Shows how to fix multiprocessing deadlock. Use it when you hit this exact problem. Skip it when your error message or symptom looks different.
TL;DR
What was reported: Issue PrefectHQ/prefect#10794 (closed, 24 comments): ### First check - [X] I added a descriptive title to this issue. - [X] I used the GitHub search to find a similar issue and didn't find it. - [X] I searched the Prefect documentation for this issue. - [X] I checked that this issue is related to Prefect and not one of its dependencies. ### Bug summary Running native python multiprocessing with more than 1 worker leads to a deadlock at the.join() step.
Steps
- What was reported:
Issue PrefectHQ/prefect#10794 (closed, 24 comments): ### First check - [X] I added a descriptive title to this issue. - [X] I used the GitHub search to find a similar issue and didn't find it. - [X] I searched the Prefect documentation for this issue. - [X] I checked that this issue is related to Prefect and not one of its dependencies. ### Bug summary Running native python multiprocessing with more than 1 worker leads to a deadlock at the .join() step. I am running this in a docker container. ### Reproduction ```python3 Inside a docker container: import multiprocessing as mp def mptask(args): return 1 @flow def mpflow(): n_cpu = mp.
- What works:
Native Python multiprocessing deadlocks at .join() when a Prefect flow runs inside a Docker container with more than one worker: in a container, multiprocessing creates new processes with fork, and forking a multithreaded process is unsafe (per the Python docs). On a laptop the same code uses spawn, which is why it only breaks in Docker. Workaround (maintainer-suggested): set the multiprocessing start method to spawn before any pool is created, e.g. multiprocessing.setstartmethod('spawn'), configured before the third-party code that creates the pool ever runs. Prefect itself does not create the pool, so the fix belongs in your code, not Prefect's.
When to use
You are seeing this: I am running this in a docker container. ### Reproduction ```python3 Inside a docker container: import multiprocessing as mp def mptask(args): return 1 @flow def mpflow(): n_cpu = mp. Use this skill when you run into "Multiprocessing deadlock".
When not to use
If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.
Versions
No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.
Why this happens
The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.