Immutable solution version
Multiprocessing deadlock
Complete solution
Native Python multiprocessing deadlocks at .join() when a Prefect flow runs inside a Docker container with more than one worker: in a container, multiprocessing creates new processes with fork, and forking a multithreaded process is unsafe (per the Python docs). On a laptop the same code uses spawn, which is why it only breaks in Docker.
What was reported: Issue PrefectHQ/prefect#10794 (closed, 24 comments): ### First check - [X] I added a descriptive title to this issue. - [X] I used the GitHub search to find a similar issue and didn't find it. - [X] I searched the Prefect documentation for this issue. - [X] I checked that this issue is related to Prefect and not one of its dependencies. ### Bug summary Running native python multiprocessing with more than 1 worker leads to a deadlock at the .join() step. I am running this in a docker container. ### Reproduction ```python3 Inside a docker container: import multiprocessing as mp def mptask(args): return 1 @flow def mpflow(): n_cpu = mp.
What works: Native Python multiprocessing deadlocks at .join() when a Prefect flow runs inside a Docker container with more than one worker: in a container, multiprocessing creates new processes with fork, and forking a multithreaded process is unsafe (per the Python docs). On a laptop the same code uses spawn, which is why it only breaks in Docker. Workaround (maintainer-suggested): set the multiprocessing start method to spawn before any pool is created, e.g. multiprocessing.setstartmethod('spawn'), configured before the third-party code that creates the pool ever runs. Prefect itself does not create the pool, so the fix belongs in your code, not Prefect's.
Evidence
In the GitHub issue, the reporter asked: Multiprocessing deadlock. The working answer that came out of the thread: Native Python multiprocessing deadlocks at .join() when a Prefect flow runs inside a Docker container with more than one worker: in a container, multiprocessing creates new processes with fork, and forking a multithreaded process is unsafe (per the Python docs). On a laptop the same code uses spawn, which is why it only breaks in Docker. Workaround (maintainer-suggested): set the multiprocessing start method to spawn before any pool is created, e.g. multiprocessing.setstartmethod('spawn'), configured before the third-party code that creates the pool ever runs. Prefect itself does not create the pool, so the fix belongs in your code, not Prefect's. Source: https://github.com/PrefectHQ/prefect/issues/10794
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_ClyLANseKelb40sTi779nQ" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Multiprocessing deadlock <span>— Vectle</span></a>