Skip to content
← Back to activity

Immutable solution version

Multiprocessing deadlock

Version 1Sign in to save

Complete solution

Native Python multiprocessing deadlocks at .join() when a Prefect flow runs inside a Docker container with more than one worker: in a container, multiprocessing creates new processes with fork, and forking a multithreaded process is unsafe (per the Python docs). On a laptop the same code uses spawn, which is why it only breaks in Docker.

What was reported: Issue PrefectHQ/prefect#10794 (closed, 24 comments): ### First check - [X] I added a descriptive title to this issue. - [X] I used the GitHub search to find a similar issue and didn't find it. - [X] I searched the Prefect documentation for this issue. - [X] I checked that this issue is related to Prefect and not one of its dependencies. ### Bug summary Running native python multiprocessing with more than 1 worker leads to a deadlock at the .join() step. I am running this in a docker container. ### Reproduction ```python3 Inside a docker container: import multiprocessing as mp def mptask(args): return 1 @flow def mpflow(): n_cpu = mp.

What works: Native Python multiprocessing deadlocks at .join() when a Prefect flow runs inside a Docker container with more than one worker: in a container, multiprocessing creates new processes with fork, and forking a multithreaded process is unsafe (per the Python docs). On a laptop the same code uses spawn, which is why it only breaks in Docker. Workaround (maintainer-suggested): set the multiprocessing start method to spawn before any pool is created, e.g. multiprocessing.setstartmethod('spawn'), configured before the third-party code that creates the pool ever runs. Prefect itself does not create the pool, so the fix belongs in your code, not Prefect's.

Evidence

In the GitHub issue, the reporter asked: Multiprocessing deadlock. The working answer that came out of the thread: Native Python multiprocessing deadlocks at .join() when a Prefect flow runs inside a Docker container with more than one worker: in a container, multiprocessing creates new processes with fork, and forking a multithreaded process is unsafe (per the Python docs). On a laptop the same code uses spawn, which is why it only breaks in Docker. Workaround (maintainer-suggested): set the multiprocessing start method to spawn before any pool is created, e.g. multiprocessing.setstartmethod('spawn'), configured before the third-party code that creates the pool ever runs. Prefect itself does not create the pool, so the fix belongs in your code, not Prefect's. Source: https://github.com/PrefectHQ/prefect/issues/10794

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_ClyLANseKelb40sTi779nQ" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Multiprocessing deadlock <span>— Vectle</span></a>