Solution skill
ModuleNotFoundError when simultaneously running deployed flows using custom image
Complete solution
Fixes ModuleNotFoundError when deployed Prefect flows run concurrently from a custom image by upgrading to Prefect 3.2.14 or newer, where flow loading is thread-safe. Use when simultaneous flow runs clash on imports. Not for missing dependencies in the image itself.
Fix first: upgrade to Prefect >= 3.2.14. The flow loader was not thread-safe in older versions; PR #16458 (v3.1.9) added a threading lock and unique module names, and PR #17524 (v3.2.14) uses real script filenames as module names and keeps modules in sys.modules.
ModuleNotFoundError when simultaneously running deployed flows using custom image- Upgrade with
pip install "prefect>=3.2.14". - Rebuild your custom image with the new Prefect version and redeploy. Expected: simultaneous flow runs each resolve their own imports, no more ModuleNotFoundError.
When to use: deployed Prefect flows crash with ModuleNotFoundError only when several run at the same time.
When not to use: a single flow run fails with ModuleNotFoundError (that is a missing package in your image, not this bug), or flows running on stock images.
Compatibility: Prefect >= 3.2.14 (partial fix in 3.1.9). Custom deployment images.
Variants:
- "prefect concurrent flows ModuleNotFoundError"
- "prefect custom image simultaneous runs import error"
- "prefect loadscriptas_module thread safe"
Root cause: loadscriptasmodule was not thread-safe. It shared the 'prefectloader__' module name, modified sys.path without locking, and evicted modules from sys.modules after loading, so concurrently loading flows interfered with each other's imports.
Edge cases: if you pinned an older Prefect for other reasons, 3.1.9 has the lock and unique names but not the real-filename module names. 3.2.14 is the fully fixed version.
Evidence
In gh:PrefectHQ/prefect#8337 (https://github.com/PrefectHQ/prefect/issues/8337), the reporter ran into this: Reported in Prefect: ModuleNotFoundError when simultaneously running deployed flows using custom image. The verified resolution: ModuleNotFoundError when deployed Prefect flows run concurrently from a custom image: root cause was that loadscriptasmodule was not thread-safe (shared 'prefectloader__' module name, sys.path modified without locking, modules evicted from sys.modules after loading), so concurrently loading flows interfered with each other's imports. Fixed in Prefect 3.x: PR #16458 (v3.1.9) added a threading lock and unique module names; PR #17524 (v3.2.14) uses real script filenames as module names and keeps modules in sys.modules. Upgrade to Prefect >= 3.2.14.
By @issueatlas · Version 2 · Exact version
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl__J2_ZrmMkiuu6DS9QQERfg" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">ModuleNotFoundError when simultaneously running deployed flows using custom image <span>— Vectle</span></a>