Immutable solution version
Prefect 3 server memory grows until OOM with default in-memory messaging
Complete solution
The Prefect 3 server leak came from the in-memory event messaging path used when no Redis is configured: unbounded internal queues plus a fakeredis Lua-runtime leak under the default `memory://` Docket broker. Fix: upgrade Prefect to 3.6.16 or later, which carries the pydocket 0.17.x fixes and the server-side queue caps. If you can run Redis, also point the messaging layer at it (`PREFECT_SERVER_DOCKET_URL=redis://...`) as an extra buffer, since the in-memory path is the one that leaked.
Prefect 3 server memory grows until OOM with default in-memory messaging
The problem
Self-hosted Prefect 3 servers showed memory climbing steadily until Kubernetes OOM-killed the pod, a regression from Prefect 2 which was flat. The reporter ran without Redis and noted Prefect 2 had no such problem. Prefect maintainers and collaborators dug in and found several contributing leaks: unbounded queues in the in-memory messaging implementation, and a fakeredis Lua-runtime leak behind the default memory:// Docket broker, fixed in pydocket 0.17.2 and 0.17.3. Prefect also capped the in-memory queues so they warn and drop instead of growing forever. Two reporters confirmed the leak was gone on 3.6.16, one with 30 days of stable graphs.
The verified fix
The Prefect 3 server leak came from the in-memory event messaging path used when no Redis is configured: unbounded internal queues plus a fakeredis Lua-runtime leak under the default memory:// Docket broker. Fix: upgrade Prefect to 3.6.16 or later, which carries the pydocket 0.17.x fixes and the server-side queue caps. If you can run Redis, also point the messaging layer at it (PREFECT_SERVER_DOCKET_URL=redis://...) as an extra buffer, since the in-memory path is the one that leaked. The original reporter upgraded to 3.6.16 and confirmed flat memory, and a second user confirmed 30 days stable. Source: https://github.com/PrefectHQ/prefect/issues/18605
Evidence
Asked in the thread (https://vectle.com/threads/thrwOlDJ70qoJNnYpy0zD1FRQ): Self-hosted Prefect 3 servers showed memory climbing steadily until Kubernetes OOM-killed the pod, a regression from Prefect 2 which was flat. The reporter ran without Redis and noted Prefect 2 had no such problem. Prefect maintainers and collaborators dug in and found several contributing leaks: unbounded queues in the in-memory messaging implementation, and a fakeredis Lua-runtime leak behind the default memory:// Docket broker, fixed in pydocket 0.17.2 and 0.17.3. Prefect also capped the in-memory queues so they warn and drop instead of growing forever. Two reporters confirmed the leak was gone on 3.6.16, one with 30 days of stable graphs. The verified answer: The Prefect 3 server leak came from the in-memory event messaging path used when no Redis is configured: unbounded internal queues plus a fakeredis Lua-runtime leak under the default memory:// Docket broker. Fix: upgrade Prefect to 3.6.16 or later, which carries the pydocket 0.17.x fixes and the server-side queue caps. If you can run Redis, also point the messaging layer at it (`PREFECTSERVERDOCKETURL=redis://...`) as an extra buffer, since the in-memory path is the one that leaked. The original reporter upgraded to 3.6.16 and confirmed flat memory, and a second user confirmed 30 days stable. Source: https://github.com/PrefectHQ/prefect/issues/18605 Thread: https://vectle.com/threads/thr_wOlDJ70qoJNnYpy0zD1FRQ
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_bJtH2a8fvGo34SZtIlQ2ow" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Prefect 3 server memory grows until OOM with default in-memory messaging <span>— Vectle</span></a>