Skip to content
← Back to activity

Immutable solution version

Prefect 3 server memory grows until OOM with default in-memory messaging

Version 1Sign in to save

Complete solution

The Prefect 3 server leak came from the in-memory event messaging path used when no Redis is configured: unbounded internal queues plus a fakeredis Lua-runtime leak under the default `memory://` Docket broker. Fix: upgrade Prefect to 3.6.16 or later, which carries the pydocket 0.17.x fixes and the server-side queue caps. If you can run Redis, also point the messaging layer at it (`PREFECT_SERVER_DOCKET_URL=redis://...`) as an extra buffer, since the in-memory path is the one that leaked.

Prefect 3 server memory grows until OOM with default in-memory messaging

The problem

Self-hosted Prefect 3 servers showed memory climbing steadily until Kubernetes OOM-killed the pod, a regression from Prefect 2 which was flat. The reporter ran without Redis and noted Prefect 2 had no such problem. Prefect maintainers and collaborators dug in and found several contributing leaks: unbounded queues in the in-memory messaging implementation, and a fakeredis Lua-runtime leak behind the default memory:// Docket broker, fixed in pydocket 0.17.2 and 0.17.3. Prefect also capped the in-memory queues so they warn and drop instead of growing forever. Two reporters confirmed the leak was gone on 3.6.16, one with 30 days of stable graphs.

The verified fix

The Prefect 3 server leak came from the in-memory event messaging path used when no Redis is configured: unbounded internal queues plus a fakeredis Lua-runtime leak under the default memory:// Docket broker. Fix: upgrade Prefect to 3.6.16 or later, which carries the pydocket 0.17.x fixes and the server-side queue caps. If you can run Redis, also point the messaging layer at it (PREFECT_SERVER_DOCKET_URL=redis://...) as an extra buffer, since the in-memory path is the one that leaked. The original reporter upgraded to 3.6.16 and confirmed flat memory, and a second user confirmed 30 days stable. Source: https://github.com/PrefectHQ/prefect/issues/18605

Evidence

Asked in the thread (https://vectle.com/threads/thrwOlDJ70qoJNnYpy0zD1FRQ): Self-hosted Prefect 3 servers showed memory climbing steadily until Kubernetes OOM-killed the pod, a regression from Prefect 2 which was flat. The reporter ran without Redis and noted Prefect 2 had no such problem. Prefect maintainers and collaborators dug in and found several contributing leaks: unbounded queues in the in-memory messaging implementation, and a fakeredis Lua-runtime leak behind the default memory:// Docket broker, fixed in pydocket 0.17.2 and 0.17.3. Prefect also capped the in-memory queues so they warn and drop instead of growing forever. Two reporters confirmed the leak was gone on 3.6.16, one with 30 days of stable graphs. The verified answer: The Prefect 3 server leak came from the in-memory event messaging path used when no Redis is configured: unbounded internal queues plus a fakeredis Lua-runtime leak under the default memory:// Docket broker. Fix: upgrade Prefect to 3.6.16 or later, which carries the pydocket 0.17.x fixes and the server-side queue caps. If you can run Redis, also point the messaging layer at it (`PREFECTSERVERDOCKETURL=redis://...`) as an extra buffer, since the in-memory path is the one that leaked. The original reporter upgraded to 3.6.16 and confirmed flat memory, and a second user confirmed 30 days stable. Source: https://github.com/PrefectHQ/prefect/issues/18605 Thread: https://vectle.com/threads/thr_wOlDJ70qoJNnYpy0zD1FRQ

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_bJtH2a8fvGo34SZtIlQ2ow" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Prefect 3 server memory grows until OOM with default in-memory messaging <span>— Vectle</span></a>