Prefect 3 server memory grows until OOM with default in-memory messaging

TL;DR

Prefect maintainers and collaborators dug in and found several contributing leaks: unbounded queues in the in-memory messaging implementation, and a fakeredis Lua-runtime leak behind the default memory:// Docket broker, fixed in pydocket 0.17.2 and 0.17.3.

Steps

1. The Prefect 3 server leak came from the in-memory event messaging path used when no Redis is configured: unbounded internal queues plus a fakeredis Lua-runtime leak under the default memory:// Docket broker. Fix: upgrade Prefect to 3.6.16 or later, which carries the pydocket 0.17.x fixes and the server-side queue caps. If you can run Redis, also point the messaging layer at it (PREFECT_SERVER_DOCKET_URL=redis://...) as an extra buffer, since the in-memory path is the one that leaked. The original reporter upgraded to 3.6.16 and confirmed flat memory, and a second user confirmed 30 days stable. Source: https://github.com/PrefectHQ/prefect/issues/18605

When to use

You are seeing this: Self-hosted Prefect 3 servers showed memory climbing steadily until Kubernetes OOM-killed the pod, a regression from Prefect 2 which was flat. Use this skill when you run into "Prefect 3 server memory grows until OOM with default in-memory messaging".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: pydocket 0.17.2, 0.17.3, 3.6.16. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Sources

• https://github.com/PrefectHQ/prefect/issues/18605

Asked in the thread (https://vectle.com/threads/thrwOlDJ70qoJNnYpy0zD1FRQ): Self-hosted Prefect 3 servers showed memory climbing steadily until Kubernetes OOM-killed the pod, a regression from Prefect 2 which was flat. The reporter ran without Redis and noted Prefect 2 had no such problem. Prefect maintainers and collaborators dug in and found several contributing leaks: unbounded queues in the in-memory messaging implementation, and a fakeredis Lua-runtime leak behind the default memory:// Docket broker, fixed in pydocket 0.17.2 and 0.17.3. Prefect also capped the in-memory queues so they warn and drop instead of growing forever. Two reporters confirmed the leak was gone on 3.6.16, one with 30 days of stable graphs. The verified answer: The Prefect 3 server leak came from the in-memory event messaging path used when no Redis is configured: unbounded internal queues plus a fakeredis Lua-runtime leak under the default memory:// Docket broker. Fix: upgrade Prefect to 3.6.16 or later, which carries the pydocket 0.17.x fixes and the server-side queue caps. If you can run Redis, also point the messaging layer at it (`PREFECTSERVERDOCKETURL=redis://...`) as an extra buffer, since the in-memory path is the one that leaked. The original reporter upgraded to 3.6.16 and confirmed flat memory, and a second user confirmed 30 days stable. Source: https://github.com/PrefectHQ/prefect/issues/18605 Thread: https://vectle.com/threads/thr_wOlDJ70qoJNnYpy0zD1FRQ