Immutable solution version
Dagster webserver and daemon memory usage grows fast until OOM
Complete solution
How to fix: Dagster webserver and daemon memory usage grows fast until OOM. This memory leak was traced to grpcio, not Dagster itself: grpcio versions after 1.57.0 leak in Dagster's webserver/daemon setup. Short-term fix: pin `grpcio==1.57.0` (1.59.5 was also reported stable) in your Dagster deployment image.
The problem
Dagster's webserver and daemon steadily grew in memory until containers were OOM-killed, even on small deployments. A user narrowed it down by pinning dependencies and found grpcio was the culprit: with grpcio 1.57.1 or newer installed, memory climbed fast; pinned to 1.57.0, the leak disappeared. A Dagster maintainer confirmed the finding and added a grpcio pin to the Helm chart's user-deployment images so fresh installs wouldn't pull the bad version. A later user reported that upgrading to grpcio 1.68+ also fixed it, with a month of stable operation. The reporter who found the pin confirmed two weeks of flat memory on the pinned version.
What fixed it
This memory leak was traced to grpcio, not Dagster itself: grpcio versions after 1.57.0 leak in Dagster's webserver/daemon setup. Short-term fix: pin grpcio==1.57.0 (1.59.5 was also reported stable) in your Dagster deployment image. Durable fix: upgrade to grpcio>=1.68, which one user ran for a month with stable memory. Dagster also added a grpcio pin to the Helm chart images, so current chart installs pull a safe version. The original reporter confirmed flat memory for two weeks on the pinned version.
Evidence
A user asked: Dagster's webserver and daemon steadily grew in memory until containers were OOM-killed, even on small deployments. A user narrowed it down by pinning dependencies and found grpcio was the culprit: with grpcio 1.57.1 or newer installed, memory climbed fast; pinned to 1.57.0, the leak disappeared. A Dagster maintainer confirmed the finding and added a grpcio pin to the Helm chart's user-deployment images so fresh installs wouldn't pull the bad version. A later user reported that upgrading to grpcio 1.68+ also fixed it, with a month of stable operation. The reporter who found the pin confirmed two weeks of flat memory on the pinned version. The verified answer: This memory leak was traced to grpcio, not Dagster itself: grpcio versions after 1.57.0 leak in Dagster's webserver/daemon setup. Short-term fix: pin grpcio==1.57.0 (1.59.5 was also reported stable) in your Dagster deployment image. Durable fix: upgrade to grpcio>=1.68, which one user ran for a month with stable memory. Dagster also added a grpcio pin to the Helm chart images, so current chart installs pull a safe version. The original reporter confirmed flat memory for two weeks on the pinned version. Thread: https://vectle.com/threads/thr_NnekL9-MHEUe8gaW7vepTA
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_lM-T11fxjh32IXgxnyMzmA" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Dagster webserver and daemon memory usage grows fast until OOM <span>— Vectle</span></a>