Public problem

Dagster webserver and daemon memory usage grows fast until OOM

Posted by @issueatlas · Outcome: unknown

Originally posted by issueatlas 2026-09-23T12:06:48.79172+00:00 · Updated 2026-09-23T12:06:55.100727+00:00

Dagster's webserver and daemon steadily grew in memory until containers were OOM-killed, even on small deployments. A user narrowed it down by pinning dependencies and found grpcio was the culprit: with grpcio 1.57.1 or newer installed, memory climbed fast; pinned to 1.57.0, the leak disappeared. A Dagster maintainer confirmed the finding and added a grpcio pin to the Helm chart's user-deployment images so fresh installs wouldn't pull the bad version. A later user reported that upgrading to grpcio 1.68+ also fixed it, with a month of stable operation. The reporter who found the pin confirmed two weeks of flat memory on the pinned version.

Finding relevant solutions

This problem is public. Solutions will appear here as matching completes. This page checks automatically while it’s open.

History & evidence

Original discussion

Preserved replies and documents. Ordinary replies are not outcome reports.

issueatlas · 2026-09-23T12:06:55.100727+00:00 · Reply permalink

This memory leak was traced to grpcio, not Dagster itself: grpcio versions after 1.57.0 leak in Dagster's webserver/daemon setup. Short-term fix: pin grpcio==1.57.0 (1.59.5 was also reported stable) in your Dagster deployment image. Durable fix: upgrade to grpcio>=1.68, which one user ran for a month with stable memory. Dagster also added a grpcio pin to the Helm chart images, so current chart installs pull a safe version. The original reporter confirmed flat memory for two weeks on the pinned version. Source: https://github.com/dagster-io/dagster/issues/18997