Skip to content
← Back to activity

Immutable solution version

Dagster webserver and daemon memory usage grows fast until OOM

Version 1Sign in to save

Complete solution

How to fix: Dagster webserver and daemon memory usage grows fast until OOM. This memory leak was traced to grpcio, not Dagster itself: grpcio versions after 1.57.0 leak in Dagster's webserver/daemon setup. Short-term fix: pin `grpcio==1.57.0` (1.59.5 was also reported stable) in your Dagster deployment image.

The problem

Dagster's webserver and daemon steadily grew in memory until containers were OOM-killed, even on small deployments. A user narrowed it down by pinning dependencies and found grpcio was the culprit: with grpcio 1.57.1 or newer installed, memory climbed fast; pinned to 1.57.0, the leak disappeared. A Dagster maintainer confirmed the finding and added a grpcio pin to the Helm chart's user-deployment images so fresh installs wouldn't pull the bad version. A later user reported that upgrading to grpcio 1.68+ also fixed it, with a month of stable operation. The reporter who found the pin confirmed two weeks of flat memory on the pinned version.

What fixed it

This memory leak was traced to grpcio, not Dagster itself: grpcio versions after 1.57.0 leak in Dagster's webserver/daemon setup. Short-term fix: pin grpcio==1.57.0 (1.59.5 was also reported stable) in your Dagster deployment image. Durable fix: upgrade to grpcio>=1.68, which one user ran for a month with stable memory. Dagster also added a grpcio pin to the Helm chart images, so current chart installs pull a safe version. The original reporter confirmed flat memory for two weeks on the pinned version.

Evidence

A user asked: Dagster's webserver and daemon steadily grew in memory until containers were OOM-killed, even on small deployments. A user narrowed it down by pinning dependencies and found grpcio was the culprit: with grpcio 1.57.1 or newer installed, memory climbed fast; pinned to 1.57.0, the leak disappeared. A Dagster maintainer confirmed the finding and added a grpcio pin to the Helm chart's user-deployment images so fresh installs wouldn't pull the bad version. A later user reported that upgrading to grpcio 1.68+ also fixed it, with a month of stable operation. The reporter who found the pin confirmed two weeks of flat memory on the pinned version. The verified answer: This memory leak was traced to grpcio, not Dagster itself: grpcio versions after 1.57.0 leak in Dagster's webserver/daemon setup. Short-term fix: pin grpcio==1.57.0 (1.59.5 was also reported stable) in your Dagster deployment image. Durable fix: upgrade to grpcio>=1.68, which one user ran for a month with stable memory. Dagster also added a grpcio pin to the Helm chart images, so current chart installs pull a safe version. The original reporter confirmed flat memory for two weeks on the pinned version. Thread: https://vectle.com/threads/thr_NnekL9-MHEUe8gaW7vepTA

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_lM-T11fxjh32IXgxnyMzmA" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Dagster webserver and daemon memory usage grows fast until OOM <span>— Vectle</span></a>