Dagster Helm concurrency pools blocked by injected max_concurrent_runs setting

TL;DR

This was implemented by the maintainers: the Helm chart now exposes the concurrency block independently of the queued run coordinator, so you can set concurrency pools through additionalInstanceConfig without a custom run coordinator.

Steps

1. This was implemented by the maintainers: the Helm chart now exposes the concurrency block independently of the queued run coordinator, so you can set concurrency pools through additionalInstanceConfig without a custom run coordinator. If you are on an older chart version and hit the DagsterInvalidConfigError about maxconcurrentruns, that is the conflict: either upgrade the chart, or work around it by setting the concurrency config only in the instance YAML while leaving the run coordinator on defaults.

When to use

Use this skill when you run into "Dagster Helm concurrency pools blocked by injected maxconcurrentruns setting".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Source: GitHub issue dagster-io/dagster#31086 (closed, 9 comments): setting the concurrency pools block via additionalInstanceConfig in the Dagster Helm chart failed daemon startup with DagsterInvalidConfigError, because the Helm run-coordinator template always injected maxconcurrentruns, which is incompatible with the concurrency > runs config. There was no way to leave that field unset through Helm values, so concurrency limits could not be configured without defining a fully custom run coordinator.

Verified resolution: This was implemented by the maintainers: the Helm chart now exposes the concurrency block independently of the queued run coordinator, so you can set concurrency pools through additionalInstanceConfig without a custom run coordinator. If you are on an older chart version and hit the DagsterInvalidConfigError about maxconcurrentruns, that is the conflict: either upgrade the chart, or work around it by setting the concurrency config only in the instance YAML while leaving the run coordinator on defaults.

Source URL: https://github.com/dagster-io/dagster/issues/31086