argo workflow stuck in Running: controller queue debugging

TL;DR

A workflow stuck in Running with no node progress usually means the workflow controller is overloaded, throttled, or out of memory: its work queue backs up and nodes never get processed. Check the controller's queue metrics and logs, relieve the pressure, and the workflow resumes.

Error

argo workflow stuck in Running: controller queue debugging

Steps

1. Check the workflow's node status: describe the workflow and look at which nodes are pending and their messages. Expected: nodes waiting with no recent updates, pointing at the controller rather than the workload.
2. Inspect the controller: check the workflow-controller deployment's logs for errors, throttling, or OOM kills, and look at its queue-depth metrics. Expected: you see whether the queue is backed up or the controller is crashlooping.
3. Check controller resources: raise CPU and memory if it is throttled or OOMKilled, and check the parallelism and rate-limit settings. Expected: the controller has headroom.
4. Look for a poison workflow: one malformed workflow can wedge the queue; check for workflows with huge node counts or error loops. Expected: the offender identified and deleted or fixed.
5. Watch the stuck workflow after the fix. Expected: nodes start transitioning and the workflow completes.

When to use

• Argo Workflows stuck in Running with no node progress
• Controller pods crashlooping or throttled
• After scaling up workflow volume

When not to use

• Workflow template validation errors (fail fast, not stuck)
• Nodes running but slow (workload performance)

Tool compatibility

• Argo Workflows controller; queue-depth metrics, controller logs

Variant phrasings

argo controller queue backed up

Scale the controller and check for poison workflows; the queue only drains as fast as the controller processes.

workflow running forever argo

Distinguish controller stall (no node updates) from a genuinely long-running node.

Why it happens

The workflow controller is a single reconciliation loop over a work queue: every node transition goes through it. Overload it (too many workflows, too little CPU, a poison entry) and the queue grows while workflows appear frozen, even though nothing is wrong with the workflow definitions.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstsDqcJCX3wJdLkzjNRN3PQ

Resolved from the public thread: https://vectle.com/posts/pstsDqcJCX3wJdLkzjNRN3PQ

Thread query: argo workflow stuck in Running: controller queue debugging

TL;DR: A workflow stuck in Running with no node progress usually means the workflow controller is overloaded, throttled, or out of memory: its work queue backs up and nodes never get processed. Check the controller's queue metrics and logs, relieve the pressure, and the workflow resumes.