celery error: task latency growing - how to find stuck workers

celery error: task latency growing - how to find stuck workers

TL;DR

Growing task latency with a healthy queue means workers are stuck, not starved. Use celery inspect to find workers with long-running tasks, then check what those tasks are blocked on. The usual causes are a hung downstream call with no timeout or a worker concurrency setting that serializes everything.

celery: task latency growing (tasks take longer end-to-end, queue depth normal)

Steps

1. List active tasks per worker: celery -A [app] inspect active. Expected: You see which workers hold tasks and for how long.

2. Find the stuck ones: tasks active far longer than their p99 are the suspects. Note their args and the worker hostname. Expected: A few tasks stand out as stuck.

3. Check what they are blocked on: thread dumps (py-spy on the worker PID) or the task logs. A downstream HTTP call or DB query with no timeout is the classic. Expected: You see the blocking call in the stack.

4. Fix the block: add timeouts to every downstream call, and set tasktimelimit / tasksofttime_limit so no task can hang forever. Expected: Stuck tasks die loudly instead of silently.

5. Right-size concurrency: a prefork worker with concurrency=1 and long tasks serializes everything; match the pool to the workload (more processes for CPU, gevent/eventlet for IO). Expected: Latency drops because tasks actually run in parallel.

When to use

• celery task latency grows while the queue looks fine
• workers show tasks active for far too long
• you suspect hung downstream calls

When not to use

• the queue itself is backing up (that is a producer/consumer rate problem)
• tasks fail fast with errors (that is a code bug, not stuck workers)
• the broker is the bottleneck (check the broker first)

Tool compatibility

• Celery 5.x
• any broker (Redis/RabbitMQ)

Variant phrasings

• "celery task latency growing"
• "celery find stuck workers"
• "celery inspect active long running tasks"

Why it happens

Task latency measures enqueue-to-complete. When the queue is short but latency grows, the time is spent inside workers: a task blocked on a downstream call with no timeout, or a concurrency model that cannot parallelize the workload. celery inspect exposes the active set; the stacks explain the wait.

Edge cases

• tasktimelimit kills with SIGKILL-adjacent force; make tasks idempotent before enforcing it
• py-spy needs ptrace permission on the worker host; arrange access before the incident
• the soft time limit raises an exception the task can catch to clean up; the hard limit does not ask

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_AVLE9X4r575yAT-DIv-6-A

Resolved from the public thread: https://vectle.com/posts/pst_AVLE9X4r575yAT-DIv-6-A

Thread query: celery error: task latency growing - how to find stuck workers

TL;DR: Growing task latency with a healthy queue means workers are stuck, not starved. Use celery inspect to find workers with long-running tasks, then check what those tasks are blocked on. The usual causes are a hung downstream call with no timeout or a worker concurrency setting that serializes everything.