# celery error: task latency growing - how to find stuck workers

## TL;DR

Growing task latency with a healthy queue means workers are stuck, not starved. Use celery inspect to find workers with long-running tasks, then check what those tasks are blocked on. The usual causes are a hung downstream call with no timeout or a worker concurrency setting that serializes everything.

```text
celery: task latency growing (tasks take longer end-to-end, queue depth normal)
```

## Steps

1. List active tasks per worker: celery -A [app] inspect active.
   Expected: You see which workers hold tasks and for how long.

2. Find the stuck ones: tasks active far longer than their p99 are the suspects. Note their args and the worker hostname.
   Expected: A few tasks stand out as stuck.

3. Check what they are blocked on: thread dumps (py-spy on the worker PID) or the task logs. A downstream HTTP call or DB query with no timeout is the classic.
   Expected: You see the blocking call in the stack.

4. Fix the block: add timeouts to every downstream call, and set task_time_limit / task_soft_time_limit so no task can hang forever.
   Expected: Stuck tasks die loudly instead of silently.

5. Right-size concurrency: a prefork worker with concurrency=1 and long tasks serializes everything; match the pool to the workload (more processes for CPU, gevent/eventlet for IO).
   Expected: Latency drops because tasks actually run in parallel.

## When to use

- celery task latency grows while the queue looks fine
- workers show tasks active for far too long
- you suspect hung downstream calls

## When not to use

- the queue itself is backing up (that is a producer/consumer rate problem)
- tasks fail fast with errors (that is a code bug, not stuck workers)
- the broker is the bottleneck (check the broker first)

## Tool compatibility

- Celery 5.x
- any broker (Redis/RabbitMQ)

## Variant phrasings

- "celery task latency growing"
- "celery find stuck workers"
- "celery inspect active long running tasks"

## Why it happens

Task latency measures enqueue-to-complete. When the queue is short but latency grows, the time is spent inside workers: a task blocked on a downstream call with no timeout, or a concurrency model that cannot parallelize the workload. celery inspect exposes the active set; the stacks explain the wait.

## Edge cases

- task_time_limit kills with SIGKILL-adjacent force; make tasks idempotent before enforcing it
- py-spy needs ptrace permission on the worker host; arrange access before the incident
- the soft time limit raises an exception the task can catch to clean up; the hard limit does not ask

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_AVLE9X4r575yAT-DIv-6-A