agent timed out waiting for EMR spark job, job stuck

TL;DR

A Spark job that never finishes is stuck in a stage: data skew, a hanging external call, or executor loss loops. The agent should check the Spark UI for the stuck stage and its task distribution, wait a bounded time for slow-but-moving stages, and escalate with the stage named. Spark jobs rarely hang for no reason; the UI shows the reason.

agent timed out waiting for EMR spark job, job stuck

Use this when

• An agent waits on an EMR Spark job that stalls.
• Stages show no progress over multiple checks.
• The agent's timeout fires first.

Not for this skill when

• The job fails fast with an exception. Read the driver logs.
• The job is slow but progressing. That is a tuning question.

Steps

1. Open the Spark UI and identify the stuck stage and its task states. Note the stage. Verify: the stuck stage is named
2. Check task distribution: a few slow tasks means skew; all tasks stuck means a systemic block. Note the pattern. Verify: skew versus systemic is decided
3. Look at executor health: lost executors, blacklisted nodes, or spot terminations. Note the executor state. Verify: infrastructure is checked
4. Wait a bounded time if tasks are moving, even slowly. Note the deadline. Verify: slow progress gets patience, not panic
5. Escalate with the stage, the pattern, and the job id. Note the escalation. Verify: a human gets a precise report

Variant phrasings

spark job stuck stage

The Spark-shaped search.

emr spark job hanging

The symptom phrasing.

Compatibility: Spark on EMR. The Spark UI and stage model are version-independent.

Why it happens

Spark exposes its execution as stages and tasks, which makes hangs diagnosable: the stuck stage is visible, and the task distribution across it tells you whether the problem is skew (a few tasks with all the data) or systemic (everything blocked on a shuffle service, a metastore, or lost executors). Agents that only watch the job state see stuck; agents that read the stage see why. The EMR layer adds spot-instance churn as a classic cause of executor loss loops.

Edge cases / pitfalls

• Data skew looks like a hang on the last few tasks. It is slow, not stuck; the fix is salting or repartitioning, not waiting.
• Spot terminations during shuffles can loop executor loss forever on aggressive bid strategies. Check the instance fleet.
• A stage blocked on an external call (a slow API per row) hangs all tasks identically. Look for per-row external I/O.
• Killing a stuck Spark job is safe but loses the shuffle data. Prefer diagnosis before killing when the job is expensive.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_XTgYtIUv9d0bm6qRGgj2Sg