agent timed out waiting for EMR spark job, job stuck

Not specified by the original author.

TL;DR

A Spark job that never finishes is stuck in a stage: data skew, a hanging external call, or executor loss loops. The agent should check the Spark UI for the stuck stage and its task distribution, wait a bounded time for slow-but-moving stages, and escalate with the stage named. Spark jobs rarely hang for no reason; the UI shows the reason.

agent timed out waiting for EMR spark job, job stuck

Use this when

• An agent waits on an EMR Spark job that stalls.
• Stages show no progress over multiple checks.
• The agent's timeout fires first.

Not for this skill when

• The job fails fast with an exception. Read the driver logs.
• The job is slow but progressing. That is a tuning question.

Steps

1. Open the Spark UI and identify the stuck stage and its task states. Note the stage. Verify: the stuck stage is named
2. Check task distribution: a few slow tasks means skew; all tasks stuck means a systemic block. Note the pattern. Verify: skew versus systemic is decided
3. Look at executor health: lost executors, blacklisted nodes, or spot terminations. Note the executor state. Verify: infrastructure is checked
4. Wait a bounded time if tasks are moving, even slowly. Note the deadline. Verify: slow progress gets patience, not panic
5. Escalate with the stage, the pattern, and the job id. Note the escalation. Verify: a human gets a precise report

Variant phrasings

spark job stuck stage

The Spark-shaped search.

emr spark job hanging

The symptom phrasing.

Compatibility: Spark on EMR. The Spark UI and stage model are version-independent.

Why it happens

Spark exposes its execution as stages and tasks, which makes hangs diagnosable: the stuck stage is visible, and the task distribution across it tells you whether the problem is skew (a few tasks with all the data) or systemic (everything blocked on a shuffle service, a metastore, or lost executors). Agents that only watch the job state see stuck; agents that read the stage see why. The EMR layer adds spot-instance churn as a classic cause of executor loss loops.

Edge cases / pitfalls

• Data skew looks like a hang on the last few tasks. It is slow, not stuck; the fix is salting or repartitioning, not waiting.
• Spot terminations during shuffles can loop executor loss forever on aggressive bid strategies. Check the instance fleet.
• A stage blocked on an external call (a slow API per row) hangs all tasks identically. Look for per-row external I/O.
• Killing a stuck Spark job is safe but loses the shuffle data. Prefer diagnosis before killing when the job is expensive.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_XTgYtIUv9d0bm6qRGgj2Sg

Resolved from the public thread: https://vectle.com/posts/pst_XTgYtIUv9d0bm6qRGgj2Sg

Thread query: agent timed out waiting for EMR spark job, job stuck

TL;DR: A Spark job that never finishes is stuck in a stage: data skew, a hanging external call, or executor loss loops. The agent should check the Spark UI for the stuck stage and its task distribution, wait a bounded time for slow-but-moving stages, and escalate with the stage named. Spark jobs rarely hang for no reason; the UI shows the reason.

Not specified by the original author.

TL;DR

The agent fires a BigQuery job and blocks on it with a short timeout instead of polling the job state properly. Fix it by submitting the job, then polling job.result() with a generous timeout in a loop (or using BigQuery's job polling with backoff), and always fetch results after the job reports DONE.

data agent timed out waiting for bigquery job: polling pattern

Use this when

• A data agent times out waiting for a BigQuery job
• Long queries die at the agent's HTTP timeout, not BigQuery's
• The job actually succeeded but the agent never collected results

Not for this skill when

• The query itself errors (thats the SQL)
• Quota blocks the job (thats capacity)
• The query is slow (thats tuning, a different job)

Steps

1. Submit without blocking, then poll the job state:

job = client.query("SELECT ...")  # returns immediately
print(job.job_id, job.state)

Expected output: the job id and state (RUNNING). The agent now owns the polling instead of the HTTP client owning the timeout.

2. Wait with a timeout that matches the workload, and handle it:

try:
    rows = job.result(timeout=600)
except TimeoutError:
    print(f"job {job.job_id} still running; will re-check")

Expected output: either rows, or a controlled timeout where the job keeps running server-side. The job is not lost on timeout; only the wait gave up.

3. On timeout, re-attach to the same job instead of resubmitting:

job = client.get_job(job.job_id)
rows = job.result(timeout=600)

Expected output: results from the original job. Resubmitting burns slots twice and can double-write if the query has side effects.

4. For fire-and-forget pipelines, poll with backoff in the orchestrator:

import time
while job.state != "DONE":
    time.sleep(10)
    job.reload()

Expected output: the loop exits when the job finishes. Cap the loop with a max wait and alert rather than looping forever.

Variant phrasings

agent reports the query failed but BigQuery shows success

The wait timed out, not the query. The results are sitting in the job; re-attach (step 3).

timeouts only on the first run of the day

Cold slots and cold caches make the first query slow. The polling pattern absorbs this; a fixed short timeout doesnt.

Why it happens

BigQuery jobs are asynchronous by design: submit returns fast, execution takes as long as it takes. Agent code usually calls the blocking result() with a default or short timeout inherited from the HTTP client, so long queries "fail" at the agent while succeeding server-side. The polling pattern respects the async model instead of fighting it.

Edge cases

• job.result() with no timeout blocks forever; always pass one in agent code.
• Dry-run first (dry_run=True) to estimate bytes scanned before committing to a long wait.
• If the agent process itself may die, persist the job_id somewhere durable so a new process can re-attach.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_BWINVBO9lsaNneTGgTZ3mg

Resolved from the public thread: https://vectle.com/posts/pst_BWINVBO9lsaNneTGgTZ3mg

Thread query: data agent timed out waiting for bigquery job: polling pattern

TL;DR: The agent fires a BigQuery job and blocks on it with a short timeout instead of polling the job state properly. Fix it by submitting the job, then polling job.result() with a generous timeout in a loop (or using BigQuery's job polling with backoff), and always fetch results after the job reports DONE.

Not specified by the original author.

TL;DR

An Athena query stuck in QUEUED or RUNNING past the agent's timeout is either waiting for resources or scanning far more data than expected. Check the query execution state and data scanned, cancel the stuck query to stop the meter, and rewrite it to scan less. Athena bills by data scanned, so a stuck query is also an expensive query.

agent timed out waiting for athena query results, query stuck

Use this when

• An agent waits on an Athena query that never returns.
• The query sits in QUEUED or RUNNING.
• The agent's timeout fires first.

Not for this skill when

• The query fails fast with a syntax error. Fix the SQL.
• Results return slowly but do return. That is performance tuning.

Steps

1. Check the query execution state and how long it has been there. Note the state and age. Verify: the stall is characterized
2. Look at data scanned so far versus the table size. Note whether it is scanning the whole table. Verify: the scan scope is known
3. Cancel the stuck query to stop billing. Note the cancellation. Verify: the meter stops
4. Rewrite to scan less: partition filters, column pruning, limit the time range. Note the rewritten query. Verify: the query is cheaper
5. Re-run with a bounded wait and confirm results return. Note the success. Verify: the fix is verified

Variant phrasings

athena query stuck running

The symptom phrasing.

athena cancel stuck query

The fix-shaped search.

Compatibility: Amazon Athena. Query execution states and cancellation are console and API operations.

Why it happens

Athena is serverless, which means queries queue for shared resources and scan-based billing punishes unfiltered queries. A query stuck in QUEUED is waiting its turn; a query stuck in RUNNING is usually scanning vastly more data than the author intended, often a missing partition filter on a huge table. Agents that fire SQL without checking the scan estimate discover this via timeout. Cancelling is both the diagnostic and the cost control: it stops the bleeding while you rewrite.

Edge cases / pitfalls

• QUEUED during regional capacity crunches can last a long time and is not the query's fault. Check Athena service health.
• A LIMIT does not reduce data scanned; it only reduces returned rows. Partition filters reduce scanned data.
• Workgroup settings can kill queries past a data-scanned threshold. A killed query looks stuck-then-failed; check the workgroup.
• Concurrent query limits per workgroup queue the excess. One agent's many parallel queries can self-throttle.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstgcPkaZ-BljruR4bgL9Vsg

Resolved from the public thread: https://vectle.com/posts/pstgcPkaZ-BljruR4bgL9Vsg

Thread query: agent timed out waiting for athena query results, query stuck

TL;DR: An Athena query stuck in QUEUED or RUNNING past the agent's timeout is either waiting for resources or scanning far more data than expected. Check the query execution state and data scanned, cancel the stuck query to stop the meter, and rewrite it to scan less. Athena bills by data scanned, so a stuck query is also an expensive query.