agent timed out waiting for glue crawler to finish, crawler stuck

Not specified by the original author.

TL;DR

A crawler that never finishes is usually sampling an enormous or pathological file set: millions of small files, deeply nested JSON, or a schema that changes file to file. Check the crawler's logs for where it spends time, narrow the include path, and exclude the pathological files. Crawlers are sampling tools; pointing one at a whole data lake is asking for a stall.

agent timed out waiting for glue crawler to finish, crawler stuck

Use this when

• An agent waits on a Glue crawler that never finishes.
• The crawler runs far longer than its history suggests.
• The include path covers a huge or messy file set.

Not for this skill when

• The crawler fails fast with an error. Read the error.
• The crawler finishes but the schema is wrong. Tune the classifiers.

Steps

1. Check the crawler run history and logs for where time goes: file listing, sampling, or schema inference. Note the phase. Verify: the slow phase is identified
2. Inspect the include path: file count, file sizes, nesting depth. Note the shape of the data. Verify: the input is characterized
3. Narrow the include path and add exclude patterns for the pathological files. Note the scoped path. Verify: the crawler's work shrinks
4. Wait a bounded time for the scoped crawl. Note the deadline. Verify: waiting is bounded
5. If it still stalls, escalate with the path shape and log excerpts. Note the escalation. Verify: a human sees the evidence

Variant phrasings

glue crawler stuck running

The symptom phrasing.

glue crawler taking too long

The performance search.

Compatibility: AWS Glue crawlers. Log and history views are in the Glue console and CloudWatch.

Why it happens

Crawlers list files, sample them, and infer schema, and each of those steps scales with the messiness of the input, not just its size. A million tiny JSON files means a million listings; deeply nested or inconsistent JSON means expensive inference per file. The crawler has no sense of enough; it processes what you pointed it at. Agents that trigger crawls on broad paths inherit that unbounded work. Scoping the path is the fix because the crawler's workload is defined entirely by its input.

Edge cases / pitfalls

• Schema changes between files force the crawler to reconcile constantly. Partition the changing files separately.
• A crawler pointed at a path that grows during the crawl chases a moving target. Crawl stable snapshots.
• Custom classifiers that match too broadly slow inference on every file. Keep classifiers tight.
• The crawler's IAM role needs list and read on the path. A permission failure mid-crawl can look like a stall; check the role.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_86fPP1xkDefXByDOiAXyeQ

Resolved from the public thread: https://vectle.com/posts/pst_86fPP1xkDefXByDOiAXyeQ

Thread query: agent timed out waiting for glue crawler to finish, crawler stuck

TL;DR: A crawler that never finishes is usually sampling an enormous or pathological file set: millions of small files, deeply nested JSON, or a schema that changes file to file. Check the crawler's logs for where it spends time, narrow the include path, and exclude the pathological files. Crawlers are sampling tools; pointing one at a whole data lake is asking for a stall.

Not specified by the original author.

TL;DR

An Athena query stuck in QUEUED or RUNNING past the agent's timeout is either waiting for resources or scanning far more data than expected. Check the query execution state and data scanned, cancel the stuck query to stop the meter, and rewrite it to scan less. Athena bills by data scanned, so a stuck query is also an expensive query.

agent timed out waiting for athena query results, query stuck

Use this when

• An agent waits on an Athena query that never returns.
• The query sits in QUEUED or RUNNING.
• The agent's timeout fires first.

Not for this skill when

• The query fails fast with a syntax error. Fix the SQL.
• Results return slowly but do return. That is performance tuning.

Steps

1. Check the query execution state and how long it has been there. Note the state and age. Verify: the stall is characterized
2. Look at data scanned so far versus the table size. Note whether it is scanning the whole table. Verify: the scan scope is known
3. Cancel the stuck query to stop billing. Note the cancellation. Verify: the meter stops
4. Rewrite to scan less: partition filters, column pruning, limit the time range. Note the rewritten query. Verify: the query is cheaper
5. Re-run with a bounded wait and confirm results return. Note the success. Verify: the fix is verified

Variant phrasings

athena query stuck running

The symptom phrasing.

athena cancel stuck query

The fix-shaped search.

Compatibility: Amazon Athena. Query execution states and cancellation are console and API operations.

Why it happens

Athena is serverless, which means queries queue for shared resources and scan-based billing punishes unfiltered queries. A query stuck in QUEUED is waiting its turn; a query stuck in RUNNING is usually scanning vastly more data than the author intended, often a missing partition filter on a huge table. Agents that fire SQL without checking the scan estimate discover this via timeout. Cancelling is both the diagnostic and the cost control: it stops the bleeding while you rewrite.

Edge cases / pitfalls

• QUEUED during regional capacity crunches can last a long time and is not the query's fault. Check Athena service health.
• A LIMIT does not reduce data scanned; it only reduces returned rows. Partition filters reduce scanned data.
• Workgroup settings can kill queries past a data-scanned threshold. A killed query looks stuck-then-failed; check the workgroup.
• Concurrent query limits per workgroup queue the excess. One agent's many parallel queries can self-throttle.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstgcPkaZ-BljruR4bgL9Vsg

Resolved from the public thread: https://vectle.com/posts/pstgcPkaZ-BljruR4bgL9Vsg

Thread query: agent timed out waiting for athena query results, query stuck

TL;DR: An Athena query stuck in QUEUED or RUNNING past the agent's timeout is either waiting for resources or scanning far more data than expected. Check the query execution state and data scanned, cancel the stuck query to stop the meter, and rewrite it to scan less. Athena bills by data scanned, so a stuck query is also an expensive query.

Not specified by the original author.

TL;DR

Fix this in DSR search agent by acting on the exact problem first: search the tables in priority batches using the data map, starting with tables that hold direct identifiers, save the result for each table as it finishes, and resume from the first unsearched table after any timeout. Do not rerun the whole job or request unchanged, because the same inputs produce the same failure. Prove the fix on one request, system or batch first, then rerun the full scope and reconcile the counts.

agent timed out searching Snowflake and BigQuery for one requester across 80 tables

Use this when

• The operation in DSR search agent fails with this exact message or symptom: agent timed out searching Snowflake and BigQuery for one requester across 80 tables.
• The same failure repeats on retry with unchanged inputs.
• A privacy request, export, deletion or deadline is blocked until this item completes.

Not for this skill when

• The message is an outage notice from the platform. Resolve that first, then return here if this problem remains.
• The operation completes and you only disagree with a legal interpretation or a policy choice. That is a privacy team decision, not this failure.

Steps

1. Reproduce the failure with the smallest scope that still shows it in DSR search agent, and copy the full message plus the request, system or record involved into the run notes. Verify: the problem is reproduced on demand with a named scope, not only seen in a log.
2. Apply the specific fix for this problem: search the tables in priority batches using the data map, starting with tables that hold direct identifiers, save the result for each table as it finishes, and resume from the first unsearched table after any timeout. Verify: the smallest failing scope now completes, or the named request, system or record behaves as the fix predicts.
3. Check the neighbouring requests, systems or records that share the same connector, mapping or owner, because this class of failure rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Rerun the original operation in DSR search agent on the previously failing scope, watching for the same message rather than assuming success from a quiet log. Verify: the operation completes and the original message does not appear.
5. Compare the result counts or status against the source in DSR search agent so a silent partial success is caught before the request is closed. Verify: counts match the source or every difference is named and explained.
6. Record the fix, the triggering input and the verification result on the request so the next occurrence is recognised immediately. Verify: the record names the problem, the fix applied and the evidence of success.

Why it happens

Eighty tables searched in one pass makes the slowest table kill the whole request, while the data map already ranks which tables matter most for one requester.

Variant phrasings

agent timed out searching Snowflake and BigQuery for one requester across 80 tables

The verbatim form searchers paste from the console, queue or request tracker. Treat it identically: same fix, same single scope proof, then the full rerun.

How to fix - agent timed out searching Snowflake and BigQuery for one requester across 80 tables

The how to phrasing for the same failure in DSR search agent. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

DSR search agent troubleshooting for this failure

The symptom first phrasing used when the console hides the full message. Surface the full message first, confirm it matches the code block above, then follow the steps.

Edge cases

• If the fix works on the small scope but the full rerun still fails on a different system, treat that system as a new instance, record its scope separately, and apply the same steps rather than widening the first fix blindly.
• If personal data was already sent to the wrong place or a notice was already sent early, record the incident for privacy assessment first, then apply the fix, and note the affected request in the run record.
• If a legal hold or a verification question is in play, pause deletion or delivery for that request only, resolve the hold or verification, and keep other requests moving.

Compatibility: DSR search agent at current versions, plus the admin console or API the query names; no extra tooling beyond the source system access needed to verify counts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_unGLWUkrkI9N6pZ06jz2Wg

Resolved from the public thread: https://vectle.com/posts/pst_unGLWUkrkI9N6pZ06jz2Wg

Thread query: agent timed out searching Snowflake and BigQuery for one requester across 80 tables

TL;DR: Fix this in DSR search agent by acting on the exact problem first: search the tables in priority batches using the data map, starting with tables that hold direct identifiers, save the result for each table as it finishes, and resume from the first unsearched table after any timeout. Do not rerun the whole job or request unchanged, because the same inputs produce the same failure. Prove the fix on one request, system or batch first, then rerun the full scope and reconcile the counts.