agent timed out waiting for glue crawler to finish, crawler stuck

TL;DR

A crawler that never finishes is usually sampling an enormous or pathological file set: millions of small files, deeply nested JSON, or a schema that changes file to file. Check the crawler's logs for where it spends time, narrow the include path, and exclude the pathological files. Crawlers are sampling tools; pointing one at a whole data lake is asking for a stall.

agent timed out waiting for glue crawler to finish, crawler stuck

Use this when

• An agent waits on a Glue crawler that never finishes.
• The crawler runs far longer than its history suggests.
• The include path covers a huge or messy file set.

Not for this skill when

• The crawler fails fast with an error. Read the error.
• The crawler finishes but the schema is wrong. Tune the classifiers.

Steps

1. Check the crawler run history and logs for where time goes: file listing, sampling, or schema inference. Note the phase. Verify: the slow phase is identified
2. Inspect the include path: file count, file sizes, nesting depth. Note the shape of the data. Verify: the input is characterized
3. Narrow the include path and add exclude patterns for the pathological files. Note the scoped path. Verify: the crawler's work shrinks
4. Wait a bounded time for the scoped crawl. Note the deadline. Verify: waiting is bounded
5. If it still stalls, escalate with the path shape and log excerpts. Note the escalation. Verify: a human sees the evidence

Variant phrasings

glue crawler stuck running

The symptom phrasing.

glue crawler taking too long

The performance search.

Compatibility: AWS Glue crawlers. Log and history views are in the Glue console and CloudWatch.

Why it happens

Crawlers list files, sample them, and infer schema, and each of those steps scales with the messiness of the input, not just its size. A million tiny JSON files means a million listings; deeply nested or inconsistent JSON means expensive inference per file. The crawler has no sense of enough; it processes what you pointed it at. Agents that trigger crawls on broad paths inherit that unbounded work. Scoping the path is the fix because the crawler's workload is defined entirely by its input.

Edge cases / pitfalls

• Schema changes between files force the crawler to reconcile constantly. Partition the changing files separately.
• A crawler pointed at a path that grows during the crawl chases a moving target. Crawl stable snapshots.
• Custom classifiers that match too broadly slow inference on every file. Keep classifiers tight.
• The crawler's IAM role needs list and read on the path. A permission failure mid-crawl can look like a stall; check the role.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_86fPP1xkDefXByDOiAXyeQ