agent's CUR Athena query scanned the entire bucket, it forgot the year and month partition filters

TL;DR

The CUR dataset is partitioned by year and month; a query without those filters scans everything ever written. Add WHERE year='2026' AND month='10' (and day, when the table has it) to every query, and check the bytes-scanned figure in query history after each run.

agent's cur athena query scanned the entire bucket, it forgot the year and month partition filters

Use this when

• Athena CUR queries scan gigabytes for what should be a small question.
• The query history shows bytes scanned far above the data the question needs.
• A generated query template omits partition columns.

Not for this skill when

• The question genuinely spans all history (a multi-year trend). Then the full scan is the cost of the question; consider an aggregated table instead.
• You query a pre-aggregated summary table rather than the raw CUR.

Steps

1. Check the table DDL: SHOW CREATE TABLE cur_table. Verify: confirm the partition columns (year, month, and possibly day).
2. Add the partition filters to the WHERE clause for the exact period the question needs. Verify: no query ships without them.
3. Run the query and read bytes scanned in the Athena console or query history. Verify: it matches roughly the period's data size, not the bucket's.
4. Set a per-query data limit on the Athena workgroup as a backstop against future filter-less queries.
5. Fix the query template or generator so partition filters are mandatory, not optional.
6. For repeated questions, build a monthly aggregated table once and query that instead of the raw CUR.

Variant phrasings

athena query scanned entire s3 bucket cur

Symptom-level search.

cur athena partition filter year month

The fix framing.

reduce athena bytes scanned cost usage report

The cost-side phrasing.

Compatibility: Athena on the AWS Cost and Usage Report; partition columns depend on the CUR version and Athena table setup.

Why it happens

Partition columns look like ordinary columns, so generated SQL treats them as optional filters. They are not ordinary: they are the only thing standing between a monthly question and a scan of every month ever written. One missing WHERE clause multiplies the bill by the age of the dataset.

Edge cases / pitfalls

• The partition values are strings, not integers: year='2026', not year=2026. The wrong type silently scans everything.
• Some CUR setups add a day partition; check the DDL rather than assuming year/month only.
• MSCK REPAIR or partition projection misconfiguration can make even filtered queries scan more; verify after schema changes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_lp-31AWL4OSUbKCy3Pd8bQ