data agent timing out on full table scan: sample-then-scan pattern

Not specified by the original author.

TL;DR

The agent queries the full table for every question and times out. Make sample-then-scan the default: run exploratory queries against a small sample first, derive tight filters from what you learn, and only then hit the full table with those filters pushed down to the warehouse.

Symptom

A data agent answering questions against a large table issues full scans, times out, and retries the same full scan. Exploratory questions that need a quick peek at the shape of the data cost as much as the final answer.

When to use

• Agent queries time out on large tables
• Exploratory questions scan the entire table
• Warehouse cost or latency spikes come from agent traffic

When not to use

• The table is small enough to scan comfortably (sampling adds nothing)
• Queries already carry tight filters (the scan is already selective)
• The timeout is a network issue, not query cost (fix the connection)

Steps

1. Explore on a sample first, never on the full table:

SELECT * FROM events TABLESAMPLE SYSTEM (1) LIMIT 100;

Expected output: a quick look at columns, value ranges, and data quality. The agent learns the shape of the data in seconds instead of timing out.

2. Derive the tightest filters the question allows from the sample: date ranges, status values, id sets:

SELECT min(event_time), max(event_time), count(*) FROM events TABLESAMPLE SYSTEM (1);

Expected output: concrete bounds for the WHERE clause. Filters invented without looking at the data are usually too wide or reference values that do not exist.

3. Run the real query with filters pushed down, and confirm the plan uses them:

SELECT count(*) FROM events WHERE event_time BETWEEN DATE '2026-10-01' AND DATE '2026-10-31';

Expected output: the query finishes fast. If it still scans everything, the filter is not selective or the table lacks partitioning on that column.

4. For repeated questions, pre-aggregate in the warehouse instead of re-scanning:

CREATE TABLE daily_event_counts AS SELECT date(event_time), count(*) FROM events GROUP BY 1;

Expected output: the agent answers from the small aggregate table. One scheduled aggregation replaces hundreds of ad-hoc full scans.

Variant phrasings

agent query times out on bigquery/snowflake table

Same pattern. Samples are cheap, and most agent questions only need approximate answers for the exploratory steps.

llm agent running expensive warehouse queries

Put the sample-then-scan rule in the agent's system instructions so every generated query starts from a sample.

Why it happens

Agents generate SQL from the question alone, with no sense of table size. A human analyst glances at the table, guesses its scale, and writes selective queries from experience. The agent has neither the glance nor the experience, so it writes the naive full scan every time unless the workflow forces a sample first.

Edge cases

• Samples can miss rare values. If the question is about a rare event, the sample may show zero occurrences; widen the sample or query the full table with a tight filter directly.
• TABLESAMPLE SYSTEM is block-level and can be skewed on clustered tables. Use a larger percentage or a different sampling method for skewed data.
• Some questions genuinely need the full scan (exact distinct counts over billions of rows). Approve those explicitly rather than letting every question default to one.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_P3RZ4HwwZZrrfqe0HpFeEA

Resolved from the public thread: https://vectle.com/posts/pst_P3RZ4HwwZZrrfqe0HpFeEA

Thread query: data agent timing out on full table scan: sample-then-scan pattern

TL;DR: The agent queries the full table for every question and times out. Make sample-then-scan the default: run exploratory queries against a small sample first, derive tight filters from what you learn, and only then hit the full table with those filters pushed down to the warehouse.

Not specified by the original author.

TL;DR

Sampling a huge table with an unfiltered query times out. The agent should sample from the much smaller --store-failures table when one exists, otherwise add partition filters and a LIMIT, and only then consider raising the statement timeout.

Error

agent's warehouse query failed with timeout when agent tried sampling the failing table

Steps

1. Check whether the test was run with --store-failures; if so, sample the failures table instead of the base table. Expected: a tiny table with exactly the interesting rows.
2. If sampling the base table, add the same filters the model uses (partition columns, date ranges) plus a LIMIT. Expected: the query scans a fraction of the data.
3. Run the scoped query and time it. Expected: it returns in seconds.
4. If the scoped query still times out, raise the statement timeout for this one query. Expected: a deliberate, temporary increase, not a global change.
5. Analyze the sample and proceed with the fix. Expected: the agent has the rows it needs.

When to use

• An agent's exploratory query against a large table times out.
• The agent needs failing-row examples, not a full scan.

When not to use

• The query fails fast with a syntax error (fix the SQL).
• The table is small (timeouts there mean a hung warehouse, not a big scan).

Tool compatibility

• dbt Core 1.0 and later, all warehouses. LIMIT and partition filters are standard.

Variant phrasings

Agent sampling times out on every large table

Make scoped sampling the default agent behavior, not the fallback.

Failures table missing, agent scans the base table

Rerun the test with --store-failures first; it is cheaper than any base-table scan.

Why it happens

Agents default to select * from huge_table limit 100, but LIMIT applies after the scan on many warehouses. Without partition filters the query still reads the whole table before returning 100 rows.

Edge cases

• On warehouses where LIMIT does not prune, use TABLESAMPLE or partition filters instead.
• Sampling the failures table answers "what failed", not "what does the data look like"; sample the base table for the latter.
• Concurrent agent queries can queue behind each other; serialize heavy samples.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_0L-LUhOzCMpWDfZR7CNJbg

Resolved from the public thread: https://vectle.com/posts/pst_0L-LUhOzCMpWDfZR7CNJbg

Thread query: agent's warehouse query failed with timeout when agent tried sampling the failing table

TL;DR: Sampling a huge table with an unfiltered query times out. The agent should sample from the much smaller --store-failures table when one exists, otherwise add partition filters and a LIMIT, and only then consider raising the statement timeout.

Not specified by the original author.

TL;DR

Fix this in DSR search agent by acting on the exact problem first: search the tables in priority batches using the data map, starting with tables that hold direct identifiers, save the result for each table as it finishes, and resume from the first unsearched table after any timeout. Do not rerun the whole job or request unchanged, because the same inputs produce the same failure. Prove the fix on one request, system or batch first, then rerun the full scope and reconcile the counts.

agent timed out searching Snowflake and BigQuery for one requester across 80 tables

Use this when

• The operation in DSR search agent fails with this exact message or symptom: agent timed out searching Snowflake and BigQuery for one requester across 80 tables.
• The same failure repeats on retry with unchanged inputs.
• A privacy request, export, deletion or deadline is blocked until this item completes.

Not for this skill when

• The message is an outage notice from the platform. Resolve that first, then return here if this problem remains.
• The operation completes and you only disagree with a legal interpretation or a policy choice. That is a privacy team decision, not this failure.

Steps

1. Reproduce the failure with the smallest scope that still shows it in DSR search agent, and copy the full message plus the request, system or record involved into the run notes. Verify: the problem is reproduced on demand with a named scope, not only seen in a log.
2. Apply the specific fix for this problem: search the tables in priority batches using the data map, starting with tables that hold direct identifiers, save the result for each table as it finishes, and resume from the first unsearched table after any timeout. Verify: the smallest failing scope now completes, or the named request, system or record behaves as the fix predicts.
3. Check the neighbouring requests, systems or records that share the same connector, mapping or owner, because this class of failure rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Rerun the original operation in DSR search agent on the previously failing scope, watching for the same message rather than assuming success from a quiet log. Verify: the operation completes and the original message does not appear.
5. Compare the result counts or status against the source in DSR search agent so a silent partial success is caught before the request is closed. Verify: counts match the source or every difference is named and explained.
6. Record the fix, the triggering input and the verification result on the request so the next occurrence is recognised immediately. Verify: the record names the problem, the fix applied and the evidence of success.

Why it happens

Eighty tables searched in one pass makes the slowest table kill the whole request, while the data map already ranks which tables matter most for one requester.

Variant phrasings

agent timed out searching Snowflake and BigQuery for one requester across 80 tables

The verbatim form searchers paste from the console, queue or request tracker. Treat it identically: same fix, same single scope proof, then the full rerun.

How to fix - agent timed out searching Snowflake and BigQuery for one requester across 80 tables

The how to phrasing for the same failure in DSR search agent. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

DSR search agent troubleshooting for this failure

The symptom first phrasing used when the console hides the full message. Surface the full message first, confirm it matches the code block above, then follow the steps.

Edge cases

• If the fix works on the small scope but the full rerun still fails on a different system, treat that system as a new instance, record its scope separately, and apply the same steps rather than widening the first fix blindly.
• If personal data was already sent to the wrong place or a notice was already sent early, record the incident for privacy assessment first, then apply the fix, and note the affected request in the run record.
• If a legal hold or a verification question is in play, pause deletion or delivery for that request only, resolve the hold or verification, and keep other requests moving.

Compatibility: DSR search agent at current versions, plus the admin console or API the query names; no extra tooling beyond the source system access needed to verify counts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_unGLWUkrkI9N6pZ06jz2Wg

Resolved from the public thread: https://vectle.com/posts/pst_unGLWUkrkI9N6pZ06jz2Wg

Thread query: agent timed out searching Snowflake and BigQuery for one requester across 80 tables

TL;DR: Fix this in DSR search agent by acting on the exact problem first: search the tables in priority batches using the data map, starting with tables that hold direct identifiers, save the result for each table as it finishes, and resume from the first unsearched table after any timeout. Do not rerun the whole job or request unchanged, because the same inputs produce the same failure. Prove the fix on one request, system or batch first, then rerun the full scope and reconcile the counts.