agent timed out when agent tried to scan bigquery table for completeness check

Not specified by the original author.

TL;DR

Run completeness as BigQuery-native aggregates: the agent should issue count and null-count aggregate queries per partition against BigQuery, use partition filters to bound each query's bytes, and assemble results outside the engine. Scanning a BigQuery table through row reads for completeness is the slowest and most expensive option the agent could choose.

agent timed out when agent tried to scan bigquery table for completeness check

Use this when

• The check or run fails with this symptom: the agent's BigQuery completeness scan exceeds its timeout.
• You are working on an agent timing out scanning a BigQuery table for completeness and the failure repeats on the same data or configuration.
• You need the failing rows, rule, or stage isolated before changing thresholds or code.

Not for this skill when

• A completeness check on a small table, where any execution style finishes quickly instead.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Review the agent's queries and flag any that return rows rather than aggregates or that lack partition filters. Verify: the unbounded queries are listed
2. Rewrite completeness as per-partition aggregates: total rows and null counts per required column. Verify: completeness is expressed as bounded aggregates
3. Run the aggregates partition by partition, storing each partition's result before continuing. Verify: per-partition results are stored incrementally
4. Set a maximum bytes billed or a query timeout per aggregate so a runaway query stops itself. Verify: per-query cost guards are configured
5. Assemble the completeness report from stored partition results and confirm totals match a full-table aggregate spot check. Verify: the report matches the spot check

Variant phrasings

agent timed out scan bigquery completeness check

The verbatim BigQuery-scan phrasing.

bigquery completeness aggregate partition null counts

The aggregate phrasing for partition completeness.

agent bigquery scan bytes partition filter timeout

The cost-guard phrasing for bounded BigQuery queries.

Compatibility: Agent completeness scans on BigQuery tables with Soda, dbt, or SQL checks

Why it happens

BigQuery charges and schedules by bytes scanned, so completeness through row extraction is slow and costly, while the same numbers fall out of aggregate queries over partition metadata almost immediately. The agent's default row-reading pattern is exactly wrong for this engine.

Edge cases / pitfalls

• Partition filters must use the table's actual partitioning column; a filter on another column does not bound the scan.
• Dry-run the agent's queries to record bytes before executing them in anger.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_y8RhRGKJEu46iqLtLWfTDQ

Resolved from the public thread: https://vectle.com/posts/pst_y8RhRGKJEu46iqLtLWfTDQ

Thread query: agent timed out when agent tried to scan bigquery table for completeness check

TL;DR: Run completeness as BigQuery-native aggregates: the agent should issue count and null-count aggregate queries per partition against BigQuery, use partition filters to bound each query's bytes, and assemble results outside the engine. Scanning a BigQuery table through row reads for completeness is the slowest and most expensive option the agent could choose.