dbt cloud API rate limit hit mid-job-status-poll, poll failed

TL;DR

The agent polled job status faster than the dbt Cloud API allows and got throttled, so the poll failed and the agent lost track of the job. Poll with backoff, lengthen the interval as the wait grows, and treat 429 as slow down, not as job failed. The job is fine; the polling was greedy.

dbt cloud API rate limit hit mid-job-status-poll, poll failed

Use this when

• Agent job-status polls hit dbt Cloud 429s.
• Polling worked for one job but fails when watching several.
• The agent lost track of a job after the poll failed.

Not for this skill when

• Polls return 401. That is auth.
• The job itself failed. Read the job status.

Steps

1. Note the poll interval and the number of jobs watched: compute calls per minute. Note the rate. Verify: the polling load is measured
2. Lengthen the interval and add backoff with jitter on 429. Note the new pacing. Verify: the polling respects the limit
3. On 429, keep the last known job state and keep waiting; do not mark the job failed. Note the rule. Verify: throttle is not failure
4. Watch fewer jobs per agent or stagger the polls. Note the reduced load. Verify: the load fits the limit
5. Confirm the agent tracks jobs to completion without 429s. Note the clean run. Verify: the pattern works

Variant phrasings

dbt cloud api polling interval

The tuning phrasing.

dbt cloud 429 too many requests

The code-shaped search.

Compatibility: dbt Cloud API rate limits. Polling patterns are client-side.

Why it happens

Status polling is the greediest API consumer: it repeats the same call on a timer, and the timer that feels responsive for one job is abusive for twenty. dbt Cloud's rate limit is per account, so one agent's tight polling loop can throttle every other integration. Backoff on 429 is the API's way of asking for pacing; honoring it keeps the agent's view of the job intact instead of failing the poll and losing the thread.

Edge cases / pitfalls

• A 429 during the final poll can make a successful job look unknown. Always do a last-chance poll after backing off.
• Poll intervals should grow with the job's expected duration: seconds early, minutes late.
• Multiple agents polling the same account need coordination, not just individual backoff. Share a polling budget.
• The job status endpoint and the list endpoints share the limit. Budget the whole integration, not just the poll loop.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_socslBNWlwLKSecw34bKzw