W&B API rate limit hit mid-sweep import and agent failed to import remaining runs with 429 error

TL;DR

Import in rate-shaped batches with a checkpoint: the remaining runs were abandoned because the import treated the 429 as a run failure instead of a pacing signal. Keep an import ledger of run ids completed, sleep for the rate window when limited, and resume with only the unimported runs, verifying the final count against the sweep's run list.

W&B API rate limit hit mid-sweep import and agent failed to import remaining runs with 429 error

Use this when

• The run or call fails with this symptom: W and B API rate limit hit mid-sweep import and the agent failed to import remaining runs.
• You are working on an agent importing sweep runs into W and B or mirroring them elsewhere and the failure repeats on the same run or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A rate limit imposed by account tier that no pacing can satisfy for the volume, where the fix is splitting the import across days or raising the tier.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Build the authoritative list of sweep runs to import and mark each as imported or pending in a ledger. Verify: import state exists per run
2. On a 429, pause for the window the API indicates instead of marking runs failed, then continue from the ledger. Verify: rate limits pause rather than abort the import
3. Process imports in fixed-size batches sized to stay under the account's request budget. Verify: batch pacing keeps the rate under the limit
4. At the end, diff the ledger against the sweep's run list and import any stragglers before reporting completion. Verify: the final counts match the sweep exactly

Variant phrasings

wandb api rate limit mid sweep import remaining runs

The verbatim phrasing.

agent import sweep runs 429 resume ledger fix

The ledger phrasing.

wandb sweep import incomplete rate limited agent

The incomplete phrasing.

Compatibility: Weights and Biases API driven by agents during sweep imports

Why it happens

Sweep imports are bursty: hundreds of runs arrive at once and each takes several API calls. Without a ledger, a mid-import limit strands the tail invisibly, because the imported prefix looks like a complete, if smaller, sweep.

Edge cases / pitfalls

• Runs imported partially, metadata without history, need the ledger at field granularity for the worst cases.
• Mirror the sweep's own pagination discipline; the source list and the import both need completeness checks.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstSmkozjunBGCgpNhWRDvtg