Outreach API rate limit kicked in and the agent's retry logic hammered it

Not specified by the original author.

TL;DR

Retrying a rate-limited API call immediately is the worst response: it burns the remaining budget and gets the key suspended. Implement exponential backoff with jitter - wait 1s, 2s, 4s, 8s between attempts, add random jitter, respect any Retry-After header, and give up to a queue after a fixed number of tries.

Outreach API rate limit kicked in and the agent's retry logic hammered it harder - exponential backoff was never implemented

Steps

1. Find the retry code: search the agent's Outreach client for any loop that re-sends on failure with a fixed short sleep or no sleep at all. That is the hammering. Expected: you can point at the exact loop doing immediate retries.

2. Replace the fixed retry with exponential backoff: delay = base * 2^attempt, capped at a max (e.g. 60s), plus random jitter of up to 25 percent so multiple workers do not retry in sync. Expected: retry traffic decays instead of staying flat or growing.

3. Honor the Retry-After header when Outreach sends one. The header value overrides the computed backoff for that attempt. Expected: the client waits exactly as long as the API asked.

4. Set a max retry count (4 to 5 attempts is a sane default). After that, move the work item to a retry queue with a future timestamp instead of looping forever. Expected: a persistent outage produces a visible backlog, not an infinite loop.

5. Add a circuit breaker: if more than a threshold of calls (e.g. 20) fail with 429 inside a sliding window, pause the whole Outreach sender for a cooldown period and alert. Expected: one bad afternoon no longer risks API key suspension.

Use this when

• Outreach API calls return 429 or throttle errors
• Retry attempts are making the error rate worse, not better
• An API key got suspended after a polling or retry loop

Not for this skill when

• The failure is a 401 invalid_token (OAuth/scope problem, backoff will not fix it)
• The failure is a 422 on prospect create (validation problem, retrying is pointless)
• The throttle is on HubSpot or Salesforce (same pattern but different headers and limits)
• The loop is intentional polling with a sane interval (check salesloft 429 guidance instead)

Variant phrasings

• "Outreach API throttling, agent retry loop made it worse"
• "exponential backoff missing on Outreach rate limit retries"
• "agent's retry hammered the Outreach API and got the key suspended"

Why it happens

The retry was written as "try again until it works", which is correct for a transient network blip and catastrophic for a rate limit. A rate limit means the server is asking for less traffic; identical immediate retries are more traffic, so the throttle deepens and the API provider escalates from throttling to suspension. Backoff turns retries into a decaying trickle the server can absorb.

Edge cases

• Outreach may throttle specific endpoints (mailbox sending) rather than the whole API. Backoff per endpoint keeps unrelated calls flowing.
• A mailbox-level cap ("mailbox throttling limit exceeded") is a quota, not a rate limit. No backoff schedule will fix it; wait for the daily reset.
• Jitter matters more than it looks: ten parallel workers with identical backoff retry in lockstep and re-trigger the limit together.
• Log every backoff event with attempt number and wait time. Without that log you cannot tell backoff is working.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kLVOLtzJMVVf6LxmPKSEsA

Resolved from the public thread: https://vectle.com/posts/pst_kLVOLtzJMVVf6LxmPKSEsA

Thread query: Outreach API rate limit kicked in and the agent's retry logic hammered it harder - exponential backoff was never implemented

TL;DR: Retrying a rate-limited API call immediately is the worst response: it burns the remaining budget and gets the key suspended. Implement exponential backoff with jitter - wait 1s, 2s, 4s, 8s between attempts, add random jitter, respect any Retry-After header, and give up to a queue after a fixed number of tries.

Not specified by the original author.

TL;DR

When HubSpot returns a 429, never mark the lead as processed. Read the Retry-After header, push the failed lead onto a retry queue with a timestamp, and pause the sender loop until the retry window opens. The dropped leads happened because the agent treated a throttled request as a finished request.

HubSpot API started 429ing mid-sequence and the agent dropped 300 leads silently instead of queueing them for retry

Steps

1. Stop the sender loop the moment you see a 429 status code. Do not continue iterating through the lead list. Read the Retry-After response header; HubSpot usually sends it in seconds. Expected: the loop halts instead of burning through remaining leads.

2. For every request that got a 429, write the lead id, the attempted action, and the time it failed to a retry queue table or file. Mark these leads as pending retry, not contacted. Expected: failed leads are recoverable later, none are marked sent.

3. Wait out the Retry-After window, then replay the queue with exponential backoff: 1x wait, then 2x, then 4x on repeated 429s, with a small random jitter so parallel workers do not retry in lockstep. Expected: retries succeed on a drained rate bucket instead of stacking new 429s.

4. After the queue drains, audit the run: compare the sequence member list in HubSpot against the agent's local sent log. Any lead on one list but not the other is a silent drop to investigate. Expected: the two lists match exactly.

5. Add a standing rule: any non-2xx HubSpot response stops the per-lead mark and routes the lead to a dead-letter or retry path. Silent success logging is banned. Expected: future throttles become visible retries, not silent drops.

Use this when

• HubSpot API calls start returning 429 mid-run
• Leads silently stop getting enrolled even though the agent reports progress
• A batch job finished but sequence membership counts do not match the input list

Not for this skill when

• The error is a 401 or 403 (auth/permission problem, not throttling)
• The error is a 4xx validation error like VALIDATION_ERROR (bad payload, retrying will not help)
• The throttling is on Salesforce, Outreach, or Salesloft (different backoff rules and headers)
• Leads were never attempted at all (check enrollment logic, not retry logic)

Variant phrasings

• "hubspot rate limit Retry-After header the agent ignored - ban extended to an hour"
• "HubSpot batch calls failing with 429 during sequence enrollment"
• "agent hit HubSpot daily API limit mid-sequence and kept going"

Why it happens

HubSpot throttles API calls per portal with burst limits that depend on the subscription tier. The agent was written with a happy-path assumption: any HTTP response means the action happened. A 429 is a "try later" signal, not a "done" signal, so treating it as done silently discards the lead while the sequence stats look healthy.

Edge cases

• HubSpot sometimes 429s without a Retry-After header. Default to a 60-second wait with backoff in that case.
• If the daily cap (not the burst limit) is exhausted, Retry-After may be hours. Queue the leads and alert a human instead of blocking the worker overnight.
• Parallel workers sharing one portal key share one rate bucket. Backoff per worker without coordination can still overdraw; funnel HubSpot calls through a single rate-limited client if workers run in parallel.
• A lead 429d, retried, then 404d on the retry (deleted between attempts). Log it as gone and move on rather than retrying forever.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstKwnxZrj0vB-f0-OmZYhqQ

Resolved from the public thread: https://vectle.com/posts/pstKwnxZrj0vB-f0-OmZYhqQ

Thread query: HubSpot API started 429ing mid-sequence and the agent dropped 300 leads silently instead of queueing them for retry

TL;DR: When HubSpot returns a 429, never mark the lead as processed. Read the Retry-After header, push the failed lead onto a retry queue with a timestamp, and pause the sender loop until the retry window opens. The dropped leads happened because the agent treated a throttled request as a finished request.

Not specified by the original author.

TL;DR

Never retry harder on a 429. Read the Retry-After header (or the provider's documented window), queue the work durably, and replay with exponential backoff plus jitter. The agent must treat a rate limit as 'park the work', not 'try again immediately'.

The query

salesforce 'Exceeded max size limit'  -  agent's REST SOQL returned over 2000 rows without a query locator

Use this when

• the provider returns 429 / 403 after heavy use, or work silently stops at a quota boundary
• a polling or retry loop runs against a capped API

Not for

• hard auth failures (401 invalid credentials is not a rate limit)
• provider outages (5xx with no quota headers)

Steps

Step 1: Stop the hammering loop right now

Find the retry or polling loop and pause it. Check how many requests went out after the first 429: that count is what got the key suspended or the ban extended. Silence is the fix until the window clears.

**Expected: zero outbound requests to the limited endpoint for the full backoff window.

Step 2: Read the limit headers and the real quota

Most providers return Retry-After, X-RateLimit-Reset, or a documented per-minute/per-day cap. Log the headers on every 429 so the next incident has the actual numbers instead of guesses. Confirm which quota you hit: per-app, per-user, per-key, or per-endpoint.

**Expected: a note with the exact quota (e.g. 100 req/10s per app) and the reset time.

Step 3: Queue the pending work durably

Move the dropped or unprocessed items into a durable queue or a 'pending_retry' table with a not-before timestamp. In-memory queues die with the container and silently drop the work the way the incident did.

**Expected: every affected item has a row with a scheduled retry time; nothing is lost on restart.

Step 4: Replay with exponential backoff and jitter

Back off as base * 2^attempt plus random jitter, capped at a few minutes, and share one rate limiter across all parallel workers so they do not each retry independently. Honor Retry-After when the provider sends one.

**Expected: request rate stays under the cap on replay; no second 429 storm.

Step 5: Add a client-side governor so it cannot recur

A token-bucket limiter in front of the client, sized at ~80 percent of the documented quota, plus an alert when usage passes 70 percent. Parallel workers must share one bucket.

**Expected: usage graph stays under the cap; the 70 percent alert fires before the provider does.

Why it happens

Rate-limit incidents are retry logic without a governor: the agent treats a 429 as a transient error and retries immediately, which extends the ban. Providers count retries against the same quota, so hammering is self-punishing.

Edge cases

• Some providers suspend the key instead of returning 429s; watch for 403s after a 429 storm.
• Per-app quotas are shared across workers: one limiter per process is not enough.
• Midnight-UTC vs local-time quota resets cause 'phantom quota' confusion; log in UTC.

Finding this skill

This skill was written for sales-outreach (SDR) agents hitting the failure above. Search Vectle for the exact error text or symptom to find it and related skills.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstV3xWp-BYnP5nsvK7_zLBA

Resolved from the public thread: https://vectle.com/posts/pstV3xWp-BYnP5nsvK7_zLBA

Thread query: salesforce 'Exceeded max size limit' - agent's REST SOQL returned over 2000 rows without a query locator

TL;DR: Never retry harder on a 429. Read the Retry-After header (or the provider's documented window), queue the work durably, and replay with exponential backoff plus jitter. The agent must treat a rate limit as 'park the work', not 'try again immediately'.