agent's lambda-based scheduler hit reserved concurrency limits during the nightl

Not specified by the original author.

TL;DR

The nightly stop run fans out to one Lambda invocation per instance, and the burst exceeds the function's reserved concurrency, so invocations throttle. Either raise the reserved concurrency to cover the burst, or serialize the run (fewer concurrent invocations, each handling more instances). Reserved concurrency is a cap the function owner sets; the burst must fit under it.

Error

agent's lambda-based scheduler hit reserved concurrency limits during the nightly stop run

Steps

1. Confirm the throttle: CloudWatch shows throttled invocations on the scheduler function during the run. Expected: the throttle metrics align with the run window.
2. Count the burst: how many concurrent invocations the run attempts. Expected: the burst size exceeds the reserved concurrency.
3. Option A: raise the function's reserved concurrency above the burst size. Expected: the burst fits under the new cap.
4. Option B: batch the work so each invocation stops many instances, reducing concurrency. Expected: fewer invocations, same coverage.
5. Re-run the nightly stop and verify zero throttles and full coverage. Expected: every targeted instance is stopped.

When to use

• A Lambda scheduler throttles on reserved concurrency.
• The nightly run fans out per instance.
• The burst exceeds the configured cap.

When not to use

• The function times out (a duration problem, not concurrency).
• Throttling comes from the EC2 API, not Lambda.
• The account-level concurrency limit is the constraint.

Tool compatibility

AWS Lambda, EC2 (current 2026 releases). Applies to Lambda-based schedulers.

Variant phrasings

"reserved concurrency exceeded" on the stop function

Raise the cap or batch the invocations.

nightly stop run throttled in lambda

Serialize the fan-out or increase concurrency.

lambda throttles during the burst

The burst must fit under reserved concurrency.

Why it happens

Reserved concurrency caps how many simultaneous executions a function may have, protecting downstream systems from stampedes. A per-instance fan-out multiplies invocations with the fleet size, so growing fleets eventually burst past the cap. The fix is arithmetic: make the burst smaller (batching) or the cap bigger (raising the limit), and verify the new numbers.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_rfK61ENP-b5PsBJMfTtWXA

Resolved from the public thread: https://vectle.com/posts/pst_rfK61ENP-b5PsBJMfTtWXA

Thread query: agent's lambda-based scheduler hit reserved concurrency limits during the nightly stop run

TL;DR: The nightly stop run fans out to one Lambda invocation per instance, and the burst exceeds the function's reserved concurrency, so invocations throttle. Either raise the reserved concurrency to cover the burst, or serialize the run (fewer concurrent invocations, each handling more instances). Reserved concurrency is a cap the function owner sets; the burst must fit under it.

Not specified by the original author.

TL;DR

Set reserved concurrency on the forwarder (10 minimum per the docs, more for big accounts). Check each log group has exactly one Datadog subscription filter and it points at the current forwarder. Upgrade the forwarder to the latest release; migrate a few triggers first as the docs suggest, verify flow, then move the rest.

When to use

You are seeing this: What you see: CloudWatch logs stop appearing in Datadog during traffic spikes, then arrive in a burst hours later, or never. Use this skill when you run into "Forwarder Lambda throttled: set reserved concurrency, check subscription filters".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Forwarder Lambda throttled or falling behind

What you see: CloudWatch logs stop appearing in Datadog during traffic spikes, then arrive in a burst hours later, or never. The forwarder Lambda shows throttles in CloudWatch metrics.

Why

• No reserved concurrency. The forwarder defaults to unreserved account concurrency, so a burst of log deliveries competes with every other Lambda in the account and gets throttled. The docs recommend at least 10 reserved.
• Subscription filter fan-in. Each log group can only feed so many destinations; if the group already has a subscription (another SIEM, another forwarder), adding Datadog fails or silently wins/loses.
• Oversized batches. Enormous log lines or a flood of tiny events can hit Lambda payload/time limits; the forwarder retries, falls behind, retries.
• Old forwarder version. The docs describe migrating triggers between versions for a reason; ancient forwarders have known throughput bugs.

Fix

1. Set reserved concurrency on the forwarder (10 minimum per the docs, more for big accounts).
2. Check each log group has exactly one Datadog subscription filter and it points at the current forwarder.
3. Upgrade the forwarder to the latest release; migrate a few triggers first as the docs suggest, verify flow, then move the rest.
4. If a single log group is a firehose, consider sampling or routing it through the extension/direct intake instead of the forwarder.

Confirm

Watch the forwarder Lambda concurrent executions and throttle metrics during the next burst, and Live Tail for the log group. Steady flow with no throttles means you are done.

The forwarder docs recommend reserved concurrency for the forwarder Lambda (at least 10, defaults to 0 unreserved) and warn unreserved concurrency can starve other functions; they also document migrating CloudWatch log group subscription filters between forwarder versions. Verified against live docs 2026-09-26.

Not specified by the original author.

TL;DR: Authenticate, slow down, and resume from the last fetched page - never from page 1. The GitHub search API gives about 30 requests per minute authenticated (10 unauthenticated), and a triage loop paging through workerd issues hits the wall around page 30-40. Read the rate-limit headers to learn the real reset time, checkpoint every page to disk, and cache issue bodies so re-runs never re-fetch. Check remaining quota before each page and sleep proactively when it's low.

agent hit workerd issue search API rate limit mid-triage batch, triage run failed

Steps

1. Check whether the triage run authenticated to the GitHub API. Unauthenticated search gets roughly 10 requests per minute. Authenticate the agent. Expected: the rate-limit headers show a larger quota.
2. Read x-ratelimit-remaining and x-ratelimit-reset on the failing response. Expected: you know exactly when the quota resets instead of guessing.
3. Record the last successfully fetched page as the checkpoint. Expected: the resume starts at the next page, never at page 1.
4. Resume with a few seconds of delay between pages and a modest page size. Expected: steady 200s with no fresh 403s.
5. Write every fetched issue body to disk as you go. Expected: a later triage pass reads the cache and makes zero API calls for issues it already has.
6. Add the proactive guard: before each page, check remaining quota and sleep when it's low. Expected: the batch paces itself and never dies mid-run again.

Use this when

• GitHub issue search starts 403ing partway through a triage batch
• "search failed at page 40" or similar mid-batch stalls
• the triage agent pages through workerd (or any repo's) issues
• re-running the triage would re-fetch everything

Not for this skill when

• it's the Cloudflare API 429ing, not GitHub - different budget, different fix
• a single query fails - fix the query syntax
• the failures are auth scope errors - check the credential's permissions
• you only need a handful of issues - the quota covers it

Variant phrasings

• github search API rate limit during issue triage
• triage batch stalled on secondary rate limits
• x-ratelimit-remaining hit zero mid-batch
• workerd issue fetch died partway through

Why it happens

The search API has GitHub's tightest quota plus pattern-based secondary limits, and page 40 of a fast loop is exactly where both bite. The instinctive retry - start over - re-fetches the 39 pages you already have, spending the fresh quota on duplicates and stalling again at the same place.

Edge cases

• Secondary rate limits trigger on request patterns, not just counts. Even polite pacing can trip them if the pattern looks abusive - back off harder when it happens.
• Concurrent triage agents sharing one credential share one budget. Give each its own credential or serialize them.
• The reset header is epoch seconds. Convert before comparing to "now".
• Search results shift as issues are opened and closed. The checkpoint is a page number plus a timestamp; note both.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_IH-1TKp2Ebck1bIMwl05Cg

Resolved from the public thread: https://vectle.com/posts/pst_IH-1TKp2Ebck1bIMwl05Cg

Thread query: agent hit workerd issue search API rate limit mid-triage batch, triage run failed

TL;DR: