workerd: "Exceeded the limit of subrequests" in fan-out

TL;DR: A single request can only make a limited number of subrequests (50 by default), so unbounded fan-out always dies at the cap. Consolidate calls into fewer origin requests, bound your concurrency, or move the fan-out into a Queue where each message gets its own fresh subrequest budget.

workerd: "Exceeded the limit of subrequests" in fan-out

1. Count the fetches in the hot path. Search the handler for fetch calls inside loops or Promise.all over a list, and log how many fire per request. Expected: the count at failure matches the platform subrequest cap.

2. Confirm with tail. Run wrangler tail and reproduce the request. Expected: you see the "Exceeded the limit of subrequests" line and the request dies before the loop finishes.

3. Consolidate where the origin allows it. Replace N per-item fetches with one batched endpoint call, or one GraphQL-style query that returns many items at once. Expected: subrequest count drops from N to 1 for that section.

4. If the origin has no batch endpoint, split the work across invocations. Push one queue message per item:

   for (const url of urls) {
     await env.FANOUT_QUEUE.send({ url });
   }

   and do the single fetch per item inside the queue consumer. Expected: each consumer invocation stays far under the cap, and the original request returns immediately.

5. Where fan-out must stay in-request, bound it. Process the list in small batches with Promise.all over slices, and cache repeated fetches for the same URL within the request. Expected: peak concurrent and total subrequests stay under the cap for your largest real list.

6. Retest with the largest real list size. Expected: no subrequest error in tail, and every item gets processed.

Use this when

• a worker loops over URLs and dies with the subrequest error at a consistent count
• aggregation endpoints, sitemap builders, or multi-origin proxies fan out per request
• health-check or status-page workers poll many upstreams in one handler
• the same code works for 10 items but fails for 100

Not for this skill when

• the error is about CPU time or memory, not subrequest count
• fetches fail individually with timeouts or 5xx (that is origin health, not the cap)
• you need the results synchronously in the response and the list is small (just bound the loop)
• the cap trips inside a Durable Object or scheduled handler for other reasons (check those limits separately)

Variant phrasings

• Cloudflare Worker too many subrequests
• "Exceeded the limit of subrequests" worker
• worker fan-out fails after 50 fetches
• Promise.all fetch limit in Cloudflare Worker
• subrequest limit aggregation worker

Why it happens

workerd caps subrequests per invocation to stop one request from turning into a denial-of-service amplifier. A loop that fires one fetch per list item burns the budget linearly, so any list longer than the cap kills the request. The cap is a platform constant, not a config knob, so the fix is architectural: fewer fetches per invocation or more invocations per job.

Edge cases

• Retries count too. A fetch retried 3 times burns 3 subrequests, which can push a borderline loop over the cap.
• Redirects followed automatically also consume subrequests. A redirect chain of 5 costs 5.
• Cache hits still count as subrequests. Caching reduces origin load but does not dodge the cap.
• Queue consumers each get their own budget, but a consumer that itself fans out can still trip it. Keep consumers to one or two fetches per message.
• Durable Object to Durable Object calls and service bindings have their own accounting. Count everything, not just http fetches.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_HWnCb2ytqJbyzKxRjQ-8ZQ