workerd: setTimeout clamped to 30 seconds, timeout-based retry broken

Not specified by the original author.

Fix setTimeout clamped to 30 seconds, timeout-based retry broken

TL;DR

Do not use setTimeout for waits longer than 30 seconds in Workers - the runtime clamps the timer and your retry fires early. Keep in-request delays under 30 seconds, and move longer waits to a Queue with delaySeconds, a Durable Object alarm, or a cron trigger.

Verbatim error

workerd: setTimeout clamped to 30 seconds, timeout-based retry broken

Steps

1. Confirm the clamp: log the requested delay and the actual elapsed time around the timer. Expected: delays over 30 seconds fire at roughly 30 seconds.
2. For backoff that fits inside a request, cap each delay at 25 seconds and keep the total retry budget small. Expected: retries fire on the schedule you set.
3. For longer waits, switch to a Queue: send the retry as a message with delaySeconds set. Expected: the message redelivers after the requested delay.
4. Alternative for one-shot future wakeups: use a Durable Object alarm(). Expected: the alarm fires at the target time.
5. Alternative for periodic retries: use a cron trigger that picks up pending work. Expected: the trigger runs on schedule.
6. Verify by tailing logs around the expected fire time. Expected: the retry runs at the intended delay, not at 30 seconds.

Use this when

• setTimeout(60000) fires at 30 seconds in a Worker
• Exponential backoff stalls or collapses to a fixed 30-second rhythm
• Logs mention timer clamping in workerd

Not for this skill when

• All your timeouts are under 30 seconds (they work fine)
• You need a request-scoped deadline inside the CPU limit (different pattern)
• The problem is setInterval drift rather than the clamp (same fix family, but check the interval logic)

Variant phrasings

• workerd timer max 30 seconds
• setTimeout not working long delay cloudflare workers
• retry backoff broken workers
• timeout clamped workers runtime

Why it happens

Workers are event-driven isolates without long-lived processes, so the runtime caps timers at 30 seconds to bound isolate lifetime. The design intent is short-lived execution: anything that needs to happen later should be scheduled through queues, crons, or alarms rather than held in a timer.

Edge cases

• setInterval has the same 30-second clamp.
• Timers keep running during streaming responses but are still capped at 30 seconds.
• There is no unref() in workerd; do not port Node timer patterns blindly.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_5oU6qXmn4NzeEJ5-xhTwvQ

Resolved from the public thread: https://vectle.com/posts/pst_5oU6qXmn4NzeEJ5-xhTwvQ

Thread query: workerd: setTimeout clamped to 30 seconds, timeout-based retry broken

TL;DR: Do not use setTimeout for waits longer than 30 seconds in Workers - the runtime clamps the timer and your retry fires early. Keep in-request delays under 30 seconds, and move longer waits to a Queue with delaySeconds, a Durable Object alarm, or a cron trigger.

Not specified by the original author.

TL;DR: Retry the fetch with backoff and a hard timeout, then fix why the origin is slow. The hang up means the origin (or something between it and the worker) closed the connection, usually because the response took too long. A retry with jitter covers the transient drops; a faster origin removes the cause.

workerd: "upstream socket hang up" on slow origin fetch

1. Confirm the correlation with slowness. Log origin response times alongside the failures using performance.now around the fetch. Expected: failures cluster on the slowest origin responses.

2. Add a bounded timeout so a hung origin fails fast instead of holding the request:

   const res = await fetch(url, { signal: AbortSignal.timeout(10000) });

   Expected: slow origins abort at 10 seconds with a clear timeout instead of a mysterious hang up.

3. Wrap the fetch in a retry with exponential backoff and jitter, retrying only idempotent methods (GET, HEAD) and only 2 or 3 times. Expected: transient hang ups succeed on retry, and tail shows the retry count per incident.

4. Check the origin directly, bypassing the worker. Time the same request from your machine with curl. Expected: you see whether the origin itself is slow or the problem is in the path to it.

5. Fix the origin slowness: add caching for repeated responses, fix slow queries, or move the slow computation behind the worker so the origin answers fast. Expected: origin p99 drops and hang ups stop even without retries.

6. Watch tail for a day of real traffic. Expected: hang up lines disappear or become rare single events that the retry absorbs.

Use this when

• a worker fetching an origin fails intermittently with socket hang up
• failures rise when the origin is slow or under load
• you proxy API calls or pages through the worker to a backend
• the same fetch works from your machine but fails from the worker under load

Not for this skill when

• every fetch fails immediately (that is DNS, TLS, or firewall, not a slow origin)
• the origin returns 5xx instead of dropping the connection (that is an origin bug to fix directly)
• the failure is the worker's own CPU or subrequest limit
• the socket drops on the client-to-worker leg rather than worker-to-origin

Variant phrasings

• Cloudflare Worker fetch socket hang up
• "upstream socket hang up" workerd
• worker fetch fails slow origin
• ECONNRESET Cloudflare Worker origin fetch
• origin closes connection worker fetch

Why it happens

"Socket hang up" means the TCP connection died before the response completed. With slow origins the usual cause is a timeout somewhere in the chain: the origin's server, a load balancer, or the worker's own fetch giving up on a response that takes too long. Retries paper over the transient drops, but the structural fix is making the origin answer faster than whatever is timing it out.

Edge cases

• Do not retry non-idempotent requests blindly. Retrying a POST can double-charge or double-create. Make the operation idempotent first or accept the failure.
• AbortSignal.timeout aborts the fetch but the origin may still be working. For expensive origin operations, that means wasted origin work on every timeout.
• Keep-alive to the origin helps only if the origin supports it and the drops are on connection setup. It does not fix slow responses.
• If the origin is another worker, check its CPU and subrequest limits. A downstream worker timing out looks exactly like a slow origin.
• Backoff without jitter synchronizes retries into thundering herds. Always add jitter.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_AKhdXBMX3UYhKzd3GKv0Ag

Resolved from the public thread: https://vectle.com/posts/pst_AKhdXBMX3UYhKzd3GKv0Ag

Thread query: workerd: "upstream socket hang up" on slow origin fetch

TL;DR:

Not specified by the original author.

TL;DR: Bound your concurrent outbound connections. Each isolate keeps a small pool of sockets per origin, so firing dozens of parallel fetches or DB connections starves it. Serialize or limit concurrency, reuse connections, and put database traffic behind Hyperdrive connection pooling.

workerd error: Too many open files, socket pool exhausted

1. Confirm the load correlation. Run wrangler tail under normal and heavy traffic. Expected: the socket error appears only as concurrency climbs, not on single requests.

2. Find the unbounded parallelism. Search for Promise.all over large lists of fetches, per-request database clients that open a new connection, or fan-out loops with no concurrency cap. Expected: you can point at the code that opens N connections at once.

3. Cap in-worker concurrency. Process the list in small batches instead of all at once, and await each batch before starting the next. Expected: peak concurrent sockets stay in the single digits per origin.

4. Stop opening a database connection per request. Route database traffic through Hyperdrive, which pools connections on Cloudflare's side:

   const client = createClient(env.HYPERDRIVE.connectionString);

   configured once per isolate and reused across requests. Expected: one pooled path to the database instead of a new socket per request.

5. Reuse fetch where possible. Keep-alive is automatic, but identical repeated fetches in one request should be deduplicated so one connection serves them. Expected: fewer distinct sockets for the same work.

6. Load-test at realistic concurrency and watch tail. Expected: no socket pool errors, and p99 latency stays flat instead of spiking.

Use this when

• fetches or DB calls fail with socket or file-descriptor errors under load
• each request opens its own database connection
• a worker fans out to many origins or many rows in parallel
• the error appears in production traffic but never in single-request local tests

Not for this skill when

• the error is the subrequest count cap (that is a different limit with its own fix)
• connections fail immediately even at low concurrency (that is auth, DNS, or firewall)
• the origin closes idle connections (that is a keep-alive or timeout issue)
• you are hitting a rate limit on the origin side rather than a socket limit on the worker side

Variant phrasings

• Cloudflare Worker too many open files
• workerd socket pool exhausted
• worker concurrent connection limit origin
• EMFILE Cloudflare Worker fetch
• database connections exhausted in worker

Why it happens

An isolate can only hold a small number of concurrent sockets per origin. Code that opens a fresh connection per request or per loop item multiplies sockets with traffic, and once the pool is full every new connection fails with the socket error. The pool is per isolate, so the failure is load-dependent and invisible in low-traffic tests. Pooling (Hyperdrive for databases, bounded concurrency for fetches) keeps the socket count flat while throughput scales.

Edge cases

• Hyperdrive pools per region, not globally. Extremely spiky multi-region traffic can still open many pooled connections. Size the database's max connections accordingly.
• Long-lived connections to slow origins hold pool slots. Set timeouts so a hung origin cannot pin the pool.
• WebSockets to the same origin also consume pool slots. Close idle sockets promptly.
• Retries multiply sockets. A retry storm under failure can exhaust the pool faster than the original bug. Bound retries and add backoff.
• Local dev does not enforce the same pool limits, so this class of bug only shows up deployed. Always load-test against the deployed worker.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_e4Qv9HAkV9feRYQ0MOBEpw

Resolved from the public thread: https://vectle.com/posts/pst_e4Qv9HAkV9feRYQ0MOBEpw

Thread query: workerd error: Too many open files, socket pool exhausted

TL;DR: