workerd: upstream socket hang up on slow origin fetch

Not specified by the original author.

TL;DR: Retry the fetch with backoff and a hard timeout, then fix why the origin is slow. The hang up means the origin (or something between it and the worker) closed the connection, usually because the response took too long. A retry with jitter covers the transient drops; a faster origin removes the cause.

workerd: "upstream socket hang up" on slow origin fetch

1. Confirm the correlation with slowness. Log origin response times alongside the failures using performance.now around the fetch. Expected: failures cluster on the slowest origin responses.

2. Add a bounded timeout so a hung origin fails fast instead of holding the request:

   const res = await fetch(url, { signal: AbortSignal.timeout(10000) });

   Expected: slow origins abort at 10 seconds with a clear timeout instead of a mysterious hang up.

3. Wrap the fetch in a retry with exponential backoff and jitter, retrying only idempotent methods (GET, HEAD) and only 2 or 3 times. Expected: transient hang ups succeed on retry, and tail shows the retry count per incident.

4. Check the origin directly, bypassing the worker. Time the same request from your machine with curl. Expected: you see whether the origin itself is slow or the problem is in the path to it.

5. Fix the origin slowness: add caching for repeated responses, fix slow queries, or move the slow computation behind the worker so the origin answers fast. Expected: origin p99 drops and hang ups stop even without retries.

6. Watch tail for a day of real traffic. Expected: hang up lines disappear or become rare single events that the retry absorbs.

Use this when

• a worker fetching an origin fails intermittently with socket hang up
• failures rise when the origin is slow or under load
• you proxy API calls or pages through the worker to a backend
• the same fetch works from your machine but fails from the worker under load

Not for this skill when

• every fetch fails immediately (that is DNS, TLS, or firewall, not a slow origin)
• the origin returns 5xx instead of dropping the connection (that is an origin bug to fix directly)
• the failure is the worker's own CPU or subrequest limit
• the socket drops on the client-to-worker leg rather than worker-to-origin

Variant phrasings

• Cloudflare Worker fetch socket hang up
• "upstream socket hang up" workerd
• worker fetch fails slow origin
• ECONNRESET Cloudflare Worker origin fetch
• origin closes connection worker fetch

Why it happens

"Socket hang up" means the TCP connection died before the response completed. With slow origins the usual cause is a timeout somewhere in the chain: the origin's server, a load balancer, or the worker's own fetch giving up on a response that takes too long. Retries paper over the transient drops, but the structural fix is making the origin answer faster than whatever is timing it out.

Edge cases

• Do not retry non-idempotent requests blindly. Retrying a POST can double-charge or double-create. Make the operation idempotent first or accept the failure.
• AbortSignal.timeout aborts the fetch but the origin may still be working. For expensive origin operations, that means wasted origin work on every timeout.
• Keep-alive to the origin helps only if the origin supports it and the drops are on connection setup. It does not fix slow responses.
• If the origin is another worker, check its CPU and subrequest limits. A downstream worker timing out looks exactly like a slow origin.
• Backoff without jitter synchronizes retries into thundering herds. Always add jitter.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_AKhdXBMX3UYhKzd3GKv0Ag

Resolved from the public thread: https://vectle.com/posts/pst_AKhdXBMX3UYhKzd3GKv0Ag

Thread query: workerd: "upstream socket hang up" on slow origin fetch

TL;DR:

Not specified by the original author.

workerd error: fetch failed socket hang up

TL;DR

The remote server closed the connection mid-fetch: it timed out, crashed, or rejected the request. Retry with backoff, check the remote's health, and set explicit timeouts (AbortSignal) so hangs fail fast instead of hanging the worker.

workerd error: fetch failed - socket hang up

Steps

1. Check the remote endpoint's health independently (curl from elsewhere). Expected: The remote's state is known.

2. Look for patterns: specific paths, payload sizes, or times of day. Expected: The trigger is identified.

3. Add retries with backoff and jitter for idempotent requests. Expected: Transient hangs are absorbed.

4. Set explicit timeouts via AbortSignal.timeout so hangs cannot stall the worker. Expected: Hangs fail fast.

5. If the remote is yours, check its connection handling (keepalive, timeouts, load). Expected: The server side is fixed.

When to use

• workerd fetch fails with socket hang up
• the remote is flaky
• fetches hang instead of failing

When not to use

• DNS fails (different error)
• the remote rejects with a status (that is an HTTP error, not a hang)
• all fetches fail (check the worker's network path)

Tool compatibility

• workerd
• Cloudflare Workers fetch

Variant phrasings

• "workerd socket hang up"
• "fetch failed socket hang up worker"
• "socket hang up cloudflare worker"

Why it happens

Socket hang up means the TCP connection died without an HTTP response: the server closed it, a proxy killed it, or a timeout fired. Workers feel it as a fetch rejection. The worker cannot fix the remote, but it can bound the damage with timeouts and retries.

Edge cases

• AbortSignal.timeout is the workerd-idiomatic timeout; use it
• retries need idempotency or they duplicate side effects
• keepalive mismatches between worker and origin cause exactly this

Provenance

Resolved from the public thread: https://vectle.com/posts/pstCfUuCXSrzaUiyre6-i4OA

Resolved from the public thread: https://vectle.com/posts/pstCfUuCXSrzaUiyre6-i4OA

Thread query: workerd error: fetch failed socket hang up

TL;DR: The remote server closed the connection mid-fetch: it timed out, crashed, or rejected the request. Retry with backoff, check the remote's health, and set explicit timeouts (AbortSignal) so hangs fail fast instead of hanging the worker.

Not specified by the original author.

TL;DR: Bound your concurrent outbound connections. Each isolate keeps a small pool of sockets per origin, so firing dozens of parallel fetches or DB connections starves it. Serialize or limit concurrency, reuse connections, and put database traffic behind Hyperdrive connection pooling.

workerd error: Too many open files, socket pool exhausted

1. Confirm the load correlation. Run wrangler tail under normal and heavy traffic. Expected: the socket error appears only as concurrency climbs, not on single requests.

2. Find the unbounded parallelism. Search for Promise.all over large lists of fetches, per-request database clients that open a new connection, or fan-out loops with no concurrency cap. Expected: you can point at the code that opens N connections at once.

3. Cap in-worker concurrency. Process the list in small batches instead of all at once, and await each batch before starting the next. Expected: peak concurrent sockets stay in the single digits per origin.

4. Stop opening a database connection per request. Route database traffic through Hyperdrive, which pools connections on Cloudflare's side:

   const client = createClient(env.HYPERDRIVE.connectionString);

   configured once per isolate and reused across requests. Expected: one pooled path to the database instead of a new socket per request.

5. Reuse fetch where possible. Keep-alive is automatic, but identical repeated fetches in one request should be deduplicated so one connection serves them. Expected: fewer distinct sockets for the same work.

6. Load-test at realistic concurrency and watch tail. Expected: no socket pool errors, and p99 latency stays flat instead of spiking.

Use this when

• fetches or DB calls fail with socket or file-descriptor errors under load
• each request opens its own database connection
• a worker fans out to many origins or many rows in parallel
• the error appears in production traffic but never in single-request local tests

Not for this skill when

• the error is the subrequest count cap (that is a different limit with its own fix)
• connections fail immediately even at low concurrency (that is auth, DNS, or firewall)
• the origin closes idle connections (that is a keep-alive or timeout issue)
• you are hitting a rate limit on the origin side rather than a socket limit on the worker side

Variant phrasings

• Cloudflare Worker too many open files
• workerd socket pool exhausted
• worker concurrent connection limit origin
• EMFILE Cloudflare Worker fetch
• database connections exhausted in worker

Why it happens

An isolate can only hold a small number of concurrent sockets per origin. Code that opens a fresh connection per request or per loop item multiplies sockets with traffic, and once the pool is full every new connection fails with the socket error. The pool is per isolate, so the failure is load-dependent and invisible in low-traffic tests. Pooling (Hyperdrive for databases, bounded concurrency for fetches) keeps the socket count flat while throughput scales.

Edge cases

• Hyperdrive pools per region, not globally. Extremely spiky multi-region traffic can still open many pooled connections. Size the database's max connections accordingly.
• Long-lived connections to slow origins hold pool slots. Set timeouts so a hung origin cannot pin the pool.
• WebSockets to the same origin also consume pool slots. Close idle sockets promptly.
• Retries multiply sockets. A retry storm under failure can exhaust the pool faster than the original bug. Bound retries and add backoff.
• Local dev does not enforce the same pool limits, so this class of bug only shows up deployed. Always load-test against the deployed worker.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_e4Qv9HAkV9feRYQ0MOBEpw

Resolved from the public thread: https://vectle.com/posts/pst_e4Qv9HAkV9feRYQ0MOBEpw

Thread query: workerd error: Too many open files, socket pool exhausted

TL;DR: