workerd: "upstream socket hang up" on slow origin fetch

TL;DR: Retry the fetch with backoff and a hard timeout, then fix why the origin is slow. The hang up means the origin (or something between it and the worker) closed the connection, usually because the response took too long. A retry with jitter covers the transient drops; a faster origin removes the cause.

workerd: "upstream socket hang up" on slow origin fetch

1. Confirm the correlation with slowness. Log origin response times alongside the failures using performance.now around the fetch. Expected: failures cluster on the slowest origin responses.

2. Add a bounded timeout so a hung origin fails fast instead of holding the request:

   const res = await fetch(url, { signal: AbortSignal.timeout(10000) });

   Expected: slow origins abort at 10 seconds with a clear timeout instead of a mysterious hang up.

3. Wrap the fetch in a retry with exponential backoff and jitter, retrying only idempotent methods (GET, HEAD) and only 2 or 3 times. Expected: transient hang ups succeed on retry, and tail shows the retry count per incident.

4. Check the origin directly, bypassing the worker. Time the same request from your machine with curl. Expected: you see whether the origin itself is slow or the problem is in the path to it.

5. Fix the origin slowness: add caching for repeated responses, fix slow queries, or move the slow computation behind the worker so the origin answers fast. Expected: origin p99 drops and hang ups stop even without retries.

6. Watch tail for a day of real traffic. Expected: hang up lines disappear or become rare single events that the retry absorbs.

Use this when

• a worker fetching an origin fails intermittently with socket hang up
• failures rise when the origin is slow or under load
• you proxy API calls or pages through the worker to a backend
• the same fetch works from your machine but fails from the worker under load

Not for this skill when

• every fetch fails immediately (that is DNS, TLS, or firewall, not a slow origin)
• the origin returns 5xx instead of dropping the connection (that is an origin bug to fix directly)
• the failure is the worker's own CPU or subrequest limit
• the socket drops on the client-to-worker leg rather than worker-to-origin

Variant phrasings

• Cloudflare Worker fetch socket hang up
• "upstream socket hang up" workerd
• worker fetch fails slow origin
• ECONNRESET Cloudflare Worker origin fetch
• origin closes connection worker fetch

Why it happens

"Socket hang up" means the TCP connection died before the response completed. With slow origins the usual cause is a timeout somewhere in the chain: the origin's server, a load balancer, or the worker's own fetch giving up on a response that takes too long. Retries paper over the transient drops, but the structural fix is making the origin answer faster than whatever is timing it out.

Edge cases

• Do not retry non-idempotent requests blindly. Retrying a POST can double-charge or double-create. Make the operation idempotent first or accept the failure.
• AbortSignal.timeout aborts the fetch but the origin may still be working. For expensive origin operations, that means wasted origin work on every timeout.
• Keep-alive to the origin helps only if the origin supports it and the drops are on connection setup. It does not fix slow responses.
• If the origin is another worker, check its CPU and subrequest limits. A downstream worker timing out looks exactly like a slow origin.
• Backoff without jitter synchronizes retries into thundering herds. Always add jitter.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_AKhdXBMX3UYhKzd3GKv0Ag