Stale internode gRPC connections after pod termination when using Kubernetes Deployments

Fix first: upgrade the Temporal server to v1.31.1 or later. That patch release closes and removes references to stale connections, which the maintainer verified.

dial tcp ... i/o timeout

1. Upgrade your Temporal server to v1.31.1 or later.
2. Run a scale-down or rolling update and watch the logs. Expected: no more repeated dial attempts to stale pod IPs.

When to use: Temporal servers keep dialing stale pod IPs after Kubernetes scale-downs, rolling updates, or node drains.

When not to use: client-side connection issues (this is server-to-server internode traffic).

Compatibility: Temporal server v1.31.1+. Reported in temporalio/temporal#8719.

Variants:

• "temporal stale grpc connections kubernetes"
• "temporal dial tcp i/o timeout pod termination"
• "temporal internode connections scale down"

Root cause: the server didnt drop internode gRPC connections to pods removed from the membership ring, so it kept redialing dead IPs.

Edge cases: the client SDK version doesnt matter here; the fix is server-side.

In gh:temporalio/temporal#8719 the issue was: Issue temporalio/temporal#8719 (closed, 11 comments): ## Expected Behavior Temporal should drop internode gRPC connections to pods that no longer exist after they are removed from the membership ring, avoiding repeated dial attempts to stale pod IPs.

Actual Behavior

We are experiencing the same issue described in PR https://github.com/temporalio/temporal/pull/8586 Temporal servers keep stale internode gRPC connections after Kubernetes pods are terminated (during scale-down, rolling updates, or node drains). This results in continuous dial tcp … i/o timeout errors... The verified answer: If Temporal servers keep dialing stale pod IPs after Kubernetes scale-downs or rolling updates (continuous dial tcp i/o timeout errors), upgrade the server to v1.31.1 or later. That patch release includes the fix that closes and removes references to stale connections, which the maintainer verified. Source: https://github.com/temporalio/temporal/issues/8719