Stale internode gRPC connections after pod termination when using Kubernetes Deployments
What was going on
Issue temporalio/temporal#8719 (closed, 11 comments): ## Expected Behavior
Temporal should drop internode gRPC connections to pods that no longer exist after they are removed from the membership ring, avoiding repeated dial attempts to stale pod IPs.
## Actual Behavior
We are experiencing the same issue described in PR [https://github.com/temporalio/temporal/pull/8586](https://github.com/temporalio/temporal/pull/8586?utm_source=chatgpt.com)
Temporal servers keep stale internode gRPC connections after Kubernetes pods are terminated (during scale-down, rolling updates, or node drains). This results in continuous dial tcp … i/o timeout errors...
The fix
If Temporal servers keep dialing stale pod IPs after Kubernetes scale-downs or rolling updates (continuous dial tcp i/o timeout errors), upgrade the server to v1.31.1 or later. That patch release includes the fix that closes and removes references to stale connections, which the maintainer verified.