Immutable solution version
Stale internode gRPC connections after pod termination when using Kubernetes Deployments
Complete solution
How to fix stale internode gRPC connections after pod termination when using Kubernetes Deployments (Temporal). The verified answer from the upstream issue, distilled into steps.
Stale internode gRPC connections after pod termination when using Kubernetes Deployments
What was going on Issue temporalio/temporal#8719 (closed, 11 comments): ## Expected Behavior Temporal should drop internode gRPC connections to pods that no longer exist after they are removed from the membership ring, avoiding repeated dial attempts to stale pod IPs.
Actual Behavior
We are experiencing the same issue described in PR https://github.com/temporalio/temporal/pull/8586 Temporal servers keep stale internode gRPC connections after Kubernetes pods are terminated (during scale-down, rolling updates, or node drains). This results in continuous dial tcp … i/o timeout errors...
The fix If Temporal servers keep dialing stale pod IPs after Kubernetes scale-downs or rolling updates (continuous dial tcp i/o timeout errors), upgrade the server to v1.31.1 or later. That patch release includes the fix that closes and removes references to stale connections, which the maintainer verified.
Evidence
In gh:temporalio/temporal#8719 the issue was: Issue temporalio/temporal#8719 (closed, 11 comments): ## Expected Behavior Temporal should drop internode gRPC connections to pods that no longer exist after they are removed from the membership ring, avoiding repeated dial attempts to stale pod IPs.
Actual Behavior
We are experiencing the same issue described in PR https://github.com/temporalio/temporal/pull/8586 Temporal servers keep stale internode gRPC connections after Kubernetes pods are terminated (during scale-down, rolling updates, or node drains). This results in continuous dial tcp … i/o timeout errors... The verified answer: If Temporal servers keep dialing stale pod IPs after Kubernetes scale-downs or rolling updates (continuous dial tcp i/o timeout errors), upgrade the server to v1.31.1 or later. That patch release includes the fix that closes and removes references to stale connections, which the maintainer verified. Source: https://github.com/temporalio/temporal/issues/8719
By @issueatlas · Version 1 · Exact version
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_GOg3c1CMv3f5Vaw8dcAgiQ" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Stale internode gRPC connections after pod termination when using Kubernetes Deployments <span>— Vectle</span></a>