context deadline exceeded in worker log when starting a workflow after upgrading to 1.19.0

TL;DR

Root cause: the ringpop service-discovery startup timeout was too short in 1.19.x, so on quick restarts services could not discover each other and the cluster wedged. Fixed upstream in PR #3911; upgrade to a 1.19.x release containing it.

Steps

1. Root cause: the ringpop service-discovery startup timeout was too short in 1.19.x, so on quick restarts services could not discover each other and the cluster wedged. Fixed upstream in PR #3911; upgrade to a 1.19.x release containing it. Workarounds confirmed in the thread: wait at least 20 seconds between stopping all services and starting them again, or stop everything, clear the cluster_membership table, and restart. A reporter verified the fix by killing all pods 20 times in a row with the cluster coming back healthy every time.

When to use

You are seeing this: After upgrading Temporal from 1.18.x to 1.19.0 (fresh deployment, no data migration), the worker log shows: error starting temporal-sys-history-scanner-workflow workflow, error: context deadline exceeded. Use this skill when you run into "context deadline exceeded in worker log when starting a workflow after upgrading to 1.19.0".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: 1.19.0, 1.18.5. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Asked in the thread (https://vectle.com/threads/thrN4rntwajDUl5v4NzLtUw9Q): After upgrading Temporal from 1.18.x to 1.19.0 (fresh deployment, no data migration), the worker log shows: error starting temporal-sys-history-scanner-workflow workflow, error: context deadline exceeded. Frontend/ringpop logs show services appearing and disappearing ("Current reachable members" flapping), and the cluster stays broken. Downgrading to 1.18.5 makes it go away. The verified answer: Root cause: the ringpop service-discovery startup timeout was too short in 1.19.x, so on quick restarts services could not discover each other and the cluster wedged. Fixed upstream in PR #3911; upgrade to a 1.19.x release containing it. Workarounds confirmed in the thread: wait at least 20 seconds between stopping all services and starting them again, or stop everything, clear the clustermembership table, and restart. A reporter verified the fix by killing all pods 20 times in a row with the cluster coming back healthy every time.

Source: https://github.com/temporalio/temporal/issues/3826 Thread: https://vectle.com/threads/thr_N4rntwajDUl5v4NzLtUw9Q