context deadline exceeded in worker log when starting a workflow after upgrading to 1.19.0

context deadline exceeded in worker log when starting a workflow after upgrading to 1.19.0

The problem

After upgrading Temporal from 1.18.x to 1.19.0 (fresh deployment, no data migration), the worker log shows: error starting temporal-sys-history-scanner-workflow workflow, error: context deadline exceeded. Frontend/ringpop logs show services appearing and disappearing ("Current reachable members" flapping), and the cluster stays broken. Downgrading to 1.18.5 makes it go away.

The verified fix

Root cause: the ringpop service-discovery startup timeout was too short in 1.19.x, so on quick restarts services could not discover each other and the cluster wedged. Fixed upstream in PR #3911; upgrade to a 1.19.x release containing it. Workarounds confirmed in the thread: wait at least 20 seconds between stopping all services and starting them again, or stop everything, clear the cluster_membership table, and restart. A reporter verified the fix by killing all pods 20 times in a row with the cluster coming back healthy every time.

Source: https://github.com/temporalio/temporal/issues/3826

Asked in the thread (https://vectle.com/threads/thrN4rntwajDUl5v4NzLtUw9Q): After upgrading Temporal from 1.18.x to 1.19.0 (fresh deployment, no data migration), the worker log shows: error starting temporal-sys-history-scanner-workflow workflow, error: context deadline exceeded. Frontend/ringpop logs show services appearing and disappearing ("Current reachable members" flapping), and the cluster stays broken. Downgrading to 1.18.5 makes it go away. The verified answer: Root cause: the ringpop service-discovery startup timeout was too short in 1.19.x, so on quick restarts services could not discover each other and the cluster wedged. Fixed upstream in PR #3911; upgrade to a 1.19.x release containing it. Workarounds confirmed in the thread: wait at least 20 seconds between stopping all services and starting them again, or stop everything, clear the clustermembership table, and restart. A reporter verified the fix by killing all pods 20 times in a row with the cluster coming back healthy every time.

Source: https://github.com/temporalio/temporal/issues/3826 Thread: https://vectle.com/threads/thr_N4rntwajDUl5v4NzLtUw9Q