RudderStack server high CPU utilization even when idle starting with 1.17.0

TL;DR

The fix shipped in v1.17.3, and the original reporter confirmed the new release fixed the problem. Fix: upgrade rudder-server to v1.17.3 or later, which contains the fix for the query loop.

Steps

1. Versions 1.17.0 through 1.17.2 had a bug that made rudder-server hammer Postgres with a repeated query, spiking CPU on both even when idle. Fix: upgrade rudder-server to v1.17.3 or later, which contains the fix for the query loop. 1.16.3 was the last unaffected version if you need to roll back instead of forward. The reporter bisected the regression to 1.17.0-rc1 and confirmed v1.17.3 resolved it. Source: https://github.com/rudderlabs/rudder-server/issues/4187

When to use

You are seeing this: After upgrading rudder-server from 1.16.3 to 1.17.2, both the server and PostgreSQL CPU usage jumped by an order of magnitude, even with no traffic. Use this skill when you run into "RudderStack server high CPU utilization even when idle starting with 1.17.0".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: 1.16.3, 1.17.2, found 1.17.0-rc1, v1.17.3, Versions 1.17.0. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Sources

• https://github.com/rudderlabs/rudder-server/issues/4187

Question asked in the thread: After upgrading rudder-server from 1.16.3 to 1.17.2, both the server and PostgreSQL CPU usage jumped by an order of magnitude, even with no traffic. The reporter bisected the releases and found 1.17.0-rc1 was the first bad release. Another user captured traffic showing rudder-server looping on a Postgres query at a high rate, which explained the CPU burn. A maintainer confirmed they had identified the root cause and were working on a fix. The fix shipped in v1.17.3, and the original reporter confirmed the new release fixed the problem.

Verified answer: Versions 1.17.0 through 1.17.2 had a bug that made rudder-server hammer Postgres with a repeated query, spiking CPU on both even when idle. Fix: upgrade rudder-server to v1.17.3 or later, which contains the fix for the query loop. 1.16.3 was the last unaffected version if you need to roll back instead of forward. The reporter bisected the regression to 1.17.0-rc1 and confirmed v1.17.3 resolved it. Source: https://github.com/rudderlabs/rudder-server/issues/4187

Thread: https://vectle.com/threads/thr_igzykcjNkB2otsC-veajhw