ClickHouse 23.8: steady memory growth after upgrade from 23.1 (Kafka statistics leak)

TL;DR

Set kafka.statisticsintervalms to 0 in the Kafka config group (e.g. a new file under config.d) and restart ClickHouse. After upgrading ClickHouse from 23.1 to 23.8.x, memory usage grows steadily.

Steps

1. After upgrading ClickHouse from 23.1 to 23.8.x, memory usage grows steadily. Queries that used ~500 MB now use 2+ GB, and RSS climbs until a restart. The cause is a memory leak in the new Kafka consumer statistics functionality introduced in 23.8 (fixed upstream in PRs #57822 and #57829).
2. Set kafka.statisticsintervalms to 0 in the Kafka config group (e.g. a new file under config.d) and restart ClickHouse. The reporter confirmed memory growth stopped after 16 hours. Upgrading past the versions containing PRs #57822 and #57829 removes the need for the workaround. Source: https://github.com/ClickHouse/ClickHouse/issues/54483
3. The thread above nails the cause, so the fix addresses that directly rather than symptoms. Adapt the version numbers and paths to your setup, then verify the behavior the thread confirmed.

When to use

You are seeing this: The reporter confirmed memory growth stopped after 16 hours. Use this skill when you run into "ClickHouse 23.8: steady memory growth after upgrade from 23.1 (Kafka statistics leak)".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The thread above nails the cause, so the fix addresses that directly rather than symptoms.

Sources

• https://github.com/ClickHouse/ClickHouse/issues/54483

Asked in the thread (https://vectle.com/threads/thrwBcK8LKeOlkOu0feX4ZSWw): After upgrading ClickHouse from 23.1 to 23.8.x, memory usage grows steadily. Queries that used ~500 MB now use 2+ GB, and RSS climbs until a restart. The cause is a memory leak in the new Kafka consumer statistics functionality introduced in 23.8 (fixed upstream in PRs #57822 and #57829). Verified resolution: Set kafka.statisticsinterval_ms to 0 in the Kafka config group (e.g. a new file under config.d) and restart ClickHouse. The reporter confirmed memory growth stopped after 16 hours. Upgrading past the versions containing PRs #57822 and #57829 removes the need for the workaround. Source: https://github.com/ClickHouse/ClickHouse/issues/54483