clickhouse server ever increasing memory usage

TL;DR

Upgrade to 24.7 or newer and the growth stops. Ever-increasing ClickHouse server memory usage was a leak fixed in 24.7 (PR #65839).

Steps

1. https://github.com/ClickHouse/clickhouse/issues/65600: Issue ClickHouse/clickhouse#65600 (closed, 27 comments): ### Company or project name No response ### Question We are running a 6-node 3S-2R clickhouse cluster and I have noticed that the memory usage of the nodes keeps on increasing. Our usecase if that we are inserting the data from kafka topics and also using s3-queue to ingest alb logs data. [image] Since 2-3 days, It has reached the AWS EC2 instance memory limit and I am getting the following errors ``` 2024.06.21 12:08:39.291296 [ 2697 ] {} [Error] MergeTreeBackgroundExecutor: Exception while executing background task {0c2385f1-5800-4375-afa1-ad7c65c4d333::20240613649301377339_149}: Code: 2

When to use

Use this skill when you run into "clickhouse server ever increasing memory usage".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: 2024.06.21. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Sources

• https://github.com/ClickHouse/clickhouse/issues/65600:

Issue gh:ClickHouse/clickhouse#65600: the reporter asked: Issue ClickHouse/clickhouse#65600 (closed, 27 comments): ### Company or project name No response ### Question We are running a 6-node 3S-2R clickhouse cluster and I have noticed that the memory usage of the nodes keeps on increasing. Our usecase if that we are inserting the data from kafka topics and also using s3-queue to ingest alb logs data. [image] Since 2-3 days, It has reached the AWS EC2 instance memory limit and I am getting the following errors ``` 2024.06.21 12:08:39.291296 [ 2697 ] {} [Error] MergeTreeBackgroundExecutor: Exception while executing background task {0c2385f1-5800-4375-afa1-ad7c65c4d333::20240613649301377339_149}: Code: 2 The verified answer was: Ever-increasing ClickHouse server memory usage was a leak fixed in 24.7 (PR #65839). Upgrade to 24.7 or newer and the growth stops. Evidence thread: https://github.com/ClickHouse/clickhouse/issues/65600