Performance degradation after upgrading from 8.6.1 to 8.16.1

What this teaches

How to performance degradation after upgrading from 8.6.1 to 8.16.1.

The problem

Issue elastic/elasticsearch#118623 (closed, 21 comments): Performance degradation after upgrading from 8.6.1 to 8.16.1. Investigation pointed at file descriptor leaks on the upgraded nodes.

What worked (verified answer)

If your cluster slows down after upgrading from 8.6.x to 8.16.x, check for file descriptor pressure before adding hardware. The thread traced the degradation to a file descriptor leak on the upgraded nodes: lots of descriptors held on deleted docvalues .dvd files, a much higher flush rate, and high CPU. Setting sharedArenaMaxPermits=1 resolved it for the reporters, who confirmed it against the same symptoms in a separate discuss thread.

Source

gh:elastic/elasticsearch#118623 - https://github.com/elastic/elasticsearch/issues/118623

In gh:elastic/elasticsearch#118623, the reporter ran into: Issue elastic/elasticsearch#118623 (closed, 21 comments): Performance degradation after upgrading from 8.6.1 to 8.16.1. Investigation pointed at file descriptor leaks on the upgraded nodes. The verified answer was: If your cluster slows down after upgrading from 8.6.x to 8.16.x, check for file descriptor pressure before adding hardware. The thread traced the degradation to a file descriptor leak on the upgraded nodes: lots of descriptors held on deleted docvalues .dvd files, a much higher flush rate, and high CPU. Setting sharedArenaMaxPermits=1 resolved it for the reporters, who confirmed it against the same symptoms in a separate discuss thread. Source issue: https://github.com/elastic/elasticsearch/issues/118623