Performance degradation after upgrading from 8.6.1 to 8.16.1

TL;DR

Investigation pointed at file descriptor leaks on the upgraded nodes. The thread traced the degradation to a file descriptor leak on the upgraded nodes: lots of descriptors held on deleted docvalues.dvd files, a much higher flush rate, and high CPU.

Steps

1. Issue elastic/elasticsearch#118623 (closed, 21 comments): Performance degradation after upgrading from 8.6.1 to 8.16.1. Investigation pointed at file descriptor leaks on the upgraded nodes.
2. If your cluster slows down after upgrading from 8.6.x to 8.16.x, check for file descriptor pressure before adding hardware. The thread traced the degradation to a file descriptor leak on the upgraded nodes: lots of descriptors held on deleted docvalues .dvd files, a much higher flush rate, and high CPU. Setting sharedArenaMaxPermits=1 resolved it for the reporters, who confirmed it against the same symptoms in a separate discuss thread.

When to use

You are seeing this: Issue elastic/elasticsearch#118623 (closed, 21 comments): Performance degradation after upgrading from 8.6.1 to 8.16.1. Use this skill when you run into "Performance degradation after upgrading from 8.6.1 to 8.16.1".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: 8.6.1, 8.16.1. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Sources

• https://github.com/elastic/elasticsearch/issues/118623

In gh:elastic/elasticsearch#118623, the reporter ran into: Issue elastic/elasticsearch#118623 (closed, 21 comments): Performance degradation after upgrading from 8.6.1 to 8.16.1. Investigation pointed at file descriptor leaks on the upgraded nodes. The verified answer was: If your cluster slows down after upgrading from 8.6.x to 8.16.x, check for file descriptor pressure before adding hardware. The thread traced the degradation to a file descriptor leak on the upgraded nodes: lots of descriptors held on deleted docvalues .dvd files, a much higher flush rate, and high CPU. Setting sharedArenaMaxPermits=1 resolved it for the reporters, who confirmed it against the same symptoms in a separate discuss thread. Source issue: https://github.com/elastic/elasticsearch/issues/118623