OpenSearch 3.x snapshot repository fails on S3-compatible storage: "trailing checksum is not supported"

TL;DR

The repository-s3 plugin in 3.0.0 upgraded the AWS SDK to 2.30.31, and testing a repository failed with: "Caused by: software.amazon.awssdk.services.s3.model.S3Exception: trailing checksum is not supported (Service: S3, Status Code: 400...)". Users confirmed the bug persisted in 3.1.0 and 3.2.0, blocking upgrades.

When to use

You are seeing this: After upgrading to OpenSearch 3.0.0, snapshot repositories on S3-compatible storage (e.g. Use this skill when you run into "OpenSearch 3.x snapshot repository fails on S3-compatible storage: "trailing checksum is not supported"".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: OpenSearch 3.0.0, 2.30.31, 3.1.0, 3.2.0, 3.3.2, SDK 2.29.52. If you are on something much newer or older, the details may have shifted.

What was going on

After upgrading to OpenSearch 3.0.0, snapshot repositories on S3-compatible storage (e.g. Outscale) stopped working. The repository-s3 plugin in 3.0.0 upgraded the AWS SDK to 2.30.31, and testing a repository failed with: "Caused by: software.amazon.awssdk.services.s3.model.S3Exception: trailing checksum is not supported (Service: S3, Status Code: 400 ...)". The documented workaround of disabling checksum calculation via env vars did not help. Users confirmed the bug persisted in 3.1.0 and 3.2.0, blocking upgrades.

The fix that worked

The 3.0.0 repository-s3 plugin's newer AWS SDK sends trailing checksums that S3-compatible stores reject. Verified workaround (confirmed by multiple users): set legacy MD5 checksum calculation per repository via the create-repository API, e.g. settings {"bucket": "bucketname", "basepath": "pathinbucket", "legacymd5checksumcalculation": "true"}. Permanent fix: maintainer cwperks raised PR #19788 to make this settable in opensearch.yml; it shipped in 3.3.2, where users confirmed s3.client.default.legacymd5checksumcalculation works without per-repo config. Resolution: upgrade to OpenSearch 3.3.2+ and set s3.client.default.legacymd5checksumcalculation: true, or apply the per-repository setting on older 3.x. (An earlier community workaround was rebuilding the repository-s3 plugin zip against AWS SDK 2.29.52.) Source: https://github.com/opensearch-project/OpenSearch/issues/18240

Where this came from

https://github.com/opensearch-project/OpenSearch/issues/18240

Someone asked: After upgrading to OpenSearch 3.0.0, snapshot repositories on S3-compatible storage (e.g. Outscale) stopped working. The repository-s3 plugin in 3.0.0 upgraded the AWS SDK to 2.30.31, and testing a repository failed with: "Caused by: software.amazon.awssdk.services.s3.model.S3Exception: trailing checksum is not supported (Service: S3, Status Code: 400 ...)". The documented workaround of disabling checksum calculation via env vars did not help. Users confirmed the bug persisted in 3.1.0 and 3.2.0, blocking upgrades. The verified resolution: The 3.0.0 repository-s3 plugin's newer AWS SDK sends trailing checksums that S3-compatible stores reject. Verified workaround (confirmed by multiple users): set legacy MD5 checksum calculation per repository via the create-repository API, e.g. settings {"bucket": "bucketname", "basepath": "pathinbucket", "legacymd5checksumcalculation": "true"}. Permanent fix: maintainer cwperks raised PR #19788 to make this settable in opensearch.yml; it shipped in 3.3.2, where users confirmed s3.client.default.legacymd5checksumcalculation works without per-repo config. Resolution: upgrade to OpenSearch 3.3.2+ and set s3.client.default.legacymd5checksumcalculation: true, or apply the per-repository setting on older 3.x. (An earlier community workaround was rebuilding the repository-s3 plugin zip against AWS SDK 2.29.52.) Source: https://github.com/opensearch-project/OpenSearch/issues/18240 Thread: https://vectle.com/threads/thr_HIxu0iyHYrZrRuq3Ctn1ig