Immutable solution version
OpenSearch 3.x snapshot repository fails on S3-compatible storage: "trailing checksum is not supported"
Complete solution
What caused "OpenSearch 3.x snapshot repository fails on S3-compatible storage: "trailing checksum is not supported"" and the fix that actually worked. Distilled from the thread where it was verified.
OpenSearch 3.x snapshot repository fails on S3-compatible storage: "trailing checksum is not supported"
What was going on
After upgrading to OpenSearch 3.0.0, snapshot repositories on S3-compatible storage (e.g. Outscale) stopped working. The repository-s3 plugin in 3.0.0 upgraded the AWS SDK to 2.30.31, and testing a repository failed with: "Caused by: software.amazon.awssdk.services.s3.model.S3Exception: trailing checksum is not supported (Service: S3, Status Code: 400 ...)". The documented workaround of disabling checksum calculation via env vars did not help. Users confirmed the bug persisted in 3.1.0 and 3.2.0, blocking upgrades.
The fix that worked
The 3.0.0 repository-s3 plugin's newer AWS SDK sends trailing checksums that S3-compatible stores reject. Verified workaround (confirmed by multiple users): set legacy MD5 checksum calculation per repository via the create-repository API, e.g. settings {"bucket": "bucketname", "basepath": "pathinbucket", "legacymd5checksumcalculation": "true"}. Permanent fix: maintainer cwperks raised PR #19788 to make this settable in opensearch.yml; it shipped in 3.3.2, where users confirmed s3.client.default.legacymd5checksumcalculation works without per-repo config. Resolution: upgrade to OpenSearch 3.3.2+ and set s3.client.default.legacymd5checksumcalculation: true, or apply the per-repository setting on older 3.x. (An earlier community workaround was rebuilding the repository-s3 plugin zip against AWS SDK 2.29.52.) Source: https://github.com/opensearch-project/OpenSearch/issues/18240
Where this came from
https://github.com/opensearch-project/OpenSearch/issues/18240
Evidence
Someone asked: After upgrading to OpenSearch 3.0.0, snapshot repositories on S3-compatible storage (e.g. Outscale) stopped working. The repository-s3 plugin in 3.0.0 upgraded the AWS SDK to 2.30.31, and testing a repository failed with: "Caused by: software.amazon.awssdk.services.s3.model.S3Exception: trailing checksum is not supported (Service: S3, Status Code: 400 ...)". The documented workaround of disabling checksum calculation via env vars did not help. Users confirmed the bug persisted in 3.1.0 and 3.2.0, blocking upgrades. The verified resolution: The 3.0.0 repository-s3 plugin's newer AWS SDK sends trailing checksums that S3-compatible stores reject. Verified workaround (confirmed by multiple users): set legacy MD5 checksum calculation per repository via the create-repository API, e.g. settings {"bucket": "bucketname", "basepath": "pathinbucket", "legacymd5checksumcalculation": "true"}. Permanent fix: maintainer cwperks raised PR #19788 to make this settable in opensearch.yml; it shipped in 3.3.2, where users confirmed s3.client.default.legacymd5checksumcalculation works without per-repo config. Resolution: upgrade to OpenSearch 3.3.2+ and set s3.client.default.legacymd5checksumcalculation: true, or apply the per-repository setting on older 3.x. (An earlier community workaround was rebuilding the repository-s3 plugin zip against AWS SDK 2.29.52.) Source: https://github.com/opensearch-project/OpenSearch/issues/18240 Thread: https://vectle.com/threads/thr_HIxu0iyHYrZrRuq3Ctn1ig
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_DsmdwnPO3TeaKVKJS3lpDQ" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">OpenSearch 3.x snapshot repository fails on S3-compatible storage: "trailing checksum is not supported" <span>— Vectle</span></a>