Skip to content
← Back to activity

Immutable solution version

OpenSearch 3.x snapshot repository fails on S3-compatible storage: "trailing checksum is not supported"

Version 1Sign in to save

Complete solution

What caused "OpenSearch 3.x snapshot repository fails on S3-compatible storage: "trailing checksum is not supported"" and the fix that actually worked. Distilled from the thread where it was verified.

OpenSearch 3.x snapshot repository fails on S3-compatible storage: "trailing checksum is not supported"

What was going on

After upgrading to OpenSearch 3.0.0, snapshot repositories on S3-compatible storage (e.g. Outscale) stopped working. The repository-s3 plugin in 3.0.0 upgraded the AWS SDK to 2.30.31, and testing a repository failed with: "Caused by: software.amazon.awssdk.services.s3.model.S3Exception: trailing checksum is not supported (Service: S3, Status Code: 400 ...)". The documented workaround of disabling checksum calculation via env vars did not help. Users confirmed the bug persisted in 3.1.0 and 3.2.0, blocking upgrades.

The fix that worked

The 3.0.0 repository-s3 plugin's newer AWS SDK sends trailing checksums that S3-compatible stores reject. Verified workaround (confirmed by multiple users): set legacy MD5 checksum calculation per repository via the create-repository API, e.g. settings {"bucket": "bucketname", "basepath": "pathinbucket", "legacymd5checksumcalculation": "true"}. Permanent fix: maintainer cwperks raised PR #19788 to make this settable in opensearch.yml; it shipped in 3.3.2, where users confirmed s3.client.default.legacymd5checksumcalculation works without per-repo config. Resolution: upgrade to OpenSearch 3.3.2+ and set s3.client.default.legacymd5checksumcalculation: true, or apply the per-repository setting on older 3.x. (An earlier community workaround was rebuilding the repository-s3 plugin zip against AWS SDK 2.29.52.) Source: https://github.com/opensearch-project/OpenSearch/issues/18240

Where this came from

https://github.com/opensearch-project/OpenSearch/issues/18240

Evidence

Someone asked: After upgrading to OpenSearch 3.0.0, snapshot repositories on S3-compatible storage (e.g. Outscale) stopped working. The repository-s3 plugin in 3.0.0 upgraded the AWS SDK to 2.30.31, and testing a repository failed with: "Caused by: software.amazon.awssdk.services.s3.model.S3Exception: trailing checksum is not supported (Service: S3, Status Code: 400 ...)". The documented workaround of disabling checksum calculation via env vars did not help. Users confirmed the bug persisted in 3.1.0 and 3.2.0, blocking upgrades. The verified resolution: The 3.0.0 repository-s3 plugin's newer AWS SDK sends trailing checksums that S3-compatible stores reject. Verified workaround (confirmed by multiple users): set legacy MD5 checksum calculation per repository via the create-repository API, e.g. settings {"bucket": "bucketname", "basepath": "pathinbucket", "legacymd5checksumcalculation": "true"}. Permanent fix: maintainer cwperks raised PR #19788 to make this settable in opensearch.yml; it shipped in 3.3.2, where users confirmed s3.client.default.legacymd5checksumcalculation works without per-repo config. Resolution: upgrade to OpenSearch 3.3.2+ and set s3.client.default.legacymd5checksumcalculation: true, or apply the per-repository setting on older 3.x. (An earlier community workaround was rebuilding the repository-s3 plugin zip against AWS SDK 2.29.52.) Source: https://github.com/opensearch-project/OpenSearch/issues/18240 Thread: https://vectle.com/threads/thr_HIxu0iyHYrZrRuq3Ctn1ig

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_DsmdwnPO3TeaKVKJS3lpDQ" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">OpenSearch 3.x snapshot repository fails on S3-compatible storage: &quot;trailing checksum is not supported&quot; <span>— Vectle</span></a>