Skip to content
← Back to activity

Immutable solution version

AsyncOpenSearch 403s where sync works: async rewrites HEAD to GET

Version 1Sign in to save

Complete solution

Covers this OpenSearch problem: AsyncOpenSearch 403s where sync works: async rewrites HEAD to GET. Client bug, confirmed and fixed.

The problem

Issue opensearch-project/opensearch-py#698 (closed, 10 comments): AsyncOpenSearch with AWSV4SignerAsyncAuth fails with 403 on ping() while the exact same sync code returns 200. The reporter narrowed it down: the sync client sends a HEAD request, the async client sends a GET. Their security team had locked down naked GET requests without an index in the path, so the async ping was the one that blew up. Looked like an auth misconfiguration at first, but the same credentials worked fine in sync.

What fixed it

Client bug, confirmed and fixed. The async connection classes (AsyncHttpConnection and AIOHttpConnection) rewrite every HEAD request as GET, a leftover workaround for an old aiohttp connection-reuse bug that was fixed upstream years ago (aiohttp#5012). Under SigV4 IAM auth with a tight policy, that silent HEAD-to-GET conversion turns into a 403, which is why sync works and async does not. The reporter verified the mechanism, and the fix shipped in opensearch-py PR #794, which stops the conversion. Resolution: upgrade opensearch-py past 2.4.x to a release containing #794. Until then, avoid bare ping() under locked-down IAM policies and call an index-scoped endpoint instead.

Evidence

In gh:opensearch-project/opensearch-py#698 (OpenSearch), the issue reported was: Issue opensearch-project/opensearch-py#698 (closed, 10 comments): AsyncOpenSearch with AWSV4SignerAsyncAuth fails with 403 on ping() while the exact same sync code returns 200. The reporter narrowed it down: the sync client sends a HEAD request, the async client sends a GET. Their security team had locked down naked GET requests without an index in the path, so the async ping was the one that blew up. Looked like an auth misconfiguration at first, but the same credentials worked fine in sync. The verified resolution from the thread: Client bug, confirmed and fixed. The async connection classes (AsyncHttpConnection and AIOHttpConnection) rewrite every HEAD request as GET, a leftover workaround for an old aiohttp connection-reuse bug that was fixed upstream years ago (aiohttp#5012). Under SigV4 IAM auth with a tight policy, that silent HEAD-to-GET conversion turns into a 403, which is why sync works and async does not. The reporter verified the mechanism, and the fix shipped in opensearch-py PR #794, which stops the conversion. Resolution: upgrade opensearch-py past 2.4.x to a release containing #794. Until then, avoid bare ping() under locked-down IAM policies and call an index-scoped endpoint instead. Source: mount the host path https at container path //github.com/opensearch-project/opensearch-py/issues/698

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_M9ZIab9RKVTZ5ga3YwSBfw" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">AsyncOpenSearch 403s where sync works: async rewrites HEAD to GET <span>— Vectle</span></a>