python client to AWS Elasticsearch: requests must be SigV4-signed

python client to AWS Elasticsearch: requests must be SigV4-signed

TL;DR

[accepted answer]: AWS-hosted Elasticsearch requires every request to be signed with SigV4; plain unsigned requests are rejected, which is why curl (if it worked) or the default client fails. Wire the python client like this: from elasticsearch import Elasticsearch, RequestsHttpConnection from requestsaws4auth import AWS4Auth Build an AWS4Auth signer from your AWS access key, your AWS secret key, the region, and the service name "es". Then: es = Elasticsearch(hosts=[{"host": "[your-domain-endpoint]", "port": 443}], httpauth=[signer], usessl=True, verifycerts=True, connection_class=RequestsHttpConnection) Use the domain endpoint (my-domain.us-east-1.es.amazonaws.com style), NOT the ec2 compute hostname, and port 443 with SSL.

Use this when you run into the situation in the title.

When not to use this skill: unrelated tasks. It covers only the procedure above.

Compatibility

Tool: python. The steps above apply to the commands named in them. This skill does not pin a version, so if a flag looks different on your machine, check your installed version's docs first.

Details

Put the credentials in env vars and read them at runtime; never hardcode them.

Context: Stack Overflow #38528839 (accepted answer, 11 votes): the python elasticsearch client works from curl against an AWS-hosted Elasticsearch instance but fails from python. The asker passes the AWS host as a plain URL and does not sign requests.

Source: Stack Overflow #38528839 (accepted answer, 11 votes): the python elasticsearch client works from curl against an AWS-hosted Elasticsearch instance but fails from python. The asker passes the AWS host as a plain URL and does not sign requests.

Verified resolution: [accepted answer]: AWS-hosted Elasticsearch requires every request to be signed with SigV4; plain unsigned requests are rejected, which is why curl (if it worked) or the default client fails. Wire the python client like this: from elasticsearch import Elasticsearch, RequestsHttpConnection from requestsaws4auth import AWS4Auth Build an AWS4Auth signer from your AWS access key, your AWS secret key, the region, and the service name "es". Then: es = Elasticsearch(hosts=[{"host": "[your-domain-endpoint]", "port": 443}], httpauth=[signer], usessl=True, verifycerts=True, connection_class=RequestsHttpConnection) Use the domain endpoint (my-domain.us-east-1.es.amazonaws.com style), NOT the ec2 compute hostname, and port 443 with SSL. Put the credentials in env vars and read them at runtime; never hardcode them.

Source URL: https://stackoverflow.com/questions/38528839