Unable to update Elasticsearch mapping: elastic: Error 403 on Create Custom Attributes

Not specified by the original author.

TL;DR: if temporal operator search-attribute create fails with unable to update Elasticsearch mapping: elastic: Error 403 even as the OpenSearch master user, check for dual visibility mode. Adding search attributes is not supported when both visibilityStore (Postgres) and advancedVisibilityStore (OpenSearch) are set; that mode exists only for migrating visibility stores. Keep only advancedVisibilityStore and remove the visibilityStore and secondaryVisibilityStore keys plus the datastores.visibility section.

unable to update Elasticsearch mapping: elastic: Error 403 (Forbidden): security_exception

Steps

1. Inspect your Temporal config for both visibilityStore and advancedVisibilityStore being set.
2. Remove the visibilityStore and secondaryVisibilityStore keys and the datastores.visibility section, keeping only advancedVisibilityStore pointing at OpenSearch.

3. Restart Temporal and re-run the search-attribute create.

   Expected: the attribute creates. Success check: the custom search attribute is usable in queries.

When to use this

• Self-hosted Temporal 1.20.4 with AWS OpenSearch, 403 on attribute creation despite full cluster access.
• Dual visibility mode is configured.

When NOT to use this

• Single visibility store already; the 403 is a real permission issue then.
• You are mid-migration and need dual mode; finish the migration first, then add attributes.

Compatibility

• Temporal 1.20.4 with OpenSearch visibility.

Variant phrasings

• "temporal unable to update elasticsearch mapping 403"
• "temporal search attribute dual visibility"

Root cause

The Postgres visibility store does not support custom search attributes anyway, so the dual-mode write path rejects the mapping update with a 403 that looks like a permissions error.

Edge cases

• Do not try to work around it with broader OpenSearch permissions; the master user already failed.

Source: https://vectle.com/threads/thr_gqbJvitEqUKtGa9gjfLBtQ

In the thread https://vectle.com/threads/thrgqbJvitEqUKtGa9gjfLBtQ, the question was: Self-hosted Temporal 1.20.4 with AWS OpenSearch 2.5 and RDS Postgres. Creating a custom search attribute (temporal operator search-attribute create --name CustomTest --type Keyword) fails with: unable to update Elasticsearch mapping: elastic: Error 403 (Forbidden): securityexception, from the AddESMappingFieldActivity, even when using the OpenSearch master user with full cluster access. The verified resolution was: The cluster was in dual visibility mode: both visibilityStore (pointing at Postgres) and advancedVisibilityStore (pointing at OpenSearch) were set. Adding search attributes is not supported in dual visibility mode, which exists only to support migrating visibility stores. The Temporal maintainer's guidance: keep only advancedVisibilityStore and remove the visibilityStore and secondaryVisibilityStore keys plus the datastores.visibility section, since the Postgres visibility store does not support custom search attributes anyway. Diagnosis confirmed by a Temporal maintainer in the thread.

Not specified by the original author.

TL;DR

It's most likely that your instance has been replaced, and therefore the password has been reset. You can check that by going to the tab Instance health -> select an instance -> maximize a graph -> increase the duration to e.g. one week and check for any gaps in the data. When the instance is replaced, all this is gone.

When to use

You are seeing this: Settings like the password are stored on this instance, but I've also noticed it for internal users and index patterns for dashboards. Use this skill when you run into "AWS OpenSearch Service: Master user gets invalid from time to time".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

It's most likely that your instance has been replaced, and therefore the password has been reset. You can check that by going to the tab Instance health -> select an instance -> maximize a graph -> increase the duration to e.g. one week and check for any gaps in the data. Settings like the password are stored on this instance, but I've also noticed it for internal users and index patterns for dashboards. When the instance is replaced, all this is gone. You can prevent this in a couple of ways: - In case you're using an instance in the T range, check if it has high CPU usage. For t3.small, you can run out of CPU credits and the baseline utilization per vCPU is 20% Consider using a non T range instance type. - Use more powerful nodes - Add more nodes

Context: Stack Overflow #79450830 (accepted answer, 5 votes): We are using an AWS stack containing OpenSearch. We have several AWS accounts for different customers, but mainly a similar setup for OpenSearch in all accounts: In each AWS account, there is one PROD and one STAGING OpenSearch domain (PROD and STAGING are our way of configuring and using it, not anything AWS provides). The problem is that from time to time, our master users stop working. Luckily only on the staging domains yet, but who knows? I can't log in to the OpenSearch dashboard with it any more and my app can't via API any more. Our solution is to "create (another?) master user" using...

Source: Stack Overflow #79450830 (accepted answer, 5 votes): We are using an AWS stack containing OpenSearch. We have several AWS accounts for different customers, but mainly a similar setup for OpenSearch in all accounts: In each AWS account, there is one PROD and one STAGING OpenSearch domain (PROD and STAGING are our way of configuring and using it, not anything AWS provides). The problem is that from time to time, our master users stop working. Luckily only on the staging domains yet, but who knows? I can't log in to the OpenSearch dashboard with it any more and my app can't via API any more. Our solution is to "create (another?) master user" using...

Verified resolution: It's most likely that your instance has been replaced, and therefore the password has been reset. You can check that by going to the tab Instance health -> select an instance -> maximize a graph -> increase the duration to e.g. one week and check for any gaps in the data. Settings like the password are stored on this instance, but I've also noticed it for internal users and index patterns for dashboards. When the instance is replaced, all this is gone. You can prevent this in a couple of ways: - In case you're using an instance in the T range, check if it has high CPU usage. For t3.small, you can run out of CPU credits and the baseline utilization per vCPU is 20% Consider using a non T range instance type. - Use more powerful nodes - Add more nodes

Source URL: https://stackoverflow.com/questions/79450830/aws-opensearch-service-master-user-gets-invalid-from-time-to-time

Not specified by the original author.

TL;DR: 403 means you logged in fine but your API key is not allowed to do that. The read-only sounding roles do not cover everything the MCP server needs (like listing indices). Recreate the API key with the missing privileges.

{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/get] is unauthorized for API key id [...]"}]},"status":403}

Fix it

1. Identify the denied action in the error. indices:admin/get is list-indices, indices:data/read/search is search.
2. In Kibana, Stack Management, Security, API Keys, create a new key with explicit role descriptors covering what the server needs. Example for a search workload:

{
  "elasticsearch": {
    "cluster": ["monitor"],
    "indices": [{"names": ["logs-*"], "privileges": ["read", "view_index_metadata"]}]
  }
}

For a quick unblock, superuser works, then scope it down.

3. Replace ES_API_KEY in the MCP server config with the new encoded key. Restart the client.

   Expected: the denied tools now work.

When to use this

• Auth succeeds (no 401) but tools fail with 403 unauthorized.
• Some tools work and others do not. The working ones are within the key's privileges.

When NOT to use this

• The error is 401 missing authentication credentials. No credentials are being sent at all.
• The error is 401 unable to authenticate. The key itself is bad. Make a new one rather than broadening roles.

Compatibility

• elastic/mcp-server-elasticsearch.
• Elasticsearch 8.x with API key auth.

Why it happens

API keys carry role descriptors that gate every action. The minimal-looking roles do not include the admin-ish actions MCP tools need, like listing indices or getting mappings. People create a key with just read and then wonder why half the tools 403.

Edge cases

• Invalidating the old key after switching avoids confusion about which key is live.
• Key privileges are fixed at creation. You cannot widen an existing key; you must create a new one.
• The elastic superuser password works everywhere but should not be the MCP server's daily credential. Use scoped API keys.

The elk-setup troubleshooting table (cloud-prakhar/elk-setup elk-curl-auth-operations.md) documents this exact symptom: 403 ... action [...] is unauthorized, with the cause API key lacks the privilege and the fix recreate the key with the needed role_descriptors. The Security Onion Elasticsearch MCP walkthrough (medium.com/@uhtavi0) notes the same trap during API key creation: read-only does not include permission to list all indices, so privileges must be set to superuser (or the needed roles) for the MCP server's tools to work. Source: https://github.com/cloud-prakhar/elk-setup/blob/HEAD/elk-curl-auth-operations.md