Unable to update Elasticsearch mapping: elastic: Error 403 on Create Custom Attributes

TL;DR: if temporal operator search-attribute create fails with unable to update Elasticsearch mapping: elastic: Error 403 even as the OpenSearch master user, check for dual visibility mode. Adding search attributes is not supported when both visibilityStore (Postgres) and advancedVisibilityStore (OpenSearch) are set; that mode exists only for migrating visibility stores. Keep only advancedVisibilityStore and remove the visibilityStore and secondaryVisibilityStore keys plus the datastores.visibility section.

unable to update Elasticsearch mapping: elastic: Error 403 (Forbidden): security_exception

Steps

1. Inspect your Temporal config for both visibilityStore and advancedVisibilityStore being set.
2. Remove the visibilityStore and secondaryVisibilityStore keys and the datastores.visibility section, keeping only advancedVisibilityStore pointing at OpenSearch.

3. Restart Temporal and re-run the search-attribute create.

   Expected: the attribute creates. Success check: the custom search attribute is usable in queries.

When to use this

• Self-hosted Temporal 1.20.4 with AWS OpenSearch, 403 on attribute creation despite full cluster access.
• Dual visibility mode is configured.

When NOT to use this

• Single visibility store already; the 403 is a real permission issue then.
• You are mid-migration and need dual mode; finish the migration first, then add attributes.

Compatibility

• Temporal 1.20.4 with OpenSearch visibility.

Variant phrasings

• "temporal unable to update elasticsearch mapping 403"
• "temporal search attribute dual visibility"

Root cause

The Postgres visibility store does not support custom search attributes anyway, so the dual-mode write path rejects the mapping update with a 403 that looks like a permissions error.

Edge cases

• Do not try to work around it with broader OpenSearch permissions; the master user already failed.

Source: https://vectle.com/threads/thr_gqbJvitEqUKtGa9gjfLBtQ

In the thread https://vectle.com/threads/thrgqbJvitEqUKtGa9gjfLBtQ, the question was: Self-hosted Temporal 1.20.4 with AWS OpenSearch 2.5 and RDS Postgres. Creating a custom search attribute (temporal operator search-attribute create --name CustomTest --type Keyword) fails with: unable to update Elasticsearch mapping: elastic: Error 403 (Forbidden): securityexception, from the AddESMappingFieldActivity, even when using the OpenSearch master user with full cluster access. The verified resolution was: The cluster was in dual visibility mode: both visibilityStore (pointing at Postgres) and advancedVisibilityStore (pointing at OpenSearch) were set. Adding search attributes is not supported in dual visibility mode, which exists only to support migrating visibility stores. The Temporal maintainer's guidance: keep only advancedVisibilityStore and remove the visibilityStore and secondaryVisibilityStore keys plus the datastores.visibility section, since the Postgres visibility store does not support custom search attributes anyway. Diagnosis confirmed by a Temporal maintainer in the thread.