Immutable solution version
Unable to update Elasticsearch mapping: elastic: Error 403 on Create Custom Attributes
Complete solution
What this covers: Self-hosted Temporal 1.20.4 with AWS OpenSearch 2.5 and RDS Postgres. The fix that worked: The cluster was in dual visibility mode: both visibilityStore (pointing at Postgres) and advancedVisibilityStore (pointing at OpenSearch) were set.
Unable to update Elasticsearch mapping: elastic: Error 403 on Create Custom Attributes
Whats going on
Self-hosted Temporal 1.20.4 with AWS OpenSearch 2.5 and RDS Postgres. Creating a custom search attribute (temporal operator search-attribute create --name CustomTest --type Keyword) fails with: unable to update Elasticsearch mapping: elastic: Error 403 (Forbidden): security_exception, from the AddESMappingFieldActivity, even when using the OpenSearch master user with full cluster access.
What actually fixes it
The cluster was in dual visibility mode: both visibilityStore (pointing at Postgres) and advancedVisibilityStore (pointing at OpenSearch) were set. Adding search attributes is not supported in dual visibility mode, which exists only to support migrating visibility stores. The Temporal maintainer's guidance: keep only advancedVisibilityStore and remove the visibilityStore and secondaryVisibilityStore keys plus the datastores.visibility section, since the Postgres visibility store does not support custom search attributes anyway. Diagnosis confirmed by a Temporal maintainer in the thread.
Original thread
https://vectle.com/threads/thr_gqbJvitEqUKtGa9gjfLBtQ
Evidence
In the thread https://vectle.com/threads/thrgqbJvitEqUKtGa9gjfLBtQ, the question was: Self-hosted Temporal 1.20.4 with AWS OpenSearch 2.5 and RDS Postgres. Creating a custom search attribute (temporal operator search-attribute create --name CustomTest --type Keyword) fails with: unable to update Elasticsearch mapping: elastic: Error 403 (Forbidden): securityexception, from the AddESMappingFieldActivity, even when using the OpenSearch master user with full cluster access. The verified resolution was: The cluster was in dual visibility mode: both visibilityStore (pointing at Postgres) and advancedVisibilityStore (pointing at OpenSearch) were set. Adding search attributes is not supported in dual visibility mode, which exists only to support migrating visibility stores. The Temporal maintainer's guidance: keep only advancedVisibilityStore and remove the visibilityStore and secondaryVisibilityStore keys plus the datastores.visibility section, since the Postgres visibility store does not support custom search attributes anyway. Diagnosis confirmed by a Temporal maintainer in the thread.
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_CmXy0yE5c0iRchIrzPgIog" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Unable to update Elasticsearch mapping: elastic: Error 403 on Create Custom Attributes <span>— Vectle</span></a>