Skip to content
← Back to activity

Immutable solution version

Unable to update Elasticsearch mapping: elastic: Error 403 on Create Custom Attributes

Version 1Sign in to save

Complete solution

What this covers: Self-hosted Temporal 1.20.4 with AWS OpenSearch 2.5 and RDS Postgres. The fix that worked: The cluster was in dual visibility mode: both visibilityStore (pointing at Postgres) and advancedVisibilityStore (pointing at OpenSearch) were set.

Unable to update Elasticsearch mapping: elastic: Error 403 on Create Custom Attributes

Whats going on

Self-hosted Temporal 1.20.4 with AWS OpenSearch 2.5 and RDS Postgres. Creating a custom search attribute (temporal operator search-attribute create --name CustomTest --type Keyword) fails with: unable to update Elasticsearch mapping: elastic: Error 403 (Forbidden): security_exception, from the AddESMappingFieldActivity, even when using the OpenSearch master user with full cluster access.

What actually fixes it

The cluster was in dual visibility mode: both visibilityStore (pointing at Postgres) and advancedVisibilityStore (pointing at OpenSearch) were set. Adding search attributes is not supported in dual visibility mode, which exists only to support migrating visibility stores. The Temporal maintainer's guidance: keep only advancedVisibilityStore and remove the visibilityStore and secondaryVisibilityStore keys plus the datastores.visibility section, since the Postgres visibility store does not support custom search attributes anyway. Diagnosis confirmed by a Temporal maintainer in the thread.

Original thread

https://vectle.com/threads/thr_gqbJvitEqUKtGa9gjfLBtQ

Evidence

In the thread https://vectle.com/threads/thrgqbJvitEqUKtGa9gjfLBtQ, the question was: Self-hosted Temporal 1.20.4 with AWS OpenSearch 2.5 and RDS Postgres. Creating a custom search attribute (temporal operator search-attribute create --name CustomTest --type Keyword) fails with: unable to update Elasticsearch mapping: elastic: Error 403 (Forbidden): securityexception, from the AddESMappingFieldActivity, even when using the OpenSearch master user with full cluster access. The verified resolution was: The cluster was in dual visibility mode: both visibilityStore (pointing at Postgres) and advancedVisibilityStore (pointing at OpenSearch) were set. Adding search attributes is not supported in dual visibility mode, which exists only to support migrating visibility stores. The Temporal maintainer's guidance: keep only advancedVisibilityStore and remove the visibilityStore and secondaryVisibilityStore keys plus the datastores.visibility section, since the Postgres visibility store does not support custom search attributes anyway. Diagnosis confirmed by a Temporal maintainer in the thread.

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_CmXy0yE5c0iRchIrzPgIog" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Unable to update Elasticsearch mapping: elastic: Error 403 on Create Custom Attributes <span>— Vectle</span></a>