Skip to content
← Back to activity

Immutable solution version

`airbyte-cron`: RESOURCE_EXHAUSTED namespace rate limit exceeded

Version 1Sign in to save

Complete solution

How to airbyte-cron: RESOURCE_EXHAUSTED namespace rate limit exceeded. Verified in gh:airbytehq/airbyte#30691.

What this teaches

How to airbyte-cron: RESOURCE_EXHAUSTED namespace rate limit exceeded.

The problem

Issue airbytehq/airbyte#30691 (closed, 28 comments): ### Topic Temporal issue ### Revelant information Airbyte version: 0.50.21 We are observing abnormal amount of rate limit errors from airbyte-cron. We are not using airbyte schedulers, only one cron job is setup on the Airbyte UI. The following error message is emitted every few seconds as soon as we start the docker compose. ```2023-09-22 15:27:03 ERROR i.m.s.DefaultTaskExceptionHandler(handle):47 - Error invoking scheduled task for bean [io.airbyte.cron.jobs.SelfHealTemporalWorkflows@31e9f894] RESOURCEEXHAUSTED: namespace rate limit exceeded io.grpc.StatusRuntimeException: RESOURCEEXHAUSTE

What worked (verified answer)

Hit with RESOURCEEXHAUSTED namespace rate limit errors from airbyte-cron flooding every few seconds right after startup. The fix that worked across multiple environments: 1. Helm uninstall your Airbyte release. 2. Delete the Airbyte namespace. 3. Drop the temporal and temporalvisibility databases in your external Postgres (the stuck Temporal workflow state lives there). 4. Reinstall Airbyte fresh. The bad state is in the Temporal DBs, so reinstalling without dropping them just brings the errors back.

Source

gh:airbytehq/airbyte#30691 - https://github.com/airbytehq/airbyte/issues/30691

Evidence

In gh:airbytehq/airbyte#30691, the reporter ran into: Issue airbytehq/airbyte#30691 (closed, 28 comments): ### Topic Temporal issue ### Revelant information Airbyte version: 0.50.21 We are observing abnormal amount of rate limit errors from airbyte-cron. We are not using airbyte schedulers, only one cron job is setup on the Airbyte UI. The following error message is emitted every few seconds as soon as we start the docker compose. ```2023-09-22 15:27:03 ERROR i.m.s.DefaultTaskExceptionHandler(handle):47 - Error invoking scheduled task for bean [io.airbyte.cron.jobs.SelfHealTemporalWorkflows@31e9f894] RESOURCEEXHAUSTED: namespace rate limit... The verified answer was: Hit with RESOURCEEXHAUSTED namespace rate limit errors from airbyte-cron flooding every few seconds right after startup. The fix that worked across multiple environments: 1. Helm uninstall your Airbyte release. 2. Delete the Airbyte namespace. 3. Drop the temporal and temporal_visibility databases in your external Postgres (the stuck Temporal workflow state lives there). 4. Reinstall Airbyte fresh. The bad state is in the Temporal DBs, so reinstalling without dropping them just brings the errors back. Source issue: https://github.com/airbytehq/airbyte/issues/30691

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl__IRQLxtnQ-i8sdZJlqnExQ" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">`airbyte-cron`: RESOURCE_EXHAUSTED namespace rate limit exceeded <span>— Vectle</span></a>