Solution skill
Unexpected behaviour from S3 State Backend
Complete solution
Shows how to fix unexpected behaviour from S3 State Backend. Use it when you hit this exact problem. Skip it when your error message or symptom looks different.
TL;DR
Attempting to run a pipeline with S3 State Backend set. When I set MELTANO\STATE\BACKEND\_URI='s3://my-data-lake/state/ the files are saved to s3://my-data-lake/state/state/[env]:[pipeline name]/ so whatever subdirectory I use in my key is captured and reused as part of the key.
Steps
- Two strange issues occurred.
- When I set
MELTANO_STATE_BACKEND_URI='s3://my-data-lake/state/the files are saved tos3://my-data-lake/state/state/[env]:[pipeline name]/so whatever subdirectory I use in my key is captured and reused as part of the key. It looks like the/state/part of my URI is correctly being parsed out as a prefix but is not being removed from the base URI... - If Meltano's S3 state backend saves files under a duplicated path (s3://bucket/state/state/...), the bug only triggers when the MELTANOSTATEBACKEND_URI ends with a trailing slash: the prefix is parsed out for reads and writes but not for deletes, so the subdirectory gets doubled. Short-term fix: remove the trailing slash from your state backend URI. The fix was merged upstream and the upgrade logic copies existing state files from the duplicated paths to the correct ones, so upgrading past it is safe.
When to use
You are seeing this: Issue meltano/meltano#7938 (closed, 10 comments): ### Meltano Version 2.19.0 Use this skill when you run into "Unexpected behaviour from S3 State Backend".
When not to use
If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.
Versions
Versions mentioned in the source: 2.19.0, 3.8, OS 13.4.1. If you are on something much newer or older, the details may have shifted.
Why this happens
The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.
Sources
- https://github.com/meltano/meltano/issues/7938
Evidence
In gh:meltano/meltano#7938, the reporter ran into: Issue meltano/meltano#7938 (closed, 10 comments): ### Meltano Version
2.19.0
Python Version
3.8
Bug scope
API
Operating System
Mac OS 13.4.1
Description
Attempting to run a pipeline with S3 State Backend set.
Two strange issues occurred.
- When I set
MELTANO_STATE_BACKEND_URI='s3://my-data-lake/state/the files are saved tos3://my-data-lake/state/state/[env]:[pipeline name]/so whatever subdirectory I use in my key is captured and reused as part of the key. It looks like the/state/part of my URI is correctly being parsed out as a... The verified answer was: If Meltano's S3 state backend saves files under a duplicated path (s3://bucket/state/state/...), the bug only triggers when the MELTANOSTATEBACKEND_URI ends with a trailing slash: the prefix is parsed out for reads and writes but not for deletes, so the subdirectory gets doubled. Short-term fix: remove the trailing slash from your state backend URI. The fix was merged upstream and the upgrade logic copies existing state files from the duplicated paths to the correct ones, so upgrading past it is safe. Source issue: https://github.com/meltano/meltano/issues/7938
By @issueatlas · Version 2 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_MSFQyf6lTjhspR6nHGIN-w" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Unexpected behaviour from S3 State Backend <span>— Vectle</span></a>