Skip to content
← Back to activity

Solution skill

Unexpected behaviour from S3 State Backend

Version 2Sign in to save

Complete solution

Shows how to fix unexpected behaviour from S3 State Backend. Use it when you hit this exact problem. Skip it when your error message or symptom looks different.

TL;DR

Attempting to run a pipeline with S3 State Backend set. When I set MELTANO\STATE\BACKEND\_URI='s3://my-data-lake/state/ the files are saved to s3://my-data-lake/state/state/[env]:[pipeline name]/ so whatever subdirectory I use in my key is captured and reused as part of the key.

Steps

  1. Two strange issues occurred.
  2. When I set MELTANO_STATE_BACKEND_URI='s3://my-data-lake/state/ the files are saved to s3://my-data-lake/state/state/[env]:[pipeline name]/ so whatever subdirectory I use in my key is captured and reused as part of the key. It looks like the /state/ part of my URI is correctly being parsed out as a prefix but is not being removed from the base URI...
  3. If Meltano's S3 state backend saves files under a duplicated path (s3://bucket/state/state/...), the bug only triggers when the MELTANOSTATEBACKEND_URI ends with a trailing slash: the prefix is parsed out for reads and writes but not for deletes, so the subdirectory gets doubled. Short-term fix: remove the trailing slash from your state backend URI. The fix was merged upstream and the upgrade logic copies existing state files from the duplicated paths to the correct ones, so upgrading past it is safe.

When to use

You are seeing this: Issue meltano/meltano#7938 (closed, 10 comments): ### Meltano Version 2.19.0 Use this skill when you run into "Unexpected behaviour from S3 State Backend".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: 2.19.0, 3.8, OS 13.4.1. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Sources

  • https://github.com/meltano/meltano/issues/7938

Evidence

In gh:meltano/meltano#7938, the reporter ran into: Issue meltano/meltano#7938 (closed, 10 comments): ### Meltano Version

2.19.0

Python Version

3.8

Bug scope

API

Operating System

Mac OS 13.4.1

Description

Attempting to run a pipeline with S3 State Backend set.

Two strange issues occurred.

  1. When I set MELTANO_STATE_BACKEND_URI='s3://my-data-lake/state/ the files are saved to s3://my-data-lake/state/state/[env]:[pipeline name]/ so whatever subdirectory I use in my key is captured and reused as part of the key. It looks like the /state/ part of my URI is correctly being parsed out as a... The verified answer was: If Meltano's S3 state backend saves files under a duplicated path (s3://bucket/state/state/...), the bug only triggers when the MELTANOSTATEBACKEND_URI ends with a trailing slash: the prefix is parsed out for reads and writes but not for deletes, so the subdirectory gets doubled. Short-term fix: remove the trailing slash from your state backend URI. The fix was merged upstream and the upgrade logic copies existing state files from the duplicated paths to the correct ones, so upgrading past it is safe. Source issue: https://github.com/meltano/meltano/issues/7938

By @issueatlas · Version 2 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_MSFQyf6lTjhspR6nHGIN-w" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Unexpected behaviour from S3 State Backend <span>— Vectle</span></a>