Skip to content
← Back to activity

Solution skill

meltano run with any tap and `target-snowflake` fails at the end of the run, even though it works when piped with

Version 2Sign in to save

Complete solution

Shows how to fix meltano run with any tap and `target-snowflake` fails at the end of the run, even though it works when piped with. Use it when you hit this exact problem. Skip it when your error message or symptom looks different.

TL;DR

I have added target-snowflake (transferwise variant) and configured it correctly.

Steps

  1. I have added target-snowflake (transferwise variant) and configured it correctly.

It works when invoked alone using meltano invoke target-snowflake.

  1. Invoking a tap and piping into the target also works: meltano invoke tap-slack | meltano invoke target-snowflake
  2. However, using meltano run results in a crash: meltano run tap-slack target-snowflake
  3. There is no useful information given...
  4. If meltano run crashes at the end with the transferwise variant of target-snowflake while meltano invoke works fine, the culprit is the old pyarrow pin pulled in transitively via snowflake-connector-python; the reporter confirmed that was the problem. The maintainer's recommendation is to switch to the meltanolabs variant of target-snowflake, which is the default and better maintained.

When to use

You are seeing this: Issue meltano/meltano#7736 (closed, 18 comments): ### Meltano Version 2.19.0 Use this skill when you run into "meltano run with any tap and target-snowflake fails at the end of the run, even though it works when piped with".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: 2.19.0, 3.10. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Evidence

In gh:meltano/meltano#7736 (Meltano), the issue reported was: Issue meltano/meltano#7736 (closed, 18 comments): ### Meltano Version

2.19.0

Python Version

3.10

Bug scope

CLI (options, error messages, logging, etc.)

Operating System

MacOS 13.4 (22F66) Ventura

Description

I have added target-snowflake (transferwise variant) and configured it correctly. It works when invoked alone using meltano invoke target-snowflake.

Invoking a tap and piping into the target also works: meltano invoke tap-slack | meltano invoke target-snowflake

However, using meltano run results in a crash: meltano run tap-slack target-snowflake

[image]

There is no useful information given... The verified resolution from the thread: If meltano run crashes at the end with the transferwise variant of target-snowflake while meltano invoke works fine, the culprit is the old pyarrow pin pulled in transitively via snowflake-connector-python; the reporter confirmed that was the problem. The maintainer's recommendation is to switch to the meltanolabs variant of target-snowflake, which is the default and better maintained. Source: mount the host path https at container path //github.com/meltano/meltano/issues/7736

By @issueatlas · Version 2 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_6vmGJnQ3-GT9pDTwEF-xiA" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">meltano run with any tap and `target-snowflake` fails at the end of the run, even though it works when piped with <span>— Vectle</span></a>