Solution skill
meltano run with any tap and `target-snowflake` fails at the end of the run, even though it works when piped with
Complete solution
Shows how to fix meltano run with any tap and `target-snowflake` fails at the end of the run, even though it works when piped with. Use it when you hit this exact problem. Skip it when your error message or symptom looks different.
TL;DR
I have added target-snowflake (transferwise variant) and configured it correctly.
Steps
- I have added
target-snowflake(transferwise variant) and configured it correctly.
It works when invoked alone using meltano invoke target-snowflake.
- Invoking a tap and piping into the target also works:
meltano invoke tap-slack | meltano invoke target-snowflake - However, using
meltano runresults in a crash:meltano run tap-slack target-snowflake - There is no useful information given...
- If meltano run crashes at the end with the transferwise variant of target-snowflake while meltano invoke works fine, the culprit is the old pyarrow pin pulled in transitively via snowflake-connector-python; the reporter confirmed that was the problem. The maintainer's recommendation is to switch to the meltanolabs variant of target-snowflake, which is the default and better maintained.
When to use
You are seeing this: Issue meltano/meltano#7736 (closed, 18 comments): ### Meltano Version 2.19.0 Use this skill when you run into "meltano run with any tap and target-snowflake fails at the end of the run, even though it works when piped with".
When not to use
If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.
Versions
Versions mentioned in the source: 2.19.0, 3.10. If you are on something much newer or older, the details may have shifted.
Why this happens
The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.
Evidence
In gh:meltano/meltano#7736 (Meltano), the issue reported was: Issue meltano/meltano#7736 (closed, 18 comments): ### Meltano Version
2.19.0
Python Version
3.10
Bug scope
CLI (options, error messages, logging, etc.)
Operating System
MacOS 13.4 (22F66) Ventura
Description
I have added target-snowflake (transferwise variant) and configured it correctly. It works when invoked alone using meltano invoke target-snowflake.
Invoking a tap and piping into the target also works: meltano invoke tap-slack | meltano invoke target-snowflake
However, using meltano run results in a crash: meltano run tap-slack target-snowflake
[image]
There is no useful information given... The verified resolution from the thread: If meltano run crashes at the end with the transferwise variant of target-snowflake while meltano invoke works fine, the culprit is the old pyarrow pin pulled in transitively via snowflake-connector-python; the reporter confirmed that was the problem. The maintainer's recommendation is to switch to the meltanolabs variant of target-snowflake, which is the default and better maintained. Source: mount the host path https at container path //github.com/meltano/meltano/issues/7736
By @issueatlas · Version 2 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_6vmGJnQ3-GT9pDTwEF-xiA" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">meltano run with any tap and `target-snowflake` fails at the end of the run, even though it works when piped with <span>— Vectle</span></a>