dagster-airbyte breaks on Airbyte community edition 1.7+: use Basic auth path

TL;DR

This was fixed in the dagster-airbyte package: upgrade it and point the integration at your self-hosted Airbyte instance with Basic auth instead of OAuth. A community member shipped a custom resource with Basic auth support and pagination fixes as a stopgap.

Steps

1. This was fixed in the dagster-airbyte package: upgrade it and point the integration at your self-hosted Airbyte instance with Basic auth instead of OAuth. If you are stuck on an older dagster-airbyte, the verified workaround is a custom Airbyte resource class that overrides the target URL and uses Basic auth, plus pagination handling on connection listing, since the old get_connections did not paginate and silently dropped connections past the first 20. Note the old resources are being deprecated in favor of AirbyteWorkspace and AirbyteCloudWorkspace, so migrate to those rather than extending the legacy ones.

When to use

You are seeing this: If you are stuck on an older dagster-airbyte, the verified workaround is a custom Airbyte resource class that overrides the target URL and uses Basic auth, plus pagination handling on connection listing, since the old get_connections did not paginate and... Use this skill when you run into "dagster-airbyte breaks on Airbyte community edition 1.7+: use Basic auth path".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: edition 1.7.0. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Edge cases

• Note the old resources are being deprecated in favor of AirbyteWorkspace and AirbyteCloudWorkspace, so migrate to those rather than extending the legacy ones.

Source: GitHub issue dagster-io/dagster#30866 (closed, 11 comments): the dagster-airbyte integration broke against Airbyte community edition 1.7.0+, because Airbyte removed the configuration APIs in favor of their public APIs, and Dagster's revamped integration targeted Airbyte Cloud URLs and OAuth auth only. Workspaces on the community edition could not even start Dagster after upgrading to Airbyte 1.8. A community member shipped a custom resource with Basic auth support and pagination fixes as a stopgap.

Verified resolution: This was fixed in the dagster-airbyte package: upgrade it and point the integration at your self-hosted Airbyte instance with Basic auth instead of OAuth. If you are stuck on an older dagster-airbyte, the verified workaround is a custom Airbyte resource class that overrides the target URL and uses Basic auth, plus pagination handling on connection listing, since the old get_connections did not paginate and silently dropped connections past the first 20. Note the old resources are being deprecated in favor of AirbyteWorkspace and AirbyteCloudWorkspace, so migrate to those rather than extending the legacy ones.

Source URL: https://github.com/dagster-io/dagster/issues/30866