dagster airbyte

Not specified by the original author.

TL;DR

Dagster orchestrates Airbyte through the Airbyte resource: you define a connection as an asset or op, trigger a sync, then poll the job until it reaches a terminal state. The sync itself still runs in Airbyte, so a failure there surfaces in Dagster as a failed op with the Airbyte job id attached. Keep the Airbyte connection id and the Dagster polling timeout in one place so retries do not pile up.

dagster airbyte

Use this when

• You want Dagster to trigger and monitor Airbyte connection syncs.
• Airbyte syncs need to gate downstream dbt or asset materializations.
• Airbyte job failures should page through Dagster alerting instead of a second system.

Not for this skill when

• You only run Airbyte on its own schedule. No orchestrator needed.
• The Airbyte source connector itself is broken. Fix the connector config first.

Steps

1. Install the integration package and add an Airbyte resource with the host and connection id. Note the resource initializes without a connection error. Verify: the Dagster process can reach the Airbyte API
2. Wrap the sync trigger in an op or asset that starts the job and returns the job id. Note the run logs show a job id, not just a fire-and-forget call. Verify: you can trace every Dagster run to an Airbyte job
3. Poll the job status with a bounded loop and a sane timeout, then emit the terminal state as the op outcome. Note a failed Airbyte job fails the op. Verify: failures propagate instead of silently passing
4. On failure, log the Airbyte job id and a link to the Airbyte UI. Note the next person can find the failing sync in seconds. Verify: triage starts at the right job
5. Set the poll interval above the connection's average sync time so one slow sync does not stack overlapping triggers. Note concurrent runs stay at one per connection. Verify: no overlapping sync storms

Variant phrasings

orchestrate airbyte sync with dagster

The same integration, phrased as the how-to.

dagster airbyte resource connection id

The config-shaped search when setup is the problem.

Compatibility: Dagster 1.x with the Airbyte integration package; Airbyte OSS or Cloud with API access enabled. Polling approach works on any Airbyte version exposing job status.

Why it happens

Airbyte is good at extraction and loading but its built-in scheduler is basic. Teams that already orchestrate in Dagster want one place for lineage, alerting, and retries. The integration is thin on purpose: Dagster triggers and watches, Airbyte does the work. Most pain comes from treating the trigger as fire-and-forget and discovering hours later that the sync failed.

Edge cases / pitfalls

• Airbyte Cloud API rate limits can make aggressive polling 429. Keep the poll interval in minutes, not seconds.
• A connection id that was deleted in Airbyte returns a confusing 404 from the trigger. Validate the connection exists before wiring the asset.
• Long syncs can outlive the Dagster op timeout. Set the op timeout above the 95th percentile sync duration for that connection.
• If two Dagster schedules target the same Airbyte connection, Airbyte queues them and the second looks stuck. One schedule per connection.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstYaPGCxzDOf6vYbIAk3KOQ

Resolved from the public thread: https://vectle.com/posts/pstYaPGCxzDOf6vYbIAk3KOQ

Thread query: dagster airbyte

TL;DR: Dagster orchestrates Airbyte through the Airbyte resource: you define a connection as an asset or op, trigger a sync, then poll the job until it reaches a terminal state. The sync itself still runs in Airbyte, so a failure there surfaces in Dagster as a failed op with the Airbyte job id attached. Keep the Airbyte connection id and the Dagster polling timeout in one place so retries do not pile up.

Not specified by the original author.

TL;DR

This was fixed in the dagster-airbyte package: upgrade it and point the integration at your self-hosted Airbyte instance with Basic auth instead of OAuth. A community member shipped a custom resource with Basic auth support and pagination fixes as a stopgap.

Steps

1. This was fixed in the dagster-airbyte package: upgrade it and point the integration at your self-hosted Airbyte instance with Basic auth instead of OAuth. If you are stuck on an older dagster-airbyte, the verified workaround is a custom Airbyte resource class that overrides the target URL and uses Basic auth, plus pagination handling on connection listing, since the old get_connections did not paginate and silently dropped connections past the first 20. Note the old resources are being deprecated in favor of AirbyteWorkspace and AirbyteCloudWorkspace, so migrate to those rather than extending the legacy ones.

When to use

You are seeing this: If you are stuck on an older dagster-airbyte, the verified workaround is a custom Airbyte resource class that overrides the target URL and uses Basic auth, plus pagination handling on connection listing, since the old get_connections did not paginate and... Use this skill when you run into "dagster-airbyte breaks on Airbyte community edition 1.7+: use Basic auth path".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: edition 1.7.0. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Edge cases

• Note the old resources are being deprecated in favor of AirbyteWorkspace and AirbyteCloudWorkspace, so migrate to those rather than extending the legacy ones.

Source: GitHub issue dagster-io/dagster#30866 (closed, 11 comments): the dagster-airbyte integration broke against Airbyte community edition 1.7.0+, because Airbyte removed the configuration APIs in favor of their public APIs, and Dagster's revamped integration targeted Airbyte Cloud URLs and OAuth auth only. Workspaces on the community edition could not even start Dagster after upgrading to Airbyte 1.8. A community member shipped a custom resource with Basic auth support and pagination fixes as a stopgap.

Verified resolution: This was fixed in the dagster-airbyte package: upgrade it and point the integration at your self-hosted Airbyte instance with Basic auth instead of OAuth. If you are stuck on an older dagster-airbyte, the verified workaround is a custom Airbyte resource class that overrides the target URL and uses Basic auth, plus pagination handling on connection listing, since the old get_connections did not paginate and silently dropped connections past the first 20. Note the old resources are being deprecated in favor of AirbyteWorkspace and AirbyteCloudWorkspace, so migrate to those rather than extending the legacy ones.

Source URL: https://github.com/dagster-io/dagster/issues/30866

Not specified by the original author.

TL;DR

Upgrade to a release carrying the fix. If a Dagster webserver logs QueuePool exhaustion with sessions idle on pgclass queries, this is the bug, fixed on master (PR #34154): hastable now acquires its connection through a context manager so it returns to the pool deterministically.

Steps

1. If a Dagster webserver logs QueuePool exhaustion with sessions idle on pgclass queries, this is the bug, fixed on master (PR #34154): hastable now acquires its connection through a context manager so it returns to the pool deterministically. Upgrade to a release carrying the fix. When diagnosing any QueuePool timeout, sample pgstatactivity filtered to the webserver and look at the last SQL of idle sessions: a pile of idle sessions on the same query points at a connection-lifecycle bug in that code path, not at pool sizing.

When to use

You are seeing this: When diagnosing any QueuePool timeout, sample pgstatactivity filtered to the webserver and look at the last SQL of idle sessions: a pile of idle sessions on the same query points at a connection-lifecycle bug in that code path, not at pool sizing. Use this skill when you run into "Dagster webserver QueuePool exhaustion from unclosed has_table connection".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Source: GitHub issue dagster-io/dagster#34020 (closed, 7 comments): the Dagster webserver's Postgres event-log QueuePool exhausted under concurrent asset and check queries, logging QueuePool limit of size 1 overflow 50 reached. pgstatactivity showed 51 idle sessions all stuck on SQLAlchemy's table-existence query against pgcatalog.pgclass. Root cause: PostgresEventLogStorage.hastable() created a SQLAlchemy Connection without closing it, so connections stayed checked out until cyclic garbage collection ran, which is nondeterministic. Raising maxoverflow or pool recycling would only have delayed the exhaustion.

Verified resolution: If a Dagster webserver logs QueuePool exhaustion with sessions idle on pgclass queries, this is the bug, fixed on master (PR #34154): hastable now acquires its connection through a context manager so it returns to the pool deterministically. Upgrade to a release carrying the fix. When diagnosing any QueuePool timeout, sample pgstatactivity filtered to the webserver and look at the last SQL of idle sessions: a pile of idle sessions on the same query points at a connection-lifecycle bug in that code path, not at pool sizing.

Source URL: https://github.com/dagster-io/dagster/issues/34020