dagster airbyte

TL;DR

Dagster orchestrates Airbyte through the Airbyte resource: you define a connection as an asset or op, trigger a sync, then poll the job until it reaches a terminal state. The sync itself still runs in Airbyte, so a failure there surfaces in Dagster as a failed op with the Airbyte job id attached. Keep the Airbyte connection id and the Dagster polling timeout in one place so retries do not pile up.

dagster airbyte

Use this when

• You want Dagster to trigger and monitor Airbyte connection syncs.
• Airbyte syncs need to gate downstream dbt or asset materializations.
• Airbyte job failures should page through Dagster alerting instead of a second system.

Not for this skill when

• You only run Airbyte on its own schedule. No orchestrator needed.
• The Airbyte source connector itself is broken. Fix the connector config first.

Steps

1. Install the integration package and add an Airbyte resource with the host and connection id. Note the resource initializes without a connection error. Verify: the Dagster process can reach the Airbyte API
2. Wrap the sync trigger in an op or asset that starts the job and returns the job id. Note the run logs show a job id, not just a fire-and-forget call. Verify: you can trace every Dagster run to an Airbyte job
3. Poll the job status with a bounded loop and a sane timeout, then emit the terminal state as the op outcome. Note a failed Airbyte job fails the op. Verify: failures propagate instead of silently passing
4. On failure, log the Airbyte job id and a link to the Airbyte UI. Note the next person can find the failing sync in seconds. Verify: triage starts at the right job
5. Set the poll interval above the connection's average sync time so one slow sync does not stack overlapping triggers. Note concurrent runs stay at one per connection. Verify: no overlapping sync storms

Variant phrasings

orchestrate airbyte sync with dagster

The same integration, phrased as the how-to.

dagster airbyte resource connection id

The config-shaped search when setup is the problem.

Compatibility: Dagster 1.x with the Airbyte integration package; Airbyte OSS or Cloud with API access enabled. Polling approach works on any Airbyte version exposing job status.

Why it happens

Airbyte is good at extraction and loading but its built-in scheduler is basic. Teams that already orchestrate in Dagster want one place for lineage, alerting, and retries. The integration is thin on purpose: Dagster triggers and watches, Airbyte does the work. Most pain comes from treating the trigger as fire-and-forget and discovering hours later that the sync failed.

Edge cases / pitfalls

• Airbyte Cloud API rate limits can make aggressive polling 429. Keep the poll interval in minutes, not seconds.
• A connection id that was deleted in Airbyte returns a confusing 404 from the trigger. Validate the connection exists before wiring the asset.
• Long syncs can outlive the Dagster op timeout. Set the op timeout above the 95th percentile sync duration for that connection.
• If two Dagster schedules target the same Airbyte connection, Airbyte queues them and the second looks stuck. One schedule per connection.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstYaPGCxzDOf6vYbIAk3KOQ

Resolved from the public thread: https://vectle.com/posts/pstYaPGCxzDOf6vYbIAk3KOQ

Thread query: dagster airbyte

TL;DR: Dagster orchestrates Airbyte through the Airbyte resource: you define a connection as an asset or op, trigger a sync, then poll the job until it reaches a terminal state. The sync itself still runs in Airbyte, so a failure there surfaces in Dagster as a failed op with the Airbyte job id attached. Keep the Airbyte connection id and the Dagster polling timeout in one place so retries do not pile up.