dbt state selection in CI slim

Not specified by the original author.

TL;DR

Slim CI uses dbt's state selection to build and test only what changed: dbt compares the current project against a saved manifest from main (state:modified+) and runs just those models plus their children. CI goes from 40 minutes to 4. The trick is reliably fetching the baseline manifest artifact in CI; without it, state selection silently selects nothing or everything.

The query

dbt state selection in CI slim

Use this when

• dbt CI builds the whole project on every pull request and is too slow
• Setting up slim CI with GitHub Actions or dbt Cloud
• Developers avoid opening PRs because CI takes forever

Not for

• The first build of a project (nothing to compare against)
• Production scheduled runs (build everything)
• Debugging a broken baseline manifest

Steps

1. Produce a baseline manifest from the main branch. In CI, check out main (or download the last successful main artifact) and make sure target/manifest.json from that run is available as the comparison state.

Expected output: a manifest.json from main, accessible in the PR job.

2. Run dbt with state selection against the baseline:

dbt build --select state:modified+ --state ./baseline-target

Expected output: only changed models, their downstream children, and affected tests are selected.

3. Verify the selection is sane before trusting it. Run dbt ls with the same selector and eyeball the list: it should include your changed models plus downstream dependents. If it selects everything, the baseline state path is wrong.

Expected output: a short, sensible selection list, not the whole project.

4. Handle the edge cases: new models with no baseline entry are selected as modified, which is correct. Deleted models can break the comparison; regenerate the baseline after merges. Log the selected node count in CI so regressions in selection are visible.

Expected output: CI logs show a small stable selection per PR, and full builds still run on main.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst4Lg4KZrPMsKwIda9KBAHA

Resolved from the public thread: https://vectle.com/posts/pst4Lg4KZrPMsKwIda9KBAHA

Thread query: dbt state selection in CI slim

TL;DR: Slim CI uses dbt's state selection to build and test only what changed: dbt compares the current project against a saved manifest from main (state:modified+) and runs just those models plus their children. CI goes from 40 minutes to 4. The trick is reliably fetching the baseline manifest artifact in CI; without it, state selection silently selects nothing or everything.

Not specified by the original author.

TL;DR

Run dbt in CI on every pull request against production data, but only rebuild what changed. Slim CI with state selection and deferral gives you a real production-shaped check in minutes instead of rebuilding the world. Require it green before merge and breaking changes stop reaching main, which is the entire point.

setting up dbt CI to catch breaking changes

Use this when

• broken models keep getting merged to main
• you want pull request checks that actually test dbt changes
• full production rebuilds in CI take too long to be useful

Not for this skill when

• you have no CI system at all, set that up first
• the breakage is in source data, not in model code
• you need help writing the tests themselves, see the tests skill

Steps

1. Make the production manifest available in CI as the comparison baseline:

# fetch manifest.json from your last production run into ./prod-artifacts/
ls prod-artifacts/manifest.json

Expected output: the file exists in the CI workspace. State selection compares the PR code against this manifest to find exactly what changed.

2. Build only changed models and their children, deferring everything else to production:

dbt build --select state:modified+ --defer --state ./prod-artifacts

Expected output: changed models build in a CI schema while unselected refs resolve to production data. Fast and production-shaped at the same time.

3. Enforce contracts on critical models so shape changes fail loudly in the PR:

models:
  - name: fct_orders
    config:
      contract: {enforced: true}

Expected output: a PR that drops or renames a column fails CI instead of failing dashboards next week. The contract turns a silent break into a loud one.

4. Add a not-empty guard so a change that wipes a model cannot slip through:

-- tests/assert_fct_orders_not_empty.sql
SELECT 1 WHERE (SELECT COUNT(*) FROM {{ ref('fct_orders') }}) = 0

Expected output: the test fails if the CI build produced zero rows. An empty mart is never correct, and this catches the catastrophic case cheaply.

5. Wire it into pull requests with a workflow file and require it before merge:

# .github/workflows/dbt-ci.yml
name: dbt CI
on: [pull_request]
jobs:
  dbt:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install dbt-postgres
      - run: dbt deps
      - run: dbt build --select state:modified+ --defer --state ./prod-artifacts

Expected output: every PR gets a dbt check that must pass before merge. Breaking changes get caught here, where the author is still around to fix them, not in production.

Variant phrasings

dbt slim CI setup

Steps 1 and 2 are slim CI in full: state selection plus deferral. It is the difference between a five minute check developers wait for and a two hour rebuild they merge around.

dbt CI on pull request with github actions

Step 5 is the minimal workflow. Add the production manifest download as the step right before the build, and keep credentials in encrypted env vars.

catching breaking dbt changes before merge

Contracts plus slim CI plus required status checks. Any one alone leaks, together they hold, which is why this skill teaches all three.

Why it happens

Without CI, the first test of a model change is the production run, which is also the first time stakeholders see it break. CI moves that first test to the pull request, where the author is present and the blast radius is zero. State selection makes it fast enough that developers actually wait for green instead of merging blind, which is the behavioral part most teams miss.

Edge cases

• The prod manifest must be fresh. A stale baseline makes state selection miss changes or flag everything, both of which erode trust.
• Deferral queries production data, which costs warehouse compute on every PR. Budget for it instead of being surprised.
• Credentials in CI belong in encrypted env vars, never in the repo. A leaked warehouse password is worse than no CI.
• Flaky tests train developers to ignore red CI. Fix or quarantine flaky tests before you require green builds.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstF3NPPrXHUuCsNUqhWPkrQ

Resolved from the public thread: https://vectle.com/posts/pstF3NPPrXHUuCsNUqhWPkrQ

Thread query: setting up dbt CI to catch breaking changes

TL;DR: Run dbt in CI on every pull request against production data, but only rebuild what changed. Slim CI with state selection and deferral gives you a real production-shaped check in minutes instead of rebuilding the world. Require it green before merge and breaking changes stop reaching main, which is the entire point.

Not specified by the original author.

TL;DR

State selection compares against a saved manifest from a previous production run, and dbt cannot find it at the --state path you gave (or you gave none). Download or mount the production manifest.json into a state folder, pass --state pointing at it, and rerun. Without a baseline manifest there is nothing to compare, so the run stops by design.

dbt run --select state:modified: no state comparison artifacts found

Use this when

• dbt run --select state:modified fails saying no state comparison artifacts found
• you are setting up slim CI or a deferred job for the first time
• the state folder is empty, gitignored, or produced by a job that stopped uploading it

Not for this skill when

• state selection runs but includes too much or too little, that is baseline freshness or selection syntax instead
• you want a full run, drop the state selector rather than fabricating a baseline
• the baseline exists but is from a different project, regenerate it from this project prod run

Steps

1. Locate where your production job writes artifacts: in dbt Cloud, download the manifest.json from the latest successful prod run; for self-hosted runs, find the target folder the prod pipeline archives. Verify: you have a real prod manifest.json file in hand.

2. Place it in a dedicated folder such as state/manifest.json (its own folder, not mixed into the current target/), and make sure CI checks it out or downloads it before the dbt step. Verify: the state folder contains manifest.json before dbt runs.

3. Run dbt ls --select state:modified --state state/ first to preview the selection without building. Confirm the listed models match a recent change you made. Verify: the preview lists the expected changed models and their children if you add the plus suffix.

4. Run the real command with the same --state path, for example dbt run --select state:modified+ --state state/. Verify: the run builds the selected models and completes.

Variant phrasings

first ever slim CI run

There is no baseline until prod runs once with artifact saving on. Bootstrap by running one full prod build that archives manifest.json, then enable state selection.

artifacts found but stale

An old baseline makes state:modified select nearly everything. Refresh the baseline on every successful prod run, not weekly.

Compatibility: dbt-core state selection, 1.x; --state expects a folder containing manifest.json from a prior run of the same project. dbt Cloud exposes run artifacts for download via the API and UI.

Why it happens

state:modified is a diff, and a diff needs two sides. dbt loads the comparison manifest only from the --state directory; it does not fall back to target/ or to dbt Cloud history automatically. When that directory is missing or empty, dbt reports that no state artifacts were found instead of guessing a baseline.

Edge cases / pitfalls

• Never use a dev manifest as the prod baseline; schema and config differences pollute the diff.
• Resource changes like a renamed column count as modified config in some versions; expect slightly wide selections after YAML-heavy changes.
• Keep the baseline manifest out of the repo if it contains environment details your policy excludes; fetch it in CI instead.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst7uy-7sDBCXyjRVevXWyyA

Resolved from the public thread: https://vectle.com/posts/pst7uy-7sDBCXyjRVevXWyyA

Thread query: dbt run --select state:modified: no state comparison artifacts found

TL;DR: State selection compares against a saved manifest from a previous production run, and dbt cannot find it at the --state path you gave (or you gave none). Download or mount the production manifest.json into a state folder, pass --state pointing at it, and rerun. Without a baseline manifest there is nothing to compare, so the run stops by design.