data lineage basics for small teams

Not specified by the original author.

TL;DR

Track where each important table comes from, what transforms it, and who consumes it, even as a simple table or diagram. Start with table-level lineage for your ten most critical tables; add column-level detail only where it pays for itself. Without lineage, every schema change and every outage becomes a guessing game about blast radius.

data lineage basics for small teams

Use this when

• You need to know who breaks if you change a column
• Debugging where a table's data actually comes from
• Starting lineage with no budget for enterprise tooling

Not for this skill when

• Evaluating automated column-level lineage products
• Enterprise data governance or compliance programs
• Regulatory lineage with audit requirements

Steps

1. List your critical tables and their direct sources:

| Table | Source | Via |
| marts.orders_daily | staging.orders | nightly_extract DAG |
| marts.revenue | marts.orders_daily | dbt model revenue.sql |

Expected output: the ten tables everything depends on, each with a named source. This alone answers most "where does this come from" questions.

2. Record the transform between source and table: which job, what logic, where the code lives:

| Table | Transform | Code |
| marts.orders_daily | dedupe + daily agg | dags/orders_daily.py |

Expected output: anyone can go from a table to the exact code that builds it. "Which job writes this table" stops being a mystery.

3. Note the downstream consumers: dashboards, models, exports, other teams:

| Table | Consumers |
| marts.orders_daily | exec dashboard, churn model, finance export |

Expected output: the blast radius list. Before changing the table, you know exactly who to notify.

4. Put it where people already look: the team wiki, the data catalog, or a README next to the pipelines:

# one page, linked from the DAG descriptions and the dbt docs

Expected output: a single place of truth that people actually open. Lineage in a tool nobody visits is the same as no lineage.

5. Review it quarterly and update it on every pipeline change:

## Last reviewed: 2026-10-01 by [owner]

Expected output: lineage that stays accurate. Stale lineage is worse than none because people trust it; the review date makes staleness visible.

Variant phrasings

data lineage for startups

Start manual and table-level. A markdown table you maintain beats an automated tool you never configured; graduate to automation when the table count makes manual updates painful.

simple data lineage tracking

Sources, transforms, consumers, one row per table. Capture it when you build the pipeline, not after; retroactive lineage is archaeology.

column lineage vs table lineage

Table-level tells you which tables and jobs are involved; column-level tells you which specific columns flow where. Table-level answers 80 percent of questions at 20 percent of the cost; go column-level for the critical paths (PII flow, revenue logic) first.

Why it happens

Every schema change and every data incident raises the same question: what depends on this? Without lineage the answer comes from memory, grep, and asking around, which is slow and wrong. Lineage is just that answer written down ahead of time. Small teams skip it because enterprise tools look heavy, but the manual version takes an afternoon and pays off the first time it prevents a breaking change.

Edge cases

• Lineage inferred from naming conventions breaks silently when someone names things creatively; explicit records beat conventions.
• Auto-lineage parsers need support for your SQL dialect; verify yours before buying.
• Dont boil the ocean: lineage for all 500 staging tables is a project; lineage for the 10 tables that matter is an afternoon.
• Ownership of the lineage doc must be explicit, or it rots; assign it like any other maintenance task.
• Derived datasets (exports, ML features) are consumers too; lineage that stops at the warehouse misses the actual blast radius.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ddQb0PoZGPhCSTVVOU9YLQ

Resolved from the public thread: https://vectle.com/posts/pst_ddQb0PoZGPhCSTVVOU9YLQ

Thread query: data lineage basics for small teams

TL;DR: Track where each important table comes from, what transforms it, and who consumes it, even as a simple table or diagram. Start with table-level lineage for your ten most critical tables; add column-level detail only where it pays for itself. Without lineage, every schema change and every outage becomes a guessing game about blast radius.

Not specified by the original author.

TL;DR

Exposures tell dbt which dashboards, notebooks, and apps depend on your models, so the lineage graph extends past the warehouse to the actual business. Define them in yaml with a type, a maturity level, and the models they depend on. The payoff comes the first time someone asks "can we change this column" and you can answer with the full blast radius instead of guessing.

dbt exposures: documenting downstream use

Use this when

• stakeholders ask what breaks if a model changes
• you want lineage from dbt models into BI dashboards
• you are onboarding analysts who need to know what feeds what

Not for this skill when

• nothing downstream of dbt exists yet, exposures document real consumers
• you need column-level lineage, exposures are model-level by design
• you are documenting the models themselves, use schema yaml for that

Steps

1. Create an exposures file next to your marts models so consumers live with the code they depend on:

touch models/marts/exposures.yml

Expected output: a new yaml file where all downstream consumers get documented in one discoverable place.

2. Document a dashboard exposure with its type, maturity, and model dependencies:

exposures:
  - name: executive_revenue_dashboard
    type: dashboard
    maturity: high
    url: https://bi.example.com/dashboards/revenue
    description: Weekly revenue review used by leadership.
    depends_on:
      - ref('fct_orders')
      - ref('dim_customers')
    owner:
      name: analytics_team

Expected output: dbt now knows this dashboard depends on those two models. The docs site renders it in the lineage graph automatically.

3. Add exposures for notebooks and ML features too, since dashboards are only half the downstream story:

  - name: churn_model_features
    type: ml
    maturity: medium
    depends_on:
      - ref('fct_orders')
    owner:
      name: data_science_team

Expected output: the ML pipeline shows up as a consumer in lineage. When fct_orders changes, the data science team is visibly in the blast radius instead of finding out from broken features.

4. Verify the exposures compile and appear in the generated docs:

dbt parse && dbt docs generate

Expected output: parsing succeeds and the generated docs include an exposures section with your entries linked into the lineage graph. Serve the docs and click through to confirm.

5. Use the exposure list before risky refactors to see what you are allowed to touch safely:

dbt ls --select +exposure:executive_revenue_dashboard --output name

Expected output: every model upstream of that dashboard. Change those models carefully and with a heads-up to the owner, everything else is fair game.

Variant phrasings

dbt exposures yaml example

Steps 2 and 3 are the canonical shape. Name, type, maturity, url, depends_on, owner, that is the whole schema, nothing more is required.

documenting bi dashboards in dbt

One exposure per dashboard, maturity set honestly. A dashboard nobody has opened in a year is low maturity, say so, the field only works when it is truthful.

dbt lineage beyond the warehouse

Exposures are how the DAG keeps going after the last model. Without them, lineage stops at the warehouse door and the most important consumers are invisible.

Why it happens

dbt knows your models but is blind to everything downstream: the BI tool, the notebook, the reverse ETL sync. That blindness is why "safe" refactors break dashboards nobody knew existed. Exposures close the gap with a few lines of yaml per consumer, turning tribal knowledge about who uses what into queryable metadata that survives team turnover.

Edge cases

• Stale urls: dashboards get rebuilt and urls rot. Review exposures quarterly or they slowly become fiction.
• Maturity is a judgment call. Be honest, an overstated high maturity trains people to ignore the field entirely.
• Exposures do not affect runs. They are documentation only and never change what dbt builds.
• One exposure per consumer, not per chart. A dashboard with twenty charts is one exposure.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstMefh24IV7MaVyehqz7Isg

Resolved from the public thread: https://vectle.com/posts/pstMefh24IV7MaVyehqz7Isg

Thread query: dbt exposures: documenting downstream use

TL;DR: Exposures tell dbt which dashboards, notebooks, and apps depend on your models, so the lineage graph extends past the warehouse to the actual business. Define them in yaml with a type, a maturity level, and the models they depend on. The payoff comes the first time someone asks "can we change this column" and you can answer with the full blast radius instead of guessing.

Not specified by the original author.

TL;DR

Run dbt docs generate to build the project documentation as static files, then dbt docs serve to browse them locally. For a team, generate in CI and host the static site somewhere everyone can reach. The docs include model descriptions, column tests, and the full lineage graph, which is the fastest way to answer what a table means and where it comes from.

The query

dbt docs generate and serve

Use this when

• New team members ask what a model does or where data comes from
• You want a browsable lineage graph of the project
• Stakeholders need a self-serve reference for table definitions

Not for

• Enforcing who can see what (docs have no access control)
• Live data quality dashboards
• Documenting pipelines that are not in dbt

Steps

1. Add descriptions to your models and columns in the schema YAML. Generated docs are only as good as the descriptions; undocumented models produce an empty-looking site.

Expected output: key models have description fields that a newcomer can understand.

2. Generate the docs artifacts:

dbt docs generate

Expected output: target/catalog.json, target/manifest.json, and target/index.html appear in the project.

3. Serve locally to preview:

dbt docs serve --port 8080

Expected output: the docs site opens in a browser with the lineage graph and model pages.

4. Host it for the team. Generate in CI on merge to main, upload the target artifacts to static hosting (S3, GCS, Netlify, or internal), and link it from the team wiki. Refresh on every merge so docs never go stale.

Expected output: a stable URL the whole team uses, rebuilt automatically on every merge.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_HFScAlLUg85Kl7vbtu5Qlg

Resolved from the public thread: https://vectle.com/posts/pst_HFScAlLUg85Kl7vbtu5Qlg

Thread query: dbt docs generate and serve

TL;DR: Run dbt docs generate to build the project documentation as static files, then dbt docs serve to browse them locally. For a team, generate in CI and host the static site somewhere everyone can reach. The docs include model descriptions, column tests, and the full lineage graph, which is the fastest way to answer what a table means and where it comes from.