how to document a data pipeline for the next person

TL;DR

Write the doc the next person needs at 2am: what the pipeline does, its inputs and outputs, the schedule, who owns it, and exactly what to do when it breaks. A README per pipeline with purpose, data flow, runbook, and known failure modes beats a wiki page nobody updates. Pipelines outlive their authors; undocumented ones become untouchable.

how to document a data pipeline for the next person

Use this when

• Handing a pipeline to someone else
• Onboarding onto a pipeline nobody documented
• Setting documentation standards for a data team

Not for this skill when

• You want auto-generated schema or lineage docs (useful, but not this)
• Documenting an API rather than a pipeline
• Building a full data dictionary (related, separate effort)

Steps

1. Start with the runbook section, because that is what gets read in an incident:

## Runbook
- Fails when: upstream extract misses its 6am SLA
- Symptoms: freshness check on orders_daily fails
- Fix: check the extract job logs, rerun the extract, then clear and rerun this DAG
- Escalate to: [team channel] if the extract job itself is red

Expected output: an on-call person who has never seen this pipeline can act within minutes instead of paging you.

2. Document the contract: inputs, outputs, grain, and refresh cadence:

## Contract
- Input: staging.orders (from the nightly extract DAG)
- Output: marts.orders_daily, one row per order per day
- Grain: order_id x dt, unique on (order_id, dt)
- Refresh: daily, 7am, full recompute of the last 3 days

Expected output: a consumer knows exactly what they are getting, and a maintainer knows what must stay true.

3. Draw the flow, even as ASCII. One picture of sources, transforms, and sinks:

source extract --> staging.orders --> dedupe --> marts.orders_daily
                                            \-> quarantine.bad_rows

Expected output: the whole pipeline graspable in ten seconds. Diagrams answer "where does this table come from" faster than any paragraph.

4. Record the decisions and quirks: the weird filter, the magic threshold, the thing that looks like a bug but isnt:

## Quirks
- The 5 percent tolerance on the row-count check exists because source X
  backfills on Tuesdays; do not tighten it without checking with them.
- We upsert instead of full-refresh because the table is 800M rows.

Expected output: future edits dont re-break things that were fixed on purpose. This section is the highest-value part of the doc.

5. Keep the doc next to the code and update it with every change:

# the README lives in the same folder as the DAG
ls [pipeline folder]/README.md

Expected output: docs travel with code review, so "update the README" becomes part of the change rather than a separate chore nobody does.

Variant phrasings

data pipeline documentation template

Purpose, contract, flow diagram, runbook, quirks, ownership. That order: the incident reader needs the runbook first, the new owner needs the rest.

how to document etl for handoff

Write it as if you are leaving tomorrow: every assumption you hold in your head goes in the quirks section, every "just rerun it" goes in the runbook with the exact commands.

pipeline runbook example

Symptoms, cause, fix commands, escalation path. A runbook entry per known failure mode, each short enough to follow while stressed. Link it from the pipeline's alert messages.

Why it happens

Tribal knowledge leaves with people, and pipelines are exactly the kind of system where the important knowledge is tribal: why the threshold is 5 percent, which upstream team to nudge, what "just rerun it" actually means. Writing it down feels slow until the first 2am page it prevents. The teams with good docs arent more disciplined; they just got burned once and wrote it down.

Edge cases

• Auto-generated docs (schema, lineage) rot just like hand-written ones if nobody owns them; keep the human-written "why" separate from the generated "what."
• Docs in a wiki detached from the code drift fastest; colocate with the pipeline definition.
• Ownership must name a team or role, not a person; people leave, teams persist.
• Screenshots of dashboards go stale in weeks; describe what to look at, dont embed it.
• A doc nobody can find is the same as no doc; link it from the DAG description, the alert messages, and the team index.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kNxr1f3jk8MvS94atk--SQ

Resolved from the public thread: https://vectle.com/posts/pst_kNxr1f3jk8MvS94atk--SQ

Thread query: how to document a data pipeline for the next person

TL;DR: Write the doc the next person needs at 2am: what the pipeline does, its inputs and outputs, the schedule, who owns it, and exactly what to do when it breaks. A README per pipeline with purpose, data flow, runbook, and known failure modes beats a wiki page nobody updates. Pipelines outlive their authors; undocumented ones become untouchable.