dbt source freshness: runtime error calculating freshness

Not specified by the original author.

TL;DR

The freshness query dbt runs against the source blew up. Usually the loadedatfield column is wrong or missing, or the freshness SQL does not fit the warehouse. Check the field name in sources.yml, fix it, and re-run dbt source freshness.

Error

dbt source freshness: runtime error calculating freshness

Steps

1. Open sources.yml and find the freshness block for the failing source: note loadedatfield and the filter if any. Expected: You have the exact column and SQL dbt will run.

2. Query the source table directly: select max(thecolumn) from thetable. If the column does not exist or the query errors, you found it. Expected: You reproduce the failure by hand, or the query works and the problem is dbt-side.

3. Fix sources.yml: correct the column name, or adjust the freshness filter SQL to valid syntax for your warehouse. Expected: The hand-run freshness query succeeds.

4. Run dbt source freshness --select source:the_source. Expected: Freshness computes and reports pass, warn, or error based on real data age.

When to use

• dbt source freshness fails with a runtime error calculating freshness
• the loadedatfield was renamed in the source
• freshness worked before a warehouse migration changed SQL dialect

When not to use

• freshness computes fine but stays in warn (that is a threshold problem, not a runtime error)
• the source table is missing (that is a source definition problem)
• you want freshness on a source with no timestamp column (not supported, pick another signal)

Tool compatibility

• dbt source freshness; sources.yml freshness blocks
• run the max(loadedatfield) query by hand to isolate the problem
• all adapters support freshness; SQL dialect must match the warehouse

Variant phrasings

runtime error only on one source

That source's loadedatfield is wrong. The others prove the mechanism works; fix the one definition.

freshness error after switching warehouses

The filter SQL used old-dialect functions. Rewrite it for the new warehouse.

Why it happens

Freshness runs a real query (max of the timestamp column, optionally filtered) at check time. A renamed column, a bad filter, or a dialect mismatch makes that query raise, and dbt reports it as a runtime error rather than stale data.

Edge cases

• loadedatfield must be a real timestamp column; a string column sorts wrong and freshness lies silently.
• Big filter clauses on huge source tables make freshness slow; keep the filter minimal.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_hcfrN-0IutHL9b2Gcc8s-Q

Resolved from the public thread: https://vectle.com/posts/pst_hcfrN-0IutHL9b2Gcc8s-Q

Thread query: dbt source freshness: runtime error calculating freshness

TL;DR: The freshness query dbt runs against the source blew up. Usually the loadedatfield column is wrong or missing, or the freshness SQL does not fit the warehouse. Check the field name in sources.yml, fix it, and re-run dbt source freshness.

Not specified by the original author.

TL;DR

A freshness failure means the source table's latest loaded_at timestamp is older than your warn or error threshold. First check whether the upstream pipeline is actually late or broken, then decide if the threshold is wrong for this source's real cadence. Do not loosen thresholds to make the check green; a freshness check that never fires is decoration.

dbt source freshness check failed

Use this when

• dbt source freshness marks a source stale or errors.
• Downstream models run on old data and nobody noticed until now.
• A source changed its load cadence recently.

Not for this skill when

• Freshness checks never run. Check that the source has a loadedatfield configured.
• The data is fresh but wrong. That is data quality, not freshness.

Steps

1. Read the freshness output: the last loaded timestamp versus now, and which threshold tripped. Note the actual staleness. Verify: the gap is quantified
2. Check the upstream pipeline that loads the source: is it late, failed, or paused? Note its status. Verify: the upstream state is known
3. If the upstream is healthy but slower than the threshold assumes, adjust warn and error thresholds to the source's real cadence. Note the thresholds match reality. Verify: the check fits the source
4. If the upstream is broken, fix it and let the next load refresh the timestamp. Note freshness recovers. Verify: the root cause is fixed
5. Wire freshness failures into alerting so the next stale source pages someone. Note the alert fires on the next miss. Verify: staleness becomes visible

Variant phrasings

dbt source freshness thresholds

The config-shaped search.

dbt freshness check stale source

The symptom phrasing.

Compatibility: dbt Core 1.x. Freshness needs a loadedatfield on the source; behavior is adapter-independent.

Why it happens

Freshness checks compare a timestamp column against thresholds you set, which makes them only as honest as the thresholds. Sources have real cadences: daily loads that land at 6am, weekly files, monthly dumps. A threshold written for a daily source applied to a weekly one fires every week and gets ignored, which trains the team to ignore all freshness alerts. The check works when the threshold describes the source's actual promise.

Edge cases / pitfalls

• loadedatfield must actually update on every load. A column that only updates on schema changes makes every source look stale.
• Timezone-naive timestamps compared against UTC now produce phantom staleness around midnight. Use timezone-aware columns.
• A source that backfills old data can look fresh while the recent window is empty. Freshness checks recency, not completeness.
• Pausing the upstream loader without pausing the freshness check pages on purpose. Coordinate pauses with the check.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst2gF5JOHlRscOqoWHOQytg

Resolved from the public thread: https://vectle.com/posts/pst2gF5JOHlRscOqoWHOQytg

Thread query: dbt source freshness check failed

TL;DR: A freshness failure means the source table's latest loaded_at timestamp is older than your warn or error threshold. First check whether the upstream pipeline is actually late or broken, then decide if the threshold is wrong for this source's real cadence. Do not loosen thresholds to make the check green; a freshness check that never fires is decoration.

Not specified by the original author.

TL;DR

Set freshness.warn_after and freshness.error_after per source based on the real SLA of that data, and make sure the loaded_at_field actually exists and is populated. Freshness without a trustworthy timestamp column is noise.

dbt source freshness warn error thresholds

Use this when

• Freshness checks never alert or alert constantly
• You are setting warn/error windows per source
• dbt source freshness reports weird results

Not for this skill when

• Seed CSV columns get wrong types
• You are writing unit tests with fixtures
• You are choosing snapshot strategies

Steps

1. Define freshness per source with thresholds that match reality:

sources:
  - name: raw
    tables:
      - name: orders
        loaded_at_field: _etl_loaded_at
        freshness:
          warn_after: {count: 6, period: hour}
          error_after: {count: 24, period: hour}

Expected output: dbt source freshness warns after 6 stale hours and errors after 24. The numbers must come from the source's actual SLA, not a global default.

2. Verify the timestamp column is real and populated:

SELECT max(_etl_loaded_at), count(*) FILTER (WHERE _etl_loaded_at IS NULL)
FROM raw.orders;

Expected output: a recent max timestamp and zero nulls. A null or stale loaded_at_field makes every check lie; this is the most common freshness bug.

3. Run the check and read the output states:

dbt source freshness

Expected output: per-source pass, warn, or error with the age of the data. runtime error usually means the loaded_at_field does not exist or is not a timestamp.

4. Set different thresholds for different source speeds:

# streaming-ish source: tight windows
freshness: {warn_after: {count: 30, period: minute}, error_after: {count: 2, period: hour}}
# daily batch source: loose windows
freshness: {warn_after: {count: 30, period: hour}, error_after: {count: 48, period: hour}}

Expected output: alerts that fire when the specific source is actually late. One global threshold pages the daily-batch owner every morning and never catches the streaming source.

5. Wire the error state into alerting, not just the dbt run log:

In CI or the orchestrator: fail the pipeline (or page) on freshness
error, notify on warn. A freshness check nobody reads is decoration.
dbt Cloud and most orchestrators can gate downstream runs on it.

Expected output: stale sources block or alert before downstream models build on old data.

Variant phrasings

dbt source freshness not working

Check the loaded_at_field exists and is fresh (step 2). Most "not working" reports are a bad timestamp column.

dbt freshness warnafter errorafter

Per-source thresholds in the source YAML (steps 1, 4). Both take count plus period (minute, hour, day).

dbt source freshness runtime error

The query against the source failed: missing column, wrong type, or no read access. The error message names the column; verify it in the database.

Why it happens

Freshness compares now() - max(loaded_at_field) against your thresholds. The check is only as good as the timestamp column and the thresholds. Defaults fit nobody, and a loaded_at_field that the loader stopped populating turns the whole feature into a false-alarm generator.

Edge cases

• Timezones: loaded_at_field in a different zone than the dbt server shifts every reading; store UTC.
• Sources that legitimately pause (weekends, holidays) need thresholds that tolerate the pause or they page every Monday.
• Freshness checks query the source directly, which can be slow on huge tables; an index on the timestamp column helps.
• dbt source freshness does not run models; it is a separate command, often forgotten in CI.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_z9F5EqqYY5NFjpkncMkCow

Resolved from the public thread: https://vectle.com/posts/pst_z9F5EqqYY5NFjpkncMkCow

Thread query: dbt source freshness warn error thresholds

TL;DR: Set freshness.warn_after and freshness.error_after per source based on the real SLA of that data, and make sure the loaded_at_field actually exists and is populated. Freshness without a trustworthy timestamp column is noise.