agent misconfigured the dagster freshness threshold, false stale alerts

Not specified by the original author.

TL;DR

A freshness threshold is a number somebody chose, and if an agent generated it from a guess, such as assuming hourly data for a daily asset, the alerts are fiction. Derive the threshold from the asset's actual cadence and the business's tolerance: measure the real interval between materializations over recent weeks, set the lag window comfortably above the normal interval plus the run duration, and if the source is calendar shaped, use a cron based policy that knows about weekends instead of a flat lag.

agent misconfigured the dagster freshness threshold, false stale alerts

Use this when

• Stale alerts fire on assets whose latest data is as recent as the source allows.
• Alerts follow a pattern, such as firing every weekend or before a daily run lands.
• The freshness policy in code was generated or copied from another asset.

Not for this skill when

• The asset really is older than the business tolerance because a run failed. Fix the run; the alert did its job.
• There is no freshness policy and the stale label comes from a different monitoring tool. Fix that tool's threshold there.

Steps

1. Read the current policy in the asset definition and note the exact threshold: a maximum lag value or a cron window, and who or what set it. Note the number and its origin if recorded. Verify: the current promise is stated precisely
2. Measure the asset's real cadence from materialization history over several weeks: typical interval, worst normal interval, and the run duration. Note the three numbers. Verify: the threshold discussion uses measured cadence, not vibes
3. Compare the threshold with the schedule that feeds the asset, including the upstream source's own publishing cadence, and note any period where the threshold is mathematically impossible to meet, such as a lag window shorter than the source interval. Note the impossible window. Verify: impossible promises are identified explicitly
4. Reset the policy to fit reality and need: lag above the worst normal interval plus run time, or a cron based freshness policy matching the business calendar. Record the reasoning in a comment beside the policy. Note the new values. Verify: the policy is achievable on a normal bad day, not only on a perfect one
5. Watch the policy over at least one full cadence cycle including a weekend if relevant, and confirm no false stale fires while a genuinely delayed test still would. Note the cycle result. Verify: the alert is quiet for healthy data and still meaningful for late data

Variant phrasings

dagster freshness policy threshold too tight false alerts daily asset

The cadence mismatch phrasing.

dagster maximum lag vs cron freshness policy weekends

The policy type choice search.

Compatibility: Dagster 1.x freshness policies on assets, lag based and cron based. The measurement approach uses materialization history available in the Dagster UI and storage.

Why it happens

Freshness thresholds fail silently in design and loudly in operation. A threshold is a claim about a pipeline's rhythm, and a generated or copied value encodes some other pipeline's rhythm, or none at all. Flat lag windows are especially brittle for calendar shaped data, since a daily weekday pipeline is more than forty eight hours old by Monday morning while being perfectly healthy, so a twenty four hour lag policy manufactures a weekly false alarm that trains everyone to ignore the real one. Matching the policy type to the data's calendar is the actual fix; widening a flat number only moves the false alarm.

Edge cases / pitfalls

• Do not set the threshold from the best week in the history. Use a bad but normal week, or the first incident will re-open the tuning argument.
• Multi asset chains inherit the slowest upstream cadence. Set freshness on the end asset from the chain's rhythm, not the asset's own run duration.
• A policy change is a code change. Route it through review like any other contract change, with the measurement attached.
• If the business tolerance is genuinely tighter than the source cadence, the answer is a conversation about the source, not a policy that alerts forever.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst8Ij9XUzZfffZXnt_NfS9w

Resolved from the public thread: https://vectle.com/posts/pst8Ij9XUzZfffZXnt_NfS9w

Thread query: agent misconfigured the dagster freshness threshold, false stale alerts

TL;DR: A freshness threshold is a number somebody chose, and if an agent generated it from a guess, such as assuming hourly data for a daily asset, the alerts are fiction. Derive the threshold from the asset's actual cadence and the business's tolerance: measure the real interval between materializations over recent weeks, set the lag window comfortably above the normal interval plus the run duration, and if the source is calendar shaped, use a cron based policy that knows about weekends instead of a flat lag.

Not specified by the original author.

TL;DR

A freshness policy is a promise: this asset will be materialized within a stated window of its upstream data or on a stated cadence. Dagster marks the asset stale when the latest materialization is older than that promise allows, which usually means an upstream asset stopped landing, a schedule or sensor did not fire, or the policy window is tighter than the pipeline can physically run. Compare the policy deadline with the actual upstream and asset timestamps before loosening anything.

dagster freshness policy violation, asset marked stale

Use this when

• Dagster shows a freshness policy violation or stale badge on an asset.
• Freshness alerts fire on a rhythm, such as every Monday or after holidays.
• Upstream assets look recent but the governed asset is flagged stale.

Not for this skill when

• The asset run is actively failing on code errors. Freshness is a symptom there, not the diagnosis.
• There is no freshness policy on the asset. A plain old asset is just unscheduled, not stale.

Steps

1. Read the asset freshness policy in code: the maximum lag or the cron based window it declares. Note the exact deadline rule. Verify: the promise being measured is known word for word
2. In the asset view, compare the latest materialization time of this asset and of each upstream asset against that deadline. Note which link in the chain is actually old. Verify: the stale link is located, upstream or here
3. Check the automation that should have run it: schedule, sensor, or automation condition evaluations around the missed window, including whether the daemon was up. Note any skipped or errored evaluation. Verify: the missed trigger is explained
4. Look for calendar traps: a cron window written in a different timezone than the team assumes, or a weekday only cadence measured against a policy that counts weekends. Note any mismatch. Verify: timezone and calendar effects are confirmed or cleared
5. Fix the real link: restore the upstream run or automation, or, if the policy itself promises more than the business needs, change the policy in code with the reason recorded. Then watch one full cycle. Note the asset returns inside its window. Verify: freshness holds for a full cycle, not just one manual run

Variant phrasings

dagster asset stale freshness policy maximum lag

The lag based policy phrasing.

dagster freshness check failed cron window

The cron based policy version.

Compatibility: Dagster 1.x freshness policies, both lag based and cron based, evaluated by the Dagster daemon. Exact policy classes differ across 1.x releases; the timestamp comparison is the constant.

Why it happens

Freshness is computed from timestamps, not from intentions. Dagster takes the newest materialization, applies the policy arithmetic, and reports a violation the moment the deadline passes, whether the cause is a broken upstream, a paused schedule, or a policy written during a hack week that promised hourly data from a daily source. Recurring violations on a calendar pattern are the tell for a policy and cadence mismatch rather than a broken pipeline, because broken pipelines do not respect weekends.

Edge cases / pitfalls

• Manually materializing once clears the badge and proves nothing. Judge the fix over the next scheduled cycle.
• A partitioned asset can be fresh on recent partitions and stale on the partition the policy evaluates. Check which partitions the policy covers.
• Backfills rewrite old partitions without making the asset fresh. Freshness follows the relevant latest data, not run activity.
• Alert routing matters: a stale asset nobody owns generates noise until people mute it. Assign the policy an owner when you write it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Q2i32oSaGFIYtyHJEC5e0g

Resolved from the public thread: https://vectle.com/posts/pst_Q2i32oSaGFIYtyHJEC5e0g

Thread query: dagster freshness policy violation, asset marked stale

TL;DR: A freshness policy is a promise: this asset will be materialized within a stated window of its upstream data or on a stated cadence. Dagster marks the asset stale when the latest materialization is older than that promise allows, which usually means an upstream asset stopped landing, a schedule or sensor did not fire, or the policy window is tighter than the pipeline can physically run. Compare the policy deadline with the actual upstream and asset timestamps before loosening anything.

Not specified by the original author.

TL;DR

Stop polling a table whose source is dead: the agent should check the loader's state once, report the staleness with its cause, and schedule a single recheck after the expected repair window instead of polling the same stale timestamp. Polling measures the table, but the fault lives in the loader, and the loop delays the escalation that would fix it.

agent loop detected: repeated freshness check failure loop on stale table

Use this when

• The check or run fails with this symptom: the agent repeats freshness checks on a table that stays stale.
• You are working on an agent looping on repeated freshness checks against a stale table and the failure repeats on the same data or configuration.
• You need the failing rows, rule, or stage isolated before changing thresholds or code.

Not for this skill when

• Active polling during a live incident where the loader is being repaired and frequent checks are deliberate instead.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Review the agent's polling history: how many freshness checks ran, at what interval, against which timestamp. Verify: the polling pattern is documented
2. Have the agent check the feeding job's state and last success directly, once, and attach that state to its report. Verify: the loader state is included in the agent's report
3. Replace open-ended polling with a stop rule: after two identical stale results, escalate and schedule one recheck at a stated time. Verify: the stop and recheck rule is configured
4. Record the staleness start and the escalation in the run log so the incident has a timeline. Verify: the timeline is preserved in the log
5. Test the rule against a deliberately stale table and confirm the agent escalates instead of looping. Verify: the test ends in escalation after the configured count

Variant phrasings

agent loop repeated freshness check stale table

The verbatim freshness-loop phrasing.

agent stuck polling freshness status loader state

The loader-state phrasing for checking the pipeline, not the table.

freshness polling stop rule escalate stale data agent

The stop-rule phrasing for bounded rechecks.

Compatibility: Agent freshness polling on stale warehouse tables with Soda or custom checks

Why it happens

Freshness only changes when a load lands, and loads are driven by pipelines the polling agent does not control. Without a stop rule keyed to the loader's state, the agent spends its budget re-measuring a known fact while the repair waits for a human the agent never notified.

Edge cases / pitfalls

• Poll with backoff even in the allowed window. Fixed-interval polling of a dead table is pure waste.
• Escalations should carry the last good timestamp and the loader's identity so the human starts ahead.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_etzdrXVsjLfoJXAEiEYY1Q

Resolved from the public thread: https://vectle.com/posts/pst_etzdrXVsjLfoJXAEiEYY1Q

Thread query: agent loop detected: repeated freshness check failure loop on stale table

TL;DR: Stop polling a table whose source is dead: the agent should check the loader's state once, report the staleness with its cause, and schedule a single recheck after the expected repair window instead of polling the same stale timestamp. Polling measures the table, but the fault lives in the loader, and the loop delays the escalation that would fix it.