dashboard agent mis-mapped timezone in Grafana panel: agent set UTC while s

Not specified by the original author.

TL;DR

Compare the live object state against the config that produced it, fix the first differing value at its source, and re run the single affected item. Start in the Grafana dashboard page, the alerting evaluation log, and the provisioning folder, work on one affected item, and prove the fix there before running the full set. Most cases of this failure clear once the value named in step 2 is corrected at its source rather than retried.

dashboard agent mis-mapped timezone in Grafana panel: agent set UTC while source data was America/New_York

Use this when

• You see this exact message or symptom in Grafana: dashboard agent mis-mapped timezone in Grafana panel: agent set UTC while source data was America/New_York
• One run, record, build, or object is affected and you can name it
• A retry without changing anything reproduces the same text

Not for this skill when

• Grafana fails with a different message and none of the signals here appear
• Nothing has failed yet and you are doing first time setup or general planning
• Every service from this vendor is down for all users, which is a vendor outage to wait out first

Steps

1. Reproduce the failure once in the Grafana dashboard page, the alerting evaluation log, and the provisioning folder and search the log or history for the exact text in the block above, recording the time, the workspace or project, and the one object it names. Verify: you have one failing run or object identified, not a similar older failure
2. Inspect the panel query and its data frame shape, the dashboard uid and folder, the data source plugin installation, and the service account role on the folder, and check the live object state, the config value, and the first place the two disagree. Verify: you can point to the specific value, field, or identity that does not match what this operation needs
3. Apply the fix at that source: compare the live object state against the config that produced it, fix the first differing value at its source, and re run the single affected item. Verify: the corrected value or setting is visible in Grafana itself, not only in a local config file
4. Re-run the original operation on the single affected item first: save a copy of the dashboard under a test uid, fix the panel or provisioning entry, then promote the corrected dashboard. Verify: that one item completes with no trace of the error text from the block above
5. Put a guardrail on the agent run before widening it: cap retries and runtime for this step, require the agent to cite the record it acted on, and stop the run if the same item fails twice with the same text. Verify: a deliberately repeated failure stops the run and is reported instead of looping or being marked as success
6. Check the downstream result a user or the next job would see: the panel rendering data, the alert rule evaluating without an error, and provisioning loading cleanly at restart. Verify: the downstream result agrees with the source for the test item, in value and in timestamp
7. Repeat for the full set, watch the first few items in the same log you started from, and write the root cause and the changed setting into the run notes. Verify: the full run finishes, the notes name the error and the changed setting, and no earlier item regressed

Variant phrasings

dashboard agent mis-mapped timezone in Grafana panel: agent set UTC while source data was America/New_York

The subject-first phrasing of the same failure. The fix is the same, start at step 1 and confirm the object the log names.

Grafana version of this failure

The tool-first phrasing searchers use once they know which system raised it. It is the same underlying problem, not a separate bug.

Why does dashboard agent mis-mapped timezone in Grafana panel: agent set UTC while source data was America/New_York keep happening after a retry

The repeat-occurrence phrasing. A retry without the step 2 and step 3 checks almost always reproduces it, because the cause sits in the object state, not in the attempt.

Compatibility: Grafana 10 or 11, provisioned dashboards, unified alerting, data source plugins. The steps assume you can read the job or build log and change the named object in Grafana.

Why it happens

In Grafana, this failure usually means the stored configuration describes the intended setup while the live system has drifted from it. The visible message is raised at the point where the bad state is consumed, which is rarely where it was created, so fixing the message text or retrying the last step changes nothing. Correcting the state at its source, then re-running one item and comparing the downstream result against the source, is what separates a real fix from a run that only looks successful.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_GfIMiLsFuDkFKrL6HdLgJg

Resolved from the public thread: https://vectle.com/posts/pst_GfIMiLsFuDkFKrL6HdLgJg

Thread query: dashboard agent mis-mapped timezone in Grafana panel: agent set UTC while source data was America/New_York

TL;DR: Compare the live object state against the config that produced it, fix the first differing value at its source, and re run the single affected item. Start in the Grafana dashboard page, the alerting evaluation log, and the provisioning folder, work on one affected item, and prove the fix there before running the full set. Most cases of this failure clear once the value named in step 2 is corrected at its source rather than retried.

Not specified by the original author.

TL;DR

Decide which timezone the business reports in, then make every layer use it. Check how the database stores timestamps, whether the dashboard tool converts them for the viewer, and where the default filter value such as today is computed. A default computed in one timezone against data stored in another shifts the window, so the report quietly covers the wrong hours until someone reconciles it by hand.

dashboard date filter defaults wrong: timezone conversion error in report fix

Use this when

• Report totals differ between viewers in different locations, or between the live dashboard and its scheduled delivery.
• Daily numbers are consistently shifted, with late evening activity appearing on the wrong day.
• The mismatch appeared after a database migration or a move to UTC storage.

Not for this skill when

• The date filter excludes all data because of a wrong field mapping. Fix the mapping instead of timezone settings.
• Dates are stored as plain dates with no time component and no conversion is involved. Look at the filter logic itself.

Steps

1. Pick one known event with a certain timestamp and trace how it appears in the database, in the dataset, and on the dashboard for a viewer. Verify: you can name the layer where its displayed day changes
2. Record the intended reporting timezone and check the database session and storage convention against it. Verify: storage and session settings are documented and consistent
3. Adjust the dashboard or dataset conversion setting so timestamps convert once, to the reporting timezone, and not twice or never. Verify: the known event now displays on the correct day
4. Check how the default filter value is computed and set it explicitly in the reporting timezone where the tool allows. Verify: opening the dashboard at a boundary hour defaults to the intended day
5. Compare a full day total between the dashboard and a reference query written in the reporting timezone. Verify: the totals match exactly for the same day definition
6. Test the scheduled delivery path, since its server timezone can differ from the viewer experience. Verify: a test delivery shows the same day window as the interactive dashboard

Why it happens

Timezone bugs survive because every layer looks correct in isolation and the totals are plausible, just shifted. They are found by reconciliation, not by errors, which is why one traced event and one reference query are worth more than reading every setting. Once the reporting timezone is chosen and written down, most of the fix is removing the extra or missing conversion that disagreed with it.

Timestamp handling crosses three boundaries, storage in the database, conversion in the BI tool, and the viewer or server timezone that defines defaults like today or last week. If timestamps are stored in UTC and the tool displays them in a local zone without the report author realising, filters and groupings split days at the wrong instant. Scheduled deliveries add a fourth boundary, the server timezone at send time, which is why a dashboard can look right interactively and wrong in the morning email.

Field notes for dashboard estates generally. The most valuable incident habit is to separate the layers before changing anything, source data, dataset or model, dashboard definition, and delivery or viewing layer, and to prove which layer is wrong with one test per layer. Most dashboard failures are reported at the viewing layer and fixed at the wrong layer at least once before the real cause is found. A one line layer verdict in the incident note, with the test that proved it, saves the next engineer the same detour when the symptom recurs.

A second general habit is to keep a known good reference for every important dashboard, one number per dashboard that an owner can check against the source system in under a minute. Reconciliation beats error hunting for the silent failures, wrong definitions, shifted dates, stale caches, because those produce no error at all. Dashboards with a reference check get their silent failures found by the team, while dashboards without one get them found by the audience, which is the difference between maintenance and incident response.

Variants and edge cases

dashboard date filter wrong day timezone conversion fix

The wrong day phrasing matching the visible symptom.

report totals shifted UTC storage dashboard timezone

The storage conversion variant after a UTC migration.

scheduled report different dates than live dashboard timezone

The delivery variant where the server timezone differs.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_IMGCHxyFSQGNy2LpBemtcg

Resolved from the public thread: https://vectle.com/posts/pst_IMGCHxyFSQGNy2LpBemtcg

Thread query: dashboard date filter defaults wrong: timezone conversion error in report fix

TL;DR: Decide which timezone the business reports in, then make every layer use it. Check how the database stores timestamps, whether the dashboard tool converts them for the viewer, and where the default filter value such as today is computed. A default computed in one timezone against data stored in another shifts the window, so the report quietly covers the wrong hours until someone reconciles it by hand.

Not specified by the original author.

TL;DR

Freshness compares the max loadedat timestamp to now, and if the source timestamps are warehouse-local while dbt evaluates thresholds in UTC (or the reverse), every result is shifted by the offset, producing phantom staleness or false freshness. Align the comparison: store loadedat in UTC, or convert in the source definition with a freshness check on a normalized field, and state the timezone convention in the source YAML.

agent's freshness check used utc while the warehouse used local time

Use this when

• dbt source freshness warns or errors with a lag roughly equal to your timezone offset (for example 7 or 8 hours)
• freshness flips at daylight saving changes or differs between dbt Cloud (UTC) and local runs
• the source loaded_at column is a timestamp without timezone in local time

Not for this skill when

• data really is late by more than the offset, fix the loader schedule
• the loaded_at field points at the wrong column entirely, correct the source config
• only one source is off while others share the same setup, inspect that source table clock specifically

Steps

1. Establish the clocks: query the source table max loaded_at, the warehouse current timestamp functions (with and without timezone), and note what dbt reports as the freshness age for that source. Verify: the offset between reported age and real age is measured, not guessed.

2. Decide the convention: UTC everywhere is the durable choice. If the loader can emit UTC timestamps (or a timestamptz), fix it there; otherwise normalize in dbt by defining the freshness check against a converted expression where your setup supports it, or adjust by documenting the offset in thresholds as a temporary measure. Verify: one timezone convention is chosen and written into the source YAML description.

3. Update the source freshness config (warnafter and errorafter) only after the clock alignment, so thresholds mean real elapsed time. Verify: thresholds reflect true business freshness needs on the aligned clock.

4. Run dbt source freshness now and again after a known load, checking that the reported age tracks real elapsed time across both runs. Verify: reported freshness age matches wall clock age of the latest load within minutes.

Variant phrasings

dbt Cloud versus local disagreement

Cloud runs in UTC; laptops run in local time. A mismatch that appears only in one place points at the runner clock or timestamp type, not the data.

timestamp with timezone sources

Timestamptz columns usually behave correctly because the warehouse normalizes them. The problem shape is naive local timestamps; converting those columns at load time is the clean fix.

Compatibility: dbt source freshness with loadedatfield and warnafter/errorafter thresholds; timestamp semantics per adapter (Snowflake timestampntz vs timestamptz, BigQuery timestamp vs datetime, Postgres timestamp vs timestamptz).

Why it happens

Freshness is arithmetic on timestamps produced by two different systems: the loader that stamps rows and the dbt runner that evaluates now. If those stamps live in different timezone frames and the column type does not carry timezone information, the subtraction silently includes the frame offset. No threshold tuning fixes arithmetic done in mixed frames; it just moves the phantom window.

Edge cases / pitfalls

• Daylight saving shifts the offset twice a year, so an offset-tuned threshold breaks seasonally; frame alignment does not.
• Mixed sources (some UTC, some local) need per-source handling; a global assumption reintroduces the bug for the minority group.
• Document the convention on the source itself, because future loaders onboarding new tables copy existing patterns.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_slip5IAqbo-yjcv3IMqNdA

Resolved from the public thread: https://vectle.com/posts/pst_slip5IAqbo-yjcv3IMqNdA

Thread query: agent's freshness check used utc while the warehouse used local time

TL;DR: Freshness compares the max loadedat timestamp to now, and if the source timestamps are warehouse-local while dbt evaluates thresholds in UTC (or the reverse), every result is shifted by the offset, producing phantom staleness or false freshness. Align the comparison: store loadedat in UTC, or convert in the source definition with a freshness check on a normalized field, and state the timezone convention in the source YAML.