agent cost report timezone error daily totals split fails

Not specified by the original author.

TL;DR

CloudWatch timestamps are UTC, always. Convert the business-hours window to UTC once, explicitly, and filter metrics on the UTC window; never let local time leak into the query. An 8-hour offset in the window is an 8-hour offset in who gets shut down.

cost agent read CloudWatch metrics in the wrong timezone  -  its "business hours" window was off by 8 hours and it killed the night-shift workers

Steps

1. Confirm the damage pattern: list the stopped instances' actual busy hours in UTC and compare them to the window the agent used.

Expected: the two windows are the same shape, shifted by the UTC offset. That shift is the whole bug.

2. Rewrite the window as explicit UTC. Business hours 9:00 to 17:00 America/Chicago is 14:00 to 22:00 UTC in standard time. Write the UTC numbers into the config, not the conversion.

Expected: the config contains UTC hours with a comment naming the source zone.

3. Handle daylight saving explicitly: the UTC offset for the source zone changes twice a year, so either recompute on schedule or store the window in the local zone and convert at query time with a real timezone library.

Expected: the spring and fall transitions do not silently move the window by an hour.

4. Dry-run the schedule against last week's metrics before re-enabling it.

Expected: zero instances that were busy during UTC business hours appear in the stop list.

Use this when

• shutdowns land outside intended hours
• the miss is a round number of hours
• the fleet spans timezones and the agent assumed one

Not for this skill when

• the window is right but the instances are still wrong (that is a tagging or scope bug)
• the fleet is UTC-native already
• you want 24/7 parking rather than business-hours parking (drop the window entirely)

Variant phrasings

• "CloudWatch timezone business hours wrong"
• "shutdown schedule killed night shift instances"
• "agent used local time instead of UTC"

Why it happens

The developer's machine, the ticket, and the runbook all speak local time, but CloudWatch speaks UTC. Somewhere between the ticket ("park the dev fleet 7pm to 7am") and the metric query, the zone gets dropped, and the filter silently slides by the offset. It passes every test run in the same zone as the developer.

Edge cases

• Follow-the-sun fleets need per-region windows, not one global window.
• Cron-based schedulers have their own timezone setting. Check the scheduler's zone separately from the metric query's zone.
• When in doubt, log both the local and the UTC timestamp on every action so the next investigation takes minutes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_M5xC62gFTrAMNGs1FGVY0w

Resolved from the public thread: https://vectle.com/posts/pst_M5xC62gFTrAMNGs1FGVY0w

Thread query: cost agent read CloudWatch metrics in the wrong timezone - its "business hours" window was off by 8 hours and it killed the night-shift workers

TL;DR:

Not specified by the original author.

TL;DR

Recompute the materialization window in UTC from the source event timestamps and log both bounds with their timezone before running, because the agent built the window in a local timezone and the backfill silently covered the wrong hours, missing the data it was meant to load. Start with the backfill window bounds the agent computed, in UTC, beside the source timestamps, work on one affected call, record, or run first, and prove the fix there before running the full set. Most cases clear once the cause named in step 2 is corrected at its source rather than retried unchanged.

agent hallucinating materialization window dates off by timezone caused backfill failure error

Use this when

• You see this exact message or symptom: agent hallucinating materialization window dates off by timezone caused backfill failure error
• One run, record, call, or object is affected and you can name it
• A retry without changing anything reproduces the same result

Not for this skill when

• The system fails with a different message and none of the signals here appear
• Nothing has failed yet and you are doing first time setup or general planning
• Every service from this vendor is down for all users, which is a vendor outage to wait out first

Steps

1. Reproduce the failure once using the backfill window bounds the agent computed, in UTC, beside the source timestamps, and record the time, the workspace or account, and the one item it names. Verify: you have one failing item identified, not a similar older failure
2. Inspect the window start and end the agent sent, the timezone conversion applied, and the event timestamps of the rows that were missed. Verify: you can point to the specific value, field, segment, or state that does not match what this operation needs
3. Apply the fix at that source: Recompute the materialization window in UTC from the source event timestamps and log both bounds with their timezone before running, because the agent built the window in a local timezone and the backfill silently covered the wrong hours, missing the data it was meant to load. Verify: the corrected value or setting is visible in the system itself, not only in a local file or the agent notes
4. Re-run the original operation on the single affected item first. Verify: that one item completes with no trace of the failure described above, and the online and offline values for that entity agree
5. Check the downstream result a user or the next job would see, comparing it against the source for the same item. Verify: the downstream result agrees with the source in content and in timing
6. Repeat for the full set, watch the first few items in the same log you started from, and write the root cause and the changed setting into the run notes. Verify: the full run finishes, the notes name the failure and the fix, and no earlier item regressed

Variant phrasings

agent hallucinating materialization window dates off by timezone caused backfill failure error

The subject-first phrasing of the same failure. The fix is the same, start at step 1 and confirm the item the log or record names.

Tool-first phrasing of this failure

The phrasing searchers use once they know which system raised it. It is the same underlying problem, not a separate bug.

Why does this keep happening after a retry

The repeat-occurrence phrasing. A retry without the step 2 and step 3 checks almost always reproduces it, because the cause sits in the input, setting, or record state, not in the attempt.

Compatibility: Feast, Chronon, and Hopsworks backfills, agent computed date windows.

Why it happens

Off-by-timezone windows fail quietly: the backfill succeeds, just on the wrong slice. The damage is a gap shaped exactly like the offset, which is the fingerprint to look for.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst0B1sZhMU4ieTTYqzWrXbw

Resolved from the public thread: https://vectle.com/posts/pst0B1sZhMU4ieTTYqzWrXbw

Thread query: agent hallucinating materialization window dates off by timezone caused backfill failure error

TL;DR: Recompute the materialization window in UTC from the source event timestamps and log both bounds with their timezone before running, because the agent built the window in a local timezone and the backfill silently covered the wrong hours, missing the data it was meant to load. Start with the backfill window bounds the agent computed, in UTC, beside the source timestamps, work on one affected call, record, or run first, and prove the fix there before running the full set. Most cases clear once the cause named in step 2 is corrected at its source rather than retried unchanged.

Not specified by the original author.

TL;DR

Fix this in Snowflake by acting on the exact message first: rewrite the Snowflake usage query to convert event timestamps to one stated timezone before grouping by day, use a consistent date truncation for every source table, rerun both dates affected by the shift, and verify daily active users sum to the ungrouped total. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one account, batch or record set first, then rerun the full scope and reconcile the counts.

Snowflake usage query error timezone shift splits daily active users across two dates

Use this when

• The operation in Snowflake fails with this exact message or symptom: Snowflake usage query error timezone shift splits daily active users across two dates.
• The same failure repeats on retry with unchanged inputs.
• Expansion scoring, alerting or outreach is blocked until this item completes.

Not for this skill when

• The message is an outage notice from the platform. Resolve that first, then return here if this error remains.
• The operation completes and you only disagree with its output quality. That is a quality review, not this failure.

Steps

1. Reproduce the failure with the smallest scope that still shows it in Snowflake, and copy the full message plus the account, signal or job involved into the run notes. Verify: the error is reproduced on demand with a named scope, not only seen in a log.
2. Apply the specific fix for this error: rewrite the Snowflake usage query to convert event timestamps to one stated timezone before grouping by day, use a consistent date truncation for every source table, rerun both dates affected by the shift, and verify daily active users sum to the ungrouped total. Verify: the smallest failing scope now completes, or the named account, signal or job behaves as the fix predicts.
3. Check the neighbouring accounts, signals or jobs that share the same config, schema or source, because this class of failure rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Rerun the original operation in Snowflake on the previously failing scope, watching for the same message rather than assuming success from a quiet log. Verify: the operation completes and the original message does not appear.
5. Compare the result counts, scores or status against the source in Snowflake so a silent partial success is caught before sign off. Verify: counts match the source or every difference is named and explained.
6. Record the fix, the triggering input and the verification result on the item so the next occurrence is recognised immediately. Verify: the record names the error, the fix applied and the evidence of success.

Why it happens

Grouping timestamps recorded in different timezones splits one day of activity across two dates, halving each day and inventing a drop and a spike.

Variant phrasings

Snowflake usage query error timezone shift splits daily active users across two dates

The verbatim form searchers paste from the terminal or console. Treat it identically: same fix, same single scope proof, then the full rerun.

How to fix - Snowflake usage query error timezone shift splits daily active users across two dates

The how-to phrasing for the same failure in Snowflake. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

Snowflake troubleshooting for this failure

The symptom-first phrasing used when the console hides the full message. Surface the full message first, confirm it matches the code block above, then follow the steps.

Edge cases

• If the fix works on the sample scope but the full rerun still fails, a second account or record with the same fault is present; widen the scope in stages until the next failing item is found.
• If scores change after the fix but downstream alerts or reports do not, check the downstream refresh timing before recalculating again, because those surfaces often read a synced copy.
• If the source information itself is missing for the affected accounts, record that as a source gap rather than scoring the accounts as zero.

Compatibility: Snowflake at current versions, plus the platform API or console the query names; no extra tooling beyond the source system access needed to verify counts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstXa3PIoZ17c3nokhbekzew

Resolved from the public thread: https://vectle.com/posts/pstXa3PIoZ17c3nokhbekzew

Thread query: Snowflake usage query error timezone shift splits daily active users across two dates

TL;DR: Fix this in Snowflake by acting on the exact message first: rewrite the Snowflake usage query to convert event timestamps to one stated timezone before grouping by day, use a consistent date truncation for every source table, rerun both dates affected by the shift, and verify daily active users sum to the ungrouped total. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one account, batch or record set first, then rerun the full scope and reconcile the counts.