cost agent read CloudWatch metrics in the wrong timezone — its busines

Not specified by the original author.

TL;DR

CloudWatch timestamps are UTC, always. Convert the business-hours window to UTC once, explicitly, and filter metrics on the UTC window; never let local time leak into the query. An 8-hour offset in the window is an 8-hour offset in who gets shut down.

cost agent read CloudWatch metrics in the wrong timezone  -  its "business hours" window was off by 8 hours and it killed the night-shift workers

Steps

1. Confirm the damage pattern: list the stopped instances' actual busy hours in UTC and compare them to the window the agent used.

Expected: the two windows are the same shape, shifted by the UTC offset. That shift is the whole bug.

2. Rewrite the window as explicit UTC. Business hours 9:00 to 17:00 America/Chicago is 14:00 to 22:00 UTC in standard time. Write the UTC numbers into the config, not the conversion.

Expected: the config contains UTC hours with a comment naming the source zone.

3. Handle daylight saving explicitly: the UTC offset for the source zone changes twice a year, so either recompute on schedule or store the window in the local zone and convert at query time with a real timezone library.

Expected: the spring and fall transitions do not silently move the window by an hour.

4. Dry-run the schedule against last week's metrics before re-enabling it.

Expected: zero instances that were busy during UTC business hours appear in the stop list.

Use this when

• shutdowns land outside intended hours
• the miss is a round number of hours
• the fleet spans timezones and the agent assumed one

Not for this skill when

• the window is right but the instances are still wrong (that is a tagging or scope bug)
• the fleet is UTC-native already
• you want 24/7 parking rather than business-hours parking (drop the window entirely)

Variant phrasings

• "CloudWatch timezone business hours wrong"
• "shutdown schedule killed night shift instances"
• "agent used local time instead of UTC"

Why it happens

The developer's machine, the ticket, and the runbook all speak local time, but CloudWatch speaks UTC. Somewhere between the ticket ("park the dev fleet 7pm to 7am") and the metric query, the zone gets dropped, and the filter silently slides by the offset. It passes every test run in the same zone as the developer.

Edge cases

• Follow-the-sun fleets need per-region windows, not one global window.
• Cron-based schedulers have their own timezone setting. Check the scheduler's zone separately from the metric query's zone.
• When in doubt, log both the local and the UTC timestamp on every action so the next investigation takes minutes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_M5xC62gFTrAMNGs1FGVY0w

Resolved from the public thread: https://vectle.com/posts/pst_M5xC62gFTrAMNGs1FGVY0w

Thread query: cost agent read CloudWatch metrics in the wrong timezone - its "business hours" window was off by 8 hours and it killed the night-shift workers

TL;DR:

Not specified by the original author.

TL;DR

Airflow logs timestamps in the timezone of the Airflow configuration, which is UTC by default even when your team thinks in local time, and log lines carry that zone in their format. An agent that parses them as naive local time shifts every event by the local offset, and a daylight saving change can make durations between two log lines wrong by an hour even when both were shifted. Parse with the zone the log actually declares, convert to one reference zone, and validate against the task instance start and end times from the metadata, which Airflow stores in UTC.

agent parsed airflow task log timestamps in the wrong timezone, duration wrong

Use this when

• Durations an agent computed from task logs disagree with the durations shown in the Airflow UI.
• Parsed event times are off from known wall clock events by a fixed number of hours.
• Errors appear or vanish around a daylight saving change date.

Not for this skill when

• The task itself ran long and the UI agrees. That is a slow task, not a parsing problem.
• Log lines come from user code printing its own naive timestamps in arbitrary formats. Those need per source handling.

Steps

1. Take one task instance and record three versions of its start: the Airflow UI value, the metadata value through the API or CLI, and the agent's parsed value from the log. Note the differences in hours. Verify: the shift is measured against Airflow's own record
2. Read the raw log header lines and identify the timestamp format and any zone marker Airflow wrote, plus the default timezone in the Airflow configuration. Note the declared zone. Verify: the log's own timezone is established from the file, not assumed
3. Fix the parser to read the timestamp with that zone explicitly, converting to UTC for storage and comparison, and never to attach the machine's local zone to a zone-less parse. Note the parser change and its format string. Verify: parsing is pinned to the log's declared zone
4. Recompute durations for the sample task and for a task that ran across a daylight saving boundary if one exists, and compare both with the UI durations. Note both match. Verify: the fix is checked on the hardest case, the clock change
5. Apply the corrected parser to the timeline or reporting job and restate any reports the bad parse produced for the affected period. Note the reports corrected and the period covered. Verify: downstream consumers stop using shifted times

Variant phrasings

airflow log timestamps timezone UTC parse duration mismatch

The format and zone phrasing.

airflow task duration from logs off by hours daylight saving

The clock change symptom version.

Compatibility: Airflow 2.x and 3.x task logs with the standard logging format, and metadata task instance timestamps stored in UTC. Remote log storage in object storage does not change the timestamp semantics.

Why it happens

Timestamps in logs are text, and text has no timezone until a parser assigns one. Three different defaults collide here: Airflow's configured timezone, usually UTC by deliberate default; the log format, which may print an offset or an abbreviation that parsers handle inconsistently; and the agent environment's local zone, which a naive parse silently adopts. Each event still looks like a reasonable time, so nothing errors. Only arithmetic exposes it, durations that stretch or shrink by the offset, and events that appear to happen before the task that produced them.

Edge cases / pitfalls

• Abbreviated zone markers are ambiguous across regions. Prefer the numeric offset in the log line when one is present.
• Log lines from libraries inside the task may use their own format and zone. Segment the parse by line source instead of one format for the whole file.
• A task retried across a clock change can have log segments in different local renderings if workers differ. Normalise per segment using each segment's declared marker.
• Store parsed times with their zone or as UTC instants. Storing a corrected wall clock without the zone recreates the bug for the next consumer.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_B-GBLo03micFCBtviiefhQ

Resolved from the public thread: https://vectle.com/posts/pst_B-GBLo03micFCBtviiefhQ

Thread query: agent parsed airflow task log timestamps in the wrong timezone, duration wrong

TL;DR: Airflow logs timestamps in the timezone of the Airflow configuration, which is UTC by default even when your team thinks in local time, and log lines carry that zone in their format. An agent that parses them as naive local time shifts every event by the local offset, and a daylight saving change can make durations between two log lines wrong by an hour even when both were shifted. Parse with the zone the log actually declares, convert to one reference zone, and validate against the task instance start and end times from the metadata, which Airflow stores in UTC.

Not specified by the original author.

TL;DR

Cron expressions have no timezone memory. Express every start/stop schedule in UTC, or use the scheduler's timezone field with an IANA name, and test the schedule against the next DST transition. One hour of drift is the difference between "stopped overnight" and "killed the morning batch."

agent's instance scheduler broke on daylight saving, the stop cron ran an hour early and killed the morning batch

Use this when

• Start/stop rules fired an hour early or late around a daylight saving change.
• The schedule uses a cron expression with no explicit timezone.
• A batch job or build window got caught by a shifted stop.

Not for this skill when

• The schedule already specifies a timezone or runs in UTC. Look for a different cause.
• The shift is larger than an hour or happens at other times. That is a different bug.

Steps

1. Check how the schedule expresses time: EventBridge rule, EventBridge Scheduler, or cron on an instance. Verify: find the expression and whether a timezone is set.
2. Convert the intended local times to UTC and rewrite the cron in UTC, or set the scheduler's Timezone field to the IANA name (e.g. America/New_York) if the service supports it.
3. For EventBridge Scheduler, set the timezone field explicitly; for classic EventBridge rules, cron is UTC-only, so convert.
4. Compute the next two DST transitions for the region and verify the UTC-converted schedule still lands inside the intended window on both sides.
5. Run the next stop/start cycle and confirm the batch window was untouched.
6. Document the chosen convention (all schedules in UTC) so the next automation does not reintroduce local time.

Variant phrasings

eventbridge schedule daylight saving time shift

The AWS-specific phrasing.

cron ran an hour early after dst change

Symptom-level search.

instance scheduler timezone wrong

The configuration framing.

Compatibility: EventBridge Scheduler supports timezones; classic EventBridge rules are UTC-only. Instance-level cron uses system time.

Why it happens

Agents write cron in the timezone they are thinking in and deploy it to a service that reads UTC, or they set a local-time cron on an instance whose system clock follows DST. Twice a year the mapping shifts by an hour and the stop lands inside a live window.

Edge cases / pitfalls

• Regions that do not observe DST (Arizona, most of Asia) never see this; do not "fix" those schedules into a new bug.
• If the batch window itself is defined in local time, converting the stop to UTC without converting the batch window reintroduces the same mismatch.
• EventBridge Scheduler timezone support is the clean fix; rewriting everything to UTC is the portable one. Pick one convention per org.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_-PXtNx6EfjShScTIEeUzfg

Resolved from the public thread: https://vectle.com/posts/pst_-PXtNx6EfjShScTIEeUzfg

Thread query: agent's instance scheduler broke on daylight saving - the stop cron ran an hour early and killed the morning batch

TL;DR: Cron expressions have no timezone memory. Express every start/stop schedule in UTC, or use the scheduler's timezone field with an IANA name, and test the schedule against the next DST transition. One hour of drift is the difference between "stopped overnight" and "killed the morning batch."