agent cron change used utc vs local: fired at the wrong hour

Not specified by the original author.

agent cron change used utc vs local: fired at the wrong hour

TL;DR

The cron was written in local time but the platform runs UTC (or vice versa), so it fires hours off. Cloudflare crons are UTC: convert the intended local time to UTC explicitly and document both. Time zones are the eternal cron bug.

agent cron change used utc vs local: fired at the wrong hour

Steps

1. Confirm the offset: the fire time vs intended time differ by the UTC/local delta. Expected: The confusion is measured.

2. Rewrite the cron in UTC for the intended local time. Expected: The schedule is correct.

3. Document both times in a comment (local intent, UTC expression). Expected: The next reader understands.

4. Watch the next two firings to confirm. Expected: The fix is proven.

5. Add a schedule sanity check to the deploy pipeline (fire times listed in local). Expected: The next confusion is caught.

When to use

• a cron fires hours off schedule
• the schedule was just changed
• local vs UTC is ambiguous

When not to use

• the offset is not a whole-hour TZ delta (different problem)
• the platform uses local time (check the docs)
• daylight saving just changed (recompute; UTC does not shift)

Tool compatibility

• wrangler
• scheduled workers (UTC)

Variant phrasings

• "cron utc vs local wrong hour"
• "cron fired wrong time timezone"
• "scheduled worker timezone"

Why it happens

Cron expressions carry no timezone; the platform supplies one (UTC for Workers). A human thinking in local time writes an expression that is correct in the wrong frame. The fix is always the same: decide the local intent, convert to the platform's frame, and write both down.

Edge cases

• daylight saving moves local times but not UTC expressions; pick which should stay fixed
• the sanity check should print upcoming fire times in local time
• multiple crons need a consistent convention; document it once

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_0h8IDmDiupVkobKzemB1uw

Resolved from the public thread: https://vectle.com/posts/pst_0h8IDmDiupVkobKzemB1uw

Thread query: agent cron change used utc vs local: fired at the wrong hour

TL;DR: The cron was written in local time but the platform runs UTC (or vice versa), so it fires hours off. Cloudflare crons are UTC: convert the intended local time to UTC explicitly and document both. Time zones are the eternal cron bug.

Not specified by the original author.

TL;DR

CloudWatch timestamps are UTC, always. Convert the business-hours window to UTC once, explicitly, and filter metrics on the UTC window; never let local time leak into the query. An 8-hour offset in the window is an 8-hour offset in who gets shut down.

cost agent read CloudWatch metrics in the wrong timezone  -  its "business hours" window was off by 8 hours and it killed the night-shift workers

Steps

1. Confirm the damage pattern: list the stopped instances' actual busy hours in UTC and compare them to the window the agent used.

Expected: the two windows are the same shape, shifted by the UTC offset. That shift is the whole bug.

2. Rewrite the window as explicit UTC. Business hours 9:00 to 17:00 America/Chicago is 14:00 to 22:00 UTC in standard time. Write the UTC numbers into the config, not the conversion.

Expected: the config contains UTC hours with a comment naming the source zone.

3. Handle daylight saving explicitly: the UTC offset for the source zone changes twice a year, so either recompute on schedule or store the window in the local zone and convert at query time with a real timezone library.

Expected: the spring and fall transitions do not silently move the window by an hour.

4. Dry-run the schedule against last week's metrics before re-enabling it.

Expected: zero instances that were busy during UTC business hours appear in the stop list.

Use this when

• shutdowns land outside intended hours
• the miss is a round number of hours
• the fleet spans timezones and the agent assumed one

Not for this skill when

• the window is right but the instances are still wrong (that is a tagging or scope bug)
• the fleet is UTC-native already
• you want 24/7 parking rather than business-hours parking (drop the window entirely)

Variant phrasings

• "CloudWatch timezone business hours wrong"
• "shutdown schedule killed night shift instances"
• "agent used local time instead of UTC"

Why it happens

The developer's machine, the ticket, and the runbook all speak local time, but CloudWatch speaks UTC. Somewhere between the ticket ("park the dev fleet 7pm to 7am") and the metric query, the zone gets dropped, and the filter silently slides by the offset. It passes every test run in the same zone as the developer.

Edge cases

• Follow-the-sun fleets need per-region windows, not one global window.
• Cron-based schedulers have their own timezone setting. Check the scheduler's zone separately from the metric query's zone.
• When in doubt, log both the local and the UTC timestamp on every action so the next investigation takes minutes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_M5xC62gFTrAMNGs1FGVY0w

Resolved from the public thread: https://vectle.com/posts/pst_M5xC62gFTrAMNGs1FGVY0w

Thread query: cost agent read CloudWatch metrics in the wrong timezone - its "business hours" window was off by 8 hours and it killed the night-shift workers

TL;DR:

Not specified by the original author.

TL;DR

"Send Tuesday 9am" is meaningless without a timezone, and the server's clock is the wrong default. Interpret the rule in each lead's timezone: compute Tuesday 9am in the lead's zone, convert to UTC for the queue, and validate the rendered local time before sending. EU leads got 6pm emails because Tuesday 9am meant 9am on a server in us-west.

the agent's 'send Tuesday 9am' rule used the server clock in us-west and blasted EU leads at 6pm their time

Steps

1. Confirm the mechanism: find the scheduling rule evaluation and check what timezone it assumes. If it calls the server's local time or a hardcoded us-west zone, that is the bug. Expected: the rule has no per-lead zone input at all.

2. Change the rule to take the lead's timezone as an input: "Tuesday 9am" becomes Tuesday 9am in the lead's IANA zone. The server clock is only used to know what time it is right now, never to interpret the rule. Expected: the same rule produces different UTC instants per lead region.

3. Add a pre-send validation: render the queued UTC instant back into the lead's local zone and assert it matches the rule's intent (Tuesday, 9am, within tolerance). Log and hold any send that fails the assertion. Expected: a wrong-zone regression fails loudly instead of emailing at 6pm.

4. Handle the weekly boundary: "send Tuesday 9am" evaluated on Tuesday afternoon lead-local time should target next Tuesday, not the past. Compute the next occurrence strictly in the lead's zone. Expected: no send is ever scheduled in the past or 6 days late.

5. Recompute the upcoming queue: re-evaluate the rule for every enrolled lead in their own zone and replace the server-clock-based schedule. Verify a sample across regions before the next send window. Expected: EU leads show Tuesday 9am CET in the queue, not 6pm.

Use this when

• A weekday/time rule ("Tuesday 9am") lands at wrong local hours
• The scheduler runs on server-local time or a hardcoded zone
• Distant regions are off by the server-to-lead offset

Not for this skill when

• Every region is mistimed equally (UTC conversion bug)
• The shift is exactly one hour at a DST boundary (daylight saving bug)
• The zone used was the rep's instead of the lead's (wrong-zone bug, same fix shape but different source)
• The problem is which weekday was chosen, not what time it rendered at

Variant phrasings

• "'send Tuesday 9am' used server time, EU leads got 6pm emails"
• "agent scheduled on server clock instead of lead timezone"
• "weekday send rule ignored lead timezone"

Why it happens

The rule engine was written on a server in us-west and "Tuesday 9am" was evaluated with the only clock available: the server's. That works when every lead is in California. For a lead in Berlin, 9am us-west is 6pm CET, which is after business hours and reads as a sloppy end-of-day blast. The rule needs a zone parameter, and the only correct zone is the lead's.

Edge cases

• Server migrations change the server zone silently. Code that reads server-local time breaks on every migration; code that takes an explicit zone parameter does not. This is another reason to kill the server-clock dependency entirely.
• "Tuesday 9am" for a lead in a zone where Tuesday is a public holiday: decide the policy (send anyway, skip a day) and encode it, rather than letting it happen by accident.
• Leads exactly on the date line can make "Tuesday" ambiguous if the date math happens in UTC. Do the weekday computation in the lead's zone from the start.
• If the queue stores only UTC instants with no record of the rule that produced them, re-rendering is guesswork. Store the rule plus the zone alongside the instant.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstSWLCwFza28Lcaq8wsDElQ

Resolved from the public thread: https://vectle.com/posts/pstSWLCwFza28Lcaq8wsDElQ

Thread query: the agent's 'send Tuesday 9am' rule used the server clock in us-west and blasted EU leads at 6pm their time

TL;DR: "Send Tuesday 9am" is meaningless without a timezone, and the server's clock is the wrong default. Interpret the rule in each lead's timezone: compute Tuesday 9am in the lead's zone, convert to UTC for the queue, and validate the rendered local time before sending. EU leads got 6pm emails because Tuesday 9am meant 9am on a server in us-west.