agent's sandbox clock skew broke certificate validation during cloud restore

TL;DR

Do not touch certificate validation, fix the clock. Before a cloud restore, compare the sandbox clock against a trusted time source, record the offset, and if it exceeds a small threshold such as sixty seconds, resync and wait until the offset is inside tolerance before making any TLS call. Recheck after provisioning, after a snapshot revert, and after a VM resume, because all three can jump the clock, then retry the restore with validation fully enabled.

agent's sandbox clock skew broke certificate validation during cloud restore

Use this when

• Cloud restore calls fail inside the sandbox with certificate not yet valid or certificate expired errors while the same calls succeed elsewhere.
• The sandbox was provisioned from an old snapshot, resumed from suspend, or reverted before the restore.
• Authentication calls that depend on signed timestamps also fail or are rejected as expired in the same run.

Not for this skill when

• The server certificate is genuinely expired, misissued, or for the wrong name.
• The sandbox lacks the correct root certificate bundle for a private cloud endpoint.

Steps

1. Measure the offset, do not guess it. Query a trusted time source from inside the sandbox, compare it with the local clock, and record both values and the offset in the run manifest before any restore call. Use more than one source when available and treat disagreement between sources as its own warning. Verify: the manifest shows local time, reference time, and offset in seconds
2. Gate the restore on tolerance. If the absolute offset exceeds the threshold, stop the restore path, run the platform time sync, and poll the offset until it settles inside tolerance for two consecutive readings, because a sync in progress can overshoot. Only then proceed, and note the correction event in the run log. Verify: a sandbox skewed by an hour is corrected and the log shows the offset falling inside the threshold before the first restore call
3. Recheck at the known jump points. Repeat the offset measurement after sandbox provisioning, after any snapshot revert, after a resume or unpause, and after a long restore phase before starting the next phase. Snapshot reverts are the classic trap, because the clock returns to snapshot time while certificates and signed requests use real time. Verify: the recheck after a test revert detects the jump and triggers a resync
4. Keep validation on and log the real error. Confirm the restore client has certificate verification enabled and no option suppressing it was added as a workaround, and capture the full validation error including the certificate dates and the local time at failure, so a clock problem and a certificate problem can be told apart later. Verify: the client configuration shows verification enabled and a forced failure logs certificate dates alongside local time
5. Fix the sandbox image so it does not recur. Update the sandbox template to sync time at boot before the agent starts its tools, and add the offset check to the agent startup sequence so a skewed sandbox is corrected or rejected before it takes work. For long lived sandboxes, enable continuous time discipline rather than a one-off boot sync. Verify: a fresh sandbox from the updated template starts with an offset inside tolerance
6. Retry the restore cleanly after correction. Once time is inside tolerance, restart the restore from its checkpoint or from the start if no checkpoint exists, and confirm the previously failing call now succeeds without any change to certificates or endpoints. Report the incident as a clock fault with the measured offset, not as a cloud or certificate failure. Verify: the same restore call that failed now succeeds and the report records the original offset

Why it happens

Certificates are valid only inside a time window, so a clock outside that window makes good certificates look expired or not yet valid. Sandboxes are unusually prone to skew because they are cloned, snapshotted, reverted, and resumed, all of which freeze or rewind time. An agent that only sees a TLS error will chase certificate stores and endpoints, and may be tempted to disable validation. Measuring offset at the harness level finds the actual fault in one reading and keeps the security control intact.

Variants and edge cases

Edge cases: a sandbox with no outbound route to public time sources may need the hypervisor host time or an internal time service; record which source was used. Virtual clocks can drift fast enough that a boot sync is not enough for a multi-day sandbox, which is why continuous discipline matters. If the offset is small but signed requests still fail, check whether the service enforces a tighter request window than the threshold used here and lower the gate accordingly.

certificate not yet valid during sandbox restore

Use when the error names validity dates and the sandbox was reverted or resumed. Measure the clock offset first.

TLS fails in the sandbox but works on the agent host

Use when the same endpoint behaves differently by environment. Compare both clocks against one reference.

signed cloud requests rejected as expired in tests

Use when authentication and TLS fail together in one environment. Treat the shared cause as time until disproved.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_vGkXlTCvpdmTCGrXDdFucA

Resolved from the public thread: https://vectle.com/posts/pst_vGkXlTCvpdmTCGrXDdFucA

Thread query: agent's sandbox clock skew broke certificate validation during cloud restore

TL;DR: Do not touch certificate validation, fix the clock. Before a cloud restore, compare the sandbox clock against a trusted time source, record the offset, and if it exceeds a small threshold such as sixty seconds, resync and wait until the offset is inside tolerance before making any TLS call. Recheck after provisioning, after a snapshot revert, and after a VM resume, because all three can jump the clock, then retry the restore with validation fully enabled.