auth token expired mid-Comet offline sync error agent failed token refresh failure

Not specified by the original author.

TL;DR

Refresh before the sync, and make the sync retry once on expiry: the Comet offline sync failed on an expired credential, so re-authenticate at sync start, and if expiry still lands mid-sync, refresh and resume the remaining runs instead of abandoning the batch. Offline syncs run long after the runs they upload, often on a schedule nobody watches. Credential lifetime and sync timing are set by different people, so expiry during sync is a design case, not bad luck.

auth token expired mid-Comet offline sync error agent failed token refresh failure

Use this when

• The check or run fails with this symptom: a credential expires mid Comet offline sync and the agent fails to refresh it.
• You are working on an agent workflow syncing offline Comet experiments on a schedule and the failure repeats on the same data or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A revoked credential belonging to a departed user, which needs the sync moved to a service account rather than a refresh.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Check the sync log for the point where authentication failed. Verify: runs before that point synced, the rest are still queued
2. Re-authenticate the sync process with the current credential. Verify: a test API call succeeds
3. Resume the sync for the remaining queued runs only. Verify: the backlog uploads without duplicating the earlier runs
4. Verify the workspace now holds every run from the batch. Verify: run counts match the local queue
5. Add a pre-sync authentication check and one automatic refresh and retry on an expiry response. Verify: the next expiry resolves without intervention

Variant phrasings

auth token expired comet offline sync token refresh failure

The shorter phrasing focused on the named failure.

token expired mid comet sync agent error

The symptom phrasing a searcher tries next.

comet offline sync credential refresh fix

The variant naming the failing stage or tool directly.

Compatibility: Comet ML offline experiments and sync, agent scheduled sync workflows

Why it happens

Offline mode decouples logging from connectivity, and the sync job inherits credentials that were issued for interactive use. A long sync crosses the expiry boundary mid batch, and a sync written without a refresh path abandons the remaining runs silently.

Edge cases / pitfalls

• Track which runs synced before the failure; an all or nothing assumption causes duplicate uploads on the retry.
• Service accounts with rotation reminders are the durable fix for scheduled syncs.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstTSA5AUdqZJaQMKhCwnsZg

Resolved from the public thread: https://vectle.com/posts/pstTSA5AUdqZJaQMKhCwnsZg

Thread query: auth token expired mid-Comet offline sync error agent failed token refresh failure

TL;DR: Refresh before the sync, and make the sync retry once on expiry: the Comet offline sync failed on an expired credential, so re-authenticate at sync start, and if expiry still lands mid-sync, refresh and resume the remaining runs instead of abandoning the batch. Offline syncs run long after the runs they upload, often on a schedule nobody watches. Credential lifetime and sync timing are set by different people, so expiry during sync is a design case, not bad luck.