glacier restore stuck in progress for days

Not specified by the original author.

TL;DR

Glacier retrievals legitimately take hours, and bulk tier can take days, but stuck for days beyond the tier promise means the job expired, was never properly initiated, or the restored copy temporary window lapsed before use. Check the job status and retrieval tier via the API or console, confirm the restored copy availability window, and re-initiate with standard or expedited tier if the original job is gone.

glacier restore stuck in progress for days

Use this when

• A restore from Glacier or S3 Glacier Deep Archive shows in progress far beyond its tier time.
• The restored object is still not downloadable after the expected window.
• Inventory reports lag behind reality for archive storage classes.

Not for this skill when

• The object was never archived; it sits in standard storage with a different problem.
• Restore initiation itself returns errors. That is a request problem, not a stuck job.

Steps

1. Query the restore job or object restore status directly and record the tier and initiation time. Verify: actual job state and tier are known
2. Compare elapsed time with the promised window for that tier, noting bulk tier can legitimately run to days. Verify: whether the job is truly overdue is decided by tier
3. Check whether a previous restore completed and its temporary copy expired before anyone downloaded it. Verify: expiry is ruled in or out
4. If the job is absent or expired, re-initiate the restore at standard tier or expedited for urgent data. Verify: a new job shows in progress with a fresh timestamp
5. When the restore completes, copy the object to standard storage permanently if it will be needed again soon. Verify: a durable copy exists outside the temporary restore window

Variant phrasings

glacier retrieval stuck days bulk tier

The tier phrasing.

s3 glacier restore in progress not completing

The status variant.

deep archive restore expired temporary copy

The expiry wording.

Compatibility: Amazon S3 Glacier Flexible Retrieval and Deep Archive storage classes, vault-based Glacier legacy archives, expedited, standard and bulk tiers.

Why it happens

Archive tiers trade retrieval latency for cost, and their status reporting is asynchronous and inventory-delayed, so consoles can look stuck when a bulk job is merely slow. The genuine stuck cases are expired temporary copies, jobs initiated against vault archives with different tooling, and restores nobody monitored to completion. Tier-aware status checks separate patience from action.

Edge cases / pitfalls

• Set a calendar reminder for the restore completion window. Expired restores are the most common self-inflicted delay.
• Expedited tier has capacity limits and can be throttled. Standard is the reliable urgent choice.
• Restoring a whole archive to fetch one file wastes days. Use range or selective restore options where supported.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstTcHuBLymE-5rEQHQQi0uA

Resolved from the public thread: https://vectle.com/posts/pstTcHuBLymE-5rEQHQQi0uA

Thread query: glacier restore stuck in progress for days

TL;DR: Glacier retrievals legitimately take hours, and bulk tier can take days, but stuck for days beyond the tier promise means the job expired, was never properly initiated, or the restored copy temporary window lapsed before use. Check the job status and retrieval tier via the API or console, confirm the restored copy availability window, and re-initiate with standard or expedited tier if the original job is gone.

Not specified by the original author.

TL;DR

Stop waiting on an inventory as if it were a quick lookup. Vault inventories are asynchronous jobs that can take hours, so initiate the inventory well before the restore test window, persist its job id in the run store, and poll it on a slow schedule with the test marked as waiting rather than failed. Keep and reuse the last completed inventory while a fresh one runs, select the archive from that inventory, and only start the retrieval clock once the archive identity is confirmed.

agent timed out waiting for glacier vault inventory and the restore test failed

Use this when

• A restore test from an archive tier fails with a timeout while waiting for the vault inventory to be ready.
• The agent re-requests an inventory on every retry, resetting or duplicating the wait.
• Archive restores are reported unreliable even though retrievals succeed when run manually over a longer window.

Not for this skill when

• The archive object itself is missing from a completed inventory.
• A standard or instant retrieval tier object is slow for a storage performance reason.

Steps

1. Start the inventory before the test window. When a restore test is scheduled against an archive vault, initiate the inventory job at scheduling time or at the previous run close, not at test start, and store the job id, vault name, and initiation time in the run record. If an inventory job is already in progress for that vault, attach to it instead of starting another. Verify: at test start the run record already holds an inventory job id or a completed inventory
2. Poll slowly with a persisted state machine. Model the wait as explicit states, inventory requested, inventory ready, retrieval requested, retrieval ready, restoring, with each state and its entry time persisted. Poll at intervals measured in tens of minutes with jitter, update the state on each poll, and set the overall deadline for the inventory phase in hours, aligned to the service behaviour for the vault size. Verify: a run can be inspected mid-wait and shows its state, last poll, and next poll time
3. Never mark waiting as failed. While the inventory is pending inside its deadline, the test status is waiting, the orchestrator heartbeat keeps running from the polling loop, and no failure alert fires. Only a job reported failed by the service, or a deadline genuinely exceeded, converts the test to failed, and that report must include how long it waited. Verify: a test whose inventory takes three hours shows waiting throughout and no failure notification
4. Reuse the last good inventory. Cache the most recent completed inventory with its timestamp and use it to select archives for tests when a fresh inventory is still running, flagging selections as based on an inventory of that age. Archives are immutable once written, so a recent inventory is a sound selection source for restore testing. Verify: a test can proceed to retrieval using the cached inventory while a refresh runs in the background
5. Confirm the archive before starting retrieval. Once the inventory is ready, match the wanted archive by id, size, and creation time against the backup catalog record, and stop with a clear catalog mismatch if they differ, rather than retrieving the wrong archive slowly. Record the matched archive id in the checkpoint. Verify: the retrieval request references the archive id taken from the inventory, not only a description string
6. Budget the whole test realistically and report the phases. Publish expected durations per phase when the test is scheduled, track actual time per phase, and alert on phase overruns rather than one opaque total timeout. Feed the observed inventory durations back into scheduling so archive tests get windows that can actually contain them. Verify: the final report breaks out inventory, retrieval, and restore times separately

Why it happens

Archive tiers trade speed for cost, and their inventory is a periodic snapshot produced by a background job, not a live listing. Agent harnesses built around fast object storage assume listing is instant, apply a normal request timeout, and then retry in ways that multiply inventory jobs. The failure is a mismatch of time models. Persisting the job and polling it on its own timescale aligns the harness with how the archive service actually works.

Variants and edge cases

Edge cases: very large vaults can take far longer than small ones, so a single deadline will misfire at one end; scale the deadline by vault size or archive count from observed history. An inventory is a snapshot, so an archive written after it started will not appear; tests for very recent backups must wait for the next inventory or use a different tier copy. If the service expires inventory output after a window, download and store it promptly once ready.

glacier restore test timed out on inventory

Use when the wait phase is the failure. Convert the wait into a persisted polling state before lengthening any timeout.

agent requested a new vault inventory on every retry

Use when retries reset progress. Attach to the existing job id stored in the run record.

archive restore works manually but fails in automated tests

Use when the manual run simply waited longer. Schedule the inventory at test scheduling time, not at test start.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2v4TmEKtdVZ33BxwIIjQRQ

Resolved from the public thread: https://vectle.com/posts/pst_2v4TmEKtdVZ33BxwIIjQRQ

Thread query: agent timed out waiting for glacier vault inventory and the restore test failed

TL;DR: Stop waiting on an inventory as if it were a quick lookup. Vault inventories are asynchronous jobs that can take hours, so initiate the inventory well before the restore test window, persist its job id in the run store, and poll it on a slow schedule with the test marked as waiting rather than failed. Keep and reuse the last completed inventory while a fresh one runs, select the archive from that inventory, and only start the retrieval clock once the archive identity is confirmed.

Not specified by the original author.

TL;DR

The mount had already failed and the agent polled its stale in-progress view for a day: mount operations have terminal failure states the poller must read from the task's own status, not infer from the mount's absence. Fetch the operation's status directly, map its failure states to an immediate report, and add a poll budget after which the agent escalates with the last status instead of continuing to call the operation pending.

agent timed out polling a Rubrik mount that had already failed and reported the restore as still in progress for a day

Use this when

• The job or restore fails with this symptom: agent timed out polling a Rubrik mount that had already failed and reported the restore as still in progress for a day.
• You are working on an agent waiting on Rubrik live mount operations during restore tests and the failure repeats on the same target or schedule.
• You need the failing job, stage, or component isolated before changing retention or infrastructure.

Not for this skill when

• A mount that is genuinely long-running on a huge dataset, where the fix is a budget matched to the operation class rather than a short one.
• The error text names a different product, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Retrieve the mount operation's current status and history directly and find when it actually failed. Verify: the true timeline is established
2. Identify why the poller missed it: cached response, a status field it never checks, or polling the wrong object. Verify: the blind spot is named
3. Rewrite polling to read the operation's status each cycle and exit on any terminal state, success or failure. Verify: terminal states end polling immediately
4. Add a maximum poll duration with escalation that reports the last observed status verbatim. Verify: a stuck operation escalates with evidence
5. Re-run a mount test, including a deliberately failing one, and confirm failure is reported within one cycle. Verify: both outcomes report promptly

Variant phrasings

agent polling rubrik mount already failed reported in progress day

The verbatim phrasing.

agent mount poll terminal state missed failure budget fix

The terminal-state phrasing.

rubrik live mount agent timeout stale status operation failed

The stale-status phrasing.

Compatibility: Agent polling of Rubrik live mount operations with terminal state handling

Why it happens

Long operations fail silently in the gap between the operation's own state and whatever proxy the poller watches. A day of in-progress reports for a dead mount is worse than no report, because it actively reassures everyone watching the restore test programme.

Edge cases / pitfalls

• Escalations should carry the operation id so a human can open the exact task, not search by time.
• Apply the same pattern to every long operation the agent polls; the blind spot is a habit, not a one-off.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_yBnuyTfp8Boaj2w7sYb2AQ

Resolved from the public thread: https://vectle.com/posts/pst_yBnuyTfp8Boaj2w7sYb2AQ

Thread query: agent timed out polling a Rubrik mount that had already failed and reported the restore as still in progress for a day

TL;DR: The mount had already failed and the agent polled its stale in-progress view for a day: mount operations have terminal failure states the poller must read from the task's own status, not infer from the mount's absence. Fetch the operation's status directly, map its failure states to an immediate report, and add a poll budget after which the agent escalates with the last status instead of continuing to call the operation pending.