agent timed out waiting for tape library inventory and the restore test failed

TL;DR

Stop waiting on a single blocking inventory call: query inventory status in a poll loop with a budget longer than a full library scan, accept a cached or partial inventory when the needed cartridge is already listed, and only request a fresh full inventory when the target tape is missing. Most libraries take minutes to scan a full magazine set, far longer than a default agent timeout.

agent timed out waiting for tape library inventory and the restore test failed

Use this when

• The restore test agent fails before restoring anything, while waiting for tape library inventory to return.
• Inventory completes when run manually or from the library console, but not inside the agents time limit.
• The needed cartridge is known by barcode but the agent waits for every slot to be scanned.

Not for this skill when

• The library reports a genuine robotic, door open, or offline fault. Fix the hardware state first.
• The agent times out during data transfer from tape, not during inventory. That is a throughput problem.

Steps

1. Reproduce with timing: start an inventory from the library interface, time how long a full scan takes with all magazines and drives present, and compare it with the agents configured request and step timeouts. Verify: full inventory duration and the agent timeout are both measured and the mismatch is explicit
2. Change the agent to start inventory as an asynchronous job and poll its status endpoint with backoff, instead of holding one request open. Persist the inventory job identifier so a restarted agent resumes polling instead of starting a second scan. Verify: the agent survives an inventory longer than its old single request timeout without restarting the scan
3. Add a fast path for known barcodes: check the last good cached inventory and the element status for the specific cartridge first, and proceed to load that tape if its slot and drive assignment are confirmed. Verify: a test with a known barcode skips the full inventory wait and reaches the load step
4. Handle partial and stale results explicitly: if inventory returns partial data, record which ranges are missing, retry only those, and mark any restore decision based on cached data older than your threshold. Verify: partial inventory is labelled partial in the run log and never treated as an empty library
5. Coordinate with the library: check for open doors, a running inventory or audit by another system, import or export operations, and mailslot activity that pause or restart scans, and make the agent wait for library ready state before starting its own inventory. Verify: the agent logs library ready before requesting inventory and does not compete with another audit
6. Set an overall test deadline separate from the inventory budget, with a clear failure reason that distinguishes inventory timeout from restore failure, so the backup itself is not marked bad. Verify: a forced slow inventory produces an inventory timeout result, not a verification failed result for the restore point

Why it happens

A tape library inventory is a physical process, not a database query. The robot must visit slots, read barcodes, and reconcile drives and mailslots, which takes minutes in a large library and restarts or pauses when doors open or another host requests an audit. Agent frameworks, in contrast, usually impose request timeouts of tens of seconds and step timeouts tuned for API calls, so the agent abandons or retries the call while the library is still working - and a retry can restart the scan, making the wait endless. Cached inventory complicates the picture: the library may answer quickly from cache that predates a tape import, so the agent concludes the cartridge is absent. The failure is therefore a mismatch between physical time, cache freshness, and agent timeout design, and the restore under test may be perfectly healthy.

Variants and edge cases

inventory returns but the target tape is missing

Check cache age, whether the tape was imported after the last inventory, and whether the barcode was read as a different value due to a label problem. Request a targeted inventory of that magazine or slot range rather than another full scan.

agent starts a second inventory while the first runs

Missing idempotency on the start call is the cause. Store the inventory job identifier in run state, poll it after any agent restart, and make the start operation return the existing job when one is already running.

timeout only on the first run of the day

Libraries often run their own audit after power events, door closures, or scheduled maintenance, so the first caller waits behind that work. Have the agent check for an in progress library audit and wait on it instead of issuing a competing request. Edge cases: some libraries report inventory complete per partition, so an agent reading a multi partition library can finish one partition and wrongly conclude a tape in another partition is absent; always scope inventory checks to the partition that owns the restore. Element addresses also shift after firmware updates or magazine changes, so match cartridges by barcode, not by remembered slot. Log inventory start, cache age, partitions covered, and completion time in every restore test run - when this failure returns, those four fields show immediately whether the agent waited too little, read stale data, or looked in the wrong partition.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_eaxxafOFz-m9G8WI8a1hoQ