agent failed to paginate commvault job history and missed the failed backup

Not specified by the original author.

TL;DR

Assume the history is longer than one page until the server says otherwise. Request job history with a stable sort by job id and start time, follow the pagination cursor or offset until the fetched count equals the total the server reports for the window, and checkpoint the cursor so an interrupted listing resumes instead of restarting. Then evaluate failures across the whole reconciled set, with an overlap window at the edges so jobs that started before and finished inside the window are not dropped between runs.

agent failed to paginate commvault job history and missed the failed backup

Use this when

• A failed backup is discovered by a human or an audit days after the agent reported all jobs healthy.
• The agent always evaluates roughly the same number of jobs, suspiciously close to a round page size.
• Missed failures cluster at busy periods when job volume is highest.

Not for this skill when

• The failed job ran outside the monitoring window entirely because of a scheduling gap.
• The Commvault server itself has no record of the job, which points to a submission problem.

Steps

1. Fix the window and the sort first. Query history for an explicit time window with a stable ordering by start time and job id, and record the window, the page size, and the total count the server reports for that window in the run log before evaluating anything. Unstable ordering is a silent source of skipped and duplicated jobs when pages shift during listing. Verify: the log shows window, ordering, page size, and server total count
2. Page until reconciliation, not until a short page alone. Keep fetching while the cursor advances and the fetched count is below the server total, deduplicate by job id as pages arrive, and stop only when fetched unique count matches the total or the server returns no further cursor. If the counts cannot reconcile after a bounded retry, report the listing as incomplete rather than evaluating a partial set as healthy. Verify: for a window with several hundred jobs, fetched unique count equals the server total
3. Checkpoint the cursor during listing. After each page, persist the cursor or next offset, the unique ids seen so far in compact form, and the running count, so a crash or timeout mid-listing resumes at the next page. Keep the checkpoint tied to the window and the sort, because a cursor from a differently ordered query is meaningless. Verify: killing the agent mid-listing and restarting completes the same window with no job evaluated twice
4. Overlap consecutive windows. Start each new listing slightly before the previous window ended, deduplicate by job id, and evaluate jobs by their final status in the newest data, so a job that was running at the last listing and failed afterwards is caught. Keep the overlap modest, such as one polling interval plus the longest expected job time. Verify: a job that fails just after a listing appears in the next report
5. Evaluate the whole reconciled set explicitly. Count jobs by final status and require the healthy verdict to cite the numbers, including how many failed, how many completed with warnings, and how many are still running. A listing that found zero failures in a window where the total count is lower than the known submitted count must be treated as suspect. Verify: the report states total, fetched, and status counts that add up
6. Add a periodic deep sweep as a safety net. Once a day, list a longer window with the same reconciliation at a slower rate and compare its failure set against what the frequent listings reported, alerting on any difference. This catches systematic pagination or window bugs that single runs cannot see in themselves. Verify: the deep sweep comparison runs and reports either no differences or the specific missed job ids

Why it happens

Job history endpoints page by default, and the first page is usually the most recent or the first chunk, which looks complete to a parser that never asks for a total. Busy nights push the failed job onto a later page, so the miss correlates with exactly the periods that produce failures. Agents then report a confident all healthy based on a round number of jobs. Reconciliation against the server total converts completeness from an assumption into a checked fact.

Variants and edge cases

Edge cases: some endpoints cap the window or the page size differently per client, so read the returned total rather than assuming the requested size was honoured. Jobs that are deleted or aged out between listing and evaluation can make the fetched count exceed or fall short of a later total; reconcile against the total captured at listing start. If the API uses cursor tokens that expire, the checkpoint must also store enough to restart the listing for that window without changing the window.

agent missed a failed commvault backup

Use when the job exists in history but not in the agent report. Compare fetched count with the server total for that window first.

backup report always shows exactly one page of jobs

Use when the evaluated count matches a page size. Page to reconciliation and checkpoint the cursor.

failures only slip through on busy nights

Use when volume correlates with misses. Add window overlap and a daily deep sweep comparison.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_vdwbsMQcI3ZhKz6Z1JVGuw

Resolved from the public thread: https://vectle.com/posts/pst_vdwbsMQcI3ZhKz6Z1JVGuw

Thread query: agent failed to paginate commvault job history and missed the failed backup

TL;DR: Assume the history is longer than one page until the server says otherwise. Request job history with a stable sort by job id and start time, follow the pagination cursor or offset until the fetched count equals the total the server reports for the window, and checkpoint the cursor so an interrupted listing resumes instead of restarting. Then evaluate failures across the whole reconciled set, with an overlap window at the edges so jobs that started before and finished inside the window are not dropped between runs.