agent hallucinated an RMAN catalog entry and reported a database backup

Not specified by the original author.

TL;DR

Verify the catalogue entry against the backup storage before reporting the backup exists: the agent hallucinated an entry because it generated the report row from the schedule's expectation instead of the catalog's content. Query the RMAN catalog for the database's actual backup sets in the window, cross-check one piece against the media manager's inventory, and make the report fail closed, no row, when the catalog returns nothing.

agent hallucinated an RMAN catalog entry and reported a database backup that was never taken

Use this when

• The job or restore fails with this symptom: agent hallucinated an RMAN catalog entry and reported a database backup that was never taken.
• You are working on an agent reporting Oracle database backup status from RMAN data and the failure repeats on the same target or schedule.
• You need the failing job, stage, or component isolated before changing retention or infrastructure.

Not for this skill when

• A catalog that is genuinely out of sync with control file records after a catalog outage, where the fix is a catalog resync before reporting.
• The error text names a different product, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Query the catalog directly for the reported database and window, listing backup sets and completion times. Verify: the catalog's actual contents are known
2. Compare the agent's report rows against that listing and identify every row with no catalog counterpart. Verify: fabricated rows are enumerated
3. Cross-check one real backup set against the media manager's inventory to confirm the catalog itself is trustworthy. Verify: catalog-to-media agreement is sampled
4. Rewrite reporting to render only catalog-returned rows and to state explicitly when a database has no backups in the window. Verify: absence is reported as absence

Variant phrasings

agent hallucinated rman catalog entry backup never taken

The verbatim phrasing.

agent reported oracle backup not in catalog verify fix

The verify phrasing.

rman backup report fabricated entry catalog query agent

The catalog phrasing.

Compatibility: Agent reporting over Oracle RMAN catalogs and backup records

Why it happens

Backup schedules create strong expectations, and a reporter that fills its table from the schedule plus plausibility will happily certify backups that never ran. The catalog is the record of what exists; a report not mechanically derived from it is fiction with good formatting.

Edge cases / pitfalls

• Cross-check by control file records when the catalog and the report disagree; catalog sync issues cut both ways.
• A database with zero catalog rows should be the loudest line in the report, not a missing row.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_QOgVEnBRBnQ5XhgScuVO6g

Resolved from the public thread: https://vectle.com/posts/pst_QOgVEnBRBnQ5XhgScuVO6g

Thread query: agent hallucinated an RMAN catalog entry and reported a database backup that was never taken

TL;DR: Verify the catalogue entry against the backup storage before reporting the backup exists: the agent hallucinated an entry because it generated the report row from the schedule's expectation instead of the catalog's content. Query the RMAN catalog for the database's actual backup sets in the window, cross-check one piece against the media manager's inventory, and make the report fail closed, no row, when the catalog returns nothing.

Not specified by the original author.

TL;DR

Find what is growing inside the catalog before deleting anything: file level indexes, per job history, audit records, and orphaned entries for expired backups are the usual causes. Prune or disable the specific index that is exploding, expire orphaned catalog data with the product tool, and resize the catalog volume from measured daily growth.

backup catalog grows too fast database runs out of space

Use this when

• Catalog database or its volume grows daily and now threatens backup operations.
• Catalog growth is much faster than the growth of the backup data it describes.
• Jobs slow down as catalog queries, browsing, and reporting get heavier.

Not for this skill when

• The volume is full of backup data files, not catalog or index data. Manage repository retention instead.
• The catalog is corrupt and jobs cannot open it. Use the catalog rebuild skill, not pruning.

Steps

1. Measure the catalog by component: main database tables, file level or granular restore indexes, job history and logs, thumbnails or content indexes, and temporary files on the same volume. Verify: the largest component is named with its size and daily growth rate
2. Match growth to a change: a new file level indexing option, a new granular restore feature, shorter job intervals, many small jobs, or a retention change that kept catalog records after data expired. Verify: the growth start date aligns with one configuration or workload change
3. Check for orphans: catalog entries for deleted jobs, expired restore points, removed clients, and failed index builds that were never cleaned. Run the product consistency or catalogue cleanup report in read only mode first. Verify: orphan volume is quantified before any cleanup runs
4. Apply the targeted fix for the driver: reduce file indexing depth, index only selected folders, shorten job history retention, or disable indexing for workloads that never use granular restore. Verify: the next backup cycle shows lower catalog growth per job
5. Run the vendor supported prune, vacuum, or compact operation during a window with no running jobs, after taking a catalog backup, and confirm free space is actually returned to the volume. Verify: catalog size and volume free space both improve and jobs still browse old restore points
6. Resize and monitor properly: size the catalog volume from measured growth for your full retention period, move indexes to their own volume where supported, and alert on growth rate. Verify: a forecast shows the catalog fits for the whole retention cycle with headroom

Why it happens

A catalog stores metadata for every decision the backup makes, and some metadata scales with files, not with gigabytes. File level indexing records an entry per file per restore point, so a file server with ten million files backed up daily creates ten million catalog rows a day even if almost nothing changed. Granular indexes for mailboxes, database objects, and virtual machine files multiply that further. History and audit retention often default to far longer than backup retention, and when backup data expires the catalog rows do not always expire with it, leaving orphans that still cost space and query time. Database files also grow in high water marks: deleting rows frees pages inside the file but not space on the volume until a compact or vacuum runs. Growth therefore accelerates quietly until the volume fills and every job that needs the catalog stops at once.

Variants and edge cases

growth started after enabling granular restore

Mailbox, item level, or file level indexing is the price of fast granular restore. Keep it only for workloads that use it, reduce indexed scope to critical folders or mailboxes, and put the index store on its own sized volume.

catalog fine, volume still full

Look outside the database file: index caches, extracted restore indexes, job logs, and temporary staging often share the volume and are counted as catalog growth by operators. Clean with the product tools and check log retention settings.

prune runs but size never shrinks

Many catalog databases need an offline compact, vacuum, or reindex to return space, and some need free working space equal to a large fraction of the file to do it. Plan the compact with temporary space and a catalog backup first, never by truncating files by hand. Edge cases: never delete catalog tables or index files directly from the file system to free space in a hurry; an inconsistent catalog can make healthy backups unrestorable through the console. In clustered or replicated catalog setups, prune on the primary and confirm the replica applies the same cleanup, or the replica will reintroduce the bloat on failover. Record catalog bytes per restore point and per indexed file count weekly: those two ratios predict the next fill months ahead and show immediately when a new indexing feature has changed the economics of a job.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_6-sCq1MA88FCIezKulvbfg

Resolved from the public thread: https://vectle.com/posts/pst_6-sCq1MA88FCIezKulvbfg

Thread query: backup catalog grows too fast database runs out of space

TL;DR: Find what is growing inside the catalog before deleting anything: file level indexes, per job history, audit records, and orphaned entries for expired backups are the usual causes. Prune or disable the specific index that is exploding, expire orphaned catalog data with the product tool, and resize the catalog volume from measured daily growth.

Not specified by the original author.

TL;DR

Coverage must be computed from the backup system's own enrolment, not from an inventory list that ages independently: the agent reported decommissioned servers as covered because the stale list still named them and nobody reconciled it against reality. Reconcile three lists, the estate inventory, the backup enrolment, and the report's claims, flagging servers in any list but missing from another, and refresh the inventory source or its owner as part of the fix.

agent reported backup coverage for servers that had been decommissioned months earlier from a stale inventory list

Use this when

• The job or restore fails with this symptom: agent reported backup coverage for servers decommissioned months earlier from a stale inventory list.
• You are working on an agent reporting backup coverage across a server estate and the failure repeats on the same target or schedule.
• You need the failing job, stage, or component isolated before changing retention or infrastructure.

Not for this skill when

• Servers intentionally excluded from backup, such as ephemeral build nodes, which belong on an explicit exclusion list rather than being reconciled away.
• The error text names a different product, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Pull the backup system's enrolled clients and the agent's inventory list into one comparison. Verify: the two lists' differences are enumerated
2. Classify each differing server: decommissioned, renamed, excluded by policy, or genuinely unenrolled and at risk. Verify: every difference has an explanation
3. Remove decommissioned servers from the reporting inventory and record the authoritative inventory source and its owner. Verify: the inventory has a named source of truth
4. Add the genuinely unenrolled servers to enrolment or to the risk report, not silently to the covered set. Verify: real gaps become visible and owned
5. Re-issue the coverage report with the reconciliation summary attached. Verify: the corrected coverage is published with its method

Variant phrasings

agent reported coverage decommissioned servers stale inventory

The verbatim phrasing.

backup coverage report reconcile inventory enrolment agent fix

The reconciliation phrasing.

agent coverage ghosts decommissioned servers backup report wrong

The ghosts phrasing.

Compatibility: Agent backup coverage reporting from configuration management inventories

Why it happens

Inventories decay the day they are written, while backup enrolment changes with every provisioning event. A coverage report computed across two unreconciled lists certifies ghosts and, worse, can hide real servers missing from both lists, which no list comparison alone will catch without an estate source.

Edge cases / pitfalls

• Renamed servers look like one ghost plus one gap; match on stable identifiers, not names, where the estate provides them.
• Schedule the reconciliation; a one-off cleanup decays back within a quarter.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Tv6cd3sS5E5LcRYLE-Li6g

Resolved from the public thread: https://vectle.com/posts/pst_Tv6cd3sS5E5LcRYLE-Li6g

Thread query: agent reported backup coverage for servers that had been decommissioned months earlier from a stale inventory list

TL;DR: Coverage must be computed from the backup system's own enrolment, not from an inventory list that ages independently: the agent reported decommissioned servers as covered because the stale list still named them and nobody reconciled it against reality. Reconcile three lists, the estate inventory, the backup enrolment, and the report's claims, flagging servers in any list but missing from another, and refresh the inventory source or its owner as part of the fix.