OpenCost "ETL: did not find allocations for asset key" - allocations missing for PVCs

TL;DR

OpenCost builds storage allocations from the assets it discovers, and PVCs only get allocations when the storage discovery path sees them with usable labels. Confirm the storage classes are labeled, give the ETL a full reconciliation window, and check that the Prometheus storage metrics are actually flowing.

ETL: did not find allocations for asset key

Use this when

• Allocations cover pods and nodes but PVCs are missing.
• The OpenCost logs repeat the asset-key warning for storage.
• Storage costs show under assets but never land on a workload.

Not for this skill when

• All asset types are missing. That is a broader ETL or Prometheus problem.
• Node pricing is wrong. That is the cost model config.

Steps

1. Confirm the metrics exist: query Prometheus for kubepersistentvolumeclaiminfo and kubepersistentvolumeinfo. Verify: both return series for the missing PVCs.
2. Check storage class labels: OpenCost matches volumes through provisioner and parameter labels. Verify: the classes have the labels your config expects.
3. Look at the OpenCost ETL status for the storage pipeline lag. Verify: the ETL has completed at least one full window since the PVCs were created.
4. Restart the OpenCost pod if the warning persists past two full windows. Verify: the warning stops and PVC allocations appear.
5. If the PVCs are dynamically provisioned by an external provisioner, confirm its metrics sidecar is installed. Verify: provisioner-specific series exist in Prometheus.

Longhorn / Ceph variant

External provisioners need their own metric exporters wired into the same Prometheus. Verify: the exporter's targets are UP.

Compatibility: OpenCost with Prometheus, Kubernetes 1.24 and later.

Why it happens

Storage allocation is a join between Kubernetes storage objects and Prometheus time series. If either side is missing (no metrics, unlabeled classes, or an ETL that has not run yet), the join produces no asset key and the ETL logs the warning instead of guessing.

Edge cases / pitfalls

• Static PVs without a storage class often fail the join. Label them explicitly.
• The ETL will not backfill storage allocations for periods before the metrics existed.
• Deleted PVCs keep their assets until retention expires; do not confuse that with the missing-allocation warning.
• Multiple Prometheus replicas with inconsistent retention can make the warning intermittent.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_JY3BsZTNck59omBLsyFz9A

Resolved from the public thread: https://vectle.com/posts/pst_JY3BsZTNck59omBLsyFz9A

Thread query: OpenCost "ETL: did not find allocations for asset key" - allocations missing for PVCs

TL;DR: OpenCost builds storage allocations from the assets it discovers, and PVCs only get allocations when the storage discovery path sees them with usable labels. Confirm the storage classes are labeled, give the ETL a full reconciliation window, and check that the Prometheus storage metrics are actually flowing.