kubecost-cost-analyzer pod not getting ready - metrics endpoint works but the allocation UI stays empty
Fixes a Kubecost install where the cost-analyzer pod never becomes ready. Covers readiness probe tuning, resource requests, PVC binding, and port conflicts. Use when the pod is stuck pending or failing probes. Not for allocation math issues.
TL;DR
The cost-analyzer pod usually fails readiness because the probe fires before the app finishes loading its caches, or because it is starved of CPU and memory. Raise the probe delays and the resource requests, confirm the PVC binds, and the pod will settle.
Use this when
kubectl get podsshows cost-analyzer in a non-ready state for many minutes.- The metrics endpoint responds but the UI stays empty.
- The pod restarts repeatedly (CrashLoopBackOff).
Not for this skill when
- The pod is ready but allocations are zero. That is a scrape-cycle problem.
- The UI loads but numbers look wrong. That is a pricing problem.
Steps
- Describe the pod:
kubectl describe pod -l app=cost-analyzer. Verify: you can see whether it is the readiness probe, the PVC, or OOMKills. - If the probe fails, increase
initialDelaySecondsandfailureThresholdon the readiness probe in the values. Verify: the pod passes readiness after the app finishes warming. - If you see OOMKilled, raise the memory request and limit. Verify: the pod stays up for a full hour.
- If the PVC is pending, check the storage class and available PVs. Verify: the claim binds.
- Check for port conflicts with an existing Prometheus on 9001/9003. Verify: the pod logs show it bound its ports.
Small node variant
On small dev clusters the default requests can exceed node capacity. Verify: kubectl describe node shows allocatable headroom above the requests.
Compatibility: Kubecost Helm chart, Kubernetes 1.24 and later.
Why it happens
cost-analyzer loads Prometheus metadata and builds in-memory caches at startup, which takes minutes on large clusters. The default readiness probe assumes a small cluster, so it starts failing before the app is actually up, and Kubernetes restarts it into a loop.
Edge cases / pitfalls
- Raising probe delays too far hides real startup failures. Watch the logs during the first boot.
- The bundled Prometheus has the same probe sensitivity; fix both if both flap.
- Node pressure evictions look like CrashLoopBackOff. Check node conditions, not just the pod.
- Persistent volume data from a previous failed install can corrupt the cache; a fresh PVC sometimes fixes what tuning cannot.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_s5bymn2C3lsDjEn1oq-7fA