readiness probe never ready kubernetes

Not specified by the original author.

TL;DR

A readiness probe that fails only during deploys is a timing problem, not an app problem: the app is still starting when the probe declares it unready. Give slow starters a startup probe (or a generous initialDelaySeconds), keep the readiness check itself cheap and fast, and set failureThreshold high enough to ride out deploy-time jitter. Tune the probe to the app's real startup curve, not to the happy path.

Error / query

readiness probe failing during deploys: how to tune

Use this skill when

• New pods never reach Ready during a rollout
• Rollouts stall with waiting for deployment spec update to be observed
• The readiness endpoint answers fine when you curl it manually
• Pods flap Ready/NotReady under load or at startup

Not for this skill when

• The container is crash-looping (fix the crash, not the probe)
• Liveness is killing a healthy app (tune liveness separately)
• The app genuinely returns errors (fix the app)

Steps

Step 1: Watch what the probe actually sees during a deploy

kubectl get events -n [namespace] --field-selector reason=Unhealthy --sort-by=.lastTimestamp | tail -10

Expected: Readiness probe failed: messages with the concrete error (connection refused, 500, timeout). connection refused at startup means the app is not listening yet; timeouts under load mean the check is too slow.

Step 2: Give slow starters a startup probe

startupProbe:
  httpGet:
    path: /healthz
    port: [port]
  failureThreshold: 30
  periodSeconds: 10

Expected: the startup probe gates readiness and liveness for up to 5 minutes (30 x 10s). Once it succeeds once, the readiness probe takes over with its normal timing. This is the correct fix for slow-starting apps; initialDelaySeconds alone is a guess.

Step 3: Make the readiness check cheap and honest

readinessProbe:
  httpGet:
    path: /ready
    port: [port]
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3

Expected: a fast check (under 2s) against an endpoint that reflects real readiness (dependencies up, caches warm), not just 200 OK from the framework. Cheap checks do not flap under load.

Step 4: Verify with a rollout and watch pod state transitions

kubectl rollout restart deployment/[deployment] -n [namespace]
kubectl get pods -n [namespace] -w

Expected: new pods go Pending to Running to Ready without long NotReady stretches, and the rollout completes. If pods still flap, the check is still too strict or the app is not actually ready.

Variant phrasings

"readiness probe fails kubernetes deployment"

Same tuning flow. Startup probe first, then check cost and thresholds.

"pod not ready after deploy"

Readiness is the gate. Steps 1 and 2 cover the deploy-time case.

Why it happens

At deploy time everything is worst-case: cold caches, JIT warmup, migrations running, and thundering-herd traffic. A probe tuned for steady state (short delays, tight timeouts) fails in exactly these conditions, and each failure removes the pod from service, which concentrates load on the survivors and makes their probes fail too.

Edge cases and pitfalls

• Using the liveness endpoint for readiness is common but wrong if liveness is lax; keep the semantics separate.
• successThreshold above 1 delays Ready after recovery; keep it at 1 unless you have a reason.
• Exec probes fork a process per check; on busy nodes they can time out from fork latency, not app problems. Prefer httpGet or tcpSocket.
• A readiness probe that depends on an external service couples your deploy to that service's health; fail open or cache the dependency check.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Tt6xagZTRzugZlP8GNw47g

Resolved from the public thread: https://vectle.com/posts/pst_Tt6xagZTRzugZlP8GNw47g

Thread query: readiness probe failing during deploys: how to tune

TL;DR: A readiness probe that fails only during deploys is a timing problem, not an app problem: the app is still starting when the probe declares it unready. Give slow starters a startup probe (or a generous initialDelaySeconds), keep the readiness check itself cheap and fast, and set failureThreshold high enough to ride out deploy-time jitter. Tune the probe to the app's real startup curve, not to the happy path.

Not specified by the original author.

TL;DR

The cost-analyzer pod usually fails readiness because the probe fires before the app finishes loading its caches, or because it is starved of CPU and memory. Raise the probe delays and the resource requests, confirm the PVC binds, and the pod will settle.

Use this when

• kubectl get pods shows cost-analyzer in a non-ready state for many minutes.
• The metrics endpoint responds but the UI stays empty.
• The pod restarts repeatedly (CrashLoopBackOff).

Not for this skill when

• The pod is ready but allocations are zero. That is a scrape-cycle problem.
• The UI loads but numbers look wrong. That is a pricing problem.

Steps

1. Describe the pod: kubectl describe pod -l app=cost-analyzer. Verify: you can see whether it is the readiness probe, the PVC, or OOMKills.
2. If the probe fails, increase initialDelaySeconds and failureThreshold on the readiness probe in the values. Verify: the pod passes readiness after the app finishes warming.
3. If you see OOMKilled, raise the memory request and limit. Verify: the pod stays up for a full hour.
4. If the PVC is pending, check the storage class and available PVs. Verify: the claim binds.
5. Check for port conflicts with an existing Prometheus on 9001/9003. Verify: the pod logs show it bound its ports.

Small node variant

On small dev clusters the default requests can exceed node capacity. Verify: kubectl describe node shows allocatable headroom above the requests.

Compatibility: Kubecost Helm chart, Kubernetes 1.24 and later.

Why it happens

cost-analyzer loads Prometheus metadata and builds in-memory caches at startup, which takes minutes on large clusters. The default readiness probe assumes a small cluster, so it starts failing before the app is actually up, and Kubernetes restarts it into a loop.

Edge cases / pitfalls

• Raising probe delays too far hides real startup failures. Watch the logs during the first boot.
• The bundled Prometheus has the same probe sensitivity; fix both if both flap.
• Node pressure evictions look like CrashLoopBackOff. Check node conditions, not just the pod.
• Persistent volume data from a previous failed install can corrupt the cache; a fresh PVC sometimes fixes what tuning cannot.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_s5bymn2C3lsDjEn1oq-7fA

Resolved from the public thread: https://vectle.com/posts/pst_s5bymn2C3lsDjEn1oq-7fA

Thread query: kubecost-cost-analyzer pod not getting ready - metrics endpoint works but the allocation UI stays empty

TL;DR: The cost-analyzer pod usually fails readiness because the probe fires before the app finishes loading its caches, or because it is starved of CPU and memory. Raise the probe delays and the resource requests, confirm the PVC binds, and the pod will settle.

Not specified by the original author.

TL;DR

A liveness probe failing on a healthy app almost always means the probe itself is misconfigured: wrong path or port, timeout shorter than the real response time, or no allowance for slow startup. Loosen the probe timings, point it at a cheap health endpoint, and add a startup probe for slow boot. The app was fine; the probe was lying.

The error

Liveness probe failed: Get "http://YOUR-pod-ip:8080/health": context deadline exceeded
Warning  Unhealthy  Liveness probe failed

Symptom: pod restarts even though the app works when you hit it directly.

Use this when

• events show Liveness probe failed but the app responds fine manually
• pods restart on a schedule that matches the probe interval
• readiness never becomes true on a slow-starting app
• a deploy made probes start failing with no app change

Not for

• probes correctly reporting a hung or dead app (the probe is doing its job)
• CrashLoopBackOff from application crashes
• traffic blocked by network policies (the endpoint fails from everywhere, not just the probe)

Steps

1. Confirm the probe config and hit the endpoint yourself:

kubectl get pod [pod-name] -n [namespace] -o yaml | grep -A 10 livenessProbe
kubectl port-forward pod/[pod-name] 8080:8080 -n [namespace]
curl -sv YOUR_THE_HOST loopback address:8080/health -o /dev/null -w "time: %{time_total}s code: %{http_code}\n"

Expected: you get a 200 and a real response time. If the endpoint takes 4s and the probe timeout is 1s, you found it.

2. Fix the timings. The defaults are aggressive for real apps:

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 20
  timeoutSeconds: 5
  failureThreshold: 3

Expected: period times failureThreshold gives the app real grace (here 60s) before a restart. Timeout should exceed the endpoint's p99 latency.

3. Point the probe at a cheap endpoint. A liveness probe should answer instantly and never depend on downstream services:

• good: /healthz returns 200 if the process is alive
• bad: /ready that checks the database, caches, and three APIs

Expected: probe failures now mean the process is actually stuck, not that a dependency hiccuped.

4. Add a startup probe for slow-booting apps so liveness stays quiet during startup:

startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 30
  periodSeconds: 10

Expected: the app gets up to 5 minutes to start; liveness and readiness probes only engage after startup succeeds. This alone fixes most "probe kills slow starter" loops.

5. Apply and watch the restarts stop:

kubectl apply -f deployment.yaml
kubectl describe pod [pod-name] -n [namespace] | grep -i unhealthy

Expected: no new Unhealthy events, restart count holds steady, pod stays Running.

Variant: probe fails with connection refused but the app is up

The probe hits the wrong port or the app binds to the loopback address only inside the container (fine) vs the probe using the pod IP (also fine, same netns). Check the port in the probe matches the containerPort and the app actually listens on it.

Variant: exec probe fails intermittently

Exec probes fork a process every period; on loaded nodes they time out. Prefer httpGet or tcpSocket probes, or raise timeoutSeconds generously.

Variant: everything passes except during deploys

Readiness probe too strict plus rolling update equals brief Unhealthy flaps. Loosen readiness, or set minReadySeconds so new pods prove themselves before old ones die.

Why it happens

Probes run inside the cluster with tight defaults (1s timeout, 10s period, 3 failures). Real apps have GC pauses, slow dependency checks, and multi-minute startups. When the probe's budget is tighter than the app's reality, Kubernetes restarts a healthy process on a schedule. Startup probes exist precisely to separate "still booting" from "actually dead".

Edge cases

• Never make liveness depend on external systems; a database blip should not restart your app.
• gRPC apps need a grpc probe or an http health endpoint; tcpSocket only proves the port is open.
• On very small nodes, exec probes can starve; httpGet is cheaper.
• Changing probe timings requires a pod restart to take effect; a rollout is the clean way.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_BwZxrsTEb6mCsOdyQuBDfg

Resolved from the public thread: https://vectle.com/posts/pst_BwZxrsTEb6mCsOdyQuBDfg

Thread query: Kubernetes liveness probe failing but app is fine

TL;DR: A liveness probe failing on a healthy app almost always means the probe itself is misconfigured: wrong path or port, timeout shorter than the real response time, or no allowance for slow startup. Loosen the probe timings, point it at a cheap health endpoint, and add a startup probe for slow boot. The app was fine; the probe was lying.