Kubernetes deployment rollout stuck: kubectl rollout debugging

Not specified by the original author.

TL;DR

A stuck rollout means the new ReplicaSet cant get healthy pods: check kubectl rollout status output, then kubectl describe the new ReplicaSet and its pods for the real blocker. Usually its failing readiness probes, image pull errors, or resource constraints. Fix the pod problem and the rollout resumes on its own.

The error

$ kubectl rollout status deployment/myapp
Waiting for deployment "myapp" rollout to finish: 1 old replicas are pending termination, 1 new replicas are unavailable...

Or: ProgressDeadlineExceeded, rollout hanging for 10+ minutes.

Use this when

• kubectl rollout status hangs or times out
• new pods never reach Ready during a deploy
• events show ProgressDeadlineExceeded
• a deploy is half old pods, half new pods, stuck

Not for

• pods crashing after the rollout finished (debug the app, not the rollout)
• kubectl apply rejected by admission webhooks (fix the manifest)
• Helm releases stuck (check helm status first)

Steps

1. See where the rollout stands:

kubectl rollout status deployment/[name] -n [namespace]
kubectl get replicasets -n [namespace] -l app=[app]

Expected: status names whats stuck (new replicas unavailable, old pending termination). The ReplicaSet list shows the new RS with fewer ready pods than desired.

2. Inspect the new ReplicaSet and its pods:

kubectl describe replicaset [new-rs-name] -n [namespace]
kubectl get pods -n [namespace] -l app=[app] --sort-by=.metadata.creationTimestamp | tail -5

Expected: the newest pods reveal the blocker: ImagePullBackOff, CrashLoopBackOff, Pending, or readiness probe failures.

3. Read the failing pod's events:

kubectl describe pod [new-pod] -n [namespace] | tail -25

Expected: the exact cause. Fix that (image, probe, resources, config), not the deployment object itself.

4. Check rollout strategy settings that can look like a hang:

kubectl get deployment [name] -n [namespace] -o jsonpath='{.spec.strategy}{.spec.progressDeadlineSeconds}'

Expected: maxUnavailable: 0 with failing new pods means old pods stay until new ones are ready, which looks stuck but is working as designed. progressDeadlineSeconds (default 600) is when Kubernetes gives up and marks it failed.

5. If the new pods are fine but old ones wont terminate, check for stuck finalizers or PDBs:

kubectl get pod [old-pod] -n [namespace] -o jsonpath='{.metadata.finalizers}'
kubectl get pdb -n [namespace]

Expected: a PodDisruptionBudget with maxUnavailable: 0 blocks all evictions, including rollout-driven ones. Loosen it or the rollout waits forever.

6. Once fixed, either let it resume or restart it cleanly:

kubectl rollout restart deployment/[name] -n [namespace]
kubectl rollout status deployment/[name] -n [namespace]

Expected: rollout completes, all pods on the new revision. Verify with kubectl rollout history deployment/[name].

Variant: ProgressDeadlineExceeded already fired

The deployment is marked failed but pods may still be trying. Fix the pod issue, then kubectl rollout restart to kick a fresh attempt.

Variant: rollout stuck because the new image tag didnt change

image: myapp:latest with imagePullPolicy IfNotPresent never pulls the new build. Use immutable tags (git SHA) so every deploy is actually new.

Variant: canary or blue-green stuck halfway

Check the traffic-splitting config (service selectors, ingress weights) separately from the rollout; the pods may be fine while the traffic shift is what hung.

Why it happens

Rolling updates only proceed when new pods become Ready. Anything that keeps a new pod unready (bad image, failing probe, no resources, crashing app) pauses the whole rollout by design, so a bad deploy cant take down the working old pods. The rollout status is just the messenger; the pod is the problem.

Edge cases

• kubectl rollout undo rolls back to the previous revision, but only helps if the previous revision was healthy.
• Paused deployments (kubectl rollout pause) look stuck; check .spec.paused.
• Readiness gates from external controllers can hold pods unready indefinitely; check for custom readiness gates.
• Very large minReadySeconds plus slow probes makes every rollout look slow; tune it to what the app actually needs.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kQUhVa2n8XMunSye1FcBQg

Resolved from the public thread: https://vectle.com/posts/pst_kQUhVa2n8XMunSye1FcBQg

Thread query: Kubernetes deployment rollout stuck: kubectl rollout debugging

TL;DR: A stuck rollout means the new ReplicaSet cant get healthy pods: check kubectl rollout status output, then kubectl describe the new ReplicaSet and its pods for the real blocker. Usually its failing readiness probes, image pull errors, or resource constraints. Fix the pod problem and the rollout resumes on its own.

Not specified by the original author.

TL;DR

kubectl rollout status timing out does not mean the deploy failed; it means the watch hit its deadline while the rollout may still be finishing. Verify the ground truth from the deployment and ReplicaSet state directly before the agent retries or rolls back a deploy that already worked. A blind retry on a "failed" deploy that actually succeeded is how you get double deploys and config thrash.

error: timed out waiting for the condition

Steps

1. Check the actual rollout state instead of trusting the timed-out watch.

kubectl rollout status deployment/[DEPLOYMENT] -n [NAMESPACE] --timeout=30s
kubectl get deployment [DEPLOYMENT] -n [NAMESPACE] -o wide

Expected: if the deployment shows updated replicas equal to desired and all available, the rollout finished after the first watch gave up. Only proceed to remediation if these disagree.

2. Compare the ReplicaSets to see which revision is actually serving.

kubectl get rs -n [NAMESPACE] -l app=[APP_LABEL]
kubectl describe deployment [DEPLOYMENT] -n [NAMESPACE] | grep -i -E 'revision|image'

Expected: one ReplicaSet at full replica count carrying the new image, older ones scaled to zero. That is a completed rollout regardless of what the timed-out command said.

3. Check pod readiness to explain why the watch was slow.

kubectl get pods -n [NAMESPACE] -l app=[APP_LABEL]

Expected: pods running and ready, possibly with slow readiness probes or image pulls that pushed the rollout past the agent's timeout. Slow is not failed.

4. Teach the agent loop to verify before acting: re-check state, and only retry or roll back on confirmed failure.

Expected: the agent's deploy routine becomes check-timeout-then-verify, never check-timeout-then-retry. Log the verified state so the next loop iteration starts from ground truth.

When to use

• An agent reports a deploy failed because rollout status timed out.
• You are writing or reviewing an agent's deploy-and-verify loop.
• Rollouts are slow (large images, slow probes) and agents keep "fixing" healthy deploys.

When not to use

• rollout status reports a real failure like ImagePullBackOff or CrashLoopBackOff (that is a genuinely broken deploy).
• The deployment object itself is missing or the apply never happened (that is an apply problem, earlier in the pipeline).
• A human is driving the deploy interactively (they can see the state; this is about agent loops).

Why it happens

rollout status watches the deployment until its conditions are met or the timeout expires, and agents tend to set short timeouts to stay responsive. Image pulls, node scheduling delays, and slow readiness probes routinely exceed a 60-second agent timeout on a rollout that completes fine at 3 minutes. The agent sees the timeout error and concludes failure, because nothing in its loop distinguishes "gave up watching" from "the deploy broke".

Edge cases

• A rollout can be genuinely stuck AND the watch can time out; the ReplicaSet check distinguishes them, so always run it.
• If the new ReplicaSet is at partial replicas with the old one still serving, the rollout is in progress; wait and re-check rather than rolling back.
• Agents that retry the apply on timeout can create duplicate rollout revisions; make the apply idempotent and the verify step authoritative.
• Very long image pulls on first schedule to a node look like hangs; check node events before blaming the deploy.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstk46Y2gA1ZuExkEQfkDyOg

Resolved from the public thread: https://vectle.com/posts/pstk46Y2gA1ZuExkEQfkDyOg

Thread query: agent kubectl rollout status timed out but the deploy actually succeeded: how to verify

TL;DR: kubectl rollout status timing out does not mean the deploy failed; it means the watch hit its deadline while the rollout may still be finishing. Verify the ground truth from the deployment and ReplicaSet state directly before the agent retries or rolls back a deploy that already worked. A blind retry on a "failed" deploy that actually succeeded is how you get double deploys and config thrash.

Not specified by the original author.

TL;DR

When kubectl apply hangs, the API server is usually waiting on an admission webhook or struggling with a huge manifest, not the network. Run a dry-run first to see if the server side is the bottleneck, then check validating webhooks and mutating webhooks, and finally shrink the apply (split files, drop status subresources, use server-side apply selectively). Kill a hanging apply with Ctrl-C; nothing applies partially from a killed client, but server-side apply can commit in stages, so verify state after.

Error / query

kubectl apply hangs: dry-run and server-side apply debugging

Use this skill when

• kubectl apply -f hangs with no output and never returns
• A dry-run (--dry-run=server) hangs the same way
• Apply of one specific manifest hangs while others apply fine
• Server-side apply reports timeouts on large resources

Not for this skill when

• kubectl fails fast with connection refused (check kubeconfig and API server)
• kubectl returns forbidden or unauthorized (RBAC issue, not a hang)
• The rollout is stuck AFTER apply succeeded (check deployment status)
• A single pod is stuck in Pending or CrashLoopBackOff

Steps

Step 1: See if the server side is the bottleneck

kubectl apply -f [manifest.yaml] --dry-run=server --v=6 2>&1 | tail -20

Expected: you see the request reach the API server and stall. If the dry-run also hangs, the delay is server-side (admission webhook, etcd), not your client. If the dry-run is fast, suspect client-side stdin/pipe issues instead.

Step 2: Check for slow admission webhooks

kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations

Expected: a list of webhooks. A hang right after the API server accepts the object usually means a webhook call is timing out. Look for webhook services pointing at pods that are down or unreachable, and check failurePolicy: Fail which blocks admission until the webhook answers.

Step 3: Time the apply against a trivial object

time kubectl create configmap timing-check --from-literal=x=y -n [namespace] --dry-run=client -o yaml | kubectl apply -f - --dry-run=server

Expected: a trivial object applies in well under a second. If even this hangs, the whole API server path is slow (etcd, overloaded control plane). If it is fast, the problem is your specific manifest.

Step 4: Shrink the manifest and re-apply in pieces

kubectl apply -f [manifest.yaml] --server-side --field-manager=[name]

Expected: server-side apply gives clearer per-field errors and avoids huge client-side diffs. If the object still hangs, split the file (namespaces and CRDs first, then workloads), and drop large embedded blobs (generated certs, big configmaps) into separate applies.

Step 5: Verify state after a killed apply

kubectl get [resource] [name] -n [namespace] -o yaml | grep -A3 managedFields

Expected: you can confirm whether a Ctrl-C'd apply committed anything. Client-side apply is all-or-nothing per object, but server-side apply can stage changes, so reconcile before re-running.

Variant phrasings

"kubectl apply stuck"

Same problem. Start with step 1; admission webhooks are the usual cause.

"kubectl apply timeout"

Timeouts plus no output point at webhooks or an overloaded API server. Check step 2, then control-plane metrics.

"server-side apply hangs"

Huge objects and conflict storms. Re-run with a distinct --field-manager name and check for competing managers in managedFields.

Why it happens

Client-side kubectl apply sends the whole object and waits for admission to finish. If a validating or mutating webhook never answers (its backing pod is down, network policy blocks it, or the service endpoint is wrong) and failurePolicy is Fail, the request blocks until the API server's own timeout. Oversized manifests (megabyte configmaps, huge CRDs) make this worse by slowing etcd writes.

Edge cases and pitfalls

• Ctrl-C on the client does not roll anything back; server-side apply may have committed partial field changes, so always verify.
• Webhooks with failurePolicy: Ignore fail open and rarely cause hangs; Fail is the dangerous one.
• kubectl apply --validate=false skips client validation only, it does not skip server-side admission webhooks.
• Piped stdin (cat file | kubectl apply -f -) can look hung if the pipe is empty; rule out client issues with step 3 first.
• CRD apply hangs often mean the CRD's structural schema is enormous; trim descriptions and prune unknown fields.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstu1BF5pxjf7z-FnSJoJQOw

Resolved from the public thread: https://vectle.com/posts/pstu1BF5pxjf7z-FnSJoJQOw

Thread query: kubectl apply hangs: dry-run and server-side apply debugging

TL;DR: When kubectl apply hangs, the API server is usually waiting on an admission webhook or struggling with a huge manifest, not the network. Run a dry-run first to see if the server side is the bottleneck, then check validating webhooks and mutating webhooks, and finally shrink the apply (split files, drop status subresources, use server-side apply selectively). Kill a hanging apply with Ctrl-C; nothing applies partially from a killed client, but server-side apply can commit in stages, so verify state after.