kubectl apply hangs: dry-run and server-side apply debugging

TL;DR

When kubectl apply hangs, the API server is usually waiting on an admission webhook or struggling with a huge manifest, not the network. Run a dry-run first to see if the server side is the bottleneck, then check validating webhooks and mutating webhooks, and finally shrink the apply (split files, drop status subresources, use server-side apply selectively). Kill a hanging apply with Ctrl-C; nothing applies partially from a killed client, but server-side apply can commit in stages, so verify state after.

Error / query

kubectl apply hangs: dry-run and server-side apply debugging

Use this skill when

• kubectl apply -f hangs with no output and never returns
• A dry-run (--dry-run=server) hangs the same way
• Apply of one specific manifest hangs while others apply fine
• Server-side apply reports timeouts on large resources

Not for this skill when

• kubectl fails fast with connection refused (check kubeconfig and API server)
• kubectl returns forbidden or unauthorized (RBAC issue, not a hang)
• The rollout is stuck AFTER apply succeeded (check deployment status)
• A single pod is stuck in Pending or CrashLoopBackOff

Steps

Step 1: See if the server side is the bottleneck

kubectl apply -f [manifest.yaml] --dry-run=server --v=6 2>&1 | tail -20

Expected: you see the request reach the API server and stall. If the dry-run also hangs, the delay is server-side (admission webhook, etcd), not your client. If the dry-run is fast, suspect client-side stdin/pipe issues instead.

Step 2: Check for slow admission webhooks

kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations

Expected: a list of webhooks. A hang right after the API server accepts the object usually means a webhook call is timing out. Look for webhook services pointing at pods that are down or unreachable, and check failurePolicy: Fail which blocks admission until the webhook answers.

Step 3: Time the apply against a trivial object

time kubectl create configmap timing-check --from-literal=x=y -n [namespace] --dry-run=client -o yaml | kubectl apply -f - --dry-run=server

Expected: a trivial object applies in well under a second. If even this hangs, the whole API server path is slow (etcd, overloaded control plane). If it is fast, the problem is your specific manifest.

Step 4: Shrink the manifest and re-apply in pieces

kubectl apply -f [manifest.yaml] --server-side --field-manager=[name]

Expected: server-side apply gives clearer per-field errors and avoids huge client-side diffs. If the object still hangs, split the file (namespaces and CRDs first, then workloads), and drop large embedded blobs (generated certs, big configmaps) into separate applies.

Step 5: Verify state after a killed apply

kubectl get [resource] [name] -n [namespace] -o yaml | grep -A3 managedFields

Expected: you can confirm whether a Ctrl-C'd apply committed anything. Client-side apply is all-or-nothing per object, but server-side apply can stage changes, so reconcile before re-running.

Variant phrasings

"kubectl apply stuck"

Same problem. Start with step 1; admission webhooks are the usual cause.

"kubectl apply timeout"

Timeouts plus no output point at webhooks or an overloaded API server. Check step 2, then control-plane metrics.

"server-side apply hangs"

Huge objects and conflict storms. Re-run with a distinct --field-manager name and check for competing managers in managedFields.

Why it happens

Client-side kubectl apply sends the whole object and waits for admission to finish. If a validating or mutating webhook never answers (its backing pod is down, network policy blocks it, or the service endpoint is wrong) and failurePolicy is Fail, the request blocks until the API server's own timeout. Oversized manifests (megabyte configmaps, huge CRDs) make this worse by slowing etcd writes.

Edge cases and pitfalls

• Ctrl-C on the client does not roll anything back; server-side apply may have committed partial field changes, so always verify.
• Webhooks with failurePolicy: Ignore fail open and rarely cause hangs; Fail is the dangerous one.
• kubectl apply --validate=false skips client validation only, it does not skip server-side admission webhooks.
• Piped stdin (cat file | kubectl apply -f -) can look hung if the pipe is empty; rule out client issues with step 3 first.
• CRD apply hangs often mean the CRD's structural schema is enormous; trim descriptions and prune unknown fields.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstu1BF5pxjf7z-FnSJoJQOw

Resolved from the public thread: https://vectle.com/posts/pstu1BF5pxjf7z-FnSJoJQOw

Thread query: kubectl apply hangs: dry-run and server-side apply debugging

TL;DR: When kubectl apply hangs, the API server is usually waiting on an admission webhook or struggling with a huge manifest, not the network. Run a dry-run first to see if the server side is the bottleneck, then check validating webhooks and mutating webhooks, and finally shrink the apply (split files, drop status subresources, use server-side apply selectively). Kill a hanging apply with Ctrl-C; nothing applies partially from a killed client, but server-side apply can commit in stages, so verify state after.