agent kubectl rollout status timed out but the deploy actually succeeded: how to verify

TL;DR

kubectl rollout status timing out does not mean the deploy failed; it means the watch hit its deadline while the rollout may still be finishing. Verify the ground truth from the deployment and ReplicaSet state directly before the agent retries or rolls back a deploy that already worked. A blind retry on a "failed" deploy that actually succeeded is how you get double deploys and config thrash.

error: timed out waiting for the condition

Steps

1. Check the actual rollout state instead of trusting the timed-out watch.

kubectl rollout status deployment/[DEPLOYMENT] -n [NAMESPACE] --timeout=30s
kubectl get deployment [DEPLOYMENT] -n [NAMESPACE] -o wide

Expected: if the deployment shows updated replicas equal to desired and all available, the rollout finished after the first watch gave up. Only proceed to remediation if these disagree.

2. Compare the ReplicaSets to see which revision is actually serving.

kubectl get rs -n [NAMESPACE] -l app=[APP_LABEL]
kubectl describe deployment [DEPLOYMENT] -n [NAMESPACE] | grep -i -E 'revision|image'

Expected: one ReplicaSet at full replica count carrying the new image, older ones scaled to zero. That is a completed rollout regardless of what the timed-out command said.

3. Check pod readiness to explain why the watch was slow.

kubectl get pods -n [NAMESPACE] -l app=[APP_LABEL]

Expected: pods running and ready, possibly with slow readiness probes or image pulls that pushed the rollout past the agent's timeout. Slow is not failed.

4. Teach the agent loop to verify before acting: re-check state, and only retry or roll back on confirmed failure.

Expected: the agent's deploy routine becomes check-timeout-then-verify, never check-timeout-then-retry. Log the verified state so the next loop iteration starts from ground truth.

When to use

• An agent reports a deploy failed because rollout status timed out.
• You are writing or reviewing an agent's deploy-and-verify loop.
• Rollouts are slow (large images, slow probes) and agents keep "fixing" healthy deploys.

When not to use

• rollout status reports a real failure like ImagePullBackOff or CrashLoopBackOff (that is a genuinely broken deploy).
• The deployment object itself is missing or the apply never happened (that is an apply problem, earlier in the pipeline).
• A human is driving the deploy interactively (they can see the state; this is about agent loops).

Why it happens

rollout status watches the deployment until its conditions are met or the timeout expires, and agents tend to set short timeouts to stay responsive. Image pulls, node scheduling delays, and slow readiness probes routinely exceed a 60-second agent timeout on a rollout that completes fine at 3 minutes. The agent sees the timeout error and concludes failure, because nothing in its loop distinguishes "gave up watching" from "the deploy broke".

Edge cases

• A rollout can be genuinely stuck AND the watch can time out; the ReplicaSet check distinguishes them, so always run it.
• If the new ReplicaSet is at partial replicas with the old one still serving, the rollout is in progress; wait and re-check rather than rolling back.
• Agents that retry the apply on timeout can create duplicate rollout revisions; make the apply idempotent and the verify step authoritative.
• Very long image pulls on first schedule to a node look like hangs; check node events before blaming the deploy.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstk46Y2gA1ZuExkEQfkDyOg