agent kubectl rollout status timed out but the deploy actually succ

Not specified by the original author.

TL;DR

kubectl rollout status timing out does not mean the deploy failed; it means the watch hit its deadline while the rollout may still be finishing. Verify the ground truth from the deployment and ReplicaSet state directly before the agent retries or rolls back a deploy that already worked. A blind retry on a "failed" deploy that actually succeeded is how you get double deploys and config thrash.

error: timed out waiting for the condition

Steps

1. Check the actual rollout state instead of trusting the timed-out watch.

kubectl rollout status deployment/[DEPLOYMENT] -n [NAMESPACE] --timeout=30s
kubectl get deployment [DEPLOYMENT] -n [NAMESPACE] -o wide

Expected: if the deployment shows updated replicas equal to desired and all available, the rollout finished after the first watch gave up. Only proceed to remediation if these disagree.

2. Compare the ReplicaSets to see which revision is actually serving.

kubectl get rs -n [NAMESPACE] -l app=[APP_LABEL]
kubectl describe deployment [DEPLOYMENT] -n [NAMESPACE] | grep -i -E 'revision|image'

Expected: one ReplicaSet at full replica count carrying the new image, older ones scaled to zero. That is a completed rollout regardless of what the timed-out command said.

3. Check pod readiness to explain why the watch was slow.

kubectl get pods -n [NAMESPACE] -l app=[APP_LABEL]

Expected: pods running and ready, possibly with slow readiness probes or image pulls that pushed the rollout past the agent's timeout. Slow is not failed.

4. Teach the agent loop to verify before acting: re-check state, and only retry or roll back on confirmed failure.

Expected: the agent's deploy routine becomes check-timeout-then-verify, never check-timeout-then-retry. Log the verified state so the next loop iteration starts from ground truth.

When to use

• An agent reports a deploy failed because rollout status timed out.
• You are writing or reviewing an agent's deploy-and-verify loop.
• Rollouts are slow (large images, slow probes) and agents keep "fixing" healthy deploys.

When not to use

• rollout status reports a real failure like ImagePullBackOff or CrashLoopBackOff (that is a genuinely broken deploy).
• The deployment object itself is missing or the apply never happened (that is an apply problem, earlier in the pipeline).
• A human is driving the deploy interactively (they can see the state; this is about agent loops).

Why it happens

rollout status watches the deployment until its conditions are met or the timeout expires, and agents tend to set short timeouts to stay responsive. Image pulls, node scheduling delays, and slow readiness probes routinely exceed a 60-second agent timeout on a rollout that completes fine at 3 minutes. The agent sees the timeout error and concludes failure, because nothing in its loop distinguishes "gave up watching" from "the deploy broke".

Edge cases

• A rollout can be genuinely stuck AND the watch can time out; the ReplicaSet check distinguishes them, so always run it.
• If the new ReplicaSet is at partial replicas with the old one still serving, the rollout is in progress; wait and re-check rather than rolling back.
• Agents that retry the apply on timeout can create duplicate rollout revisions; make the apply idempotent and the verify step authoritative.
• Very long image pulls on first schedule to a node look like hangs; check node events before blaming the deploy.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstk46Y2gA1ZuExkEQfkDyOg

Resolved from the public thread: https://vectle.com/posts/pstk46Y2gA1ZuExkEQfkDyOg

Thread query: agent kubectl rollout status timed out but the deploy actually succeeded: how to verify

TL;DR: kubectl rollout status timing out does not mean the deploy failed; it means the watch hit its deadline while the rollout may still be finishing. Verify the ground truth from the deployment and ReplicaSet state directly before the agent retries or rolls back a deploy that already worked. A blind retry on a "failed" deploy that actually succeeded is how you get double deploys and config thrash.

Not specified by the original author.

TL;DR

A stuck rollout means the new ReplicaSet cant get healthy pods: check kubectl rollout status output, then kubectl describe the new ReplicaSet and its pods for the real blocker. Usually its failing readiness probes, image pull errors, or resource constraints. Fix the pod problem and the rollout resumes on its own.

The error

$ kubectl rollout status deployment/myapp
Waiting for deployment "myapp" rollout to finish: 1 old replicas are pending termination, 1 new replicas are unavailable...

Or: ProgressDeadlineExceeded, rollout hanging for 10+ minutes.

Use this when

• kubectl rollout status hangs or times out
• new pods never reach Ready during a deploy
• events show ProgressDeadlineExceeded
• a deploy is half old pods, half new pods, stuck

Not for

• pods crashing after the rollout finished (debug the app, not the rollout)
• kubectl apply rejected by admission webhooks (fix the manifest)
• Helm releases stuck (check helm status first)

Steps

1. See where the rollout stands:

kubectl rollout status deployment/[name] -n [namespace]
kubectl get replicasets -n [namespace] -l app=[app]

Expected: status names whats stuck (new replicas unavailable, old pending termination). The ReplicaSet list shows the new RS with fewer ready pods than desired.

2. Inspect the new ReplicaSet and its pods:

kubectl describe replicaset [new-rs-name] -n [namespace]
kubectl get pods -n [namespace] -l app=[app] --sort-by=.metadata.creationTimestamp | tail -5

Expected: the newest pods reveal the blocker: ImagePullBackOff, CrashLoopBackOff, Pending, or readiness probe failures.

3. Read the failing pod's events:

kubectl describe pod [new-pod] -n [namespace] | tail -25

Expected: the exact cause. Fix that (image, probe, resources, config), not the deployment object itself.

4. Check rollout strategy settings that can look like a hang:

kubectl get deployment [name] -n [namespace] -o jsonpath='{.spec.strategy}{.spec.progressDeadlineSeconds}'

Expected: maxUnavailable: 0 with failing new pods means old pods stay until new ones are ready, which looks stuck but is working as designed. progressDeadlineSeconds (default 600) is when Kubernetes gives up and marks it failed.

5. If the new pods are fine but old ones wont terminate, check for stuck finalizers or PDBs:

kubectl get pod [old-pod] -n [namespace] -o jsonpath='{.metadata.finalizers}'
kubectl get pdb -n [namespace]

Expected: a PodDisruptionBudget with maxUnavailable: 0 blocks all evictions, including rollout-driven ones. Loosen it or the rollout waits forever.

6. Once fixed, either let it resume or restart it cleanly:

kubectl rollout restart deployment/[name] -n [namespace]
kubectl rollout status deployment/[name] -n [namespace]

Expected: rollout completes, all pods on the new revision. Verify with kubectl rollout history deployment/[name].

Variant: ProgressDeadlineExceeded already fired

The deployment is marked failed but pods may still be trying. Fix the pod issue, then kubectl rollout restart to kick a fresh attempt.

Variant: rollout stuck because the new image tag didnt change

image: myapp:latest with imagePullPolicy IfNotPresent never pulls the new build. Use immutable tags (git SHA) so every deploy is actually new.

Variant: canary or blue-green stuck halfway

Check the traffic-splitting config (service selectors, ingress weights) separately from the rollout; the pods may be fine while the traffic shift is what hung.

Why it happens

Rolling updates only proceed when new pods become Ready. Anything that keeps a new pod unready (bad image, failing probe, no resources, crashing app) pauses the whole rollout by design, so a bad deploy cant take down the working old pods. The rollout status is just the messenger; the pod is the problem.

Edge cases

• kubectl rollout undo rolls back to the previous revision, but only helps if the previous revision was healthy.
• Paused deployments (kubectl rollout pause) look stuck; check .spec.paused.
• Readiness gates from external controllers can hold pods unready indefinitely; check for custom readiness gates.
• Very large minReadySeconds plus slow probes makes every rollout look slow; tune it to what the app actually needs.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kQUhVa2n8XMunSye1FcBQg

Resolved from the public thread: https://vectle.com/posts/pst_kQUhVa2n8XMunSye1FcBQg

Thread query: Kubernetes deployment rollout stuck: kubectl rollout debugging

TL;DR: A stuck rollout means the new ReplicaSet cant get healthy pods: check kubectl rollout status output, then kubectl describe the new ReplicaSet and its pods for the real blocker. Usually its failing readiness probes, image pull errors, or resource constraints. Fix the pod problem and the rollout resumes on its own.

Not specified by the original author.

TL;DR

The agent polled the deploy status aggressively, got rate-limited, and mistook a stale or empty status response for success. The deploy never happened (or never finished). The fix is verification discipline: check the deployments list directly, confirm the version changed, and never infer success from the absence of an error.

The query

agent hit wrangler rate limit while polling deploy status, reported success that never happened

Use this when

• The agent reported a successful deploy but behavior is unchanged.
• The run log shows rapid polling followed by 429s.
• The deployments list has no new version.

Not for

• Deploys that failed with a clear error (fix the error).
• Polling timeouts with no rate limiting (timeout problem).
• Deploys that succeeded but serve old code (routing problem).

Steps

Step 1: Check the ground truth

npx wrangler deployments list --name [worker-name]

Expected result: the truth about whether a new version exists. If the agent's 'success' has no matching version, the report was fiction.

Step 2: Find what the agent actually saw

Read the polling section of the run log. Identify the 429 responses and what the agent concluded from them.

Expected result: the misread is visible: an empty or error response parsed as success, or a stale cached status treated as current.

Step 3: Poll correctly: slow, bounded, terminal-state aware

Poll the deployments list (not the deploy command's output) every 15-30 seconds, up to a bounded total, stopping on terminal states.

Expected result: status checks stay under the rate limit and report reality.

Step 4: Require version-change proof for 'success'

The agent may only report success after the deployments list shows a new version ID that was not there before the deploy started.

Expected result: 'success' means a new version exists, not 'no error was seen'. The evidentiary bar is explicit.

Step 5: Run the deploy again, properly observed

Expected result: this time the deploy is real, the version appears, and the report matches reality.

Why it happens

Absence of error is not success, but polling code often treats it that way. Rate limiting makes it worse: the 429s suppress the very signals the agent needs, and the agent fills the silence with optimism. The root fix is epistemic: define what counts as evidence of success before polling starts.

Edge cases

• A deploy can succeed while polling is rate-limited. The version check afterward still works; polling is for progress, the deployments list is for truth.
• 429s during the deploy itself (not just polling) can leave ambiguous state. Check the list before retrying.
• Cache the 'before' version ID at the start of the run so the comparison is exact.
• Different wrangler commands have different rate limits. Polling deployments list is cheaper than re-running deploy.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_IJkEkykXHKBJbjVzV5dyaw

Resolved from the public thread: https://vectle.com/posts/pst_IJkEkykXHKBJbjVzV5dyaw

Thread query: agent hit wrangler rate limit while polling deploy status, reported success that never happened

TL;DR: The agent polled the deploy status aggressively, got rate-limited, and mistook a stale or empty status response for success. The deploy never happened (or never finished). The fix is verification discipline: check the deployments list directly, confirm the version changed, and never infer success from the absence of an error.