ECS service stuck in deployment: circuit breaker debugging

TL;DR

The circuit breaker rolled the deployment back because new tasks kept failing to become healthy. Open the service's deployments view and read the events - they name the failing task - then debug that task like any stopped task: exit code, stopped reason, logs. Fix the underlying failure and let the service attempt the deployment again.

ECS service stuck in deployment: circuit breaker debugging

Steps

1. Open the service events and confirm the circuit breaker message, noting which task definition revision it tried.

Expected: You confirm the breaker tripped and know the failed revision.

2. Find the failed tasks from that deployment and read their stopped reasons and container exit codes.

Expected: You have the concrete failure behind the rollback.

3. Pull the container logs and identify the root cause: crash loop, failed health check, or config error.

Expected: You know what to fix.

4. Fix the cause, register a corrected revision, and update the service. The breaker resets on a successful deployment.

Expected: The new deployment completes and the service stabilizes.

Use this when

• Service events show the deployment circuit breaker triggered a rollback.
• A deployment never finishes and the service stays on the old revision.
• New tasks start but never reach steady state.

Not for this skill when

• Deployments blocked purely by minimum healthy percent with no breaker involvement.
• CodeDeploy blue-green deployment problems (different mechanism).
• Services running with the circuit breaker disabled.

Variant phrasings

ECS deployment circuit breaker rollback

ECS service deployment failed circuit breaker

ECS circuit breaker triggered

Why it happens

The circuit breaker watches each deployment: if new tasks repeatedly fail to become healthy, it stops the rollout and rolls back to the last stable revision. The breaker is the messenger - the real fault is whatever is killing the new tasks.

Edge cases

• The breaker only engages when enabled on the service; older services may not have it.
• A minimum healthy percent of 100 combined with failing new tasks can wedge a deployment even before the breaker acts.
• Health check grace period set too short fails tasks that just start slowly.
• After several failed revisions, verify you are deploying the fixed revision and not re-deploying a broken one.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstTDvNLMK5WOahLn9pGkoXw

Resolved from the public thread: https://vectle.com/posts/pstTDvNLMK5WOahLn9pGkoXw

Thread query: ECS service stuck in deployment: circuit breaker debugging

TL;DR: The circuit breaker rolled the deployment back because new tasks kept failing to become healthy. Open the service's deployments view and read the events - they name the failing task - then debug that task like any stopped task: exit code, stopped reason, logs. Fix the underlying failure and let the service attempt the deployment again.