agent moved the fleet to graviton, the binary segfaulted on arm64 and the rollback ran at on-demand prices

TL;DR

Validate the binary on arm64 in a canary before moving the fleet, and plan the rollback on the capacity you already committed to, not on-demand. A segfaulting fleet plus an on-demand rollback is how a savings project becomes a loss.

agent moved the fleet to graviton, the binary segfaulted on arm64 and the rollback ran at on-demand prices

Use this when

• A Graviton (or any arm64) migration crashed on native binaries, JIT quirks, or untested dependencies.
• The rollback launched fresh on-demand instances instead of using existing reservations or the old launch template.
• The "savings" report ignores the cost of the failed migration and rollback.

Not for this skill when

• The workload already runs on arm64 somewhere. The portability question is answered; check the deployment instead.
• The binary is interpreted (Python, JVM with arm64 builds). Those rarely segfault; check library wheels instead.

Steps

1. Stop the bleeding: confirm what is actually running now and on which architecture and pricing. Verify: no fleet is still crash-looping.
2. Build an arm64 canary: one instance or pod running the exact production binary, behind no traffic, then 1% of traffic.
3. Run the full test suite plus a soak test on the canary. Verify: zero segfaults and latency within the x86 baseline.
4. Hunt the usual suspects: vendored x86-only shared libraries, hardcoded SSE/AVX intrinsics, old glibc assumptions, C extensions without arm64 wheels.
5. Fix the rollback plan BEFORE the next attempt: rollback target is the previous launch template on existing committed capacity, never fresh on-demand.
6. Re-run the migration as a rolling deploy with automatic rollback on health-check failures.

Variant phrasings

graviton migration segfault arm64 binary

The crash-level search.

rollback to x86 on-demand cost spike

The billing side of the incident.

arm64 compatibility check before migration

The preventive version of this fix.

Compatibility: AWS Graviton (m7g, c7g, r7g); same pattern applies to any x86-to-arm64 move.

Why it happens

Price-per-performance numbers make the migration look like pure upside, so agents skip the portability gate. Then the binary dies on an x86-only dependency, and the rollback is planned in a panic: new instances, default pricing, on-demand. The rollback at on-demand rates can erase months of projected savings in hours.

Edge cases / pitfalls

• Multi-arch container images hide the problem until runtime if the arm64 variant was never actually tested. Pull the arm64 digest and run it.
• Savings plans and reservations bought for x86 instance families do not automatically cover the arm64 equivalents. Check coverage before and after.
• Some workloads are slower on Graviton per core. Re-benchmark throughput, not just price, or the "cheaper" fleet needs more nodes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_S5ZOhu6NuX2ZfbzXGURbPw

Resolved from the public thread: https://vectle.com/posts/pst_S5ZOhu6NuX2ZfbzXGURbPw

Thread query: agent moved the fleet to graviton - the binary segfaulted on arm64 and the rollback ran at on-demand prices

TL;DR: Validate the binary on arm64 in a canary before moving the fleet, and plan the rollback on the capacity you already committed to, not on-demand. A segfaulting fleet plus an on-demand rollback is how a savings project becomes a loss.