Anyscale on EKS: InsufficientInstanceCapacity or vCPU limit errors on GPU node groups

TL;DR

Also check disk: on Bottlerocket node groups, container images or Ray object spill can fill the ~20 GiB AMI-default data volume silently, so size the data volume explicitly for training workloads. Request a quota increase for the relevant Running On-Demand P/G instances quota in the AWS console before scaling a GPU node group above the desired size.

The error

InsufficientInstanceCapacity or vCPU limit errors on GPU node groups

Fix

1. Also check disk: on Bottlerocket node groups, container images or Ray object spill can fill the ~20 GiB AMI-default data volume silently, so size the data volume explicitly for training workloads. Expected: You see the expected output instead of the problem.

2. Re-run the original operation and confirm the error is gone. Expected: no error, normal output.

When to use

• You hit this exact error with Anyscale on EKS.
• The symptom matches: InsufficientInstanceCapacity or vCPU limit errors on GPU node groups.

When NOT to use

• A different error message from Anyscale on EKS; the cause here is specific to this error.
• Unrelated Anyscale on EKS issues (different feature, different failure).
• You need general documentation for the tool; check the official docs instead.

Compatibility

Reported against Anyscale on EKS. Source: https://github.com/anyscale/terraform-provider-anyscale/blob/HEAD/examples/aws-eks-basic/README.md.

Variant phrasings

InsufficientInstanceCapacity or vCPU limit errors on GPU node groups

Why it happens

Scaling an Anyscale GPU node group on AWS EKS fails with InsufficientInstanceCapacity or vCPU service-quota errors, even though the Terraform apply otherwise succeeded. Most AWS accounts start with a very low (often zero) service quota for g4dn, g5, p4d and p5 instance families.

Edge cases

• If your error message differs even slightly, this is probably a different issue; search the exact text.
• If the fix does not help, capture the full error output and check the source link for updates.

Source

https://github.com/anyscale/terraform-provider-anyscale/blob/HEAD/examples/aws-eks-basic/README.md

Source: Scaling an Anyscale GPU node group on AWS EKS fails with InsufficientInstanceCapacity or vCPU service-quota errors, even though the Terraform apply otherwise succeeded. Most AWS accounts start with a very low (often zero) service quota for g4dn, g5, p4d and p5 instance families.

Verified resolution: [Anyscale terraform-provider-anyscale example troubleshooting]: Request a quota increase for the relevant Running On-Demand P/G instances quota in the AWS console before scaling a GPU node group above the desired size. Also check disk: on Bottlerocket node groups, container images or Ray object spill can fill the ~20 GiB AMI-default data volume silently, so size the data volume explicitly for training workloads.

Source URL: https://github.com/anyscale/terraform-provider-anyscale/blob/HEAD/examples/aws-eks-basic/README.md