Anyscale on EKS: InsufficientInstanceCapacity or vCPU limit errors on GPU node groups

[Anyscale terraform-provider-anyscale example troubleshooting]: Request a quota increase for the relevant Running On-Demand P/G instances quota in the AWS console before scaling a GPU node group above the desired size. Also check disk: on Bottlerocket node groups, container images or Ray object spill can fill the ~20 GiB AMI-default data volume silently, so size the data volume explicitly for training workloads.

Context: Scaling an Anyscale GPU node group on AWS EKS fails with InsufficientInstanceCapacity or vCPU service-quota errors, even though the Terraform apply otherwise succeeded. Most AWS accounts start with a very low (often zero) service quota for g4dn, g5, p4d and p5 instance families.

Source: Scaling an Anyscale GPU node group on AWS EKS fails with InsufficientInstanceCapacity or vCPU service-quota errors, even though the Terraform apply otherwise succeeded. Most AWS accounts start with a very low (often zero) service quota for g4dn, g5, p4d and p5 instance families.

Verified resolution: [Anyscale terraform-provider-anyscale example troubleshooting]: Request a quota increase for the relevant Running On-Demand P/G instances quota in the AWS console before scaling a GPU node group above the desired size. Also check disk: on Bottlerocket node groups, container images or Ray object spill can fill the ~20 GiB AMI-default data volume silently, so size the data volume explicitly for training workloads.

Source URL: https://github.com/anyscale/terraform-provider-anyscale/blob/HEAD/examples/aws-eks-basic/README.md