Baseten autoscaling runs on in-flight tokens, and engine config names must match the engine

TL;DR

Configure autoscaling around in-flight tokens, not concurrency, and keep at least one replica alive since scale-to-zero is unsupported. Match engine_config field names to the engine you actually run, vLLM names for vLLM and TRT-LLM names for TRT-LLM, or the deploy fails at startup.

Steps

1. Configure autoscaling around in-flight tokens, not concurrency, and keep at least one replica alive since scale-to-zero is unsupported. Match engineconfig field names to the engine you actually run, vLLM names for vLLM and TRT-LLM names for TRT-LLM, or the deploy fails at startup. For cache-sensitive workloads, keep maxscaledownrate low and scaledowndelay in the 300 to 600 second range so replicas drain gently and your time-to-first-token stays stable.

When to use

You are seeing this: Match engine_config field names to the engine you actually run, vLLM names for vLLM and TRT-LLM names for TRT-LLM, or the deploy fails at startup. Use this skill when you run into "Baseten autoscaling runs on in-flight tokens, and engine config names must match the engine".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Source: A Baseten usage reference documents the autoscaling and config gotchas that cause failed or slow deploys. LLM serving autoscales on in-flight tokens, not request concurrency, so setting concurrencytarget is rejected and only targetinflighttokens applies, with 50K to 150K a sane starting range. Scale-to-zero is not supported, so minreplica must be at least 1. engineconfig field names are engine-native: vLLM uses maxnumseqs while TRT-LLM uses maxbatchsize, and mixing the two fails at startup. Scale-down also erodes KV cache, causing TTFT spikes unless maxscaledown_rate stays low.

Verified resolution: Configure autoscaling around in-flight tokens, not concurrency, and keep at least one replica alive since scale-to-zero is unsupported. Match engineconfig field names to the engine you actually run, vLLM names for vLLM and TRT-LLM names for TRT-LLM, or the deploy fails at startup. For cache-sensitive workloads, keep maxscaledownrate low and scaledowndelay in the 300 to 600 second range so replicas drain gently and your time-to-first-token stays stable.

Source URL: https://github.com/dsingal0/dotfiles/blob/HEAD/skills/baseten/SKILL.md