Baseten autoscaling runs on in-flight tokens, and engine config names must match the engine

Configure autoscaling around in-flight tokens, not concurrency, and keep at least one replica alive since scale-to-zero is unsupported. Match engineconfig field names to the engine you actually run, vLLM names for vLLM and TRT-LLM names for TRT-LLM, or the deploy fails at startup. For cache-sensitive workloads, keep maxscaledownrate low and scaledowndelay in the 300 to 600 second range so replicas drain gently and your time-to-first-token stays stable.

Context: A Baseten usage reference documents the autoscaling and config gotchas that cause failed or slow deploys. LLM serving autoscales on in-flight tokens, not request concurrency, so setting concurrencytarget is rejected and only targetinflighttokens applies, with 50K to 150K a sane starting range. Scale-to-zero is not supported, so minreplica must be at least 1. engineconfig field names are engine-native: vLLM uses maxnumseqs while TRT-LLM uses maxbatchsize, and mixing the two fails at startup. Scale-down also erodes KV cache, causing TTFT spikes unless maxscaledown_rate stays low.

Matched source

Source: Published skill Original query: "Baseten autoscaling runs on in-flight tokens, and engine config names must match the engine" Key terms: autoscaling, baseten, config, engine, flight, match, must, names, runs, tokens

Source: A Baseten usage reference documents the autoscaling and config gotchas that cause failed or slow deploys. LLM serving autoscales on in-flight tokens, not request concurrency, so setting concurrencytarget is rejected and only targetinflighttokens applies, with 50K to 150K a sane starting range. Scale-to-zero is not supported, so minreplica must be at least 1. engineconfig field names are engine-native: vLLM uses maxnumseqs while TRT-LLM uses maxbatchsize, and mixing the two fails at startup. Scale-down also erodes KV cache, causing TTFT spikes unless maxscaledown_rate stays low.

Verified resolution: Configure autoscaling around in-flight tokens, not concurrency, and keep at least one replica alive since scale-to-zero is unsupported. Match engineconfig field names to the engine you actually run, vLLM names for vLLM and TRT-LLM names for TRT-LLM, or the deploy fails at startup. For cache-sensitive workloads, keep maxscaledownrate low and scaledowndelay in the 300 to 600 second range so replicas drain gently and your time-to-first-token stays stable.

Source URL: https://github.com/dsingal0/dotfiles/blob/HEAD/skills/baseten/SKILL.md