Roll out OpenAI prompt caching: stable prefixes, verify cached_tokens, measure savings

Prompt caching rollout

The steps

1. Measure the baseline first: log usage.prompt_tokens_details on a sample of requests and note what is billable today. You cannot prove savings without a before number.
2. Restructure prompts so the stable content comes first: system instructions, tool definitions, few-shot examples, then the variable user content last. Caching works on prefixes, so anything that changes per request must sit at the end.
3. Check the minimum: the cacheable prefix must reach at least 1,024 tokens for current models. Short prompts do not qualify; do not expect caching on a 200-token system message.
4. Keep the prefix byte-identical across requests. One dynamic timestamp or user id injected early in the prompt breaks the cache for everything after it. Move volatile fields to the end or out of the prompt entirely.
5. Verify with cached_tokens in the usage object. If cached tokens stay at zero, the prefix is not stable or is under the minimum; fix the structure, not the API call.
6. For workloads that need separate cache accounting or routing control, use a stable prompt_cache_key. This keeps distinct workflows from evicting each other's cached prefixes.

The trap

Assuming caching is automatic. The API caches opportunistically on matching prefixes; a prompt that shuffles content order per request gets zero hits and the team concludes caching "does not work."

Checklist

• Baseline token usage logged before the rollout.
• Stable prefix of 1,024+ tokens, volatile content last.
• cached_tokens verified nonzero in production traffic.
• promptcachekey set where workflows share a model.
• Savings re-measured monthly; prompt edits re-verified for cache hits.

OpenAI's prompt caching guide documents the 1,024-token minimum, discounts up to 90%, cachedtokens in usage, and promptcache_key. Verified live at https://developers.openai.com/api/docs/guides/prompt-caching.