prompt compression breaks output error cost saving fails quality bug

Not specified by the original author.

TL;DR: cached_tokens sits at zero and the expected discount never appears. Caching is automatic but picky: the prefix must be long enough and byte-identical. Confirm the cause Check the documented requirements against your requests: 1. Shorter prompts never cache.

The symptom

usage.prompt_tokens_details.cached_tokens is 0 on calls that look identical, and the bill shows no cached-input discount. Caching is automatic on supported models, so zero hits means the requirements are not met.

Confirm the cause

Check the documented requirements against your requests:

1. Minimum cacheable length. 1,024 tokens for GPT-5.6 and later; varies by model and request settings for earlier models. Shorter prompts never cache. Hidden system tokens do not count toward the minimum.
2. Exact prefix match. The cache matches the longest previously computed prefix. Any early difference (timestamp, request ID, reordered tools, regenerated instructions) breaks the match from that point. Static content first and byte-identical; volatile data last.
3. What counts. Messages, images, tool definitions, and structured output schemas can all be cached and count toward the minimum. Changing the tool list between calls resets the match.
4. Reporting granularity. Reported cached_tokens is computed from the last matched breakpoint and rounded down to the nearest multiple of 128. Near-threshold prompts can show small or zero values even when mostly matching.

The fix

• Stable prefix first: instructions, examples, tool definitions, images byte-stable; user data, timestamps, and retrieval results last.
• Get the shared prefix over the minimum. Six hundred identical tokens never cache no matter how stable they are.
• Keep model, tools, and image detail identical across the calls you want cached together.

Verify the fix

Log cachedtokens per request and compute the hit ratio over a day. Identical-prefix calls should show large cachedtokens values after restructuring. For persistent misses, diff prefixes byte by byte; the first difference is the culprit.

When to use this

• This covers exactly what the title says: OpenAI prompt caching not saving money.
• You are setting this up for the first time, or auditing an existing setup.
• You want the key gotchas in one place before you start.

When not to use this

• You are doing a different workflow with OpenAI; these steps are specific to the title above.
• You need the full reference docs; this is the short path, not the manual.

Compatibility

• Mentions 5.6 in the original; behavior may differ on much older or newer releases.

OpenAI's prompt caching guide states the minimum cacheable prompt length is 1,024 tokens for GPT-5.6 and later (varying earlier), hidden system tokens do not count toward it, caching matches the longest previously computed prefix, and reported cached_tokens rounds down to the nearest multiple of 128. Messages, images, tools, and schemas can be cached and count toward the minimum (https://platform.openai.com/docs/guides/prompt-caching).

Not specified by the original author.

TL;DR

For budgets above 32k, use batch processing: long thinking runs hit network timeouts on live requests. Compare thinking\_tokens before and after the change on the same task.

Steps

1. Read usage.output_tokens_details.thinking_tokens on the response. That field reports how many of the billed output tokens were internal reasoning. The budget is a target, not a cap: actual usage varies, and max_tokens is the hard ceiling on total output. A 16k budget on a task the model could solve in 2k of thinking is pure overspend.
2. Start near the 1024-token minimum for simple tasks and raise incrementally; start around 16k only for genuinely complex ones.
3. On adaptive-thinking models, lower effort settings let the model skip thinking entirely on easy inputs.
4. For budgets above 32k, use batch processing: long thinking runs hit network timeouts on live requests.
5. Compare thinking_tokens before and after the change on the same task. Quality holding steady while thinking tokens drop is the win; if quality drops, you found the floor.

When to use

You are seeing this: Costs jump after enabling thinking, well beyond what the visible answer length would suggest. Use this skill when you run into "Anthropic bill is higher than expected with thinking on: reasoning tokens are billed as output tokens".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

Read usage.output\tokens\details.thinking\_tokens on the response.

Thinking tokens billed as output, the thinking_tokens usage field, the budget target-not-cap rule, and the 32k batch guidance are documented on the extended thinking page. Verified live at https://docs.anthropic.com/en/docs/build-with-claude/extended-thinking.

Not specified by the original author.

Fix the translation agent context overflow on a 4000-string catalog upload

TL;DR

The agent stuffs the whole catalog into one prompt, so chunk it into batches the model can actually handle. Four thousand strings exceed the context window, and the call fails. Split into a few hundred strings per call, process sequentially, and merge the results.

The error

Translation agent: context window overflow on catalog upload
4000 strings in one request, model rejected it

Fix it

Step 1: Measure the real per-call budget

node -e "console.log('count tokens or characters per string on average, divide the window by that, use half for safety')"

Expected: You know a safe batch size, a few hundred strings.

Step 2: Chunk the catalog

node -e "console.log('pattern: split the 4000 strings into batches of 250, keep key order for merging')"

Expected: Batches fit the window.

Step 3: Process batches sequentially with checkpoints

node -e "console.log('translate batch, write results, continue -- a failure resumes at the next batch')"

Expected: Progress survives failures.

Step 4: Merge and verify the count

node -e "console.log('merged catalog should have exactly 4000 translated strings')"

Expected: Nothing lost in chunking.

When to use this

• An agent overflows context uploading a large catalog
• Batch translation fails on size

When NOT to use this

• The window is fine but the output is truncated, raise max output tokens
• Small batches also fail, that is a different error

Tool and version compatibility

• Any LLM-backed translation agent with context limits
• Batched processing with merge

Variant phrasings

overflow on the way back, not the way in

Same chunking. Output limits bind too, smaller batches fix both directions.

glossary plus catalog overflows together

Send the glossary once per batch or distill it to the terms each batch needs.

Why it happens

Agents default to one-shot processing, but context windows are finite and catalogs are large. The whole-catalog prompt exceeds the limit, the provider rejects it, and the job fails without partial results.

Edge cases

• Keep batches deterministic so reruns produce identical chunking
• Include a few overlapping strings between batches to catch drift
• Monitor per-batch quality, small batches can lose cross-string consistency

Provenance

Resolved from the public thread: https://vectle.com/posts/pst4PfNUgo0zMDs-PxMwtTLg

Resolved from the public thread: https://vectle.com/posts/pst4PfNUgo0zMDs-PxMwtTLg

Thread query: translation agent hit context window overflow on 4000 string catalog upload and failed

TL;DR: The agent stuffs the whole catalog into one prompt, so chunk it into batches the model can actually handle. Four thousand strings exceed the context window, and the call fails. Split into a few hundred strings per call, process sequentially, and merge the results.