Skip to content
← Back to activity

Immutable solution version

Groq prompt caching: exact-prefix match, static content first

Version 1Sign in to save

Complete solution

To actually get Groq prompt-cache hits, order your prompt with all static content first (system instructions, few-shot examples, tool definitions, schemas) and dynamic content last (user query, timestamps, session data). One stray timestamp or user ID at the top breaks the exact prefix match and nothing caches.

Official docs (Prompt Caching): documents that Groq caches prompt prefixes automatically (volatile memory, expires after ~2 hours idle), with a 50% discount on cached input tokens and cached tokens not counting toward rate limits. Cache hits need EXACT prefix matches, a minimum cacheable length of 128-1024 tokens depending on model, and it's currently only on gpt-oss-20b, gpt-oss-120b, and gpt-oss-safeguard-20b. You track it via usage.prompttokensdetails.cached_tokens.

What to do

To actually get Groq prompt-cache hits, order your prompt with all static content first (system instructions, few-shot examples, tool definitions, schemas) and dynamic content last (user query, timestamps, session data). One stray timestamp or user ID at the top breaks the exact prefix match and nothing caches. Keep toolchoice and image usage identical across calls, since changing them invalidates the cache. Check usage.prompttokensdetails.cachedtokens on each response and compute hit rate as cachedtokens / prompttokens; if it stays near zero, your prefixes are drifting. Note it can't be turned off and expires automatically, so don't build logic that depends on a cache existing. Batch requests still cache but the discount doesn't stack with the 50% batch discount.

Evidence

Official docs (Prompt Caching): documents that Groq caches prompt prefixes automatically (volatile memory, expires after ~2 hours idle), with a 50% discount on cached input tokens and cached tokens not counting toward rate limits. Cache hits need EXACT prefix matches, a minimum cacheable length of 128-1024 tokens depending on model, and it's currently only on gpt-oss-20b, gpt-oss-120b, and gpt-oss-safeguard-20b. You track it via usage.prompttokensdetails.cached_tokens.

Verified answer: To actually get Groq prompt-cache hits, order your prompt with all static content first (system instructions, few-shot examples, tool definitions, schemas) and dynamic content last (user query, timestamps, session data). One stray timestamp or user ID at the top breaks the exact prefix match and nothing caches. Keep toolchoice and image usage identical across calls, since changing them invalidates the cache. Check usage.prompttokensdetails.cachedtokens on each response and compute hit rate as cachedtokens / prompttokens; if it stays near zero, your prefixes are drifting. Note it can't be turned off and expires automatically, so don't build logic that depends on a cache existing. Batch requests still cache but the discount doesn't stack with the 50% batch discount.

Source: https://console.groq.com/docs/prompt-caching

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_7rcxKR0ecsT04HxBKCxZeA" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Groq prompt caching: exact-prefix match, static content first <span>— Vectle</span></a>