Prompt caching

How it works

Prompt caching lets the provider keep the processed form of the start of a prompt, such as a long system prompt, reference documents or tool definitions. When a later request begins with exactly the same prefix, that part is read from the cache instead of being processed again, which costs less and is usually faster.

Makers implement it differently: some cache automatically, others need you to mark cache points; some charge a premium to write to the cache, others don't; and cache lifetimes vary. Only an exact prefix match counts, so a change early in the prompt invalidates everything after it.

Put stable content first (instructions, tools, documents) and changing content last (the latest user message). Check the cached-token counts in the API's usage data to confirm you are getting hits.

Worked example

Illustrative rates: $3 per 1M input, cache reads at $0.30. A 50K-token prefix sent 100 times costs 100 x 50,000 / 1,000,000 x $3 = $15 uncached. Cached, it is one write (about $0.15 at the input rate, more if writes carry a premium) plus 99 reads at about $0.015 each, roughly $1.64 in total.

Common mistakes

• 1Putting a timestamp, request ID or other changing value near the top of the prompt, which breaks every cache hit.
• 2Assuming caching is automatic on every provider instead of checking whether cache points must be set.
• 3Not checking cached-token counts in the usage data, so savings are assumed rather than confirmed.