What OpenAI API prompt caching actually reuses
OpenAI API prompt caching is a feature for requests that begin with the same prompt prefix. When a model processes input, it calculates intermediate key-value (KV) states that help it refer to earlier tokens as it processes more input and generates output. Caching preserves those states for the unchanged beginning of a prompt. If a later request finds a matching cache entry for that prefix, the model can reuse the saved work rather than process those tokens again. It must still process new input and generate a response to the new request.
That distinction matters: the cache stores KV tensors, not the prompt tokens themselves. Nor is it a store of finished answers. A cached prefix can reduce work on familiar input while leaving the model to handle what comes next. OpenAI says prompt caching is enabled by default for supported models; the feature does not require a request to return the same answer as an earlier one.
Why the shared prefix determines a cache hit
The word “prefix” is doing important work here. OpenAI describes the cache as applying to the unchanged tokens at the beginning of a prompt, not to any repeated passage found somewhere in a request. Its documentation says the full rendered context can include OpenAI-provided instructions, developer messages, tool definitions, and conversation history with text, images, documents, or supported audio. Reuse requires the entire rendered prefix to match.
Consider a hypothetical pair of requests that start with the same instructions and tool definitions, then ask different questions. Their shared beginning could be reusable if a matching, eligible cache entry is found; the differing questions would still need processing. Now imagine changing a tool description near the beginning of the second request. Content after that change cannot match the existing prefix in the same way. This is why two requests that look broadly similar to a reader are not necessarily equivalent for caching.
A changed request does not automatically erase an existing cache entry. The relevant question is whether a subsequent request retains the same prefix and finds an eligible matching breakpoint. OpenAI’s selected evidence source frames cache behavior around that match, rather than a general judgment that two prompts cover the same topic.
What this means for cost and latency
Reusing processed input can reduce the work performed before a response starts. OpenAI also says reused tokens are charged at the model’s reduced cached-input rate, with discounts of up to 95%. That figure is an upper bound, not a promise about any particular request. Pricing varies by model, and input tokens use the applicable uncached-input, cached-input, or cache-write rate; cache-write pricing is not an additional fee layered onto another input rate.
The practical outcome depends on reuse, not merely on sending repeated-looking prompts. OpenAI provides a Prompt Caching Dashboard for monitoring cache read hit rates and a diagnostics tool for investigating misses. Those measurements can distinguish a genuinely reused prefix from one that only appears similar on inspection.
What the cache preserves during processing
As a model reads input tokens, it calculates intermediate key-value (KV) states that help it refer back to earlier tokens while processing new input and generating output. The prompt cache preserves those states for an unchanged prefix. It does not store the prompt tokens themselves as the cached object. When a later request finds an eligible matching entry, the model can reuse the saved states for that prefix, then process the new input and generate a response.
That distinction matters when thinking about what is saved. A hypothetical request that keeps its opening instructions but asks a different question at the end could reuse work associated with the unchanged opening. It would not reuse the work needed to process the new question or produce the answer. OpenAI API prompt caching is therefore about avoiding repeated processing of matching input, not retrieving an earlier response.
The rendered context is broader than the visible question
The prefix OpenAI considers is the model’s full rendered context. It can include OpenAI-provided instructions, developer messages, tool definitions, and conversation history. That history may contain text, images, documents, or supported audio. Someone accustomed to browsing ChatGPT guides might focus on the wording of a user’s question, but identical questions alone do not establish that two API requests have identical rendered prefixes.
Tool configuration deserves particular attention in an application using AI Tools. Changes to tool names, descriptions, schemas, ordering, or tool-specific instructions can affect the rendered prefix. The documented parallel_tool_calls setting can also change instructions about calling multiple tools in one turn. If a relevant change occurs before a cache breakpoint, the prefix after that change cannot match the existing entry. This does not mean that editing a request deletes an older cache entry; a subsequent request may still match that entry if it presents the same eligible prefix.
Why a continuing session still needs measurement
Agents API model calls follow the same prompt-caching behavior as Responses API calls. Reusing context within a session can preserve a shared prefix, but a session by itself does not guarantee a hit. For example, if a hypothetical agent preserves its earlier context while changing a tool definition near the beginning of the rendered input, the later portion may no longer match the entry it hoped to reuse. The deciding factor is the eligible matching prefix, not simply the fact that the conversation continued.
OpenAI points to the Prompt Caching Dashboard for monitoring cache read hit rates and the Prompt Cache Diagnostics tool for investigating misses. Those are more reliable checks than assuming a stable-looking conversation is being reused. Cost also needs careful interpretation: pricing varies by model, and input tokens use an uncached-input, cached-input, or cache-write rate. Cache-write pricing is not an additional charge layered onto one of the other input rates.
Which changes can interrupt reuse?
When an application sends similar requests, it is tempting to compare only the user’s latest message. For OpenAI API prompt caching, the more useful check is whether the full rendered prefix remains the same. That context can include developer messages, tool definitions and conversation history, as well as OpenAI-provided instructions. A change before an eligible breakpoint means the portion after that change cannot match the existing entry.
Tool configuration deserves particular attention in applications that send tools with each request. Changing a tool’s name, description, schema, order or tool-specific instructions can affect the prefix. The parallel_tool_calls setting can also change instructions about calling multiple tools in one turn. If you are comparing two otherwise similar requests, check these fields alongside the conversation content. Changing a request does not necessarily discard an existing cache entry; the question is whether a later request can still find an eligible matching prefix.
Who should check usage rather than assume a hit?
Developers using supported OpenAI models get prompt caching by default, but availability is not a promise that every repeated-looking request will reuse cached work. This matters for systems that assemble prompts from several parts: a stable-looking user question may sit behind changing instructions or tool definitions. It also matters for agents. Agents API model calls have the same prompt-caching behavior as Responses API calls, and reusing context within a session can preserve a shared prefix. Maintaining that session alone does not guarantee a hit.
For a real application, use the Prompt Caching Dashboard to monitor cache read hit rates. If reuse is lower than expected, the Prompt Cache Diagnostics tool is the source-supported place to investigate misses. For Agents API workloads, OpenAI also points to observability and usage information for session usage fields and subagent accounting. Those checks are more informative than treating a continuing conversation as evidence that every model call benefited from caching.
How should cost estimates handle cached input?
A cache hit changes how reused input tokens are charged, but there is no single cached-input price to apply across models. OpenAI says pricing varies by model and directs readers to its API pricing page for current cached-input and cache-write rates. The reduced cached-input rate can discount reused tokens by up to 95%; that upper bound should not be treated as the rate for a particular model or request.
One distinction is easy to miss in a cost estimate: cache-write pricing is not an extra charge added on top of another input rate. Input tokens use the applicable uncached-input, cached-input or cache-write rate. If you are evaluating a hypothetical workload with a large repeated instruction block, check the model’s current rates and observed cache reads before estimating savings. That keeps the estimate tied to actual reuse rather than the mere presence of repeated text.
What the documentation cannot tell you about your requests
OpenAI’s prompt caching guide explains the condition for reuse: a later request needs a matching rendered prefix and an eligible cache entry. It cannot tell you whether a particular request met that condition. A conversation may look unchanged to a reader while its rendered context includes developer messages, tool definitions, or other material beyond the visible question. Conversely, changing a request does not necessarily discard an existing cache entry; what matters is whether a matching prefix remains available at a breakpoint.
The guide also describes lower cached-input rates and less time spent processing reused input before a response starts. Those are reasons to investigate caching, not a measured saving for your application. The pricing that applies depends on the model, and the amount of reusable input depends on the requests. If you are evaluating a specific workload, treat its cache behavior, cost, and latency as questions to check against usage evidence rather than conclusions drawn from the feature description.
Where to check the primary evidence
Start with the Prompt Caching Dashboard, which OpenAI identifies as the place to monitor cache read hit rates. If the rate raises questions, OpenAI points to its Prompt Cache Diagnostics tool for diagnosing misses and improving reuse. These sources serve different purposes: the dashboard helps you see whether cache reads are occurring, while diagnostics helps you examine why expected reuse may not be happening.
For an Agents API workflow, consult OpenAI’s observability and usage documentation for session usage fields and subagent accounting. Agents API model calls follow the same prompt-caching behavior as Responses API calls, but a continuing session alone does not establish a hit. Finally, check the current API pricing page for the model’s cached-input and cache-write rates. The guide says cache-write pricing is not an extra fee layered onto another input charge: input tokens take the applicable uncached-input, cached-input, or cache-write rate.
FAQ
Can I confirm a cache hit just by comparing two prompts?
No. The visible text may not show the full rendered context, which can include instructions, tool definitions, and conversation history. Use the Prompt Caching Dashboard to check cache read hit rates, and the diagnostics tool if reuse is lower than expected.
Does a cache miss mean an earlier entry was deleted?
Not necessarily. OpenAI says a changed request does not automatically discard an existing entry. A subsequent request still needs the same prefix and an eligible matching breakpoint. A miss by itself does not identify which part of that condition was unmet.
Which price should I use when estimating future requests?
Check the current API pricing page for the model you intend to use, then distinguish input billed at uncached-input, cached-input, or cache-write rates. Do not apply the cached-input rate to every token merely because prompt caching is enabled; actual reuse needs to be checked.
