AI Dictionary: Requests, Tokens, and Caching
- Published
How providers process requests, account for token usage, reuse prompt prefixes, and manage conversation state.
This dictionary focuses on LLM-based coding agents. Product behavior varies by model, API, and harness. Entries explicitly marked as informal describe working vocabulary used here, rather than universal technical definitions.
Model provider
The service or runtime that makes a model available for inference. It may be a remote API or a local runtime such as Ollama or llama.cpp. A harness sends requests to the provider or runtime and handles the responses.
For a hosted model, the provider operates the serving hardware and receives requests through its service. Rate limits, service capacity, and outages can therefore affect results or availability independently of the model and harness. For a local runtime, the model may run on the user’s own hardware instead. When a hosted agent stalls or repeatedly errors, checking provider status is one useful diagnostic step.
Providers set their own availability and commercial terms, which may include different rates for input, output, and cached tokens. The serving provider and model developer can be different organizations; cloud platforms and aggregators may offer models developed by others.
Running a model locally can provide more control over where model inputs are processed and may avoid per-token API charges. Capabilities and hardware requirements vary, and connected tools, telemetry, or other services can still send data off-device.
Model provider request
One exchange in which a harness or application submits input to a model provider and receives a response, such as a tool call or final answer. A user turn can involve multiple requests when tools are used; after a tool result, the harness may send a follow-up request so the model can continue.
The model needs relevant context for each response, but how that context is supplied depends on the API. A harness may resend conversation history and tool results, or a stateful API may retain conversation items and accept a conversation identifier. Provider-side storage does not give the model built-in memory. Prompt caching can reduce the processing cost of eligible repeated prefixes, but does not eliminate usage charges for prior context.
For usage-priced APIs, requests commonly report token usage and costs are calculated from the applicable rates and billable features. A short user message can lead to substantial usage if it triggers many requests or requires large context; subscriptions and other product plans may bill differently.
It’s worth keeping the request distinct from the turn. A turn is one exchange with you, and a single turn — “fix the failing test” — plays out as a chain of requests:
| Request | Model returns | Harness then |
|---|---|---|
| 1 | Tool call: run the tests | Runs them and returns the failure output |
| 2 | Tool call: read the test file | Returns the file contents |
| 3 | Tool call: read the source file | Returns the file contents |
| 4 | Tool call: edit the source file | Applies the edit and reports the result |
| 5 | Tool call: run the tests again | Runs them and returns the pass output |
| 6 | Final answer: “fixed, tests pass” | Shows it to you |
This example uses six requests in one turn. Depending on the API, each request may receive context from the harness or use provider-managed conversation state. When estimating usage, consider both the number of requests and the context processed by each.
Input tokens
Tokens supplied to a model for a request, including any instructions, conversation content, and tool results included in its context. Usage-based APIs commonly price input and output tokens separately, but which category costs less and by how much depends on the provider and model.
In some AI coding workflows, input tokens can make up a large share of usage because requests include substantial instructions, history, and tool results. A harness may resend history, summarize or compact it, or rely on provider-managed state; the exact input and billing behavior depends on the product and API. A single request can process far more input than the visible user message suggests.
Prompt caching can reduce the price or processing time for eligible repeated input prefixes, according to provider-specific rules. If context usage remains high, reducing irrelevant history or compacting between tasks may help.
For example, a ten-token user question can be part of a request with thousands of input tokens once instructions, history, and file contents are included.
Output tokens
Tokens generated by the model, including visible text and, for some models and APIs, non-visible reasoning or other output. Usage-based services often price output differently from input; the rates and ratio vary by model and provider.
Generated prose, code, and tool-call content may count as output usage. Some reasoning models also report and bill non-visible reasoning tokens as output tokens. Whether those tokens are exposed or billed that way depends on the model and service; higher reasoning settings can use more tokens.
Output generation is often sequential, so longer responses can take more time. Total latency also depends on input processing, reasoning, provider load, network time, and any tool calls; a long wait does not necessarily mean a long answer is coming.
For example, a tool call asking to read a file is generated output; the returned file contents become input for a subsequent inference.
Prefix cache
Prompt caching is a provider feature that can reuse computation for an eligible shared prefix across requests. Matching rules, minimum prefix lengths, cache lifetime, latency effects, and pricing vary by provider and model. Cached input may be billed at a different rate from ordinary input.
Caching can help when successive requests contain an eligible, unchanged prefix, as often happens when a conversation grows by appending new messages. The provider may reuse computation for that prefix, subject to its cache rules. Conversation state and prompt caching are separate features: a stateful API can retain conversation items without making every item a cache hit.
Cache entries can expire, but their lifetime varies by provider, model, and configuration; some services offer multiple cache durations. After expiry, a later request may need to process the prefix again. Check the provider’s current documentation for the applicable behavior and pricing, such as OpenAI’s prompt-caching guide.
For example, keep stable instructions and tool definitions before changing task data when the provider caches matching prefixes. Editing the first instruction can invalidate reuse of later content.
Cache tokens
Cached input tokens are the portion of a request that a provider identifies as eligible for prompt-cache reuse. The provider may report and price them separately from uncached input. Cache availability and savings depend on the provider, model, request, and cache rules.
This matters when a workflow submits a repeated context prefix. Depending on the API, the harness may send that context directly or refer to provider-managed conversation state. Eligible cached input may receive a discount, but the amount varies and caching does not necessarily reduce every request’s total cost or make a long session inexpensive.
A simplified example shows prefix matching. Assume each letter stands for an eligible block, the first request populates a cold cache, later requests arrive before expiry, and the provider can reuse each matching block:
| Request sends | Cached | Uncached input | Why |
|---|---|---|---|
AB |
nothing | AB |
Cold cache — no reusable entry |
ABC |
AB |
C |
AB is an exact prefix of the previous request |
ABCD |
ABC |
D |
Prefix still intact |
AXCD |
A |
XCD |
An edit changed B to X; the match fails there |
Many prompt caches match unchanged prefixes, so a change near the beginning can reduce how much later content matches. The exact matching, cache breakpoints, minimum lengths, expiry, and billing rules are provider-specific. If usage changes unexpectedly, inspect the provider’s usage report and cache documentation where available.
Stateless
Does not retain state between interactions. For ordinary LLM inference, a model does not automatically remember a previous request or update its parameters from it. Continuity depends on information made available to the next inference.
That does not mean every API request must resend the entire transcript. A harness may send selected history, a provider may store conversation items, and a serving system may reuse cached computation. These are forms of system state, distinct from learning new parameters. OpenAI’s conversation-state guide shows both client-managed history and provider-managed conversations.
For example, tell a model “use TypeScript,” then make an independent request with neither that instruction nor a reference to stored history. The new request has no reason to preserve your choice. Put the convention in project instructions that future sessions load if it needs to persist.
Stateful
Retains information between interactions. A harness can keep conversation history across turns, a provider can store conversation items, and a memory system can carry selected information across sessions. Counterpart to stateless.
Separate the kinds of state:
| Layer | Example state | What it means |
|---|---|---|
| Model parameters | Patterns learned in training | Ordinary inference does not update these |
| Inference runtime | Cached attention keys and values | Reuses computation; not learning from the user |
| Session or provider | Messages, tool results, compaction state | Supplies continuity for later inference |
| Memory system | Saved preferences or project notes | Retrieves selected information across sessions |
| Environment | Files, database records | Persists only according to its storage lifecycle |
For example, a saved preference can make a new session use TypeScript without the model having learned that preference in its weights. Persistent files can survive a session, while files in a disposable sandbox may disappear when it is destroyed.