AI Dictionary: How Models Generate Language

This dictionary focuses on LLM-based coding agents. Product behavior varies by model, API, and harness. Entries explicitly marked as informal describe working vocabulary used here, rather than universal technical definitions.

Dictionary overview

AI

Artificial intelligence: the broad field of building systems that perform tasks associated with intelligence, such as recognizing patterns, using language, planning, or making decisions. AI includes symbolic systems, machine learning, robotics, and more; large language models are one part of it. John McCarthy’s introduction describes the broader field.

The label also moves in everyday conversation. The AI effect names the tendency to stop calling a capability AI once it becomes familiar. That observation helps explain changing usage, but it is not a technical definition of the field.

For example, a spam classifier and a coding assistant can both use AI, even though only the assistant generates conversational text. When someone says “AI can’t reason,” ask which system and which task they mean.

LLM

Large language model: a language model trained at large scale to model patterns in language, often capable of generating text, following instructions, and producing code. “Large” has no fixed parameter threshold. Some related models also process images or audio.

For example, an LLM can draft a function from a description. Tools, project context, and an execution loop turn that capability into part of a coding agent. LLM, AI, model, and agent name different levels of the system.

Model

A model is a learned system, including its parameters and the architecture that uses them. A text-generation model predicts tokens and has no built-in access to files, commands, or tools. Examples of model families include GPT and Claude; a model needs a surrounding system to act as an agent.

By itself, a text-generation model cannot read files, run commands, browse the web, or remember prior interactions. The surrounding harness or product can provide tools, manage conversation state, and run a loop in which the model proposes actions and receives results. Multimodal models can also accept or generate non-text content.

Providers often offer models with different tradeoffs in capability, speed, and price; a smaller model may be faster and less expensive, while a larger one may perform better on some tasks. The tradeoff varies by model and workload. Some harnesses let you select or switch models during a session.

Being strict about the word also sharpens diagnosis. The same model can behave differently with another harness or context. Before attributing a disappointing result to the model, check the instructions, available tools, supplied context, and settings as well.

For example, the same model given a file-reading tool can inspect your repository; without that tool or supplied file contents, it can only respond from its existing input.

Parameters

The learned numerical values used by a model - often billions of them - adjusted during training. They encode patterns that shape the model’s outputs; this is sometimes described as the model’s parametric knowledge. Parameters are also called weights.

For a text-generation model, the parameters help transform tokenized input into a probability distribution for the next token. They are not a literal database or code lookup table. Information reflected in the model’s outputs may come from patterns learned during training, while information can also be supplied at inference time through context or retrieval.

For a given model version, ordinary inference leaves its parameters unchanged; feedback in a session does not automatically teach that deployed model. Parameters can be changed through further training or fine-tuning where supported, producing an updated or derived model. Project-specific information is often more simply provided through context or retrieval rather than model training.

For example, correcting an API signature in a chat can change the next answer because the correction is in context, while the model’s weights remain unchanged.

Training

Training adjusts a model’s parameters using data. Pre-training often teaches broad patterns through prediction tasks, while post-training can refine behavior such as instruction following. Providers commonly train their own models, and some also offer fine-tuning; researchers and organizations can train or adapt models when they have the data, expertise, and compute.

One common pre-training method repeatedly gives a model text and adjusts its parameters to improve predictions of later tokens, using very large datasets. Other training stages can use different objectives and data. The resulting parameters encode learned patterns rather than a conventional database of facts.

Providers often state a knowledge cutoff for a model version, so its training knowledge may not reflect a library release from last month. When a model lacks knowledge of your codebase or internal APIs, supplying current documentation in context or through retrieval is often more practical than fine-tuning, though fine-tuning may suit some goals.

For example, training on many code examples can teach common patterns; reading your private repository during a task supplies context for inference rather than retraining the model.

Fine-tuning

Further training that adapts a model’s parameters for a particular behavior or task. It can improve a response format, style, or specialized performance when appropriate data and evaluation are available.

For example, train on approved examples to produce a consistent support-response format. Retrieve the current support policy at inference time so the answer can use updated facts. Fine-tuning and retrieval serve different purposes; neither guarantees correctness or permanently stores every supplied fact.

Inference

Running a trained model to generate output. For an autoregressive text model, this means generating tokens from the supplied context while its learned parameters remain fixed. Inference is repeated whenever a model is used; API costs depend on the provider, model, token categories, and any other billable features or product plan.

A model’s life splits into two phases:

Phase	When it happens	What it does	Parameters
Training	During development of a model version	Produces or adjusts parameters using training data	Being written
Inference	Every time anyone uses the model	Runs the frozen parameters over your context to generate tokens	Read-only

Ordinary inference does not update the model’s learned parameters, so a correction in one interaction does not by itself change the model for the next one. The model has no built-in memory of that exchange; continuity can come from context supplied again or from a separate memory or conversation-state system.

With usage-based APIs, billing commonly depends on the input and output processed for each request, though rates and billable categories vary. An agent may make multiple requests while using tools, so the amount of context and number of requests can affect cost as well as quality. Some APIs manage conversation state for you, while others expect the harness to provide relevant context on each call.

For example, ask the model to explain a test failure: that response is inference. Updating the model with a training dataset is training.

Token

For text models, a token is a unit of text processed or generated by the model. A token may represent part of a word, a whole word, punctuation, or other text; token counts affect context use and may affect cost and latency.

For text input, a tokenizer maps text into token IDs using a vocabulary and encoding associated with the model. Tokenization methods and vocabularies vary, and are generally fixed for a given model version. Text models process these token representations and generate text token by token; multimodal models may also process or produce images, audio, or other content.

For English prose, three-quarters of a word per token is a rough estimate, not a reliable conversion. Code and unusual strings can tokenize differently from ordinary prose, and the result depends on the model’s tokenizer. A short-looking file with many unusual identifiers or encoded data may therefore use more tokens than expected.

Token counts are one measure of input size, output, and context-window use. Many APIs price input, output, and cached tokens at different rates, but pricing models vary and some products use subscriptions or other billing units. Text generation speed is often reported in output tokens per second.

If a toy tokenizer splits unhelpful into un and helpful, that word uses two tokens. This is illustrative; use the relevant tokenizer for actual counts.

Next-token prediction

Autoregressive text generation produces output one token at a time: the model predicts a next token from its current context, adds it, and repeats. This describes text-generation models; multimodal models can also process or generate other kinds of content.

At each step, a text-generation model assigns scores or probabilities to possible next tokens. A decoding method selects the next token; it may sample among candidates or choose the highest-scoring token. Sampling can make repeated outputs vary, while some systems can also vary because of provider or serving behavior.

This helps explain why fluent output is not necessarily accurate: next-token prediction does not itself verify claims against the world. Earlier generated text also becomes part of the context for later generation. Since text is emitted sequentially, output-generation speed can affect how quickly a response arrives.

For example, after The capital of France is, a model may strongly favor Paris. That plausible continuation still is not a live check of a source.

Effort

Some providers and harnesses expose an effort or reasoning setting that influences how much computation or reasoning a model uses for a request. The available controls and their effects vary by model. Any internal reasoning tokens are generated at inference time; they may not appear in the visible answer.

Higher effort can increase latency and, on usage-priced services, token use and cost; the effect depends on the model and provider. Some services count hidden reasoning tokens as output tokens, while billing and reporting details vary. The general tradeoff is more computation for a potentially stronger result, with added time or expense.

Some providers and harnesses expose effort settings on a scale. The names and behavior vary; these labels illustrate a common pattern:

Level	What it’s for
Low	Mechanical edits, lookups, well-specified changes with one clear path.
Medium	Everyday coding with several steps.
High	Tricky bugs, design decisions, multi-step plans.
Max	The hardest problems, where a wrong answer is expensive to unwind.

Too little effort can miss steps on a difficult task; more effort can add latency and cost without improving a simple one. Higher effort does not guarantee correctness. These levels are illustrative, not a universal scale or guaranteed task-to-setting mapping.

Match effort to the task, not the session. Turn it up for the part that’s genuinely hard to reason about, and back down for the rote work around it.

For example, an effort increase may help investigate a race condition; a mechanical identifier rename may not benefit. Compare verified outcomes rather than assuming the higher label wins.

Sampling and temperature

Decoding determines how a model’s next-token scores become generated tokens. Sampling chooses among candidates; temperature changes how concentrated that choice distribution is. Top-p and other controls can restrict the candidates considered.

For example, two sampled explanations may phrase the same answer differently. Lower temperature often reduces variation, but does not guarantee correctness or identical end-to-end runs. Some models do not expose these controls, and temperature is distinct from reasoning effort.

The OpenAI Responses reference documents its sampling controls; check model compatibility before using them.

Non-determinism

The same input can produce different output, though it is not guaranteed to do so. With sampling enabled, repeated runs may differ even when the prompt and context are unchanged.

Variation can come from sampling choices as well as provider or harness behavior. Sampling adds randomness by design; deterministic decoding can reduce that source of variation, but may not make every end-to-end system reproducible. A different early token can also lead generation down a different path.

Repeated runs may vary in quality, and a retry can produce a better or worse result. For consequential work, verify the output rather than assuming an agent will repeat the same behavior. Automated checks can catch some errors, while review may still be needed for qualities those checks do not measure.

Be careful not to over-interpret a small number of runs. A short streak of disappointing results does not by itself show that a model or service has changed; compare repeated results and check for changes in the model, harness, settings, or context.

For example, two runs of “write a parser” may choose different algorithms. Evaluate both against the same tests instead of expecting identical code.