AI Dictionary: Workflows, Verification, and Review

This dictionary focuses on LLM-based coding agents. Product behavior varies by model, API, and harness. Entries explicitly marked as informal describe working vocabulary used here, rather than universal technical definitions.

Dictionary overview

Human-in-the-loop (HITL)

Human participation in a system’s decision or feedback loop, through supervision, approval checkpoints, review, or collaboration. This dictionary focuses on active pairing during an agent session, a narrower form of HITL.

The contrast is with AFK work, where the agent runs unattended and you judge the result afterwards. Human-in-the-loop means catching problems while they’re still cheap: you see the agent reach for the wrong file, misread the requirement, or start down a dead end, and you redirect it in one sentence — rather than discovering twenty minutes of confident work built on that mistake. Agents can miss that they are off track; an unattended run may keep building on a wrong assumption unless its checks or stop conditions catch it.

Which pattern fits depends on the work. Well-specified, low-risk, easy-to-verify tasks suit AFK. Tasks that are ambiguous, irreversible, or where you’d struggle to review the finished result — a schema migration, a tricky design decision, anything touching production — suit staying in the loop. The judgement call is essentially: how expensive is a wrong turn, and how late would you catch it?

Some work is in-the-loop by nature, because your reactions are the input. Grilling only works with you there to answer the questions; prototyping only works with you there to react to the artifact.

Staying in the loop costs your attention, which is the scarce resource. Part of getting better with agents is moving more work safely out of the loop — with plans, automated checks, and human review at the end instead of supervision throughout. A software factory takes this further by starting sessions from triggers, so even kicking off the work doesn’t need you.

For example, watching an agent propose a database migration and correcting its assumption before execution is active pairing. Approving the migration at a defined checkpoint is also HITL, with less continuous supervision.

AFK

Away from keyboard. A working pattern where the user kicks off a session and leaves the agent to run unattended. The throughput multiplier of AI coding — many AFK sessions can run in parallel while you sleep, eat, or work on something else. Needs enough pre-authorized capability to finish the task, explicit stop conditions, and appropriate isolation; it does not require bypassing all approvals.

When you’re not there, the agent handles ambiguity differently. While you’re watching, an ambiguous decision surfaces as a question and you answer it; once you’ve walked away, an agent may pick a default, stop, or leave a question for your return, and if it guesses, later decisions can build on that guess. One failure pattern is coming back to hours of finished, confident work built on a wrong call made in the first ten minutes. The work isn’t sloppy — it’s coherent, just coherent about the wrong thing.

Since you can’t give input during the run, give it before and after instead. Before: resolve the ambiguity up front — a grilling session, a written spec — so there are fewer gaps for the agent to fill alone. During: automated checks and automated review stand in for the attention you’re not giving, failing fast on what can be caught mechanically. After: the run ends in something reviewable — a PR, not changes already merged. AFK does not require removing human review; review can wait until the end, which is why what arrives at the end has to be worth reviewing. This is also why AX matters most in AFK runs — with no one watching, the environment, instructions, and checks support the agent without live supervision.

For example, ask an agent to fix one failing test, run the relevant checks, and leave a diff for review. Tell it to stop if the fix requires changing the public API rather than guessing unattended.

Automated check

A verification implemented in code — tests, type checks, lints, build rules, or hook checks. It applies explicit assertions or rules rather than an open-ended model judgment. Repeatability under controlled inputs is the goal, not a guarantee for every test. A useful failure gives the agent feedback it can investigate without waiting for a person. A flaky test is still a check, but an unreliable one; randomized and environment-dependent tests need controlled seeds or other measures to interpret results.

Self-correction works as a loop. The agent makes a change, runs the check as a tool call, and the failure output lands in its context window — a type error with a file and line, a failing assertion with expected and actual values. That can be enough to investigate and fix the problem, then run the check again without human intervention. A failure report may still need more evidence, and a repeated pass does not prove the fix is sound. Determinism is what makes the loop trustworthy: the same controlled inputs and environment should produce the same verdict, so a pass means something. A flaky check poisons this — the agent “fixes” code that was fine, or retries past a real failure.

This is why good checks are a large part of a codebase’s AX. An agent in a repo with strict types, a fast test suite, and a linter has useful ways to catch mistakes before you see them; an agent in a repo without them has less automated feedback to challenge its output. The difference matters most in AFK runs, where checks can provide feedback without a person watching. But a check only catches what it asserts — green checks mean the asserted properties hold, not that the code is right. The judgement-shaped gaps are what automated review and human review are for.

For example, a unit test asserting that empty input returns an empty list is an automated check. A test that depends on a live service can vary when that service changes, even if the test code is unchanged.

Evaluation (eval)

A structured measurement of model or agent behavior on defined tasks using specified success criteria. An eval can use executable checks, human judgments, model-based graders, or a combination. It is a process for measuring quality, not synonymous with automated review.

For example, run an agent repeatedly on ten representative parser bugs, then measure whether it fixes the behavior, preserves unrelated tests, and stays within scope. Record the model, harness, settings, dataset, and environment so comparisons are meaningful.

A benchmark score may not predict performance on your repository. Inspect failures, use held-out cases, and account for run-to-run variation. Anthropic’s eval guide discusses agent tasks, trials, and graders.

Automated review

An agent reviewing another agent’s work, often with a different model or system prompt. It forms a judgment rather than applying only fixed rules, and results may vary between runs. Runs anywhere — pre-merge on a PR, post-hoc on commit history, mid-session as a subagent. An LLM-as-judge in CI is automated review, not an automated check; what the assertion does decides the category, not where it runs.

Separation from the working context can make review more independent. Asking the authoring agent to review its own work can find problems too, but can preserve its original assumptions — the session that produced the bug also contains the reasoning that produced it, and the agent reads its own conclusions back as confirmation. A reviewer with fresh context avoids some of those inherited assumptions, though it can still share the same blind spots: it sees the diff the way a stranger would, which is what review depends on. A different model or a review-specific system prompt may help — different blind spots, and a system prompt scoped to what you actually care about (security, API contracts, performance) rather than a vague “look for problems”.

It slots between the other review layers. Automated checks are deterministic and catch what can be asserted mechanically; human review is expensive and scales worst. Automated review sits in the middle: it can catch judgement-shaped problems — a misleading function name, a missed edge case — at machine cost. Because it’s non-deterministic, it can miss things and flag non-issues; treat it as a filter that raises the floor before a human looks, not a gate that replaces one.

For example, a fresh reviewer flags that a new route skips authorization. Confirm the execution path and fix it if valid; do not accept or dismiss the finding just because a model wrote it.

Human review

A person inspecting the agent’s actual output and relevant evidence, then forming a judgment. For code, this includes the diff, surrounding implementation, and validation results. Reading the diff or the changed files counts; reading the agent’s description of what it did does not — narration is not the artifact. The description is a secondary source, written by the party being reviewed; the diff is the primary source, and review means reading it.

Agents can raise the volume of code produced, making human review a bottleneck. One useful idea is layering different review strategies. Automated checks catch the mechanical failures, automated review can flag issues requiring interpretation, and human review is reserved for what only you can judge — whether the change is the right change, whether the approach fits the codebase, whether this should exist at all.

Review is also cheaper earlier. Reading a plan before work starts, or a small diff mid-flight, takes minutes; excavating a finished branch after an AFK run takes longer. Where you place the review checkpoint is a human-in-the-loop decision, not an afterthought.

For example, read the actual authorization change and its failure-case tests before approving it. An agent saying “security is unchanged” is a claim to inspect, not the review itself.

Vibe coding

A working pattern where the user accepts the agent’s code without human review. The diff is treated as opaque — what matters is whether the program behaves, not what’s inside. Automated review and automated checks may still run; vibe coding is silent on both.

The term comes from Andrej Karpathy, who coined it in early 2025: the stance is to describe what you want, accept what comes back, and judge it by running it while largely setting aside inspection of the implementation.

Vibe coding trades inspection for speed. Reading diffs can become a bottleneck in agent-driven work, so dropping it removes the main bottleneck. For code whose failures are cheap — prototypes, one-off scripts, internal tools — that’s a reasonable trade. The risk scales with the code’s lifespan and stakes.

Potential costs appear later. Vibe-coded changes accumulate into a codebase nobody has read, and behavior may have been the main thing checked — so anything behavior doesn’t surface, like a secret written to logs, a missing edge case, or quietly wrong data handling, can ship unseen. Debugging may become the first careful reading of the code. With human review gone, whatever automated verification still runs — tests, types, automated review — is the only gate the code passes through. Applied to an automated production workflow, this overlaps with what this dictionary calls a dark factory.

Software factory

In this dictionary, a system of work where agent sessions are started by triggers — an issue being created, a schedule, a CI failure, another session finishing — rather than by a human, so more work runs AFK and human attention is spent on the human-in-the-loop decisions that remain.

In a manual agent workflow, each session waits for someone to start it. Even fully AFK work waits on a person to open the session, point it at the ticket, and set it going. Teams want to ship more than that allows. A factory takes the human out of starting sessions, and not necessarily out of anything else.

Common triggers and the sessions they start:

Trigger	Session it starts	Example
Issue created or labelled	Exploration, bug fix, implementation	An issue labelled ready-for-agent gets a session that opens a PR
Schedule (cron)	Recurring maintenance	One lint rule fixed per night
CI failure or monitoring alert	Diagnosis, fix attempt	A failing build on main gets a session that finds the breaking commit and proposes a fix
Another session finishing	Follow-on work	A PR opened by one agent triggers an automated review, whose comments trigger a fix-up session

A factory doesn’t have to cover the whole software process. Even one recurring job can illustrate this pattern; broader uses of “software factory” include other forms of organized or automated production. Starting that small is useful: a narrow loop produces small, similar PRs, and reviewing them shows how far the loop can be trusted before it’s widened.

Humans can sit anywhere in a factory — writing and labelling the issues that trigger sessions, approving a plan before implementation starts, doing human review before merge. Deciding which of those decisions stay human is the main design question. A codebase, or part of one, where no human reviews the factory’s output is a dark factory.

Dark factory

In this dictionary, a software-production workflow in which agents make changes and no human routinely reads each implementation before it proceeds. The phrase borrows from automated “lights-out” manufacturing; its exact scope varies in industry usage.

For example, an unattended maintenance pipeline opens, tests, and merges dependency patches under a pre-approved policy. Humans define the policy and investigate exceptions, but do not read every patch.

Lack of per-change human review increases reliance on automated checks, evaluations, observability, rollback, and limits on what can change. Those controls can miss problems. Humans may still review requirements, audit samples, or handle incidents, so a dark factory need not mean nobody ever inspects code.

When something breaks, traceability still matters: recorded commits, diffs, logs, and release identifiers can show what changed even if no one read it beforehand.