AI agent evaluation verifies whether an agent reached the correct outcome and whether the tool calls and retrievals that produced it were appropriate. A team that scores only the final output may never check how the agent arrived at that output, because an agent can return a correct-looking output through a broken sequence of tool calls.
This guide covers what to measure at each layer of an agent run and how five evaluation methods differ in what they catch and miss.
For the production-tracing side of the same problem, explore AI agent observability, which answers the question of what is happening during a live run.
What is AI agent evaluation?
AI agent evaluation scores an agent against your own tasks, using your own tools and your own data. It checks whether the agent produced the correct outcome by following the steps you defined. AI agent evaluation measures the system you deployed, rather than an abstract model.
There’s also a test for model benchmarking that scores a model on a fixed public task set, with the same set used for everyone. Agent evaluation scores your agent on your own workflows: a collections call that updates your CRM, or a rebooking that honors your fare rules.
A public benchmark cannot catch the failures that matter in your own workflows, because it never tests your internal tools or your compliance rules. Agent evaluation catches the agent who calls the wrong internal tool and the agent who breaks a disclosure rule that your compliance team has set.
Why output accuracy is the wrong metric
Output accuracy is the wrong primary metric for an AI agent, because two different failures both score as correct on an answer-only check. An agent can reach the right answer through a broken process, or it can execute every step correctly and still fail to finish the task.
The agent fails to follow the predefined steps
The agent completes the task using the wrong path because the check only looks at the final answer, not the path that produced it. If the agent skips a required step, reads the wrong record, or calls the wrong tool, it can still produce output that looks correct in that run.
For instance, a support agent can ask for a customer's checking balance and read the savings account instead. The two balances may match that day, but the wrong lookup returns a right-looking number.
Common reasons this happens:
- Wrong data source: The lookup pulls the wrong record, account, or field, not the one the task specified.
- Wrong tool call: The agent calls a tool that doesn't match the step it is on, or passes it the wrong argument.
- Skipped step: The workflow advances to the next step before a required check, like an identity or policy verification, completes.
- Lost context: An earlier value in the conversation carries forward instead of the value a later turn corrected.
The agent follows every step but never produces an output
This is a different scenario wherein an agent can pass every step-level check and still fail to complete the task. A refund agent would choose the correct tool, pass valid arguments, receive a valid response at each step, yet still stop before the final write to the payment system.
Every step-level score reads as correct. The refund never posts, and the customer's payment record never updates.
Here’s why this happens:
- Muted write failure: The update call returns without confirming the write completed, and the agent never checks.
- Confirmation mistaken for completion: The agent treats a valid response from an intermediate step as the end of the task.
- Missing final step: The workflow ends one step before the action that commits the change, like the write to the payment or CRM system.
- Turn or cost limit hit: The agent reaches its turn or cost ceiling before the last step runs, and it returns whatever it has.
What to actually measure?
Agent evaluation operates at three layers, and each layer scores a different part of the run. The step layer scores a single action, the trace layer scores whether the run reached its objective, and the outcome layer scores the business result the run produced.
A team that measures only one of these layers cannot see what happened at the other two. Here’s what to measure across layers.
The step level scores each action
Step-level evaluation scores each action inside a run, before the run finishes. A wrong action can still produce a right-looking output, so a business that checks only the final answer never sees the step that almost failed.
The following steps are performed to verify the correctness of this step.
- Tool selection: Checks whether the agent calls the explicitly requested tool, built for this step, rather than a similar one that returns the incorrect result.
- Tool arguments: Verifies if the agent passes the correct account, order, or record ID into the tool it called.
- Retrieval quality: Confirms if the document, policy, or record the agent pulled actually match the question it was answering.
- Reasoning coherence: Runs a check to make sure each step follows logically from the one before it, without contradicting an earlier decision in the same run.
The trace level scores the whole run
Trace-level evaluation scores the whole run an agent made against the objective the agent was given, and not just each action on its own.
A business needs this layer because a run can pass every step and still fail the task: a refund agent can call every right tool and still stop before the payment record updates. Trace-level scoring catches that gap between correct steps and a completed task. This is also the layer that a compliance team reads for reporting.
It requires aligning these factors in a test:
- Task completion: To check whether the run reached the state the task required, such as a posted refund or a confirmed booking.
- Policy adherence: Verify whether they follow the rules they were built to follow, such as required disclosures or escalation triggers.
- Context retention: Determine whether an agent retained a value or instruction across turns rather than losing it partway through.
- Turns and cost: How many turns did the run take, and what did it cost, compared to the same task done a shorter way?
The outcome level scores the business result
Outcome-level evaluation measures the business result the run was built to produce, usually represented as a number that leadership can read on a dashboard. It checks whether the agent's work moved a metric the business tracks, not whether any individual step or the underlying trace was correct.
The following checks occur here:
- Resolution rate: What share of runs closed the task without a person stepping in.
- Escalation rate: What share of runs got handed to a person, and why.
- Human correction rate: How often a person had to fix what the agent produced after the fact.
- Downstream business metric: Represents the number the workflow was built to move, as collections recovered, bookings completed, or tickets closed.
Most teams stop at the outcome layer, because those are the numbers a dashboard already shows the business. LangChain's State of Agent Engineering survey of 1,340 teams building production agents found that 89% have added some observability to their agents. Still, only 62% can trace an individual step or tool call, and just 37.3% run evaluations against live production traffic.
How evaluation methods compare
A misaligned LLM-as-judge would produce confident but incorrect scores, and this is the most common evaluation failure in practice. Before a team trusts a judge model, it has to check that model's scores against human labels on a sample, and repeat the check whenever the judge model changes.
| Method | What it catches | What it misses | Cost | When to use it |
|---|---|---|---|---|
| Human review | Nuance and judgment on open-ended output | Scale, since a person reviews only a few runs | High | Calibration and hard edge cases |
| Rule-based assertions | Known hard constraints, like a forbidden action or a bad format | Anything the rules don't encode | Low | Policy and safety gates |
| LLM-as-judge | Open-ended quality at scale | Whatever an unaligned judge scores as wrong | Medium | High volume, once the judge is checked against human labels |
| Regression suites from production failures | A repeat of a failure that already happened | A failure mode nobody has seen yet | Low to maintain | Stopping a known bug from returning |
| Simulation | Behavior across many generated cases before launch | The real distribution of live traffic | Medium | Pre-deployment coverage |
Offline evaluation versus continuous evaluation
A pre-deployment evaluation set is tested before an agent goes live to catch the failure modes a team already knows about. The continuous evaluation is meant to run on live production traffic after launch, catching the failure modes nobody thought to test for.
Here’s the complete difference between the evaluation and continuous evaluation.
| Process Aspect | Offline evaluation | Continuous evaluation |
|---|---|---|
| When it runs | Before the agent goes live | After launch, on live production traffic |
| What it catches | Failure modes a team already knows about | Failure modes nobody thought to test for |
| Test cases come from | A curated eval set built ahead of time | Real runs the agent that is currently handling |
| Role in the loop | Blocks a known failure from reaching production again | Adds the next production failure to the eval set |
| Risk if left unmaintained | Reports passing scores against tasks the agent no longer runs | Has no risk of going stale, since it always tests current traffic |
What connects both these evaluations is a single loop of a production failure. For example, a misread account from a live run becomes a new evaluation case, and that case becomes a regression test that prevents the same failure from recurring. This loop also keeps a pre-deployment suite useful after launch, rather than letting it go stale the day the agent ships.
An evaluation set becomes outdated whenever a team changes the agent's workflows, tools, and policies. A suite that the team stops updating is worse than no suite at all, because it reports passing scores against tasks the agent no longer runs. That suite simulates false confidence instead of real coverage. A team that runs continuous evaluation and A/B testing on live traffic keeps the suite matched to the agent's current workflows.
Choosing evaluation tooling
For an enterprise, choosing evaluation tooling means deciding where agent traces and data live, not comparing features. A bank or an airline needs to know who can see a call transcript before it picks a vendor.
Categories
The market breaks into four categories, based on where the suite lives and who runs it.
Evaluation-first platforms treat the eval suite itself as the core product and add tracing around it.
- Braintrust
- Galileo
- Maxim
- Comet
Open-source libraries run inside your own code and CI, so no vendor stores your traces.
- DeepEval
- MLflow evaluation tools
Tracing platforms with an evaluation layer capture the trace first, then score against it.
- Langfuse
- Arize Phoenix
Built into an agent platform means the platform running the agents scores them itself, so the buyer doesn't assemble a separate stack.
Data residency, whether traces can leave your environment, and who maintains the suite decide the choice, not feature count.
HappyRobot falls into this fourth category, with its governance layer able to score every agent on the platform, so an enterprise buys the evaluation rather than building it.
