Evaluating an agent means judging how it reached a result, not only what it returned — because a multi-step agent can be right by many routes and wrong for reasons the final answer never shows. This is the practitioner's version: what to score, where a model belongs in the scoring and where it doesn't, and how to run it continuously once the agent is in production.
An LLM eval scores one input and one output. An agent produces neither.
The standard LLM evaluation compares a response to a reference answer, or has a judge model score it against a rubric. Both assume the unit under test is a single turn. An agent's unit of work is a run: a plan, some number of tool calls, retrieved context, intermediate reasoning, retries, and eventually an output. Scoring only the last of those is scoring the smallest part of what happened.
Three specific things break. There is no single correct trajectory — two runs can use different tools in a different order and both be right, so matching against a stored path measures conformity, not quality. The output can be right for the wrong reasons — an agent can produce a correct answer from retrieved evidence that contradicts it, and output-only scoring books that as a pass. And the expensive failures are invisible in the output: an agent that silently retried a failing tool eleven times and an agent that got it first try return the same string.
These four get used interchangeably and mean genuinely different things. The distinction that matters is what each one can tell you when something is wrong.
Tells you what happened. Traces, spans, latencies, token counts, the sequence of tool calls. It is the raw record. Observability answers "what did the agent do," and it answers nothing about whether that was the right thing to do.
Tells you when a known quantity moved. Error rate, p99 latency, spend per run, volume. Monitoring is alerting on metrics you already decided to watch. It cannot surface a failure mode nobody anticipated.
Tells you whether behaviour held on cases you anticipated. Valuable and necessary. But a test suite is a fixed set of expectations, and the interesting agent failures are the ones nobody wrote a case for.
Tells you whether the run was any good, against criteria stated in advance, with the evidence attached. It is the only one of the four that produces a judgement someone can act on without re-deriving it themselves.
The practical consequence: a green trace is not a passing run. Observability will happily show you a clean, complete, low-latency trace of an agent confidently doing the wrong thing. Every span succeeded. The outcome was still wrong. Nothing in the observability layer is built to notice that, because noticing it requires knowing what the run was supposed to achieve — which is exactly the thing evaluation supplies and observability does not.
The longer version of that argument, including why the trace is testimony rather than proof, is in evaluation vs. observability.
The test for whether a dimension is worth scoring: does it still mean the same thing after you improve the agent? A frozen expected answer fails that test — improve the agent so it surfaces something the reference never contained, and the suite books your improvement as a regression. These six hold.
Did the run achieve what it was asked to achieve — as opposed to terminating, which is what "done" actually reports. Execution is not outcome, and the gap between them is where most silent failure lives.
Was the path sane: no loops, no silent retries, no unrecovered errors, no escalation that got no response. Scored as properties of the path, never as a match against one blessed path.
Four separate questions, scored separately: was the tool allowed, was it the right tool, were the arguments right, and was the result used correctly. Collapsing them into one number hides which is broken.
Is every factual claim traceable to something the agent actually retrieved — checked against the sources it used on that run, not against a reference answer written in advance.
What the task required that the output never addressed. Omission is the failure users notice first and fixed test suites catch last, because nothing in the output signals absence.
Tokens, latency, tool calls — a correct answer at forty tool calls is a different product from the same answer at four. And the same task run repeatedly: passing sometimes is a distinct result from passing.
Treated as a measurement framework with its own vocabulary — seven properties that fail independently, and what an agent SLO can honestly promise — this is agent reliability.
A large share of what you want to know needs no model at all, and running one anyway makes the result slower, more expensive, and less repeatable than it has to be. Sort the checks by what they actually require:
The ordering is not stylistic. Deterministic checks are reproducible, so a result you dispute can be re-derived exactly; model judgements are not, so every one of them needs its evidence attached to be worth anything. Run the cheap exact checks first and a large fraction of runs never need the expensive ambiguous ones.
The same rule applied to one common case: should you ask an LLM how similar two things are?
An LLM judge is another agent, with its own failure modes, and "the model says it's good" is circular unless something breaks the loop. Three things break it: evidence — every judgement quotes the span it is about, so a human can check it in seconds rather than re-reading the trace; criteria fixed in advance, so the judge scores against something a person agreed to rather than whatever it inferred; and deterministic checks wherever the question permits one, so the model is never asked what code could have answered.
And before scoring anything against a reference set, check the reference set: how do you know your evaluation labels are correct?
Pre-release evaluation asks should we ship this. Continuous evaluation asks is the thing we shipped still working.
They are not the same job run on a schedule. The pre-release version runs against curated cases with known-good outcomes, and you control the inputs. The production version runs against real traffic, where you control nothing, most runs have no reference answer, and volume makes scoring every run expensive.
What changes:
Sampling strategy, the four sources of drift, and why most of what changes your agent is not a change you made: continuous evaluation.
Evaluation needs the run, not just the answer — and the run is what OpenTelemetry already carries if your agent is instrumented at all. Spans for each step, tool calls with their arguments and results, retrieved context, timings, token counts, errors and retries.
That matters for a practical reason beyond convenience: telemetry is portable and framework-agnostic. An evaluation layer that reads OTel is not coupled to LangGraph or CrewAI or Bedrock or whatever you migrate to next year, and it does not require wrapping your agent in someone else's SDK to observe it.
The depth of the telemetry is what separates a shallow evaluation from a deep one. If tool arguments are not recorded, tool-argument correctness cannot be scored by anyone, no matter what they promise. Instrumentation quality is the ceiling on evaluation quality.
Which attributes unlock which checks, and the three things that bite in production: evaluating agents from OpenTelemetry.
Agent TrustKit reads standard OpenTelemetry, scores runs against a contract you write in advance, and produces a report where every score traces back to the evidence behind it — deterministic checks where the question permits one, a model only where language is genuinely the problem. It runs on your own machine, against your own data, offline.
Evaluation produces scores. Scores are an input to a decision that somebody still has to make and put their name on — and the distance between "we measured it" and "we can ship it" is not closed by a higher number. It is closed by evidence organised so a reviewer can check the claim rather than trust it.
That is the difference between an evaluation and a clearance: one tells you how the agent scored, the other is something a person is willing to sign.
Bring a contract and the OpenTelemetry you already emit. We build the first one with you.
Book a call →[email protected]