Observability answers what did the agent do. Evaluation answers should it have done that. The gap between those two questions is where every expensive agent failure lives — and no amount of tracing closes it, because a trace is produced by the system you are trying to check.
A trace with every span green is entirely compatible with an agent that did the wrong thing, confidently, end to end.
Observability is instrumentation: spans, timings, tool calls, token counts, errors. It is a faithful record of execution, and it is genuinely necessary — you cannot evaluate a run you cannot see. But every question it answers is a question about mechanics. Did the call succeed. How long did it take. What did it cost.
Evaluation asks a question observability has no vocabulary for: was this run any good. Answering it requires something the telemetry does not contain — a statement of what the run was supposed to achieve. That statement has to come from a person, in advance. It is not recoverable from the trace at any level of instrumentation detail, because it is not a fact about what happened.
This is the part that gets missed, and it is the strongest argument for keeping the two functions distinct rather than treating evaluation as a feature of an observability product.
The trace is generated by the system under investigation. It records what the agent reported doing. Those diverge more often than they should:
In all four the observability layer is working perfectly and reporting success. Detecting any of them requires reading the content of the spans against a stated expectation — which is evaluation, not tracing.
The fourth case deserves separate attention because no amount of instrumentation fixes it. Spans exist for work that was attempted. Work that was never attempted produces nothing to observe — the trace is complete, correct, and silent about the most important thing in the run. Only comparing the output against what was required surfaces it, and the requirement lives outside the telemetry by definition.
Gives you the run. Cannot tell you the run was wrong, because it has no representation of what right would have been.
Watches known quantities and pages you when they move. Cannot surface a failure mode nobody thought to threshold — and agent failures are novel by nature.
Confirms behaviour on cases you wrote down. Cannot cover the open-ended input distribution an agent actually meets in production.
Scores runs against criteria fixed in advance, with evidence. Cannot function without observability underneath it — it needs the run to read.
The relationship is layered, not competitive. Evaluation consumes observability. Teams that have invested in tracing are the ones best positioned to evaluate well, because the raw material is already there and already portable. The mistake is expecting the tracing investment to have answered the quality question by itself.
Tool error rate tripled in the last hour. That is a threshold on a known quantity, and it is exactly what monitoring is for. It tells you something moved; it does not tell you what.
You open the traces and see which tool, which spans, what latency, what sequence. Now you know the mechanics of what happened, across however many runs you care to read.
Were the answers still grounded, still complete, still inside policy while that was happening? The error rate could have tripled with no quality impact — retries absorbed it — or output quality could have collapsed while the error rate looked fine. Those are opposite incidents and the first two layers cannot distinguish them.
The failure mode that never pages anyone: every metric nominal, every span green, quality quietly degrading because a retrieval corpus went stale. Nothing in monitoring or observability is built to notice. Continuous evaluation is the only layer that reports it.
The observability vendors are excellent at what they do, and several now ship evaluation features. The question worth asking of any of them is narrow: where do the criteria come from, and what evidence backs a score?
If criteria are inferred from the trace, the system is grading the agent against its own behaviour, and it will report a confidently wrong run as fine. If a score arrives without the evidence behind it, you have replaced trusting the agent with trusting the scorer — which is the same problem one level up, and quieter.
The distinction is not vendor positioning. It is whether anything in the stack holds a statement of intent that did not come from the system being judged.
Agent TrustKit reads the OpenTelemetry your agents already emit — so it sits on top of whatever observability you have rather than replacing it — and scores runs against a contract you write in advance. Every finding quotes the span it came from. It is an investigator, not a judge: it shows the evidence and the reasoning, so a disputed score can be checked instead of argued.
Point Agent TrustKit at telemetry you already have and see what the trace didn't tell you.
Book a call →[email protected]