The trace your agent already emits contains almost everything an evaluation needs: the plan, the tool calls and their arguments, the retrieved context, the errors, the retries, the costs. Reading standard OpenTelemetry rather than a proprietary SDK means the evaluation layer outlives whichever agent framework you are using this year.
An evaluation layer coupled to your agent framework has the same lifespan as your choice of agent framework.
Agent frameworks are moving fast and teams migrate between them regularly. Anything that requires wrapping your agent in a vendor's SDK to observe it has to be reinstalled, re-instrumented, and re-validated every time that choice changes — and it constrains the choice, which is worse.
OpenTelemetry inverts that. The GenAI semantic conventions give model calls, tool invocations, and agent steps a shared vocabulary, so a consumer reading them works across LangGraph, CrewAI, the OpenAI Agents SDK, Bedrock Agents, LlamaIndex, or something that does not exist yet. Three practical consequences:
This is the concrete version of "instrument well." Each capture below is the precondition for a specific evaluation — without it, that check is impossible for anyone, regardless of tooling.
Which tool was invoked, on every call. Without it you cannot check a forbidden-tool rule, an allow-list, or a risk-tier policy — the cheapest and highest-value checks there are.
The actual payloads. Most tool failures are argument failures, and a trace recording only "search_documents succeeded" cannot surface a single one of them.
The passages or document IDs actually returned. This is the one most often missing, and without it groundedness cannot be scored at all — there is nothing to check a claim against.
Parent/child relationships and sequence. Loop detection, retry counting, and path analysis are all structural questions that need the shape of the trace, not just its contents.
Failures and what followed them. An agent that silently swallowed a tool error and proceeded looks identical to one that succeeded, unless the failure itself was recorded.
Per-call token counts, the model identifier, and timings. Cost variance is a reliability signal, and the model ID is what lets you attribute a drift event to a provider-side change.
No evaluation tool can score what the trace never recorded. If retrieved passages are absent, groundedness is unanswerable; if tool arguments are absent, argument correctness is unanswerable. Any vendor promising those scores from a trace that lacks the underlying data is inferring them, and an inferred score with no evidence behind it is the thing evaluation exists to replace. The honest response to thin telemetry is to say the check could not be run — and route to instrumentation work.
The trace records what happened. It contains no statement of what should have happened, and no amount of additional instrumentation will produce one, because intent is not an observable property of execution.
That missing half has to be authored: which tools this agent may use, what a complete answer to this class of request includes, what budget is acceptable, which actions require a human gate. A contract. Telemetry plus criteria is an evaluation; telemetry alone is a very detailed record of something you have not judged.
It is also why criteria inferred from traces are circular — an agent graded against its own behaviour will always look consistent with itself. See evaluation vs. observability for the longer version of that argument.
Agent TrustKit ingests OpenTelemetry using the GenAI semantic conventions — or Bedrock Agents trace exports — and scores runs against a contract you write, offline, on your own machine. When telemetry is too thin to answer a check, it says so and names the instrumentation gap rather than guessing.
Then most of the work is done. Bring a contract and see what your telemetry can and can't answer.
Book a call →[email protected]