Guide · OpenTelemetry

Evaluating agents from OpenTelemetry

The trace your agent already emits contains almost everything an evaluation needs: the plan, the tool calls and their arguments, the retrieved context, the errors, the retries, the costs. Reading standard OpenTelemetry rather than a proprietary SDK means the evaluation layer outlives whichever agent framework you are using this year.

OpenTelemetryGenAI Semantic ConventionsPortable TracesInstrumentationFramework-AgnosticSpans
01 Why OTel

Frameworks churn. Telemetry doesn't.

An evaluation layer coupled to your agent framework has the same lifespan as your choice of agent framework.

Agent frameworks are moving fast and teams migrate between them regularly. Anything that requires wrapping your agent in a vendor's SDK to observe it has to be reinstalled, re-instrumented, and re-validated every time that choice changes — and it constrains the choice, which is worse.

OpenTelemetry inverts that. The GenAI semantic conventions give model calls, tool invocations, and agent steps a shared vocabulary, so a consumer reading them works across LangGraph, CrewAI, the OpenAI Agents SDK, Bedrock Agents, LlamaIndex, or something that does not exist yet. Three practical consequences:

  • No code change in most cases. Teams already instrumented through LangGraph, Langfuse, Phoenix, or the OpenAI SDK are usually emitting what is needed.
  • The data is yours. Traces are an open format sitting in your own collector — not a proprietary store you have to export from.
  • Evaluation can run offline. Anything that reads a trace file works on your machine, against your data, with no phone-home. That is frequently the difference between a security review passing and stalling.
Instrument once, in the open standard. Then whatever reads it is a replaceable part, including this one.
02 What to capture

Which attributes unlock which checks.

This is the concrete version of "instrument well." Each capture below is the precondition for a specific evaluation — without it, that check is impossible for anyone, regardless of tooling.

Enables policy checks

Tool name and identity

Which tool was invoked, on every call. Without it you cannot check a forbidden-tool rule, an allow-list, or a risk-tier policy — the cheapest and highest-value checks there are.

Enables argument scoring

Tool arguments and results

The actual payloads. Most tool failures are argument failures, and a trace recording only "search_documents succeeded" cannot surface a single one of them.

Enables groundedness

Retrieved context

The passages or document IDs actually returned. This is the one most often missing, and without it groundedness cannot be scored at all — there is nothing to check a claim against.

Enables trajectory checks

Span hierarchy and order

Parent/child relationships and sequence. Loop detection, retry counting, and path analysis are all structural questions that need the shape of the trace, not just its contents.

Enables recovery checks

Errors, retries, status

Failures and what followed them. An agent that silently swallowed a tool error and proceeded looks identical to one that succeeded, unless the failure itself was recorded.

Enables cost SLOs

Tokens, model, latency

Per-call token counts, the model identifier, and timings. Cost variance is a reliability signal, and the model ID is what lets you attribute a drift event to a provider-side change.

The ceiling

Instrumentation depth bounds evaluation depth.

No evaluation tool can score what the trace never recorded. If retrieved passages are absent, groundedness is unanswerable; if tool arguments are absent, argument correctness is unanswerable. Any vendor promising those scores from a trace that lacks the underlying data is inferring them, and an inferred score with no evidence behind it is the thing evaluation exists to replace. The honest response to thin telemetry is to say the check could not be run — and route to instrumentation work.

03 Practicalities

Three things that bite in production.

  • Sampling loses the runs you want. Head-based sampling at 1% is standard for performance monitoring and wrong for evaluation: it discards 99% of the failures you most need to look at. Prefer tail-based sampling that keeps errors, outliers, and anything tripping a check — and be aware that a default collector config will quietly throw away your evidence.
  • Payloads are sensitive. Tool arguments and retrieved context contain the same data your agent handles. That is precisely why evaluation should run where the data already lives rather than shipping traces to a vendor cloud — and it is why redaction belongs at the collector, not as an afterthought.
  • Retention outlives the incident. Traces commonly rotate in days. Clearance evidence, drift baselines, and audit questions arrive in months. Whatever an evaluation produces has to be durable independently of the trace it was derived from, or you will be asked to prove something whose source no longer exists.
04 The limit

What OpenTelemetry cannot give you.

The trace records what happened. It contains no statement of what should have happened, and no amount of additional instrumentation will produce one, because intent is not an observable property of execution.

That missing half has to be authored: which tools this agent may use, what a complete answer to this class of request includes, what budget is acceptable, which actions require a human gate. A contract. Telemetry plus criteria is an evaluation; telemetry alone is a very detailed record of something you have not judged.

It is also why criteria inferred from traces are circular — an agent graded against its own behaviour will always look consistent with itself. See evaluation vs. observability for the longer version of that argument.

Agent TrustKit

Reads your OTel. Scores against your contract.

Agent TrustKit ingests OpenTelemetry using the GenAI semantic conventions — or Bedrock Agents trace exports — and scores runs against a contract you write, offline, on your own machine. When telemetry is too thin to answer a check, it says so and names the instrumentation gap rather than guessing.

What to bring · A report built from thin telemetry

Already emitting traces?

Then most of the work is done. Bring a contract and see what your telemetry can and can't answer.

Book a call [email protected]Next case study Get started