Agent TrustKit
Evidence-based evaluation and clearance for AI agents.
Agent TrustKit evaluates agentic workflows from the OpenTelemetry your agents already emit, and produces evidence-backed agent clearance reports. It answers the one question that keeps agents sitting in staging: can this be trusted to operate autonomously? — scored against a task contract you wrote down first.
Accuracy is a number. Shipping is a decision.
97.5% task completion, budgets met, policies clean — and the gate still failed. A single eval score would have shipped it.
The report below is a real one. The agent invented a sell-side downgrade and dropped a fact the portfolio manager needed. Groundedness and completeness both failed their contract thresholds, so the gate failed, and a single accuracy number would have waved the whole thing straight through.
There is a second half to this. When something does go wrong, most teams cannot name the person who was supposed to sign off in the first place. The clearance report is the artifact that person signs. The task contract is what they are signing against.
How do you evaluate agentic workflows?
You judge how the agent got there, against rules somebody wrote down before the run.
A golden dataset scores an output by diffing it against an expected answer, which assumes there is one. A briefing agent, a retrieval pipeline, or a multi-agent workflow can be right in a hundred different ways, so a fixed answer key starts logging good outputs as failures — and then decays every time you improve the system underneath it.
Agent TrustKit starts from the trace instead. Four steps, and the interesting part is which of them the model is allowed to touch.
Write the contract
One YAML file your stakeholders fill in: objective, forbidden behaviors, tool policy, budgets, tolerated failure rates, autonomy tier. Anything it does not explicitly loosen runs at zero tolerance. trustkit init scaffolds it from an archetype pack.
Point it at your traces
Standard OTel GenAI semantic conventions, or an Amazon Bedrock Agents trace export. trustkit inspect runs a read-only preflight first and names your coverage gaps before any evaluation happens.
The model investigates
The LLM scores nothing. It pulls findings with the quote attached: "unsupported claim: 'Two desks downgraded AAPL to Sell'; the source says no rating changes were published this week." On traces too big for one prompt it gets a closed, read-only toolset and walks to the evidence itself.
Deterministic Python scores it
Every number is computed in ordinary code against the contract's thresholds. Same traces, same trust profile, every time — which is what makes the report survive a risk review instead of starting an argument.
A clearance decision, with the evidence attached.
Not "89/100." A sentence your team can act on and your reviewers can check.
What it's cleared for
The report says approved for internal briefings on covered names; not approved for client-facing use or trade execution. That is the shape of evidence an AI-governance review is asking for.
Catches omissions
Everyone tests for hallucinations. Agent TrustKit also tests completeness: did the output carry every material fact that was in the sources? On briefing and research agents, the fact that got dropped is the career-ending failure mode.
A CI gate your team owns
trustkit compare diffs baseline against candidate per dimension and exits 1 on a regression. New prompt, new model, new retriever — the merge stops if trust went backwards, and nobody re-litigates it next quarter.
An offline dashboard
trustkit ui serves a static React app on localhost, reading derived output only. No auth, no telemetry, no phone-home — and a build check fails CI if a network primitive ever appears in the source.
Four reports. Same agent, same 40 runs.
What your telemetry supports decides which of these you get — and trustkit inspect tells you before you commit to anything.
One maximal demo teaches the wrong lesson. A prospect sees every section filled in, their own first report arrives with fewer, and they assume something is broken. So all four are here: the same fictional PM morning-briefing agent, 8 tickers × 5 trials, differing only in what the trace and the consent allowed.
The headline
97.5% task completion, budgets respected, policies clean — and the gate still fails on the two zero-tolerance dimensions. Two hallucinated claims, each quoted against the source sentence that contradicts it, plus one omitted material fact.
Read the sample report ↗After remediation
The same agent, one version later. The gate passes with three violations still visible, inside contract tolerance. A spotless report reads as untested.
See the approved run ↗Consent gap
Identical traces, identical agent. The client hasn't authorised the analyst step, so nobody looked at grounding or omission. Not a pass, not a failure — the evidence is on the trace, waiting for a signature.
Analyst step declined ↗Capture gap
The same defective runs, exported without retrieved passages or tool results. Deterministic checks still land; the analyst has nothing to check claims against. Consent wouldn't help — this one routes to instrumentation work, not to a report.
Thin telemetry ↗When one prompt can't hold the trace.
Real multi-agent traces don't fit in a judge prompt. So the analyst gets tools and goes looking for the evidence itself.
A single-shot judge call works until the trace outgrows it. Then you truncate, and truncation is where evidence goes to die: the analyst can't find the source that supports a claim, books it as unverifiable, and your report fills up with shrugs.
So we stopped stuffing the trace into a prompt. The investigator drives a closed, read-only toolset over the trace already in memory — stage and agent activity, authoritative diffs, retrieval sources, full-text search. It navigates to the evidence rather than being handed a window of it. Nothing leaves the box, because the tools only reach local objects. It routes only where it's needed, bounded by a 12-turn cap, and the scoring rules do not change: it gathers, the metrics engine scores.
That last number is the part that compounds. A cheap pass writes down patterns no fixed checker looks for yet; when the same one recurs across runs, it graduates into deterministic Python. An agent that got lost hunting for a config file, one that silently retried a call it knew was failing, one that escalated to a human and got no answer back — each started as a fraction-of-a-cent observation and now runs on every trace at zero LLM cost, forever.
Teams where a wrong answer costs money.
The common thread: someone downstream has to put their name on it.
Asset managers, banks, insurers
Research and briefing agents where a dropped fact or an invented rating change reaches a portfolio manager. Model risk management already expects documented, reproducible evidence; this is that evidence, for agents.
Federal & regulated programs
Evaluation runs where your data already lives, so no federal data touches a vendor system and the engagement needs no FedRAMP authorization. NIST AI RMF crosswalks generate per run and feed your system's ATO package.
Anyone shipping agents to production
You have a drift harness and a green suite, and the release meeting still stalls on "but can we rely on it?" This is the artifact that ends that meeting, plus the CI gate that keeps it ended.
High-stakes document work
Summarization and synthesis where completeness matters as much as accuracy. Omission detection is the check almost nobody runs, and it's the one that catches the fact that should have been carried forward.
Built for the questions your risk team now asks.
Evidence lined up against NIST AI RMF and the CSA Agentic Profile. There is no certification to sell here.
Agent TrustKit is the MEASURE layer feeding MANAGE — the independent measurement your GOVERN function anchors on. trustkit report --nist-appendix renders the crosswalk per run, so every engagement report doubles as an RFP-ready artifact. Evaluation runs where your data already lives, which means a self-hosted engagement needs no FedRAMP authorization at all, and our evidence feeds your system's ATO.
Our crosswalks grade our own coverage control by control as supported, partial, roadmap, or out of scope. Where the honest answer is "that's a different tool's job," the document says so. Ask and we will send you the crosswalks and the security pack.
Your first clearance report is free.
A real report on your agent, with nothing leaving your environment.
This isn't a demo on our data. Send an Amazon Bedrock Agents trace export or an OTel export of your runs, and a read-only preflight inspects the file first and tells you what it can and can't support. If your traces are thin you'll hear it from the tool, rather than from us three weeks later. Starting cold is fine too: the contract packs ship templates and task sets per archetype — RAG QA, knowledge synthesis, coding pipelines, generic agents, plus a robustness pack with an adversarial task set.
We give the first one away because one report isn't the business. Models drift, prompts get tuned, someone swaps the retriever on a Thursday, and every one of those re-opens the question the first report answered.
The Agent Trust Sprint
Fixed scope, fixed price, and it turns one baseline into a practice: a task contract written with your stakeholders, remediation guidance, a re-run with a comparison report showing what moved, and a working CI gate your team keeps running after we leave. Every artifact lands in your repo.
The things people ask on the first call.
How do you evaluate agentic workflows?
You evaluate an agentic workflow by judging how it reached its result, not only what it returned. A multi-step agent can be right in many different ways, so an expected-answer key stops working. Agent TrustKit reads the OpenTelemetry your agents already emit, extracts quoted evidence from the trace, and scores that evidence against a task contract: a written statement of the objective, prohibitions, tool policy, budgets and autonomy tier the workflow has to hold to. The output is a pass/fail clearance decision with the evidence attached to every finding.
What happens if a check could not run?
The gate returns INCONCLUSIVE. It is a third outcome alongside pass and fail, and it exits non-zero, so a CI gate stops on it. A dimension nobody checked is recorded as unevaluated rather than clean — whether that is because you declined the opt-in analyst step, or because the trace never captured what the agent retrieved. Both cases reach the same verdict for different reasons, and the remedies differ: one is a consent question, the other needs instrumentation work before a report is worth running. The point is that Agent TrustKit will not tell you something passed if it did not look.
What is an agent clearance report?
A clearance report states what an agent is approved to do, and what it is not. Instead of a single score, it reads: approved for internal briefings on covered names; not approved for client-facing use or trade execution. Every dimension in it is scored against a threshold someone wrote down before the run, and every finding quotes the trace line it came from, so a risk reviewer can check the work rather than take it on faith.
How is this different from using an LLM as a judge?
An LLM judge is asked an underdetermined question — is this output good? — and its answer moves with the prompt. Agent TrustKit makes the question determinate before it is asked: the task contract supplies the thresholds, the model is confined to extracting evidence with pointers into the trace, and the scoring is deterministic Python. The same traces produce the same trust profile every time, and the profile can be recomputed without an LLM in the loop at all.
Does Agent TrustKit need access to my data?
No. The deterministic evaluation — policy, budgets, tool use, reliability, PII and secrets checks — runs entirely against your file, in your environment, with no collector and no phone-home. The AI-analyst step that handles groundedness and omission checks is opt-in, and it can run against a Bedrock-hosted Claude inside your own AWS account. Agent TrustKit emits a trust profile and findings; it has no code path that ships your raw traces anywhere. If you decline the analyst step, the gate returns INCONCLUSIVE rather than a pass: nobody checked grounding or omission, so nothing can claim they were clean.
What do I need to bring to get started?
Two things: a short task contract describing what your agents are supposed to do, and the OpenTelemetry they already emit. If you are instrumented through LangGraph, the OpenAI Agents SDK, Langfuse, Phoenix, or Amazon Bedrock Agents, there is a good chance you do not touch your application code at all. Run trustkit inspect first and it reports what your traces can and cannot support before any evaluation happens.
Which agent frameworks and telemetry formats are supported?
Two native ingestion paths. Amazon Bedrock Agents trace exports work out of the box, including knowledge-base lookups, which means groundedness and omission checks need nothing extra. Standard OpenTelemetry using the GenAI semantic conventions is the other, covering anything instrumented through LangGraph, the OpenAI Agents SDK, Langfuse, Phoenix, ADOT, or direct OTel. If you log to something else entirely — CloudWatch, a database, flat files — a small adapter is written against your schema, usually half a day, and it is yours to keep.
What does it cost?
The first clearance report is free, generated on one of your own agents. After that the work is sold as an engagement rather than a software seat: the Agent Trust Sprint is fixed scope and fixed price, and turns that first baseline into a task contract, remediation guidance, a re-run comparison, and a CI gate your team owns. There is no per-seat licence and no certification being sold.
Does Agent TrustKit replace our existing evals or drift harness?
No, and it is not meant to. A drift harness answers whether the workflow still does what it did last week, which is good engineering and sometimes the entire job. Golden cases live inside a contract as one useful layer. What Agent TrustKit adds is the separate claim a stakeholder signs: that the run stayed inside declared rules, grounded every claim in retrieved sources, and omitted nothing material — with evidence a reviewer can check.
Who signs off on your agent?
Your first trust report is free, generated on your own agent, with nothing leaving your environment. Send a Bedrock or OpenTelemetry export and we will run the read-only preflight first, so nobody gets surprised.