Agent TrustKit
Every agent project hits the same wall: how do we prove this can be trusted? Until somebody will sign that sentence, the thing sits in staging. Or it ships and nobody signed anything, which is worse. Agent TrustKit is the definition of done that agent projects are missing.
Accuracy is a number. Shipping is a decision.
97.5% task completion, budgets met, policies clean, and the gate still failed. A single eval score would have shipped it.
The report below is a real one, and it tells the whole story in one screen. 97.5% task completion. Budgets met. Policies clean. And the gate failed anyway, because the agent invented a sell-side downgrade and dropped a fact the portfolio manager needed. A single accuracy number would have waved it straight through.
Then there's the other half of the problem. When something does go wrong, most teams can't name the person who was supposed to sign off in the first place. The trust report is the artifact that person signs. The task contract is what they're signing against.
Why not just a golden dataset?
Golden datasets earn their place in an evaluation strategy. For anything agentic, they can't be the whole of it.
A golden dataset scores an output by diffing it against an expected answer, which assumes there is one. A briefing agent, a retrieval pipeline, or a multi-agent workflow can be right in a hundred different ways, so a fixed answer key starts logging good outputs as failures. Then it decays. Swap in a better model, add a data source, tune the retriever, and the goldens you spent two weeks labeling start flagging your improvements as regressions. Multiply that across a dozen workflows and the re-labeling bill outruns anyone's appetite for paying it.
A bigger answer key doesn't fix that, and neither does handing the output to another model for a grade. We run the analyst as an investigator, not a judge. It gathers quoted evidence out of the trace and hands that evidence to deterministic Python, which scores it against a contract: the properties the output has to hold. Policy, budgets, forbidden tools, PII, and schema are all straight code. Groundedness and omissions get checked against the sources the agent actually retrieved, not a frozen expected answer. Golden cases live inside that as one useful layer. They just aren't the foundation.
Every score traces back to a line you wrote down.
Each score points at a clause in a contract you co-authored, and each finding quotes its evidence. The LLM investigates. It never grades.
The task contract
One YAML file your stakeholders fill in up front: objective, forbidden behaviors, budgets, tolerated failure rates, approved scope, autonomy tier. Every score in the report points back at a line in it. Anything the contract doesn't explicitly loosen runs at zero tolerance, so a passing report means what the person reading it assumes it means.
Investigator, not judge
The LLM scores nothing. It pulls findings with the quote attached: "unsupported claim: 'Two desks downgraded AAPL to Sell'; the source says no rating changes were published this week." The scoring is deterministic Python, so the same traces give the same profile every time. That's what makes the report survive a risk review.
Catches omissions
Everyone tests for hallucinations. Agent TrustKit also tests completeness: did the output carry every material fact that was in the sources? On briefing and research agents the fact that got dropped is the career-ending failure mode, and almost nobody checks for it.
Scoped trust, not a score
The report doesn't hand you "89/100." It says approved for internal briefings on covered names; not approved for client-facing use or trade execution. Your team can act on that sentence, and it's the shape of evidence an AI-governance review is asking for.
The analyst is validated
Label a batch of analyst findings and the calibration harness scores agreement with your humans, per dimension and per checker. So when someone asks why they should trust our evaluator, the answer is a number instead of a shrug.
Regression-proof in CI
trustkit compare diffs baseline against candidate at the dimension and finding level and exits 1 on a regression. New prompt, new model, new retriever: you get the exact per-dimension delta, and the merge stops if trust went backwards. Nobody has to re-litigate it next quarter.
It reads your OpenTelemetry, and tells you what to fix.
Point Agent TrustKit at your OTel and it reads back. trustkit inspect maps your coverage gaps first: missing outputs, no retrieved sources, no cost or token fields, handoff routes you never tagged. Then in the run itself it flags the waste you'd never spot by hand. Tool calls that moved the task nowhere. An agent firing the same search five times because it can't find the right context. A call stuck in a loop. With a dozen agents hitting a dozen tools, all of that disappears into the noise. In the trace it's obvious.
When one prompt can't hold the trace.
Real multi-agent traces don't fit in a judge prompt. So the analyst gets tools and goes looking for the evidence itself.
A single-shot judge call works fine until the trace outgrows it. Then you truncate, and truncation is where evidence goes to die: the analyst can't find the source that supports a claim, so it books the claim as unverifiable and your report fills up with shrugs.
The fix was to stop stuffing the trace into a prompt. TraceInvestigator drives the Claude SDK Tool Runner in-process over a closed, read-only toolset on the trace already in memory: stage and agent activity, authoritative diffs, retrieval sources, full-text search. It navigates to the evidence instead of being handed a window of it. Nothing leaves the box, because the tools only reach local objects. Every call is logged and cached, and the retrieval log lands on the finding, so an agentic run replays as cleanly as a single-shot one.
It routes only where it's needed. --investigator auto flips on at the point single-shot would have truncated, bounded by a 12-turn cap and 20K-char tool results. The verdict schema and every scoring rule stay exactly the same, so investigator-not-judge still holds: it gathers, the metrics engine scores.
The system gets cheaper over time.
A pattern the model notices once becomes a check that costs nothing, forever after.
On the agentic route, a separate cheap pass writes down what it noticed while navigating the trace: patterns no fixed checker looks for yet. These are field notes, not verdicts, kept structurally separate from trust findings so they can never fail a gate. When the same tag recurs across runs, the report flags it as a graduation candidate.
Three have already crossed over into free, deterministic Python: an agent that got lost hunting for a config file before finding it by search, one that silently retried a call it already knew was failing, one that escalated to a human and got no answer back. Each started as a fraction-of-a-cent observation on a small model and is now a check with zero LLM cost, running on every trace, forever.
A dashboard you run on your own machine.
Markdown reports are fine for the repo. For the room where people argue about shipping, you want something you can click.
trustkit ui serves a static React app off localhost. It reads Agent TrustKit's derived output only, the trust profile and findings, and it has no code path to your raw traces. You pick a run folder in the browser and it parses client-side. Nothing uploads. There's no auth, no telemetry, no phone-home, and a build check fails CI if a network primitive or an external URL ever shows up in the source. Your team doesn't need Node either: the CLI ships a stdlib static server bound to 127.0.0.1.
Two surfaces. The fleet dashboard is the cross-run view: gate pass rate, drift checkpoints, average cost per run, and a latest-vs-baseline table with per-dimension sparklines, so a regression like constraint adherence going 8.3% → 16.7% shows up as a red line before anyone reads a paragraph. The report viewer is one run in detail, with section toggles (hide the NIST appendix, or don't) and downloads for the HTML report, the profile JSON, and the raw findings.
The demo mode is real data.
trustkit ui --demo boots the whole thing against a bundled run set, so you can walk a stakeholder through a trust report before you've instrumented a single agent of your own. Same UI, same charts, no setup and no account.
Built for the questions your risk team now asks.
Documented, reproducible evidence lined up against NIST AI RMF and the CSA Agentic Profile. We don't sell a certification. We hand you evidence.
Agent TrustKit generates the documented, evidence-backed measurement an AI governance program needs, mapped to the frameworks your reviewers are already citing. There is no certification to sell here. What you get is evidence your risk team and your assessors can put to work.
The evidence, not a checkbox
Agent TrustKit is the MEASURE layer feeding MANAGE: the independent measurement your GOVERN function anchors on. trustkit report --nist-appendix renders the crosswalk per run, so every engagement report doubles as an RFP-ready artifact.
The agent-specific controls, shipped
Not a roadmap slide. Contracts carry a declared autonomy tier and oversight boundary (AG-GV.1), tools carry the full four-dimension risk taxonomy (AG-MP.1), and runs report delegation-chain length, per-agent fan-out, and agent-identity attestation (AG-MS.1/.3). Where a control belongs to an identity provider or a kill switch, we say so.
No custody, no FedRAMP hurdle
Evaluation runs where your data already lives and nothing comes back to us. No federal data ever touches a vendor system, so the self-hosted engagement needs no FedRAMP authorization at all, and our evidence feeds your system's ATO.
Proven in CI
A local-first tool with no server to attack. Trivy, bandit, pip-audit, and an SBOM run on every change, and there's a SIG-style questionnaire and a secure-development write-up sitting ready for your third-party review.
What we don't claim
Nobody certifies "NIST compliance," and you can't buy governance as software. Our crosswalks grade our own coverage honestly, control by control, as supported, partial, roadmap, or out of scope. Where Agent TrustKit's real answer is "that's a different tool's job," the document says that. Ask and we'll send you the crosswalks and the security pack.
Your first trust report is free.
A real report on your agent, with nothing leaving your environment. Send a Bedrock or OpenTelemetry export and we'll do the rest.
This isn't a demo built on our data. It's a trust report generated on your agent, and we've written down exactly what to send: an Amazon Bedrock Agents trace export, or an OTel export of your runs. Before anything else happens, a read-only preflight inspects your file and tells you what it can and can't support. If your traces are thin, you'll hear it from the tool instead of from us three weeks later. And if you'd rather look before sending anything at all, the sample report above is the same artifact you'd get.
We give the first one away because one report isn't the business. Models drift, prompts get tuned, someone swaps the retriever on a Thursday, and every one of those re-opens the question the first report answered. The work that follows, remediation, re-runs, baseline-vs-candidate comparisons, a scheduled trustkit drift watch, and a CI gate your team owns, is where this actually earns its keep.
Starting cold is fine too. The contract packs ship templates and task sets per archetype (RAG QA, knowledge synthesis, coding pipelines, generic agents, plus a robustness pack with an adversarial task set), so we're not staring at a blank YAML file on the first call.
The Agent Trust Sprint
Fixed scope, fixed price, and it turns one baseline into a practice: a task contract written with your stakeholders, remediation guidance, a re-run with a comparison report showing what moved, and a working CI gate your team keeps running after we leave. Every artifact lands in your repo.
Who signs off on your agent?
Claim the free trust report and find out what your risk team would actually be signing.
Book a call →[email protected]