Score accuracy, hallucination, and prompt-injection resistance against a versioned dataset, gate CI so a regression never ships, and turn every run into a signed, hash-chained Trust Scorecard that a customer, auditor, or teammate can check independently.
A run is reproducible because the dataset, the agent target, the scorer set, and the judge model are all versioned. Personal data is redacted at the source before anything is stored.
Test cases are grouped into an immutable, content-hashed dataset, so a run always points at an exact, reproducible version of your evaluation set.
Your agent is reached through one adapter method, so the harness never depends on your application internals and works with any framework.
Every result is graded for correctness and hallucination against expected outcomes, with a verdict and a rationale, not just a number.
Adversarial cases probe whether the agent can be steered off task or made to leak instructions, and the harness scores how it held up.
Lock today's metrics in as a baseline, then gate your pipeline so a build exits non-zero the moment a change drops below it.
Every run produces a signed scorecard on the same tamper-evident chain as your other evidence, verifiable by someone who was not in the room.
Most teams evaluate once, at launch, then stop. An agent's behavior drifts with every model update and every prompt change. The harness makes re-evaluation a five-minute CI step instead of a quarterly project, so the evidence stays current.
Reach a first verified run in about sixty seconds with the built-in sample agent, fully offline, then point it at your own agent when you are ready.
Pure standard library, no dependencies for the core. pip install trustra-eval, then trustra eval init and trustra eval run.
trustra eval gate exits non-zero if a metric drops below your locked baseline, before the change reaches production.
Add the Flight Recorder reader extra and scorecards land on the same tamper-evident record as your production logs.
The offline package ships with a sample agent and a starter dataset, so you see a real Trust Scorecard before wiring up your own agent.
Tamper-evident history, risk findings, and audit-ready reports for customer-facing AI.
Replay multi-step agent runs, tool calls, and decision paths end to end.
Enforce input, output, action, and cost limits at runtime.
Trust scoring, verification, and the Trustra Verified badge.