Evaluation harnesses
that name a cause and an owner for each failure.
An evaluation harness runs every scenario repeatedly against a pinned version and classifies failures by cause, so each class points at an owner and a fix. It matters when a score moves and nobody can say whether the agent did.
The real question
Whether a failed run names a cause and an owner, or sends the team back to reading every transcript.
Step 04 of 05 · AI & Agent Quality
The loop
From scenario
to release decision.
Five steps, each feeding the next. This page is one of them.
- Agent evaluations
- Workflow and handoff testing
- Deterministic checks
- Evaluation harnesses (this page)
- Release criteria
Running exampleAn illustration, not a case study.
A support agent is asked to refund a duplicated charge. The run repeats the scenario from a reset state. A second refund after a retry is classed as a tool-handling fault, which points at the integration’s owner, not at the prompt.
01Questions
The questions
it answers.
What exactly did this run test?
Each run records the model version, prompt, tool definitions, sampling settings, judge and scenario set, so a difference between runs can be named.
Did the scenario pass, or pass this time?
Each scenario runs several times and reports how often it passed; the number of trials is a cost decision your team has made and written down.
Did the workflow fail, or did the run?
A rate limit, an environment that did not reset or a grader that crashed is counted as a harness fault, apart from failures of the workflow.
Who owns this failure, and what kind of fix does it need?
Each failure has a cause class, such as bad tool data, a wrong tool chosen or context lost at a handoff, and each class has an owner.
02Pitfalls
Where it
usually breaks.
A pass rate from one trial per scenario
Each scenario runs once, so one that passes only some of the time reads green one day and red the next. The pass rate moves by chance, and a regression looks the same as a bad run.
Failure labels that describe the symptom
Failures are tagged “hallucination” or “wrong answer”: how each looked, not what broke. Someone still reads every transcript, and one cause is fixed many times as separate cases.
A harness running a different agent from production
The evaluation drives the model through its own prompt, tool set or context handling. It scores an agent that never ships, and the one in production goes unmeasured.
A comparison where everything moved at once
Between two runs the model, prompt, scenarios and judge all changed. The score moved, and nothing shows which of them moved it.
03Approach
How I work
on it.
Pin the run and measure its noise
With your engineers I pin the model version, prompt, tools, scenarios and starting state of a run, then repeat it unchanged to see how far the result moves.
Agree cause classes and their owners
With your team I read real failures and agree a small set of cause classes and who owns each, dropping any that could not lead to action.
Keep the classes and graders current
I review unclassified failures with your team and re-check model judges against human labels when a judge or the agent’s model changes, so runs stay comparable.
What you leave with
- A run definition: the versions and starting state every run records, and the number of trials per scenario.
- A small set of failure classes, each with a named owner and a kind of fix, and harness faults kept apart.
- Rules that assign a class from the trace where the trace can decide, and a review step for the failures left over.
- Results and traces stored per scenario and trial, with cost and latency, ready to report against your release bar.
04Measures
What I
would measure.
Signals to read, not targets to hit: what counts as good enough is your team’s decision.
Run-to-run spread on an unchanged system
When it widens, something in the run is no longer pinned, and small differences between versions stop being evidence.
Share of failures left unclassified
A growing share means the workflow is failing in ways the classes do not cover, and those failures have no owner.
Share of failed trials that are harness faults
A rise means the run is measuring its own infrastructure, and the pass rate has stopped describing the workflow.
05Evidence
The topic,
in practice.
From my employment: a case study in which unstable tests and assertions were fixed until a failed eval pointed at the product, with the limits of its evidence stated.
Case studies
Making AI evaluations trustworthy at an AI-native language-technology platform.
Improving unstable tests and assertions raised the eval pass rate from just over half to three in four within a month, and the failures that remained were known product issues, not faults in the tests.
Need credible verification
of an AI workflow?
Describe the workflow, what it is meant to achieve and where confidence is thin.