Release criteria
agreed before the results are in.
A set of release criteria is the bar an AI workflow must clear, with each release reported against it in a form product and engineering can act on. It matters when a model or prompt change ships on a score nobody set in advance.
The real question
What would have stopped this release, and who agreed it before the results were in.
Step 05 of 05 · AI & Agent Quality
The loop
From scenario
to release decision.
Five steps, each feeding the next. This page is one of them.
- Agent evaluations
- Workflow and handoff testing
- Deterministic checks
- Evaluation harnesses
- Release criteria (this page)
Running exampleAn illustration, not a case study.
A support agent is asked to refund a duplicated charge. A duplicated refund is in the class that blocks a release on one occurrence. An awkward sentence in the message to the customer is not; it is tracked as a trend.
01Questions
The questions
it answers.
What does the workflow have to clear to ship?
Scenarios are grouped by what a failure costs; each group has a written bar, and failures that block on one occurrence are listed, not averaged away.
Did the change make it better, or only different?
A difference between two releases, measured scenario by scenario, is evidence of a change only when it is wider than the run-to-run spread.
Who sets the threshold, and who can waive it?
Product and engineering set each threshold together, and every waiver records who granted it and why.
What does a missed criterion tell the team to do?
A missed criterion is reported with the failed scenarios and their failure classes, so the choice to fix, hold or waive rests on causes, not a score.
02Pitfalls
Where it
usually breaks.
A threshold agreed after the results are in
The run finishes, the number is discussed and the bar lands just under it. Every release passes, so the criteria can no longer say no.
One pass rate for the whole scenario set
Common cases improve, the consequential ones regress and the average rises. The report calls the release better when it got worse on the actions you cannot undo.
A waiver agreed in a meeting and never recorded
A missed criterion is argued away and the release goes out. Nobody records who waived it or why, so no one can say how often the bar has actually held.
Prompt and model changes that skip the bar
A prompt is edited in configuration, or the model behind an alias is updated. Nothing is deployed, so nothing is run and the change reaches users unmeasured.
03Approach
How I work
on it.
Agree the bar before the results
With product and engineering I write down the bar for each scenario group and what blocks on one occurrence, such as an unapproved refund, before results exist.
Define a release and a real change
With your team I define what a release is, prompt and model updates included, and how large a gap beyond the harness’s measured spread counts as a real change.
Report each release against the bar
For each release I report every criterion as met or missed, with the change since the last release and the failure class behind each miss.
What you leave with
- Written criteria per scenario group: the bar, its owner, what blocks on one occurrence and how a waiver is recorded.
- A definition of a release: prompt, model and tool changes as well as code, with a scheduled run for upstream changes.
- A rule for telling a real change from run-to-run variation, using the spread the harness measured.
- A report per release against the bar: criteria met or missed, the change since the last and the cause of each miss.
04Measures
What I
would measure.
Signals to read, not targets to hit: what counts as good enough is your team’s decision.
Releases shipped on a waiver
A rising share says the bar sits where the team will not hold it, so the threshold or the workflow has to move.
Changes that reached users without a run
Each one is a prompt, model or tool change the definition of a release missed; a rise says the definition has fallen behind how the workflow changes.
Cost and latency per completed task
A rise at a steady pass rate says the result was bought with more calls or retries, which the outcome rate alone hides.
05Evidence
The topic,
in practice.
From my employment: a case study on AI evaluations, and one in which I led quality for two mobile apps and their platform. Each states the limits of its evidence.
Case studies
Making AI evaluations trustworthy at an AI-native language-technology platform.
Dashboards monitored the evals’ effectiveness, execution time and cost, so that improvements had data to support them, and the failures left after the clean-up were known product issues.
Sustaining quality across two mobile apps and their content delivery platform.
Not an AI workflow: for two mobile apps and their content platform, I made testing repeatable, performance and stability measurable, and release decisions grounded in evidence.
Need credible verification
of an AI workflow?
Describe the workflow, what it is meant to achieve and where confidence is thin.