Making AI evaluations trustworthy at an AI-native language-technology platform.

Challenge

The platform’s AI features were checked by automated evaluations, or evals: tests with assertions, and judges that score the features’ output against defined criteria. Evals earn trust only when a pass means the feature works and a failure points to a real problem. My main focus was whether the evals were actually testing the features, rather than just satisfying the judges’ criteria.

Two further challenges sat alongside it: transparency and reporting on how effective the evals were, and the time and cost of running them.

Baseline

When I started, test stability was poor: the evals’ pass rate was just over half, and many tests and assertions needed improving before a result could be relied on. With failures that frequent, it was hard to tell a product problem from a test problem. Reviewing the evals also turned up a lot of tests that brought no real value.

My role

As Head of Customer Trust, I reviewed the evals’ tests and assertions, corrected the tests that were not adding value, and introduced dashboards for both the evals and the live product.

Intervention

Testing the feature, not just the judges’ criteria

I reviewed the tests and assertions behind the evals, asking of each one whether it tested what the feature should do or only what the judges’ criteria rewarded. I found a lot of tests that were not bringing any value and corrected them to test the feature itself.

Making the results stable

Stability was the other half of trust. Improving the unstable tests and assertions raised the pass rate from just over half to three in four within a month. The failures that remained were known product issues: a failed eval now pointed at the product, not at the test.

Making effectiveness, time and cost visible

For transparency, I introduced dashboards that monitor the evals’ effectiveness, execution time and cost, so that there is sufficient data to support improvements.

Connecting the evals to live customer data

I also introduced observability dashboards that bring together live customer data from Langfuse and other sources. They gave a clearer view of the success rate of each part of the agentic system’s user journeys, so features could be improved, and coverage added, where success rates were low.

Evidence

Within a month, the evals’ pass rate rose from just over half to three in four, and the failures that remained were known product issues rather than faults in the tests.

Starting point

Just over half

End point

Three in four, a month later

Unit

Pass rate of the evals

Scope

The automated evals I worked on, for the platform’s AI features

My contribution

I reviewed the evals’ tests and assertions, corrected the tests that were adding no value, and introduced the dashboards for the evals and for live customer data.

Caveat. Both are approximate and given in words; the exact figures, and the dashboards’ time, cost and success-rate figures, are not published.

Limitations

A higher pass rate here means fewer failures caused by the tests themselves; it is not, on its own, a measure of the AI features’ quality. With the dashboards in place, the next improvements, including to execution time and cost, can be measured against data.