Clearing the way for a release train at an AI-native language-technology platform.

Challenge

Two things slowed most changes on their way to production: manual handovers and constant rebasing. A change waited for a person at each handover. Meanwhile, so many changes landed on the shared branch that by the time an engineer had brought one up to date, the branch had moved on again. Every rebase cost an engineer’s time and another full pipeline run across dozens of services and the whole test suite.

The company already released often, but not to a timetable. The goal was a release train: releases that leave on schedule from the last green build and carry whatever is ready, so nobody has to chase a release.

Baseline

What the analysis found:

• Most of a change’s time was waiting. Between first commit and production, a change spent longer with nobody working on it than with anyone working on it.
• Approved changes kept falling behind. Most merge requests passed their checks, then fell behind the shared branch before they could merge.
• The shared branch was often red, so a train leaving at a fixed time would regularly have missed its departure.
• A passing build could not be trusted. In August, fewer than half of the Playwright test runs passed without a retry.
• People sat in the release path. A release needed manual steps and an approval that failed it if nobody answered in time.

My role

As Head of Customer Trust, I led the analysis of where each change’s time went and wrote the plan for getting to a release train. With the QA engineers, I lead QA’s part of it: making sure a passing build can be trusted.

Intervention

The plan

Five moves, in order:

1. Keep the shared branch green by default.
2. Make green mean green, so a passing build can be trusted.
3. Cut the review wait, most of which turned out to be on the author’s side, not the reviewer’s.
4. Take people out of the release path.
5. Make failure measurable, by linking every incident to the release that caused it.

For the rebasing, I have proposed merge trains, which line approved changes up and test them together so nobody has to rebase by hand. They are not in place yet. The first release train can leave once the shared branch has stayed reliably green for two weeks running and a release needs no manual steps.

What has changed so far

QA’s part is the first to show results, and it matters early: a train cannot leave on a build nobody trusts.

• Playwright test packs are now measured on whether each run passes without a retry, not only on whether it passes in the end.
• A common cause of timing failures in the tests was fixed, with an automatic check that stops it coming back.
• Tests that had been set aside as unreliable came back once they passed first time, so more of the suite now checks each build.

What comes next

The API tests come next: making their runs quicker and getting them to pass without a retry. Each test that is set aside as unreliable will also get an owner and a deadline. The policy, and the automation that runs it, are being put in place.

Evidence

Between August and late September, the share of Playwright test runs that passed without a retry roughly doubled, while more tests checked each build. By early October the Playwright test stage took less than half as long as in August and September.

Starting point

Fewer than half of runs, August 2026

End point

Roughly twice the August share, late September to early October 2026

Unit

Share of gating Playwright runs that passed without a job retry

Scope

The Playwright test packs that must pass for each build of the shared branch.

Measurement period

August to October 2026

Source

Pipeline job history

My contribution

I lead QA’s part of the work. QA engineers and the teams that own each test pack made the fixes.

Other factors

Two changes around the shared test environment likely did much of the work: lighter load from the API tests, then more capacity for the Playwright jobs. The gains have not been isolated from one another.

Caveat. Results so far, over a short period. Individual tests could still be retried within a run, under the same limit before and after, so the comparison holds.

Limitations

On the test-automation side, most of the work the release train needs is done; what remains is optimising the API tests. Their runs take longer for now, partly as a deliberate trade: they run with fewer workers, the change that likely steadied Playwright. The effect on waiting and rebasing can be measured once the release train and merge trains are running, and change failure rate and recovery time once incidents are linked to releases.