In June, 20 of 40 production releases at a client failed. Half of them. In July, 4 of 39 failed, about 10%.
Nobody rewrote the pipeline in between. No new CI vendor, no big migration. The change came from looking at where releases actually failed and fixing those places, one at a time.
I pulled every production release run from the start of the engagement in mid-May to late August, 150 runs in total, and grouped them by month.
| Month | Runs | Failed | Failure rate | Median successful run | |---|---|---|---|---| | May (from mid-month) | 31 | 13 | 42% | 15.1 min | | June | 40 | 20 | 50% | 35.0 min | | July | 39 | 4 | 10% | 18.6 min | | August (to late month) | 40 | 7 | 17.5% | 15.0 min |
Two things in that table matter as much as the headline.
August went back up to 17.5%. Still far better than June, but not a straight line down. I would rather show that than round it away.
And the median successful release took about 15 minutes in May and about 15 minutes in August. Releases did not get faster. June was slow, at 35 minutes, and July was close to normal again, but the typical good release took the same time at the end as at the start. This post is about releases that finish, not releases that finish quicker.
A 50% failure rate changes how people work. When every second release fails, a red run stops meaning "something is wrong with my change". It means "run it again". People retry without reading the logs. Real failures hide among the flaky ones, and the release becomes something you babysit instead of something you trust.
That is the worst part of a flaky pipeline. It is not the lost minutes. It is that the signal is gone.
The first thing I built was not a fix. It was a set of dashboards for CI and releases: how many runs, how many failed, at which step, and how long each step took.
Until then, the failure rate was a feeling. Everyone knew releases were flaky. Nobody knew whether it was one step failing most of the time or ten steps failing a little. Those need very different fixes, and guessing wrong wastes weeks.
With failures grouped by step, the picture got much simpler. Most of the red runs came from a few places, and one step stood out.
Two causes stood out, and both were about the release waiting on things that were not really part of the release.
A deploy wait that failed the whole release. After pushing a new version, the release waited for the GitOps tool to report the application as synced and healthy. When that wait timed out, the whole release was marked failed, even when the new version was fine. That step alone was behind about 90% of the flaky production merges. I made the wait non-fatal: it still runs and still reports, but a slow sync no longer fails a release that actually worked. Application health is watched by monitoring and alerting, which is where it belongs, not by a timer in a pipeline.
That is a trade-off, and it is worth being honest about. A non-fatal check can hide a real problem if nothing else is watching. It only makes sense when production alerting is good enough to catch a bad deploy on its own.
A staging environment that no longer did anything. The release path still went through a staging environment that nobody used. Every release paid for that stage: more steps, more places to fail, more waiting. I rewired the release to skip it, and later the unused environment was removed entirely as part of a cost cleanup.
Production releases were not the only flaky part. Pull request validation was failing about 48% of the time over 30 days in July. By August, the 7-day rate was down to about 35%. Better, but still not good, and I will not pretend otherwise.
Two rules went in from late July: a required check on pull request titles and linear history on main. In September a merge queue went live, so changes were tested together, in batches of up to five, before landing on main. In its first days, 23 of 23 queue runs were green.
The dashboards also showed how much one bad test can do. On the day the merge queue went live, a single end-to-end test that depended on a live third-party sandbox, with no credentials configured, was behind all 18 failed runs. The pull request failure rate jumped to 63%, and merges to main fell from about 19 a day to 1. One test, one external dependency, and the whole team was blocked. Grouping failures by test is what turned that from "CI is broken again" into one clear cause.
Build the release dashboard on the first day, not after a month of retries. It costs very little and it turns "CI is flaky" into a list.
Be careful about what a release step waits for. Every external wait, whether it is a sync, a health check, a staging deploy or a third-party sandbox, is a place where the release can fail for reasons unrelated to the change. Keep the ones that protect production, and make the rest report instead of block.
And measure failure rate and duration separately. They are different problems. Fixing one does not fix the other, and claiming both when you fixed one is how a team stops trusting your numbers.
If your team has started retrying releases without reading the logs, I can help you find out why they fail. See Kubernetes & Platform.
Argo CD's self-heal reverted manual canary releases within seconds. The fix was not a better GitOps setup but matching how the team really ships.
Accelerating Releases with Continuous Integration and Continuous Deployment
How DevOps is Transforming the Way Teams Work Together
