For a few months I reported a request success rate of 99.994% to a client's leadership. In late September I found out it was wrong. Measured the way customers experience the product, the same kind of period came out at 99.9575%.
That gap looks small on paper. It is not. 99.994% means about 6 failed requests in every 100,000. 99.9575% means about 42. The real failure rate was roughly seven times what I had been reporting.
And the part that hurt more: I had also said 5xx errors were down about 80%. With the corrected method, total 5xx had gone from about 13,300 to about 12,900 over the same comparison. Flat.
The SLO counted a request as failed only when the application itself returned a 5xx. Everything else counted as a success.
The problem is what that leaves out. A request can fail before the application ever sees it. The load balancer returns a 502 when the target closes the connection badly, a 503 when there is no healthy target, and a 504 when the target takes too long. The application never returned those errors, so the SLI never counted them. From its point of view, those requests did not fail.
When I put the load balancer's own 5xx counts next to the application's, the load balancer errors were about 90% of all the 5xx that clients actually received. My SLI had been measuring the 10%.
Nothing was hidden on purpose. The SLO was set up early, when the goal was simply to have SLOs at all, and the number looked healthy. A healthy number does not invite questions. It took putting the load balancer's error counts next to the SLO to see that the two did not match.
I rewrote the SLI to count every 5xx at the load balancer, whatever produced it, against all requests the load balancer received. I added a request-based SLO for client request success, counting all 5xx, with a 99.95% target over 30 days and 7 days. Then I recalculated the previous period with the new method, which is where the 99.9575% came from.
The new number also changed the picture across all the SLOs. With honest inputs, only about three in four were being met, and some of the front-end ones were far off target.
Then came the uncomfortable part. I had sent the old number to leadership. I wrote the correction down next to the original figure, with the reason and the corrected method, for the next report.
An SLO that only ever moves in your favour is not measuring anything. If I want leadership to trust the next number, they need to see me correct this one.
It is also a better place to work from. "Errors were flat" is a clearer problem statement than "everything is fine". The weekday-morning spike turned out to have three causes, a nightly job running inside a web pod, a keep-alive timeout mismatch and a capped node pool, and after the fixes there were seven weekday mornings in a row with zero load balancer 5xx.
While rebuilding the SLOs I also had to explain why some numbers moved differently from others.
A time-slice SLO splits time into windows, say 5 minutes, and marks each window good or bad. Availability is the share of good windows. A request-based SLO counts requests: failed over total.
They answer different questions. A time-slice SLO asks how much of the time the service was healthy. A request-based SLO asks how many customer requests failed. One bad 5-minute window during a quiet night costs a time-slice SLO as much as one during the busiest hour. A request-based SLO weighs the busy hour far more, because more people were affected.
For service health, time slices are fine. For anything you would promise a customer, count requests. That is also why my advice for the SLA was a request success number at the load balancer, not a time-slice uptime figure.
Once the corrected numbers were in, there was a fair question: if a whole day can go by with six 5xx, why not promise zero errors.
Because one deploy, one cloud provider incident or one cache failover will produce a handful of errors, and a zero-error promise breaks on the first one. Zero errors is a good internal engineering target. It is a bad contract. The proposal I made was 99.95% request success, measured at the load balancer, counting all 5xx. I wrote more about that in a separate post on writing an SLA you can measure.
Build the first SLO at the edge of your system, where the customer is, not inside the application. The load balancer, or the CDN if you have one, sees every failure the customer sees. Application metrics are great for finding which service failed. They are a poor source for "did the customer get an answer".
And check the SLO against the incident log early. If you had an outage and your SLO did not notice, the SLO is wrong.
If you are not sure your SLOs describe what customers actually see, I can help you check. Get in touch.
