The problem
There were no SLOs, no shared dashboards, and a self-hosted metrics stack running next to the paid observability tool. Two production problems early in the engagement made the gap obvious: one caused repeated hard refreshes for users, the other was a memory leak growing pods by about 110 MiB an hour. Both were found by people, not by alerts.
What Nuno did
Nuno built shared dashboards and about 40 production SLOs, grouped into four tiers. The critical path targets 99.9% and counts five times in a weighted composite score. Core services target 99.5%, and supporting and background work target 99%. Service health uses time-slice SLOs, and the main user flows use synthetic checks. Nuno set stability standards for every production service: at least two replicas with a disruption budget, customer-facing pods protected from node consolidation, and no consolidation during peak hours. Nuno wrote the incident playbook, the severity levels and the postmortem template, and ran incidents as commander.
Why the first measure was wrong
In late September Nuno checked the main request success SLO against what clients actually received. It counted only errors returned by the application. Errors produced by the load balancer itself (502, 503 and 504) were not counted, and those were about 90% of the 5xx responses clients saw. With the corrected method, success for late August to late September was 99.9575%, not the 99.9944% Nuno had reported, and total 5xx had stayed flat instead of falling. Nuno also found that synthetic tests were 9.6% of all production requests and had barely registered a real outage. Nuno logged the correction against the earlier report, rewrote the SLI to count every 5xx at the load balancer, and moved the reliability dashboard to request-based SLOs without synthetic traffic.
The SLA proposal
When leadership asked what to promise customers, Nuno's first proposal was 99.9% monthly availability from external synthetic checks. Two weeks later, once it was clear synthetics had missed a real outage, Nuno changed the proposal: a contractual SLA of 99.95% request success, measured at the load balancer and counting all 5xx. Zero errors stays as an internal engineering target, not a contract term, because a single deploy, cloud incident or cache failover would break a zero-error promise.
Results
In the first weeks of October 2026, week over week, the load balancer request success SLO went from 97.42% to 99.06% and the pod running rate from 84.77% to 99.75%. After Nuno fixed the morning spike, there were zero load balancer 5xx on seven weekday mornings in a row. One full day in mid-October had six 5xx responses in total, 99.9991% success. On the new request-based SLOs, 7-day availability was 99.993% and crash-free sessions were 99.87%. The proposed SLA is measured the same way customers experience the product.
Solutions
SLOs and dashboards
- About 40 production SLOs in four weighted tiers, rolled up to one composite score.
- Request-based success SLO counting all 5xx at the load balancer, with a 99.95% target over 30 and 7 days.
- Synthetic traffic removed from production metrics.
Alerting and incidents
- Paging on production 5xx within 5 minutes, on sustained under-replicated services and on node provisioning failures.
- New alerts backtested over 7 days before going live.
- Incident playbook, severity levels and postmortem template.
Fixes behind the numbers
- A weekday morning 5xx spike traced to a nightly job running inside a web pod, a keep-alive timeout mismatch and a capped node pool and fixed by Nuno.
- Minimum replicas and disruption budgets for every production service.
