The problem
Nuno's assessment on 1 September 2026 found the monitoring tools were all installed and none of them were wired up. There were no alert rules, no dashboards, no scrape targets for the services and no SLOs, and ingress metrics were switched off. Two alerting engines were running. The default rules in one of them fired into an Alertmanager with no receivers, so every alert it raised was dropped without a trace.
Scoping it down
Nuno's first estimate was 212 hours and covered APM, tracing, SLOs and sizing for the launch peak. In mid-September the client reset the goal to basic backend health for a pilot of 10 to 15 users. Nuno replanned it as 15 tickets and 80 hours, split into required, nice-to-have and later tiers, and the client approved it the next day. Disk and memory alerts stayed in. CPU threshold alerts went out, because they would have been noise for this workload.
What Nuno did
Nuno turned on ingress metrics first, since nothing about user traffic was visible without them. Dashboards were provisioned as code: JSON in git, delivered by Terraform and loaded by the Grafana sidecar, so nobody has to click them back together after a rebuild. Alert rules use stable IDs so they survive redeploys. Grafana reads Cloud Monitoring through Workload Identity for the managed database and storage, with no keys to rotate. Nuno added queue health for Pub/Sub, metrics from the hosted workflow engine, and scraping for the GitOps controller and certificate manager. Alerts route by severity: every alert goes to the team chat, and critical production alerts also go to the support mailbox. Nearly every alert links to a short runbook.
What the alerts found
From 21 to 29 September a secret missing in production left 15 pods unable to start. No new version of the admin frontend or of 6 workers had rolled out, and 6 database migrations were waiting. Production kept serving the old pods, so from the outside it looked healthy. Once the secret was in place, all 18 production pods ran and 16 alerts resolved within minutes. The second find was queue messages stuck for about 7 days in dev, test and production: about 27,000 undelivered in dev and 46 on a production subscription. The first real critical production alert fired on 29 September, reached the chat and the support mailbox with its runbook link, and served as the project's test incident.
Results
The platform went from 0 to 9 dashboards and from 0 to 24 alert rules in 15 tickets and 80 hours, live in dev and production on 29 September 2026. In the first week the alerts surfaced a production rollout blocked for 8 days, with 15 pods down and 6 migrations waiting, and queue messages stuck for about 7 days. There is no before and after for response time or alert noise, because before this there were no alerts at all.
Solutions
Dashboards and metrics
- 9 dashboards in 3 folders, provisioned from git through Terraform.
- Ingress metrics switched on, plus queue health, workflow engine metrics, GitOps and certificate scraping.
- Cloud Monitoring data read through Workload Identity, with no service account keys.
Alerting
- 24 alert rules in 6 groups, with stable IDs.
- Severity routing: chat for everything, email as well for critical production alerts.
- Runbook links on 22 of the 23 rules listed in the delivery report.
