The 8-Day Outage Nobody Saw

On 29 September 2026 I applied a new set of alert rules to a production GKE cluster. Within minutes they were firing. Fifteen pods had been failing to start since 21 September. Eight days. Nobody had noticed, because the site was up.

This post is about why that happens, and the two alerts I now put on every Kubernetes platform before anything else.

The stack was there, it just said nothing

The platform already had Prometheus, Grafana, Loki, Tempo and two OpenTelemetry collectors, all deployed by Terraform. On paper it had observability. In practice it had 0 alert rules, 0 alert receivers, 0 dashboards and 0 SLOs, and ingress metrics were switched off.

It was worse than empty. Two alerting engines were running. The default rules in one of them were firing into an Alertmanager with no receivers configured. So alerts were being raised and then dropped, with no log line anyone would read. If you had asked the team "do we have alerting?", the honest answer was "sort of", and "sort of" is the dangerous answer.

The work was scoped to basic backend health for a small pilot: 15 tickets, 80 hours. I ended up with 9 dashboards and 24 alert rules. Every rule got a stable ID, a severity and, for nearly all of them, a link to a short runbook. Critical production alerts went to both the team chat and the support mailbox.

Then I applied it to production, and the alerts started saying things.

What the first alerts found

A secret was missing in production. It existed in the other environments, so everything built and tested fine. In production, every pod that needed it sat in CreateContainerConfigError.

Here's the part that matters. Kubernetes did exactly what it's designed to do. The new ReplicaSet couldn't start, so the old one kept serving. The admin frontend and 6 workers were stuck on their previous versions. 6 database migrations were waiting behind them. From the outside, every URL answered. Uptime checks were green.

Once the secret was in place, all 18 production pods came up and 16 alerts resolved within minutes.

The second find was quieter. Messages on Pub/Sub subscriptions had been stuck for about 7 days, in dev, test and production. About 27,000 undelivered in dev, 46 on one production subscription. Nothing was failing loudly. Messages just weren't being consumed.

The first real critical production alert fired that same day. It reached the chat and the support mailbox with its runbook link. That became the project's test incident, which is a much better test than a synthetic one.

Running is not the same as rolling out

The 8-day outage didn't show up because every common health signal was answering a different question.

• "Is the site up?" Yes. The old pods were serving.
• "Are pods running?" Yes, enough of them. The old ones.
• "Did the deploy succeed?" Nobody was asking. The pipeline had pushed the manifest and moved on.

The question that would have caught it is "does the cluster have the version we asked for?" In Kubernetes terms, that's whether available replicas match desired replicas for the current spec, and whether any container is stuck waiting.

Queue backlogs are the same shape. A queue with no consumer doesn't throw errors. It just gets older. Message count alone can mislead too, since a queue can be small and still stuck. The useful number is the age of the oldest unacknowledged message.

The two alerts I add first now

These are generic and safe to copy. Tune the durations to your deploy times.

- alert: DeploymentReplicasMismatch
  expr: |
    kube_deployment_status_replicas_available
      < kube_deployment_spec_replicas
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "{{ $labels.namespace }}/{{ $labels.deployment }} has fewer available replicas than desired"

- alert: ContainerStuckWaiting
  expr: |
    sum by (namespace, pod, reason) (
      kube_pod_container_status_waiting_reason{
        reason=~"CreateContainerConfigError|ImagePullBackOff|CrashLoopBackOff"
      }
    ) > 0
  for: 10m
  labels:
    severity: critical

For queues, alert on the oldest unacknowledged message age per subscription, with a threshold based on how fast that consumer normally drains. On Google Cloud that's a Cloud Monitoring metric you can read straight into Grafana.

Neither alert is clever. Both would have caught a problem that sat in production for over a week.

What surprised me

I expected the first week of alerts to be mostly noise and tuning. Instead, the first two things they found were real, old and in production.

I also expected CPU and memory thresholds to matter early. For this workload they didn't, and I left CPU threshold alerts out of the pilot scope on purpose. Disk and out-of-memory alerts stayed in. CPU alerts on a service that rarely gets busy just teach people to ignore the channel.

What I'd do differently

I'd put the replicas-mismatch alert in on day one, before dashboards, before anything. It's ten lines and it needs only metrics the cluster already exposes. I spent the first part of the project on ingress metrics and dashboards, which were needed, but that alert would have caught the blocked rollout days earlier.

I'd also check every Alertmanager for receivers before writing a single rule. An Alertmanager with no receivers is worse than no Alertmanager, because it looks like coverage.

A short checklist

1. List every alerting engine in the cluster. Confirm each one has a receiver that a human reads.
2. Alert when available replicas stay below desired for longer than a normal deploy.
3. Alert on containers stuck in config, image pull or crash loop states.
4. Alert on the age of the oldest message in each queue, not just the count.
5. Fire a real test alert end to end and check it lands where you think it does.

If your monitoring stack is installed but quiet, I can help you find out what it's missing. See Observability & SRE.