Users of a production web app were getting pages that reloaded on their own. Not errors, not slow pages. The browser would do a full hard refresh in the middle of whatever they were doing.
The cause turned out to be the Horizontal Pod Autoscaler. It was scaling the main web app up and down every 1 to 2 minutes, all day.
The trouble started with a change meant to make production safer. It bundled five changes into one, including the CPU request and autoscaling settings of the main web app. Each one was reasonable on its own. Together they were hard to reason about, and when the page refreshes started, it was not obvious which of the five was to blame.
I am not going to say who wrote it, because it does not matter. It was well meant, and it is the kind of change any engineer could ship. The lesson is about the shape of the change, not the person.
The web app's pods had a CPU request of 250m. The HPA was set to keep average CPU utilization at 70% of the request. That means it tried to hold each pod at about 175m.
The pods used 400 to 500m when they were just sitting there.
Roughly what happened: the HPA saw every pod at about double its target and kept adding pods. More pods spread the load, the average moved, some pods went quiet, the HPA removed pods, the average jumped again. Utilization was measured against a number that had nothing to do with what the app needed, so small changes in traffic made big swings in the ratio. The replica count went back and forth every minute or two.
On its own, that would be wasteful but mostly invisible. Pods come and go all the time in Kubernetes.
This app used session affinity, so a user stuck to one pod. When that pod went away, the user's next request went to a different pod, and that pod could be in a different build state. When that happened, the page did a full hard refresh.
So every scale-down sent a group of users to a new pod, and some of them got a hard refresh. With the HPA flapping every 1 to 2 minutes, that was a steady stream of reloads all day.
The fix was to raise the CPU request to 1000m, so that the HPA's target matched what the pods actually used.
Within 5 minutes the app went from 16 churning pods to 4 stable ones. Four bigger pods, sized to reality, instead of sixteen small ones that the autoscaler kept creating and destroying.
That is a useful thing to show a team that thinks of right-sizing as cutting. In the same engagement I was lowering requests across most services to cut the bill. This service needed the opposite. The goal is a request that matches real use, in whichever direction that is.
While looking at the rollout, I found a second problem. The pods had a fixed 120-second delay before their readiness probe started, and it was redundant.
That delay meant every new pod sat idle for two minutes before taking traffic. Time to Ready was 134 seconds. Removing the redundant delay brought it down to 31 to 41 seconds, about 77% faster. Rolling updates went from 6 to 8 minutes to 3 to 4.
Faster readiness matters for the same reason as the flapping. The longer a new pod takes to be ready, the longer the window where traffic shifts around and builds are mixed.
The last part was about where the pods ran. Customer-facing pods were moved off interruptible capacity, so a reclaimed node would not trigger the same kind of reshuffle. That cost about $450 a month. For the pods that serve paying users, it was an easy trade.
Later in the engagement I found the mirror image on another service. Its HPA had a memory target of 1.7 GiB, but the service used 2.3 GiB at idle. Memory does not go down when you add pods the way CPU does, so the HPA could never get under the target. The service sat at its maximum replica count for 30 days.
Setting a target above idle use took it from 10 pods to 4, and the namespace's memory from 23.6 GiB to 9.5 GiB. Nothing was broken, so nothing drew attention to it. It was just expensive.
Split hardening changes. Resource requests, autoscaling targets and probes each change how a pod behaves under load. Ship them one at a time, watch each for a day, then move on. When something goes wrong, you know which one did it.
Before setting an HPA target, look at the pod's real usage at idle and at peak. If idle use is already above the target, the autoscaler will never be calm.
If your autoscaler is busier than your users, I can help you find out why. See Kubernetes & Platform.
A production rollout sat blocked for 8 days while the site looked healthy. The first week of real alerts found it, and stuck queue messages too.
Spot nodes were reclaimed, the on-demand fallback hit its own limit, and production was down for about 25 minutes. Then I moved production off spot.
Production releases failed 50% of the time. Dashboards showed where they broke, and fixing those causes took July to 10%. Duration did not improve.
