A few months into a cost engagement, production went down for about 25 minutes. The cause was part of production running on spot capacity, with an on-demand pool as the fallback.
The fallback was there. It just could not fall back far enough.
Spot nodes running production pods were reclaimed. That is normal. Spot capacity can be taken back at short notice, and the whole design assumes it will be.
The node autoscaler tried to launch replacements on spot and could not. So it turned to the on-demand pool, which was configured as the safety net.
The on-demand pool had a limit on how much CPU and memory it was allowed to provision. When the incident started it was already using 46.2 GiB of its 48 GiB memory limit. Each of the pods that needed a new home asked for 2.44 GiB. There was room for none of them.
So the pods sat pending. The autoscaler was doing exactly what it was told: never let this pool grow past its limit. Nobody had connected that limit to the question "is this enough to carry production if spot disappears".
Once the cause was clear, recovery was one change: raise the pool's limit. New nodes came up and the pending pods were scheduled within about 5 minutes.
The rest of the 25 minutes was the time it took to see why nothing was being scheduled. The pods were pending and the autoscaler was running, but nothing was alerting on the fact that it could not launch nodes.
Here is what I took from it. When you set a resource limit on a node pool, you usually think of it as a cost guardrail. It stops a runaway deployment from renting a hundred nodes. That is a good reason to have one.
But if that pool is your fallback, its limit also decides how much of production can survive losing the primary capacity. A cost control quietly became an availability setting, and nobody had sized it for that job. I had not looked at it that way either.
The same limit showed up again about two weeks later. A weekday-morning error spike turned out to have three causes, and one of them was the same capped node pool. Once you know to look for it, it is everywhere.
After the incident, I moved production off spot entirely. That costs about $1,000 a month.
I want to be clear that this was a choice, not a correction. Spot for production can work. Plenty of teams run it well, with a wide spread of instance types, a fallback sized for the whole workload, and good alerting. It saves real money.
But this was a customer-facing product. The question was not "can spot work here" but "what does an hour of downtime cost compared with $1,000 a month". For this business, the answer was easy. Earlier in the engagement I had already pinned the customer-facing web pods to regular capacity for about $450 a month, after a different problem. This finished the job.
Non-production and CI stayed on spot, because that is where interruptible capacity makes sense: a reclaimed node there costs a few minutes of someone's time, not a customer's trust.
Paying more for reliability does not mean giving up on the bill. The monthly spend had already come down by 55% earlier in the engagement, mostly from right-sizing and removing waste. That work continued, and right-sizing is the place to find the next $1,000, not spot capacity in production.
When I wrote up the cost case, I included this. Spend fell 55% and held for four months, then some of it was deliberately spent back on reliability. I would rather show that trade-off than pretend the lowest number is always the best one.
The outage was long because nothing paged on the actual problem. Afterwards I added:
Either one would have pointed at the cause much sooner. Both are cheap to set up.
I would size the fallback pool's limit from the production workload, not from a cost estimate. If the pool exists to absorb a spot loss, its limit has to fit everything that might move onto it, with room left over.
And I would test it. Cordon the spot nodes in a quiet window, or simulate a reclaim, and watch whether the fallback really catches everything. Ten minutes of a planned test is cheaper than 25 minutes of an unplanned one.
If you want a second look at where your cluster saves money and where it takes risks, I can help. See Kubernetes & Platform.
On every cluster I looked at, workloads requested 7 to 20 times the CPU they used. The node autoscaler bought nodes for the requests, not the usage.
A production rollout sat blocked for 8 days while the site looked healthy. The first week of real alerts found it, and stuck queue messages too.
A CPU request far below real use made the HPA scale up and down every 1 to 2 minutes, and users got hard refreshes. Raising one number fixed it.
