The problem
All four clusters (development, CI, staging and production) were on a Kubernetes version past standard support. EKS charges $0.60 an hour per cluster in extended support instead of $0.10, and in April the fee came to $1,728 for the month. The autoscaler and several controllers were far behind, so every upgrade also meant upgrading the tools that place and route the workloads.
What Nuno did
Nuno used the same sequence on every cluster, starting with development, then CI, then production. First the code: clean infrastructure plans before touching anything. Then the add-ons, one at a time, from the least to the most customer-visible. Then the control plane. Then the nodes, rolled one at a time, with snapshots of cluster state taken before and after. The first upgrade, 1.32 to 1.33, ran on all four clusters in the last week of May. The second, 1.33 to 1.34, ran in mid-July on the three clusters that remained after the unused staging environment was removed. Each time Nuno upgraded the node autoscaler, the load balancer and ingress controllers, the certificate manager and the storage drivers. The autoscaler went from 1.4 to 1.11 over the two upgrades.
What surprised Nuno
A few things did not behave the way the documentation suggested. A node pool disruption budget of 10% on a small pool rounds down to zero, so drifted nodes never rolled until the budget was changed. A short-lived cloud session that expired halfway through an apply produced a misleading error that pointed at the code, not the credentials. The autoscaler chart stopped shipping its custom resource definitions, so they had to be managed separately. And one permission the planning notes said was required turned out not to be.
Where the seconds came from
The production node roll covered 16 nodes, one at a time. The 15 to 30 seconds customers could see came from a cache node moving to a new host while the application reconnected. Nuno filed a follow-up so the application reconnects faster next time.
Results
Two Kubernetes upgrades between late May and mid-July 2026, with about 15 to 30 seconds of customer-visible disruption during the production upgrade. The extended support fee, $1,728 in April, dropped by about $1,440 a month. By September all three remaining clusters ran the same 1.34 patch release with no version drift, and the same sequence is ready for the next upgrade.
Solutions
Upgrade method
- One repeatable sequence for every cluster: code, add-ons, control plane, nodes, then state sync and review.
- Development first, production last.
- Cluster state snapshots before and after each upgrade.
Add-ons
- Node autoscaler, ingress and load balancer controllers, certificate manager and storage drivers upgraded one at a time.
- Infrastructure providers moved to their current major versions.
Production nodes
- Nodes rolled one at a time behind disruption budgets.
- Follow-up filed for faster cache reconnects in the application.
