At 14:44 UTC on Sunday 31 May 2026 I started moving a small k3s cluster's control plane onto a new, disk-encrypted server. A few steps in, I rolled it back. A bug in my own script had deleted the snapshot I was about to restore from.
At 19:21 I started again. At 19:32 the control plane was on the new server. By 20:27 all three app pods were running and every URL was healthy. About 66 minutes from retry to done.
This is what the script looked like, what broke, and why the rollback was boring, which is exactly what you want from one.
The control plane ran on an older bare-metal server. The plan was to move everything onto a re-provisioned server with encrypted disks. The data services (Postgres, Vault, object storage) had already been moved across to the new server in the days before. What was left was the control plane itself, plus the persistent volumes that hadn't moved yet.
Before starting I took a full backup of the databases and persistent volumes, and kept it off both servers. That backup was the real safety net. Everything else was about not needing it.
I wrote the cutover as a script in phases. Each phase did one thing, checked it, and had a matching rollback. The phases followed the obvious order: snapshot the control-plane datastore, move the control plane, point the nodes at it, then restore the remaining volume data and check the workloads.
If any check failed, the script stopped and the operator, me, ran the rollback for that phase. Not the whole thing. Just undo the step you're on and the ones before it, in reverse order.
This is more work than one long script. It's worth it because, when something goes wrong, you don't have to think. You run the next line of the rollback.
Two bugs, both mine.
The snapshot lived inside a directory the uninstaller wipes. The script staged the datastore snapshot in a folder under the cluster's data directory. A later phase ran the uninstaller, and the uninstaller removes that whole directory. So by the time I went to restore the snapshot, it was gone.
An environment variable never reached the installer. The install line looked like this:
INSTALL_OPTION=value curl -sfL https://example.com/install.sh | sh -
That sets INSTALL_OPTION for curl, not for sh. The installer never saw it. The fix is to put the variable on the other side of the pipe:
curl -sfL https://example.com/install.sh | INSTALL_OPTION=value sh -
It's a classic shell mistake.
When the restore failed, I ran the rollback for the phases I'd done and service went back to the old setup. The old node had kept serving part of the traffic throughout. I don't have a clean measure of exactly what users saw during the failed attempt, so I won't claim zero downtime. What I can say is that the rollback was a few known commands, not an investigation.
Then I fixed the two bugs. The snapshot moved to a path outside anything the installer touches, and the variable moved to the right side of the pipe.
At 19:21 I ran it again. The control plane was on the new server at 19:32. About 5 GB of persistent volume data was restored. At 20:27 all three app pods were running and every public URL returned healthy.
Stage artifacts outside every path the process can delete. Better still, outside the server. A snapshot is only a safety net if nothing in your own procedure can touch it.
Test the script end to end on a throwaway server first. I had rehearsed the disk encryption on a throwaway box, but not this exact cutover script. A dry run on a scratch machine would have caught both bugs in an afternoon, at no risk.
Check environment variables where they're used. If an installer behaves as if an option wasn't set, assume it wasn't. Print it from inside the script you're calling.
Rolling back isn't failing. Not having a way back is.
If you have a migration coming up and want help planning the cutover, get in touch.
New servers turned out 8 to 13 times slower on sequential disk than the box they replaced. Ten minutes of dd and openssl would have shown it before the move.
About twenty end-of-life VMs, single points of failure and a CDN that cached nothing. What moving AEM onto one AKS cluster took, and what I'd change.
Spot nodes were reclaimed, the on-demand fallback hit its own limit, and production was down for about 25 minutes. Then I moved production off spot.
