The problem
About 45 imperative shell scripts built everything: host bootstrap, cluster add-ons, secrets typed into Vault by hand, per-app database setup, DNS, and deploy wrappers. They had to run in the right order, and that order lived in people's heads. The pieces inside the cluster were in decent shape. The glue between Vault, Postgres, Kubernetes and DNS was not, and there was no CI for infrastructure at all.
What Nuno did
Nuno wrote a migration plan in five layers and picked one tool for the job, Terraform with cloud-init, instead of a Terraform and Ansible mix. Six modules now cover the bare-metal server and its bootstrap (private networking, Postgres, Vault, k3s, node monitoring and firewall), Vault policies and Kubernetes auth, Postgres roles and databases, DNS, object storage, and the backup bucket. The git and CI server is built first from a laptop, and CI then builds everything else. CI lints and validates every change, and applies only through a guarded manual pipeline. Before touching production Nuno proved the whole thing on a throwaway environment. Nuno also benchmarked the new hosts, found them much slower on disk, and moved production back onto the fast server, this time with disk encryption rehearsed first on a throwaway box.
What went wrong
The first control-plane cutover attempt, on a Sunday in late May, failed when a script bug wiped the staged snapshot. Nuno rolled back, fixed the script and tried again that evening. The control plane moved, and about 66 minutes after the retry started all app pods were running and every URL was healthy. Nuno had also added a GitOps controller as an upgrade, then removed it on purpose a few days later because it fought the existing manual canary releases. The release flow the client already trusted stayed.
Results
On a smoke test in May 2026, a fresh bare-metal server went from terraform apply to a validated host with all six health checks green in about 17 minutes. The next day the CI pipeline planned against it and reported no changes, so the CI-built and laptop-built state matched. The git and CI platform moved with 693 pipeline runs of history. The production cutover rolled back once and then completed in about 66 minutes the same evening. The documented rebuild from zero is about 90 minutes, with a few manual steps kept on purpose, such as unsealing Vault.
Solutions
Infrastructure as code
- About 45 hand-run scripts replaced by 6 Terraform modules and 5 environments.
- Server bootstrap through cloud-init, checked by health checks at the end of every build.
- The existing production server imported into Terraform, and a retired server later destroyed through Terraform.
CI and platform
- Lint and validate on every change, with apply behind a manual guard.
- Git and CI platform moved to its own server with all users, repositories, issues and 693 pipeline runs.
- Production data on encrypted disks, with nightly versioned backups added later.
