The problem
A redirect service has an unusual edge. It does not serve one domain, it serves every domain its customers bring. Each one needs a valid certificate, issued automatically when the customer adds the domain, and selected by name when a visitor connects. The service also counts every redirect for analytics and billing, so it needs the real client IP on every request. And as a bootstrapped product it had to run on a small monthly bill without cutting corners on reliability.
Three architectures
The first version, in March 2026, was a proof of concept on one small ARM instance. Caddy issued certificates on demand through its admin API, a managed database held the rules, and an Elastic IP gave customers a fixed address for their apex records. It cost about $24 a month and proved the idea. The second version, from July 2026, moved to the real product: an AWS Organization with one account per product and nothing in the management account, 15 Terragrunt modules, a self-hosted Git server with spot CI runners that scale to zero, and a gateway in the application that terminates TLS itself and picks each customer's certificate by name. The third version, in October 2026, put that gateway behind a Network Load Balancer with client IP preservation.
What went wrong
Certificate issuing never worked in the second design at first. The gateway validates domains over plain HTTP on port 80, but Caddy already owned ports 80 and 443 on the host and redirected every HTTP request to HTTPS, so the validation request never reached the gateway. A proxy in front would have hidden the client IP, and the application server cannot read the PROXY protocol. Nuno added a second private IP and a second Elastic IP to the host and sent their traffic to the gateway with nftables DNAT, leaving Caddy untouched and the client IP intact. A timer re-pointed the rule every 20 seconds because each deploy moved the gateway to the other port. The move to the load balancer cost about 4 minutes of edge downtime: the first attempt failed because a new AWS account cannot create load balancers until AWS Support enables them, and the Elastic IP had already been detached.
The launch test
On 8 and 9 October 2026 Nuno sent about 2.1M requests at the service. The main run was a 2-hour soak of 1.3M HTTPS requests with 62 errors, with no slowdown and no memory growth. All 33 redirect rule checks passed in each of 7 rounds. Metering counted exactly: 1,000 requests sent, 1,000 added to the count. The test found the limits too. New TLS connections topped out at about 190 a second because the gateway used one of the instance's two cores, while plain HTTP reached about 2,100 requests a second. It also found a launch blocker: after each application deploy the edge stopped syncing its configuration, and restarting the gateway caused a 2.5-minute outage.
Results
In the October 2026 launch test the service served 1.3M HTTPS requests in 2 hours with 62 errors, about 5 in every 100,000, and counted usage exactly. The test showed the next limit, about 190 new TLS connections a second, before any customer did. Before the load balancer, the whole AWS estate ran at about $38 to $45 a month: the August 2026 bill was $26.28 and September ran at about $1.50 a day. The load balancer adds an hourly charge on top, paid on purpose because the edge is the product.
Solutions
Accounts and cost
- AWS Organization with one account per product and nothing running in the management account.
- Network module that cannot create a NAT gateway, which would cost about $32 a month on its own.
- Spot CI runners that scale to zero when no pipeline runs.
Edge and TLS
- Per-customer certificates issued automatically and selected by name on each connection.
- nftables DNAT on a second address instead of a proxy, keeping the real client IP.
- Network Load Balancer with client IP preservation in front of the gateway.
Launch readiness
- 2-hour soak test with 1.3M HTTPS requests before launch.
- Metering checked by sending a known number of requests and counting the rows.
- Edge sync failure after deploys found before launch, with a gateway restart after each deploy as the workaround.
