One Wrong Keyword Silently Stopped Hundreds of Terraform Stacks

More than 400 Terraform stacks had stopped reacting to git pushes. Nothing failed. No run went red. Merges landed, and the infrastructure just didn't change.

The cause was one word in a shared policy: allow, where the policy type only understands track, propose and ignore.

How push policies work

The IaC orchestration tool these stacks ran on decides what to do with a git event using a push policy, written in Rego. For each push or pull request, the policy can say:

• track: this push should trigger a tracked run, the kind that can apply;
• propose: run a proposed plan, typically for a pull request;
• ignore: do nothing.

If none of those rules match, nothing happens. That's the important part. A push policy that matches nothing doesn't error. It quietly ignores the event.

Other policy types in the same tool use different rule names, and allow shows up in some of them and in most generic Rego examples. It's easy to bring a habit from one into another.

The bug

A shared push policy for "deploy on merge" had been written with allow rules. It was auto-attached to every stack with a certain label, which turned out to be more than 400 stacks.

# What the policy effectively said. The tool never reads "allow"
# in a push policy, so no rule matched and every push was ignored.
allow {
  input.push.branch == input.stack.branch
}

On top of that, the policy referenced an input field that no longer exists in the push policy input schema. The policy had worked at some point, so the likely story is that the input it depended on changed upstream and nobody noticed, because the failure mode is silence.

The fix

I rewrote it as a single-purpose, track-only policy using Rego v1 syntax:

package pushpolicy

import rego.v1

track if {
  input.push.branch == input.stack.branch
}

In review two good changes came in. First, add an explicit guard so only real merge commits to the tracked branch trigger an apply. Second, keep the "propose a plan on pull requests" logic in its own separate policy instead of folding it into this one. One policy, one job, so the next person can read it in ten seconds and see what it does.

That second point matters more than it sounds. When a policy does three things, a mistake in one of them hides behind the other two.

Why a bug like this goes unnoticed

Silent failures in CI are worse than loud ones, and this kind is as silent as it gets:

• each stack still shows its last run, and the last run was fine;
• merges went through review and landed normally;
• nothing in the policy log looked like an error, just "ignored";
• most stacks don't change every day, so a week without runs looks normal.

The only symptom was absence, and absence is the hardest thing to alert on.

Test policies against real events before you merge

The tool records policy evaluations, and you can replay a past input against a changed policy before you merge it. That's the cheapest test you'll ever run. Take a real push event from a stack that should be tracked, run the new policy against it, and check the result says track. Then take one that should be ignored and check that too.

For any shared policy, I now want that replay result pasted into the pull request. If the policy is attached to hundreds of stacks, a reviewer reading Rego in a diff is not enough.

Alert on silence

The bigger lesson is about monitoring. A pipeline that does nothing needs its own alert, because none of the usual signals fire. Two that would have caught this:

1. A canary stack. A small, harmless stack attached to the same shared policies, with a scheduled commit that should always trigger a run. If it doesn't run, something upstream is broken.
2. Commits without runs. Compare the latest commit on each stack's tracked branch with the commit of its latest run. If a stack has had commits for more than a day or two and no run, flag it.

Neither is hard to build. Both turn "nothing happened" into something a human sees.

What I'd do differently

I'd treat shared policies like shared libraries: versioned, tested against recorded inputs, and owned by someone. A policy attached by label to hundreds of stacks has the blast radius of a platform change. It deserves the same care, not the care you'd give a one-off stack setting.

I'd also check what labels a policy is auto-attached by, and how many stacks match, before touching it. "About 400" is a very different review from "about 4".

Checklist for shared CI policies

1. Know the rule names each policy type actually evaluates. Don't copy from another type.
2. One policy, one purpose.
3. Replay real recorded events against any change before merging, and paste the result.
4. Keep a canary stack on every shared policy.
5. Alert when tracked branches have commits but no runs.

If your infrastructure pipelines have grown to hundreds of stacks and you're not sure they all still run, I can help you check. See Fractional Platform Team.