Our login endpoint sits behind AWS WAF, and one rule in the web ACL is a rate based statement: any single address that sends more than a set number of login requests in five minutes is blocked. The web ACL is Terraform, and the number is a variable, login_rate_limit. Our pipeline runs plan and apply for production from GitHub Actions, and it passes production's settings to Terraform as environment variables, TF_VAR_login_rate_limit among them, taken from the variables of the production environment in GitHub.
That was deliberate. The runbook for credential stuffing says to lower the variable in GitHub and run the edge workflow, which changes the limit in a few minutes without waiting for a pull request.
At ten past three on a Saturday in October, logins were arriving from about eleven hundred addresses, each sending between seven and nine hundred attempts every five minutes, under our limit of two thousand. The engineer on call followed the runbook and set the variable to three hundred. The plan said No changes. Your infrastructure matches the configuration. He ran it again, printed the variable in the job to make sure it was set, and it was. Terraform said no changes again.
Terraform reads variables from several places, and later sources win. Environment variables come first. Then terraform.tfvars, then terraform.tfvars.json, then any .auto.tfvars files in lexical order, and last whatever is passed with -var or -var-file. In April somebody had added a terraform.tfvars to the edge module, holding production's current values, so that plans run on a laptop would match the pipeline. The pull request said that CI overrides these anyway. It does not. The file was read after the environment, and its two thousand replaced his three hundred on every run.
At five past four a colleague passed the value with -var in the workflow, the plan showed one change, and the attack was blocked within minutes. By then forty three customer accounts had been signed in to with reused passwords. We locked them that morning and wrote to their owners.
There is no terraform.tfvars or .auto.tfvars file in any root module now, and the pipeline fails if one appears. Each environment's values live in a file the pipeline names explicitly with -var-file. An emergency change is a workflow_dispatch input passed with -var, the strongest source Terraform has, and the workflow fails if a plan made with an override shows no changes. Every value the runbook tells people to change is also an output, so the plan shows what Terraform will really use. And the runbook is rehearsed in staging once a month, which would have found this in April.
We had built our emergency control on the first source Terraform reads, which is the one every other source overrides.
– Sergey Shinder
