Our deploy workflow runs on every push to main, and its production job names an environment called production. The environment has required reviewers, a team of six senior engineers, so the job waits until one of them presses Approve. In our security policy that is written down as a two person rule: nobody can put a change into production alone.
In September an internal audit sampled three months of deployments and pulled the approval history for each run from the API. Of 214 production deploys, 58 had been approved by the same person who had merged the pull request and so started the run. Most were small and harmless. One of them was a change to our refund limits that had been merged on a Friday evening without a code review, because branch protection on that repository had been loosened during an incident in July and never tightened again.
Nothing had been bypassed. Required reviewers means that any one person on the list can approve, and GitHub does not exclude the person who triggered the run unless you tick a separate box, Prevent self-review. We had never ticked it, because when we set the environment up the option did not exist, and nobody went back to read the settings page when it appeared. Our six reviewers were also the people who merged most often, so for a quarter of our deploys the second person was the first person.
Three changes. Prevent self-review is now on for every production environment, and a scheduled job reads the environment settings across the organisation and opens an issue for any that has it off. Branch protection on deployable repositories is managed in Terraform, so a rule relaxed by hand during an incident is put back by the next apply and shows up in its plan before that. And every week a report lists each production run with the person who triggered it and the person who approved it, and flags any row where they match.
The policy described the control we wanted. The settings page described the one we had, and for two years nobody had compared them.
– Sergey Shinder
