Our delivery notification functions run on AWS Lambda, about forty of them, all built from one Terraform module. Each writes to CloudWatch Logs under /aws/lambda followed by the function name, and some of those lines contain customer delivery addresses. In September our data protection officer asked for thirty days of retention on anything holding an address.
The module had never declared a log group. It did not need to. The execution role had the standard basic policy, which includes logs:CreateLogGroup, and on its first invocation Lambda creates the group itself. A group created that way has retention set to never expire.
So the first job was an inventory, and the inventory was the surprise. There were a hundred and three Lambda log groups. Sixty one of them belonged to functions that no longer existed, removed over two years by ordinary Terraform changes that destroyed the function and nothing else. Terraform had never created those groups, so it had no record of them and nothing to destroy. The oldest held addresses from 2022.
We exported what the auditors wanted to keep and deleted the orphans. Then we changed the module so this cannot build up again. It now declares an aws_cloudwatch_log_group with the function's name and retention_in_days set to thirty, and the function depends on it, so the group exists before anything can write. The forty existing groups were brought under Terraform with import blocks, because declaring them alone fails with ResourceAlreadyExistsException.
We also took logs:CreateLogGroup out of the execution role. With it, a function whose group goes missing quietly gets a new one that keeps everything forever. Without it, the function cannot write logs at all, which is loud, and we would rather have loud. Destroying a function now destroys its log group in the same plan, because both live in the same module instance.
Outside Terraform, the AWS Config managed rule cw-loggroup-retention-period-check now runs in every account and flags any log group with no retention or more than the maximum we allow. It found eleven more, in services that had never been near our module.
Terraform keeps track of what it made. The logs were made by Lambda on its own initiative, and they stayed when we deleted everything Terraform knew about.
– Sergey Shinder
