Our customer portal is deployed to fourteen web servers by a deploy user that copies each release into /srv/portal/releases and switches a symlink. The releases directory belongs to the www-data group, and deploy is a member, which is the only reason it can write there.
In October support asked for the deploy job to collect the service's journal after each release, so they could see startup errors without logging in. Reading the system journal needs membership of the systemd-journal group, so we added one task to our base role: the user module, name deploy, groups systemd-journal.
The play ran green on all fourteen servers on a Tuesday. Nothing happened until Thursday's release, when every server failed the copy with permission denied on the releases directory.
The user module's groups parameter does not mean add the user to these groups. It means these are the user's supplementary groups, and by default the module removes the user from every group not in the list. Adding is what append: true is for. Our task had quietly turned deploy's membership of www-data and adm into membership of systemd-journal alone. Ansible reported the task as changed, which was true, and nobody reads changed on a user task as a removal.
The two day gap was the second lesson. Group membership is read when a session starts. Processes and sessions that already existed kept the old groups, so the change only bit when Thursday's deploy opened a fresh SSH connection. By then the play that caused it was two days and three other changes in the past, and we spent the first hour suspecting the release itself.
We added append: true, ran the role again, and the release went out forty minutes late. Every user task in our roles now states append explicitly, and a lint rule rejects a user task that sets groups without it. The base role ends with a check that compares each service user's groups with a list kept in the inventory, so a group disappearing fails the play on the day it happens, not at the next release.
The task said what we wanted to add. The module read it as the complete list.
– Sergey Shinder