Our new carrier key ended with a newline and every pod refused to send it

On a Wednesday in October our largest carrier rotated our API key. They gave us a new key and twenty four hours before the old one stopped working, and the engineer on duty did it the way our runbook said. He pasted the new key into a file, built a Kubernetes Secret from it with kubectl create secret generic --from-file, sealed it and merged the change. The rollout restarted the booking service's pods with the new key, and for the next thirty five minutes no shipment could be booked with that carrier.

The carrier's dashboard showed no failed requests at all. That was the first clue, because it meant nothing had reached them.

The booking service is written in Go and puts the key in an Authorization header. Every call was failing inside our own pod with an error saying the header field value was invalid. Go's HTTP client refuses to send a header value that contains a newline, because a newline inside a header is how header injection works, and it fails before a single byte leaves the process.

The newline came from the editor. He had written the key to the file with vim, and vim ends the last line of a file with a newline by default. --from-file stores the bytes of the file exactly as they are, so the Secret held the key followed by a newline, and the environment variable built from it did too. Our staging Secret had been created with --from-literal, which takes the value from the command line and adds nothing, so the same deploy had passed there an hour earlier.

We rolled back the Secret and put the new key in that evening without the newline.

Secrets are now written with --from-literal or with printf '%s', never from an editor's file, and the runbook includes a check that decodes the Secret and shows its last byte. The service checks its own credentials at startup, and a key with leading or trailing whitespace keeps the pod from becoming ready, so a bad Secret stops a rollout with most of the old pods still serving. A Secret stores bytes, not what you meant to type. Ours stored one byte more than the carrier had given us.

– Sergey Shinder