Every small fleet reaches the same moment: you add a new engineer, and then you copy a public key into authorized_keys on eleven machines. Someone leaves, and you repeat the ritual in reverse, hoping you remembered the jump box nobody documents. Host keys are the other half of the problem — every rebuild produces a new fingerprint and another WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED.

OpenSSH certificates fix both problems without replacing SSH. Instead of every server trusting every user key, every server trusts one certificate authority (CA) key. The CA signs short-lived certificates that bind a public key to a username, a set of permissions, and an expiry time. The server checks the signature and the identity, and never needs a per-user file again.

This is not X.509. OpenSSH certificates are a much simpler format, and the operational details are different enough that most people bounce off them once. Here is the mental model, a working lab, the server configuration, host certificates, and revocation that actually works.

What an OpenSSH certificate contains

A certificate is a public key plus signed metadata. When you inspect one with ssh-keygen -L, you see:

  • Type — user or host certificate.
  • Public key — the key being certified, not a new key.
  • Signing CA — which CA key signed it.
  • Key ID — a human-readable label written to server logs.
  • Serial — a number you can use for revocation.
  • Valid — start and end times.
  • Principals — the usernames (or hostnames) this certificate may authenticate as.
  • Critical Options — restrictions the server must understand, or it rejects the certificate.
  • Extensions — permissions such as permit-pty or permit-agent-forwarding.

The private key never leaves the client. The certificate is a separate *-cert.pub file. The CA key is an ordinary SSH key; there is no special key type and no certificate hierarchy. Clients and servers trust the CA public key directly, which means there are no intermediate CAs to manage — and no chain to constrain a compromised CA.

The two defaults that make people give up

The ssh-keygen(1) manual says: “By default, generated certificates are valid for all users or hosts.” That sounds convenient. It is also a trap, because sshd_config(5) says certificates that lack a list of principals “will not be permitted for authentication using TrustedUserCAKeys.”

So a certificate signed like this:

ssh-keygen -s user_ca -I alice alice.pub

is valid forever, allows every forwarding permission OpenSSH knows about, and is rejected by a correctly configured server because it has no principals. Always pass -n:

ssh-keygen -s user_ca -I alice -n alice -V +12h alice.pub

The second default is equally sharp: without -V, the certificate is valid from the Unix epoch to the distant future. Without -O clear, it grants X11 forwarding, agent forwarding, port forwarding, PTY allocation, and ~/.ssh/rc execution. A default certificate is a permanent, fully privileged credential. Treat -V and -O clear as mandatory, not optional hardening.

Lab: a user CA in five commands

Run this on a machine you control. The CA private key should not live on the servers you log in to.

install -d -m 700 ~/.ssh/ca ssh-keygen -t ed25519 -a 100 -C "user-ca@example.com" -f ~/.ssh/ca/user_ca ssh-keygen -t ed25519 -a 100 -C "alice@laptop" -f ~/.ssh/id_ed25519 ssh-keygen -s ~/.ssh/ca/user_ca \ -I "alice@laptop-2026-11-11" \ -n alice \ -V +12h \ -z 1001 \ -O clear -O permit-pty \ ~/.ssh/id_ed25519.pub ssh-keygen -L -f ~/.ssh/id_ed25519-cert.pub

The -I value is the key ID that lands in the server log. The -n value must match the Unix account, unless you add a principals file later. -z gives the certificate a serial number for revocation. -O clear removes the default forwarding permissions, and -O permit-pty adds back only what an interactive shell needs.

On OpenSSH 10.0p2, the inspection output looks like this (fingerprints shortened):

alice-cert.pub: Type: ssh-ed25519-cert-v01@openssh.com user certificate Public key: ED25519-CERT SHA256:itwADN+Ol2Tk... Signing CA: ED25519 SHA256:HDzAN1TPyjP... Key ID: "alice@laptop-2026-11" Serial: 1001 Valid: from 2026-11-11T05:00:00 to 2026-11-11T17:01:20 Principals: alice Critical Options: source-address 203.0.113.0/24 Extensions: permit-pty

That example also used -O source-address=203.0.113.0/24. It is useful for a fixed office or VPN range, but it is a poor fit behind carrier NAT or a load balancer that changes the apparent source address. The server evaluates the address it sees, not the address the client thinks it has.

Critical options versus extensions

The distinction matters when you write your own policies. An unknown critical option causes the certificate to be refused. An unknown extension is ignored. Use critical options for restrictions that must fail closed, and extensions for capabilities that older servers may safely skip.

Server side: trust the CA, not the user

Copy only the CA public key to the server:

sudo install -o root -g root -m 644 ~/.ssh/ca/user_ca.pub /etc/ssh/user_ca.pub

Add one line to /etc/ssh/sshd_config:

TrustedUserCAKeys /etc/ssh/user_ca.pub

Validate and reload:

sudo sshd -t sudo systemctl reload ssh

On RHEL-family systems the unit is usually sshd instead of ssh. Then log in normally:

ssh -i ~/.ssh/id_ed25519 alice@server.example.com

OpenSSH automatically loads a certificate named by appending -cert.pub to the identity file, so keeping id_ed25519 and id_ed25519-cert.pub side by side is enough. If the files have different names, point at the certificate explicitly:

ssh -i ~/.ssh/id_ed25519 -o CertificateFile=~/.ssh/alice-cert.pub alice@server.example.com

Use ssh -vvv when it fails. The client prints which keys and certificates it offered, and the server log records the accepted key ID. A successful login usually appears as Accepted publickey with the certificate identity in the log line. The important part is that authorized_keys was never consulted.

Principals are not usernames

By default, the certificate's principal must equal the target Unix username. That is fine for a small team, but it couples every certificate to an account name. AuthorizedPrincipalsFile decouples them:

AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u

Create the directory and a file per account:

sudo install -d -o root -g root -m 755 /etc/ssh/auth_principals echo 'alice-admin' | sudo tee /etc/ssh/auth_principals/alice sudo chmod 644 /etc/ssh/auth_principals/alice

Now a certificate signed with -n alice-admin can log in as alice, while a certificate with principal bob-admin cannot. The %u token expands to the target username. If the file exists and the certificate has no matching principal, authentication fails even though the CA signature is valid.

For a directory-backed fleet, AuthorizedPrincipalsCommand can generate the list dynamically. The command must be owned by root and not writable by group or others, and sshd refuses to start if you set the command without AuthorizedPrincipalsCommandUser. That is a deliberate safety check, not a suggestion.

Host certificates: stop trusting fingerprints one by one

User certificates solve login sprawl. Host certificates solve the other half: clients no longer pin one host key per machine.

On the host, sign its existing public host key with the host CA:

sudo ssh-keygen -s /root/ca/host_ca \ -h \ -I "web01-2026-11" \ -n web01.example.com,10.0.0.21 \ -V +52w \ /etc/ssh/ssh_host_ed25519_key.pub

That writes /etc/ssh/ssh_host_ed25519_key-cert.pub. Add it to sshd_config:

HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub

The certificate's public key must match a private host key already configured with HostKey. The private host key is still required; the certificate only proves that the host key was signed by your CA.

On clients, add the host CA to the known-hosts file:

@cert-authority *.example.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA...

Put that line in /etc/ssh/ssh_known_hosts for a fleet-wide default, or in ~/.ssh/known_hosts for one user. The hostnames pattern is matched against the name the client used, so include every name and address people actually type in the certificate's -n list. If clients reach the host as web01, web01.example.com, and 10.0.0.21, sign all three.

This is what removes the rebuild warning. A replacement host presents a new host key with a valid certificate from the same CA, and the client accepts it without a manual fingerprint update. It also means the CA can issue a host certificate for a name the host does not own — the DNS and network still have to be right, but the cryptographic identity is centralized.

Revocation that actually works

Short validity is the primary control. A twelve-hour certificate cannot be used next month, even if nobody remembers to revoke it. Revocation is the emergency brake for a key that must die before it expires.

Generate a Key Revocation List (KRL) from a public key:

ssh-keygen -k -f revoked.krl alice.pub ssh-keygen -k -u -f revoked.krl bob.pub ssh-keygen -Q -f revoked.krl alice.pub

ssh-keygen -Q exits with a non-zero status when the key is revoked, which makes it usable in a deployment check. The KRL is a compact binary format; revoking by serial number can take as little as one bit per certificate.

You can also revoke by serial or key ID without keeping the original certificate:

printf 'serial:1001\n' > revoke.spec ssh-keygen -k -f revoked.krl -s user_ca.pub revoke.spec printf 'id:alice@laptop-2026-11\n' > revoke-id.spec ssh-keygen -k -u -f revoked.krl -s user_ca.pub revoke-id.spec

Serial zero is reserved and cannot be revoked by serial number, so give every certificate a non-zero -z value if you plan to use serial-based revocation. To revoke by serial or key ID, the CA key must be supplied with -s, as shown above.

Enable it on the server:

RevokedKeys /etc/ssh/revoked_keys

Then distribute the KRL to that path. One operational warning from the manual: if the revocation file is unreadable, public key authentication is refused for all users. A missing or mispermissioned KRL is an outage, not a silent no-op. Check it during deployment.

Host keys can be revoked client-side with an @revoked line in known_hosts, but a KRL on the server is the usual tool for user certificates.

Revocation does not terminate sessions that are already established. It applies the next time the key is presented.

Automation: keep the CA key off the servers

The CA private key is the crown jewel. Anyone who has it can mint a certificate for any principal, so treat it like a root signing key: offline, passphrase-protected, or in an HSM. Do not copy it to every host just because the public key is safe to distribute.

For a signing service, load the CA key into an agent and sign with the public half:

ssh-add ~/.ssh/ca/user_ca ssh-keygen -Us ~/.ssh/ca/user_ca.pub \ -I "$USER@$(hostname)-$(date -u +%Y%m%dT%H%MZ)" \ -n "$USER" \ -V +8h \ -O clear -O permit-pty \ ~/.ssh/id_ed25519.pub

The -U flag tells ssh-keygen that the CA key is held by ssh-agent; the public key identifies which key to use. For a PKCS#11 token or HSM, -D points at the provider library and -s identifies the CA public key. Both keep the private key out of the signing host's filesystem, though an agent or token that will sign anything is still a high-value target.

A useful CI pattern is a certificate that can do exactly one thing:

ssh-keygen -s user_ca \ -I "deploy-$CI_JOB_ID" \ -n deploy \ -V +15m \ -O clear \ -O force-command='/usr/local/bin/deploy-only' \ deploy.pub

The force-command option is signed into the certificate, so the client cannot override it. Combine it with a dedicated account and short validity, and a leaked CI key has a narrow, auditable blast radius.

Failure modes worth memorizing

  • Permission denied (publickey) and ssh-keygen -L shows Principals: (none) — you signed without -n. The CA is trusted, but sshd will not accept a principal-less certificate.
  • Certificate expired or not yet valid — check the Valid: line and the client/server clocks. A certificate with a future start time fails immediately.
  • Principal mismatch — the certificate principal must equal the target username, or appear in AuthorizedPrincipalsFile. A role principal only works if the file lists it.
  • CA not trusted — verify the path in TrustedUserCAKeys, its permissions, and that you reloaded sshd. The file may contain multiple CA keys, one per line.
  • Certificate not offered — name it <identity>-cert.pub beside the private key, or set CertificateFile. A certificate sitting in the same directory with a different name is not automatically used.
  • Host certificate mismatch — the client name is not in the certificate's principal list. Add every alias and IP clients use.
  • All public key auth fails after enabling RevokedKeys — the KRL path is unreadable. Fix permissions or remove the directive until the file is in place.
  • Root still cannot log in — certificates do not bypass PermitRootLogin, AllowUsers, PAM, account expiry, or shell policy. They replace key distribution, not authorization policy.

What certificates do not solve

A CA does not remove the need to protect accounts. It does not grant sudo, bypass PAM, or make a compromised laptop safe. A stolen private key plus a valid certificate is a bearer credential until the certificate expires; that is why short validity, -O clear, and source-address restrictions matter.

There are no intermediate CAs in OpenSSH. Every server trusts the CA key directly, so a single CA compromise is a fleet compromise. If you need finer-grained delegation, run separate CAs for separate trust domains rather than trying to build an X.509-style hierarchy that OpenSSH does not implement.

Host certificates also do not fix DNS or routing. They prove that a host key was signed by your CA for a name. The client still has to reach the right host and use a name that appears in the certificate.

Migration and rollback

Do not remove authorized_keys on day one. Add TrustedUserCAKeys, issue certificates to a pilot group, and watch the server logs for accepted certificate IDs. Keep the old keys in place until every login path — interactive, automation, backups, monitoring — has a certificate.

Rollback is equally simple: remove or comment out TrustedUserCAKeys and HostCertificate, run sshd -t, and reload. Existing authorized_keys entries still work if you left them alone. Certificates are additive; they do not mutate the user's key file.

The payoff is not just fewer files. It is a single place to answer “who can log in, as whom, from where, and until when” — with a signature that every host can verify.

Sources

Commands and certificate output in this article were verified with OpenSSH 10.0p2 (Debian 13). The revocation examples were tested with a locally generated CA and a throwaway key.