A host that is three days behind real time does not report "the clock is wrong". It reports:
SSL certificate problem: certificate is not yet valid401 token is not yet valid (nbf)- two-factor codes rejected, twice, then a lockout
-
Release file ... is not valid yetfromapt update - log lines from three services that cannot be correlated into one incident
These are all the same bug. Time is the one dependency every service shares and almost nobody graphs. NTP usually works, right up until a VM is restored from a snapshot, an RTC battery dies, a firewall blocks UDP 123, or someone "fixes" the date by hand.
This is about the mechanics underneath: why a clock has to be both correct and continuous, what the default client actually does, when to replace it with chrony, and how to prove the result without trusting the daemon that produced it.
Two clocks, and only one of them is "the time"
Linux exposes several clocks. Two matter for this discussion:
-
CLOCK_REALTIMEis the wall clock: seconds since the Unix epoch. TLS certificate validation, JWTexpandnbf, TOTP, file timestamps, and log lines all read it. It can be set, and it can jump. -
CLOCK_MONOTONICcounts from boot and is not affected by setting the wall clock. Timeouts,sleep(), and most kernel timers use it. - The RTC is the battery-backed chip on the motherboard. The kernel reads it at boot and, depending on configuration, writes the system time back to it.
The time zone is presentation only. Etc/UTC versus Asia/Dubai changes how a timestamp is rendered, not what the epoch value is. That distinction matters because changing the time zone never fixes a clock that is actually wrong, and setting the clock never fixes a time zone that is actually wrong.
The two-part requirement is the source of most confusion: the clock must be accurate (close to a reference) and continuous (never jump backwards under running software). NTP has to trade one against the other, and it does that with two different correction methods.
Stepping and slewing are different tools
A time daemon can correct an offset in two ways:
- Step: call
clock_settime()and jump. Immediate and discontinuous. Required at boot when the clock is off by hours, and dangerous while databases, build tools, or replication are running. - Slew: run the local oscillator slightly fast or slow until the offset reaches zero. Continuous and order-preserving, but slow.
How slow is slow? Chrony's default maxslewrate is 83333.333 ppm, which is 8.333% of real time (chrony.conf documentation). At that cap, correcting one second takes about 12 seconds. Correcting one hour takes about 12 hours. Linux's kernel driver allows up to 100000 ppm, but that is still only 10%.
That arithmetic is why "let it settle on its own" is not a plan for a host that booted with a dead RTC and a clock a day behind. It is also why the sane policy is to step early, during boot, and slew afterward.
date -s is a raw step. It needs CAP_SYS_TIME, it provides no discipline, and in a container it fails with Operation not permitted because the clock belongs to the host kernel, not the container.
What a healthy host looks like
Here is a Debian 13 host running systemd 257.13:
$ timedatectl Local time: Wed 2026-11-18 05:02:06 UTC Universal time: Wed 2026-11-18 05:02:06 UTC RTC time: Wed 2026-11-18 05:02:06 Time zone: Etc/UTC (UTC, +0000) System clock synchronized: yes NTP service: active RTC in local TZ: no $ timedatectl timesync-status Server: 78.40.197.59 (2.debian.pool.ntp.org) Poll interval: 34min 8s (min: 32s; max 34min 8s) Leap: normal Version: 4 Stratum: 2 Reference: C14FED0E Precision: 1us (-24) Root distance: 29.296ms (max: 5s) Offset: -21.186ms Delay: 134.379ms Jitter: 19.007ms Packet count: 119 Frequency: +8.207ppm A few things are worth reading carefully:
-
RTC in local TZ: nois the correct answer. An RTC in local time is a dual-boot convenience that breaks around daylight saving changes;timedatectl set-local-rtc 1is documented as "not fully supported" for exactly this reason. - A poll interval sitting at the 2048-second maximum is normal. The client backs off when the clock is stable.
-
Offset: -21.186msmeans the local clock is 21 ms behind the server.Root distance: 29.296ms (max: 5s)is the estimated uncertainty back to the reference clock, and it is well inside the 5-second ceiling. -
Frequency: +8.207ppmis the correction being applied to the local oscillator. Uncorrected, 8.2 ppm is about 0.7 seconds per day, or roughly four minutes per year.
For monitoring, the machine-readable form is more useful than the table:
$ timedatectl show -p NTPSynchronized -p TimeUSec -p RTCTimeUSec NTPSynchronized=yes TimeUSec=Wed 2026-11-18 05:02:14 UTC RTCTimeUSec=Wed 2026-11-18 05:02:14 UTC But note what NTPSynchronized=yes actually claims: the kernel considers the clock synchronized. It does not tell you the offset. If accuracy is a requirement, alert on the offset and root distance, not on the boolean alone.
systemd-timesyncd is a client, not an NTP implementation
The systemd man page is unusually direct: systemd-timesyncd.service "implements SNTP only" and will "step the system clock for large offsets or slowly adjust it for smaller deltas" (systemd-timesyncd.service). It is a client that sets this host's clock. It does not serve time to a network, and it does not run the full NTP selection algorithm.
Its defaults are modest and documented in timesyncd.conf(5):
-
NTP=is empty by default;FallbackNTP=carries a compiled-in list. -
RootDistanceMaxSec=5— switch servers if the estimated distance to the reference clock exceeds five seconds. -
PollIntervalMinSec=32,PollIntervalMaxSec=2048. -
ConnectionRetrySec=30,SaveIntervalSec=60.
The important behavioral detail is that the configured servers are contacted in turn, until one responds. That is failover, not a pool and not a quorum. There is no majority vote and no falseticker detection: the first server that answers and passes the root-distance test becomes the source. For a single VM this is usually fine. For a host that mints tokens, validates TOTP, or signs requests, it is a single point of trust.
Site configuration belongs in a drop-in, not in the vendor file:
# /etc/systemd/timesyncd.conf.d/10-site.conf [Time] NTP=ntp1.example.net ntp2.example.net FallbackNTP=2.debian.pool.ntp.org RootDistanceMaxSec=3 PollIntervalMinSec=32 PollIntervalMaxSec=1024 sudo systemctl restart systemd-timesyncd timedatectl timesync-status Listing two servers under NTP= does not make them peers. It makes the second one the next candidate when the first stops answering.
The boot-ordering trap: time-set.target is not time-sync.target
This is the part that bites people who did everything else right.
systemctl cat systemd-timesyncd.service shows Before=time-set.target and Wants=time-set.target — not time-sync.target. The two targets mean different things (systemd.special(7)):
-
time-set.targetis about restoring an approximately monotonic clock from a local source, such as the timestamp saved in/var/lib/systemd/timesync/clock. It does not require the network. -
time-sync.targetis about completed synchronization to a remote reference. It can delay boot, because the network may not cooperate.
systemd-time-wait-sync.service is the unit that actually waits for the kernel to report synchronization, and orders itself Before=time-sync.target. On the Debian 13 host above, systemctl is-enabled systemd-time-wait-sync.service returns disabled.
That means After=time-sync.target alone is not a guarantee of accurate time. It is a guarantee that the target was reached — possibly immediately, with no NTP packet exchanged.
If a service genuinely needs a correct wall clock, order it and pull the target in:
# /etc/systemd/system/token-api.service.d/10-time.conf [Unit] Wants=time-sync.target After=time-sync.target sudo systemctl enable --now systemd-time-wait-sync.service Two caveats come straight from the unit file. It has TimeoutStartSec=infinity, so a host with no working NTP can sit there waiting. And it carries ConditionVirtualization=!container, because a container cannot set the host clock anyway. If a bounded boot matters more than the accuracy guarantee, cap it:
# /etc/systemd/system/systemd-time-wait-sync.service.d/10-timeout.conf [Service] TimeoutStartSec=90s A timeout makes the unit fail, but ordering dependencies are still released, so boot proceeds without the guarantee. That is the trade, and it should be a deliberate one.
When the default is not enough: chrony
Chrony is the right tool when you need any of the following:
- multiple sources with selection and falseticker detection
- authenticated time (NTS)
- a server that provides time to other machines
- hardware timestamping
- long-term drift modeling rather than a single correction loop
Installing it is straightforward, but stop the other client first. Two daemons disciplining one clock will fight, and the offset will oscillate:
sudo apt install chrony sudo systemctl disable --now systemd-timesyncd sudo systemctl enable --now chrony A minimal Debian configuration:
# /etc/chrony/chrony.conf (Debian/Ubuntu; /etc/chrony.conf on Fedora/RHEL) pool 2.debian.pool.ntp.org iburst driftfile /var/lib/chrony/chrony.drift makestep 1.0 3 rtcsync minsources 2 What each line does:
-
iburstsends a burst of 4-8 requests at startup so the first correction arrives in seconds rather than minutes. -
driftfileremembers the oscillator's frequency error across restarts. -
makestep 1.0 3steps the clock if the offset exceeds one second, but only for the first three updates. After that, corrections are slewed. This is the boot-time step, expressed as policy. -
rtcsynccopies the system time to the RTC every 11 minutes on Linux. The kernel does the copy; chronyd does not track RTC drift in this mode. -
minsources 2requires two agreeing sources before the clock is updated. That improves reliability, at the cost of not syncing on a network where only one source is reachable.
After a restart, read the state:
sudo systemctl restart chrony chronyc tracking chronyc sources -v chronyc tracking reports the offset between chrony's internal NTP clock and the system clock, the current frequency correction in ppm, root delay and dispersion, the update interval, and leap status. A large System time value that is shrinking means a slew is in progress. Leap status: Not synchronised means exactly that, regardless of what the service state says.
chronyc sources -v is the one to learn. The M column is the source mode: ^ server, = peer, # local reference clock. The S column is the selection state: * is the best source, + is another selected source combined with it, - is selectable but unused, x is a falseticker, ~ has too much variability, and ? is unusable. Reach is an octal bitmask of the last eight polls, so 377 means all eight were answered. Poll is a base-2 logarithm: 6 means every 64 seconds.
If you need to jump rather than wait:
sudo chronyc makestep That cancels the remaining slew and moves the clock immediately. Do it in a maintenance window, not under a running database, because a backwards jump is exactly the discontinuity the slew was avoiding.
Authenticated time with NTS
A plain NTP exchange has no cryptographic authentication by default. Anything that can deliver a believable reply can move your clock, and a moved clock can invalidate certificates, tokens, and audit logs.
NTS (RFC 8915) fixes the authentication problem in two phases: a TLS key exchange (NTS-KE, TCP port 4460 by default), followed by NTP packets authenticated with the resulting keys. Chrony supports it, and Cloudflare's time.cloudflare.com is an NTS-capable service (Cloudflare NTS documentation).
server time.cloudflare.com iburst nts ntsdumpdir /var/lib/chrony ntsdumpdir saves the NTS cookies so a restart does not repeat the key exchange. The default key refresh interval is four weeks. Verify that authentication is actually in use:
chronyc authdata The Mode column should read NTS. If it reads -, you are still getting unauthenticated time, and the nts option is not doing what you think.
One honest limitation: NTS authenticates the server and the packets. It does not make the server's clock correct. A signed wrong time is still wrong, which is why the next section exists.
Verify without trusting the daemon
An independent cross-check is easy: compare the local clock to an HTTPS Date header. It is second-granular and includes network delay, so a second or two of difference is normal. Minutes or hours are not.
remote=$(curl -sI https://example.com | sed -n 's/^[Dd]ate: //p' | tr -d '\r') local_epoch=$(date -u +%s) remote_epoch=$(date -u -d "$remote" +%s) printf 'offset: %+d s\n' "$((local_epoch - remote_epoch))" This is not a substitute for a proper monitoring check, but it catches the failure mode that matters: a host that believes it is synchronized while being wrong by an embarrassing amount.
For real monitoring, export the offset and root distance rather than the synchronization flag. With systemd-timesyncd, parse timedatectl show-timesync or timedatectl timesync-status. With chrony, parse chronyc tracking. Alert on absolute offset and on root distance, and page on Leap status leaving Normal.
The traps, in one place
- RTC in local time.
timedatectl set-local-rtc 1is documented as not fully supported and creates daylight-saving bugs. Set it to0. - Two clients. Check
systemctl is-active systemd-timesyncd chrony. Keep exactly one owner of the clock. - Containers.
systemd-timesyncdis conditioned withConditionVirtualization=!container, andCAP_SYS_TIMEis normally absent. The clock belongs to the host; fix it there. - VMs. A hypervisor time-sync integration and a guest NTP client can both adjust the clock. Pick one, or they will argue.
- Large offsets. At the default maximum slew rate, an hour of error takes about 12 hours to remove. Step at boot, not at noon.
-
timedatectl set-ntp true. It enables the first known synchronization service it finds, not necessarily the one you intended. Know which unit is active. - Logs. Journal timestamps come from the same clock. A wrong clock makes incident forensics useless, and it makes the "when did this start" question unanswerable.
A short checklist
-
timedatectlshows NTP active andRTC in local TZ: no. - Exactly one client is running:
systemctl is-active systemd-timesyncd chrony. - The offset is visible:
timedatectl timesync-statusorchronyc tracking. - Sources look healthy: at least one
*, andReachat377inchronyc sources -v. - An independent check agrees, not just the daemon's own opinion.
- If accuracy is a hard requirement, enable
systemd-time-wait-sync.serviceand order those services aftertime-sync.target. - If the network is untrusted, use NTS and confirm it with
chronyc authdata.
A clock is infrastructure that does not announce itself. It is also one of the few problems you can fully diagnose in under a minute, provided you look at the offset instead of the status line.
References
- systemd-timesyncd.service(8) — SNTP-only behavior, stepping versus slewing, the saved clock file.
- timesyncd.conf(5) — NTP, FallbackNTP, RootDistanceMaxSec, poll intervals, retry and save intervals.
- timedatectl(1) — status, timesync-status, show-timesync, set-ntp, set-local-rtc.
- systemd.special(7) —
time-set.targetversustime-sync.target. - chrony.conf(5) —
makestep,rtcsync,maxslewrate,minsources, NTS options. - chronyc(1) —
tracking,sources,authdata,makestep,waitsync. - RFC 5905 — Network Time Protocol Version 4 — port 123, protocol operation.
- RFC 8915 — Network Time Security — NTS-KE and authenticated NTP.
- RFC 6238 — TOTP — 30-second default time step, one-step validation window.
- RFC 7519 — JWT —
expandnbfsemantics. - Cloudflare Time Services: NTS — NTS phases and
time.cloudflare.com.

