For Raspberry Pi clusters, ham radio
digital modes, logging infrastructure, or anything where precise
timestamping matters, it’s noise.
A Stratum 1 NTP server solves this permanently: GPS satellites carry atomic-clock-derived time. A GPS receiver with a pulse-per-second (PPS) signal gives you sub-microsecond accuracy. Your entire network syncs directly from that, with no upstream dependency, no internet requirement, no pool latency.
Here’s how to build one for under $20.
A staffer at the USA’s National Institute of Standards and Technology (NIST) tried to disable backup generators powering some of its Network Time Protocol infrastructure, after a power outage around Boulder, Colorado, led to errors.
As explained in a mailing list post by Jeffrey Sherman, a NIST supervisory physicist who maintains the institute’s atomic clocks, “The atomic ensemble time scale at our Boulder campus has failed due to a prolonged utility power outage.”
Sherman, whose LinkedIn bio proclaims he is “One of the few federal employee actually paid to watch the clocks all day,” says one impact of the incident “is that the Boulder Internet Time Services no longer have an accurate time reference.”
That’s bad because one of the things NIST uses its atomic clocks for is to provide a Network Time Protocol service, the authoritative source of timing information that the computing world relies on so that diverse systems can synchronize events. If NTP isn’t working, outcomes can include difficulties authenticating between systems, meaning applications can become unstable.
At this point, readers might wonder why NIST can’t just turn off the inaccurate service. Sherman said a backup generator kicked in and kept the servers running.
“I will attempt to disable them [the generators] to avoid disseminating incorrect time,” he wrote.
TL;DR: If you want to operate a secure environment you should use your own on-site stratum 1 NTP servers along with authentication. This is the only way to eliminate time spoofing attacks from the outside. Don’t reduce your overall security to a stateless and unauthenticated (read: easy-to-spoof) network protocol!
If you are using a couple of different NTP sources it might be not that easy for an attacker to spoof your time – though not unfeasible at all. And think about small routers with VPN endpoints and DNSSEC resolving enabled, or IoT devices such as cameras or door openers – they don’t even have a real-time clock with a battery inside. They fully rely on NTP.
This is what this blogpost series is all about. Let’s dig into it. ;)
Ping & traceroute
If you suspect a network problem between the monitoring system and your server, it’s helpful to have a traceroute from your NTP server to the monitoring system. You can traceroute to the monitoring network using 139.178.64.42 and 2604:1380:2:6000::15.
You can ping or traceroute from the monitoring network using HTTP, with:
curl http://trace.ntppool.org/traceroute/8.8.8.8
curl http://trace.ntppool.org/ping/8.8.8.8Enter the domain name or IP address of the NTP server you want to measure.
The NTP Pool DNS Mapper tries mapping user IP addresses to their DNS servers (and vice-versa). It also tracks which servers support EDNS-SUBNET and how well that actually matches the client IP (as seen by the HTTP server). You can see a demo at mist.ntppool.org.
It's done to research how to improve the DNS system used by the NTP Pool and potentially other similar applications.
How can I help?
Thank you for asking! The easiest way to help is to help get more data.
If you have a website you can have your users help by adding one of the following two code snippets to your site.
Museum boffins find code that crashes in 2037
A stark warning about the upcoming Epochalypse, also known as the "Year 2038 problem," has come from the past, as National Museum Of Computing system restorers have discovered an unsetting issue while working on ancient systems.
Robin Downs, a volunteer who was recently involved in an exhibition of Digital Equipment Corporation (DEC) gear at the museum, was on hand to demonstrate the problem to The Register in the museum's Large Systems Gallery, which now houses a running PDP-11/73. //
"So we found bugs that exist, pre-2038, in writing this that we didn't know about."
The Year 2038 problem occurs in systems that store Unix time – the number of seconds since the Unix epoch (00:00:00 UTC on January 1, 1970) in a signed 32-bit integer (64-bit is one modern approach, but legacy systems have a habit of lingering).
At 03:14:07 UTC on January 19, 2038, the second counter will overflow. In theory, this will result in a time and date being returned before the epoch – 20:45:52 UTC on December 13, 1901, but that didn't happen for Downs. //
zb42
As the article is specifically about date problems that occur before the 32bit unix time rollover, I think it should be mentioned that:
32bit NTP is going to roll over on the 7th of February 2036.
Maybe https://github.com/mdavids/ntptools/blob/main/go/monitor2.go
% ./monitor2 -server any.time.nl -stratum 2
Server: any.time.nl
Result: OK - test successfulAny exit code other than 0 indicates an error.
mengzhuo
2d
I’ve made a tracentp tool that can works for you
./tracentp -c 1 time.nist.gov
OK from time.nist.gov:seq=1 stratum=1 offset=10.87403ms distance=118.379889ms RTT=235.539075ms ref=.NIST.https://github.com/mengzhuo/tracentp/releases/tag/v1.0.1
Release tracentp 1.0.1 · mengzhuo/tracentp
Changelog a35e0f6 fix releaser
Barely working servers are great for the test pool. More scenarios to test!
I’ve written a simple NTP simulator (probably not the only one out there) that lets you tweak things however you like:
Installation and setup commands (for systemd managed systems). If you are using a different platform, email us or post on the community development forum.
Step 1: Add the package repository
Step 2: Install the agent
Ubuntu, Debian:
sudo apt install -y ntppool-agentEnterprise Linux, Fedora, etc:
sudo yum install -y ntppool-agentStep 3: Enable and start the service
sudo systemctl enable --now ntppool-agent@testStep 4: Monitor the logs (in one terminal window)
sudo journalctl -u ntppool-agent@\* -fStep 5: Setup the agent (in another terminal window on the same server)
sudo -u ntpmon ntppool-agent setup -e test -a 1d03ggnStep 6: Open the provided link from the setup process to authenticate the agent to your account
NTP Pool Repositories
The NTP Pool monitor and other tools are built for Linux and FreeBSD (i386, amd64 and arm64).
Yum and deb package repositories are available, in a “test” and “production” flavor.
The NTP Pool Monitoring Agent package is ntppool-agent.
WallTime is a NTP synchronized studio clock and notification system that works with your standard computer monitor or television. The clock is kept always in sync using Network Time Protocol (NTP). It can take the place of stand alone clock systems, notification systems, on-air lights and more.
Complete system ready to connect to your monitor. No PC is required.
$450
Includes WallTime device, power supply, HDMI cable and velcro for attaching to monitor.
This tool is useful to check if a given Network Time Protocol server is reachable over the internet using IPv4 and IPv6 connectivity.
It is also useful for knowing the resulting offset using the exact time of x1.ncomputers.org stratum 2 NTP public server.
The Windows Time service (W32Time) synchronizes the date and time for all computers managed by Active Directory Domain Services (AD DS). This article covers the different tools and settings used to manage the Windows Time service.
Apple runs a fleet of stratum 1 NTP servers at time.apple.com. In my experience, ntpd/chronyd are very happy with them.
It looks like, instead of doing anycast, they maybe use DNS to steer you to the closest one.
time.apple.com is a CNAME for time-osx.g.aaplimg.com.
It's worth noting that this list changes over time, and you're generally probably best off using pool time.apple.com rather than trying to rely on this maybe-outdated list.
Strange Behavior of 0.debian.pool.ntp.org (84.255.251.205:123) - Server operators - NTP Pool Project
The big internet corporations (like Google, Meta, Amazon) have independently developed their own smearing behavior, some even iterated over multiple smearing patterns. If you search for <corporation> leap smear with an internet search engine of your choice, you will find blog posts and documentation where they explain their respective approach. I suggest you research this for any upstream server you intend to use. A quick summary of the ones I came up with spontaneously:
- Google, Amazon: Smeared from noon to noon around the leap with 1/86400 of a second added to each second
- Meta: Smeared from the leap until 17 hours later in a non-linear curve
- Cloudflare: Standard-compliant insertion/deletion of a leap second and propagation of the leap indicator (that’s why they are allowed in the pool)
- Microsoft: No idea what software their time servers are running, but the Windows Time Service included in Windows machines does not handle leap seconds and just resyncs after one happens
All of this is best practice information and much less relevant since the decision was made to do away with leap seconds in the foreseeable future. //
Apple also offers good collection of stratum 1 servers that doesn’t use leap second smearing. Someone compiled a (non-definitive) list of them so you can see if they are worth using for your location:
pool time.apple.comAs for analyzing the traffic, here’s a useful oneliner that works on Rocky Linux 9. Other operating systems / versions may vary. Adjust the “10000” packet count limit as you see fit.
tcpdump -n -c 10000 inbound and ip and udp and dst port 123 | cut -d" " -f3 | cut -d. -f1-4 | sort | uniq -c | sort -rn | head
Our beloved NTP protocol appears to work in a deep space environment (as tested in a simulation with a 4 hour RTT):
you could use IPv6 for the multicast addresses replacing 224.x.y.z with ff02::xyz or ff02::x:y:z etc.
Rather, the IPv6 multicast address for NTP is ff02::101. Likewise, the multicast IPv4 address for NTP is 224.0.1.1.
PTP daemon (PTPd) is an implementation the Precision Time Protocol (PTP) version 2 as defined by 'IEEE Std 1588-2008'. PTP provides precise time coordination of Ethernet LAN connected computers. It was designed primarily for instrumentation and control systems.