A staffer at the USA’s National Institute of Standards and Technology (NIST) tried to disable backup generators powering some of its Network Time Protocol infrastructure, after a power outage around Boulder, Colorado, led to errors.

As explained in a mailing list post by Jeffrey Sherman, a NIST supervisory physicist who maintains the institute’s atomic clocks, “The atomic ensemble time scale at our Boulder campus has failed due to a prolonged utility power outage.”

Sherman, whose LinkedIn bio proclaims he is “One of the few federal employee actually paid to watch the clocks all day,” says one impact of the incident “is that the Boulder Internet Time Services no longer have an accurate time reference.”

That’s bad because one of the things NIST uses its atomic clocks for is to provide a Network Time Protocol service, the authoritative source of timing information that the computing world relies on so that diverse systems can synchronize events. If NTP isn’t working, outcomes can include difficulties authenticating between systems, meaning applications can become unstable.

At this point, readers might wonder why NIST can’t just turn off the inaccurate service. Sherman said a backup generator kicked in and kept the servers running.

“I will attempt to disable them [the generators] to avoid disseminating incorrect time,” he wrote.

TL;DR: If you want to operate a secure environment you should use your own on-site stratum 1 NTP servers along with authentication. This is the only way to eliminate time spoofing attacks from the outside. Don’t reduce your overall security to a stateless and unauthenticated (read: easy-to-spoof) network protocol!

If you are using a couple of different NTP sources it might be not that easy for an attacker to spoof your time – though not unfeasible at all. And think about small routers with VPN endpoints and DNSSEC resolving enabled, or IoT devices such as cameras or door openers – they don’t even have a real-time clock with a battery inside. They fully rely on NTP.

This is what this blogpost series is all about. Let’s dig into it. ;)

Ping & traceroute
If you suspect a network problem between the monitoring system and your server, it’s helpful to have a traceroute from your NTP server to the monitoring system. You can traceroute to the monitoring network using 139.178.64.42 and 2604:1380:2:6000::15.

You can ping or traceroute from the monitoring network using HTTP, with:

curl http://trace.ntppool.org/traceroute/8.8.8.8
curl http://trace.ntppool.org/ping/8.8.8.8

Enter the domain name or IP address of the NTP server you want to measure.

The NTP Pool DNS Mapper tries mapping user IP addresses to their DNS servers (and vice-versa). It also tracks which servers support EDNS-SUBNET and how well that actually matches the client IP (as seen by the HTTP server). You can see a demo at mist.ntppool.org.

It's done to research how to improve the DNS system used by the NTP Pool and potentially other similar applications.

How can I help?
Thank you for asking! The easiest way to help is to help get more data.

If you have a website you can have your users help by adding one of the following two code snippets to your site.

Museum boffins find code that crashes in 2037

A stark warning about the upcoming Epochalypse, also known as the "Year 2038 problem," has come from the past, as National Museum Of Computing system restorers have discovered an unsetting issue while working on ancient systems.

Robin Downs, a volunteer who was recently involved in an exhibition of Digital Equipment Corporation (DEC) gear at the museum, was on hand to demonstrate the problem to The Register in the museum's Large Systems Gallery, which now houses a running PDP-11/73. //

"So we found bugs that exist, pre-2038, in writing this that we didn't know about."

The Year 2038 problem occurs in systems that store Unix time – the number of seconds since the Unix epoch (00:00:00 UTC on January 1, 1970) in a signed 32-bit integer (64-bit is one modern approach, but legacy systems have a habit of lingering).

At 03:14:07 UTC on January 19, 2038, the second counter will overflow. In theory, this will result in a time and date being returned before the epoch – 20:45:52 UTC on December 13, 1901, but that didn't happen for Downs. //

zb42
As the article is specifically about date problems that occur before the 32bit unix time rollover, I think it should be mentioned that:

32bit NTP is going to roll over on the 7th of February 2036.

Maybe https://github.com/mdavids/ntptools/blob/main/go/monitor2.go

% ./monitor2 -server any.time.nl -stratum 2

    Server: any.time.nl
    Result: OK - test successful

Any exit code other than 0 indicates an error.

mengzhuo
2d
I’ve made a tracentp tool that can works for you

./tracentp -c 1 time.nist.gov
OK from time.nist.gov:seq=1 stratum=1 offset=10.87403ms distance=118.379889ms RTT=235.539075ms ref=.NIST.

https://github.com/mengzhuo/tracentp/releases/tag/v1.0.1

Release tracentp 1.0.1 · mengzhuo/tracentp
Changelog a35e0f6 fix releaser

Barely working servers are great for the test pool. More scenarios to test!

I’ve written a simple NTP simulator (probably not the only one out there) that lets you tweak things however you like:

Installation and setup commands (for systemd managed systems). If you are using a different platform, email us or post on the community development forum.

Step 1: Add the package repository

Step 2: Install the agent

Ubuntu, Debian:

 sudo apt install -y ntppool-agent

Enterprise Linux, Fedora, etc:

 sudo yum install -y ntppool-agent

Step 3: Enable and start the service

 sudo systemctl enable --now ntppool-agent@test

Step 4: Monitor the logs (in one terminal window)

 sudo journalctl -u ntppool-agent@\* -f

Step 5: Setup the agent (in another terminal window on the same server)

 sudo -u ntpmon ntppool-agent setup -e test -a 1d03ggn

Step 6: Open the provided link from the setup process to authenticate the agent to your account

NTP Pool Repositories
The NTP Pool monitor and other tools are built for Linux and FreeBSD (i386, amd64 and arm64).

Yum and deb package repositories are available, in a “test” and “production” flavor.

The NTP Pool Monitoring Agent package is ntppool-agent.

WallTime is a NTP synchronized studio clock and notification system that works with your standard computer monitor or television.  The clock is kept always in sync using Network Time Protocol (NTP).  It can take the place of stand alone clock systems, notification systems, on-air lights and more.

Complete system ready to connect to your monitor. No PC is required.

$450
Includes WallTime device, power supply, HDMI cable and velcro for attaching to monitor.

This tool is useful to check if a given Network Time Protocol server is reachable over the internet using IPv4 and IPv6 connectivity.

It is also useful for knowing the resulting offset using the exact time of x1.ncomputers.org stratum 2 NTP public server.

The Windows Time service (W32Time) synchronizes the date and time for all computers managed by Active Directory Domain Services (AD DS). This article covers the different tools and settings used to manage the Windows Time service.

Apple runs a fleet of stratum 1 NTP servers at time.apple.com. In my experience, ntpd/chronyd are very happy with them.

It looks like, instead of doing anycast, they maybe use DNS to steer you to the closest one.

time.apple.com is a CNAME for time-osx.g.aaplimg.com.

It's worth noting that this list changes over time, and you're generally probably best off using pool time.apple.com rather than trying to rely on this maybe-outdated list.

The big internet corporations (like Google, Meta, Amazon) have independently developed their own smearing behavior, some even iterated over multiple smearing patterns. If you search for <corporation> leap smear with an internet search engine of your choice, you will find blog posts and documentation where they explain their respective approach. I suggest you research this for any upstream server you intend to use. A quick summary of the ones I came up with spontaneously:

• Google, Amazon: Smeared from noon to noon around the leap with 1/86400 of a second added to each second
• Meta: Smeared from the leap until 17 hours later in a non-linear curve
• Cloudflare: Standard-compliant insertion/deletion of a leap second and propagation of the leap indicator (that’s why they are allowed in the pool)
• Microsoft: No idea what software their time servers are running, but the Windows Time Service included in Windows machines does not handle leap seconds and just resyncs after one happens

All of this is best practice information and much less relevant since the decision was made to do away with leap seconds in the foreseeable future. //

Apple also offers good collection of stratum 1 servers that doesn’t use leap second smearing. Someone compiled a (non-definitive) list of them so you can see if they are worth using for your location:

pool time.apple.com

As for analyzing the traffic, here’s a useful oneliner that works on Rocky Linux 9. Other operating systems / versions may vary. Adjust the “10000” packet count limit as you see fit.

tcpdump -n -c 10000 inbound and ip and udp and dst port 123 | cut -d" " -f3 | cut -d. -f1-4 | sort | uniq -c | sort -rn | head

Our beloved NTP protocol appears to work in a deep space environment (as tested in a simulation with a 4 hour RTT):

https://deepspaceip.github.io/testbed/ntp.pdf

you could use IPv6 for the multicast addresses replacing 224.x.y.z with ff02::xyz or ff02::x:y:z etc.

Rather, the IPv6 multicast address for NTP is ff02::101. Likewise, the multicast IPv4 address for NTP is 224.0.1.1.

PTP daemon (PTPd) is an implementation the Precision Time Protocol (PTP) version 2 as defined by 'IEEE Std 1588-2008'. PTP provides precise time coordination of Ethernet LAN connected computers. It was designed primarily for instrumentation and control systems.

nexpensive, compact GPS unit for highly accurate serial PPS time.

Reviewed in the United States on July 2, 2012

I have the Garmin 18x LVC GPS attached to a serial card installed in a Debian Linux 'Squeeze' box (which was configured using the Setserial, GPSD, and NTP packages).

USB based GPS units are useless for accurately setting the time.

The 18x LVC GPS unit requires 5 volts of power (red wire), and for this I use a specific serial card, the StarTech.com 1 Port PCI RS232 Powered Serial Adapter Card (PCI1S650PW). The card can be configured to supply either 5 or 12 volts (or none) on pin 9 of the RS-232 DB-9 connector, and can draw power either from the PCI bus itself or from the PC power supply directly via the onboard Molex floppy power connector. (I used the latter option). Once the jumpers had been set and the card installed no drivers were required for Debian, Setserial recognizing the card as having a 16550A UART with a baud base of 921600.

The GPS unit itself is mounted externally on the chimney, as the GPS satellite signals are fairly weak, so installing outside is the best option as the GPS itself is weatherproof. For me, this location gives the GPS a clear view of the sky, but the 5 meter (16 feet) cable would not be long enough to reach the computer inside the house, so I replaced almost all of the original serial cable (which is molded onto the 18x LVC). The wires (especially the three signal wires, which includes the PPS) are also VERY thin. So, I cut the original cable close to the GPS unit itself and extended it with my own cable which runs all the way back to the computer and is terminated with a serial DB-9 connector. If you do this, remember to use a shielded cable. There are basically five wires needed for the serial connection: Measurement pulse output (PPS) from the GPS is yellow -> pin 1 (DCD) on the DB-9, transmit is white -> pin 2 (RXD), receive is green -> pin 3 (TXD), ground is black -> pin 5 (GND), and power is red -> pin 9. For the replacement I used a shielded CAT 6 cable and joined each of the pairs together at both ends and used these to give me four of the connections: PPS, transmit, receive, and power (90 mA). I then connected all of the black wires together at the GPS end of the cable and used the drain wire as the ground, connecting it to pin 5 on the DB-9 at the other end. For mounting there is a metric (M3) threaded brass recess on the underside of the 18x LVC.

I also have the StarTech.com PEX1S553LP 1 Port Low Profile Native RS232 PCI Express Serial Card with 16550 UART card installed in a Windows 7 PC which is only used to run firmware updates, etc. on the GPS when needed (as these cannot be performed under Linux). My 18x LVC is updated to version 3.80 (26th March, 2012). For what it's worth, my time1 setting in ntp.conf is 0.035. This gives me the lowest offset and jitter for PPS.

Works like a charm, with extremely low latency PPS time (accurate to a few microseconds).