“A man with a watch knows what time it is. A man with two watches is never sure.” – Segal’s Law
Introduction
Keeping accurate time is critical for a number of things at an ISP. Accurate time is needed for debugging to see what devices were affected at what time. If you had one reference clock that everything synced to, it really wouldn’t matter if it was 5mS off or 5 minutes off. So long as everything was based on that clock. Where it matters if you are trying to match your logs and data with devices that are not synced to your clock but to some other clock. So long as the other set of devices have synced to the same coordinated set of clocks you are, then you may have a good chance of lining up your logs to these other devices.
In tracking an event you are looking for timestamps that are “newer” or “older”. The need for accurate time may be in the milli or microseconds to see fast-moving events. In order to have a device to determine what time it is, it will normally use a protocol like the Network Time Protocol (NTP). NTP will send a packet out to another and trusted the NTP server to see what time it is. As there is latency on the Internet and in the protocol, NTP will also try to calculate the latency and adjust for it.
Trusted NTP servers will be classified on how they get their time such as based on an internal, highly accurate clock (stratum 0) or based on another NTP server (stratum 1, 2, etc.). For instance, a stratum 1 server will get its time from a stratum 0 server. A stratum 2 server will get its time from a stratum 1 server and so on.
One does everything they can to minimize the workload on these servers so that when a server is queried, it can respond as quickly and reliably as possible. So you may not have all of your devices connect to your only reference clock. You would distribute a second tier of NTP servers that would sync to your reference clock that your devices get their time from. As we are building a stratum 1 clock, you would deploy a set of NTP servers that would be stratum 2 servers for the rest of your devices. The stratum 2 servers would also sync with other NTP servers on the net to confirm that the reference clock is accurate.
The NTP Server
This server uses the Adafruit Global Positioning System (GPS) receiver hat sitting on a Raspberry Pi 3 to get its time from GPS. GPS has to have a highly accurate time to have accurate positioning reporting.
Setup
As mentioned above, our server will get its time from a stratum 0 server. In this case, a GPS clock is considered a stratum 0 server as it is a highly stable “coordinated clock”. Coordinated means that the time it calculates is coordinated with other clocks, such as the rest of the GPS satellite clocks as well as established clocks on the ground. Drift between these clocks is noted and adjusted as needed in order to keep all coordinated clocks in sync.
GPS receivers indicated time a couple of ways. Every second, this string is sent to the Pi. The actual timestamp of what time it is is sent to the Raspberry Pi via a 9600 baud serial port. As the string can vary in length as well as various other delays such as the computer servicing serial port interrupts, this is not an accurate method of finding out what time it is. It can actually vary several hundred mS. An additional method needs to be conveyed to the Raspberry Pi indicating the actual moment when the second occurs. This is typically a General Purpose I/O (GPIO) pin going high for a 100 mS and then going low. This GPIO pin is exposed to the computer as the Packet Per Second (PPS) interface such as /dev/pps0. The moment the GPIO pin goes high is the exact moment the second occurs. This is called the rising edge. So, timing software on the computer will get the time from the serial port and know the exact moment from the GPIO pin going high.
On a computer, you will have one process looking at the data from the GPS and one process handing the NTP protocol. The second process will get the timestamp and could be configured to assume that the first process is looking at the /dev/pps0 interface or it could use the /dev/pps0 interface. In the current configuration, the NTP server (CHRONYd) is getting the GPS timestamp from the GPSd process via shared memory and the NTP server is looking at the /dev/pps0 interface for the exact moment that the second occurs.
The GPSd daemon simply listens to the serial port and reports this via shared memory to the NTP server. We are using a compiled (source) GPSd as the packaged version was several revisions old for the Raspberry Pi distro. We are using an NTP server that is also a recent release and locally compiled called Chrony. In this case, Chrony is listening to the PPS interface to understand when the actual second starts.
Debugging
You can check the status of the NTP server by running the Chrony command-line interface called “chronyc”. A couple of commands are worth note are “sources” and “tracking”. The “sources” command will list the sources it is using for time. In the line below it shows PPS0 which is the Pulse Per Second GPIO pin. It is within the error rate tolerable of +/- 8mS and is around 200nS. GPS0 is the GPS NEMA strings via shared memory from GPSd to CHRONYd. Normally this would be off around 500mS or so we compensate for that by putting an offset in the chrony config of 0.500 seconds. PSM0 and PST0 are GPS NEMA strings and PPS signals via shared memory. Since the GPS is not handling PPS, these entries will be zero. The rest of the various NTP servers that chrony can sync against and have some confidence that the GPS data is sane.
chronyc> sources 210 Number of sources = 8 MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== #* PPS0 0 4 377 10 +227ns[ +516ns] +/- 8186ns #? GPS0 0 3 377 5 -12ms[ -12ms] +/- 100ms #? PSM0 0 3 0 - +0ns[ +0ns] +/- 0ns #? PST0 0 3 0 - +0ns[ +0ns] +/- 0ns ^- jane.qotw.net 2 4 377 10 +20us[ +20us] +/- 88ms ^- ntp18.kashra-server.com 2 4 177 10 +1485us[+1485us] +/- 95ms ^- time.iqnet.com 2 4 377 4 -1098us[-1097us] +/- 141ms ^- www.almaprovence.fr 3 4 377 5 -1196us[-1196us] +/- 115ms chronyc>
The Chrony tracking command gives you an update of stratum Chrony thinks it is. In this case Stratum 1, it has some confidence about slaving from the GPS receiver.
chronyc> tracking Reference ID : 50505330 (PPS0) Stratum : 1 Ref time (UTC) : Fri Aug 21 22:01:59 2020 System time : 0.000000248 seconds fast of NTP time Last offset : +0.000000319 seconds RMS offset : 0.000000776 seconds Frequency : 0.923 ppm slow Residual freq : +0.000 ppm Skew : 0.034 ppm Root delay : 0.000000001 seconds Root dispersion : 0.000025404 seconds Update interval : 16.0 seconds Leap status : Normal chronyc>
Configuration files
GPSd:
/etc/default/gpsd
# /etc/default/gpsd
## mod_install_stratum_one
# Default settings for the gpsd init script and the hotplug wrapper.
# Start the gpsd daemon automatically at boot time
START_DAEMON="true"
# Use USB hotplugging to add new USB devices automatically to the daemon
USBAUTO="true"
# Devices gpsd should collect to at boot time.
# They need to be read/writeable, either by user gpsd or the group dialout.
# DEVICES="/dev/ttyAMA0 /dev/pps0"
# in case you have two pps devices connected
# DEVICES="/dev/ttyAMA0 /dev/pps0 /dev/pps1"
# We tell GPSD to use the console serial port. Alas, it will try to open /etc/pps0.
# Run Chrony first to grab the port so GPSd doesn't get it.
DEVICES="/dev/ttyAMA0"
# Other options you want to pass to gpsd
GPSD_OPTIONS="-n -r -b"
GPSD_SOCKET="/var/run/gpsd.sock"
BAUDRATE="9600"
# Serial setup
# For serial interfaces, options such as low_latency are recommended
# Also, http://catb.org/gpsd/upstream-bugs.html#tiocmwait recommends
# setting the baudrate with stty
# Uncomment the following lines if using a serial device:
/bin/stty -F ${DEVICE} raw ${BAUDRATE} cs8 clocal -cstopb
# /bin/setserial ${DEVICE} low_latency
Chrony:
/etc/chrony/stratum1/10-refclocks-pps0.conf:
###################################################################### # PPS0 # PPS: /dev/pps0: Kernel-mode PPS ref-clock for the precise seconds # refclock PPS /dev/pps0 refid PPS0 precision 1e-7 poll 3 trust noselect lock PSM0 refclock PPS /dev/pps0 refid PPS0 trust # SHM(0), gpsd: NMEA data from shared memory provided by gpsd # refclock SHM 0 refid GPS0 precision 1e-1 poll 3 trust noselect offset 0.0 refclock SHM 0 refid GPS0 precision 1e-1 poll 3 trust noselect offset 0.500 # SHM(1), gpsd: PPS0 data from shared memory provided by gpsd refclock SHM 1 refid PSM0 precision 1e-7 poll 3 trust prefer # SOCK, gpsd: PPS0 data from socket provided by gpsd refclock SOCK /var/run/chrony.pps0.sock refid PST0 precision 1e-7 poll 3 trust noselect
/etc/chrony/stratum1/20-ntp-servers.conf:
# NTP Servers pool 2.debian.pool.ntp.org iburst minpoll 4 maxpoll 4
# for the GPS... # Start Chrony first to grab /etc/pps0 before GPSd does... /usr/local/sbin/chronyd sleep 2 stty -F /dev/serial0 raw 9600 cs8 clocal -cstopb /usr/local/sbin/gpsd -n -b /dev/ttyAMA0 -F /var/run/gpsd.sock
References
-
[ntp:questions] How to get shared memory access from gpsd to ntpd to work?
-
Description of NMEA data strings.
