A GPS Stratum 1 NTP Server

“A man with a watch knows what time it is. A man with two watches is never sure.” – Segal’s Law

Introduction

Keeping accurate time is critical for a number of things at an ISP. Accurate time is needed for debugging to see what devices were affected at what time.   If you had one reference clock that everything synced to, it really wouldn’t matter if it was 5mS off or 5 minutes off.  So long as everything was based on that clock.  Where it matters if you are trying to match your logs and data with devices that are not synced to your clock but to some other clock.  So long as the other set of devices have synced to the same coordinated set of clocks you are, then you may have a good chance of lining up your logs to these other devices.

In tracking an event you are looking for timestamps that are “newer” or “older”. The need for accurate time may be in the milli or microseconds to see fast-moving events. In order to have a device to determine what time it is, it will normally use a protocol like the Network Time Protocol (NTP).  NTP will send a packet out to another and trusted the NTP server to see what time it is. As there is latency on the Internet and in the protocol, NTP will also try to calculate the latency and adjust for it. 

Trusted NTP servers will be classified on how they get their time such as based on an internal, highly accurate clock (stratum 0) or based on another NTP server (stratum 1, 2, etc.). For instance, a stratum 1 server will get its time from a stratum 0 server. A stratum 2 server will get its time from a stratum 1 server and so on.

One does everything they can to minimize the workload on these servers so that when a server is queried, it can respond as quickly and reliably as possible.  So you may not have all of your devices connect to your only reference clock.  You would distribute a second tier of NTP servers that would sync to your reference clock that your devices get their time from.  As we are building a stratum 1 clock, you would deploy a set of NTP servers that would be stratum 2 servers for the rest of your devices.  The stratum 2 servers would also sync with other NTP servers on the net to confirm that the reference clock is accurate.

The NTP Server

This server uses the Adafruit Global Positioning System (GPS) receiver hat sitting on a Raspberry Pi 3 to get its time from GPS. GPS has to have a highly accurate time to have accurate positioning reporting. 

Setup

As mentioned above, our server will get its time from a stratum 0 server. In this case, a GPS clock is considered a stratum 0 server as it is a highly stable “coordinated clock”. Coordinated means that the time it calculates is coordinated with other clocks, such as the rest of the GPS satellite clocks as well as established clocks on the ground. Drift between these clocks is noted and adjusted as needed in order to keep all coordinated clocks in sync.

GPS receivers indicated time a couple of ways. Every second, this string is sent to the Pi. The actual timestamp of what time it is is sent to the Raspberry Pi via a 9600 baud serial port. As the string can vary in length as well as various other delays such as the computer servicing serial port interrupts, this is not an accurate method of finding out what time it is. It can actually vary several hundred mS. An additional method needs to be conveyed to the Raspberry Pi indicating the actual moment when the second occurs. This is typically a General Purpose I/O (GPIO) pin going high for a 100 mS and then going low. This GPIO pin is exposed to the computer as the Packet Per Second (PPS) interface such as /dev/pps0. The moment the GPIO pin goes high is the exact moment the second occurs. This is called the rising edge. So, timing software on the computer will get the time from the serial port and know the exact moment from the GPIO pin going high.

On a computer, you will have one process looking at the data from the GPS and one process handing the NTP protocol. The second process will get the timestamp and could be configured to assume that the first process is looking at the /dev/pps0 interface or it could use the /dev/pps0 interface. In the current configuration, the NTP server (CHRONYd) is getting the GPS timestamp from the GPSd process via shared memory and the NTP server is looking at the /dev/pps0 interface for the exact moment that the second occurs.

The GPSd daemon simply listens to the serial port and reports this via shared memory to the NTP server. We are using a compiled (source) GPSd as the packaged version was several revisions old for the Raspberry Pi distro. We are using an NTP server that is also a recent release and locally compiled called Chrony. In this case, Chrony is listening to the PPS interface to understand when the actual second starts.

Debugging

You can check the status of the NTP server by running the Chrony command-line interface called “chronyc”.   A couple of commands are worth note are “sources” and “tracking”.  The “sources” command will list the sources it is using for time.  In the line below it shows PPS0 which is the Pulse Per Second GPIO pin.  It is within the error rate tolerable of +/- 8mS and is around 200nS. GPS0 is the GPS NEMA strings via shared memory from GPSd to CHRONYd. Normally this would be off around 500mS or so we compensate for that by putting an offset in the chrony config of 0.500 seconds. PSM0 and PST0 are GPS NEMA strings and PPS signals via shared memory. Since the GPS is not handling PPS, these entries will be zero. The rest of the various NTP servers that chrony can sync against and have some confidence that the GPS data is sane.

chronyc> sources
210 Number of sources = 8
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
#* PPS0                          0   4   377    10   +227ns[ +516ns] +/- 8186ns
#? GPS0                          0   3   377     5    -12ms[  -12ms] +/-  100ms
#? PSM0                          0   3     0     -     +0ns[   +0ns] +/-    0ns
#? PST0                          0   3     0     -     +0ns[   +0ns] +/-    0ns
^- jane.qotw.net                 2   4   377    10    +20us[  +20us] +/-   88ms
^- ntp18.kashra-server.com       2   4   177    10  +1485us[+1485us] +/-   95ms
^- time.iqnet.com                2   4   377     4  -1098us[-1097us] +/-  141ms
^- www.almaprovence.fr           3   4   377     5  -1196us[-1196us] +/-  115ms
chronyc>

The Chrony tracking command gives you an update of stratum Chrony thinks it is. In this case Stratum 1, it has some confidence about slaving from the GPS receiver.

chronyc> tracking
Reference ID    : 50505330 (PPS0)
Stratum         : 1
Ref time (UTC)  : Fri Aug 21 22:01:59 2020
System time     : 0.000000248 seconds fast of NTP time
Last offset     : +0.000000319 seconds
RMS offset      : 0.000000776 seconds
Frequency       : 0.923 ppm slow
Residual freq   : +0.000 ppm
Skew            : 0.034 ppm
Root delay      : 0.000000001 seconds
Root dispersion : 0.000025404 seconds
Update interval : 16.0 seconds
Leap status     : Normal
chronyc>

Configuration files

GPSd:

/etc/default/gpsd
# /etc/default/gpsd
## mod_install_stratum_one
# Default settings for the gpsd init script and the hotplug wrapper.
# Start the gpsd daemon automatically at boot time
START_DAEMON="true"
# Use USB hotplugging to add new USB devices automatically to the daemon
USBAUTO="true"
# Devices gpsd should collect to at boot time.
# They need to be read/writeable, either by user gpsd or the group dialout.
# DEVICES="/dev/ttyAMA0 /dev/pps0"
# in case you have two pps devices connected
# DEVICES="/dev/ttyAMA0 /dev/pps0 /dev/pps1"
# We tell GPSD to use the console serial port.  Alas, it will try to open /etc/pps0.  
# Run Chrony first to grab the port so GPSd doesn't get it.
DEVICES="/dev/ttyAMA0"
# Other options you want to pass to gpsd
GPSD_OPTIONS="-n -r -b"
GPSD_SOCKET="/var/run/gpsd.sock"
BAUDRATE="9600"
# Serial setup
# For serial interfaces, options such as low_latency are recommended
# Also, http://catb.org/gpsd/upstream-bugs.html#tiocmwait recommends
#   setting the baudrate with stty
# Uncomment the following lines if using a serial device:
/bin/stty -F ${DEVICE} raw ${BAUDRATE} cs8 clocal -cstopb
# /bin/setserial ${DEVICE} low_latency

Chrony:

/etc/chrony/stratum1/10-refclocks-pps0.conf:
######################################################################
# PPS0
# PPS: /dev/pps0: Kernel-mode PPS ref-clock for the precise seconds
# refclock  PPS /dev/pps0                   refid PPS0  precision 1e-7  poll 3  trust  noselect  lock PSM0
refclock PPS /dev/pps0 refid PPS0 trust
# SHM(0), gpsd: NMEA data from shared memory provided by gpsd
# refclock  SHM 0                           refid GPS0  precision 1e-1  poll 3  trust  noselect  offset 0.0
refclock  SHM 0                           refid GPS0  precision 1e-1  poll 3  trust  noselect  offset 0.500
# SHM(1), gpsd: PPS0 data from shared memory provided by gpsd
refclock  SHM 1                           refid PSM0  precision 1e-7  poll 3  trust  prefer
# SOCK, gpsd: PPS0 data from socket provided by gpsd
refclock  SOCK /var/run/chrony.pps0.sock  refid PST0  precision 1e-7  poll 3  trust  noselect
/etc/chrony/stratum1/20-ntp-servers.conf:
# NTP Servers
pool  2.debian.pool.ntp.org  iburst  minpoll 4  maxpoll 4
rc.local:
# for the GPS...
# Start Chrony first to grab /etc/pps0 before GPSd does...
/usr/local/sbin/chronyd
sleep 2
stty -F /dev/serial0 raw 9600 cs8 clocal -cstopb
/usr/local/sbin/gpsd -n -b /dev/ttyAMA0 -F /var/run/gpsd.sock

References

Be the first to like this

Compliance Mode for Ubiquiti AirMax Radios

Sometime after “AirOS 5.3.3.9634.110726.1142” Ubiquiti took out what was called “Compliance Mode” that enabled frequencies outside of the Part 15 bands.  This was handy if you wanted to run these radios in Part 97 or Amateur or “Ham” Bands.   There was a simple way of doing this right after they took this out by just doing a …

 touch /etc/persistent/ct
 save
 reboot

Alas, this was taken out after a while.

After googling around and banging on this a bit, the “Country Code” for “Compliance Mode” that had an ID of 511 is now called “Licensed”.  You can find it in /var/etc/ccodes.inc.  The trick is to set the Atheros chipset on startup to the county code you want.   This would be done with a custom script that runs when the radio boots up.  You will need firmware that supports that that has “cs” in the file name.  Go to:

https://community.ui.com/questions/airOS-5-6-with-Custom-Script-Support-/4bd71c6b-7e9c-43ee-9d9d-666a7a9708fd#comment/e6d675ef-2e50-46f4-bf2a-7ef94dd12068

and find the firmware for your board (eg XM, XW or TI).  Upgrade the radio with this firmware.

After the upgrade and reboot, you can use this script that you should call “/etc/persistent/rc.poststart” with the following:

#!/bin/sh
# This is dependent on v6.3.0 of the firmware.  You may need 
# to point to the ubnt_spectral module someplace else if you
# have a different firmware version.
insmod /lib/modules/2.6.32.71/ubnt_spectral.ko
iwpriv wifi0 setCountry Ub
ifconfig ath0 down
ifconfig wifi0 down
sleep 5
rmmod /lib/modules/2.6.32.71/ubnt_spectral.ko
ifconfig ath0 up
ifconfig wifi0 up
echo "countrycode=511" > /var/etc/atheros.conf
# '840' is the Country Code for the US
sed -i 's/840/511/g' /tmp/system.cfg
echo "<option value="511">Compliance Test</option>" >> /var/etc/ccodes.inc

After creating this script, make sure you “chmod a+x /etc/persistent/rc.poststart” and the type “save” and “reboot”.

Once the radio has rebooted, you should be able to go to the “WIRELESS” tab and see that the Country Code is set for  Licensed.  If you select the “Frequency Scan List”, you should see a bunch of frequencies you can now select (see below).  Alas, the script does not survive the reboot, but that is fine as the change to “License” will survive reboots.  You can confirm this by ssh’ing into the radio and running the commands:

XM.v6.3.0# iwpriv wifi0 getCountry
wifi0     getCountry:UB 
XM.v6.3.0# iwpriv wifi0 getCountryID
wifi0     getCountryID:511 
XM.v6.3.0# 

I guess the Country “UB” is “Ubiquiti” 🙂

Protecting your Asterisk PBX…

Disclaimer:  This is likely not everything you should do in securing your FreePBX box, but it is a good starting point.   It is meant to limit your exposure to the public Internet, but this doesn’t cover exploits that can happen after you lock that down such as from address space you are allowing.  Having good credentials, among other security measures is also expected.

FreePBX‘s GUI makes setting up Asterisk a breeze.  This is a great product that Sangoma has been supporting for a while.  You can get the main package at https://www.freepbx.org/downloads/freepbx-distro for running on your own server or VM.  I am running RasPBX that is a slightly slimmed down version that runs very nicely on a Raspberry Pi.

The packages are built on a Debian-based Linux but have some exposure that one needs to do to really lock it down.  There are a bunch of applications that are on FreePBX that are listening and waiting for the rest of the Internet to exploit them.  You can easily see them by running the command “lsof -i | grep LISTEN”.  You may see something like:

root@asterisk:/etc/network# lsof -i | grep LISTEN
sshd        612        root    3u  IPv4   10163      0t0  TCP *:ssh (LISTEN)
sshd        612        root    4u  IPv6   10165      0t0  TCP *:ssh (LISTEN)
mysqld     1149       mysql   10u  IPv4   10996      0t0  TCP localhost:mysql (LISTEN)
exim4      1592 Debian-exim    4u  IPv4   11165      0t0  TCP localhost:smtp (LISTEN)
exim4      1592 Debian-exim    5u  IPv6   11166      0t0  TCP localhost:smtp (LISTEN)
asterisk   1971    asterisk   13u  IPv6   15624      0t0  TCP *:omniorb (LISTEN)
asterisk   1971    asterisk   14u  IPv6   15625      0t0  TCP *:8089 (LISTEN)
asterisk   1971    asterisk   15u  IPv4   13884      0t0  TCP localhost:5038 (LISTEN)
node       6868    asterisk   13u  IPv6 2435746      0t0  TCP *:8001 (LISTEN)
node       6868    asterisk   14u  IPv6 2435747      0t0  TCP *:8003 (LISTEN)
apache2   24214    asterisk    4u  IPv6   10221      0t0  TCP *:http (LISTEN)
apache2   24214    asterisk    6u  IPv6   10225      0t0  TCP *:https (LISTEN)
apache2   24215    asterisk    4u  IPv6   10221      0t0  TCP *:http [...]

root@asterisk:/etc/network# 

The first column shows the application that is bound and listening to that port.  The last column shows the protocol, what IP address it is bound to and what port.  Anything that shows “*:…” means it is listening to all IP addresses on the server.  Anything that shows “localhost:…” means it is listening to a local IP address and is not exposed to the public Internet.

So what are these applications and why do they need to be open to the net?  The first one is “sshd” which you likely are using to log into the server with.

mysqld” is the daemon for the MySQL database that FreePBX uses for configuration and it is bound to the “localhost” so no worry here.  If it isn’t you can see https://serversforhackers.com/c/mysql-network-security for a guide on how to lock down MySQL.

exim4” is a mail server.  Normally this is used for delivery of mail to users when there is voicemail or there is a report that needs to be mailed out to an administrator. 

“asterisk” has a number of ports open.  The “omniorb” port really is port “8088”.  You can see this in the file /etc/services.  The 8089 port is used by the same feature in Asterisk.  This is the “Asterisk GUI” framework.  FreePBX uses this for interaction with Asterisk.  It really should be locked down to the localhost IP address.    You can change this to the localhost IP address of “127.0.0.1” in FreePBX’s “Settings” -> “Advance Settings” tab.  Change the HTTP and HTTPS bind address entries for these ports and click on “Submit” and then “Apply Config”.  Port 5038 is used for the API for Asterisk.  By the default config, it is already bound to locahost / 127.0.0.1.

“node” that is listening on ports 8001 and 8003 is the FreePBX nodejs application that is used for the “User Control Panel (UCP) Node Server” that FreePBX uses.  This is used for a user to check voicemail, etc.   You can map Asterisk to use the localhost IP address by going into the “Settings” and then “Advanced Settings” and go to the section called “UCP NodeJS Server” and update the “Bind Address” to “127.0.0.1”.  (This seems to be not on FreePBX 15 and is already mapped to the localhost.)

Session Initiation Protocol (SIP)

SIP is going to be your biggest risk.   Depending on the distribution of FreePBX/RasPBX/etc., asterisk really has to listen to the SIP ports (5060/5061) so your phones and providers can send commands to Asterisk to do things like making a call.  Having Asterisk just exposed to the Internet at large means that your Asterisk box will be cracked in 30 seconds once you put it on the net.  Hackers love to find an exposed SIP server out there to make outbound calls through your box.  This obscures who is making the call and they don’t have to pay for the VoIP service.  You are.

There are a number of ways to deploy an Asterisk box.  You can either do it behind a firewall with rules on the firewall to allow certain IP addresses to be passed to the Asterisk box or you can have Linux do the filtering.   FreePBX and RasPBX both have FAIL2BAN as an option.   Fail2Ban is a program that scans the logs of various programs like SSH and Asterisk and looks for times that an IP address is trying to crack into your site and was unsuccessful.  If so, it will add a firewall rule on Linux to prevent that IP address from accessing the box for some time frame.  By default, this is about 10 minutes.  After that, the rule is taken out and that IP address can try again.  This can retard the progress of someone trying to crack into your box, but do you really want the hassle?  What happens if they do make through?  Then you may find some rather expensive bills from your VoIP provider.   Unless you really want to be serving up SIP to the rest of the Internet, let’s lock this puppy down.

My installation has a public IP address and I rely on using Linux’s “netfilter” to firewall.  Netfilter can be configured a couple of ways.  The more traditional is to use IPTABLES.  I am not going into detail on how to use IPTABLES as there are plenty of sites that can show you that.

First off, identify the IP addresses you need to allow to connect to your Asterisk box.  This will include your VoIP provider’s session border controller (SBC) that will be protecting their SIP server.  I use VoIP.ms out of Canada and they list their servers on their wiki.  Also, identify the IP addresses that your phones will be using.  This may be on your local LAN or they may be coming from other sites.  Hopefully, these are IP addresses that won’t be changing much.  In times past, I had to run an Asterisk server that had a number of home employees, so they were coming from cable providers that would change the IP address on occasion.  Normally this wouldn’t change that often so it was manageable.  Let’s say you have the following IP  addresses…

  • VOIP provider: 1.1.1.1
  • Your local LAN’s prefix 192.168.0.0/24 assuming your Asterisk box is directly connected.
  • Your local LAN’s public IP address 2.2.2.2/32 if your Asterisk box is not connected to your LAN
  • An offsite worker: 3.3.3.3/32.

You will need to create a file like /etc/iptables/rules.v4 that looks something like:

*filter
# SIP...
-A INPUT -s 1.1.1.1 -p all -m all --dport 5060 -m comment --comment "VoIP Provider" -j ACCEPT
-A INPUT -s 1.1.1.1 -p all -m all --dport 5061 -m comment --comment "VoIP Provider" -j ACCEPT
-A INPUT -s 2.2.2.2 -p all -m all --dport 5060 -m comment --comment "Public IP of Office" -j ACCEPT
-A INPUT -s 2.2.2.2 -p all -m all --dport 5061 -m comment --comment "Public IP of Office" -j ACCEPT
-A INPUT -s 3.3.3.3 -p all -m all --dport 5060 -m comment --comment "Remote Worker" -j ACCEPT
-A INPUT -s 3.3.3.3 -p all -m all --dport 5061 -m comment --comment "Remote Worker" -j ACCEPT
-A INPUT -p all -m all --dport 5060 -m comment --comment "Drop all other SIP connections" -j DROP
-A INPUT -p all -m all --dport 5061 -m comment --comment "Drop all other SIP-TLS connections" -j DROP

In this file, you are allowing SIP access for the IP addresses listed to port 5060 (UDP) and port 5061 (TCP/TLS).  The very last two rules say, if IP addresses don’t match the above, then DROP those packets.

Secure Shell (SSH)

This is another popular service that as soon as you stand up a server, you will see bots knocking at your door.  Again, fail2ban can address this, but why would you want the exposure?  Let’s just lock it down to a couple of IP addresses with IP tables.  Updating the /etc/iptables/rules.v4 file and using the example IP addresses we used for SIP…

# SSH...
-A INPUT -s 2.2.2.2 -p tcp -m tcp --dport 22 -m comment --comment "Office Public IP Address" -j ACCEPT
-A INPUT -s 3.3.3.3 -p tcp -m tcp --dport 22 -m comment --comment "Offisite IT Provider" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -m comment --comment "Drop all other SSH TCP connections" -j DROP
Apache

This is the webserver to get into the FreePBX GUI.  Let’s just lock it down to a couple of IP addresses with IP tables.  Again, updating the /etc/iptables/rules.v4 file and using the example IP addresses we used for SIP…

# Apache...
-A INPUT -s 2.2.2.2 -p tcp -m tcp --dport 80 -m comment --comment "Office Public IP Address" -j ACCEPT
-A INPUT -s 3.3.3.3 -p tcp -m tcp --dport 80 -m comment --comment "Offisite IT Provider" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -m comment --comment "Drop all other Web TCP connections" -j DROP
-A INPUT -s 2.2.2.2 -p tcp -m tcp --dport 443 -m comment --comment "Office Public IP Address" -j ACCEPT 
-A INPUT -s 3.3.3.3 -p tcp -m tcp --dport 443 -m comment --comment "Offisite IT Provider" -j ACCEPT 
-A INPUT -p tcp -m tcp --dport 443 -m comment --comment "Drop all other Web SSL TCP connections" -j DROP
IPTABLES That is Persistent

I mentioned added these rules to /etc/iptables/rules.v4. We are going to add a bit of config to the networking section of FreePBX to have the server add these rules on reboot.

Simply you need to add the line “pre-up /sbin/iptables-restore /etc/iptables/rules.v4″ for the network interface you are using to /etc/network/interfaces.  It should look something like:

auto eth0
iface eth0 inet static
    pre-up /sbin/iptables-restore /etc/iptables/rules.v4

This tells the server to load the rules you created before it turns up the network interface.

The more recent FreePBX/Raspbx uses dhcpcd.  There is a set of scripts that get run in /lib/dhcpcd/dhcpcd-hooks/.  You can create a file called “/lib/dhcpcd/dhcpcd-hooks/05-iptables” and put “/sbin/iptables-restore /etc/iptables/rules.v4” it in.  This will add the rules before the rest of the network gets turned up.

Check Your work

Reboot your box and see if the IPTABLES loaded.  If you have fail2ban loaded, you will also see entries from it there.

root@asterisk:/home/pozar# iptables --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
fail2ban-asterisk  tcp  --  anywhere             anywhere             tcp dpt:sip
fail2ban-asterisk  udp  --  anywhere             anywhere             udp dpt:sip
fail2ban-asterisk  tcp  --  anywhere             anywhere             tcp dpt:sip-tls
fail2ban-ssh  tcp  --  anywhere             anywhere             multiport dports ssh
ACCEPT     any  --  1.1.1.1  anywhere             any dpt:sip /* voip provider */
ACCEPT     tcp  --  1.1.1.1  anywhere             tcp dpt:sip-tls /* voip provider */
ACCEPT     any  --  2.2.2.2       anywhere             any dpt:sip /* Office Public IP */
ACCEPT     tcp  --  2.2.2.2       anywhere             tcp dpt:sip-tls /* Office Public IP */
ACCEPT     any  --  3.3.3.3       anywhere             any dpt:sip /* Joe's Home Office */
ACCEPT     tcp  --  3.3.3.3       anywhere             tcp dpt:sip-tls /* Joe's Home Office */
DROP       any  --  anywhere             anywhere             any dpt:sip /* Drop all other SIP UDP connections */
DROP       tcp  --  anywhere             anywhere             tcp dpt:sip-tls /* Drop all other SIP-TLS connections */
ACCEPT     tcp  -- 2.2.2.2              anywhere             tcp dpt:ssh /* Office Public IP address */
DROP       tcp  --  anywhere             anywhere             tcp dpt:ssh /* Drop all other SSH TCP connections */

Now look at the daemons and see if they are bound to the IP addresses you are expected:

root@asterisk:/home/pozar# lsof -i | grep LIST
sshd       604        root    3u  IPv4  12384      0t0  TCP *:ssh (LISTEN)
sshd       604        root    4u  IPv6  12386      0t0  TCP *:ssh (LISTEN)
apache2   1087        root    4u  IPv6  12480      0t0  TCP *:http (LISTEN)
apache2   1087        root    6u  IPv6  12484      0t0  TCP *:https (LISTEN)
mysqld    1148       mysql   10u  IPv4  13502      0t0  TCP localhost:mysql (LISTEN)
apache2   1176    asterisk    4u  IPv6  12480      0t0  TCP *:http (LISTEN)
apache2   1176    asterisk    6u  IPv6  12484      0t0  TCP *:https (LISTEN)
[...]
exim4     1562 Debian-exim    4u  IPv4  12786      0t0  TCP localhost:smtp (LISTEN)
exim4     1562 Debian-exim    5u  IPv6  12787      0t0  TCP localhost:smtp (LISTEN)
asterisk  1802    asterisk   13u  IPv4  13291      0t0  TCP localhost:omniorb (LISTEN)
asterisk  1802    asterisk   14u  IPv4  13292      0t0  TCP localhost:8089 (LISTEN)
asterisk  1802    asterisk   15u  IPv4  13293      0t0  TCP localhost:5038 (LISTEN)
node      1993    asterisk   13u  IPv4  14461      0t0  TCP localhost:8001 (LISTEN)
node      1993    asterisk   14u  IPv4  14462      0t0  TCP localhost:8003 (LISTEN)
Conclusion

As mentioned in my disclaimer, this likely isn’t everything you need to do but it will make a dent in having your server hijacked for unwanted SIP calls to countries you never thought you called.

 

Mark in a Box

Some months back a friend of mine, Mark Hernandez passed away. Mark was a smart, funny and strange guy. He was also active with KFCF in Fresno and KPFA in Berkeley. I was disappointed that there wasn’t a memorial (or one I heard about) for him. I started to work on an “art piece” that I think Mark would have liked.

Fortunately, we have audio of Mark from KFCF’s RPU program that was broadcast in the 1980s. I took a number of cuts of the audio from the “show” and put them on an SSD card, put that on an MP3 shield for a Mega Arduino board. Stuck the electronics in a box that I painted “urn” gold and put a lit button on it. Whenever you push the button, you get silly audio of Mark cut played back.

I started off with a 6 sided (hexagon) box where the edges are routed so they slip into each other.  Below you can see the joint corner.

I wanted the box to look pretty simple so I hid the power connector in the back and the speaker is on the bottom with a bit of wood for feet to lift the box about 1/4 inch off the surface it is sitting on to not muffle the sound.

Inside the box you can see the Arduino Mega with the MP3 shield, speaker under the blue tape and cable to the button and power connector.

The completed box after painting it gold to look a bit like an urn…

 

 

 

 

 

 

 

 

Here is a quick and small video of it doing its thing.

The hacky source that I really should clean up can be found at https://github.com/pozar/mark_in_a_box.  The parts list can be found in the Docs directory.

Long Live Slim Device’s Squeezebox server…

Back in 2000 a couple of guys Sean Adams and Dean Blackketter started a company called Slim Devices, Inc in Mountain View.  The made these little cute audio boxes that would talk to a back-end music server written in Perl.  The boxes first called SIMP3 and then “Squeezebox”, at first had just an ethernet NIC, a power jack and a couple of RCA pin plugs to run audio out to your stereo system.

The back end server called SlimServer, was a chunk of Perl code that would scan your music library, create a database and wait around for a request for the directory and stream an audio file to the Squeezebox player.  As the code was open, folks started to hack on it to add plugins and features to the server.   For instance, Dan Aronson added a plugin to playback music from Internet Archive’s Live Music Archive (etree.org).

To playback music, you could either use an IR remote control to scan through your music library via the Squeezebox display or go to the server’s web interface.   Slim Devices put some thought into the web UI so it was pretty clean and intuitive.   The system had some nice features including syncing up multiple players so your whole house can be playing the same thing, very easy search by what you would expect such as genre, artist, album, etc.

In 2006, Logitech bought Slim Devices and the usual story happened.  The acquiring company didn’t know what to do with it and ignored it.  Logitech did come out with a player or so but it really didn’t last long as they didn’t do much to market the devices so they discontinued the line in 2012.   Sonos was directly competing with Slim Devices and doing a much better job in marketing to high-end home audio installations and when you think of audio players for rooms, you likely will think of Sonos.

I loved my Squeezebox players and the SlimServer.  When my players started to fail, I started to buy new/old Squeezeboxes on eBay, but they started to fail too. I looked at replacing the system with Sonos.  Two things stopped me, they are EXPENSIVE and they can only handle a music library that has less than 65,000 cuts.  At this very moment, my SlimServer reports I have 117,749 tracks, 13,857 albums, 15,885 artists in 468 genres with a total playing time of 8,375 hours or about a year.  As Sonos can’t manage something this size, it meant that it isn’t an option.

Fortunately, there seems to be a large Slimserver fan base.  Enough that the server itself is still under active community development with nightly builds.  On the player side of things, there are a number of efforts that will make a Raspberry Pi look like a Squeezebox.  Examples are Pi MusicBox and Max2Play.  My personal favorite is piCorePlayer as three isn’t any licensing but you can send them a donation, the package is kept up and has some great features built into the distribution.

As with any of these distributions, you download an IMG file, use something like Etcher to write the image to the SD card you are going to use in your Raspberry Pi, plug in the SD card, power up your Raspberry Pi and with a little bit of setup, you have a player that you can toss behind a pair of powered speakers.  You would use the Slimserver web interface to tell the player what to playback.  This actually works pretty well.  But I wanted a bit more.

I found the digital to analog converter in the Raspberry Pi’s a bit lacking as they tend to be a bit noisy and not what I would call “audiophile”.  Poking around a bit I found JustBoom’s HAT for the Raspberry Pi that has a TI TAS5756M amplifier and DAC chip that can do up to 192KHz and 32 bits.  It can do 30 watts per channel which is just fine for the bedroom.  If I wanted something with a bit more power, I would get the JustBoom’s DAC HAT without the amplifier and run it into my audio system.

Having a “headless” player without a screen is fine for a speaker system you want to bury and be out of the way, but in my bedroom, I wanted a display that shows cover art, has some nice screensavers like VU meters wiggling to the music, etc.  It should turn off or have some dim clock display.   In fact, the piCorePlayer supports all of that if you hook up a display to your Pi.  It integrates very well with the Raspberry Pi Foundation’s 7-inch touch display.   You really just need one more thing, some container to hold it all together with and be a stand for your nightstand.  Most cases to support the Pi and the 7-inch screen are designed for only those devices.  If you want to add a HAT, things will not fit.  I did find one company called Smarticase that has a case called the “SmartiPi Touch” that supports Pi’s with HATs.

So the cost breakdown is:

  1. Raspberry Pi 3 model B – $35
  2. Smarticase – $28
  3. 32 GB SD card – $10
  4. Raspberry Pi power supply – $10
  5. Justboom DAC/Amp HAT – $75
  6. Raspberry Pi Foundation 7-inch touch screen – $72

So for about $230, you can have a pretty fancy audiophile Squeezebox that is going to be supported and being developed.  The software is free.  If you can go headless and are happy with the Pi’s onboard DAC, you can bring this price down to about $60.

A quick addendum…  There is a mail list discussion where users can discuss Squeezebox devices and the Slimserver at http://lists.slimdevices.com/mailman/listinfo/discuss

 

Syncing and backing up two desktops

Unison
A Bi-directional File Synchronizer.

I tend to live on two virtual locations: My MacBook Pro running OS X and my main server KUMR.LNS.COM running Linux.  I should say, for decades before I switched to Linux, I ran various versions of *BSD on KUMR.

I have a home directory on both boxes with a subfolder called “projects” that has various things I have been working on for the last 30 or so years.  I want these directories and files in both locations for access and this gives me some semblance of a backup as well.  (Of course I have other backup methods including TimeMachine and other off-site backups.)

Additionally, if I am doing development, I will tend to use my Mac but OS X does have some peculiarities in how various packages like Python, etc. get installed and running package managers like home-brew may not load what I need for an environment that will be deployed on some server so I will do that work on KUMR.  (Ya… I know about containers, and VMs.).

The challenge is how to keep things in sync with each other.  For quite a while I have been using Unison, a file synchronizer that is bi-directional and uses rsync’s rather efficient method of file transfer.  I will skip describing the rsync protocol, but you can check out the paper at https://rsync.samba.org/how-rsync-works.html for the details.

Unison is extremely efficent in working through large file collections.  I currently have about 305 GB with 215,450 files and 27,888 directories just in my “project” folder.  If I was just using “rsync”, it would take a large amount of time in walking through each file, computing the hash, seeing if the hash was the same on the other server and starting up a transfer if it isn’t.  Unison will make a similar crawl of all the files once and then keep track of files via a hash of the file and put it into an archive in ~/.unison directory.    This means that the first time I run Unison it may take a hours to crawl through all the files, but subsequent runs may take less than a minute to scan and transfer, depending on what was changed last.

If you are worried that Unison is missing anything with this system, just go into the .unison directory and delete the archives in both the local an remote servers.  Normally they start with “ar*” or “fp*” and then run unison again.

Unison also knows when you have just moved a file or folder with the same material.  If it sees a new file with the same hash and name of an existing file, it just moves that file as a “shortcut”.  A big win in just moving a folder with a large number of files or large files.

Since I have been using Unison for a while, I have had some tweaks to the Unison configuration file (~/.unison/default.prf).  I thought I would share mine here with some comments that detail the config file itself.  This, by no means is a complete set of options for Unison.  You can see all of it detailed in the manual, of which I would highly suggest reviewing.

# Unison preferences file
# Local root directory...
root = /Users/pozar/
# Remote server, ssh port number and root directory...
root = ssh://lns.com:22//home/pozar
# The program to use to copy files with. In this case rsync...
copyprog = rsync -aX --rsh='ssh -p 22' --inplace --compress
# The program to use to copy files wit that supports partial transfers...
copyprogrest = rsync -aX --rsh='ssh -p 22' --partial --inplace --compress
# maximum number of simultaneous file transfers.  Good to have more than one to really use the pipe.
maxthreads = 5
# synchronize resource forks and HFS meta-data. (true/false/default)
# I'm not interested in seeing AppleDouble files on my Linux box...
rsrc = false 
# Filename and directories to ignore...
ignore = Name projects/Mail/.imap
ignore = Name .FBCIndex
ignore = Name .FBCLockFolder 
ignore = Name {Cache*,.Trash*,.VolumeIcon.icns,.HSicon,Temporary*,.Temporary*,TheFindByContentFolder}
ignore = Name {TheVolumeSettingsFolder,.Metadata,.filler.idsff,.Spotlight,.DS_Store,.CFUserTextEncoding} 
# ignore all files with .unison somewhere in their full path
ignore = Path .unison
# Normally ignore the giant VM Harddrive files...
ignore = Name Documents/Parallels
# Don't try to copy this socket...
ignore = Name projects/gnupg/S.gpg-agent
# ignore = Name Music/iTunes
ignore = Name .lnk
ignore = Name .DS_Store
ignore = Name Thumbs.db
# Directory paths off of root to sync...
path = projects
path = Desktop
path = Documents
path = Downloads
# Keep ssh open to help with long syncs such as the initial one
sshargs = -C -o TCPKeepAlive=yes
# synchronize modification times
times = true
# Create a local log of what happened.
log = true
# Uses the modification time and length of a file as a quick check if a file has changed.
fastcheck=true

There is a handy command line argument called “-batch” which will avoid asking about what to do with each file it found to sync.  Normally it will figure things out by looking at the date stamp.  In some cases, it won’t know that to do.  You can see below an example where the permissions or times of a file may be in conflict with various Django files I have.  In this case I want to propagate this meta data to my remote server “KUMR”.  I would normally use the “>” key to tell it to go from left (Local) to right (KUMR)…

# unison
Contacting server...
Connected [//Tims-MBP.local//Users/pozar -> //kumr//home/pozar]
Looking for changes
  Waiting for changes from server                                       
Reconciling changes
local          kumr               
props    ====> props      Desktop/django/lib/python3.6/site-packages/django/contrib/contenttypes/locale/lb/LC_MESSAGES/django.mo  [] >
props    ====> props      Desktop/django/lib/python3.6/site-packages/django/contrib/contenttypes/locale/lb/LC_MESSAGES/django.po  [] >
props    <-?-> props      Desktop/django/lib/python3.6/site-packages/django/contrib/contenttypes/locale/lt/LC_MESSAGES/django.mo  [] y
Unrecognized command 'y': try again  [type '?' for help]
props    <-?-> props      Desktop/django/lib/python3.6/site-packages/django/contrib/contenttypes/locale/lt/LC_MESSAGES/django.mo  []

With the “-batch” command it avoids this so you can just script this to run from a cronjob if you like.  I normally run unison in batch mode using an alias for bash like:

alias unison='unison -batch'

Of course, if you have a situation like above, it won’t get “fixed”.  That may be fine 99.9% of the time.  Occasionally I run unison without the batch argument just to get things fully in sync.

But what happens if I have a thousand files like this?  Say for some reason, the modify times on a bunch of files got change on both sides.  Typically you would just use the UNIX “yes” command to tell it to send ‘>’ to the program with something like “yes \>”.  Unison will take this input happily until it comes to the last question where it asks if you want to propagate the changes.  Then it is looking for a ‘y’ or ‘n’.  Fortunately a ‘y’ is ignored when Unison is asking what direction to propagate the files (see above).  So you can use the bash command:

while true; do echo ">"; echo "y" ;done | unison

This sends a ‘>’ and then a ‘y’ continuously into unison.  Eventually it will ask if it should propagate the changes and it will get a ‘y’.

I should say this command should be considered a bit “dangerous” unless you are sure, the meta data and files you are propagating are what you want on the other side.

Hope this gives you some insight on this rather handy tool.  Drop me a line if you have comments or questions.

 

Blocking traffic by MAC address on Ubiquiti EdgeRouters by Time of Day

See the update at the bottom of this post – Tim 20180211

I have an Ubiquiti EdgeRouter PoE at the house as my main router.  In order to manage “resources” at the house,  I wanted a way to block a couple of MAC addresses at a certain time each day.   I created a filter that blocks by MAC address that looks something like:

set firewall name SWITCH0_IN default-action accept
set firewall name SWITCH0_IN description 'Used for blocking local users'
set firewall name SWITCH0_IN rule 1 action drop
set firewall name SWITCH0_IN rule 1 description APhone
set firewall name SWITCH0_IN rule 1 disable
set firewall name SWITCH0_IN rule 1 log disable
set firewall name SWITCH0_IN rule 1 protocol all
set firewall name SWITCH0_IN rule 1 source mac-address '66:55:44:33:22:11'
set firewall name SWITCH0_IN rule 2 action drop
set firewall name SWITCH0_IN rule 2 description iPhone
set firewall name SWITCH0_IN rule 2 disable
set firewall name SWITCH0_IN rule 2 log disable
set firewall name SWITCH0_IN rule 2 protocol all
set firewall name SWITCH0_IN rule 2 source mac-address '11:22:33:44:55:66'
set firewall name SWITCH0_IN rule 3 action drop
set firewall name SWITCH0_IN rule 3 description Desktop
set firewall name SWITCH0_IN rule 3 disable
set firewall name SWITCH0_IN rule 3 log disable
set firewall name SWITCH0_IN rule 3 protocol all
set firewall name SWITCH0_IN rule 3 source mac-address '12:34:56:78:90:ab'

I applied this rule to the “switch0” interface that talks to my LAN interfaces at eth2, eth3 and eth4.

For the rulesets above, I want to enable rule #2 and #3 for the devices “iPhone” and “Desktop” to block traffic from them.  Two hours later, I want to disable this rule to pass traffic again.  This script does just that…

#!/bin/vbash                                              
# A script to disable access for two MAC addresses at     
# 0330 to 0530 UTC (1930 to 2130 Pacific) everyday.
unblock=0
if [ $# == 1 ]; then
  unblock=1
fi
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
WR="/opt/vyatta/sbin/vyatta-cfg-cmd-wrapper"
$WR begin
if [ $unblock == 1 ]; then
  $WR set firewall name SWITCH0_IN rule 2 disable
  $WR set firewall name SWITCH0_IN rule 3 disable
else
  $WR delete firewall name SWITCH0_IN rule 2 disable
  $WR delete firewall name SWITCH0_IN rule 3 disable
fi
$WR commit                                     
$WR end

There are a couple of ways to configure the router with scripts.  Ubiquiti suggests using the /opt/vyatta/etc/functions/script-template script like:

 

#!/bin/vbash
     
if [ $# == 0 ]; then
  echo usage: $0 
  exit 1
fi
new_ip=$1;
     
source /opt/vyatta/etc/functions/script-template
configure
set interfaces tunnel tun0 local-ip $new_ip
commit
save
exit

This actually breaks due to a bug.  I have had to use the /opt/vyatta/sbin/vyatta-cfg-cmd-wrapper as part of my script.  Works just fine.

The CRON entries that will run the scripts at 0330 and 0530 UTC…

30 3 * * * /home/joeuser/block_mac.sh
30 5 * * * /home/joeuser/block_mac.sh unblock

Updated on Feb 11th 2018...

It seems that either I missed this feature or Ubiquiti just added it.  You can add times to enable and disable the rule.  For instance, in the case of Rule #2 above, you would add starttime and stoptime statements.  You can also specify date of the week or date such as day/month/year.  This has been in Vyatta for a while now.

set firewall name SWITCH0_IN rule 2 action drop
set firewall name SWITCH0_IN rule 2 description iPhone
set firewall name SWITCH0_IN rule 2 disable
set firewall name SWITCH0_IN rule 2 log disable
set firewall name SWITCH0_IN rule 2 protocol all
set firewall name SWITCH0_IN rule 2 source mac-address '11:22:33:44:55:66'
set firewall name SWITCH0_IN rule 2 time starttime '21:00:00'
set firewall name SWITCH0_IN rule 2 time stoptime '22:00:00'
set firewall name SWITCH0_IN rule 2 time utc

Radio Air Checks…

Radio air checks are normally used by radio talent and program directors to go back and listen to the talent’s performance. Many times these air checks are recordings of a DJ’S shift but was “telescoped” so it only recorded the time that the talent’s microphone was turned on (aka ‘open’). Air checks were also collected by radio fans that wanted to either record a DJ they liked or perhaps event grab that hit song they can listen to later.

Mike Schweizer was a bit of a radio fan boy in his early years. Later on he became a radio engineer specializing in remote broadcasts and working as an engineer for stations like KUSF, KYA and KSFO. As a kid and through his adult life, he made recordings of stations and collected hundreds of reels of tape. Unfortunately his life was cut short and passed away in 2011. Before he passed, he transferred a good number of these airchecks to digital. I have many of these up at my site that you can listen to.  Since I published these some years ago, Internet Archive has recently copied these recordings and has put them up there.

Many of these are classic as they are recordings of early free form radio such as KMPX or air checks of Wolfman Jack on XEPRS.

I should say that not all of these were recorded by Mike, it seems that there are a handful that were recorded by others that have snuck in here.  If you find that one of those are yours, please drop me a note and I can remove it if you deem it necessary.

Librenms’ API

Librenms is a very flexible network and server monitoring and alerting system.  I have it deployed at a number of companies based on the ease of installation, the fact that it auto discovers devices, it is updated frequently (typically multiple times a week) and supports pretty much every network device you can think of.

On top of that, the alerting can be tuned to match very specific cases as the back end is MySQL so you alerting conditions can match almost anything you can write a SQL query for.  A good example would be to only alert on certain interfaces that have a specific description in them such as “TRANSIT” where the device has a hostname of “edge” and is only a 10Gbs connection (the interface name is ‘xe’).  Because you can group things by description or part of a hostname, you can just say anything with the string “edge” in the hostname should be considered a “edge router” so a group “ER” can be created for these devices.  With autodiscovery, as soon as you add a device, it will get automatically be put into the group that the rule/regular expression matches it.

One of the more interesting features is Libre’s API.  You can get pretty much any detail you want out of what Libre has collected and stored in the DB.  It will also create graphs for you on the fly.  One case I have had in the past is to create daily and weekly total bandwidth graphs for a set of specific ports on a group of switches.  The switch ports have a particular unique string I can match on so I was able to create a “group” called “peering” that included these ports over all of the switches.

I wrote this simple script called create_public_graphs.sh that asked for a graph for daily and weekly time frames.  I also added various options to the request such as don’t show the legend of interfaces and make the in and out directions all one color.  The other option is to make different colors for each interface.  We wanted a clean look so we went for a solid color.  The API doesn’t do everything you may want such as titling the graph.  This is where I use the “convert” program from imagemagick to overlay some text at the top of the graph.  You can see the final result at the SFMIX site.

Getting Mediainfo Data into a CSV

Mediainfo is a pretty handy tool to examine media files like MP4 containers and the streams in it such as the video and audio streams. It will also spit out a couple of formats in order parse the data such as XML. But I just want to get a certain set of data and want to move it into a CSV file so I can bring it into something like Google Sheets or Excel. Seems the “Inform” argument can get me some of the way there. You can tell it to give you multiple data points about a particular stream or “General” aspect about the file. You can’t mix say “Audio” and “Video”. That’s ok. I just want a handful of things about the Video stream and the filename. Woops. The file name is in the “General” bucket. So I am going to cheat a little with the “echo” command and tell it to print the file name and a comma and not to print out a new line in order to have it create the first column for the CSV row.

So this little script will find my MP4 movie files in the “/foo/*” directories and subdirectories, assign the name to the “movie” variable, print out the name and comma without a new line and then spit out a bunch of stuff about the video stream to get me a nice CSV output…

#!/bin/sh
echo "Filename,PixelWidth,PixelHeight,DisplayRatio,Bitrate,FrameRate"
find /foo/ -type f | grep -i \.MP4 | while read movie;do
echo -n "$movie,"
mediainfo --Inform="Video;%Width%,%Height%,%DisplayAspectRatio/String%,%BitRate%,%FrameRate%" $movie
done

And I get something like:

Filename,PixelWidth,PixelHeight,DisplayRatio,Bitrate,FrameRate
/foo/fill1.mp4,1440,1080,4:3,15000000,23.976
/foo/fill2.mp4,1440,1080,4:3,15000000,23.976
/foo/fill3.mp4,1920,1080,16:9,15000000,29.970

You can get a list of Mediainfo parameters to use with “–Info-Parameters” argument.

[Update: I created a mediainfo template and script that will create a nice CSV with lots of info from an MP4 container. Assuming one video and audio track for the container.  You can see it on github at: https://github.com/pozar/mediainfo2csv]