Protecting your Asterisk PBX…

Disclaimer:  This is likely not everything you should do in securing your FreePBX box, but it is a good starting point.   It is meant to limit your exposure to the public Internet, but this doesn’t cover exploits that can happen after you lock that down such as from address space you are allowing.  Having good credentials, among other security measures is also expected.

FreePBX‘s GUI makes setting up Asterisk a breeze.  This is a great product that Sangoma has been supporting for a while.  You can get the main package at https://www.freepbx.org/downloads/freepbx-distro for running on your own server or VM.  I am running RasPBX that is a slightly slimmed down version that runs very nicely on a Raspberry Pi.

The packages are built on a Debian-based Linux but have some exposure that one needs to do to really lock it down.  There are a bunch of applications that are on FreePBX that are listening and waiting for the rest of the Internet to exploit them.  You can easily see them by running the command “lsof -i | grep LISTEN”.  You may see something like:

root@asterisk:/etc/network# lsof -i | grep LISTEN
sshd        612        root    3u  IPv4   10163      0t0  TCP *:ssh (LISTEN)
sshd        612        root    4u  IPv6   10165      0t0  TCP *:ssh (LISTEN)
mysqld     1149       mysql   10u  IPv4   10996      0t0  TCP localhost:mysql (LISTEN)
exim4      1592 Debian-exim    4u  IPv4   11165      0t0  TCP localhost:smtp (LISTEN)
exim4      1592 Debian-exim    5u  IPv6   11166      0t0  TCP localhost:smtp (LISTEN)
asterisk   1971    asterisk   13u  IPv6   15624      0t0  TCP *:omniorb (LISTEN)
asterisk   1971    asterisk   14u  IPv6   15625      0t0  TCP *:8089 (LISTEN)
asterisk   1971    asterisk   15u  IPv4   13884      0t0  TCP localhost:5038 (LISTEN)
node       6868    asterisk   13u  IPv6 2435746      0t0  TCP *:8001 (LISTEN)
node       6868    asterisk   14u  IPv6 2435747      0t0  TCP *:8003 (LISTEN)
apache2   24214    asterisk    4u  IPv6   10221      0t0  TCP *:http (LISTEN)
apache2   24214    asterisk    6u  IPv6   10225      0t0  TCP *:https (LISTEN)
apache2   24215    asterisk    4u  IPv6   10221      0t0  TCP *:http [...]

root@asterisk:/etc/network# 

The first column shows the application that is bound and listening to that port.  The last column shows the protocol, what IP address it is bound to and what port.  Anything that shows “*:…” means it is listening to all IP addresses on the server.  Anything that shows “localhost:…” means it is listening to a local IP address and is not exposed to the public Internet.

So what are these applications and why do they need to be open to the net?  The first one is “sshd” which you likely are using to log into the server with.

mysqld” is the daemon for the MySQL database that FreePBX uses for configuration and it is bound to the “localhost” so no worry here.  If it isn’t you can see https://serversforhackers.com/c/mysql-network-security for a guide on how to lock down MySQL.

exim4” is a mail server.  Normally this is used for delivery of mail to users when there is voicemail or there is a report that needs to be mailed out to an administrator. 

“asterisk” has a number of ports open.  The “omniorb” port really is port “8088”.  You can see this in the file /etc/services.  The 8089 port is used by the same feature in Asterisk.  This is the “Asterisk GUI” framework.  FreePBX uses this for interaction with Asterisk.  It really should be locked down to the localhost IP address.    You can change this to the localhost IP address of “127.0.0.1” in FreePBX’s “Settings” -> “Advance Settings” tab.  Change the HTTP and HTTPS bind address entries for these ports and click on “Submit” and then “Apply Config”.  Port 5038 is used for the API for Asterisk.  By the default config, it is already bound to locahost / 127.0.0.1.

“node” that is listening on ports 8001 and 8003 is the FreePBX nodejs application that is used for the “User Control Panel (UCP) Node Server” that FreePBX uses.  This is used for a user to check voicemail, etc.   You can map Asterisk to use the localhost IP address by going into the “Settings” and then “Advanced Settings” and go to the section called “UCP NodeJS Server” and update the “Bind Address” to “127.0.0.1”.  (This seems to be not on FreePBX 15 and is already mapped to the localhost.)

Session Initiation Protocol (SIP)

SIP is going to be your biggest risk.   Depending on the distribution of FreePBX/RasPBX/etc., asterisk really has to listen to the SIP ports (5060/5061) so your phones and providers can send commands to Asterisk to do things like making a call.  Having Asterisk just exposed to the Internet at large means that your Asterisk box will be cracked in 30 seconds once you put it on the net.  Hackers love to find an exposed SIP server out there to make outbound calls through your box.  This obscures who is making the call and they don’t have to pay for the VoIP service.  You are.

There are a number of ways to deploy an Asterisk box.  You can either do it behind a firewall with rules on the firewall to allow certain IP addresses to be passed to the Asterisk box or you can have Linux do the filtering.   FreePBX and RasPBX both have FAIL2BAN as an option.   Fail2Ban is a program that scans the logs of various programs like SSH and Asterisk and looks for times that an IP address is trying to crack into your site and was unsuccessful.  If so, it will add a firewall rule on Linux to prevent that IP address from accessing the box for some time frame.  By default, this is about 10 minutes.  After that, the rule is taken out and that IP address can try again.  This can retard the progress of someone trying to crack into your box, but do you really want the hassle?  What happens if they do make through?  Then you may find some rather expensive bills from your VoIP provider.   Unless you really want to be serving up SIP to the rest of the Internet, let’s lock this puppy down.

My installation has a public IP address and I rely on using Linux’s “netfilter” to firewall.  Netfilter can be configured a couple of ways.  The more traditional is to use IPTABLES.  I am not going into detail on how to use IPTABLES as there are plenty of sites that can show you that.

First off, identify the IP addresses you need to allow to connect to your Asterisk box.  This will include your VoIP provider’s session border controller (SBC) that will be protecting their SIP server.  I use VoIP.ms out of Canada and they list their servers on their wiki.  Also, identify the IP addresses that your phones will be using.  This may be on your local LAN or they may be coming from other sites.  Hopefully, these are IP addresses that won’t be changing much.  In times past, I had to run an Asterisk server that had a number of home employees, so they were coming from cable providers that would change the IP address on occasion.  Normally this wouldn’t change that often so it was manageable.  Let’s say you have the following IP  addresses…

  • VOIP provider: 1.1.1.1
  • Your local LAN’s prefix 192.168.0.0/24 assuming your Asterisk box is directly connected.
  • Your local LAN’s public IP address 2.2.2.2/32 if your Asterisk box is not connected to your LAN
  • An offsite worker: 3.3.3.3/32.

You will need to create a file like /etc/iptables/rules.v4 that looks something like:

*filter
# SIP...
-A INPUT -s 1.1.1.1 -p all -m all --dport 5060 -m comment --comment "VoIP Provider" -j ACCEPT
-A INPUT -s 1.1.1.1 -p all -m all --dport 5061 -m comment --comment "VoIP Provider" -j ACCEPT
-A INPUT -s 2.2.2.2 -p all -m all --dport 5060 -m comment --comment "Public IP of Office" -j ACCEPT
-A INPUT -s 2.2.2.2 -p all -m all --dport 5061 -m comment --comment "Public IP of Office" -j ACCEPT
-A INPUT -s 3.3.3.3 -p all -m all --dport 5060 -m comment --comment "Remote Worker" -j ACCEPT
-A INPUT -s 3.3.3.3 -p all -m all --dport 5061 -m comment --comment "Remote Worker" -j ACCEPT
-A INPUT -p all -m all --dport 5060 -m comment --comment "Drop all other SIP connections" -j DROP
-A INPUT -p all -m all --dport 5061 -m comment --comment "Drop all other SIP-TLS connections" -j DROP

In this file, you are allowing SIP access for the IP addresses listed to port 5060 (UDP) and port 5061 (TCP/TLS).  The very last two rules say, if IP addresses don’t match the above, then DROP those packets.

Secure Shell (SSH)

This is another popular service that as soon as you stand up a server, you will see bots knocking at your door.  Again, fail2ban can address this, but why would you want the exposure?  Let’s just lock it down to a couple of IP addresses with IP tables.  Updating the /etc/iptables/rules.v4 file and using the example IP addresses we used for SIP…

# SSH...
-A INPUT -s 2.2.2.2 -p tcp -m tcp --dport 22 -m comment --comment "Office Public IP Address" -j ACCEPT
-A INPUT -s 3.3.3.3 -p tcp -m tcp --dport 22 -m comment --comment "Offisite IT Provider" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -m comment --comment "Drop all other SSH TCP connections" -j DROP
Apache

This is the webserver to get into the FreePBX GUI.  Let’s just lock it down to a couple of IP addresses with IP tables.  Again, updating the /etc/iptables/rules.v4 file and using the example IP addresses we used for SIP…

# Apache...
-A INPUT -s 2.2.2.2 -p tcp -m tcp --dport 80 -m comment --comment "Office Public IP Address" -j ACCEPT
-A INPUT -s 3.3.3.3 -p tcp -m tcp --dport 80 -m comment --comment "Offisite IT Provider" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -m comment --comment "Drop all other Web TCP connections" -j DROP
-A INPUT -s 2.2.2.2 -p tcp -m tcp --dport 443 -m comment --comment "Office Public IP Address" -j ACCEPT 
-A INPUT -s 3.3.3.3 -p tcp -m tcp --dport 443 -m comment --comment "Offisite IT Provider" -j ACCEPT 
-A INPUT -p tcp -m tcp --dport 443 -m comment --comment "Drop all other Web SSL TCP connections" -j DROP
IPTABLES That is Persistent

I mentioned added these rules to /etc/iptables/rules.v4. We are going to add a bit of config to the networking section of FreePBX to have the server add these rules on reboot.

Simply you need to add the line “pre-up /sbin/iptables-restore /etc/iptables/rules.v4″ for the network interface you are using to /etc/network/interfaces.  It should look something like:

auto eth0
iface eth0 inet static
    pre-up /sbin/iptables-restore /etc/iptables/rules.v4

This tells the server to load the rules you created before it turns up the network interface.

The more recent FreePBX/Raspbx uses dhcpcd.  There is a set of scripts that get run in /lib/dhcpcd/dhcpcd-hooks/.  You can create a file called “/lib/dhcpcd/dhcpcd-hooks/05-iptables” and put “/sbin/iptables-restore /etc/iptables/rules.v4” it in.  This will add the rules before the rest of the network gets turned up.

Check Your work

Reboot your box and see if the IPTABLES loaded.  If you have fail2ban loaded, you will also see entries from it there.

root@asterisk:/home/pozar# iptables --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
fail2ban-asterisk  tcp  --  anywhere             anywhere             tcp dpt:sip
fail2ban-asterisk  udp  --  anywhere             anywhere             udp dpt:sip
fail2ban-asterisk  tcp  --  anywhere             anywhere             tcp dpt:sip-tls
fail2ban-ssh  tcp  --  anywhere             anywhere             multiport dports ssh
ACCEPT     any  --  1.1.1.1  anywhere             any dpt:sip /* voip provider */
ACCEPT     tcp  --  1.1.1.1  anywhere             tcp dpt:sip-tls /* voip provider */
ACCEPT     any  --  2.2.2.2       anywhere             any dpt:sip /* Office Public IP */
ACCEPT     tcp  --  2.2.2.2       anywhere             tcp dpt:sip-tls /* Office Public IP */
ACCEPT     any  --  3.3.3.3       anywhere             any dpt:sip /* Joe's Home Office */
ACCEPT     tcp  --  3.3.3.3       anywhere             tcp dpt:sip-tls /* Joe's Home Office */
DROP       any  --  anywhere             anywhere             any dpt:sip /* Drop all other SIP UDP connections */
DROP       tcp  --  anywhere             anywhere             tcp dpt:sip-tls /* Drop all other SIP-TLS connections */
ACCEPT     tcp  -- 2.2.2.2              anywhere             tcp dpt:ssh /* Office Public IP address */
DROP       tcp  --  anywhere             anywhere             tcp dpt:ssh /* Drop all other SSH TCP connections */

Now look at the daemons and see if they are bound to the IP addresses you are expected:

root@asterisk:/home/pozar# lsof -i | grep LIST
sshd       604        root    3u  IPv4  12384      0t0  TCP *:ssh (LISTEN)
sshd       604        root    4u  IPv6  12386      0t0  TCP *:ssh (LISTEN)
apache2   1087        root    4u  IPv6  12480      0t0  TCP *:http (LISTEN)
apache2   1087        root    6u  IPv6  12484      0t0  TCP *:https (LISTEN)
mysqld    1148       mysql   10u  IPv4  13502      0t0  TCP localhost:mysql (LISTEN)
apache2   1176    asterisk    4u  IPv6  12480      0t0  TCP *:http (LISTEN)
apache2   1176    asterisk    6u  IPv6  12484      0t0  TCP *:https (LISTEN)
[...]
exim4     1562 Debian-exim    4u  IPv4  12786      0t0  TCP localhost:smtp (LISTEN)
exim4     1562 Debian-exim    5u  IPv6  12787      0t0  TCP localhost:smtp (LISTEN)
asterisk  1802    asterisk   13u  IPv4  13291      0t0  TCP localhost:omniorb (LISTEN)
asterisk  1802    asterisk   14u  IPv4  13292      0t0  TCP localhost:8089 (LISTEN)
asterisk  1802    asterisk   15u  IPv4  13293      0t0  TCP localhost:5038 (LISTEN)
node      1993    asterisk   13u  IPv4  14461      0t0  TCP localhost:8001 (LISTEN)
node      1993    asterisk   14u  IPv4  14462      0t0  TCP localhost:8003 (LISTEN)
Conclusion

As mentioned in my disclaimer, this likely isn’t everything you need to do but it will make a dent in having your server hijacked for unwanted SIP calls to countries you never thought you called.

 

Syncing and backing up two desktops

Unison
A Bi-directional File Synchronizer.

I tend to live on two virtual locations: My MacBook Pro running OS X and my main server KUMR.LNS.COM running Linux.  I should say, for decades before I switched to Linux, I ran various versions of *BSD on KUMR.

I have a home directory on both boxes with a subfolder called “projects” that has various things I have been working on for the last 30 or so years.  I want these directories and files in both locations for access and this gives me some semblance of a backup as well.  (Of course I have other backup methods including TimeMachine and other off-site backups.)

Additionally, if I am doing development, I will tend to use my Mac but OS X does have some peculiarities in how various packages like Python, etc. get installed and running package managers like home-brew may not load what I need for an environment that will be deployed on some server so I will do that work on KUMR.  (Ya… I know about containers, and VMs.).

The challenge is how to keep things in sync with each other.  For quite a while I have been using Unison, a file synchronizer that is bi-directional and uses rsync’s rather efficient method of file transfer.  I will skip describing the rsync protocol, but you can check out the paper at https://rsync.samba.org/how-rsync-works.html for the details.

Unison is extremely efficent in working through large file collections.  I currently have about 305 GB with 215,450 files and 27,888 directories just in my “project” folder.  If I was just using “rsync”, it would take a large amount of time in walking through each file, computing the hash, seeing if the hash was the same on the other server and starting up a transfer if it isn’t.  Unison will make a similar crawl of all the files once and then keep track of files via a hash of the file and put it into an archive in ~/.unison directory.    This means that the first time I run Unison it may take a hours to crawl through all the files, but subsequent runs may take less than a minute to scan and transfer, depending on what was changed last.

If you are worried that Unison is missing anything with this system, just go into the .unison directory and delete the archives in both the local an remote servers.  Normally they start with “ar*” or “fp*” and then run unison again.

Unison also knows when you have just moved a file or folder with the same material.  If it sees a new file with the same hash and name of an existing file, it just moves that file as a “shortcut”.  A big win in just moving a folder with a large number of files or large files.

Since I have been using Unison for a while, I have had some tweaks to the Unison configuration file (~/.unison/default.prf).  I thought I would share mine here with some comments that detail the config file itself.  This, by no means is a complete set of options for Unison.  You can see all of it detailed in the manual, of which I would highly suggest reviewing.

# Unison preferences file
# Local root directory...
root = /Users/pozar/
# Remote server, ssh port number and root directory...
root = ssh://lns.com:22//home/pozar
# The program to use to copy files with. In this case rsync...
copyprog = rsync -aX --rsh='ssh -p 22' --inplace --compress
# The program to use to copy files wit that supports partial transfers...
copyprogrest = rsync -aX --rsh='ssh -p 22' --partial --inplace --compress
# maximum number of simultaneous file transfers.  Good to have more than one to really use the pipe.
maxthreads = 5
# synchronize resource forks and HFS meta-data. (true/false/default)
# I'm not interested in seeing AppleDouble files on my Linux box...
rsrc = false 
# Filename and directories to ignore...
ignore = Name projects/Mail/.imap
ignore = Name .FBCIndex
ignore = Name .FBCLockFolder 
ignore = Name {Cache*,.Trash*,.VolumeIcon.icns,.HSicon,Temporary*,.Temporary*,TheFindByContentFolder}
ignore = Name {TheVolumeSettingsFolder,.Metadata,.filler.idsff,.Spotlight,.DS_Store,.CFUserTextEncoding} 
# ignore all files with .unison somewhere in their full path
ignore = Path .unison
# Normally ignore the giant VM Harddrive files...
ignore = Name Documents/Parallels
# Don't try to copy this socket...
ignore = Name projects/gnupg/S.gpg-agent
# ignore = Name Music/iTunes
ignore = Name .lnk
ignore = Name .DS_Store
ignore = Name Thumbs.db
# Directory paths off of root to sync...
path = projects
path = Desktop
path = Documents
path = Downloads
# Keep ssh open to help with long syncs such as the initial one
sshargs = -C -o TCPKeepAlive=yes
# synchronize modification times
times = true
# Create a local log of what happened.
log = true
# Uses the modification time and length of a file as a quick check if a file has changed.
fastcheck=true

There is a handy command line argument called “-batch” which will avoid asking about what to do with each file it found to sync.  Normally it will figure things out by looking at the date stamp.  In some cases, it won’t know that to do.  You can see below an example where the permissions or times of a file may be in conflict with various Django files I have.  In this case I want to propagate this meta data to my remote server “KUMR”.  I would normally use the “>” key to tell it to go from left (Local) to right (KUMR)…

# unison
Contacting server...
Connected [//Tims-MBP.local//Users/pozar -> //kumr//home/pozar]
Looking for changes
  Waiting for changes from server                                       
Reconciling changes
local          kumr               
props    ====> props      Desktop/django/lib/python3.6/site-packages/django/contrib/contenttypes/locale/lb/LC_MESSAGES/django.mo  [] >
props    ====> props      Desktop/django/lib/python3.6/site-packages/django/contrib/contenttypes/locale/lb/LC_MESSAGES/django.po  [] >
props    <-?-> props      Desktop/django/lib/python3.6/site-packages/django/contrib/contenttypes/locale/lt/LC_MESSAGES/django.mo  [] y
Unrecognized command 'y': try again  [type '?' for help]
props    <-?-> props      Desktop/django/lib/python3.6/site-packages/django/contrib/contenttypes/locale/lt/LC_MESSAGES/django.mo  []

With the “-batch” command it avoids this so you can just script this to run from a cronjob if you like.  I normally run unison in batch mode using an alias for bash like:

alias unison='unison -batch'

Of course, if you have a situation like above, it won’t get “fixed”.  That may be fine 99.9% of the time.  Occasionally I run unison without the batch argument just to get things fully in sync.

But what happens if I have a thousand files like this?  Say for some reason, the modify times on a bunch of files got change on both sides.  Typically you would just use the UNIX “yes” command to tell it to send ‘>’ to the program with something like “yes \>”.  Unison will take this input happily until it comes to the last question where it asks if you want to propagate the changes.  Then it is looking for a ‘y’ or ‘n’.  Fortunately a ‘y’ is ignored when Unison is asking what direction to propagate the files (see above).  So you can use the bash command:

while true; do echo ">"; echo "y" ;done | unison

This sends a ‘>’ and then a ‘y’ continuously into unison.  Eventually it will ask if it should propagate the changes and it will get a ‘y’.

I should say this command should be considered a bit “dangerous” unless you are sure, the meta data and files you are propagating are what you want on the other side.

Hope this gives you some insight on this rather handy tool.  Drop me a line if you have comments or questions.

 

Librenms’ API

Librenms is a very flexible network and server monitoring and alerting system.  I have it deployed at a number of companies based on the ease of installation, the fact that it auto discovers devices, it is updated frequently (typically multiple times a week) and supports pretty much every network device you can think of.

On top of that, the alerting can be tuned to match very specific cases as the back end is MySQL so you alerting conditions can match almost anything you can write a SQL query for.  A good example would be to only alert on certain interfaces that have a specific description in them such as “TRANSIT” where the device has a hostname of “edge” and is only a 10Gbs connection (the interface name is ‘xe’).  Because you can group things by description or part of a hostname, you can just say anything with the string “edge” in the hostname should be considered a “edge router” so a group “ER” can be created for these devices.  With autodiscovery, as soon as you add a device, it will get automatically be put into the group that the rule/regular expression matches it.

One of the more interesting features is Libre’s API.  You can get pretty much any detail you want out of what Libre has collected and stored in the DB.  It will also create graphs for you on the fly.  One case I have had in the past is to create daily and weekly total bandwidth graphs for a set of specific ports on a group of switches.  The switch ports have a particular unique string I can match on so I was able to create a “group” called “peering” that included these ports over all of the switches.

I wrote this simple script called create_public_graphs.sh that asked for a graph for daily and weekly time frames.  I also added various options to the request such as don’t show the legend of interfaces and make the in and out directions all one color.  The other option is to make different colors for each interface.  We wanted a clean look so we went for a solid color.  The API doesn’t do everything you may want such as titling the graph.  This is where I use the “convert” program from imagemagick to overlay some text at the top of the graph.  You can see the final result at the SFMIX site.