This article shows you how to build a powerful, secure and easy-to-maintain web, database or application server suitable for e-commerce applications. We’ll start with a fresh install of FreeBSD, a very popular modern BSD Unix operating system.
We’ll setup a secure jail system using ezjail and then install web and database servers into different jails. For the purpose of the article, the web server is Apache with PHP, Perl, Python and Ruby, but it could easily be any other desired application environment such as Plone, Tomcat, Ruby on Rails, Java and others. For the database we’ll use MySQL although these instructions should also work well for other databases. With the web and database server now in their own jails we can monitor for security threats from the safety of the host. We’ll use two different intrusion-detection approaches: file integrity monitoring and an intrusion detection system. This server is suitable for e-commerce applications.
Layered security suggests this article should start with a brief tutorial on SSH Host Key encryption which uses keys to make access more difficult than using passwords. From a base FreeBSD system we turn off all non-essential services and turn on the pf firewall with all ports closed except SSH access on a high port and common web ports like ports 80 and 443.
File integrity is monitored simply using the tripwire-like AIDE, and finally a Snort IDS perhaps with a Snorby front-end provides an excellent detection engine with multiple rulesets monitoring traffic flows and visual displays of security threats.
Getting started
Start by installing the FreeBSD operating system with the developer tools package option, or just provision a FreeBSD server in the public cloud using your favourite cloud provider like Digital Ocean or Amazon AWS. Use the FreeBSD manual to quickly and easily get a system up and running. For most people, the default settings (including the default filesystem layout) when set to “automatic” are just fine. Don’t worry, most things that are installed in the default configuration can be changed or turned off/on at a later date, including the building of a custom kernel, so a default install of FreeBSD is all that’s required for now.
*** please note, I wrote this article where all applications are compiled on the machine using the ports system. I now use the pkg system instead as it is much faster. I plan to update this article to reflect pkg on FreeBSD 11 and 10 systems. ***
1. Install the ports tree & enable console logging
Download the latest snapshot ports tree, as this will be newer than the one on the reader’s install media. The ports tree is used to add more software to the system in a simple way, and it will custom fit that software to your specific computer by compiling it onto your specific architecture and CPU. Make sure you are ‘root’, the superuser, when performing all of the following commands, except where noted.
# portsnap fetch extract
In the future, the reader can use the command ‘portsnap fetch update’ to keep the ports tree current.
Now is a good time to install your favorite editor, such as the simple nano or pico editors if you are unfamiliar with Unix’ default editor, vi. Nano a great editor for new users, simple to use and perfect for editing configuration files. It’s a clone of the pico editor that comes with popular Unix mail readers like pine, but has a few more advanced features such as syntax highlighting and being somewhat customizable.
# cd /usr/ports/editors/nano
# make install clean
Let’s enable console logging, so we don’t ever need to a have a monitor attached to our server but can still see all console messages. First you will need to touch the file and give it restricted permissions in /var/log. Then you will edit the /etc/syslog.conf file and uncomment the console.info line. And then finally, restart syslogd:
# nano /etc/syslog.conf
# touch /var/log/console.log
# chmod 600 /var/log/console.log
# /etc/rc.d/syslogd restart
2. Install portaudit for security vulnerability checking
Portaudit checks your installed ports for security audits, and once installed it will be run automatically before you install other ports or packages in the future. It will also be run on a daily basis, with the results e-mailed to you for offsite review.
cd /usr/ports/ports-mgmt/portaudit
make install clean
Then run it for the first time, which will also install the latest database:
portaudit -Fda
FreeBSD sends out two daily e-mails to the administrator (assuming the system’s mail aliases have been setup), and one of them is a daily security e-mail. This e-mail will include the output from a daily run of portaudit, and will draw attention to any new vulnerabilities discovered in your installed ports. This is very useful! The output from portaudit will look something like this:
Checking for a current audit database:
Database created: Fri Jan 9 02:40:01 MST 2009
Checking for packages with security vulnerabilities:
0 problem(s) in your installed packages found.
3. Secure SSH
Adjust and secure SSH remote access immediately. Start with changing the default port it runs on to any arbitrary high port number to “hide” your SSH server from casual passer-bys. The most common attacks such as “brute force” password attack bots, which are automated bots that try to guess common passwords, often assume that the server is running on port 22, the default port. It doesn’t need to be. Changing the port only adds a very small extra layer of security, as any hacker with a port scanner can still find it easily, but it “reduces noise” in the /var/log/authlog file, making it easier to read, because the logs won’t be filled with the most mindless of attacks.
# cd /etc/ssh
# nano sshd_config
Search for the “Port” keyword and change the port to something else, to any number of your choice between 1,024 and 65,535. That’s a large number of ports to scan with a network scanner like nmap, so it takes time and dedication by a hacker to find your sshd port.
# Port 22
Port 12345
Having a line such as the above in your /etc/ssh/sshd_config file prevents automated bots, which assume your SSH server is running on port 22, from cluttering your logs with connection attempts. A dedicated attacker will still have no problem finding your SSH server, so we’ll have to continue to secure it in other ways.
One your have edited the file, save it and restart the sshd server:
# ps -aux|grep sshd
# kill -s HUP [sshd process number]
The next step creates a SSH private key on a typical Unix laptop or desktop, such as Mac OS X or Linux. Private keys can also be easily created on Windows 2000/XP and Vista using the popular and free PuTTY tools, located at http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html. If you’re using Windows, please follow the instructions for using the PuTTYgen tool to create an RSA key. Otherwise, for Mac and Linux users, simply setup your private key (if you haven’t already) using the commands below. In our case we’ll create an RSA key insteads of DSA key because it allows us to specify a larger key size, which presumably makes things a little more secure. Make sure this is on your *client* and not on the server we’ve been working on up to this point!
# ssh-keygen -t rsa -b 4096
# cd ~/.ssh
# cat id_rsa
# cat id_rsa.pub
Note that the use of a passphrase is entirely optional. Most people will want to use a passphrase, which require the key plus the passphrase in order to access the server. If there is no passphrase, anyone possessing the key will have full access to the server.
Now copy your private key to the server you plan to work on. Regardless of whether you created your key using Mac OS X, Linux or Windows, copy your public (.pub) file to the server securely using a scp or sftp command such as the one below:
# scp id_rsa.pub ssh-server:id_rsa_mydesktop.pub
Now login to the server and add this key to the your user’s authorized_keys file:
# mkdir .ssh
# chmod 700 .ssh
# cd .ssh
# cat ../id_rsa_mydesktop.pub >> authorized_keys
Make sure the permissions on authorized_keys are set paranoid:
# chmod 600 authorized_keys
Test that you can login now with SSH using key authentication instead of a password. If you can, then you’re ready to remove the password authentication from /etc/sshd_config, and then restart the sshd server again. That way, no one will ever be able to guess your password, because you use encryption keys (which can optionally have their own pass phrases embedded in them) instead of passwords.
At this point it’s a good idea to remove password authentication from the SSH daemon, so that only you with your authorized key will be able to login remotely. From the console, you can always still use a password. As the root superuser:
# nano /etc/ssh/sshd_config
Make sure there is a line that reads PasswordAuthentication no
4. Setup the Packet Filter firewall, “pf”
Setting up the pf firewall will block access to all ports except those we explicitly want open, such as ports 80 and 443 for Apache and Apache/SSL, and an arbitrary port of your choosing for SSH. It is not required to have a blocked.txt file, but it useful if you get repeated attacks from hackers in certain parts of the world: you can simply add their ISP’s entire CIDR address range into this blocked.txt file, and they will be blocked at the firewall.
#cd /etc
#touch blocked.txt
#mv pf.conf pf.conf.orig
#nano pf.conf
Before we create a pf.conf ruleset, we need to determine the addresses of our jails and configure pf to forward web requests to our applications in the correct jail. For the purposes of this article, the following is assumed:
- The server is on the Internet (and either has static IP addresses or uses DHCP)
- The jail will be at address 10.0.0.1
- We wish to filter traffic sent to both the jail and the host
Add the following simple ruleset (adjust interfaces as required)
#change interface below as required
ext_if="nfe0"
#change sshd port below to some other high port
sshd_port="45678"
webjail="10.0.0.1"
table <blockgeoip> persist file "/etc/blocked.txt"
scrub in
#Redirect web traffic to the jail.
rdr on $ext_if proto tcp from any to ($ext_if) port http -> $webjail port http
rdr on $ext_if proto tcp from any to ($ext_if) port https -> $webjail port https
#Allow jail traffic to nat back to anywhere
nat on $ext_if from $webjail to any -> ($ext_if)
#FILTERING RULES
block in
#block in quick inet6 all
block in log quick on $ext_if from <blockgeoip> to any
pass out keep state
pass quick on { lo }
antispoof log quick for { lo }
pass in on $ext_if proto tcp to ($ext_if) port { $sshd_port } keep state
pass out on $ext_if proto tcp to ($ext_if) port { $sshd_port } keep state
#Additional webjail rules. "pass out" not required because "nat pass" bypasses outgoing filters
pass in on $ext_if proto tcp to $webjail port { http, https } keep state
pass out on $ext_if proto tcp to $webjail port { http, https } keep state
Note that the brackets around the external interface referenced in the redirect (rdr) commands are to ensure that the pf ruleset is loaded even if the interface doesn’t yet have an IP address. In other words, if your IP address comes from DHCP, sometimes pf will start before the IP address is assigned, and pf will generate an error without this adjustment to the configuration. If you are using static IP address this won’t matter.
Finally, make sure pf is added into /etc/rc.conf so that it starts automatically. We could edit rc.conf manually, or simply issue the following command and save a step:
echo pf_enable="YES" >> /etc/rc.conf
Either reboot for the changes to take effect, or simply issue the following command to start pf for the first time without rebooting:
/etc/rc.d/pf start
Note that we are using pf’s excellent table command, which allows us to load a set of IP addresses to block from the /etc/blocked.txt file. This file can contain a list of individual IP addresses or a range of addresses in CIDR notation, one entry per line, in any combination. It is a great way to use geographic IP filtering at the firewall if, for example, your business does not require anyone from Asia to access your server but you see many web-based attacks from Asian IP addresses when you read your Apache logs.
In our ruleset above, if we had used rdr pass and nat pass instead of just rdr or nat, the redirection and nat rules would be performed without any filtering. In our case, we do want to filter requests that are sent to the jail, particularly the set of addresses in the table. The bonus is that the table filtering will work for services on both the virtualized jail and the host.
5. Setup a virtualized jail environment
A jail / CHROOT environment helps limit the impact of a server compromise by separating the web server from the main system. In the event of an incident where a web application vulnerability is exploited, we can be reasonably confident that the damage will be limited to the web server’s jail. FreeBSD’s jail capabilities extend well beyond simple CHROOT and provides virtualized environments that can interact with users on the Internet without exposing access to the host server in the event of a compromise. Later in this article, we’ll also add a file integrity monitor to watch for unexpected changes in the jail so we’ll know if a web application was compromised.
We’ll use the ezjail tool to help create our jail environment, because it greatly simplifies the process and makes running multiple jails easy and resource-friendly.
#cd /usr/ports/sysutils/ezjail/
#make install clean
#cp /usr/local/etc/ezjail.conf.sample /usr/local/etc/ezjail.conf
Edit the /usr/local/etc/ezjail.conf file and adjust as required. For our simple purposes, most of the defaults will be fine. You may want to change some of the defaults such as the jail locations, normally found in /usr/jails. Now install the base jail using the following command.
We will not install a copy of the ports tree inside the jail because later we’ll just mount the host’s port tree temporarily and just use that. Then we don’t need to worry about updating two sets of ports trees.
ezjail-admin install
This will proceed to download a fresh copy of FreeBSD and install it as a basejail. If the administrator happens to run this command on an updated FreeBSD system that has security patches that are newer than a major release such as FreeBSD-RELEASE-7.2-p4 instead of RELEASE-7.2, the above command with query the administrator as to which version to install as the basejail. In this case simply select the RELEASE-7.2, which can be easily updated to security patch p4 after install.
Make sure the jails are started automatically at boot time my adding ezjail_enable=”YES” to /etc/rc.conf:
echo ezjail_enable="YES" >> /etc/rc.conf
We will use the loopback interface for our jail. In this case, the address will be aliased to 10.0.0.1 since the jail needs to have an IP address.
#ifconfig lo1 create
#ifconfig lo1 inet 10.0.0.1 netmask 255.255.255.0
#ifconfig lo2 create
#ifconfig lo2 inet 10.0.0.2 netmask 255.255.255.0
Adjust as required depending on your unique requirements; some administrators might choose an obscure address like 10.53.27.90 but others say simpler is best. Whatever you choose, remember to add it to /etc/rc.conf so the interface is started at boot.
nano /etc/rc.conf
Add the following two lines after your primary network interface:
cloned_interfaces=“lo1 l02"
ifconfig_lo1="inet 10.0.0.1 netmask 255.255.255.0"
ifconfig_lo2="inet 10.0.0.2 netmask 255.255.255.0”
Now add the following entry to the /etc/hosts file so the host system knows where to find the jail. You can use any name you like; in our case we’ll call our jail “webjail”.
127.0.0.1 webjail
127.0.0.1 dbjail
Create the jail using the same IP address and name as you chose above.
root# ezjail-admin create webjail 10.0.0.1
root# ezjail-admin create dbjail 10.0.0.2
The jail will be created quite quickly. The jail exists and is located at /usr/jails/webjail. It now needs a basic configuration just like any fresh FreeBSD system would. The configuration file to start up the jail is located at /usr/local/etc/ezjail/webjail in case you need to customize any of its default startup settings.
Copy the network resolver configuration file to the jail, so both systems can see the Internet.
root# cp /etc/resolv.conf /usr/jails/webjail/etc
root# cp /etc/resolv.conf /usr/jails/dbjail/etc
Start the jail. On slow machines this make take some time.
/usr/local/etc/rc.d/ezjail start
Mount the host’s ports tree inside the jail in read-only mode by issuing the following commands in the shell of the host:
D=/usr/jails/webjail
rm -rf $D/usr/ports
mkdir -p $D/usr/ports
mount_nullfs -o rw /usr/ports $D/usr/ports
Optional: It can be very useful to have the ports tree mounted inside the jail automatically upon boot. First, verify that the mount worked correctly by visiting /usr/jails/webjail/usr/ports and ensuring the ports three is there. Then edit the /etc/fstab of the host server and add the following entry to make it mount automatic. Caution! Any errors in the /etc/fstab file will prevent the server from booting completely, and will drop you into single-user mode upon boot until you edit and fix the problem in the fstab. If you reach this point you can edit /etc/fstab using vi and change or delete the line below and reboot to a working system without this automatic mount.
# Device Mountpoint FStype Options Dump Pass#
/usr/ports /usr/jails/webjail/usr/ports nullfs rw 0 0
All set! Enter either jail as root. There are several ways to do this, the recommended way to enter the jail for the first time is to use the following command:
jexec webjail /bin/sh
You are now in a fresh new FreeBSD jail, much like a virtualized server within your host server. Another way to enter the jail is to use the jls command by itself to determine the number of your jail (in this case, we only have jail number 1) and then use the jexec command to enter the jail and specify a shell, such as this: jexec 1 /bin/bash However please note that you must have that binary (bash) installed into the jail before you can use it.
Personally, I prefer the Bash shell so I’ll immediately visit /usr/ports/shells/bash and then do a make install clean. And finally I’ll run chsh and change the shell intro to /usr/local/bin/bash to make it my default shell.
jls
jexec 1 /bin/sh
Many administrators will immediately try to ping an outside host from within the jail, to verify the network settings. This will not work however, as raw sockets are not allowed by default from within a jail. To temporarily enable ping to work from within the jail, issue the command sysctl security.jail.allow_raw_sockets=1 from the host environment.
Now would be an excellent time to install portaudit within the jail. Since you’re essentially in a new server, with the ports tree mounted, you can install it the same way as was done with the host server:
jailroot# cd /usr/ports/ports-mgmt/portaudit
jailroot# make install clean
It’s a good idea to add an entry in your jail’s /etc/hosts file for itself:
127.0.0.1 webjail webjail
6. Install MySQL into the jail using ports
First, make sure you are inside the jail using one of the methods outlined previously. We plan to install the latest MySQL (version 5.6 as of this writing – install whichever version suits your fancy).
First, from the host shell let’s jump into the dbjail, which shows up as jail #2 when we issue the jls command.
# jexec dbjail /bin/csh
# pkg install mysql56-server
Add the MySQL startup to rc.conf, so that it starts automatically when the system boots:
root# echo 'mysql_enable="YES"' >> /etc/rc.conf
Start the server
root# /usr/local/etc/rc.d/mysql-server start
Set the root password! This step should never be skipped!
root# /usr/local/bin/mysqladmin -u root password 'new-password1234+'
7. Install Apache version 2.4.x into the jail
Install and configure the latest Apache 2.4.x using the ports tree. There is no longer any reason to use the old Apache 1.3.x branch. It’s a very good idea to setup virtual hosting immediately as it can provide a small additional layer of security. We will also set the default web page to redirect visitors to a search engine (or other site).
Name-based virtual hosting makes web site enumeration more difficult for a hacker, as he does not know how many websites are being hosted by a given server, and thus cannot reach the domain we’re using for eCommerce without knowing more than just the IP address. If Apache’s name-based virtual hosting is setup immediately, an attacker would need to query a full DNS or use the somewhat limited DNS tools for name-based virtual hosting available on the web to try to enumerate all the virtual host names hosted at a given IP address – this is a little harder than it may first appear. Additionally, since no web traffic should be expected on the default virtual host (which receives all traffic directed to the host’s IP address but not a specific virtual host, such as a .com or .net), it will be interesting for the operator to watch the error logs of the default virtual host, and optionally log those requests to a different log file.
Start by installing Apache from the ports tree. Make sure you add the MySQL option.
# pkg install apache24
Configure Apache 2.4 as you would normally. If you are not sure what you’re doing, don’t change much without consulting the Apache documentation. For our purposes, we will make sure we enable MySQL support and disable mod_dav and mod_dav_fs, and leave the rest as defaults.
From within the jail, Apache’s main configuration file is located at /usr/local/etc/apache24/httpd.conf, and from the host system, the same configuration file would intuitively be found at /usr/jails/webjail/usr/local/etc/apache24/httpd.conf (simply add /usr/jails/webjail to access the file when not within the jail). For our purposes, we need to edit and change DirectoryIndex to one that includes PHP file that end with .php:
DirectoryIndex index.php index.html index.htm
You will also need to change Apache’s Listen and ServerName directives in httpd.conf to the jail’s private address and port, or else Apache will not be able to start:
Listen 10.0.0.1:80
ServerName 10.0.0.1:80
Check your jail’s /etc/rc.conf and make sure the hostname= is an FQDN (fully qualified domain name), else Apache will not start correctly. Alternatively, you could also set the hostname in /etc/rc.conf to anything of your choosing as long as you have a similar entry in the /etc/hosts file. For the purposes of this article, we’ll assume you need to setup your jails domain name as well.
Edit your jail’s /etc/rc.conf and add the following, putting in your domain name below:
hostname="somesite.com"
accf_http_load="YES"
apache22_enable="YES"
Start apache with apachectl start and if it works, surf to the page and verify it’s running
Reboot if you like, and Apache should start automatically.
Change the default Apache page to redirect to Google:
root# cd /usr/local/www/apache22/data
root# nano index.html
Put the following data into index.html:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML>
<HEAD>
<TITLE>Google</TITLE>
<META HTTP-EQUIV="REFRESH" CONTENT="0; URL=http://www.google.com">
</HEAD>
<BODY>
</BODY>
</HTML>
The rest of your website(s) should be name-based virtual hosts, where the configuration is located in a different directory, such as /usr/local/www/apache22/sites.
8. Install PHP 5 into the jail using ports
Installing PHP 5 inside using the ports system makes it easy to stay up-to-date. In this case, we’ll install PHP inside our jail with the Suhosin Hardened PHP security patch, which otherwise would need to be applied manually. Let’s get going!
Note that at the time of this writing, there is a warning in the PHP port to not install the Suhosin patch within a jailed environment. A workaround has been discovered which we will get to in a few moments. For now, simply install PHP and include the Suhosin patch when prompted for compile options:
root# pkg install php56
Make sure the PHP extensions are installed too! As well as the mysql extension for our connectivity to the database.
root# pkg install php56-extensions
root# pkg install mod_php56
root# pkg install php56-mysql
Copy the default php.ini file to its correct location
root# cp /usr/local/etc/php.ini-production /usr/local/etc/php.ini
Add the php type to the Apache httpd.conf file by running the following commands. In our case, we want the server to parts .html and .htm files to look for PHP code, if you do not want this then adjust the first line below accordingly.
echo "AddType application/x-httpd-php .php .html .htm" >> /usr/local/etc/apache24/httpd.conf
echo "AddType application/x-httpd-php-source .phps" >> /usr/local/etc/apache24/httpd.conf
Add a workaround that allows the Suhosin patch to work fine from within a jailed environment. Edit your jail’s /usr/local/etc/php.ini file and add the following line at the end:
suhosin.apc_bug_workaround = 1
Now restart the Apache server using apachectl restart and you should be golden.
At this point the server’s ability to server up dynamic web pages and content is complete. The reader can proceed to load his or her web pages into the server, typically by adding virtual hosts into the Apache configuration file. Even if only a singe domain or site is going to be hosted, it can be useful to install the site (and all other sites – many are possible on a single server) using name-based virtual hosts.
A sample name-based virtual host configuration is listed below. It can be added to the configuration file found in /usr/local/etc/apache22/extra/httpd-vhosts.conf, which is read automatically when Apache is started.
#
# newdomain.com
#
<VirtualHost *:80>
ServerAdmin me@myemail.com
DocumentRoot "/usr/local/www/apache22/sites/newdomain.com"
ServerName newdomain.com
ServerAlias www.newdomain.com
<Directory />
AllowOverride None
Order deny,allow
Deny from all
</Directory>
<Directory "/usr/local/www/apache22/sites/newdomain.com">
Options Indexes FollowSymLinks
AllowOverride Options FileInfo
Order allow,deny
Allow from all
</Directory>
</VirtualHost>
Once this entry has been made into the configuration file, the user will need to ensure that the directory exists before Apache is started, else it will not be able to find the web pages to serve up.
root# mkdir /usr/local/www/apache22/sites/newdomain.com
At this point we can restart Apache again (with apachectl restart) and start serving up dynamic, database-driven web pages with MySQL and PHP inside an Apache name-based host on a SSL-capable virtualized jail server with FreeBSD. The server could safely deliver e-commerce applications, but could still be hardened for security in a number of additional ways. The next steps are simple forays into additional defense-in-depth security.
9. Install the ModSecurity web application firewall into the jail
Now that the server is fully functional, let’s start to secure it. We’ll add ModSecurity which looks specifically for web application attacks and it does a pretty good job of finding and blocking the most common attacks.
cd /usr/ports/www/mod_security
make install clean
chmod 755 /usr/local/libexec/apache22/mod_security2.so
Ensure that mod_unique is already in your /usr/local/etc/apache22/httpd.conf configuration file, and add it if not:
LoadModule unique_id_module libexec/apache22/mod_unique_id.so
Also add an entry into httpd.conf so that ModSecurity is also loaded:
LoadModule security2_module libexec/apache22/mod_security2.so
At the time of this writing, the port installed the core ruleset automatically into /usr/local/etc/apache22/Includes/mod_security2/. Additionally, logging is done to /var/log/httpd-modsec2*.log which makes it easy to review ModSecurity-specific logs at a glance, right next to the default location of regular Apache log files in /var/log.
Note that the default configuration of ModSecurity is to detect attacks only; ModSecurity will not automatically block web-based attacks until it is told to do so. The administrator is advised to review the Apache error logs located in /var/log/httpd-error.log for some time to monitor the types of errors generated, make changes to the ModSecurity configuration as required, and ensure legitimate access to the web application will not be blocked. Once the administrator is confident of his configuration, he can change ModSecurity to “On” instead of “DetectOnly,” the default setting.
For now, we’ll restart Apache so the changes are put into effect:
apachectl restart
10. Maintain integrity of the filesystem in the jail
We will install a file integrity monitor to watch for unexpected changes on our new server. In the unlikely event that a PHP/MySQL web application is hacked and the server is compromised, we need to know exactly what files changed. We will install AIDE (Advanced Intrusion Detection Environment), which an open-source file integrity monitor similar to Tripwire, a commercial product. If any of your web applications are compromised (often because the operator failed to update those applications with the latest security patches), unwanted files may appear on the system. A good operator should read logs daily, particularly the output from the integrity monitor.
Use the ports tree to get AIDE installed quickly and easily.
root# cd /usr/ports/security/aide
root# make install clean
Edit AIDE’s configuration file, located in /usr/local/etc, and add your web directory, home directories, and so on. Below are a few example entries that could be added, you will want to add more entries based on what you are trying to monitor:
/home/kel R-tiger-rmd160-sha1
/home/kel/.bash_history L
/root/.bash_history L
/usr/share/man L
/usr/share/openssl/man L
/usr/local/www R-tiger-rmd160-sha1
Since AIDE does not run as a daemon and does not have e-mail or cron capabilities built-in, we have to find another way to run AIDE automatically on a regular basis and get the results sent offsite for review by a human being. In our case, to keep things simple we’ll use the built-in FreeBSD periodic scripts to do this on a daily basis. The easiest way is to simply create an /etc/daily.local file that will get run each day, with the contents automatically e-mailed to the administrator offsite (assuming the administrator has put his e-mail address in the mail aliases files, as described below). Issuing the following command will create the daily.local file if it doesn’t exist and put one command in it:
echo "/usr/local/bin/aide --check" >> /etc/daily.local
Please note that many ISPs block the ability for your server to send e-mails directly – they often want you to use their SMTP mail server as a gateway to send mail, so they can catch spammers. If a day or two has passed after building your server and you have not received any of the automated daily e-mails FreeBSD servers generate by default, your ISP is likely blocking outgoing mail sent directly from your server.
Unfortunately Sendmail, an age-old mailer well known on the Internet, is rather difficult to configure to use an outgoing gateway without the user having prior experience of Sendmail, so in the next
11. Change the mailer to something easier to configure (optional)
Install postfix and configure the SMTP server to use a local gateway.
# cd /usr/ports/mail/postfix
# make install clean
Edit the rc.conf file to enable postfix at startup, and add the following:
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
postfix_enable="YES"
Edit the /etc/defaults/periodic.conf file so that it reflects the following:
daily_clean_hoststat_enable="NO"
daily_status_mail_rejects_enable="NO"
daily_status_include_submit_mailq="NO"
daily_submit_queuerun="NO"
Finally, determine if you can send mail directly on the Internet, or if you need to relay your mail through a 3rd party SMTP gateway. The latter is common practice among many cable and DSL broadband ISPs, so if this applies to you you’ll need to edit the /usr/local/etc/postfix/main.cf file to reflect the SMTP server. Search for ‘relayhost’ and then add your relay address nearby. In my case, my ISP is Shaw… please use your own ISP’s mail relay and not mine below, otherwise it won’t work.
relayhost = [shawmail.cg.shawcable.net]
Further, if your ISP requires SMTP authentication and uses SSL/TLS then you will need to perform one more additional steps, as detailed in the next section
Regardless of which method you use to send mail, for all situations you will need to configure your system’s mail aliases file, so that the system knows where to send mail to.
Edit the aliases file located at /etc/mail/aliases. For some unknown reason, postfix ignores the aliases file created at /usr/local/etc/postfix/aliases which gets created by some versions of postfix, so please use the /etc/mail/aliases file that sendmail would otherwise use and you will have no problems. Ensure that a valid external e-mail address is set. This ensures that you get the daily logs generated by FreeBSD. Make sure the newaliases command is run manually after the file has been changed, or else your changes won’t get loaded.
root# nano /etc/mail/aliases
root# newaliases
Finally, start the server to make the configuration funcational.
root# postfix start
In the future, if you change postfix’ configuration you can simply issue the command postfix reload to apply the changes without needing to reboot.
- Enable SMTP via SSL/TLS (optional)
In our case, our server is hosted in the Cloud and we need to use a third-part relay (Gmail) that uses SSL/TLS. So we will need to ensure that Postfix was compiled with SASL2 and SSL/TLS enabled. If it wasn’t, you can simply delete the options file located in /var/db/posts/postfix, which is used to determine compile options when to build or rebuild a package. Delete that file and then rebuild postfix using the command portmaster postfix, and the port will ask you again using a text-based GUI which options you’ll like to compile in:
root# cd /var/db/ports/postfix
root# delete options
root# portmaster postfix
We have just a bit more work to do, by adding the following into /usr/local/etc/postfix/main.cf:
relayhost = [smtp.gmail.com]:587
smtp_sasl_auth_enable = yes
smtp_sasl_security_options = noanonymous
smtp_sasl_password_maps = hash:/usr/local/etc/postfix/sasl_passwd
smtp_sasl_mechanism_filter = plain, login
smtp_use_tls = yes
data_directory = /var/db/postfix
Then you need to create a /usr/local/etc/postfix/sasl_passwd file that contains something like this:
[smtp.gmail.com]:587 myaccount@gmail.com:mypassword
Finally, you will need to issue the following two commands to enable everything:
# postmap /usr/local/etc/postfix/sasl_passwd
# postfix reload
You should now send a test email to yourself and verify that everything works.
12. General configuration & E-Mailing of Daily Logs
Now would be a good time to ensure usernames like root and kel have the correct “names” associated to this server. Take a look at /etc/passwd to see if it’s correct, and if not issue something like:
pw usermod root -c 'Myphatserver &,,,'
pw usermod kel -c ‘Jim Nippholes,,,’
Of course this will only be useful if you correctly setup your mail aliases file earlier in this document, so that the system knows where to send your mail to.
13. Keeping Time
Time synchronization is an important part of a server’s function. As you build our your server and rely on applications, shell scripts and batch processing that gets transferred between this server and other servers, you will see how important time synchronization is. FreeBSD ships with the ntpd daemon for keeping your clock in-sync with time servers on the Internet. It is already installed and just needs to be configured.
The main configuration file for ntpd is found in /etc/ntp.conf. The defaults that were introduction in FreeBSD 8.x will be fine for most users, however one recommendation would be to uncomment the line that reads restrict default ignore so that no external connections will be accepted by ntpd.
Alternatively, if your version of FreeBSD doesn’t have a default /etc/ntp.conf file (which is common in earlier versions of FreeBSD, such as 6.x and 7.x), you can create a simple one using the entries below:
server 0.freebsd.pool.ntp.org iburst maxpoll 9
server 1.freebsd.pool.ntp.org iburst maxpoll 9
server 2.freebsd.pool.ntp.org iburst maxpoll 9
restrict default ignore
To make the server startup automatically, let’s add an entry to our rc.conf file.
echo ntpd_enable="YES" >> /etc/rc.conf
Now let’s run the server for the first time, by calling the daemon itself:
ntpd
14. Tell other servers we’re alive (optional)
This step looks at a rudimentary way of tracking the location (IP address) and status of all your servers from one location. We can do this by writing a script in PHP to keep a database of this information, such that we can glance at a single page and see what servers are alive and doing fine.
Basically, w’ere going to install the text-based web browser Lynx and then create a cron job to ping one of our own websites to notify we’re alive. The contents of the “alive.php” script is outside the scope of this article… it could be just about anything you want it to be.
# cd /usr/ports/www/lynx
# make install clean
Try lynx and see if it works:
/usr/local/bin/lynx -dump "http://www.mywebsite.complete/serverdb/alive.php"
Add the following to /etc/rc.local using the nano editor
#ping another server to say we're alive!
echo "ping www.mywebsite to say we're alive..."
/usr/local/bin/lynx -dump "http://www.mywebsite.com/serverdb/alive.php"
Add a crontab entry using the crontab -e command. Important Note: use a normal user’s crontab, not root’s!
# ping insurancelink.ca to say we're alive!!
10 * * * * /usr/local/bin/lynx -dump http://www.myserver.com/servers/saturn.php > /dev/null
15. Staying Up-To-Date
An important part of keeping a system secure is to keep on top of new security vulnerabilities and install patches as soon as they become available. FreeBSD shines is this area with a number of tools that make this easy. However the first step is to ensure you on are the security-advisories@freebsd.org mailing list, which will notify you when new system patches are available.
To keep the system kernel and userland updated, there are a number of options. The easiest of these is the freebsd-update command. To fetch new updates, use the freebsd-update fetch command, followed by the freebsd-update install command. If any parts of the kernel were updated, you will need to reboot to install the kernel, otherwise you can continue to use the system. This command provides binary updates to the kernel and userland environment, but it also updates the source code in your /usr/src tree.
Since the jail environment uses the host’s kernel, it’s very important to keep the FreeBSD kernel up-to-date with security patches. The freebsd-update command while inside the host machine will keep the kernel updated plus the host’s userland, but it will not update the jails userland.
To update the jail’s user binaries, we need to issue the command ezjail-admin update -u. This will update the base-jail using freebsd-update. So, when you receive an announcement of a new vulnerability in the FreeBSD kernel or user binaries, you’d perform a simple series of steps such as the following:
freebsd-update fetch
freebsd-update install
ezjail-admin update -u
Then your host and all jails will be updated with the latest patches.
To keep the ports tree updated, we will periodically run the command portsnap fetch update. This will update the ports tree in the host environment as well as the jail environment since our ports tree is mounted inside the jail as well.
We can easily check for known vulnerabilities in our installed ports using the portaudit -Fa command. Simply installing the tool makes it run automatically on a daily basis, as part of the /etc/periodic set of scripts, and this output will be e-mailed daily to the root user. So read the daily emails sent to you from this server, and it will indicate when it finds known vulnerabilities in installed software – a very cool security feature that covers many thousands of apps.
To check our installed ports versus the latest ports tree, we’ll use the pkg_version -v command to compare what is installed versus the latest version. There are a number of ways to upgrade installed ports, one of the easiest and most hassle-free is the portmaster utility, which can be found in /usr/ports/ports-mgmt:
cd /usr/ports/ports-mgmt/portmaster
make install clean
Then when you need to update a port, such as portaudit-0.5.14, you simply issue the portmaster portaudit command and it will be automatically compiled and upgrade to the latest version. Of course, use portmaster [installed port] to upgrade whichever port needs upgrading.
- Conclusion
Comments, feedback, and suggestions for improvement are welcome!
This document was authored by Kelly Martin, kelly@kelly3.com.
