Secure FreeBSD web application server and e-commerce system

This article will setup a FreeBSD web application server secured with two jails, one each for Apache and MySQL, and two intrusion-detection systems running on the host system. Together this system provides a fast and scalable web application server with a focus on security and easy updates with simple maintenance examples.

It will be an update to my previous article, Secure FreeBSD web server with Apache, PHP and MySQL in secured jails with file integrity monitoring.

I just have to find the time to write it. Stay tuned.

Why FreeBSD is a great secure server

It’s important to keep computer systems updated and this process should be as simple as possible to stay safe and secure. A server that runs web applications should be simple to build, easy to patch, update and upgrade, and it should have a great manual. Then being able to monitor problems remotely and be alerted to issues helps the website operator ensure that everything is running correctly. E-commerce systems require high security and are subject to PCI-DSS Compliance since they process financial transactions and should be audited periodically including self-audits by the operator to ensure there is no data or financial leakage.

We’ll take a freshly installed FreeBSD operating system (versions 9 and 10 were used with this article) and build a jail system where we’ll install a secure, fully functional server capable of processing credit card transactions online quickly and simply, keeping hackers out at the same time.

With the popularity of complex web-based applications, a common problem for administrators is how to keep web servers secure without limiting the many types of web applications and scripts that can be installed. Too often, software is installed by users will contains security vulnerabilities that will be found  today or in the future, vulnerabilities that can and will allow someone to compromise a system. It’s not a matter of if, it’s just a questions of when will your server be exploited? Administrators require layered security to severely limit the breach and have notification systems that monitor for any unexpected activity on the server. There are thousands of automated scripts out there written by hackers who search the Internet for vulnerable software that is easy to exploit. A good administrator needs to minimize this risk and the risk that all web applications provide once they are installed, and use planning, architecture, secure builds and operational tools that run, secure, monitor and report on the state of a web server.

It is not very difficult to get a basic FreeBSD system running. On modern hardware, a default install from removable media can take as little as five or ten minutes. But a base system is not a adequate or secure enough for today’s business and e-commerce needs. In this article, we’ll take the most common web server configuration, Apache/PHP/MySQL, and put it in a CHROOT/jail virtualization environment, add strong firewall, a web application firewall, a file integrity monitor to notify us of any changes to the system, and we’ll strong public-key authentication for remote access.

Secure FreeBSD web server with Apache, PHP and MySQL in secured jails with file integrity monitoring

This article shows you how to build a powerful, secure and easy-to-maintain web, database or application server suitable for e-commerce applications. We’ll start with a fresh install of FreeBSD, a very popular modern BSD Unix operating system.

We’ll setup a secure jail system using ezjail and then install web and database servers into different jails. For the purpose of the article, the web server is Apache with PHP, Perl, Python and Ruby, but it could easily be any other desired application environment such as Plone, Tomcat, Ruby on Rails, Java and others. For the database we’ll use MySQL although these instructions should also work well for other databases. With the web and database server now in their own jails we can monitor for security threats from the safety of the host. We’ll use two different intrusion-detection approaches: file integrity monitoring and an intrusion detection system. This server is suitable for e-commerce applications.

Layered security suggests this article should start with a brief tutorial on SSH Host Key encryption which uses keys to make access more difficult than using passwords. From a base FreeBSD system we turn off all non-essential services and turn on the pf firewall with all ports closed except SSH access on a high port and common web ports like ports 80 and 443.

File integrity is monitored simply using the tripwire-like AIDE, and finally a Snort IDS perhaps with a Snorby front-end provides an excellent detection engine with multiple rulesets monitoring traffic flows and visual displays of security threats.

Getting started

Start by installing the FreeBSD operating system with the developer tools package option, or just provision a FreeBSD server in the public cloud using your favourite cloud provider like Digital Ocean or Amazon AWS. Use the FreeBSD manual to quickly and easily get a system up and running. For most people, the default settings (including the default filesystem layout) when set to “automatic” are just fine. Don’t worry, most things that are installed in the default configuration can be changed or turned off/on at a later date, including the building of a custom kernel, so a default install of FreeBSD is all that’s required for now.

*** please note, I wrote this article where all applications are compiled on the machine using the ports system. I now use the pkg system instead as it is much faster. I plan to update this article to reflect pkg on FreeBSD 11 and 10 systems. ***

1. Install the ports tree & enable console logging

Download the latest snapshot ports tree, as this will be newer than the one on the reader’s install media. The ports tree is used to add more software to the system in a simple way, and it will custom fit that software to your specific computer by compiling it onto your specific architecture and CPU. Make sure you are ‘root’, the superuser, when performing all of the following commands, except where noted.

# portsnap  fetch extract

In the future, the reader can use the command ‘portsnap fetch update’ to keep the ports tree current.

Now is a good time to install your favorite editor, such as the simple nano or pico editors if you are unfamiliar with Unix’ default editor, vi. Nano a great editor for new users, simple to use and perfect for editing configuration files. It’s a clone of the pico editor that comes with popular Unix mail readers like pine, but has a few more advanced features such as syntax highlighting and being somewhat customizable.

# cd /usr/ports/editors/nano
# make install clean

Let’s enable console logging, so we don’t ever need to a have a monitor attached to our server but can still see all console messages. First you will need to touch the file and give it restricted permissions in /var/log. Then you will edit the /etc/syslog.conf file and uncomment the console.info line. And then finally, restart syslogd:

# nano /etc/syslog.conf
# touch /var/log/console.log
# chmod 600 /var/log/console.log
# /etc/rc.d/syslogd restart

2. Install portaudit for security vulnerability checking

Portaudit checks your installed ports for security audits, and once installed it will be run automatically before you install other ports or packages in the future. It will also be run on a daily basis, with the results e-mailed to you for offsite review.

cd /usr/ports/ports-mgmt/portaudit
make install clean

Then run it for the first time, which will also install the latest database:

portaudit -Fda

FreeBSD sends out two daily e-mails to the administrator (assuming the system’s mail aliases have been setup), and one of them is a daily security e-mail. This e-mail will include the output from a daily run of portaudit, and will draw attention to any new vulnerabilities discovered in your installed ports. This is very useful! The output from portaudit will look something like this:

Checking for a current audit database:

Database created: Fri Jan  9 02:40:01 MST 2009
Checking for packages with security vulnerabilities:
0 problem(s) in your installed packages found.

3. Secure SSH

Adjust and secure SSH remote access immediately. Start with changing the default port it runs on to any arbitrary high port number to “hide” your SSH server from casual passer-bys. The most common attacks such as “brute force” password attack bots, which are automated bots that try to guess common passwords, often assume that the server is running on port 22, the default port. It doesn’t need to be. Changing the port only adds a very small extra layer of security, as any hacker with a port scanner can still find it easily, but it “reduces noise” in the /var/log/authlog file, making it easier to read, because the logs won’t be filled with the most mindless of attacks.

# cd /etc/ssh
# nano sshd_config

Search for the “Port” keyword and change the port to something else, to any number of your choice between 1,024 and 65,535. That’s a large number of ports to scan with a network scanner like nmap, so it takes time and dedication by a hacker to find your sshd port.

# Port 22
Port 12345

Having a line such as the above in your /etc/ssh/sshd_config file prevents automated bots, which assume your SSH server is running on port 22, from cluttering your logs with connection attempts. A dedicated attacker will still have no problem finding your SSH server, so we’ll have to continue to secure it in other ways.

One your have edited the file, save it and restart the sshd server:

# ps -aux|grep sshd
# kill -s HUP [sshd process number]

The next step creates a SSH private key on a typical Unix laptop or desktop, such as Mac OS X or Linux. Private keys can also be easily created on Windows 2000/XP and Vista using the popular and free PuTTY tools, located at http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html. If you’re using Windows, please follow the instructions for using the PuTTYgen tool to create an RSA key. Otherwise, for Mac and Linux users, simply setup your private key (if you haven’t already) using the commands below. In our case we’ll create an RSA key insteads of DSA key because it allows us to specify a larger key size, which presumably makes things a little more secure. Make sure this is on your *client* and not on the server we’ve been working on up to this point!

# ssh-keygen -t rsa -b 4096
# cd ~/.ssh
# cat id_rsa
# cat id_rsa.pub

Note that the use of a passphrase is entirely optional. Most people will want to use a passphrase, which require the key plus the passphrase in order to access the server. If there is no passphrase, anyone possessing the key will have full access to the server.

Now copy your private key to the server you plan to work on. Regardless of whether you created your key using Mac OS X, Linux or Windows, copy your public (.pub) file to the server securely using a scp or sftp command such as the one below:

# scp id_rsa.pub ssh-server:id_rsa_mydesktop.pub

Now login to the server and add this key to the your user’s authorized_keys file:

# mkdir .ssh
# chmod 700 .ssh
# cd .ssh
# cat ../id_rsa_mydesktop.pub >> authorized_keys

Make sure the permissions on authorized_keys are set paranoid:

# chmod 600 authorized_keys

Test that you can login now with SSH using key authentication instead of a password. If you can, then you’re ready to remove the password authentication from /etc/sshd_config, and then restart the sshd server again. That way, no one will ever be able to guess your password, because you use encryption keys (which can optionally have their own pass phrases embedded in them) instead of passwords.

At this point it’s a good idea to remove password authentication from the SSH daemon, so that only you with your authorized key will be able to login remotely. From the console, you can always still use a password. As the root superuser:

# nano /etc/ssh/sshd_config

Make sure there is a line that reads PasswordAuthentication no

4. Setup the Packet Filter firewall, “pf”

Setting up the pf firewall will block access to all ports except those we explicitly want open, such as ports 80 and 443 for Apache and Apache/SSL, and an arbitrary port of your choosing for SSH. It is not required to have a blocked.txt file, but it useful if you get repeated attacks from hackers in certain parts of the world: you can simply add their ISP’s entire CIDR address range into this blocked.txt file, and they will be blocked at the firewall.

#cd /etc
#touch blocked.txt
#mv pf.conf pf.conf.orig
#nano pf.conf

Before we create a pf.conf ruleset, we need to determine the addresses of our jails and configure pf to forward web requests to our applications in the correct jail. For the purposes of this article, the following is assumed:

  • The server is on the Internet (and either has static IP addresses or uses DHCP)
  • The jail will be at address 10.0.0.1
  • We wish to filter traffic sent to both the jail and the host

Add the following simple ruleset (adjust interfaces as required)

#change interface below as required
ext_if="nfe0"
#change sshd port below to some other high port
sshd_port="45678"
webjail="10.0.0.1"

table <blockgeoip> persist file "/etc/blocked.txt"

scrub in

#Redirect web traffic to the jail.
rdr on $ext_if proto tcp from any to ($ext_if) port http -> $webjail port http
rdr on $ext_if proto tcp from any to ($ext_if) port https -> $webjail port https

#Allow jail traffic to nat back to anywhere
nat on $ext_if from $webjail to any -> ($ext_if)

#FILTERING RULES
block in
#block in quick inet6 all
block in log quick on $ext_if from <blockgeoip> to any
pass out keep state
pass quick on { lo }
antispoof log quick for { lo }
pass in on $ext_if proto tcp to ($ext_if) port { $sshd_port } keep state
pass out on $ext_if proto tcp to ($ext_if) port { $sshd_port } keep state

#Additional webjail rules. "pass out" not required because "nat pass" bypasses outgoing filters
pass in on $ext_if proto tcp to $webjail port { http, https } keep state
pass out on $ext_if proto tcp to $webjail port { http, https } keep state

Note that the brackets around the external interface referenced in the redirect (rdr) commands are to ensure that the pf ruleset is loaded even if the interface doesn’t yet have an IP address. In other words, if your IP address comes from DHCP, sometimes pf will start before the IP address is assigned, and pf will generate an error without this adjustment to the configuration. If you are using static IP address this won’t matter.

Finally, make sure pf is added into /etc/rc.conf so that it starts automatically. We could edit rc.conf manually, or simply issue the following command and save a step:

echo pf_enable="YES" >> /etc/rc.conf

Either reboot for the changes to take effect, or simply issue the following command to start pf for the first time without rebooting:

/etc/rc.d/pf start

Note that we are using pf’s excellent table command, which allows us to load a set of IP addresses to block from the /etc/blocked.txt file. This file can contain a list of individual IP addresses or a range of addresses in CIDR notation, one entry per line, in any combination. It is a great way to use geographic IP filtering at the firewall if, for example, your business does not require anyone from Asia to access your server but you see many web-based attacks from Asian IP addresses when you read your Apache logs.

In our ruleset above, if we had used rdr pass and nat pass instead of just rdr or nat, the redirection and nat rules would be performed without any filtering. In our case, we do want to filter requests that are sent to the jail, particularly the set of addresses in the table. The bonus is that the table filtering will work for services on both the virtualized jail and the host.

5. Setup a virtualized jail environment

A jail / CHROOT environment helps limit the impact of a server compromise by separating the web server from the main system. In the event of an incident where a web application vulnerability is exploited, we can be reasonably confident that the damage will be limited to the web server’s jail. FreeBSD’s jail capabilities extend well beyond simple CHROOT and provides virtualized environments that can interact with users on the Internet without exposing access to the host server in the event of a compromise. Later in this article, we’ll also add a file integrity monitor to watch for unexpected changes in the jail so we’ll know if a web application was compromised.

We’ll use the ezjail tool to help create our jail environment, because it greatly simplifies the process and makes running multiple jails easy and resource-friendly.

#cd /usr/ports/sysutils/ezjail/
#make install clean
#cp /usr/local/etc/ezjail.conf.sample /usr/local/etc/ezjail.conf

Edit the /usr/local/etc/ezjail.conf file and adjust as required. For our simple purposes, most of the defaults will be fine. You may want to change some of the defaults such as the jail locations, normally found in /usr/jails. Now install the base jail using the following command.

We will not install a copy of the ports tree inside the jail because later we’ll just mount the host’s port tree temporarily and just use that. Then we don’t need to worry about updating two sets of ports trees.

ezjail-admin install

This will proceed to download a fresh copy of FreeBSD and install it as a basejail. If the administrator happens to run this command on an updated FreeBSD system that has security patches that are newer than a major release such as FreeBSD-RELEASE-7.2-p4 instead of RELEASE-7.2, the above command with query the administrator as to which version to install as the basejail. In this case simply select the RELEASE-7.2, which can be easily updated to security patch p4 after install.

Make sure the jails are started automatically at boot time my adding  ezjail_enable=”YES” to /etc/rc.conf:

echo ezjail_enable="YES" >> /etc/rc.conf

We will use the loopback interface for our jail. In this case, the address will be aliased to 10.0.0.1 since the jail needs to have an IP address.

#ifconfig lo1 create
#ifconfig lo1 inet 10.0.0.1 netmask 255.255.255.0
#ifconfig lo2 create
#ifconfig lo2 inet 10.0.0.2 netmask 255.255.255.0

Adjust as required depending on your unique requirements; some administrators might choose an obscure address like 10.53.27.90 but others say simpler is best. Whatever you choose, remember to add it to /etc/rc.conf so the interface is started at boot.

nano /etc/rc.conf

Add the following two lines after your primary network interface:

cloned_interfaces=“lo1 l02"
ifconfig_lo1="inet 10.0.0.1 netmask 255.255.255.0"
ifconfig_lo2="inet 10.0.0.2 netmask 255.255.255.0”

Now add the following entry to the /etc/hosts file so the host system knows where to find the jail. You can use any name you like; in our case we’ll call our jail “webjail”.

127.0.0.1 webjail
127.0.0.1 dbjail

Create the jail using the same IP address and name as you chose above.

root# ezjail-admin create webjail 10.0.0.1
root# ezjail-admin create dbjail 10.0.0.2

The jail will be created quite quickly. The jail exists and is located at /usr/jails/webjail. It now needs a basic configuration just like any fresh FreeBSD system would. The configuration file to start up the jail is located at /usr/local/etc/ezjail/webjail in case you need to customize any of its default startup settings.

Copy the network resolver configuration file to the jail, so both systems can see the Internet.

root# cp /etc/resolv.conf /usr/jails/webjail/etc
root# cp /etc/resolv.conf /usr/jails/dbjail/etc

Start the jail. On slow machines this make take some time.

/usr/local/etc/rc.d/ezjail start

Mount the host’s ports tree inside the jail in read-only mode by issuing the following commands in the shell of the host:

D=/usr/jails/webjail
rm -rf $D/usr/ports
mkdir -p $D/usr/ports
mount_nullfs -o rw /usr/ports $D/usr/ports

Optional: It can be very useful to have the ports tree mounted inside the jail automatically upon boot. First, verify that the mount worked correctly by visiting /usr/jails/webjail/usr/ports and ensuring the ports three is there. Then edit the /etc/fstab of the host server and add the following entry to make it mount automatic. Caution! Any errors in the /etc/fstab file will prevent the server from booting completely, and will drop you into single-user mode upon boot until you edit and fix the problem in the fstab. If you reach this point you can edit /etc/fstab using vi and change or delete the line below and reboot to a working system without this automatic mount.

# Device    Mountpoint                    FStype  Options  Dump    Pass#
/usr/ports /usr/jails/webjail/usr/ports  nullfs rw       0       0

All set! Enter either jail as root. There are several ways to do this, the recommended way to enter the jail for the first time is to use the following command:

jexec webjail /bin/sh

You are now in a fresh new FreeBSD jail, much like a virtualized server within your host server. Another way to enter the jail is to use the jls command by itself to determine the number of your jail (in this case, we only have jail number 1) and then use the jexec command to enter the jail and specify a shell, such as this: jexec 1 /bin/bash However please note that you must have that binary (bash) installed into the jail before you can use it.

Personally, I prefer the Bash shell so I’ll immediately visit /usr/ports/shells/bash and then do a make install clean. And finally I’ll run chsh and change the shell intro to /usr/local/bin/bash to make it my default shell.

jls
jexec 1 /bin/sh

Many administrators will immediately try to ping an outside host from within the jail, to verify the network settings. This will not work however, as raw sockets are not allowed by default from within a jail. To temporarily enable ping to work from within the jail, issue the command sysctl security.jail.allow_raw_sockets=1 from the host environment.

Now would be an excellent time to install portaudit within the jail. Since you’re essentially in a new server, with the ports tree mounted, you can install it the same way as was done with the host server:

jailroot# cd /usr/ports/ports-mgmt/portaudit
jailroot# make install clean

It’s a good idea to add an entry in your jail’s /etc/hosts file for itself:

127.0.0.1 webjail webjail

6. Install MySQL into the jail using ports

First, make sure you are inside the jail using one of the methods outlined previously. We plan to install the latest MySQL (version 5.6 as of this writing – install whichever version suits your fancy).

First, from the host shell let’s jump into the dbjail, which shows up as jail #2 when we issue the jls command.

# jexec dbjail /bin/csh
# pkg install mysql56-server

Add the MySQL startup to rc.conf, so that it starts automatically when the system boots:

root# echo 'mysql_enable="YES"' >> /etc/rc.conf

Start the server

root# /usr/local/etc/rc.d/mysql-server start

Set the root password! This step should never be skipped!

root# /usr/local/bin/mysqladmin -u root password 'new-password1234+'

7. Install Apache version 2.4.x into the jail

Install and configure the latest Apache 2.4.x using the ports tree. There is no longer any reason to use the old Apache 1.3.x branch. It’s a very good idea to setup virtual hosting immediately as it can provide a small additional layer of security. We will also set the default web page to redirect visitors to a search engine (or other site).

Name-based virtual hosting makes web site enumeration more difficult for a hacker, as he does not know how many websites are being hosted by a given server, and thus cannot reach the domain we’re using for eCommerce without knowing more than just the IP address. If Apache’s name-based virtual hosting is setup immediately, an attacker would need to query a full DNS or use the somewhat limited DNS tools for name-based virtual hosting available on the web to try to enumerate all the virtual host names hosted at a given IP address – this is a little harder than it may first appear. Additionally, since no web traffic should be expected on the default virtual host (which receives all traffic directed to the host’s IP address but not a specific virtual host, such as a .com or .net), it will be interesting for the operator to watch the error logs of the default virtual host, and optionally log those requests to a different log file.

Start by installing Apache from the ports tree. Make sure you add the MySQL option.

# pkg install apache24

Configure Apache 2.4 as you would normally. If you are not sure what you’re doing, don’t change much without consulting the Apache documentation. For our purposes, we will make sure we enable MySQL support and disable mod_dav and mod_dav_fs, and leave the rest as defaults.

From within the jail, Apache’s main configuration file is located at /usr/local/etc/apache24/httpd.conf, and from the host system, the same configuration file would intuitively be found at /usr/jails/webjail/usr/local/etc/apache24/httpd.conf (simply add /usr/jails/webjail to access the file when not within the jail). For our purposes, we need to edit and change DirectoryIndex to one that includes PHP file that end with .php:

DirectoryIndex index.php index.html index.htm

You will also need to change Apache’s Listen and ServerName directives in httpd.conf to the jail’s private address and port, or else Apache will not be able to start:

Listen 10.0.0.1:80
ServerName 10.0.0.1:80

Check your jail’s /etc/rc.conf and make sure the hostname= is an FQDN (fully qualified domain name), else Apache will not start correctly. Alternatively, you could also set the hostname in /etc/rc.conf to anything of your choosing as long as you have a similar entry in the /etc/hosts file. For the purposes of this article, we’ll assume you need to setup your jails domain name as well.

Edit your jail’s /etc/rc.conf and add the following, putting in your domain name below:

hostname="somesite.com"
accf_http_load="YES"
apache22_enable="YES"

Start apache with apachectl start and if it works, surf to the page and verify it’s running

Reboot if you like, and Apache should start automatically.

Change the default Apache page to redirect to Google:

root# cd /usr/local/www/apache22/data
root# nano index.html

Put the following data into index.html:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML>

<HEAD>
<TITLE>Google</TITLE>
<META HTTP-EQUIV="REFRESH" CONTENT="0; URL=http://www.google.com">
</HEAD>

<BODY>

</BODY>
</HTML>

The rest of your website(s) should be name-based virtual hosts, where the configuration is located in a different directory, such as /usr/local/www/apache22/sites.

8. Install PHP 5 into the jail using ports

Installing PHP 5 inside using the ports system makes it easy to stay up-to-date. In this case, we’ll install PHP inside our jail with the Suhosin Hardened PHP security patch, which otherwise would need to be applied manually. Let’s get going!

Note that at the time of this writing, there is a warning in the PHP port to not install the Suhosin patch within a jailed environment. A workaround has been discovered which we will get to in a few moments. For now, simply install PHP and include the Suhosin patch when prompted for compile options:

root# pkg install php56

Make sure the PHP extensions are installed too! As well as the mysql extension for our connectivity to the database.

root# pkg install php56-extensions
root# pkg install mod_php56
root# pkg install php56-mysql

Copy the default php.ini file to its correct location

root# cp /usr/local/etc/php.ini-production /usr/local/etc/php.ini

Add the php type to the Apache httpd.conf file by running the following commands. In our case, we want the server to parts .html and .htm files to look for PHP code, if you do not want this then adjust the first line below accordingly.

echo "AddType application/x-httpd-php .php .html .htm" >> /usr/local/etc/apache24/httpd.conf

echo "AddType application/x-httpd-php-source .phps" >> /usr/local/etc/apache24/httpd.conf

Add a workaround that allows the Suhosin patch to work fine from within a jailed environment. Edit your jail’s /usr/local/etc/php.ini file and add the following line at the end:

suhosin.apc_bug_workaround = 1

Now restart the Apache server using apachectl restart and you should be golden.

At this point the server’s ability to server up dynamic web pages and content is complete. The reader can proceed to load his or her web pages into the server, typically by adding virtual hosts into the Apache configuration file. Even if only a singe domain or site is going to be hosted, it can be useful to install the site (and all other sites – many are possible on a single server) using name-based virtual hosts.

A sample name-based virtual host configuration is listed below. It can be added to the configuration file found in /usr/local/etc/apache22/extra/httpd-vhosts.conf, which is read automatically when Apache is started.

#
# newdomain.com
#
<VirtualHost *:80>

    ServerAdmin me@myemail.com
    DocumentRoot "/usr/local/www/apache22/sites/newdomain.com"

    ServerName newdomain.com
    ServerAlias www.newdomain.com

        <Directory />
            AllowOverride None
            Order deny,allow
            Deny from all
        </Directory>

        <Directory "/usr/local/www/apache22/sites/newdomain.com">
           Options Indexes FollowSymLinks
            AllowOverride Options FileInfo
            Order allow,deny
            Allow from all
        </Directory>
</VirtualHost>

Once this entry has been made into the configuration file, the user will need to ensure that the directory exists before Apache is started, else it will not be able to find the web pages to serve up.

root# mkdir /usr/local/www/apache22/sites/newdomain.com

At this point we can restart Apache again (with apachectl restart) and start serving up dynamic, database-driven web pages with MySQL and PHP inside an Apache name-based host on a SSL-capable virtualized jail server with FreeBSD. The server could safely deliver e-commerce applications, but could still be hardened for security in a number of additional ways. The next steps are simple forays into additional defense-in-depth security.

9. Install the ModSecurity web application firewall into the jail

Now that the server is fully functional, let’s start to secure it. We’ll add ModSecurity which looks specifically for web application attacks and it does a pretty good job of finding and blocking the most common attacks.

cd /usr/ports/www/mod_security
make install clean
chmod 755 /usr/local/libexec/apache22/mod_security2.so

Ensure that mod_unique is already in your /usr/local/etc/apache22/httpd.conf configuration file, and add it if not:

LoadModule unique_id_module libexec/apache22/mod_unique_id.so

Also add an entry into httpd.conf so that ModSecurity is also loaded:

LoadModule security2_module libexec/apache22/mod_security2.so

At the time of this writing, the port installed the core ruleset automatically into /usr/local/etc/apache22/Includes/mod_security2/. Additionally, logging is done to /var/log/httpd-modsec2*.log which makes it easy to review ModSecurity-specific logs at a glance, right next to the default location of regular Apache log files in /var/log.

Note that the default configuration of ModSecurity is to detect attacks only; ModSecurity will not automatically block web-based attacks until it is told to do so. The administrator is advised to review the Apache error logs located in /var/log/httpd-error.log for some time to monitor the types of errors generated, make changes to the ModSecurity configuration as required, and ensure legitimate access to the web application will not be blocked. Once the administrator is confident of his configuration, he can change ModSecurity to “On” instead of “DetectOnly,” the default setting.

For now, we’ll restart Apache so the changes are put into effect:

apachectl restart

10. Maintain integrity of the filesystem in the jail

We will install a file integrity monitor to watch for unexpected changes on our new server. In the unlikely event that a PHP/MySQL web application is hacked and the server is compromised, we need to know exactly what files changed. We will install AIDE (Advanced Intrusion Detection Environment), which an open-source file integrity monitor similar to Tripwire, a commercial product. If any of your web applications are compromised (often because the operator failed to update those applications with the latest security patches), unwanted files may appear on the system. A good operator should read logs daily, particularly the output from the integrity monitor.

Use the ports tree to get AIDE installed quickly and easily.

root# cd /usr/ports/security/aide
root# make install clean

Edit AIDE’s configuration file, located in /usr/local/etc, and add your web directory, home directories, and so on. Below are a few example entries that could be added, you will want to add more entries based on what you are trying to monitor:

/home/kel                      R-tiger-rmd160-sha1
/home/kel/.bash_history   L
/root/.bash_history     L
/usr/share/man          L
/usr/share/openssl/man  L
/usr/local/www          R-tiger-rmd160-sha1

Since AIDE does not run as a daemon and does not have e-mail or cron capabilities built-in, we have to find another way to run AIDE automatically on a regular basis and get the results sent offsite for review by a human being. In our case, to keep things simple we’ll use the built-in FreeBSD periodic scripts to do this on a daily basis. The easiest way is to simply create an /etc/daily.local file that will get run each day, with the contents automatically e-mailed to the administrator offsite (assuming the administrator has put his e-mail address in the mail aliases files, as described below). Issuing the following command will create the daily.local file if it doesn’t exist and put one command in it:

echo "/usr/local/bin/aide --check" >> /etc/daily.local

Please note that many ISPs block the ability for your server to send e-mails directly – they often want you to use their SMTP mail server as a gateway to send mail, so they can catch spammers. If a day or two has passed after building your server and you have not received any of the automated daily e-mails FreeBSD servers generate by default, your ISP is likely blocking outgoing mail sent directly from your server.

Unfortunately Sendmail, an age-old mailer well known on the Internet, is rather difficult to configure to use an outgoing gateway without the user having prior experience of Sendmail, so in the next

11. Change the mailer to something easier to configure (optional)

Install postfix and configure the SMTP server to use a local gateway.

# cd /usr/ports/mail/postfix
# make install clean

Edit the rc.conf file to enable postfix at startup, and add the following:

sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
postfix_enable="YES"

Edit the /etc/defaults/periodic.conf file so that it reflects the following:

daily_clean_hoststat_enable="NO"
daily_status_mail_rejects_enable="NO"
daily_status_include_submit_mailq="NO"
daily_submit_queuerun="NO"

Finally, determine if you can send mail directly on the Internet, or if you need to relay your mail through a 3rd party SMTP gateway. The latter is common practice among many cable and DSL broadband ISPs, so if this applies to you you’ll need to edit the /usr/local/etc/postfix/main.cf file to reflect the SMTP server. Search for ‘relayhost’ and then add your relay address nearby. In my case, my ISP is Shaw… please use your own ISP’s mail relay and not mine below, otherwise it won’t work.

relayhost = [shawmail.cg.shawcable.net]

Further, if your ISP requires SMTP authentication and uses SSL/TLS then you will need to perform one more additional steps, as detailed in the next section

Regardless of which method you use to send mail, for all situations you will need to configure your system’s mail aliases file, so that the system knows where to send mail to.

Edit the aliases file located at /etc/mail/aliases. For some unknown reason, postfix ignores the aliases file created at /usr/local/etc/postfix/aliases which gets created by some versions of postfix, so please use the /etc/mail/aliases file that sendmail would otherwise use and you will have no problems. Ensure that a valid external e-mail address is set. This ensures that you get the daily logs generated by FreeBSD. Make sure the newaliases command is run manually after the file has been changed, or else your changes won’t get loaded.

root# nano /etc/mail/aliases
root# newaliases

Finally, start the server to make the configuration funcational.

root# postfix start 

In the future, if you change postfix’ configuration you can simply issue the command postfix reload to apply the changes without needing to reboot.

  1. Enable SMTP via SSL/TLS (optional)

In our case, our server is hosted in the Cloud and we need to use a third-part relay (Gmail) that uses SSL/TLS. So we will need to ensure that Postfix was compiled with SASL2 and SSL/TLS enabled. If it wasn’t, you can simply delete the options file located in /var/db/posts/postfix, which is used to determine compile options when to build or rebuild a package. Delete that file and then rebuild postfix using the command portmaster postfix, and the port will ask you again using a text-based GUI which options you’ll like to compile in:

root# cd /var/db/ports/postfix
root# delete options
root# portmaster postfix

We have just a bit more work to do, by adding the following into /usr/local/etc/postfix/main.cf:

relayhost = [smtp.gmail.com]:587
smtp_sasl_auth_enable = yes
smtp_sasl_security_options = noanonymous
smtp_sasl_password_maps = hash:/usr/local/etc/postfix/sasl_passwd
smtp_sasl_mechanism_filter = plain, login
smtp_use_tls = yes
data_directory = /var/db/postfix

Then you need to create a /usr/local/etc/postfix/sasl_passwd file that contains something like this:

[smtp.gmail.com]:587 myaccount@gmail.com:mypassword

Finally, you will need to issue the following two commands to enable everything:

# postmap /usr/local/etc/postfix/sasl_passwd
# postfix reload

You should now send a test email to yourself and verify that everything works.

12. General configuration & E-Mailing of Daily Logs

Now would be a good time to ensure usernames like root and kel have the correct “names” associated to this server. Take a look at /etc/passwd to see if it’s correct, and if not issue something like:

pw usermod root -c 'Myphatserver &,,,'
pw usermod kel -c ‘Jim Nippholes,,,’

Of course this will only be useful if you correctly setup your mail aliases file earlier in this document, so that the system knows where to send your mail to.

13. Keeping Time

Time synchronization is an important part of a server’s function. As you build our your server and rely on applications, shell scripts and batch processing that gets transferred between this server and other servers, you will see how important time synchronization is. FreeBSD ships with the ntpd daemon for keeping your clock in-sync with time servers on the Internet. It is already installed and just needs to be configured.

The main configuration file for ntpd is found in /etc/ntp.conf. The defaults that were introduction in FreeBSD 8.x will be fine for most users, however one recommendation would be to uncomment the line that reads restrict default ignore so that no external connections will be accepted by ntpd.

Alternatively, if your version of FreeBSD doesn’t have a default /etc/ntp.conf file (which is common in earlier versions of FreeBSD, such as 6.x and 7.x), you can create a simple one using the entries below:

server 0.freebsd.pool.ntp.org iburst maxpoll 9
server 1.freebsd.pool.ntp.org iburst maxpoll 9
server 2.freebsd.pool.ntp.org iburst maxpoll 9
restrict default ignore

To make the server startup automatically, let’s add an entry to our rc.conf file.

echo ntpd_enable="YES" >> /etc/rc.conf

Now let’s run the server for the first time, by calling the daemon itself:

ntpd

14. Tell other servers we’re alive (optional)

This step looks at a rudimentary way of tracking the location (IP address) and status of all your servers from one location. We can do this by writing a script in PHP to keep a database of this information, such that we can glance at a single page and see what servers are alive and doing fine.

Basically, w’ere going to install the text-based web browser Lynx and then create a cron job to ping one of our own websites to notify we’re alive. The contents of the “alive.php” script is outside the scope of this article… it could be just about anything you want it to be.

# cd /usr/ports/www/lynx
# make install clean

Try lynx and see if it works:

/usr/local/bin/lynx -dump "http://www.mywebsite.complete/serverdb/alive.php"

Add the following to /etc/rc.local using the nano editor

#ping another server to say we're alive!
echo "ping www.mywebsite to say we're alive..."
/usr/local/bin/lynx -dump "http://www.mywebsite.com/serverdb/alive.php"

Add a crontab entry using the crontab -e command. Important Note: use a normal user’s crontab, not root’s!

# ping insurancelink.ca to say we're alive!!
10   *   *   *   *   /usr/local/bin/lynx -dump http://www.myserver.com/servers/saturn.php > /dev/null

15. Staying Up-To-Date

An important part of keeping a system secure is to keep on top of new security vulnerabilities and install patches as soon as they become available. FreeBSD shines is this area with a number of tools that make this easy. However the first step is to ensure you on are the security-advisories@freebsd.org mailing list, which will notify you when new system patches are available.

To keep the system kernel and userland updated, there are a number of options. The easiest of these is the freebsd-update command. To fetch new updates, use the freebsd-update fetch command, followed by the freebsd-update install command. If any parts of the kernel were updated, you will need to reboot to install the kernel, otherwise you can continue to use the system. This command provides binary updates to the kernel and userland environment, but it also updates the source code in your /usr/src tree.

Since the jail environment uses the host’s kernel, it’s very important to keep the FreeBSD kernel up-to-date with security patches. The freebsd-update command while inside the host machine will keep the kernel updated plus the host’s userland, but it will not update the jails userland.

To update the jail’s user binaries, we need to issue the command ezjail-admin update -u. This will update the base-jail using freebsd-update. So, when you receive an announcement of a new vulnerability in the FreeBSD kernel or user binaries, you’d perform a simple series of steps such as the following:

freebsd-update fetch
freebsd-update install
ezjail-admin update -u

Then your host and all jails will be updated with the latest patches.

To keep the ports tree updated, we will periodically run the command portsnap fetch update. This will update the ports tree in the host environment as well as the jail environment since our ports tree is mounted inside the jail as well.

We can easily check for known vulnerabilities in our installed ports using the portaudit -Fa command. Simply installing the tool makes it run automatically on a daily basis, as part of the /etc/periodic set of scripts, and this output will be e-mailed daily to the root user. So read the daily emails sent to you from this server, and it will indicate when it finds known vulnerabilities in installed software – a very cool security feature that covers many thousands of apps.

To check our installed ports versus the latest ports tree, we’ll use the pkg_version -v command to compare what is installed versus the latest version. There are a number of ways to upgrade installed ports, one of the easiest and most hassle-free is the portmaster utility, which can be found in /usr/ports/ports-mgmt:

cd /usr/ports/ports-mgmt/portmaster
make install clean

Then when you need to update a port, such as portaudit-0.5.14, you simply issue the portmaster portaudit command and it will be automatically compiled and upgrade to the latest version. Of course, use portmaster [installed port] to upgrade whichever port needs upgrading.

  1. Conclusion

Comments, feedback, and suggestions for improvement are welcome!

This document was authored by Kelly Martin, kelly@kelly3.com.

WordPress SFTP into FreeBSD jail

FreeBSD jail with WordPress SFTP plugin article image

The purpose of this article is to make it easy for WordPress and its 3rd party SFTP plugin to work on a FreeBSD host where the website and web server daemon runs within a jail, but the sshd daemon runs on the host thereby causing problems that prevent the WordPress SFTP plugin from working properly. In this article we enable remote read/write access to the WordPress application to enable it to self-update, using an sftp user who is restricted from seeing or accessing the rest of host’s  filesystem. The jail configuration for WordPress significantly limits the scope of a breach should WordPress or any one of its plugins become compromised by an attacker down the road. This article assumes any modern FreeBSD version is used. The author is using FreeBSD 10 and this very likely works on earlier versions too.

We’ll start by creating a special sftp/ssh user for WordPress on the FreeBSD host.

First create a new group called “chroot”:

pw groupadd chroot

Next, create a new user such as “wordpress-larry” and ensure their primary login group is set to “chroot”. You should likely add the user to an additional, secondary group such as “www” so they can access/edit files for the website.

adduser

Run the adduser command and follow the prompts to create a new user with a STRONG password. In our case we’ll create a new user with the username “wordpress-larry”. Now let’s switch to that user and create a new SSH private/public key pair for logging in, since it’s safer to use keys than a password:

su myuser
cd ~
ssh-keygen -t rsa -b 4096

At this point we need to edit the /etc/ssh/sshd_config file to ensure the user is chroot’ed within their home directory only. The goal is to prevent a rogue user from having any access to the rest of the filesystem, we want to keep him in the protected jail. Edit the file and add a new group for chroot, like this:

nano /etc/ssh/sshd_config

  Match group          chroot
    ChrootDirectory    /home/%u
    X11Forwarding      no
    AllowTcpForwarding no
    ForceCommand       internal-sftp

One the configuration has been updated, restart the SSH daemon, sshd:

/etc/rc.d/sshd restart

The user can only navigate within their home directory thanks to the ChrootDirectory configuration. However we do need them to access part of a website with files located elsewhere, so let’s first mirror the directory structure as seen within a jailed login, so it looks like the wordpress location when the user logs in:

mkdir –p /home/myuser/wordpress

Next we’ll use a filesystem mount to mount the website’s directory into the user’s home directory using mount_nullfs, where “source” is the /full/path to the files you want to make available to the user:

mount_nullfs /path/to/wordpress/files /home/myuser/wordpress

Confirm the mount was successful then create an entry in /etc/fstab related to the above, making sure the entry is all on one line:

/path/to/wordpress/files/ /home/myuser/wordpress/ nullfs rw 0 0

For remote access, it’s critical that the user’s directory be owned by “root” and also be part of the “chroot” group, or else SSH/SFTP will not allow the user to even login:

chown –R root:chroot /home/myuser
chmod 755 /home/myuser

Finally, test the connection (using SFTP, not SSH) to make sure it’s working. Logging in manually will let you confirm that the WordPress files are there, in the user’s home directory, and so the SFTP plugin will be able to function in WordPress with the added security of only being able to see and edit files located in the website’s jail.

Done.

Kelly’s MySQL quick tutorial

You don’t need tools like MySQLAdmin when basic tasks like creating and deleting databases, adding users and adjusting permissions can easily be done from the command line. I run these commands on a FreeBSD system and is likely the same on Linux, Windows, and other systems.

Basic tasks

Log into mysql:

#mysql –u root –p
password: ******

Welcome to MySQL………

Create database:

>create database Frankenstein;
>use Frankenstein;

Create user:

>GRANT ALL PRIVILEGES ON Frankenstein.* TO Bob@localhost IDENTIFIED BY “bobspassword69” WITH GRANT OPTION;

Then reload the grant tables in the database using flush privileges. If you forget this step you can also run “mysqladmin reload” to reload the grant tables. Then exit.

FLUSH PRIVILEGES;
exit;

At this point you should be golden.

On a side note, if you experience a problem where MySQL keeps exiting with an error about mysql.sock on FreeBSD, it’s likely a permissions problem in your system. If you have this problem, try something like this:

“do you already have another mysqld server running on socket: /var/run/mysql/mysql.sock ?”

#chown _mysql /var/run/mysql

What about backup and recovery?

Now that we have amazing content in our database, we must backup the database to keep it safe like this:

mysqldump –u bob –p Frankenstein > frankenstein-backup-01Jan2023.sql

Whew. Now let’s restore the database into a running MySQL server of our choice:

mysql –u bob –p Frankenstein < frankenstein-db-backup-01Jan2023.sql

Anything else? Consult the manual, search the web, or add a comment here.