Press "Enter" to skip to content

FreeBSD jails with Apache, PHP, and MySQL in separate jails

Setup a secure web server on FreeBSD with Apache/PHP and MySQL each in separate jails to help contain security incidents, then setup a file integrity monitor on the host to monitor any changes inside the jails.

Why jails, or virtual containers, are useful

Why use FreeBSD jails? The jail helps limit the impact of a complete system compromise by separating applications and services from each other and from the main system, putting Internet-facing applications in virtual containers. For example, a web server and a database server will be installed in separate jails that cannot see each other and cannot see all service of the host they run on.

When an Internet-facing application is compromised, the attacker may gain the privileges of the user account used by the service (such as httpd, Apache) within the jail. The attacker can make many changes on the system, perform additional attacks and attempt privilege escalation to the jail’s root capability. But he cannot reach the host’s root or other applications thanks to the jail.

From the host we run monitoring tools to first discover the incident, alert someone, and monitor what has changed on the system so we can fix it with certainty. Combined with regular backups, it becomes much easier to recover at a system level from web-based vulnerabilities that are frequently exploited.

Getting started

To get started, on a running FreeBSD system we’ll  install the the ezjail tool  using FreeBSD’s built-in package system, pkg. This tool helps build our base jail environment and makes it easy to deploy as many jails as we need for different applications and services. All commands in the article are made as root (with a ‘#” prompt) unless otherwise noted.

# pkg install ezjail

Open the/usr/local/etc/ezjail.conf configuration file in an editor and update the configuration to point the jails to where they are stored. For some people the defaults are fine, in our case we’ll be using ZFS. Note that the ezjail_jailzfs entry below does not have a leading / in the path:

ezjail_jaildir=/usr/jails
ezjail_use_zfs="YES"
ezjail_use_zfs_for_jails="YES"
ezjail_jailzfs="usr/jails"

For our basic need, the rest of the ezjail defaults will be fine. Now we can install the base jail using the following command.

# ezjail-admin install

This will proceed to download a fresh copy of FreeBSD and install the it into a basejail. Make sure all the jails are started automatically at boot time by adding ezjail_enable=”YES” to /etc/rc.conf as follows:

# echo ezjail_enable="YES" >> /etc/rc.conf

We will use the loopback interface for our jails. In this case, the address will be aliased to 10.0.0.1 for lo1 and 10.0.0.2 for lo2, and so on, because each jail needs to have a unique private IP address to communicate with the host and other systems.

#ifconfig lo1 create inet 192.168.0.1 netmask 255.255.255.0
#ifconfig lo2 create inet 192.168.0.2 netmask 255.255.255.0
#ifconfig lo3 create inet 192.168.0.3 netmask 255.255.255.0

For each interface added, remember to also put a corresponding entry into /etc/rc.conf so the interface will be started at boot. Edit the configuration adding the following information right after the entry for your primary network interface:

cloned_interfaces="lo1 lo2 lo3"
ifconfig_lo1="inet 192.168.0.1 netmask 255.255.255.0"
ifconfig_lo2="inet 192.168.0.2 netmask 255.255.255.0”
ifconfig_lo3="inet 192.168.0.3 netmask 255.255.255.0”

Now make corresponding entries in your/etc/hosts file for the jail names so the host knows these hostnames are on the loopback. You can use any names you like. Edit the file, in our case we’ll create three jails so we’ll add host entries for each one:

127.0.0.1      app
127.0.0.1      web
127.0.0.1      db

We’re ready to make our containers now and the jails will be created quite quickly. This is a testament to the lightweight nature of a jail, we are installing only userland components. We’ll create three jails using the name/address pair that match the entries above.

# ezjail-admin create app 192.168.0.1
# ezjail-admin create web 192.168.0.2
# ezjail-admin create db 192.168.0.3

We can now see the jails at /usr/jails and under each jail name is the / root directory of the jail. Each jail will think it is a complete FreeBSD system of its own. Because of that, be prepared to do some basic configuration within each jail just like you would with any freshly installed FreeBSD system. At the very least, we should create a hostname within each jail such as hostname=”app.mydomain.com” entry within each jail’s /etc/rc.conf.

Before starting the jail it’s important to copy the network resolver configuration to the jail, so both systems can use the host’s DNS nameservers. Using symbolic links to this file in each jail is an option and might help avoid configuration problems in the future if the resolver is updated on the host but not the jails. However in our case we’ll just copy the configuration file to ensure the jails are completely isolated. If you anticipate changes the nameservers used in resolv.conf then write a short script and add a cron job to copy this resolv.conf into the jails when the file changes so the jails stay up-to-date.

# cp /etc/resolv.conf /usr/jails/app/etc
# cp /etc/resolv.conf /usr/jails/web/etc
# cp /etc/resolv.conf /usr/jails/db/etc

Now start the jails. On slow machines this make take some time. On OCI, it flies.

# service ezjail start 

Now the jails are running, go into each jail and change the root password. There are several ways to do this, the simplest may be to just open your favorite shell within each jail (csh, bash, or something else) using the jexec command:

# jexec app csh
# passwd
# exit
# jexec web csh
# passwd
# exit
# jexec db csh
# passwd
# exit

From here you can go back into each jail and configure each operating system, install applications and add jailed users as desired. If the timezone in the jails is not correct run tzsetup to fix it. Going in and out of many jails things can get confusing so it’s a good idea to use a visual style of prompt in your shell of choice so that it’s easier to see when you’re inside the host or when you’re working on a certain jail.

Just for fun, below is a color csh prompt I use with csh inside my jails:

set prompt="%t %{\e[31;40;1m%}`whoami`%{\e[0m%}%{\e[36m%}@%{\e[0m%}%{\e[31;40m%}`hostname -s`%{\e[31m%}:%{\e[0m%}%{\e[31m%}%~%{\e[32;40;0m%}"\#"%{\e[0m%}%{\e[32m%}%{\e[0m%}"

Prompts are personal preference, the above one looks as follows on a dark terminal background. The red background and hostname tells me I’m inside a jail on a production server:

Other useful jail commands run from the host include jls which lists the jails by number. You can reference the jail by number so for example if a jls shows the web jail is #1, then  jexec 1 csh opens a command shell on the first jail, provided the jail is running. Running jexec web csh does the same thing, running a command in a jail by name without needing to know the jail’s number. Restarting jails will change their number so it is easier to call the jail by name.

An important command to keep the jail up-to-date with the latest patches is ezjail-admin update -u and should be run every time you update the host, whether binary patches were applied (freebsd-update fetch install) or some other method is used to keep the host up to date.

 Once inside a new system, which is what a new jail feels like, many administrators will immediately try to ping an outside host to verify the network settings. This will not work inside a jail however, as it requires access to raw sockets which by default are not allowed. To temporarily enable ICMP ping to work from within the jail to the internet or elsewhere, issue the command sysctl security.jail.allow_raw_sockets=1 from the host environment.

Keeping Time

I like my servers on the correct time. On newer versions of FreeBSD, ntpd just needs to be turned on. You can also add it to your startup at boot in /etc/rc.conf quickly by running these two commands:

#service ntpd start
#echo ntpd_enable="YES" >> /etc/rc.conf

Note that the daemon will be listening on the interface for connections so be aware and block it at the firewall for external connections. Alternatively, you can also prevent ntp from listening for connections by editing /etc/ntp.conf and uncomment the line(s) that reads restrict default ignore.

Install MySQL into a jail

There are numerous different MySQL versions in package repository, all are installed in a similar manner. For this article we’ll install version 7.0 – install whichever version suits your fancy.

First, from the host shell let’s jump open a csh shell in the db (database) jails install mysql-server:

# jexec db csh
# pkg install mysql80-server

Now add a startup entry to rc.conf so it starts automatically at boot:

# echo 'mysql_enable="YES"' >> /etc/rc.conf

Start the server

# /usr/local/etc/rc.d/mysql-server start

Set the root password! This step should never be skipped!

# /usr/local/bin/mysqladmin -u root password 'my-new-password1234+@#$'

Install Apache 2.4 into another jail

Install and configure the latest Apache 2.4.x using the ports tree. It’s a very good idea to setup virtual hosting immediately as it can provide a small additional layer of security. We will also set the default web page to redirect visitors to a search.

Name-based virtual hosting makes web site enumeration more difficult when a hacker does not know how many website and which ones are being hosted at each IP address. A hacker needs to query a full DNS of the site. Since no web traffic should be expected on the default virtual host reached only by IP address, we can direct that traffic somewhere else.

Start by entering the web jail from the host and install the Apache package.

# jexec web csh
# pkg install apache24

Configure Apache 2.4 as you would normally and refer to the Apache documentation if needed. For our purposes, we will make sure we enable MySQL support and disable mod_dav and mod_dav_fs, and leave the rest as defaults.

From within the jail, Apache’s main configuration file is located at /usr/local/etc/apache24/httpd.conf. Since we’ll be installing PHP next, we need to edit and change DirectoryIndex to include files that end with .php:

DirectoryIndex index.php index.html index.htm

You will also need to change Apache’s Listen and ServerName directives in httpd.conf to the jail’s private address and port, or else Apache will not be able to start:

Listen 192.168.0.1:80 
ServerName 192.168.0.1:80

Check your jail’s /etc/rc.conf and make sure the hostname= is an FQDN (fully qualified domain name), else Apache will not start correctly. Alternatively, you could also set the hostname in /etc/rc.conf to anything of your choosing as long as you have a similar entry in the /etc/hosts file. For the purposes of this article, we’ll assume you need to setup your jails domain name as well.

Edit your jail’s /etc/rc.conf and add the following, putting in your domain name below:

hostname="somesite.com"
accf_http_load="YES"
apache24_enable="YES"

Start apache with apachectl start and if it works, surf to the page and verify it’s running.

Restart the jail by first exiting the jail and then from the host issue the command ezjail-admin web restart and then go back into the jail and verify that apache does indeed start at boot.

For fun, change the default Apache page to redirect to Google or some other page, since we will be using virtual hosts in our configuration. Having a default page as a redirection reduces the amount of information given out to an attacked when the reach the web server by public IP address only (such as when found by scripts through a port scan), instead of reaching a specific site by its domain name. 

# cd /usr/local/www/apache22/data
# nano index.html

Put the following data into index.html:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML>
<HEAD>
<TITLE>Google</TITLE>
<META HTTP-EQUIV="REFRESH" CONTENT="0; URL=http://www.google.com">
</HEAD>
<BODY>
</BODY>
</HTML>

The rest of your website(s) should be name-based virtual hosts, where the configuration is located in a different directory, such as /usr/local/www/apache22/sites. 

Install PHP into the web jail

Installing PHP inside using the ports system makes it easy to stay up-to-date. In this case, we’ll install PHP inside our jail with the Suhosin Hardened PHP security patch, which otherwise would need to be applied manually. Let’s get going! 

# pkg install php70 

Make sure the PHP extensions are installed too! As well as the mysql extension for our connectivity to the database.

# pkg install php72-extensions
# pkg install mod_php72
# pkg install php72-mysql

Copy the default php.ini file to its correct location

root# cp /usr/local/etc/php.ini-production /usr/local/etc/php.ini

Add the php type to the Apache httpd.conf file by running the following commands. In our case, we want the server to parts .html and .htm files to look for PHP code, if you do not want this then adjust the first line below accordingly.

#echo "AddType application/x-httpd-php .php .html .htm" >> /usr/local/etc/apache24/httpd.conf

#echo "AddType application/x-httpd-php-source .phps" >> /usr/local/etc/apache24/httpd.conf

Now restart the Apache server using apachectl restart and you should be golden.

At this point the server’s ability to server up dynamic web pages and content is complete. The reader can proceed to load his or her web pages into the server, typically by adding virtual hosts into the Apache configuration file. Even if only a singe domain or site is going to be hosted, it can be useful to install the site (and all other sites – many are possible on a single server) using name-based virtual hosts.

A sample name-based virtual host configuration is listed below. It can be added to the configuration file found in /usr/local/etc/apache22/extra/httpd-vhosts.conf, which is read automatically when Apache is started.

#
# newdomain.com
#

<VirtualHost *:80>
    ServerAdmin me@myemail.com
    DocumentRoot "/usr/local/www/apache22/sites/newdomain.com"
    ServerName newdomain.com
    ServerAlias www.newdomain.com

        <Directory />
            AllowOverride None
            Order deny,allow
            Deny from all
        </Directory>
 
        <Directory "/usr/local/www/apache22/sites/newdomain.com">
           Options Indexes FollowSymLinks
            AllowOverride Options FileInfo
            Order allow,deny
            Allow from all
        </Directory>
</VirtualHost>

Once this entry has been made into the configuration file, the user will need to ensure that the directory exists before Apache is started, else it will not be able to find the web pages to serve up.

# mkdir /usr/local/www/apache22/sites/newdomain.com

At this point we can restart Apache again (with apachectl restart) and start serving up dynamic, database-driven web pages with MySQL and PHP inside an Apache name-based host on a SSL-capable virtualized jail server with FreeBSD. The server could safely deliver e-commerce applications, but could still be hardened for security in a number of additional ways. The next steps are simple forays into additional defense-in-depth security.

Communicating between jails

Jails may often need to communicate with each other, and the mysql jail in particular needs to be reached from two web jails. There are two general ways to achieve this: sockets or using a network. Sockets were once significantly faster than the network but the performance gap seems to has narrowed significantly over the years. Both approaches are discussed here with their advantages and disadvantages.

Option 1: using sockets

For some people this might be preferred method when the db, web and app tier all exist on the same server in different jails. The shared mysql socket needs to be reachable by the web jail. To do this we’ll have to merge directories on three filesystems. I borrowed the idea from this article that was quite useful. The only caveat is, I do not believe unionfs works fully with ZFS today, but since our base OS is using UFS and only the jails use ZFS with block storage, it works.

In parent system create subdirectory /usr/mysqltmp. Then in each jail create subdirectory /mysqltmp, as follows:

mkdir /usr/mysqltmp
mkdir /block/jails/web/mysqltmp
mkdir /block/jails/db/mysqltmp

Merge these directories onto the host using:

mount_unionfs /usr/mysqltmp /block/jails/web/mysqltmp
mount_unionfs /usr/mysqltmp /block/jails/db/mysqltmp

Now edit (or create, if needed) the mysql configuration file /usr/local/etc/mysql/my.cnf and add “skipnetworking” under the [mysqld] section to ensure no external communication works over the network, and then restart mysql:

#jexec db /bin/csh
#nano -w /usr/local/etc/mysql/my.cnf
#/usr/local/etc/rc.d/mysql-server restart
#exit

Then finally go into the web jail and configure the application to use a socket instead of the network to communicate by replacing a standard private IP address with localhost:/mysqlsock/mysql.sock. In our case we’re using WordPress which has a configuration file name wp-config.php and a database host address that looks like this:

define('DB_HOST', 'localhost:/mysqlsock/mysql.sock');

That’s it, now just test the web application to ensure it can communicate with the database without networking. In addition to performance improvements, this has the added benefit where MySQL can be inside a jail and does not need to accept incoming network connections in order to serve requests. Permissions on the linked directories should be kept as strict as possible and monitored for unusual activity.

Note: as a final setp, you will need to add an entry in /etc/fstab to make the above union permanent, or else the union won’t survive a reboot and you’ll have to repeat the above process. Unfortunately I never got the fstab quite right and errors in the fstab will keep the machine from booting… since I didn’t have console access to troubleshoot the issue, I decided to go with Option 2, below, which is simpler in some ways and scales better if you need to scale the system onto multiple servers down the road.

Option 2: use network communication

The traditional method is to use TCP/IP networking to communicate between jails, in our case the web, app and db jails. This method uses localhost and only  NAT capabilities between the jails, so we’ll use pf, which is a fast and powerful firewall built-in to FreeBSD. NOTE: you will need to setup a basic pf.conf configuration, if you do not have a pf firewall setup, try this one for a single NIC configuration, a short blog post which I created just for this article.

We are also redirecting incoming traffic on two of the host’s network interfaces to the two web application jails. Then it’s simple to configure the web application to use the database over the network with the db jail’s private IP address and port number.

Edit the host’s pf.conf file:

#Redirect web traffic to the two web jails.

rdr on $ext_if_app proto tcp from any to ($ext_if_app) port http -> $app port http

rdr on $ext_if_app proto tcp from any to ($ext_if_app) port https -> $app port https

rdr on $ext_if_web proto tcp from any to $ext_if_web port http -> $web port http

rdr on $ext_if_web proto tcp from any to $ext_if_web port https -> $web port https

#Allow jail traffic to NAT back to anywhere
nat on $ext_if_app from $app to any -> ($ext_if_app)
nat on $ext_if_web from $web to any -> ($ext_if_web)
nat on $ext_if_ins from $db to any -> ($ext_if_ins)

Reload pf’s configuration file using “pfctl -f /etc/pf.conf” and then specify the db’s private IP address when asked for the host address of the database when first configuring the application.

Maintain integrity of the filesystems in the jails

Now we install a very simple file integrity monitor to watch for unexpected changes on our new server. In the unlikely event that a PHP/MySQL web application is hacked and the server is compromised, we need to know exactly what files changed. We will install AIDE (Advanced Intrusion Detection Environment), which an open-source file integrity monitor similar to Tripwire, a commercial product. If any of your web applications are compromised (often because the operator failed to update those applications with the latest security patches), unwanted files may appear on the system. A good operator should read logs daily, particularly the output from the integrity monitor.

Use the ports tree to get AIDE installed quickly and easily.

# cd /usr/ports/security/aide
# make install clean

Edit AIDE’s configuration file, located in /usr/local/etc, and add your web directory, home directories, and so on. Below are a few example entries that could be added, you will want to add more entries based on what you are trying to monitor:

/home/myaccount              R-tiger-rmd160-sha1
/home/myaccount/.bash_history          L
/root/.bash_history          L
/usr/share/man               L
/usr/share/openssl/man       L
/usr/local/www               R-tiger-rmd160-sha1

Since AIDE does not run as a daemon and does not have e-mail or cron capabilities built-in, we have to find another way to run AIDE automatically on a regular basis and get the results sent offsite for review by a human being. In our case, to keep things simple we’ll use the built-in FreeBSD periodic scripts to do this on a daily basis. The easiest way is to simply create an /etc/daily.local file that will get run each day, with the contents automatically e-mailed to the administrator offsite (assuming the administrator has put his e-mail address in the mail aliases files, as described below). Issuing the following command will create the daily.local file if it doesn’t exist and put one command in it:

#echo "/usr/local/bin/aide --check" >> /etc/daily.local

If you’re using an ISP instead of a hosting in the cloud, please note that many ISPs block the ability for your server to send e-mails directly – they often want you to use their SMTP mail server as a gateway to send mail, so they can catch spammers. If a day or two has passed after building your server and you have not received any of the automated daily e-mails FreeBSD servers generate by default, your ISP is likely blocking outgoing mail sent directly from your server.

Final Thoughts

I use this configuration frequently, but don’t always remember to update the documentation here on this site. So if you have a problem or something doesn’t work, add a comment below and I’ll try to help you out and then fix this article. Also, the FreeBSD community forums are an excellent source of information with many helpful people there.

Be First to Comment

Leave a Reply

Your email address will not be published. Required fields are marked *