Isolate websites using Nginx, PHP-FPM, Acme.sh, MySQL and AIDE on FreeBSD

This article seeks to isolate each website on a server to minimize threat exposure. If one website gets compromised we want to limit lateral movement so that other websites are unaffected and busy servers running dozens or hundreds of website domains can continue to function.

Objectives, how, and why

We setup php-fpm and nginx to use a unique user process for each website. The will communicate using a Unix socket.

Far more common is to have all websites run under the same user process. The problem with the default approach is when one website is compromised, the attacker can often traverse directories to read and write files across all websites on the server’s filesystem, risking exposure of confidential information and personal data.

This approach was inspired by an article on the same topic but written for Linux, so I wanted to make a FreeBSD version.

We’ll make SSL easy with acme.sh, then install the Tripwire-like AIDE filesystem monitor. The use of acme.sh to simplify the process of installing certificates is inpired by an article on Devopscraft. They make it easy for all websites to support strong encryption and one can even force all traffic to use encryption if this is desired.

Note: At the time of writing the versions used were FreeBSD 13.2, nginx 1.24, PHP 8.2.x, MySQL 8.x, Acme.sh 3.x, AIDE 0.27.

Step 1, Configure nginx and php-fpm, unique user/group and socket

If you don’t have nginx or php installed yet, let’s get started:

#pkg install nginx
#pkg install php82
#pkg install php82-extensions

If you need help getting these configured and starting after every reboot, there are some good articles on getting a basic nginx/php-fpm/mysql set up using FreeBSD (examples: 1, 2, 3 – these are all similar, so for those in a hurry just read the first one).

After you have a basic setup of Nginx and PHP-FPM, it’s time to secure the environment. The key to have each website run under its own user process so if one site is compromised due to an application vulnerability, the attacker will have a more difficult time affecting other sites on the same server.

Frst, add a new group and user on the system for the new website domain, making sure the user has “nologin” privileges when prompted for the shell type, no password is set, and the account is locked after creation.

Change the “newdomain.com” used in all parts of this article to your app’s domain. All commands below are performed as a root or sudo.

#pw groupadd newdomain.com
#adduser newdomain.com

For the “adduser” command you would answer the questions like this:

Username: newdomain.com
Full name: 
Uid (Leave empty for default): 
Login group [newdomain.com]: 
Login group is newdomain.com. Invite newdomain.com into other groups? []: 
Login class [default]: nologin
Shell (sh csh tcsh bash rbash nologin) [sh]: nologin
Home directory [/home/newdomain.com]: 
Home directory permissions (Leave empty for default): 
Use password-based authentication? [yes]: no
Lock out the account after creation? [no]: yes
Username   : newdomain.com
Password   : <disabled>
Full Name  : 
Uid        : 1007
Class      : nologin
Groups     : newdomain.com 
Home       : /home/newdomain.com
Home Mode  : 
Shell      : /usr/sbin/nologin
Locked     : yes
OK? (yes/no): yes

Now let’s create the site’s configuration profile.

#cd /usr/local/etc/php-fpm.d
#nano newdomain.com.conf

You create a new config file for each domain being hosted, knowing that all configuration files in this directory with a *.conf suffix are loaded by the php-fpm daemon automatically when the application is started and restarted. A standard php-fpm installation on FreeBSD 12 has set this behaviour in the default configuration file.

[newdomain.com]
user = newdomain.com
group = newdomain.com
listen = /var/run/php-fpm-newdomain.com.sock
listen.owner = www
listen.group = www
php_admin_value[disable_functions] = exec,passthru,shell_exec,system
php_admin_flag[allow_url_fopen] = off
pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3
chdir = /

Save the file and restart the php-fpm daemon.

#service php-fpm restart

Check to see if the newdomain.com has a running process now:

#ps aux|grep newdomain.com

If you do see it running under its own user process, go on to step 2.

Step 2, Configure nginx

Edit your nginx configuration to add the new website with its own socket by following these instructions. There are different styles of configuration for nginx on different Unix-like platforms but this author’s own preference is to simply add a line at the bottom of the default config file that tells nginx to also read all configurations in the /usr/local/etc/nginx/sites-enabled directory. Then create a new configuration file for each website, making it easy to add and remove many websites at a time.

#nano /usr/local/etc/nginx.conf

Append the following entry somewhere in this config file:

#include sites-enabled/*;

A sample configuration is below, customize it to meet your needs and the needs of your application. This example has extra bits added to help a WordPress website run little better (fixes broken permalink issues). The SSL configuration below should be #commented out for now but will be used later.

Pay special attention to all the “newdomain.com” entries especially the “fastcgi_pass unix:/var/run/php-fpm-newdomain.com.sock;” entry near the end of the configuration which ensures nginx has a unique Unix socket for each php-fpm process hosting this website . Every user process will use a different socket to help with isolation.

Create a new nginx config file for this website:

#nano /usr/local/etc/nginx/sites-enabled/newdomain.com

Then, add an entry such as the one below, keeping the commented items as well for now:

server {
    listen 80;
#    listen 443 ssl;


    root /usr/local/www/nginx/newdomain.com;
    index index.php index.html index.htm;


    server_name newdomain.com www.newdomain.com;


#    ssl_certificate      /root/.acme.sh/newdomain.com/newdomain.com.cer;
#    ssl_certificate_key  /root/.acme.sh/newdomain.com/newdomain.com.key;


#    ssl_session_cache    shared:SSL:1m;
#    ssl_session_timeout  5m;


    ssl_ciphers  HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers  on;


    gzip on;
    gzip_disable "msie6";
    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 5;
    gzip_buffers 16 8k;
    gzip_http_version 1.0;


    location / {
        #below is for WordPress permalinks
        try_files $uri $uri/ /index.php?$args;
    }


    #rewrite /wordpress to top level directory
    location ^~ /wordpress/(.+) {
      rewrite ^/wordpress/(.+)$ $1 last;
    }


    # Add trailing slash to */wp-admin requests.
    rewrite /wp-admin$ $scheme://$host$uri/ permanent;


    # Directives to send expires headers and turn off 404 error logging.
    location ~* ^.+\.(ogg|ogv|svg|svgz|eot|otf|woff|mp4|ttf|rss|atom|jpg|jpeg|gif|png|ico|zip|tgz|gz|rar|bz2|doc|xls|exe|ppt|tar|mid|midi|wav|bmp|rtf)$ {
           access_log off; log_not_found off; expires max;
    }

    location ~ \.php$ {
        try_files $uri =404;
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_pass unix:/var/run/php-fpm-newdomain.com.sock;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include fastcgi_params;
        fastcgi_read_timeout 540;
    }

    #add some caching info
    set $skip_cache 0;


    # POST requests and URLs with a query string should always go to PHP
    if ($request_method = POST) {
        set $skip_cache 1;
    }
}

Make a directory to store your web files that matches the above root directory.

It’s really important that all files for this domain are owned by the user process. Without this step there is no point in doing user isolation. No user other than the website user process should be able to write files in the website’s directory.

#mkdir /usr/local/www/nginx/newdomain.com
#chown -R newdomain.com:newdomain.com /usr/local/www/nginx/newdomain.com

Restart nginx

#service nginx restart

Step 3, Enable SSL/TLS

First install the acme.sh package to manage our free Let’s Encrypt keys.:

#pkg install acme.sh

Issue a certificate. You need to first have the DNS A-record for the domain pointing to this server’s IP address before the next step will work correctly.

#acme.sh --issue -d newdomain.com --webroot /usr/local/www/nginx/newdomain.com

If acme prints a key you have success.

Troubleshooting: If acme fails with an error and does not provide a key, it likely means either your DNS for the domain is not yet pointing to this server IP address or acme may not have write permissions in the directory. Note that acme uses Let’s Encrypt to generate the certificates and to prove ownership before issuing the cert, acme.sh creates a temporary web page to be served on port 80 that is created and deleted automatically.

Need a wildcard certificate instead? And happen to be using Cloudflare for DNS? Here’s a good article on how to get an API key from Cloudflare so acme.sh, with more updated info on the Cloudflare API token configuration here on Github.

With SSL key in hand, update the nginx configuration to use the cert for this site. These simple steps using acme are a fast way to get a free SSL certificate for encrypted web traffic.

#nano /usr/local/etc/nginx/sites-enabled/newdomain.com

If you used the above sample configuration, you can just uncomment the SSL lines after ensuring your keys are setup correctly, and make sure the SSL entries map to the location of the key and certificate.

Note about errors: there are at least 6 locations in the above nginx config file where you must replace “yourdomain.com” with your own website domain. It’s easy to forget the last entry, updating the socket, because even if that setting is missed the process for this may appear to run fine under another user but would not be able to read/write files in the correct website directory, causing all sorts of errors. In this article we are configuring php-fpm with nginx specifically to running each website in its own process so we have greater control over isolation. If the website is ever compromised by a bad actor hacker, this approach limits the damage zone on a multi-site server.

Restart nginx to take effect.

#service nginx restart

At this point the website is fully functional. It just needs a database, an app and some content.

Step 4 Secure the app’s config files

If you are indeed running WordPress, remember to change permissions on the “wp-config.php” file to keep out those pesky intruders who may have compromised a nearby website on the same host.

If you’re not running WordPress, with other applications there is usually a similar config file with the database details saved in plain text so it’s important to set permissions on it to 600, which means read/write access for the website process only and no access by any other user or group.

#chmod 600 /usr/local/www/nginx/newdomain.com/wp-config.php

If this is the only website you will ever need to create, have a good life, thanks for reading and goodbye. Otherwise continue on.

Step 5, Create a database for the app

Fortunately, I wrote short article on using MySQL from the command line.

If you have not yet installed MySQL, use the pkg system on FreeBSD and secure the installation berfore creating a database. There are many articles on the web about how to do this.

Step 6, Rinse and repeat

Now rapidly repeat the above steps for every website you want to add. You might have dozens.

1. Create a new group and a new user, such as xyz.com
2. Create xyz.com config for php-fpm.d with a unique user, group, and socket
3. Restart php-fpm
4. Edit nginx config, add an entry for xyz.com, port 80 only
5. Restart nginx
6. Point the website’s DNS A-record to this server’s IP address
7. Install SSL certificate, update nginx config for SSL, restart nginx
8. Create database and db user
9. Install application
10. Test the web app

Step 6, AIDE filesystem integrity check

Let’s get AIDE running, it’s simple:

#pkg install aide
#nano /usr/local/etc/aide.conf

Edit the config file and ensure your website directories are part of your configuration. At a bare minimum, as your websites:

/usr/local/www/nginx    R-tiger-rmd160-sha1

Then initialize AIDE. This will take a while depending on how many files/directories you have configured.

AIDE does not run as a daemon, run it manually or as a cron job. After it’s initialized, move the database from aide.db.new to aide.db, then create a test file and check to see if AIDE finds it:

#aide --init
#cd /var/db/aide/databases
#mv aide.db.new aide.db
#touch /usr/local/www/nginx/JUST_A_TEST
#aide --check

That’s it. Appreciate any comments or feedback. I update this article regularly.