Host multiple web sites securely with nginx, PHP-FPM and MySQL 8 on FreeBSD 12

This article focuses on providing security and isolation on a FreeBSD 12 web server hosting numerous websites. We use php-fpm and nginx to host many website, each under its own user process which can be contained by the system. The approach was inspired by the DigitalOcean article on the same topic but written for Linux and I wanted to make a FreeBSD version for the benefit of all FreeBSD users.

We’ll go a step further by adding SSL with free SSL certificates for every website from Let’s Encrypt to ensure all traffic to and from the website can be encrypted. Encryption is a basic requirement for all websites now to fully function on the web. The use of acme.sh to simplify the process of installing certificates is inpired by an article on Devopscraft. They make it easy for every website to support strong encryption and even force all traffic to use encryption if this is desired.

Step 1, Configure the system and php-fpm

This article assumes you have nginx and php-fpm already installed and running, and ideally MySQL too. If you don’t have the prerequisites, it’s not hard to pkg add each of the applications, make each service to start automatically and then secure the installation. There are some good articles on getting nginx/php-fpm/mysql set this up on FreeBSD already out there for FreeBSD (examples: 1, 2, 3 – these are similar but each has some unique bits too).

Starting with a standard nginx, php-fpm and mysql install, we’re going to make each website run under its own user process so if the site is compromised, it will not have permission to affect other sites on the same server.

We first add a group and user on the system for the new website, making sure the user has “nologin” privileges when prompted for the shell type, no password is set, and the account is locked after creation. Then we create a new configuration file in the php-fpm.d directory which will be automatically loaded when php-fpm is run or restarted.

You obviously must change the “newdomain.com” used in all parts of this article to the domain name you looking to host. All commands below are to be performed as a root user or sudo.

pw groupadd newdomain.com
adduser newdomain.com
cd /usr/local/etc/php-fpm.d
nano newdomain.com.conf

Then you create a new config file for each domain being hosted, knowing that all configuration files in this directory with a *.conf suffix are loaded by the php-fpm daemon automatically when the application is started and restarted. A standard php-fpm installation on FreeBSD 12 has set this behaviour in the default configuration file.

[newdomain.com]
user = newdomain.com
group = newdomain.com
listen = /var/run/php-fpm-newdomain.com.sock
listen.owner = www
listen.group = www
php_admin_value[disable_functions] = exec,passthru,shell_exec,system
php_admin_flag[allow_url_fopen] = off
pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3
chdir = /

Save the file and restart the php-fpm daemon.

service php-fpm restart

Check to see if the newdomain.com has a running process now:

ps aux|grep newdomain.com

If you do see it running under its own user process, go on to step 2.

Step 2, Configure nginx

Edit your nginx configuration to add the new website with its own socket. There are different styles of configuration for nginx on different Unix-like platforms but this author’s own preference is to simply add a line at the bottom of the default config file that tells nginx to also read all configurations in the /usr/local/etc/nginx/sites-enabled directory. Then create a new configuration file for each website, making it easy to add and remove many websites at a time.

nano /usr/local/etc/nginx.conf

Append the following entry somewhere in this config file:

include sites-enabled/*;

A sample configuration is below, customize it to meet your needs and the needs of your application. This example has extra bits added to help a WordPress website run little better (fixes broken permalink issues). The SSL configuration below should be #commented out for now but will be used later.

Pay special attention to all the “newdomain.com” entries especially the “fastcgi_pass unix:/var/run/php-fpm-newdomain.com.sock;” entry near the end of the configuration which ensures nginx has a unique Unix socket for each php-fpm process hosting this website . Every user process will use a different socket to help with isolation.

Create a new nginx config file for this website:

nano /usr/local/etc/nginx/sites-enabled/newdomain.com

Then, add an entry such as the one below:

server {
    listen 80;
#    listen 443 ssl;


    root /usr/local/www/nginx/newdomain.com;
    index index.php index.html index.htm;


    server_name newdomain.com www.newdomain.com;


#    ssl_certificate      /root/.acme.sh/newdomain.com/newdomain.cer;
#    ssl_certificate_key  /root/.acme.sh/newdomain.com/newdomain.com.key;


#    ssl_session_cache    shared:SSL:1m;
#    ssl_session_timeout  5m;


    ssl_ciphers  HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers  on;


    gzip on;
    gzip_disable "msie6";
    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 5;
    gzip_buffers 16 8k;
    gzip_http_version 1.0;


    location / {
        #below is for WordPress permalinks
        try_files $uri $uri/ /index.php?$args;
    }


    #rewrite /wordpress to top level directory
    location ^~ /wordpress/(.+) {
      rewrite ^/wordpress/(.+)$ $1 last;
    }


    # Add trailing slash to */wp-admin requests.
    rewrite /wp-admin$ $scheme://$host$uri/ permanent;


    # Directives to send expires headers and turn off 404 error logging.
    location ~* ^.+\.(ogg|ogv|svg|svgz|eot|otf|woff|mp4|ttf|rss|atom|jpg|jpeg|gif|png|ico|zip|tgz|gz|rar|bz2|doc|xls|exe|ppt|tar|mid|midi|wav|bmp|rtf)$ {
           access_log off; log_not_found off; expires max;
    }

    location ~ \.php$ {
        try_files $uri =404;
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_pass unix:/var/run/php-fpm-newdomain.com.sock;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include fastcgi_params;
        fastcgi_read_timeout 540;
    }

    #add some caching info
    set $skip_cache 0;


    # POST requests and URLs with a query string should always go to PHP
    if ($request_method = POST) {
        set $skip_cache 1;
    }
}

Make a directory to store your web files that matches the above root directory.

mkdir /usr/local/www/nginx/newdomain.com
chown -R newdomain.com:newdomain.com newdomain.com

Restart nginx

service nginx restart

Step 3, Enable SSL/TLS

First install the acme.sh package to manage our free Let’s Encrypt keys.:

pkg install acme.sh

Issue a certificate. You need to first have the DNS A-record for the domain pointing to this server’s IP address before the next step will work correctly.

acme.sh --issue -d newdomain.com --webroot /usr/local/www/nginx/newdomain.com

If acme prints a key you have success.

Troubleshooting: If acme fails with an error and does not provide a key, it likely means either your DNS for the domain is not yet pointing to this server IP address or acme may not have write permissions in the directory. Note that acme uses Let’s Encrypt to generate the certificates and to prove ownership before issuing the cert, acme.sh creates a temporary web page to be served on port 80 that is created and deleted automatically.

With key in hand, update the nginx configuration to use the cert for this site. These simple steps using acme are a fast way to get a free SSL certificate for encrypted web traffic.

nano /usr/local/etc/nginx/nginx.com

If you used the above sample configuration, you can just uncomment the SSL lines after ensuring your keys are setup correctly, and make sure the SSL entries map to the location of the key and certificate.

Note about errors: there are at least 6 locations in the above nginx config file where you must replace “yourdomain.com” with your own website domain. It’s easy to forget the last entry, updating the socket, because even if that setting is missed the process for this may appear to run fine under another user but would not be able to read/write files in the correct website directory, causing all sorts of errors. In this article we are configuring php-fpm with nginx specifically to running each website in its own process so we have greater control over isolation. If the website is ever compromised by a bad actor hacker, this approach limits the damage zone on a multi-site server.

Restart nginx to take effect.

service nginx restart

At this point the website is fully functional. It just needs a database, an app and some content.

Step 4 Setup a database

The reader may need to setup a database and a database user for this website before installing the application. In this article we use MySQL 8 with WordPress which requires a specific type of password authentication in MySQL in order to function properly, and as of this writing the required authentication mysql_nation_password is not the default mechanism.

Login as root into MySQL and create a database and a new user/password similar to below:

mysql -u root -p
create database Frankenstein;
use Frankenstein;
CREATE USER 'ed'@'localhost' IDENTIFIED WITH mysql_native_password BY 'password';
GRANT ALL ON Frankenstein.* TO 'ed'@'localhost';
exit

Obviously replace “ed” and “password” with a username/password combination that works for you. The web application is now ready to be installed (WordPress, in this example).

If you are indeed running WordPress, remember to change permissions on the “wp-config.php” file to keep out those pesky intruders who may have compromised a nearby website on the same host. And if you’re not running WordPress, with other applications there is usually a similar config file with the database details saved in plain text so it’s important to set permissions on it to 600, which means read/write access for the website process only and no access by any other user or group.

chmod 600 /usr/local/www/nginx/newdomain.com/wp-config.php

If this is the only website you will ever need to create, have a good life, thanks for reading and goodbye. Otherwise continue on.

Step 5, Rinse and repeat

Now that we have a working configuration let’s summarize the steps needed for each subsequent website that needs to be added. A single server can host many dozens or even hundreds of websites and web applications provided you get acceptable performance based on their loads.

  1. Create a new group and a new user named xyz.com
  2. Create xyz.com config for php-fpm.d with a unique user, group, and socket
  3. Restart php-fpm
  4. Edit nginx config, add an entry for xyz.com, port 80 only
  5. Restart nginx
  6. Point the website’s DNS A-record to this server’s IP address
  7. Install SSL certificate, update nginx config for SSL, restart nginx
  8. Create database and db user
  9. Install application
  10. Test everything

That’s it! Appreciate any comments or feedback on this article.

-Kelly