This article enhances a basic web server setup (Nginx, PHP and MySQL on FreeBSD) with additional security. The goal is to host many different websites on the same system in a way that minimizes the threat exposure should one or more website applications get compromised. We want to limit lateral movement and disallow exploited apps from affecting other websites on the same server.
Goals and Objectives
We focus on least privilege and isolation. We use php-fpm and nginx to host a website in isolation by running each site under its own user process instead of having all websites running under the same process, which is much more common and less secure. The problem with the common method without isolation is when one website is compromised, the attacker can potentially traverse directories to read and write files across all websites on the server’s filesystem. Isolating separate users and providing less system access provides better security. The approach was inspired by the DigitalOcean article on the same topic but written for Linux, so I wanted to make a FreeBSD version for the benefit of all FreeBSD users.
We’ll then go a step further by making SSL easy with acme.sh and free SSL certificates for all hosted websites from Let’s Encrypt to ensure all traffic to and from the website can be encrypted. Encryption is a basic requirement now on the web. The use of acme.sh to simplify the process of installing certificates is inpired by an article on Devopscraft. They make it easy for every website to support strong encryption and even force all traffic to use encryption if this is desired.
Note: most modern versions of FreeBSD, nginx, php-fpm and mysql will work just fine with this configuration. Most often, the newest versions are the best ones to run on a web server. At the time this article was published the system was running FreeBSD 12, nginx 1.22, PHP 8.2, and MySQL 8.0.
Step 1, Configure nginx and php-fpm with user-specific sockets
This article assumes you have nginx and php-fpm already installed and running, and ideally MySQL too. If you don’t have the prerequisites, don’t worry it’s easy. Start using “pkg add” for each of the applications, make each service to start automatically and then secure the installation. There are some good articles on getting a basic nginx/php-fpm/mysql set up on FreeBSD (examples: 1, 2, 3 – these are similar, so for those in a hurry just read the first one). After you have a basic setup, come back here to further secure the environment.
The key to this article is making each website run under its own user process in FreeBSD so if the site is compromised, the attacker will not have the ability to affect other sites on the same server.
We first add a group and user on the system for the new website, making sure the user has “nologin” privileges when prompted for the shell type, no password is set, and the account is locked after creation. Then we create a new configuration file in the php-fpm.d directory which will be automatically loaded when php-fpm is run or restarted.
You obviously must change the “newdomain.com” used in all parts of this article to the domain name you looking to host. All commands below are to be performed as a root user or sudo.
pw groupadd newdomain.com
adduser newdomain.comFor the “adduser” command you would answer the questions like this:
Username: newdomain.com
Full name:
Uid (Leave empty for default):
Login group [newdomain.com]:
Login group is newdomain.com. Invite newdomain.com into other groups? []:
Login class [default]: nologin
Shell (sh csh tcsh bash rbash nologin) [sh]: nologin
Home directory [/home/newdomain.com]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]: no
Lock out the account after creation? [no]: yes
Username : newdomain.com
Password : <disabled>
Full Name :
Uid : 1007
Class : nologin
Groups : newdomain.com
Home : /home/newdomain.com
Home Mode :
Shell : /usr/sbin/nologin
Locked : yes
OK? (yes/no): yesNow let’s create the site’s configuration profile.
cd /usr/local/etc/php-fpm.d
nano newdomain.com.confYou create a new config file for each domain being hosted, knowing that all configuration files in this directory with a *.conf suffix are loaded by the php-fpm daemon automatically when the application is started and restarted. A standard php-fpm installation on FreeBSD 12 has set this behaviour in the default configuration file.
[newdomain.com]
user = newdomain.com
group = newdomain.com
listen = /var/run/php-fpm-newdomain.com.sock
listen.owner = www
listen.group = www
php_admin_value[disable_functions] = exec,passthru,shell_exec,system
php_admin_flag[allow_url_fopen] = off
pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3
chdir = /Save the file and restart the php-fpm daemon.
service php-fpm restartCheck to see if the newdomain.com has a running process now:
ps aux|grep newdomain.comIf you do see it running under its own user process, go on to step 2.
Step 2, Configure nginx
Edit your nginx configuration to add the new website with its own socket by following these instructions. There are different styles of configuration for nginx on different Unix-like platforms but this author’s own preference is to simply add a line at the bottom of the default config file that tells nginx to also read all configurations in the /usr/local/etc/nginx/sites-enabled directory. Then create a new configuration file for each website, making it easy to add and remove many websites at a time.
nano /usr/local/etc/nginx.confAppend the following entry somewhere in this config file:
include sites-enabled/*;A sample configuration is below, customize it to meet your needs and the needs of your application. This example has extra bits added to help a WordPress website run little better (fixes broken permalink issues). The SSL configuration below should be #commented out for now but will be used later.
Pay special attention to all the “newdomain.com” entries especially the “fastcgi_pass unix:/var/run/php-fpm-newdomain.com.sock;” entry near the end of the configuration which ensures nginx has a unique Unix socket for each php-fpm process hosting this website . Every user process will use a different socket to help with isolation.
Create a new nginx config file for this website:
nano /usr/local/etc/nginx/sites-enabled/newdomain.comThen, add an entry such as the one below, keeping the commented items as well for now:
server {
listen 80;
# listen 443 ssl;
root /usr/local/www/nginx/newdomain.com;
index index.php index.html index.htm;
server_name newdomain.com www.newdomain.com;
# ssl_certificate /root/.acme.sh/newdomain.com/newdomain.com.cer;
# ssl_certificate_key /root/.acme.sh/newdomain.com/newdomain.com.key;
# ssl_session_cache shared:SSL:1m;
# ssl_session_timeout 5m;
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
gzip on;
gzip_disable "msie6";
gzip_vary on;
gzip_proxied any;
gzip_comp_level 5;
gzip_buffers 16 8k;
gzip_http_version 1.0;
location / {
#below is for WordPress permalinks
try_files $uri $uri/ /index.php?$args;
}
#rewrite /wordpress to top level directory
location ^~ /wordpress/(.+) {
rewrite ^/wordpress/(.+)$ $1 last;
}
# Add trailing slash to */wp-admin requests.
rewrite /wp-admin$ $scheme://$host$uri/ permanent;
# Directives to send expires headers and turn off 404 error logging.
location ~* ^.+\.(ogg|ogv|svg|svgz|eot|otf|woff|mp4|ttf|rss|atom|jpg|jpeg|gif|png|ico|zip|tgz|gz|rar|bz2|doc|xls|exe|ppt|tar|mid|midi|wav|bmp|rtf)$ {
access_log off; log_not_found off; expires max;
}
location ~ \.php$ {
try_files $uri =404;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_pass unix:/var/run/php-fpm-newdomain.com.sock;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
fastcgi_read_timeout 540;
}
#add some caching info
set $skip_cache 0;
# POST requests and URLs with a query string should always go to PHP
if ($request_method = POST) {
set $skip_cache 1;
}
}Make a directory to store your web files that matches the above root directory.
It’s really important that all files for this domain are owned by the user process. Without this step there is no point in doing user isolation. No user other than the website user process should be able to write files in the website’s directory.
mkdir /usr/local/www/nginx/newdomain.com
chown -R newdomain.com:newdomain.com /usr/local/www/nginx/newdomain.comRestart nginx
service nginx restartStep 3, Enable SSL/TLS
First install the acme.sh package to manage our free Let’s Encrypt keys.:
pkg install acme.shIssue a certificate. You need to first have the DNS A-record for the domain pointing to this server’s IP address before the next step will work correctly.
acme.sh --issue -d newdomain.com --webroot /usr/local/www/nginx/newdomain.comIf acme prints a key you have success.
Troubleshooting: If acme fails with an error and does not provide a key, it likely means either your DNS for the domain is not yet pointing to this server IP address or acme may not have write permissions in the directory. Note that acme uses Let’s Encrypt to generate the certificates and to prove ownership before issuing the cert, acme.sh creates a temporary web page to be served on port 80 that is created and deleted automatically.
Need a wildcard certificate instead? And happen to be using Cloudflare for DNS? Here’s a good article on how to get an API key from Cloudflare so acme.sh, with more updated info on the Cloudflare API token configuration here on Github.
With SSL key in hand, update the nginx configuration to use the cert for this site. These simple steps using acme are a fast way to get a free SSL certificate for encrypted web traffic.
nano /usr/local/etc/nginx/sites-enabled/newdomain.comIf you used the above sample configuration, you can just uncomment the SSL lines after ensuring your keys are setup correctly, and make sure the SSL entries map to the location of the key and certificate.
Note about errors: there are at least 6 locations in the above nginx config file where you must replace “yourdomain.com” with your own website domain. It’s easy to forget the last entry, updating the socket, because even if that setting is missed the process for this may appear to run fine under another user but would not be able to read/write files in the correct website directory, causing all sorts of errors. In this article we are configuring php-fpm with nginx specifically to running each website in its own process so we have greater control over isolation. If the website is ever compromised by a bad actor hacker, this approach limits the damage zone on a multi-site server.
Restart nginx to take effect.
service nginx restartAt this point the website is fully functional. It just needs a database, an app and some content.
Step 4 Setup a database
The reader may need to setup a database and a database user for this website before installing the application. In this article we use MySQL 8 with WordPress which requires a specific type of password authentication in MySQL in order to function properly, and as of this writing the required authentication mysql_nation_password is not the default mechanism.
Login as root into MySQL and create a database and a new user/password similar to below:
mysql -u root -p
create database Frankenstein;
use Frankenstein;
CREATE USER 'ed'@'localhost' IDENTIFIED WITH mysql_native_password BY 'password';
GRANT ALL ON Frankenstein.* TO 'ed'@'localhost';
exitObviously replace “ed” and “password” with a username/password combination that works for you. The web application is now ready to be installed (WordPress, in this example).
If you are indeed running WordPress, remember to change permissions on the “wp-config.php” file to keep out those pesky intruders who may have compromised a nearby website on the same host. And if you’re not running WordPress, with other applications there is usually a similar config file with the database details saved in plain text so it’s important to set permissions on it to 600, which means read/write access for the website process only and no access by any other user or group.
chmod 600 /usr/local/www/nginx/newdomain.com/wp-config.phpIf this is the only website you will ever need to create, have a good life, thanks for reading and goodbye. Otherwise continue on.
Step 5, Rinse and repeat
Now that we have a working configuration let’s summarize the steps needed for each subsequent website that needs to be added. A single server can host many dozens or even hundreds of websites and web applications provided you get acceptable performance based on their loads.
- Create a new group and a new user named xyz.com
- Create xyz.com config for php-fpm.d with a unique user, group, and socket
- Restart php-fpm
- Edit nginx config, add an entry for xyz.com, port 80 only
- Restart nginx
- Point the website’s DNS A-record to this server’s IP address
- Install SSL certificate, update nginx config for SSL, restart nginx
- Create database and db user
- Install application
- Test everything
That’s it! Appreciate any comments or feedback on this article.
-Kelly