I was excited to try GCP (Google Compute Platform) with FreeBSD since community images are available to deploy. DigitalOcean stopped supporting FreeBSD so I’ve moved on to GCP.
In GCP’s Compute engine, if you need multiple interfaces, you must create a VM instance with multiple NICs at creation time — you cannot add a second NIC after the VM has been provisioned. At the same time I created multiple ephemeral public IP addresses too so all interfaces can serve traffic to the public internet. But no matter what I tried, I could not make a second NIC send/receive traffic. I searched forums and read articles but just could not determine how to make it work, all the guides I found were for Linux. Google’s guide in GCP is for Linux and does not work with FreeBSD. The way Linux and FreeBSD support multiple interfaces is quite different. This article might help fill that gap.
FreeBSD supports multiple default routes but only one works by default. First let’s check how many routing tables are available in your kernel using the command:
sysctl -n net.fibsDefault is just one. A custom kernel is not required since support for multiple routing tables was added into the generic kernel way back in 2012. However it must be configured in /boot/loader.conf. We need to append an entry as follows so there are at least two FIBS, so we can have one default route for each interface.
echo 'net.fibs=2' >> /boot/loader.confYou will need to reboot to activate it in the kernel.
For the purposes of this article, our two network interfaces are vtnet0 and vtnet1. Both interfaces are 10G connections getting IPv4 addresses from DHCP. The first interface will work just fine with a default route but the second NIC will require additional configuration.
Our goal is to get Apache running on the second NIC because we already have Nginx running on the first interface on TCP port 80 and 443.
Edit /etc/rc.conf to tell Apache to startup at boot time and to use FIB 1 instead of FIB 0. Note the FreeBSD man page for rc.conf clearly shows <name>_fib is the syntax to set the setfib value to run a service under:
apache24_enable="YES"
apache24_fib="1"For this example, our primary interface is vtnet0 and is given an address of 10.188.0.3 by DHCP with a default route of 10.188.0.1. Our secondary NIC is given an address of 10.189.0.2 from DHCP as well, but the default route given to it will not work. It will need a default route of 10.189.0.1.
We’re going to manually configure the network on our second NIC, vtnet1 as follows. The wonderful setfib command is the key here, I first learned about it from this post on the FreeBSD forums, plus this one. The setfib utility runs another utility with a different routing table. It lets us add a second default route for the network on this interface once it has an address:
#ifconfig vtnet1 10.189.0.2 netmask 255.255.255.0 broadcast 10.189.0.255 mtu 1430
#setfib 1 route add 10.189.0.2/24 -iface vtnet1
#setfib 1 route add default 10.189.0.1You can verify the second default route is now working by comparing the output of setfib 0 netstat -ar, which displays routing information for the vtnet0, versus setfib 1 netstat -ar, which displays routing information for vtnet1. In our example we now have two different default routes, one for each network and interface:
#setfib 0 netstat -ar
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default 10.188.0.1 UGS vtnet0
10.188.0.0/20 10.188.0.1 UGS vtnet0
10.188.0.1 link#1 UHS vtnet0
venus.northamerica link#1 UH lo0
10.189.0.0/24 link#2 U vtnet1
10.189.0.2 link#2 UHS lo0
localhost link#3 UH lo0
#setfib 1 netstat -ar
Routing tables (fib: 1)
Internet:
Destination Gateway Flags Netif Expire
default 10.189.0.1 UGS vtnet1
10.189.0.0/24 link#2 US vtnet1
localhost link#3 UHS lo0
We must also configure Apache’s httpd.conf to only listen on address 10.189.0.2. To do that, we edit /usr/local/etc/apache24/httpd.conf so there’s an entry that looks like this:
Listen 10.189.0.2:80Restart Apache and see if you can connect to it externally. If everything is working, make the change permanent by adding the following two entries to /etc/rc.local so they get run at startup:
setfib 1 route add 10.189.0.2/24 -iface vtnet1
setfib 1 route add default 10.189.0.1Reboot and double check both default routes are still working correctly. Most likely the second interface was already getting an IPv4 address so we didn’t need to an an ifconfig entry in /etc/rc.conf (here are some posts with examples), we only need the above two setfib commands to add the network and default route.
That’s it. As simple as that!
Obviously, once you have Apache listening on vtnet1, port 80, you should then enable SSL by setting up a TLS certificate then force all traffic onto TCP port 443, then perhaps even close port 80 at the firewall when it’s no longer needed since all web traffic should be encrypted nowadays. Those steps are beyond the scope of this article, but I’ve written about how to do exactly that with FreeBSDand Nginx here and it’s a very similar with Apache. Most people will start by editing /usr/local/etc/apache24/extra/httpd-ssl.conf then use something like Acme.sh with Let’s Encrpt to provide the certificate and keep renewing it when it expires.
Conclusion
As you can see, having multiple interfaces with multiple default routes in FreeBSD is not only possible, it’s not even that difficult once you know how to do it. Most of the time the excellent FreeBSD Handbook tells us almost everything we need to know about system administration. But in this case I couldn’t find the documentation and it took me a while to figure out what setfib was.
Other uses
Multiple interfaces can be used for so many things. Yet the setup is different on FreeBSD than on Linux or Windows so examples are often the best way to learn.
One example is running the SSH daemon on multiple interfaces at the same time. This article from way back in 2011 explains how to do it, basically running multiple instances of SSH on different FIBS. Since that article is old, note that a custom kernel is no longer required.
Another example is running a VPN on one interface. Here’s a discussion on the FreeBSD forums about that, from way back in 2015.
Yet another example with different vlans.
Comments welcome.
Kelly
Leave a Reply