GCP nearline storage is good for file level backups as it’s relatively inexpensive. At the time of this writing, 100GB of file storage on nearline costs under $2/month. After setting up restic, we’ll create a script and a cronjob to run it daily; restic only updates the files that change.

We can use restic to restore individual files and directories. It’s a nice way to have file-level backup and restore capability and is a great addition to any system. File-level backups can work in conjunction with other data protection approaches such as snapshots of the VM which are very easy to configure in GCP on a schedule.

1. GCP Bucket & Service Account Setup

1. Create a Cloud Storage Bucket in the GCP Console and set the default class to Nearline.
2. Navigate to IAM & Admin > Service Accounts and create a service account named restic-backup.
3. Assign the Storage Object Admin role for the newly created bucket.
4. Generate and download the JSON key file. Upload it to your FreeBSD server at /etc/restic-gcp-key.json.
5. Set restrictive permissions on the key file:

chmod 600 /etc/restic-gcp-key.json

2. Install & Configure

Install Restic via the FreeBSD package manager:

pkg install restic

Create the environment configuration file /etc/restic.env. Update your JSON key and bucket name in the config below! And note, we prefix the variables with export so they are properly exposed to the environment when sourced by sh scripts:

export GOOGLE_APPLICATION_CREDENTIALS="/etc/restic-gcp-key.json"
export RESTIC_REPOSITORY="gs:your-bucket-name:/sys-backup"
export RESTIC_PASSWORD="YourStrongPasswordHere"

Lock down permissions and initialize the remote repository (run only once). Because FreeBSD’s default interactive shell is often csh (which does not support the . operator), wrap interactive commands in a sh -c subshell so the required variables are loaded just when the tool is run:

chmod 700 /etc/restic.env
sh -c '. /etc/restic.env && restic init'

3. Automate A Backup Script

Create the backup script and located it wherever you keep you scripts, for this article we’ll put the script in /usr/local/libexec/run-restic-backup.sh. Note that even if your user shell is csh, this script natively uses #!/bin/sh, so sourcing the `.env` file directly will work normally here.

We’re going to backup all the files on the server except a few areas we need to exclude. Adjust as required if you only need to backup a subset of the entire server.

#!/bin/sh
set -e

# Load environment configuration
. /etc/restic.env

# Run full system backup targeting /
restic backup \
  --one-file-system \
  --exclude="/tmp/*" \
  --exclude="/var/tmp/*" \
  --exclude="/usr/obj/*" \
  --exclude="/dev/*" \
  --exclude="/proc/*" \
  /

# Prune old snapshots (keep 7 daily, 4 weekly)
restic forget --keep-daily 7 --keep-weekly 4 --prune

Make the script executable and schedule it via /etc/crontab:

chmod 700 /usr/local/libexec/run-restic-backup.sh

0  2  *  *  *  root  /usr/local/libexec/run-restic-backup.sh > /var/log/restic-backup.log 2>&1

4. Operational Commands (For Interactive csh Users)

When running these manually in a csh terminal, we will wrap them in an sh subshell so that the /etc/restic.env gets loaded just for each restic command invocation.

That’s it. Monitor your logs at /var/log/restic-backup.log periodically to ensure backups keep continuing automatically. Quick and easy, we are done.

In GCP’s Compute engine, if you need multiple interfaces, you must create a VM instance with multiple NICs at creation time — you cannot add a second NIC after the VM has been provisioned. At the same time I created multiple ephemeral public IP addresses too so all interfaces can serve traffic to the public internet. But no matter what I tried, I could not make a second NIC send/receive traffic. I searched forums and read articles but just could not determine how to make it work, all the guides I found were for Linux. Google’s guide in GCP is for Linux and does not work with FreeBSD. The way Linux and FreeBSD support multiple interfaces is quite different. This article might help fill that gap.

FreeBSD supports multiple default routes but only one works by default. First let’s check how many routing tables are available in your kernel using the command:

sysctl -n net.fibs

Default is just one. A custom kernel is not required since support for multiple routing tables was added into the generic kernel way back in 2012. However it must be configured in /boot/loader.conf. We need to append an entry as follows so there are at least two FIBS, so we can have one default route for each interface.

echo 'net.fibs=2' >> /boot/loader.conf

You will need to reboot to activate it in the kernel.

For the purposes of this article, our two network interfaces are vtnet0 and vtnet1. Both interfaces are 10G connections getting IPv4 addresses from DHCP. The first interface will work just fine with a default route but the second NIC will require additional configuration.

Our goal is to get Apache running on the second NIC because we already have Nginx running on the first interface on TCP port 80 and 443.

Edit /etc/rc.conf to tell Apache to startup at boot time and to use FIB 1 instead of FIB 0. Note the FreeBSD man page for rc.conf clearly shows <name>_fib is the syntax to set the setfib value to run a service under:

apache24_enable="YES"
apache24_fib="1"

For this example, our primary interface is vtnet0 and is given an address of 10.188.0.3 by DHCP with a default route of 10.188.0.1. Our secondary NIC is given an address of 10.189.0.2 from DHCP as well, but the default route given to it will not work. It will need a default route of 10.189.0.1.

We’re going to manually configure the network on our second NIC, vtnet1 as follows. The wonderful setfib command is the key here, I first learned about it from this post on the FreeBSD forums, plus this one. The setfib utility runs another utility with a different routing table. It lets us add a second default route for the network on this interface once it has an address:

#ifconfig vtnet1 10.189.0.2 netmask 255.255.255.0 broadcast 10.189.0.255 mtu 1430
#setfib 1 route add 10.189.0.2/24 -iface vtnet1
#setfib 1 route add default 10.189.0.1

You can verify the second default route is now working by comparing the output of setfib 0 netstat -ar, which displays routing information for the vtnet0, versus setfib 1 netstat -ar, which displays routing information for vtnet1. In our example we now have two different default routes, one for each network and interface:

#setfib 0 netstat -ar
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.188.0.1         UGS      vtnet0
10.188.0.0/20      10.188.0.1         UGS      vtnet0
10.188.0.1         link#1             UHS      vtnet0
venus.northamerica link#1             UH          lo0
10.189.0.0/24      link#2             U        vtnet1
10.189.0.2         link#2             UHS         lo0
localhost          link#3             UH          lo0

#setfib 1 netstat -ar
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.189.0.1         UGS      vtnet1
10.189.0.0/24      link#2             US       vtnet1
localhost          link#3             UHS         lo0

We must also configure Apache’s httpd.conf to only listen on address 10.189.0.2. To do that, we edit /usr/local/etc/apache24/httpd.conf so there’s an entry that looks like this:

Listen 10.189.0.2:80

Restart Apache and see if you can connect to it externally. If everything is working, make the change permanent by adding the following two entries to /etc/rc.local so they get run at startup:

setfib 1 route add 10.189.0.2/24 -iface vtnet1
setfib 1 route add default 10.189.0.1

Reboot and double check both default routes are still working correctly. Most likely the second interface was already getting an IPv4 address so we didn’t need to an an ifconfig entry in /etc/rc.conf (here are some posts with examples), we only need the above two setfib commands to add the network and default route.

That’s it. As simple as that!

Obviously, once you have Apache listening on vtnet1, port 80, you should then enable SSL by setting up a TLS certificate then force all traffic onto TCP port 443, then perhaps even close port 80 at the firewall when it’s no longer needed since all web traffic should be encrypted nowadays. Those steps are beyond the scope of this article, but I’ve written about how to do exactly that with FreeBSDand Nginx here and it’s a very similar with Apache. Most people will start by editing /usr/local/etc/apache24/extra/httpd-ssl.conf then use something like Acme.sh with Let’s Encrpt to provide the certificate and keep renewing it when it expires.

Conclusion

As you can see, having multiple interfaces with multiple default routes in FreeBSD is not only possible, it’s not even that difficult once you know how to do it. Most of the time the excellent FreeBSD Handbook tells us almost everything we need to know about system administration. But in this case I couldn’t find the documentation and it took me a while to figure out what setfib was.

Other uses

Multiple interfaces can be used for so many things. Yet the setup is different on FreeBSD than on Linux or Windows so examples are often the best way to learn.

One example is running the SSH daemon on multiple interfaces at the same time. This article from way back in 2011 explains how to do it, basically running multiple instances of SSH on different FIBS. Since that article is old, note that a custom kernel is no longer required.

Another example is running a VPN on one interface. Here’s a discussion on the FreeBSD forums about that, from way back in 2015.

Yet another example with different vlans.

Comments welcome.

Kelly

This article is for building a web server that isolates multiple websites to reduce threat exposure. Web applications are commonly vulnerable to compromise if they are not kept up-to-date. The goal is to limit lateral movement, keeping other websites and systems on that server unaffected. Busy servers running many website domains can continue to function. We will use native FreeBSD sockets and user and group permissions to prevent a compromised domain from having access to the filesystems of any other web domains or the host itself.

Objectives, plus the how and why

We want to use php-fpm and nginx on FreeBSD, with php-fpm configured to run each web domain as a unique user process. PHP with php-fpm and Nginx will communicate using a FreeBSD socket.

It’s far more common to see environments where many websites run under one user process. The problem with that approach is when one website is compromised, the attacker may traverse directories to read and write files across all websites on the server, risking further compromise and greater exposure of confidential information and/or personal data from other organizations and individuals hosting websites on that server. We can do better.

This approach was inspired by an article on the same topic but written for Linux, so I made a FreeBSD version. It works on any currently supported FreeBSD version.

We’ll make SSL certificates look easy with acme.sh, then finally we’ll install a simple Tripwire-like filesystem monitor known as AIDE to watch for filesystem changes to specific parts of the system.

The use of acme.sh to simplify the process of installing certificates is inpired by an article on Devopscraft. The tool makes it easy for all websites to have encryption and one can even force all traffic to use encryption if this is desired.

Note: At the time of writing the versions used were FreeBSD 13.2, nginx 1.24, PHP 8.2.x, MySQL 8.x, Acme.sh 3.x, AIDE 0.17.

Step 1, Setup nginx and php-fpm with a unique user, group and socket

If you don’t have nginx or php installed yet, let’s get started. In this article the # symbol represents commands that must be run as root or sudo.

#pkg install nginx
#pkg install php82
#pkg install php82-extensions

If you need help getting these configured and starting after every reboot, there are some good articles on getting a basic nginx/php-fpm/mysql set up using FreeBSD (examples: 1, 2, 3 – these are all similar, so for those in a hurry just read the first one).

After you have a basic setup of Nginx and PHP-FPM, it’s time to secure the environment. The key to have each website run under its own user process so if one site is compromised due to an application vulnerability, the attacker will have a more difficult time affecting other sites on the same server.

Frst, add a new group and user on the system for the new website domain, making sure the user has “nologin” privileges when prompted for the shell type, no password is set, and the account is locked after creation.

Change the “newdomain.com” used in all parts of this article to your app’s domain. All commands below are performed as a root or sudo.

#pw groupadd newdomain.com
#adduser newdomain.com

For the “adduser” command you would answer the questions like this:

Username: newdomain.com
Full name: 
Uid (Leave empty for default): 
Login group [newdomain.com]: 
Login group is newdomain.com. Invite newdomain.com into other groups? []: 
Login class [default]: nologin
Shell (sh csh tcsh bash rbash nologin) [sh]: nologin
Home directory [/home/newdomain.com]: 
Home directory permissions (Leave empty for default): 
Use password-based authentication? [yes]: no
Lock out the account after creation? [no]: yes
Username   : newdomain.com
Password   : <disabled>
Full Name  : 
Uid        : 1007
Class      : nologin
Groups     : newdomain.com 
Home       : /home/newdomain.com
Home Mode  : 
Shell      : /usr/sbin/nologin
Locked     : yes
OK? (yes/no): yes

Now let’s create the site’s configuration profile.

#cd /usr/local/etc/php-fpm.d
#nano newdomain.com.conf

You create a new config file for each domain being hosted, knowing that all configuration files in this directory with a *.conf suffix are loaded by the php-fpm daemon automatically when the application is started and restarted. A standard php-fpm installation on FreeBSD 12 has set this behaviour in the default configuration file.

[newdomain.com]
user = newdomain.com
group = newdomain.com
listen = /var/run/php-fpm-newdomain.com.sock
listen.owner = www
listen.group = www
php_admin_value[disable_functions] = exec,passthru,shell_exec,system
php_admin_flag[allow_url_fopen] = off
pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3
chdir = /

Save the file and restart the php-fpm daemon.

#service php_fpm restart

Check to see if the newdomain.com has a running process now:

#ps aux|grep newdomain.com

If you do see it running under its own user process, go on to step 2.

Step 2, Configure nginx

Edit your nginx configuration to add the new website with its own socket by following these instructions. There are different styles of configuration for nginx on different Unix-like platforms but this author’s own preference is to simply add a line at the bottom of the default config file that tells nginx to also read all configurations in the /usr/local/etc/nginx/sites-enabled directory. Then create a new configuration file for each website, making it easy to add and remove many websites at a time.

#nano /usr/local/etc/nginx.conf

Append the following entry somewhere in this config file:

#include sites-enabled/*;

A sample configuration is below, customize it to meet your needs and the needs of your application. This example has extra bits added to help a WordPress website run little better (fixes broken permalink issues). The SSL configuration below should be #commented out for now but will be used later.

Pay special attention to all the “newdomain.com” entries especially the “fastcgi_pass unix:/var/run/php-fpm-newdomain.com.sock;” entry near the end of the configuration which ensures nginx has a unique Unix socket for each php-fpm process hosting this website . Every user process will use a different socket to help with isolation.

Create a new nginx config file for this website:

#nano /usr/local/etc/nginx/sites-enabled/newdomain.com

Then, add an entry such as the one below, keeping the commented items as well for now:

server {
    listen 80;
#    listen 443 ssl;


    root /usr/local/www/nginx/newdomain.com;
    index index.php index.html index.htm;


    server_name newdomain.com www.newdomain.com;


#    ssl_certificate      /root/.acme.sh/newdomain.com/newdomain.com.cer;
#    ssl_certificate_key  /root/.acme.sh/newdomain.com/newdomain.com.key;


#    ssl_session_cache    shared:SSL:1m;
#    ssl_session_timeout  5m;


    ssl_ciphers  HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers  on;


    gzip on;
    gzip_disable "msie6";
    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 5;
    gzip_buffers 16 8k;
    gzip_http_version 1.0;


    location / {
        #below is for WordPress permalinks
        try_files $uri $uri/ /index.php?$args;
    }


    #rewrite /wordpress to top level directory
    location ^~ /wordpress/(.+) {
      rewrite ^/wordpress/(.+)$ $1 last;
    }


    # Add trailing slash to */wp-admin requests.
    rewrite /wp-admin$ $scheme://$host$uri/ permanent;


    # Directives to send expires headers and turn off 404 error logging.
    location ~* ^.+\.(ogg|ogv|svg|svgz|eot|otf|woff|mp4|ttf|rss|atom|jpg|jpeg|gif|png|ico|zip|tgz|gz|rar|bz2|doc|xls|exe|ppt|tar|mid|midi|wav|bmp|rtf)$ {
           access_log off; log_not_found off; expires max;
    }

    location ~ \.php$ {
        try_files $uri =404;
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_pass unix:/var/run/php-fpm-newdomain.com.sock;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include fastcgi_params;
        fastcgi_read_timeout 540;
    }

    #add some caching info
    set $skip_cache 0;


    # POST requests and URLs with a query string should always go to PHP
    if ($request_method = POST) {
        set $skip_cache 1;
    }
}

Make a directory to store your web files that matches the above root directory.

It’s really important that all files for this domain are owned by the user process. Without this step there is no point in doing user isolation. No user other than the website user process should be able to write files in the website’s directory.

#mkdir /usr/local/www/nginx/newdomain.com
#chown -R newdomain.com:newdomain.com /usr/local/www/nginx/newdomain.com

Restart nginx

#service nginx restart

Step 3, Enable SSL/TLS

First install the acme.sh package to manage our free Let’s Encrypt keys.:

#pkg install acme.sh

Issue a certificate. You need to first have the DNS A-record for the domain pointing to this server’s IP address before the next step will work correctly.

#acme.sh --issue -d newdomain.com --webroot /usr/local/www/nginx/newdomain.com

If acme prints a key you have success.

Troubleshooting: If acme fails with an error and does not provide a key, it likely means either your DNS for the domain is not yet pointing to this server IP address or acme may not have write permissions in the directory. Note that acme uses Let’s Encrypt to generate the certificates and to prove ownership before issuing the cert, acme.sh creates a temporary web page to be served on port 80 that is created and deleted automatically.

Need a wildcard certificate instead? And happen to be using Cloudflare for DNS? Here’s a good article on how to get an API key from Cloudflare so acme.sh, with more updated info on the Cloudflare API token configuration here on Github.

With SSL key in hand, update the nginx configuration to use the cert for this site. These simple steps using acme are a fast way to get a free SSL certificate for encrypted web traffic.

#nano /usr/local/etc/nginx/sites-enabled/newdomain.com

If you used the above sample configuration, you can just uncomment the SSL lines after ensuring your keys are setup correctly, and make sure the SSL entries map to the location of the key and certificate.

Note about errors: there are at least 6 locations in the above nginx config file where you must replace “yourdomain.com” with your own website domain. It’s easy to forget the last entry, updating the socket, because even if that setting is missed the process for this may appear to run fine under another user but would not be able to read/write files in the correct website directory, causing all sorts of errors. In this article we are configuring php-fpm with nginx specifically to running each website in its own process so we have greater control over isolation. If the website is ever compromised by a bad actor hacker, this approach limits the damage zone on a multi-site server.

Restart nginx to take effect.

#service nginx restart

At this point the website is fully functional. It just needs a database, an app and some content.

Step 4, update your user permissions

Let’s make your user can edit all those wonderful websites that will be hosted. Regular users should be able to edit the files their own websites, but only if you add them to the domain’s group otherwise they will not be able to write to the directory.

Add the user robert to the group of each domain newdomain.com like this:

pw group mod newdomain.com -m robert

Step 5 Secure the app’s config files

If you are indeed running WordPress, remember to change permissions on the “wp-config.php” file to keep out those pesky intruders who may have compromised a nearby website on the same host.

If you’re not running WordPress, with other applications there is usually a similar config file with the database details saved in plain text so it’s important to set permissions on it to 600, which means read/write access for the website process only and no access by any other user or group.

#chmod 600 /usr/local/www/nginx/newdomain.com/wp-config.php

If this is the only website you will ever need to create, have a good life, thanks for reading and goodbye. Otherwise continue on.

Step 6, Create a database for the app

Fortunately, I wrote short article on using MySQL from the command line.

If you have not yet installed MySQL, use the pkg system on FreeBSD and secure the installation berfore creating a database. There are many articles on the web about how to do this.

Step 7, Rinse and repeat

Now rapidly repeat the above steps for every website you want to add. You might have dozens.

1. Create a new group and a new user, such as xyz.com
2. Create xyz.com config for php-fpm.d with a unique user, group, and socket
3. Restart php-fpm
4. Edit nginx config, add an entry for xyz.com, port 80 only
5. Restart nginx
6. Point the website’s DNS A-record to this server’s IP address
7. Install SSL certificate, update nginx config for SSL, restart nginx
8. Create database and db user
9. Install application
10. Test the web app

Step 6, AIDE filesystem integrity check

Let’s get AIDE running, it’s simple:

#pkg install aide
#nano /usr/local/etc/aide.conf

Edit the config file and ensure your website directories are part of your configuration. At a bare minimum, as your websites:

/usr/local/www/nginx    R-tiger-rmd160-sha1

Then initialize AIDE. This will take a while depending on how many files/directories you have configured.

AIDE does not run as a daemon, run it manually or as a cron job. After it’s initialized, move the database from aide.db.new to aide.db, then create a test file and check to see if AIDE finds it:

#aide --init
#cd /var/db/aide/databases
#mv aide.db.new aide.db
#touch /usr/local/www/nginx/JUST_A_TEST
#aide --check

That’s it. Appreciate any comments or feedback. I update this article regularly.

You don’t need tools like MySQLAdmin for basic tasks like creating and deleting databases, adding users and adjusting permissions. This can easily be done from the command line. 

Basic tasks

Log into mysql:

#mysql –u root –p
password: ******

Welcome to MySQL monitor.
...
root@localhost [(none)]>

Create a database:

>create database Frankenstein;
>use Frankenstein;

Create a user and grant privileges:

>CREATE USER 'Frank'@'localhost' identified by 'Fr@nksP@s$w0rD';
>GRANT ALL ON Frankenstein.* TO 'Frank'@'localhost';

Then reload the grant tables in the database using flush privileges. If you forget this step you can also run “mysqladmin reload” to reload the grant tables.

>FLUSH PRIVILEGES;
>exit;

Change user password:

>ALTER USER 'user-name'@'localhost' IDENTIFIED BY 'NEW_USER_PASSWORD';
>FLUSH PRIVILEGES;

Grant special permissions to a user, such as the PROCESS permission, required for certain backup operations:

>GRANT PROCESS ON *.* TO 'Frank'@'localhost';

The above fixes a permissions change introduced a few years ago (around 2020, Covid era) into MySQL 8.0 and 5.7 that resulted in permission errors like the one below when doing backups:

mysqldump: Error: 'Access denied; you need (at least one of) the PROCESS privilege(s) for this operation' when trying to dump tablespaces

Backup and Recovery?

Backup the database often:

mysqldump –u Frank –p Frankenstein > frankenstein-backup-25Aug2023.sql

Restore the database on another system, or revert back if you need to: 

mysql –u Frank –p Frankenstein < frankenstein-db-backup-01Jan2023.sql

Database too big? 

Sometimes databases are much bigger than they should be.  Rogue apps and some WordPress plugins can create large tables of useless data.  Find the offending table by running a query like this within your Frankenstein database:

>SELECT
  TABLE_NAME AS `Table`,
  ROUND((DATA_LENGTH + INDEX_LENGTH) / 1024 / 1024) AS `Size (MB)`
FROM
  information_schema.TABLES
WHERE
  TABLE_SCHEMA = "Frankenstein"
ORDER BY
  (DATA_LENGTH + INDEX_LENGTH)
DESC;

For this and other sizing examples, thanks to AJ Welch for his article on Chartio.

Any other suggestions? Leave a comment.

The best article I found on the subject shows exactly how to do this on multiple platforms: various types of Linux and on FreeBSD. I’m reproducing the instructions below just for FreeBSD so I have this info handy for myself and anyone who might be visiting my FreeBSD-focused site.

First, install the postfix_sasl package. You need the sasl version for the TLS support. All commands below are run as root.

# pkg install postfix-sasl

Very important: if this is your first time installing postfix, follow the detail instructions given by the package installer if you want to permanently replace sendmail on your system.  You only need one active mail daemon.

Edit the postfix configuration file located at /usr/local/etc/postfix/main.cf and add the following under the relayhost section:

relayhost = [smtp.gmail.com]:587
smtp_use_tls = yes
smtp_sasl_auth_enable = yes
smtp_sasl_security_options =
smtp_sasl_password_maps = hash:/usr/local/etc/postfix/sasl_passwd
smtp_tls_CAfile = /etc/mail/certs/cacert.pem

Create a new file at /usr/local/etc/postfix/sasl_password for your Gmail username/password by entering one line into that file in the following format:

[smtp.gmail.com]:587    myemail@mydomain.com:MyP@$$w0rdHere

Fix permissions on this file:

# chmod 600 /usr/local/etc/postfix/sasl_passwd

Process the password file by issuing the following command:

# postmap /usr/local/etc/postfix/sasl_passwd

Now restart postfix and send a test message

# service postfix reload
# mail -s "Test subject" recipient@anywhere.com

That’s it, you’re done. It’s worthwhile to check the postfix logs at /var/log/maillog to see if there are any errors that you need to correct. Thanks for reading this article.

Kelly

Disclaimer: at the time of writing, I work for Oracle. I wrote this article for fun on my free time. FreeBSD isn’t supported on OCI so I just had to find a way to get it running anyway and use the capabilities of Oracle Cloud Infrastructure since I love the high-performance, elastic nature of OCI.

Before going through this article you need to get FreeBSD running on OCI, and it’s not hard. The easiest way is to create a VMDK (VMware virtual disk image) or QCOW2 VM image on your computer, upload it to OCI object storage, and then use that VM image to create a Custom Image in OCI  compute that can be launched whenever needed. The OCI documentation is great and the relevant section will walk you through this process, which is written about importing Linux images but works just fine for FreeBSD too, running in “emulation mode.”

Attaching block storage

First, provision some block storage in the OCI web console and attach it to your FreeBSD compute instance. We will then need to run OS commands inside the instance so the operating system can see the storage as an iSCSI device.

We’ll initially test the connection works without setting up CHAP authentication and then if successful we’ll add on these authentication credentials to improve our security, and finally make this all permanent by creating a configuration file to mount the block storage at boot.

The OCI web console provides iSCSI command documentation for Linux and also separates out the pertinent information we need to connect block storage in FreeBSD. All this information except the Linux commands will be needed. We need to provide the block storage IP address, port and the volume IQN from the web console using the following command, whereby the -A attach option is used in the following format:

# service iscsictld start
# iscsictl -A -p 10.10.10.10:port-t iqn.myIQNnumberHere

Note that the IQN that Oracle’s Cloud Infrastructure gives you does not have a :target0 at the end of it, so you do not need to add one – simply use the IQN that is provided along with the correct IP address and port shown in the OCI console. If the above command is successful, nothing will be output to the shell. To check if the iSCSI attach was successful we simply issue the iscsictl command on its own and you should see something like this:

# iscsictl
Target name         Target portal    State
iqn.myIQNnumberHere 10.10.10.10:3260 Connected: da0

This indicates a successful connection. If you do not have output similar to the above that shows the iSCSI device is now connected, and instead see errors like “connection lost” or “waiting” or “not found” or “authentication failed” then the FreeBSD Handbook page entitled iSCSI Initiator and Target Configuration has some troubleshooting ideas for you.

Now we need to mount the block storage into the filesystem. In our case we’ll mount the block storage as /usr/jails, feel free to pick another location if you have something else in mind.

Since we want the performance, data integrity, and pooled storage capabilities of the ZFS filesystem, we need to startup the service. Below I shamelessly reproduce a portion of the excellent quick start guide to ZFS from the Handbook because it’s perfectly written.

There is a startup mechanism that allows FreeBSD to mount ZFS pools during system initialization. To enable it, add this line to /etc/rc.conf:

zfs_enable="YES"

Then start the service:

# service zfs start

To create a simple, non-redundant ZFS pool using a single disk device:

# zpool create block /dev/da0

To view the new pool, review the output of df:

# df Filesystem      
1K-blocks    Used     Avail Capacity  Mounted on 
/dev/gpt/rootfs  45696284 1668492  40372092     4%    /devfs                   1       1         0   100%    /devblock           516030224      92 516030132     0%    /block

This output shows that the /block pool has been created and mounted. It is now accessible as a file system. Now let’s create a filesystem within the pool:

# zfs create block/jails

With block storage now attached and working, let’s make the configuration permanent and also more secure. We’ll start by unmounting /jails and then remove the iSCSI block storage so we can reattach it using CHAP authentication credentials at boot time.

# zfs umount block
# iscsictl -R -p 10.10.10.10:port-t iqn.myIQNnumberHere

Now go into the OCI web console and detach the storage, then add it again but this time enable authentication using CHAP credentials. Make sure you select CHAP when you add the block storage again.

To connect using a configuration file, create /etc/iscsi.conf and ensure give it chmod 600 permissions so that only root users can read the file. Then edit the fil with contents like this, inserting the information you received in the OCI console for each placeholder below:

b0 {
        TargetAddress   = 10.10.10.10
        TargetName      = iqn.myIQNnumberHere
        AuthMethod      = CHAP
        chapIName       = user
        chapSecret      = mySecretSecret
}

The b0 specifies a nickname within the configuration file. It will be used by the initiator to specify which configuration to use if there are more than one. The other lines specify the parameters to use during connection. The TargetAddress and TargetName are mandatory, whereas the CHAP authentication is optional. In this example, the CHAP username and secret are shown .

To connect to the specific target, specify the nickname:

# iscsictl -An b0

Alternately, to connect to all targets defined in the configuration file, use:

# iscsictl -Aa

To make the initiator automatically connect to all targets in/etc/iscsi.conf, as well as ensure zfs starts at boot, add the following to /etc/rc.conf:

iscsictl_enable="YES"
iscsictl_flags="-Aa"
zfs_enable="YES"

Also make sure to add an entry for ZFS in /boot/loader.conf:

zfs_load="YES"

Now our system is fully configured and we can build out the jails to isolate our apps and services. In our cases we’ll isolate the database, application and web tiers into their own virtual containers. This process helps to heighten the overall security of the system because if one tier becomes compromised, the other tiers and the host will all remain secure. We will also monitor the jails from the all-knowing host using one or more intrusion detection systems in order to detect a breach.

Final Thoughts

I ran into some minor challenges with the boot order of services in FreeBSD, where my block storage mount attempt was happening before ZFS was enabled, causing it to fail, and then ezjail would fail to start because it couldn’t see the storage.

To resolve this issue I added an entry to /etc/rc.local pointing to a simple script I wrote with just a few lines,  that first runs the command zfs mount -a (which mounts all ZFS volumes), waits a few moments to ensure the storage is available, and then runs ezjail. Since this all happens at the end of the boot cycle when executived from /etc/rc.local, it worked just fine.

Comments or questions? Leave a comment below or send me an email.

Some of the lines below wrap on smaller screens so I’ve included extra spaces between the rdr entries to make them easier to read:

ext_if="em0"
sshd_port="45678"
web="192.168.0.1"
db="192.168.0.2"

scrub in

#Redirect web traffic to the jail.
rdr on $ext_if proto tcp from any to ($ext_if) port http -> $web port http

rdr on $ext_if proto tcp from any to ($ext_if) port https -> $web port https

rdr on $ext_if2 proto tcp from any to ($ext_if2) port http -> $web port http

#Allow jail traffic to nat back to anywhere
nat on $ext_if from $web to any -> ($ext_if)
nat on $ext_if from $db to any -> ($ext_if)
nat on $ext_if2 from $web to any -> ($ext_if)

#FILTERING RULES
block in
block in quick inet6 all
pass out keep state
pass quick on { lo }
antispoof log quick for { lo }
pass in on $ext_if proto tcp to ($ext_if) port { $sshd_port } keep state

pass out on $ext_if proto tcp to ($ext_if) port { $sshd_port } keep state

#Additional ins web rules. "pass out" not required because "nat pass" bypasses outgoing filters

pass in on $ext_if proto tcp to $web port { http, https } keep state

pass in on $ext_if2 proto tcp to $web port { http } keep state

The first step is to use Oracle’s free VirtualBox virtualization app to create a new VM image. Download VirtualBox if you don’t have it and then create a new image.  In my case, I’m creating a FreeBSD VM by first booting from a FreeBSD “boot only” ISO image. I create the VM with two virtual CPUs (vCPUs), 7.5GB of RAM and 50GB of disk since that is the default shape of the smallest image on the Oracle Public Cloud (OPC).

By default VirtualBox creates the VM with VMDK disk files. This doesn’t work with OPC so we need to convert the disk image to an .img format. The easiest way is to follow a two-step procedure, starting with a command line utility that gets installed on the Mac when you first installed VirtualBox. Navigate to the directory for your VM and issue the following command in a terminal window:

VBoxManage clonehd /path/to/MyVMfile.vmdk --format VDI /path/to/MyVMfile.vdi

The next step is to convert the VDI image into an IMG file. Note that this will be a huge file, in this case 50GB with lots of empty space inside, so make sure you have enough space before proceeding:

VBoxManage internalcommands converttoraw /path/to/MyVMfile.vdi /path/to/MyVMfile.img

On Linux and Windows, we would then create a sparse copy of this file to significantly reduced its size. Unfortunately, Mac is based on BSD and their HFS+ filesystem does not support sparse files. Your options are to either copy the file into Linux or Windows and perform a sparse copy before creating a compressed tarball to upload to OPC, or just suck it up and use the massive 50GB file.

Finally, we’ll tar/gzip it up before uploading to Oracle Public Cloud:

tar -czSf /path/to/MyVMfile_sparse.tar.gz /path/to/MyVMfile_sparse.img

You might as well delete the huge .img file now to save space.

Now you can upload this file to the Storage side of OPC using a container named compute_images, then register the image and finally deploy a new instance using your new private image.

If you have any problems, leave a comment here or email me and I’ll see if I can help you out.

Kelly

The purpose of this article is to make it easy for WordPress and its 3rd party SFTP plugin to work on a FreeBSD host where the website and web server daemon runs within a jail, but the sshd daemon runs on the host thereby causing problems that prevent the WordPress SFTP plugin from working properly. In this article we enable remote read/write access to the WordPress application to enable it to self-update, using an sftp user who is restricted from seeing or accessing the rest of host’s  filesystem. The jail configuration for WordPress significantly limits the scope of a breach should WordPress or any one of its plugins become compromised by an attacker down the road. This article assumes any modern FreeBSD version is used. The author is using FreeBSD 10 and this very likely works on earlier versions too.

We’ll start by creating a special sftp/ssh user for WordPress on the FreeBSD host.

First create a new group called “chroot”:

pw groupadd chroot

Next, create a new user such as “wordpress-larry” and ensure their primary login group is set to “chroot”. You should likely add the user to an additional, secondary group such as “www” so they can access/edit files for the website.

adduser

Run the adduser command and follow the prompts to create a new user with a STRONG password. In our case we’ll create a new user with the username “wordpress-larry”. Now let’s switch to that user and create a new SSH private/public key pair for logging in, since it’s safer to use keys than a password:

su myuser
cd ~
ssh-keygen -t rsa -b 4096

At this point we need to edit the /etc/ssh/sshd_config file to ensure the user is chroot’ed within their home directory only. The goal is to prevent a rogue user from having any access to the rest of the filesystem, we want to keep him in the protected jail. Edit the file and add a new group for chroot, like this:

nano /etc/ssh/sshd_config

  Match group          chroot
    ChrootDirectory    /home/%u
    X11Forwarding      no
    AllowTcpForwarding no
    ForceCommand       internal-sftp

One the configuration has been updated, restart the SSH daemon, sshd:

/etc/rc.d/sshd restart

The user can only navigate within their home directory thanks to the ChrootDirectory configuration. However we do need them to access part of a website with files located elsewhere, so let’s first mirror the directory structure as seen within a jailed login, so it looks like the wordpress location when the user logs in:

mkdir –p /home/myuser/wordpress

Next we’ll use a filesystem mount to mount the website’s directory into the user’s home directory using mount_nullfs, where “source” is the /full/path to the files you want to make available to the user:

mount_nullfs /path/to/wordpress/files /home/myuser/wordpress

Confirm the mount was successful then create an entry in /etc/fstab related to the above, making sure the entry is all on one line:

/path/to/wordpress/files/ /home/myuser/wordpress/ nullfs rw 0 0

For remote access, it’s critical that the user’s directory be owned by “root” and also be part of the “chroot” group, or else SSH/SFTP will not allow the user to even login:

chown –R root:chroot /home/myuser
chmod 755 /home/myuser

Finally, test the connection (using SFTP, not SSH) to make sure it’s working. Logging in manually will let you confirm that the WordPress files are there, in the user’s home directory, and so the SFTP plugin will be able to function in WordPress with the added security of only being able to see and edit files located in the website’s jail.

Done.