pf Tables

pf has tables, which are extremely useful when creating and destroying jails. The given link has all the information needed, but below are some quick commands to get started.

A series of jails will be created, with IP addresses 10.10.10.100-105. A small excerpt from pf.conf follows, in which a table is created, IPs given, and a rule to use it.

table <jails> persist { \
        10.10.10.100, \
        10.10.10.101, \
        10.10.10.102, \
        10.10.10.103, \
}

pass in on $ext_if proto tcp from any to <jails> port 22

This should permit SSH to the jails table. A new jail has just been created, add it to the table and then display the table contents.

$ sudo pfctl -t jails -T add 10.10.10.104
$ sudo pfctl -t jails -T show
10.10.10.100
10.10.10.101
10.10.10.102
10.10.10.103
10.10.10.104

The pf.conf file must still be modified to include this new IP address if it is to persist across reboots.

Packet Filter (pf)

There are several firewall options for FreeBSD - this is about pf.

Kernel Options
While pf can be loaded as a kernel module, ALTQ cannot. If the kernel is to be recompiled, may as well add both. Below are all the pieces available, but not all are required depending on usage. If there is no intended use of ALTQ, then kernel modules may be used instead.

# pf - manually added
device          pf              # OpenBSD Packet Filter firewall
device          pflog           # pseudo network device for logging
device          pfsync          # state change log interface (HA)
# ALTQ - queues
options         ALTQ            # ALTQ framework
options         ALTQ_CBQ        # Class Based Queueing
options         ALTQ_RED        # Random Early Detection
options         ALTQ_RIO        # Random Early Detection In and Out
options         ALTQ_HFSC       # Hierarchical Fair Service Curve Sched.
options         ALTQ_PRIQ       # Priority Queueing. high traffic first
options         ALTQ_NOPCC      # SMP support. Required on SMP systems

There are some pieces for rc.conf:

# pf
pf_enable="YES"                 # enable pf
pf_rules="/etc/pf.conf"         # rules definition file for pf
pf_flags=""                     # additional flags for pfctl
pflog_enable="YES"              # start pflogd(8)
pflog_logfile="/var/log/pflog"  # where pflogd stores logfile
pflog_flags=""                  # additional flags for pflogd

Example configurations for pf can be found in /usr/share/examples/pf on FreeBSD. Or, read through the documentation.

Recompile the kernel and restart. Or, if using modules: kldload pf.

ezjail Flavour

About Flavours
ezjail provides a template system called "flavours." A flavour can be specified when creating an ezjail jail (-f flag), making it extremely easy to create similar jails over and over.

Flavours live in BASE_EZJAIL_DIR/flavours - the rest of this post will assume BASE_EZJAIL_DIR is /usr/jails and ezjail-admin version is 3.0.

Create A Flavour

$ cd /usr/jails/flavours
$ sudo cp -r example theusual
$ cd theusual/etc
$ sudo vi resolv.conf
$ cd ..

The above block copied the example directory to theusual and then created resolv.conf so the new jail knows the local nameserver.

Now to use it.

$ sudo ezjail-admin create -f theusual jail1 10.10.10.101
$ sudo ezjail-admin start jail1
$ sudo ezjail-admin console jail1
# telnet www.google.com 80

The above assumes network settings are correct and a firewall does not block out going port 80 traffic.

On the first start-up, the new jail will run ezjail.flavour, which is a shell script. This can be modified to add users, packages, start services, and so on. With a working network, hopefully setup with the above resolv.conf, packages can even be installed remotely.

The example ezjail.flavour, which was copied, looks for a pkg directory and tries to load packages from there.

$ cd /usr/jails/flavours/theusual
$ sudo mkdir pkg
$ sudo cp /some/dir/with/some_package.tbz pkg/

Upon initial start-up, the next jail created with this flavour will install the local package some_package. The example script also has examples for adding users, groups, and starting services.

A Full Example
The example flavour, provided by ezjail, has a few lines for adding groups, users, and packages. Below is a full example, with the few modifications clearly labeled. This is used to create an environment for Hadoop by installing java (in pkg directory), creating a hadoop user, and setting up and starting sshd.

#!/bin/sh
#

# Groups
#########
#
# You will probably start with some groups your users should be in

pw groupadd -q -n hadoop # -g 1004

# Users
########
#
# You might want to add some users. The password is to be provided in the
# encrypted form as found in /etc/master.passwd.
# The example password here is "admin"
# Refer to crypt(3) and pw(8) for more information

# add our hadoop user here
# user: hadoop, group: hadoop, uid: 110
pw useradd -n hadoop -u 110 -g hadoop -s /bin/sh -m -d /home/hadoop -c 'hadoop account'
mkdir -p /home/hadoop/.ssh
chmod 700 /home/hadoop/.ssh

echo -n 'NOTE: THIS WOULD BE SSH KEY' >> /home/hadoop/.ssh/authorized_keys2
chown -R hadoop:hadoop /home/hadoop

# Files
########
#
# You can now give files to users just created

# /usr/hadoop is where hadoop will live
tar -zxf /usr/hadoop/hadoop*tar.gz -C /usr/hadoop/
rm /usr/hadoop/hadoop-*.tar.gz
chown -R hadoop:hadoop /usr/hadoop

# Packages
###########
#
# Install all packages previously put to /pkg
# Remove package files afterwards

# load the staged packages
[ -d /pkg ] && PACKAGESITE=file:// pkg_add -r /pkg/*
rm -rf /pkg

# Postinstall
##############
#
# Your own stuff here, for example set login shells that were only
# installed just before.
hname=`uname -n`
# sshd
echo "ListenAddress $hname" >> /etc/ssh/sshd_config
echo 'sshd_enable="YES"' >> /etc/rc.conf
/etc/rc.d/sshd start

First FreeBSD Jail

I want a jail to build packages while leaving other jails clean. This should result in only one jail getting cluttered with various source files and what-not. The steps below assume ezjail has never been initialized.

Initialize ezjail
In a previous post, I provided a few quick steps to getting a system ready for jails. The next step is to run ezjail-admin for the first time.

$ sudo ezjail-admin install -p

This will have initialized the jail system for the first time and created a copy of the ports tree. The default directory for the install is /usr/jails and will be created if not already present. Easy.

Create First Jail
A jail needs an IP address. If aliases were setup in rc.conf (and rebooted), then things should be good. If not, here is the syntax:

$ sudo ifconfig em0 inet 10.10.10.110 netmask 255.255.255.255 alias

Create and start the first jail.

$ sudo ezjail-admin create builder 10.10.10.110
$ sudo ezjail-admin start builder

Jail Configuration
There is a good chance the jail is somewhat useless at this point, as network settings and other pieces are missing. Time to use the console.

$ sudo ezjail-admin console builder

The above command results in a root account on the named jail. Fix the network, add a user, and install a few packages. The values below should be changed to match your environment.

# echo "nameserver 10.10.10.1" >> /etc/resolv.conf
# pw useradd -n builder -g builder -s /bin/sh -m -d /home/builder -c 'builder account'
# passwd builder
# pkg_add -r sudo
# sudoedit /usr/local/etc/sudoers
# vi /etc/ssh/sshd_config
# echo 'sshd_enable="YES"' >> /etc/rc.conf
# exit

Quick explanation:

1. Create resolv.conf and add our nameserver/router
2. Add user "builder" (-n builder)
3. Set builder's password
4. Install and then configur sudo
5. Edit sshd_config so "builder" can log in. Be sure to change ListenAddress (10.10.10.110 for this example)
6. Enable sshd in rc.conf

Restart the jail and log in.

$ sudo ezjail-admin stop builder
$ sudo ezjail-admin start builder
$ ssh 10.10.10.110

Create Packages
The original goal for this jail was to be able to build packages for other jails. Here is a quick run-down of how.

Logged in as "builder" to 10.10.10.110 jail:

$ cd /usr/ports/benchmarks/bonnie
$ sudo make install
$ cd ~
$ mkdir bonnie
$ cd bonnie
$ pkg_info | grep bonnie
$ pkg_create -b bonnie-2.0.6_1
$ ls
bonnie-2.0.6_1.tbz

Now to copy this out of the jail and into another. This can either be done by file transfer between jails (scp, ftp, etc) or from the host as shown below.

$ sudo cp /usr/jails/builder/home/builder/bonnie/bonnie-2.0.6_1.tbz /usr/jails/jail2/tmp/

And now load it in the other jail.

$ sudo pkg_add /tmp/bonnie-2.0.6_1.tbz

FreeBSD Jail Host Set-up

This post summarizes what steps were taken to configure and prepare a host system for FreeBSD jails. This is not perfect, I am not a FreeBSD professional, so please be careful if you choose to follow any of these steps.

This is for FreeBSD 8.2-STABLE and assumes a system already installed and running.

Install Sources
Using the sysinstall tool, install all sources: Configure > Distributions > Src > All

This will place the sources needed to compile a custom kernel into the /usr/src/ directory. 

Custom Kernel
Now to customize, build, and install a new kernel. For the purposes below, the hostname "COEUS" will be used - replace this with whatever hostname is correct for your machine. The FreeBSD Handbook provides a great page on how to compile a custom kernel.

# cd /usr/src/sys/`uname -p`/conf
# cp GENERIC COEUS
# vi COEUS

The Configuration File is explained in detail in the FreeBSD Handbook. The ident GENERIC line should be changed to ident COEUS and superfluous devices should be commented out. It is also advisable to remove all options NFS* unless required.

Build And Install The Kernel
Building the kernel is pretty straight forward. Remember the name of your file (COEUS in this example).

# cd /usr/src
# make -j`sysctl -n hw.ncpu` buildkernel KERNCONF=COEUS

If no errors were displayed, then it should be safe to install the kernel and then reboot:

# make installkernel KERNCONF=COEUS
# shutdown -r now

Upon reboot, check whether things were successful.

$ uname -a
FreeBSD coeus.local 8.2-RELEASE FreeBSD 8.2-RELEASE #4: Tue Nov 15 00:11:11 PST 2011     root@coeus.local:/usr/obj/usr/src/sys/COEUS  amd64

Install Packages
The next step is to install some packages. This portion assumes you have a working connection.

# pkg_add -r sudo
# pkg_add -r ezjail

It is recommended to configure and use sudo over always-on root. The remaining steps assume this has been setup correctly and a normal user has been logged in.

Jail Considerations
There is some prepwork to be done in order to get jails setup correctly. First, each jail will need an IP address. Will they be in order? Is a block of address already set aside? How are jails expected to be named? A general strategy now will help out later.

Once the answer to jail names, IP address, and how many have been answered, continue on.

Configuring /etc/rc.conf
Below is a working /etc/rc.conf and some comments. Your file may include a lot more than the example below. Use your best judgement.

# Enable network daemons for user convenience.
# Please make all changes to this file, not to /etc/defaults/rc.conf.
# This file now contains just the overrides from /etc/defaults/rc.conf.
#
hostname="coeus.local"          # set hostname
#
# syslogd should not list on any IP address
# this allows syslogd in jails
syslogd_flags="-ss"             # additional flags for syslogd
#
# some services:
sshd_enable="YES"               # enable sshd
sendmail_enable="NO"            # disable sendmail
#
# ifconfig
ifconfig_em0="inet 192.168.1.180 netmask 255.255.255.0"
defaultrouter="192.168.1.1"
#
# jail aliases
ifconfig_em0_alias0="192.168.1.181 netmask 255.255.255.255"
ifconfig_em0_alias1="192.168.1.190 netmask 255.255.255.255"
ifconfig_em0_alias2="192.168.1.191 netmask 255.255.255.255"
ifconfig_em0_alias3="192.168.1.192 netmask 255.255.255.255"
ifconfig_em0_alias4="192.168.1.193 netmask 255.255.255.255"
ifconfig_em0_alias5="192.168.1.194 netmask 255.255.255.255"
#
# jails
jail_enable="YES"               # enable jails
jail_set_hostname_allow="NO"    # disable hostname changes in jails
jail_sysvipc_allow="YES"        # needed for postgresql
#
# ezjail GO!
ezjail_enable="YES"             # enable ezjails

Quick breakdown:

1. syslogd_flags will stop syslogd from listening on all IPs. 
2. sshd_enable and sendmail_enable control those services and how I want them on startup.
3. ifconfig_em0 and defaultrouter set network settings for the host
4. ifconfig_em0_alias1 .. 5 are the IP address for the jails. Note the netmask of all 255's.
5. jail_enable enables jails
6. ezjail_enable allows ezjail to be used

Reboot.