Table of Contents

Installing FreeBSD 9.0 (or later) Root on ZFS using GPT

This article covers installing FreeBSD 9.x with the root file system on ZFS with bsdinstall.

Creating a bootable ZFS filesystem

Note: This walkthrough assumes that a zfs mirror of two disks is created, but the instructions work equally well for a single disk or a raidz or raidz2 setup, just replace 'mirror' as needed in the examples.
  1. Boot the FreeBSD install CD/DVD or USB Memstick.
  1. Go through the initial setup as usual.
  1. When the partitioning dialogue in bsdinstall comes up, choose the 'Shell' option
  1. Create new GPT partitioned disks, repeat this for all disks.
# gpart destroy -F ada0
# gpart destroy -F ada1
# gpart create -s gpt ada0
# gpart create -s gpt ada1
  1. Add partitions for the boot loader and swap then install the protective MBR and gptzfsboot boot loader. It is possible to have swap partitions on zfs pools, but this might lead to deadlocks in low memory situations. All partitions are aligned to 4k for optimal performance with 'advanced format' drives.
# gpart add -s 222 -a 4k -t freebsd-boot -l boot0 ada0
# gpart add -s 8g -a 4k -t freebsd-swap -l swap0 ada0
# gpart add -a 4k -t freebsd-zfs -l disk0 ada0
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
  1. Now duplicate this layout for each additional disk:
# gpart add -s 222 -a 4k -t freebsd-boot -l boot1 ada1
# gpart add -s 8g -a 4k -t freebsd-swap -l swap1 ada1
# gpart add -a 4k -t freebsd-zfs -l disk1 ada1
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
  1. If using an 'advanced format' drive, the following commands will create virtual devices with 4k sectors, to ensure ZFS uses the correct block size:
Note: FreeBSD 10.1 or later: Load the zfs kernel module and set the minimum ashift value with sysctl vfs.zfs.min_auto_ashift=12 in stead of using gnop devices 
# gnop create -S 4096 /dev/gpt/disk0
# gnop create -S 4096 /dev/gpt/disk1
  1. Load the necessary kernel modules
# kldload zfs
  1. Create the zfs pool.

Replace the 'mirror' keyword as needed, and list all disks that should be part of the pool. 

# zpool create -o altroot=/mnt -O canmount=off -m none zroot mirror /dev/gpt/disk0.nop /dev/gpt/disk1.nop

This will create a zpool called 'zroot' which will not be mounted. This zpool is only used to derive other file systems from.

Installing FreeBSD to the ZFS file system

  1. Create ZFS filesystem hierarchy

Notes:
  1. Compression may be set to on, off, lz4, lzjb, gzip, gzip-N (where N is an integer from 1 (fastest) to 9 (best compresion ratio. gzip is equivalent to gzip-6);
    • lz4 provides the best trade-off (significantly faster compression and decompression with moderately higher compression ratios);
    • Prior to FreeBSD 8.4 and 9.2, lz4 compression is not available;
  2. Use the fletcher4 checksum algorithm, it is more robust than the old default fletcher2 algorithm.
  3. Set atime to off, this avoid writing a metadata change every time a file is accessed, a serious performance penalty. It can also cause snapshots to grow unexpectedly.
# zfs set checksum=fletcher4						zroot
# zfs set atime=off							zroot
# zfs create   -o mountpoint=none					zroot/ROOT
# zfs create   -o mountpoint=/						zroot/ROOT/default
# zfs create   -o mountpoint=/tmp -o compression=lz4	-o setuid=off	zroot/tmp
# chmod 1777 /mnt/tmp
# zfs create   -o mountpoint=/usr					zroot/usr
# zfs create								zroot/usr/local
Note:

If /home is mounted from someplace else, such as using NFS there is no need to create zroot/home below. 

# zfs create   -o mountpoint=/home			-o setuid=off	zroot/home
Note:

If you use nullfs or nfs to mount /usr/ports to different locations/servers, you will also need to nullfs/nfs mount /usr/ports/distfiles and/or /usr/ports/packages. 

# zfs create 	-o compression=lz4 			-o setuid=off	zroot/usr/ports
# zfs create 	-o compression=off	-o exec=off	-o setuid=off	zroot/usr/ports/distfiles
# zfs create 	-o compression=off	-o exec=off	-o setuid=off	zroot/usr/ports/packages
# zfs create 	-o compression=lz4 	-o exec=off	-o setuid=off	zroot/usr/src
# zfs create								zroot/usr/obj
# zfs create	-o mountpoint=/var					zroot/var
# zfs create 	-o compression=lz4 	-o exec=off	-o setuid=off	zroot/var/crash
# zfs create				-o exec=off	-o setuid=off	zroot/var/db
# zfs create 	-o compression=lz4 	-o exec=on	-o setuid=off	zroot/var/db/pkg
# zfs create				-o exec=off	-o setuid=off	zroot/var/empty
# zfs create 	-o compression=lz4 	-o exec=off	-o setuid=off	zroot/var/log
# zfs create 	-o compression=gzip 	-o exec=off	-o setuid=off	zroot/var/mail
# zfs create				-o exec=off	-o setuid=off	zroot/var/run
# zfs create 	-o compression=lz4 	-o exec=on	-o setuid=off	zroot/var/tmp
# chmod 1777 /mnt/var/tmp
  1. Set the dataset to boot from
# zpool set bootfs=zroot/ROOT/default zroot
  1. Add the swap devices to fstab, so that they will automatically show up when the system starts.
# cat << EOF > /tmp/bsdinstall_etc/fstab
# Device			Mountpoint		FStype	Options		Dump	Pass#
/dev/gpt/swap0			none			swap	sw		0	0
/dev/gpt/swap1			none			swap	sw		0	0
EOF
  1. Once all of the zfs filesystems have been created, type 'exit' in the shell and proceed with the installation as normal.
  1. Once the installation is complete choose 'Exit' from the main menu
  1. The next dialogue will will offer the option to 'open a shell in the new system', choose this option
  1. Configure zfs to load and mount the file systems automatically at boot
# mount -t devfs devfs /dev
# echo 'zfs_enable="YES"' >> /etc/rc.conf
# echo 'zfs_load="YES"' >> /boot/loader.conf
  1. Set read only on /var/empty, it is supposed to be empty at all times.
# zfs set readonly=on zroot/var/empty
  1. Update the zpool cache file. This is only needed on FreeBSD 9.0 and 9.1.
# zpool set cachefile=/boot/zfs/zpool.cache   zroot
# exit
  1. To finish the installation, choose the reboot option from the dialogue, do not forget to remove the installation media from the computer.