Installing FreeBSD Root on ZFS using a FreeBSD MBR Slice

This guide is a work in progress and hasn't been fully tested

Fails to boot

BTX loader 1.00  BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive C: is disk0
BIOS 638kB/2094720kB available memory

FreeBSD/i386 bootstrap loader, Revision 1.1
(root@, Sat Jul 18 10:58:41 UTC 2009

can't load 'kernel'

Type '?' for a list of commands, 'help' for more detailed help.

Boot FreeBSD install DVD or USB Memstick

Choose Fixit option in sysinstall

Create MBR disk

If you are installing on an New drive, you will need to create the MBR geometry first:

Fixit# gpart create -s mbr ad0

Create Slice using gpart

This example already has two slices defined on the drive.

Fixit# gpart show ad0
=>       63  625142385  ad0  MBR  (289G)
         63  209712447    1  !7  (100G)
  209712510     417690    2  !136  (204M)
  210130200  415012248       - free -  (198G)

Create a third slice then create a BSD partition table on the new slice.

Fixit# gpart add -t freebsd ad0
ad0s3 added

And examine the result:

Fixit# gpart show ad0
=>       63  625142385  ad0  MBR  (289G)
         63  209712447    1  !7  (100G)
  209712510     417690    2  !136  (204M)
  210130200  415012248    3  freebsd  (198G)

Sizes and offsets are specified in sectors (1 sector is typically 512 bytes) but can also be specified in MB or GB by adding an 'M' or 'G' suffix.

Set partition active (Optional)

Some BIOSes might require setting a Slice active. Here we set partition 1 (the Windows Partiton) as the active partition.

Fixit# gpart set -a active -i 1 ad0

Load ZFS kernel module

Fixit# kldload /mnt2/boot/kernel/opensolaris.ko
Fixit# kldload /mnt2/boot/kernel/zfs.ko

Create ZFS Pool zroot

Fixit# mkdir /boot/zfs
Fixit# zpool create zroot /dev/ad0s3
Fixit# zpool set bootfs=zroot zroot

NOTE: zroot is the name of the ZFS Pool, it could be anything (i.e. tank, data, …)

Create ZFS Swap volume

Fixit# zfs create -V 2gb zroot/swap
Fixit# zfs set org.freebsd:swap=on zroot/swap
Fixit# zfs set checksum=off zroot/swap

Install boot Manager

Fixit# gpart bootcode -b /mnt2/boot/boot0 ad0
ad0 has bootcode

It's ok if you get an “unknown class” message here, but it you get “failed to install” or the like, you'll need to try again after making sure that nothing is mounted from /dev/ad0.

Install ZFS boot

- Export zroot, before installing boot code

Fixit# zpool export zroot

- Install the boot1 stage:

Fixit# dd if=/mnt2/boot/zfsboot of=/dev/ad0s3 count=1

This may fail with an “operation not permitted” error message, since the kernel likes to protect critical parts of the disk. If this happens for you, run:

Fixit# sysctl kern.geom.debugflags=0x10

- Install the boot2 zfs stage into the convienient hole in the ZFS filesystem on-disk format which is located just after the ZFS metadata (this is the seek=1024).

Fixit# dd if=/mnt2/boot/zfsboot of=/dev/ad0s3 skip=1 seek=1024

- Import zroot to continue the install

Fixit# zpool import zroot

Create ZFS filesystem hierarchy

The fletcher4 algorithm should be more robust than the fletcher2 algorithm.

Fixit# zfs set checksum=fletcher4					zroot

Fixit# zfs create -o compression=on	-o exec=on	-o setuid=off	zroot/tmp
Fixit# chmod 1777 /zroot/tmp

Fixit# zfs create							zroot/usr
Fixit# zfs create							zroot/usr/home
Fixit# cd /zroot ; ln -s /usr/home home

Fixit# zfs create -o compression=lzjb 			-o setuid=off	zroot/usr/ports
Fixit# zfs create -o compression=off	-o exec=off	-o setuid=off	zroot/usr/ports/distfiles
Fixit# zfs create -o compression=off	-o exec=off	-o setuid=off	zroot/usr/ports/packages

Note:

If you use nullfs or nfs to mount /usr/ports to different locations/servers, you will also need to nullfs/nfs mount /usr/ports/distfiles and/or /usr/ports/packages.

Fixit# zfs create -o compression=lzjb 	-o exec=off	-o setuid=off	zroot/usr/src

Fixit# zfs create							zroot/var
Fixit# zfs create -o compression=lzjb 	-o exec=off	-o setuid=off	zroot/var/crash
Fixit# zfs create			-o exec=off	-o setuid=off	zroot/var/db
Fixit# zfs create -o compression=lzjb 	-o exec=on	-o setuid=off	zroot/var/db/pkg
Fixit# zfs create			-o exec=off	-o setuid=off	zroot/var/empty
Fixit# zfs create -o compression=lzjb 	-o exec=off	-o setuid=off	zroot/var/log
Fixit# zfs create -o compression=gzip 	-o exec=off	-o setuid=off	zroot/var/mail
Fixit# zfs create			-o exec=off	-o setuid=off	zroot/var/run
Fixit# zfs create -o compression=lzjb 	-o exec=on	-o setuid=off	zroot/var/tmp
Fixit# chmod 1777 /zroot/var/tmp

Note:

Compression may be set to on, off, lzjb, gzip, gzip-N (where N is an integer from 1 (fastest) to 9 (best compression ratio. gzip is equivalent to gzip-6).
Compression will cause some latency when accessing files on the ZFS filesystems. Use compression on ZFS filesystems which will not be accessed that often.

Install FreeBSD to zroot

Fixit# cd /dist/8.0-*
Fixit# export DESTDIR=/zroot
Fixit# for dir in base catpages dict doc games info lib32 manpages ports; \
  do (cd $dir ; ./install.sh) ; done
Fixit# cd src ; ./install.sh all
Fixit# cd ../kernels ; ./install.sh generic
Fixit# cd /zroot/boot ; cp -Rlp GENERIC/* /zroot/boot/kernel/

Make /var/empty readonly

Fixit# zfs set readonly=on zroot/var/empty

chroot into /zroot

Fixit# chroot /zroot

Create /etc/rc.conf

Fixit# echo 'zfs_enable="YES"' > /etc/rc.conf
Fixit# echo 'hostname="beastie.mydomain.local"' >> /etc/rc.conf
Fixit# echo 'ifconfig_re0="DHCP"' >> /etc/rc.conf

Note:

Replace re0 with the name of the Network interface for the new system

Create /boot/loader.conf

Fixit# echo 'zfs_load="YES"' > /boot/loader.conf
Fixit# echo 'vfs.root.mountfrom="zfs:zroot"' >> /boot/loader.conf

Change root's password

Fixit# passwd

Set the local time zone

Fixit# tzsetup

Create /etc/mail/aliases.db

Fixit# cd /etc/mail
Fixit# make aliases

Exit from the /zroot

Fixit# umount /dev
Fixit# exit

Install zpool.cache to the ZFS filesystem

Fixit# cp /boot/zfs/zpool.cache /zroot/boot/zfs/zpool.cache

Create /etc/fstab

Fixit# cat << EOF > /zroot/etc/fstab
# Device			Mountpoint		FStype	Options		Dump	Pass#
EOF

export LD_LIBRARY_PATH

Fixit# export LD_LIBRARY_PATH=/mnt2/lib

Unmount all zfs filesystems

Fixit# zfs unmount -a

Change mount points for zroot pool

Fixit# zfs set mountpoint=legacy zroot
Fixit# zfs set mountpoint=/tmp zroot/tmp
Fixit# zfs set mountpoint=/usr zroot/usr
Fixit# zfs set mountpoint=/var zroot/var