Release Testing

This page contains notes from gjb@ on what testing is done during a release cycle. Once complete, this page will be integrated in to the FreeBSD Releng article in the Handbook.

For testing BETA, RC, and RELEASE builds, I use a combination of real hardware, VirtualBox, and QEMU. There is no real reason I use VirtualBox over bhyve or other hypervisors other than I find it easier to manipulate virtual machines.

[1] I test the installer by doing a few variations of options. Of note:

• amd64: mirrored ZFS, mirrored ZFS with GELI, mirrored ZFS with encrypted swap, ZFS with GELI with a space in the passphrase
• i386: UFS, default options
• all default options selected
• miscellaneous options enabled/disabled (especially any new options since the last release)
• adding user accounts with non-default pw(8) options, such as the login shell, additional GIDs, password containing a space
• When booting into the installed system, verify 'uname -a' looks sane (i.e., correct Subversion revision, $USER from the build machine did not leak, correct architecture, etc.)
• Note: Provided the installer does not change during the release cycle, the different variations are not necessarily needed each build unless there are changes to something directly used by bsdinstall(8) (i.e., a ZFS merge, GELI update, pw(8) update, etc.)
• Run the Kyua test suite on the freshly-installed system:

# pkg install -y kyua
# cd /usr/tests   # Note: /usr/tests is not built or installed by default on 10.x and earlier
# script /var/log/kyua.log kyua test -k /usr/tests/Kyuafile
# kyua report-html --output ~/public_html/tests

[2] Generally, I make sure dvd1.iso boots, but re-use the disc1.iso virtual machine for testing package installation. Since bsdconfig(8) has not supported installing packages from dvd1.iso, packages are installed as:

# mkdir -p /dist
# mount -t cd9660 /dev/cd0 /dist
# setenv REPOS_DIR /dist/packages/repos
# pkg bootstrap
# pkg update
# pkg install xorg xorg-server gnome3
# pkg del -ay
# pkg install xorg xorg-server kde

After ensuring the desktop environments start properly, install the emulators/virtualbox-ose-additions{,nox11} package from the upstream pkg(8) mirrors and enable the rc(8) scripts in rc.conf(5):

# unsetenv REPOS_DIR
# pkg update
# pkg install emulators/virtualbox-ose-additions-nox11
# echo 'vboxservice_enable="YES"' >> /etc/rc.conf
# echo 'vboxguest_enable="YES"' >> /etc/rc.conf

• Ensure each image format boots (Note: I use QEMU for this, as it is faster than VirtualBox to set up, and QEMU supports all four image formats we provide)
• Ensure 'pkg bootstrap' works
• Ensure 'uname -a' looks sane

• Verify the images boot on boards I have
• Verify SSH access works as the 'freebsd' user
• Ensure su(1) to the 'root' user works
• Ensure 'pkg bootstrap' works