Release Testing
This page contains notes from gjb@ on what testing is done during a release cycle. Once complete, this page will be integrated in to the FreeBSD Releng article in the Handbook.
Testing Environment
For testing BETA, RC, and RELEASE builds, I use a combination of real hardware, VirtualBox, and QEMU. There is no real reason I use VirtualBox over bhyve or other hypervisors other than I find it easier to manipulate virtual machines.
ISO Image Testing
| Image | What to test | Notes |
|---|---|---|
| bootonly.iso | That it boots. | Since the installer uses FTP to fetch the distribution sets which have not yet propagated to the mirrors, there is not much more to check here. |
| disc1.iso | It boots, burns to a CD-ROM without problems, and installs without issues. | [1] |
| dvd1.iso | Install packages provided on-disc. | [2] |
[1] I test the installer by doing a few variations of options. Of note:
- amd64: mirrored ZFS, mirrored ZFS with GELI, mirrored ZFS with encrypted swap, ZFS with GELI with a space in the passphrase
- i386: UFS, default options
- all default options selected
- miscellaneous options enabled/disabled (especially any new options since the last release)
- adding user accounts with non-default pw(8) options, such as the login shell, additional GIDs, password containing a space
- When booting into the installed system, verify 'uname -a' looks sane (i.e., correct Subversion revision, $USER from the build machine did not leak, correct architecture, etc.)
- Note: Provided the installer does not change during the release cycle, the different variations are not necessarily needed each build unless there are changes to something directly used by bsdinstall(8) (i.e., a ZFS merge, GELI update, pw(8) update, etc.)
- Run the Kyua test suite on the freshly-installed system:
# pkg install -y kyua # cd /usr/tests # Note: /usr/tests is not built or installed by default on 10.x and earlier # script /var/log/kyua.log kyua test -k /usr/tests/Kyuafile # kyua report-html --output ~/public_html/tests
[2] Generally, I make sure dvd1.iso boots, but re-use the disc1.iso virtual machine for testing package installation. Since bsdconfig(8) has not supported installing packages from dvd1.iso, packages are installed as:
# mkdir -p /dist # mount -t cd9660 /dev/cd0 /dist # setenv REPOS_DIR /dist/packages/repos # pkg bootstrap # pkg update # pkg install xorg xorg-server gnome3 # pkg del -ay # pkg install xorg xorg-server kde
After ensuring the desktop environments start properly, install the emulators/virtualbox-ose-additions{,nox11} package from the upstream pkg(8) mirrors and enable the rc(8) scripts in rc.conf(5):
# unsetenv REPOS_DIR # pkg update # pkg install emulators/virtualbox-ose-additions-nox11 # echo 'vboxservice_enable="YES"' >> /etc/rc.conf # echo 'vboxguest_enable="YES"' >> /etc/rc.conf
Virtual Machine Testing
- Ensure each image format boots (Note: I use QEMU for this, as it is faster than VirtualBox to set up, and QEMU supports all four image formats we provide)
- Ensure 'pkg bootstrap' works
- Ensure 'uname -a' looks sane
ARM SoC Testing
- Verify the images boot on boards I have
- Verify SSH access works as the 'freebsd' user
- Ensure su(1) to the 'root' user works
- Ensure 'pkg bootstrap' works