====== Release Testing ======
This page contains notes from gjb@ on what testing is done during a release cycle.  Once complete, this page will be integrated in to the [[https://www.freebsd.org/doc/en/articles/freebsd-releng/|FreeBSD Releng]] article in the Handbook.

===== Testing Environment =====
For testing BETA, RC, and RELEASE builds, I use a combination of real hardware, [[/VirtualBox|VirtualBox]], and QEMU.  There is no real reason I use [[/VirtualBox|VirtualBox]] over bhyve or other hypervisors other than I find it easier to manipulate virtual machines.

===== ISO Image Testing =====

^ Image ^ What to test ^ Notes ^
| bootonly.iso | That it boots. | Since the installer uses FTP to fetch the distribution sets which have not yet propagated to the mirrors, there is not much more to check here. |
| disc1.iso | It boots, burns to a CD-ROM without problems, and installs without issues. | [1] |
| dvd1.iso | Install packages provided on-disc. | [2] |

[1] I test the installer by doing a few variations of options.  Of note:
  * amd64: mirrored ZFS, mirrored ZFS with GELI, mirrored ZFS with encrypted swap, ZFS with GELI with a space in the passphrase
  * i386: UFS, default options
  * all default options selected
  * miscellaneous options enabled/disabled (especially any new options since the last release)
  * adding user accounts with non-default pw(8) options, such as the login shell, additional GIDs, password containing a space
  * When booting into the installed system, verify 'uname -a' looks sane (i.e., correct Subversion revision, $USER from the build machine did not leak, correct architecture, etc.)
  * Note:  Provided the installer does not change during the release cycle, the different variations are not necessarily needed each build unless there are changes to something directly used by bsdinstall(8) (i.e., a ZFS merge, GELI update, pw(8) update, etc.)
  * Run the Kyua test suite on the freshly-installed system:

<code>
# pkg install -y kyua
# cd /usr/tests   # Note: /usr/tests is not built or installed by default on 10.x and earlier
# script /var/log/kyua.log kyua test -k /usr/tests/Kyuafile
# kyua report-html --output ~/public_html/tests
</code>

[2] Generally, I make sure dvd1.iso boots, but re-use the disc1.iso virtual machine for testing package installation.  Since bsdconfig(8) has not supported installing packages from dvd1.iso, packages are installed as:

<code>
# mkdir -p /dist
# mount -t cd9660 /dev/cd0 /dist
# setenv REPOS_DIR /dist/packages/repos
# pkg bootstrap
# pkg update
# pkg install xorg xorg-server gnome3
# pkg del -ay
# pkg install xorg xorg-server kde
</code>

After ensuring the desktop environments start properly, install the emulators/virtualbox-ose-additions{,nox11} package from the upstream pkg(8) mirrors and enable the rc(8) scripts in rc.conf(5):

<code>
# unsetenv REPOS_DIR
# pkg update
# pkg install emulators/virtualbox-ose-additions-nox11
# echo 'vboxservice_enable="YES"' >> /etc/rc.conf
# echo 'vboxguest_enable="YES"' >> /etc/rc.conf
</code>

===== Virtual Machine Testing =====
  * Ensure each image format boots (Note: I use QEMU for this, as it is faster than [[/VirtualBox|VirtualBox]] to set up, and QEMU supports all four image formats we provide)
  * Ensure 'pkg bootstrap' works
  * Ensure 'uname -a' looks sane

===== ARM SoC Testing =====
  * Verify the images boot on boards I have
  * Verify SSH access works as the 'freebsd' user
  * Ensure su(1) to the 'root' user works
  * Ensure 'pkg bootstrap' works