Welcome to FreeBSD Test Suite project!

This project aims to equip FreeBSD with a comprehensive test suite that is easy to run out of the box and during the development of the system. The goal of the test suite is to assist both developers and users in assessing the quality of FreeBSD.

The runtime of choice to execute the test suite and generate reports is Kyua. The tests themselves can be written in a variety of languages and are not restricted to using a specific framework; the recommendation is to use the ATF libraries bundled into the base tree, but other supported options exist.

$ pkg install kyua  # this can be skipped on FreeBSD CURRENT after 23 March 2020
$ kyua test -k /usr/tests/Kyuafile

The following goals are specific to the FreeBSD test suite:

• Provide a comprehensive set of tests that can be used to assess the quality of the tree at any given point in time. The release engineering team should be able to confidently use the results of the test suite to vet a release (including alphas, betas and release candidates).

• Provide a comprehensive set of tests that exercises a release on a given hardware configuration. System administrators should be able to use this right out of the box after deploying FreeBSD to a new machine to ensure the release works as advertised on their particular hardware/software combination. Of course, the release engineering team should also use this on reference hardware.

• Provide a mechanism to implement tests for the kernel. Writing test code for the kernel should not be significantly more difficult than writing test code for user-space applications.

• Minimize divergence with NetBSD's test suite. Both systems should use the same testing tools and have a similar setup so that sharing test code is easy and possible.

The following goals are desirable for the FreeBSD test suite but they mostly describe work required on Kyua:

• Provide a per-release matrix in which the rows describe test cases and the columns describe every supported configuration (any desired combination of hardware architecture, compiler, etc.). Each cell shows the result of the latest execution, be it a test result if the build succeeded or a link to a log if the build failed.

• Provide historical evaluation of tests so that clicking on any cell of the matrix yields a list of results, their timestamps and any information associated to them. The idea is to permit automatically spotting flaky tests and to allow pinpoint which commit is at fault in the face of a test failure.

• Asynchronously notify developers (e.g. by sending an email to a mailing list) of test failures.

Highlights:

• The test suite exists in 10.0-RELEASE and newer.
• The building of the test suite is disabled on 10.x, but enabled on 11.x by default. Set WITH_TESTS=yes in src.conf(5) to enable.
• Continuous Integration testing is done via FreeBSD CI Service running Jenkins.

• tests(7): User-level introduction to the FreeBSD test suite.

• Structure: Description of the basic concepts involved in the test suite, its build infrastructure and the structure of the installed products.

• DeveloperHowTo: Tutorial on how to add new tests and quick API reference.

• freebsd-testing mailing list

• ATF, Kyua: Project page for the testing framework.
• autotest: Source code of the automation scripts and configuration used in the test cluster.

• JulioMerino (jmmv): Owner of this project, of these pages and author of Kyua. Also owner of the same project in NetBSD. Maintainer of the devel/atf, devel/kyua, etc, ports.
• RuiPaulo (rpaulo): Reviewer of patches, ex-mentor of jmmv.
• EnjiCooper (ngie): Original driver of this project. Did initial ATF import and ported lots of tests into their own private repository. Integrated (and is continuing to integrate) tests from NetBSD and legacy tests from tools/regression into the FreeBSD test suite.
• SimonGerraty (sjg) and others at Juniper: Ported many NetBSD tests to FreeBSD. Work on the share/mk infrastructure.
• AlanSomers (asomers): Former maintainer of the devel/atf, devel/kyua, etc ports. Test contributor.