Welcome to FreeBSD Test Suite project!

This project equipped FreeBSD with a comprehensive test suite that aims to be easy to run out of the box and during the development of the system. Goal of our test suite is to assist both developers and users in assessing the quality of FreeBSD.

The test framework of choice is Kyua. Tests can be written in many languages and are not restricted to using a specific framework. The recommendation is to use GitHub: jmmv/atf: Automated Testing Framework (ATF), a library collection to write test programs in C, C++ and POSIX shell., which is included in FreeBSD Base, but other supported options exist.

• Test suite is available in 10.0-RELEASE and newer.
• Test suite is not built by default on FreeBSD 10.x but enabled on 11.x by default. Set WITH_TESTS=yes in src.conf(5) before buildworld to enable.
• Continuous Integration testing is done via FreeBSD CI Service running Jenkins.

• Provide a comprehensive set of tests that can be used to assess the quality of the tree at any given point in time. The release engineering team should be able to confidently use the results of the test suite to vet releases, including alphas, betas and release candidates.

• Provide a comprehensive set of tests that exercises a release on a given hardware configuration. System administrators should be able to use this right out of the box after deploying FreeBSD to a new machine to ensure the release works as advertised on their particular hardware/software combination. Of course, the release engineering team should also use this on reference hardware.

• Provide a mechanism to easily implement tests for the FreeBSD kernel. Writing test code for the kernel should not be significantly more difficult than writing test code for user-space applications.

• Minimize divergence with NetBSD's test suite. Both systems should use the same testing tools and have a similar setup so that sharing test code is possible and easy.

The following goals are desirable for the FreeBSD test suite but they mostly describe work required on Kyua:

• Provide a per-release matrix in which the rows describe test cases and the columns describe every supported configuration (any desired combination of hardware architecture, compiler, etc.). Each cell shows the result of the latest execution, be it a test result if the build succeeded or a link to a log if the build failed.

• Provide historical evaluation of tests so that clicking on any cell of the matrix yields a list of results, their timestamps and any information associated to them. The idea is to permit automatically spotting flaky tests and to allow pinpoint which commit is at fault in the face of a test failure.

• Asynchronously notify developers (e.g. by sending an email to a mailing list) of test failures.

# pkg install kyua  # this can be skipped on FreeBSD CURRENT after 23 March 2020
$ kyua test -k /usr/tests/Kyuafile

Test logs are created in ~/.kyua/logs of the invoking user by default.

• tests(7): User-level introduction to the FreeBSD test suite.
• Structure: Description of the basic concepts involved in the test suite, its build infrastructure and the structure of the installed products.
• DeveloperHowTo: Tutorial on how to add new tests and quick API reference.

• freebsd-testing mailing list

• ATF, Kyua: Project page for the testing framework.
• autotest: Source code of the automation scripts and configuration used in the test cluster.

• JulioMerino (jmmv): Owner of this project, of these pages and author of Kyua. Also owner of the same project in NetBSD. Maintainer of the devel/atf, devel/kyua, etc, ports.
• RuiPaulo (rpaulo): Reviewer of patches, ex-mentor of jmmv.
• EnjiCooper (ngie): Original driver of this project. Did initial ATF import and ported lots of tests into their own private repository. Integrated (and is continuing to integrate) tests from NetBSD and legacy tests from tools/regression into the FreeBSD test suite.
• SimonGerraty (sjg) and others at Juniper: Ported many NetBSD tests to FreeBSD. Work on the share/mk infrastructure.
• AlanSomers (asomers): Former maintainer of the devel/atf, devel/kyua, etc ports. Test contributor.