===== FreeBSD Test Suite =====

Welcome to %%FreeBSD%% Test Suite project!

===== Introduction =====

This project equipped %%FreeBSD%% with a comprehensive test suite that aims to be easy to run out of the box and during the development of the system.  Goal of our test suite is to assist both developers and users in assessing the quality of %%FreeBSD%%.

The test framework of choice is [[https://github.com/jmmv/kyua/|Kyua]]. Tests can be written in many languages and are not restricted to using a specific framework. The recommendation is to use [[https://github.com/jmmv/atf/|GitHub: jmmv/atf: Automated Testing Framework (ATF), a library collection to write test programs in C, C++ and POSIX shell.]], which is included in %%FreeBSD%% Base, but other supported options exist.

===== Status =====

  * Test suite is available in 10.0-RELEASE and newer.
  * Test suite is not built by default on %%FreeBSD%% 10.x but enabled on 11.x by default. Set ''%%WITH_TESTS=yes%%'' in ''%%src.conf(5)%%'' before ''%%buildworld%%'' to enable.
  * Continuous Integration testing is done via [[https://ci.freebsd.org|FreeBSD CI Service]] running Jenkins.

===== Goals =====

  * Provide a comprehensive set of tests that can be used to assess the quality of the tree at any given point in time. The release engineering team should be able to confidently use the results of the test suite to vet releases, including alphas, betas and release candidates.

  * Provide a comprehensive set of tests that exercises a release on a given hardware configuration. System administrators should be able to use this right out of the box after deploying %%FreeBSD%% to a new machine to ensure the release works as advertised on their particular hardware/software combination. Of course, the release engineering team should also use this on reference hardware.

  * Provide a mechanism to easily implement tests for the %%FreeBSD%% kernel.  Writing test code for the kernel should not be significantly more difficult than writing test code for user-space applications.

  * Minimize divergence with %%NetBSD%%'s test suite. Both systems should use the same testing tools and have a similar setup so that sharing test code is possible and easy.

The following goals are desirable for the %%FreeBSD%% test suite but they mostly describe work required on Kyua:

  * Provide a per-release matrix in which the rows describe test cases and the columns describe every supported configuration (any desired combination of hardware architecture, compiler, etc.).  Each cell shows the result of the latest execution, be it a test result if the build succeeded or a link to a log if the build failed.

  * Provide historical evaluation of tests so that clicking on any cell of the matrix yields a list of results, their timestamps and any information associated to them.  The idea is to permit automatically spotting flaky tests and to allow pinpoint which commit is at fault in the face of a test failure.

  * Asynchronously notify developers (e.g. by sending an email to a mailing list) of test failures.

===== Getting Started =====

<code>
# pkg install kyua  # this can be skipped on FreeBSD CURRENT after 23 March 2020
$ kyua test -k /usr/tests/Kyuafile
</code>

Test logs are created in ''%%~/.kyua/logs%%'' of the invoking user by default.

===== Documentation =====

  * ''%%tests(7)%%'': User-level introduction to the %%FreeBSD%% test suite.
  * [[TestSuite/Structure]]: Description of the basic concepts involved in the test suite, its build infrastructure and the structure of the installed products.
  * [[TestSuite/DeveloperHowTo]]: Tutorial on how to add new tests and quick API reference.

===== Contact =====

  * [[https://lists.freebsd.org/mailman/listinfo/freebsd-testing|freebsd-testing mailing list]]

===== Links =====

  * [[https://github.com/jmmv/atf/|ATF]], [[http://github.com/jmmv/kyua/|Kyua]]: Project page for the testing framework.
  * [[https://svnweb.freebsd.org/base/user/jmmv/autotest/|autotest]]: Source code of the automation scripts and configuration used in the test cluster.

===== Credits =====

  * JulioMerino (''%%jmmv%%''): Owner of this project, of these pages and author of Kyua.  Also owner of the same project in %%NetBSD%%. Maintainer of the ''%%devel/atf%%'', ''%%devel/kyua%%'', etc, ports.
  * RuiPaulo (''%%rpaulo%%''): Reviewer of patches, ex-mentor of ''%%jmmv%%''.
  * EnjiCooper (''%%ngie%%''): Original driver of this project.  Did initial ATF import and ported lots of tests into their own private repository. Integrated (and is continuing to integrate) tests from %%NetBSD%% and legacy tests from tools/regression into the %%FreeBSD%% test suite.
  * SimonGerraty (''%%sjg%%'') **and others at Juniper**: Ported many %%NetBSD%% tests to %%FreeBSD%%.  Work on the ''%%share/mk%%'' infrastructure.
  * AlanSomers (''%%asomers%%''): Former maintainer of the ''%%devel/atf%%'', ''%%devel/kyua%%'', etc ports. Test contributor.

----

{{tag>Project}}
{{tag>HowTo}}
{{tag>Stale}}
{{tag>NeedsContent}}