This page is part of the [[/TestSuite|TestSuite]] project and provides a description of the basic concepts involved in the test suite, its build infrastructure and the structure of the installed products. **IMPORTANT: Please do not take anything for granted, ESPECIALLY if you have previously worked with and/or have seen ''%%src/tools/test/%%'' and ''%%src/tools/regression/%%''. Some details differ in important ways.** This document is based on a [[http://wiki.netbsd.org/tutorials/atf/|similar tutorial in NetBSD]] and a lot of text has been copied form there. ====== Concepts ====== ===== Kyua ===== Kyua is the runtime engine of choice to run the %%FreeBSD%% test suite. This utility implements a modular system to safely run test programs of various kinds and that collates their results into a unified database for later report generation and debugging. Kyua is available from the ports system under ''%%devel/kyua/%%'' and its homepage is at http://code.google.com/p/kyua/ ===== Test programs vs. test cases ===== The definitions in here are tricky given that the line between each concept is very thin. A **test case** is a piece of code that exercises a specific functionality of another piece of code. Commonly, test cases validate the outcome of a particular function or class method, or the validity of the execution of a command with a particular combination of flags/arguments. Test cases are supposed to be very concise and should test just //one behavior//. A **test** is more specific than a test case in the sense that a test is the actual "comparison" being performed. If you are familiar with "checks" and "requires" in the popular *Unit frameworks, a single line invoking these functions is a test. For example: in a calculator application, one could have a test case for additions, another for subtractions... and each of these could invoke different tests by using different inputs (corner cases) to the tested operations. That said, a well-designed test suite will often have one single test per test case and thus the test case and test terms refer to the same thing. There are two types of tests: ''%%CHECK%%''s and ''%%REQUIRE%%''s. A check-style test raises an error in the test case but does not cause the test case to terminate immediately. On the other hand, a require-style test terminates the test case right away. You should use require-style tests whenever it is impossible for the test to continue and yield any meaningful results if that particular test fails. A **test program** is a binary that collects and exposes a group of test cases. Typically, these test programs expose conceptually-related tests or all the tests for a particular source file. In general, //having many test programs with **just** one test case in them is wrong//. Consider some some other organization. Please note that this is extremely common in (almost?) all other test frameworks and, when used wisely, becomes an invaluable structure for your test suite. For example, suppose you have the following fictitious source files for the ls tool: * ''%%bin/ls/fs.c%%'': Provides the ''%%list_files()%%'' and ''%%stat_files()%%'' functions. * ''%%bin/ls/ui.c%%'': Provides the ''%%format_columns()%%'' function. * ''%%bin/ls/main.c%%'': The main method for ''%%ls%%''. Then, you could define the following test programs and test cases: * ''%%bin/ls/fs_test.c%%'': Provides test cases for ''%%list_files%%'' and ''%%stat_files%%''. These would be named ''%%list_files__empty_directory%%'', ''%%list_files__one_file%%'', ''%%list_files__multiple_files%%'', ''%%stat_files__directory%%'', ''%%stat_files__symlink%%'', etc. * ''%%bin/ls/ui_test.c%%'': Provides test cases for the format_columns function. These would be named ''%%format_columns__no_files%%'', ''%%format_columns__multiple_files%%'', etc. * ''%%bin/ls/integration_test.sh%%'': Provides "black box" test cases for the binary itself. These would be named ''%%lflag%%'', ''%%lflag_and_Fflag%%'', ''%%no_flags%%'', ''%%no_files%%'', etc. Try to keep your test case names as descriptive as possible so that they do not require comments to explain what they intend to test. The test case names are exposed to the user. ===== Test program types ===== Kyua supports the execution of both plain (aka legacy) test programs as well as ATF-based test programs. (Support for other test program types is possible and easy, but there are no specific plans at the moment to implement them.) **Plain test programs** are simple binaries that perform one or more tests from their ''%%main()%%'' function and return 0 on success and non-0 on failure. While plain test programs can implement various tests, they only implement a single test case (the ''%%main()%%'' function). Other than their return code, these test programs have no mechanism to communicate information to the runtime engine. **ATF-based test programs** are binaries that link against the ATF libraries. These test programs: are able to provide separate test cases that the run-time engine can execute individually; have access to rich APIs to simplify the coding of common testing patterns; and can report feedback to the run-time engine on the granularity of each test case. //The recommendation is to write ATF-based test programs// wherever possible given their versatility. Plain test programs should only be used as a transitional step while incorporating existing tests into the new %%FreeBSD%% test suite or, more importantly, to support plugging tests of third-party software over which %%FreeBSD%% has no control. ===== ATF test case parts ===== Each ATF test case is split in three parts: the **body**, which is required; and the **head** and the **cleanup**, which are optional. ==== The head ==== The **head** is used **for the sole purpose** of defining meta-data properties for the test case. The full list of available properties is provided in ''%%kyua-atf-interface(1)%%'', and these are the most common ones: * descr: A textual description of the purpose of the test case. * require.user: Set to 'root' to tell the tests runtime that this test requires root privileges. The test will later be skipped if you are running Kyua as non-root, and the test will be executed otherwise. Please do not use this unless absolutely necessary! You can most likely make your tests run as a regular user. * use.fs: Set to 'true' if the test case creates temporary files in the "current directory". If set to false, the tests runtime will set the "current directory" to an unwritable directory, which will disallow the creation of the temporary files and will make your test fail if it attempts to do so. Disabling access to the file system is generally good practice to ensure that test cases don't "leak" files unexpectedly, but is also beneficial for performance reasons. ==== The body ==== The **body** is the actual meat of the test case. This is just a regular function that executes any code you want and calls special ATF functions to report failures; see below. ===== Test case isolation ===== Kyua isolates the execution of //every test case// to prevent side-effects (such as temporary file leftovers, in-memory data corruption, etc.). In particular: * A test case is **always executed as a subprocess** that is separate from the test runner program. This implies that you cannot pass any in-memory state between the parts. * The current working directory of a test case is changed to a temporary location that gets cleaned up later on automatically. (Set the use.fs property to true in the head if you need to write to this temporary directory.) * The environment of the test case is "sanitized" to get rid of variables that can cause side-effects; e.g. ''%%LC_ALL%%'', ''%%TZ%%'', etc. ====== Structure of the installed test suite ====== The %%FreeBSD%% test suite lives in ''%%/usr/tests/%%'' and is available out of the box on any system (assuming the system was built with ''%%WITH_TESTS%%'', which defaults to disabled at the moment). ===== Test suite definitions ===== The whole ''%%/usr/tests/%%'' tree represents the full %%FreeBSD%% test suite and executing the test suite from that directory will cause all tests to be run. However, every subdirectory of ''%%/usr/tests/%%'' can be considered a test suite of its own. For example, the ''%%/usr/tests/bin/cp/%%'' subdirectory holds the test suite of the ''%%cp(1)%%'' binary and it can be executed on its own without having to run all other tests. This is an important property: any slice of the whole test suite is a test suite of its own and must be able to work independently. ===== Directory structure ===== The test suite in ''%%/usr/tests/%%'' is intended to mimic the layout of the ''%%src%%'' tree as much as possible. For example: * ''%%/usr/src/bin/cp/%%'' -> ''%%/usr/tests/bin/cp/%%'' * ''%%/usr/src/lib/libc/%%'' -> ''%%/usr/tests/lib/libc/%%'' * ''%%/usr/src/usr.bin/cut/%%'' -> ''%%/usr/tests/usr.bin/cut/%%'' ===== Kyuafile scripts ===== The way the above is implemented is by having a ''%%Kyuafile%%'' in every subdirectory of ''%%/usr/tests/%%''. A ''%%Kyuafile%%'' file is a script that defines the test suite hanging from that subdirectory onwards and does so by either registering the test programs that exist within a directory and/or by including ''%%Kyuafile%%'' files from immediate subdirectories. Here come a set of //fictitious// ''%%Kyuafile%%''s for a test suite containing some tests for ''%%cp(1)%%'' and ''%%libc%%'': ==== /usr/tests/Kyuafile ==== syntax(2) include('bin/Kyuafile') include('lib/Kyuafile') ==== /usr/tests/bin/Kyuafile ==== syntax(2) include('cp/Kyuafile') ==== /usr/tests/bin/cp/Kyuafile ==== syntax(2) atf_test_program{name='fast_test'} atf_test_program{name='huge_test'} ==== /usr/tests/lib/Kyuafile ==== syntax(2) include('libc/Kyuafile') ==== /usr/tests/lib/libc/Kyuafile ==== syntax(2) plain_test_program{name='close_test'} plain_test_program{name='open_test', require_progs='some-helper-tool'} plain_test_program{name='read_test'} plain_test_program{name='write_test'} In reality, however, things do not work in this exact way. The top-level ''%%Kyuafile%%'' and the ''%%Kyuafile%%'' of any top-level subdirectory of ''%%/usr/tests%%'' contain a smart script that is able to auto-discover any ''%%*/Kyuafile%%''s. The reasons for this are given below in the section about the build infrastructure. For actual sample ''%%Kyuafile%%'' code, please refer to ''%%src/share/examples/tests/%%''. ====== Build infrastructure for the test suite ====== ===== Basic test suite components ===== The ''%%src/tests/%%'' directory of the source tree provides basic, generic machinery to build the contents of ''%%/usr/tests/%%''. This tree may eventually also include cross-functional tests that do not fit any other directory of the source tree. In particular, the contents of ''%%src/tests/%%'' work this way: * ''%%src/tests/Kyuafile%%'' is a generic Kyua script that scans the current directory and automatically includes any ''%%*/Kyuafile%%''s that are found. This file is installed into ''%%/usr/tests/%%'' and also into any top-level subdirectory in ''%%/usr/tests/%%'' (e.g. ''%%/usr/tests/bin/' and %%''/usr/tests/lib/'). * ''%%src/tests/Makefile%%'' installs ''%%/usr/tests/Kyuafile%%'' and recurses into all other subdirectories of ''%%src/tests/%%'' to do their thing. * ''%%src/tests/bin/Makefile%%'' (and any other ''%%Makefile%%'' for top-level subdirectories) installs the corresponding ''%%Kyuafile%%'' and may recurse into other subdirectories. ===== Makefiles for test programs ===== Test programs are spread throughout the tree as the goal for the code of the test programs is to live next to the code they are testing. All testing code should be placed in a ''%%tests%%'' subdirectory of the corresponding tool or library. For example: ''%%src/lib/libcrypt/tests/%%''. These directories should be conditionally built and installed via the MK_TESTS knob. On 12.x branches and newer, this should be done as follows: ==== snippet from src/lib/libcrypt/Makefile that handles integration ==== HAS_TESTS= SUBDIR.${MK_TESTS}+= tests # `${MK_TESTS}` requires src.opts.mk or bsd.own.mk On 11.x branches and older, this should be done as follows: ==== snippet from src/lib/libcrypt/Makefile that handles integration ==== # `${MK_TESTS}` requires src.opts.mk on 11.x branches; it requires bsd.own.mk on other versions. .if ${MK_TESTS} != "no" SUBDIR+= tests .endif Every directory containing tests must have a ''%%Makefile%%'' that includes ''%%bsd.test.mk%%''. The ''%%Makefile%%'' must also set ''%%TESTSDIR%%'' to the directory in which these tests will be installed, if it does not fit the hierarchy sans the last component, e.g., ''%%/usr/tests/lib/libcrypt%%'' is the default value for ''%%src/lib/libcrypt%%''. For example: ==== src/lib/libcrypt/tests/Makefile ==== # Optional: bsd.own.mk needs to be included when referencing `${TESTSBASE}`, etc. #.include # TESTSDIR: implied value after r289158 shown below. #TESTSDIR= ${TESTSBASE}/lib/libcrypt ATF_TESTS_C= crypt_test .include For actual sample ''%%Makefile%%'' code, please refer to ''%%src/share/examples/tests/%%''. ===== Test program autodiscovery ===== As mentioned above, the top-level ''%%Kyuafile%%''s auto-discover the test programs that appear in the directories they are located. The reason for this is to simplify the definition of the test suite by decoupling the scripts that define the test suite from the actual test programs being installed. Consider, for example, the build machinery to construct ''%%/usr/tests/lib/%%''. The various libraries in ''%%src/lib/%%'' will have their tests in ''%%src/lib/*/tests%%'' and will install them into subdirectories of ''%%/usr/tests/lib/%%''. However, the ''%%/usr/tests/lib/Kyuafile%%'' file is installed from ''%%src/tests/lib/%%'', which has no knowledge of (and shouldn't have) ''%%src/lib/*/tests/%%''.