To add a new test case to the source tree, look for any test program in the tree that can assimilate it. If you find such a program, just add the test case to it: no other changes are required so your life is easy. Otherwise, you will have to create a new test program.

If you need to add a new test program to the source tree, you first need to consider the type of test program you will be creating. It is recommended that you create an ATF-based test program, as these are more versatile. Restrict yourself to using plain test programs only as a transitional step or, especiallly, if you do not own the code of the test (e.g. a test from a third-party package bundled in the base system).

Test programs should be named foo_test where foo is the name of the module or utility you are testing.

The procedure to add a new test program:

1. Locate the appropriate subdirectory in which to put your test program. In general this just means the tests subdirectory located where the code you are interested in testing lives. It is OK and expected to have multiple test programs into the same directory. Restrain yourself from creating one directory per test program!

If the subdirectory exists:

1. Choose a sane name for the test program; the name must not be so specific that it restricts the addition of future test cases into it.

1. Create the test program source file using one of the templates below. E.g. src/lib/libc/tests/write_test.c.

1. Add the new test program to the Makefile.

If the subdirectory does not exist:

1. Do the same as above.

1. Create the Makefile for the directory using the templates below.

1. Edit the parent Makefile to recurse into the new subdirectory.

1. Edit src/etc/mtree/BSD.tests.dist to register the new subdirectory.

Follow this template to create your Makefile. This assumes you are adding an ATF-based test program:

.include <bsd.own.mk>

# This must always be defined.
TESTSDIR= ${TESTSBASE}/bin/ls

# These correspond to the test programs you have in the directory.
ATF_TESTS_C+= c1_test c2_test  # Correspond to c1_test.c and c2_test.c.
ATF_TESTS_SH+= sh1_test sh2_test  # Correspond to sh1_test.c and sh2_test.c

# Define only if your tests need any data files.
FILESDIR= ${TESTSDIR}
FILES= testdata1.txt testdata2.bin  # Any necessary data files.

.include <atf.test.mk>

This section provides generic documentation on how to write ATF-based test programs both using the C interface and the shell interface.

The API to write C test programs is detailed in atf-c-api(3). Please refer to that manual page for questions.

The following code snippet provides a C test program with two test cases. The specific details as to how this works follow later:

#include <atf-c.h>

ATF_TC(tc, my_test_case);
ATF_TC_HEAD(tc, my_test_case)
{
    atf_tc_set_md_var(tc, "require.machine", "amd64");
}
ATF_TC_BODY(tc, my_test_case)
{
    ATF_CHECK(true); /* Success; continue execution. */
    ATF_CHECK(false); /* Failure; continue execution. */

    ATF_CHECK_EQ(5, 2 + 2); /* Failure; continue execution. */
    ATF_REQUIRE_EQ(5, 2 + 2); /* Failure; abort execution. */

    if (!condition)
        atf_tc_fail("Condition not met!"); /* Abort execution. */
}

ATF_TC(tc, another_test_case);
ATF_TC_HEAD(tc, another_test_case)
{
    atf_tc_set_md_var(tc, "require.progs", "gpart");
}
ATF_TC_BODY(tc, another_test_case)
{
    /* Do more tests here... */
}

ATF_TP_ADD_TCS(tp)
{
    ATF_TP_ADD_TC(tp, my_test_case);
    ATF_TP_ADD_TC(tp, another_test_case);
}

This program needs to be built with the Makefile shown below. Once built, the program automatically gains a main() method that provides a consistent user interface to all test programs. You are simply not intended to provide your own main method, nor to deal with the command-line of the invocation.

To build a C test program, append the name of the test program (without the .c extension) to the ATF_TESTS_C variable in the Makefile.

For example:

.include <bsd.own.mk>

TESTSDIR= ${TESTSBASE}/bin/ls

ATF_TESTS_C+= fs_test ui_test

.include <atf.test.mk>

The following functions are commonly used from within a test case body:

• ATF_REQUIRE(boolean_expression): Checks if the given boolean expression is true and, if not, aborts execution and marks the test as failed. Similarly ATF_CHECK performs the same test but does not abort execution: it records the failure but keeps processing the test case. For an explanation on when to use which, refer to the FAQ question below.

• ATF_REQUIRE_EQ(expected_expression, actual_expression): Checks if the two expressions match and, if not, aborts marking the test as failed. Similarly, ATF_CHECK_EQ records the error but does not abort execution.

• ATF_REQUIRE_STREQ(expected_string, actual_string): Same as ATF_REQUIRE_EQ but performs string comparisons with strcmp.

• atf_tc_skip(const char *format, ...): Marks the test case as skipped with the provided reason and exits.

• atf_tc_fail(const char *format, ...): Marks the test case as failed with the provided reason and exits.

• atf_tc_pass(void): Explicitly marks the test case as passed. This is implied when the test case function ends, so you should not use this in general.

• atf_expect_fail(const char *format, ...): Tells the atf runtime that the code following this call is expected to raise one or more failures (be it with atf_tc_fail, ATF_REQUIRE_*, etc.). Use this to mark a block of code that is known to be broken (e.g. a test that reproduces a known bug). Use the string parameter to provide an explanation about why the code is broken; if possible, provide a PR number. Lastly, to terminate the “expected failure” code block and reset the runtime to the default functionality, use the atf_expect_pass() function.

• atf_expect_death(const char *format, ...): Same as atf_expect_fail but expects an abrupt termination of the test case, be it due to a call to exit() or to the reception of a signal.

• atf_expect_exit(int exitcode, const char *format, ...): Same as atf_expect_fail but expects the test case to exit with a specific exitcode. Provide -1 to indicate any exit code.

• atf_expect_signal(int signo, const char *format, ...): Same as atf_expect_fail but expects the test case to receive a specific signal. Provide -1 to indicate any signal.

• atf_expect_timeout(const char *reason, ...): Same as atf_expect_fail but expects the test case to get stuck and time out.

• atf_tc_get_config_var("srcdir"): Returns the path to the directory containing the test program binary. This must be used to locate any data/auxiliary files stored alongside the binary.

• RL(integer_expression, integer): Used to evaluate a call to a libc function that updates errno when it returns an error and to provide correct error reporting. The integer expression is the call to such function, and the literal integer provides the expected return value when there is an error. For example: RL(open("foo", O_RDONLY), -1). This would fail the test case if open returns -1, and would record the correct error message returned by libc.

The API to write shell test programs is detailed in atf-sh-api(3). Please refer to that manual page for questions.

The following code snippet provides a shell test program with two test cases. The details on how this works are provided later:

atf_test_case my_test_case
my_test_case_head() {
    atf_set "descr" "require.machine" "amd64"
}
my_test_case_body() {
    touch file1 file2

    cat >expout <<EOF
file1
file2
EOF
    # The following call validates that the 'ls' command returns an
    # exit code of 0, that its stdout matches exactly the contents
    # previously stored in the 'expout' file and that its stderr is
    # completely empty.  See atf-check(1) for details, which is the
    # auxiliary tool invoked by the atf_check wrapper function.
    atf_check -s eq:0 -o file:expout -e empty 'ls'

    atf_check_equal 4 $((2 + 2))

    if [ 'a' != 'b' ]; then
        atf_fail "Condition not met!"  # Explicit failure.
    fi
}

atf_test_case another_test_case
another_test_case_head() {
    atf_set "require.progs" "gpart"
}
another_test_case_body() {
    # Do more tests...
}

atf_init_test_cases() {
    atf_add_test_case my_test_case
    atf_add_test_case another_test_case
}

This program needs to be built with the Makefile shown below. The program automatically gains an entry point that provides a consistent user interface to all test programs. You are simply not intended to provide your own “main method”, nor to deal with the command-line of the invocation.

To build a shell test program, append the name of the test program (without the .sh extension) to the ATF_TESTS_SH variable in the Makefile.

For example:

.include <bsd.own.mk>

TESTSDIR= ${TESTSBASE}/bin/ls

ATF_TESTS_SH+= integration_test something_else_test

.include <atf.test.mk>

If you want to run the test program yourself, you should know that shell-based test programs are processed with the atf-sh interpreter. atf-sh is just a thin wrapper over /bin/sh that loads the shared atf code and then delegates execution to your source file.

The following functions are commonly used from within a test case body:

• atf_check: This is probably the most useful function for shell-based tests. It may need some experience to get it right, but it allows, in one line, to check the execution of a command. Where check means: validate exit code, stdout and stderr. This is just a wrapper over atf-check, so please refer to atf-check(1).

• atf_check_equal value1 value2: Check that the two values are equal and, if not, abort execution.

• atf_expect_*: Same as their C counterparts; see above.

• atf_fail reason: Explicitly marks the test case as failed and aborts it.

• atf_skip reason: Explicitly marks the test case as skipped and exits.

• atf_pass: Explicitly marks the test case as passed and exits.

• atf_get_srcdir: Prints the path to the directory where the test case lives. Use as $(atf_get_srcdir)/my-static-data-file.

Let's suppose you have a program to exercise a particular piece of code. The code below, which does not use atf, implements what seem to be 4 different tests (see the calls to errx(3)) bundled in one test case (the main() function):

#include <err.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

/* This test program exercises the snprintf function. */

int main(void)
{
    char buf[1024];

    printf("Testing integers");
    snprintf(buf, sizeof(buf), "%d", 3);
    if (strcmp(buf, "3") != 0)
        errx(EXIT_FAILURE, "%d failed");
    snprintf(buf, sizeof(buf), "a %d b", 5);
    if (strcmp(buf, "a 5 b") != 0)
        errx(EXIT_FAILURE, "%d failed");

    printf("Testing strings");
    snprintf(buf, sizeof(buf), "%s", "foo");
    if (strcmp(buf, "foo") != 0)
        errx(EXIT_FAILURE, "%s failed");
    snprintf(buf, sizeof(buf), "a %s b", "bar");
    if (strcmp(buf, "a bar b") != 0)
        errx(EXIT_FAILURE, "%s failed");

    return EXIT_SUCCESS;
}

To convert this program into an atf test program, use the template above and keep this in mind:

• Split the whole main function into separate test cases. In this scenario, the calls to printf(3) delimit a good granularity for the test cases: one for the integer formatter, one for the string formatter, etc.

• Use the ATF_CHECK* and/or atf_tc_fail functions to do the comparisons and report errors. Neither errx nor any other error reporting and program termination functions (read: err, errx, warn, warnx, exit or, worse, abort) are to be used in an ATF-based test program.

The result would look like:

#include <atf-c.h>
#include <stdio.h>

ATF_TC_WITHOUT_HEAD(tc, integer_formatter);
ATF_TC_BODY(tc, integer_formatter)
{
    char buf[1024];

    snprintf(buf, sizeof(1024), "%d", 3);
    ATF_CHECK_STREQ("3", buf);

    snprintf(buf, sizeof(1024), "a %d b", 5);
    ATF_CHECK_STREQ("a 5 b", buf);
}

ATF_TC_WITHOUT_HEAD(tc, string_formatter);
ATF_TC_BODY(tc, string_formatter)
{
    char buf[1024];

    snprintf(buf, sizeof(1024), "%s", "foo");
    ATF_CHECK_STREQ("foo", buf);

    snprintf(buf, sizeof(1024), "a %s b", "bar");
    ATF_CHECK_STREQ("a bar b", buf);
}

ATF_TP_ADD_TCS(tp)
{
    ATF_TP_ADD_TC(tp, integer_formatter);
    ATF_TP_ADD_TC(tp, string_formatter);
}

Which can later be invoked as any of:

$ kyua test snprintf_test  # Normal execution method.
$ kyua debug snprintf_test:integer_formatter  # For debugging, with isolation.
$ ./snprintf_test integer_formatter  # For debugging, without isolation.

Not really, but you should.

Keep in mind that Kyua supports running these legacy test programs just fine, so you can technically keep them. Use the plain.test.mk Makefile fragment to build them.

There are two cases in which you may want to keep plain test programs:

1. Legacy tests. If you are converting existing tests, a first good step is just to move the code around to plug them into the test suite without actually changing their code. Plugging tests into the test suite has a bigger impact than spending hours tweaking their code.

However once you are done with this first step, you should spend the time to actually convert the test program into the ATF format. This will later allow the addition of new test cases into the program and will give you additional flexibility.

1. Third-party tests. Any tests that come from source not maintained by FreeBSD should be kept unmodified and plugged into the test suite “as they are”.

Use the “expectations” mechanism to define part of the test case as faulty, crashy, etc. This is for two reasons:

• As long as the bug still exists, the test case will be reported as an “expected failure”. Such expected failures do not count towards the success or failure of the whole test suite.

• When the bug gets fixed, the bug will not trigger any more in the test case, and thus the expectation of failure will not be met any more. At this point the test case will start raising a regular failure, which is usually addressed by just removing the expect_* calls (but add a comment with the PR number!).

For example, suppose we have PR lib/1 that reports a condition in which snprintf() does the wrong formatting when using %s, and PR lib/2 that mentions that another snprintf() call using %d with number 5 causes a segfault. We could do:

#include <atf-c.h>
#include <signal.h>
#include <stdio.h>

ATF_TC_WITHOUT_HEAD(tc, integer_formatter);
ATF_TC_BODY(tc, integer_formatter)
{
    char buf[1024];

    snprintf(buf, sizeof(buf), "Hello %d\n", 1);
    ATF_CHECK_STREQ("Hello 1", buf);

    atf_tc_expect_signal(SIGSEGV, "PR lib/2: %%d with 5 causes a crash");
    snprintf(buf, sizeof(buf), "Hello %d\n", 5);
    atf_tc_expect_pass();
    ATF_CHECK_STREQ("Hello 5", buf);
}

ATF_TC_WITHOUT_HEAD(tc, string_formatter);
ATF_TC_BODY(tc, string_formatter)
{
    char buf[1024];

    snprintf(buf, sizeof(buf), "Hello %s\n", "world!");
    atf_tc_expect_failure("PR lib/1: %%s does not work");
    ATF_CHECK_STREQ("Hello world!", buf);
    atf_tc_expect_pass();
}

ATF_TP_ADD_TCS(tp)
{
    ATF_TP_ADD_TC(tp, integer_formatter);
    ATF_TP_ADD_TC(tp, string_formatter);
}

No. Kyua does this automatically for you because it runs every test program in its own temporary subdirectory. Having code to remove files, or to perform any other kind of cleanup, is usually wrong and you should avoid it. Keep things simple!

ATF_CHECK logs errors but does not abort the execution of the test program. ATF_REQUIRE logs errors in a similar way but immediately terminates the execution.

You can use this distinction in the following way: use ATF_REQUIRE to check the code that “prepares” your test case. Use ATF_CHECK to do the actual functionality tests once all the set up has been performed. For example:

ATF_TC_BODY(getline) {
    FILE *f;
    char buf[1024];

    /* Opening the file is not part of the functionality under test, but it
     * must succeed before we actually test the relevant code. */
    ATF_REQUIRE((f = fopen("foo")) != NULL);

    ATF_CHECK(getline(f, buf, sizeof(buf)) > 0);
    ATF_CHECK_STREQ("line 1", buf);

    ATF_CHECK(getline(f, buf, sizeof(buf)) > 0);
    ATF_CHECK_STREQ("line 2", buf);
}