Differences

This shows you the differences between two versions of the page.

--- testsuite:developerhowto [2013/10/26 23:23] – JulioMerino
+++ testsuite:developerhowto [2023/01/06 12:45] (current) – Add ATF-python details AlexanderChernikov
@@ Line 1: / Line 1: @@
-This page is part of the [[/TestSuite|TestSuite]] project and provides a tutorial on how to add new tests to the test suite and a reference guide for developers.
+This page is part of the [[/TestSuite|TestSuite]] project and provides a tutorial on how to add new tests to the test suite and a reference guide for developers.  It is assumed that you have already gone over [[TestSuite/Structure]], especially the section detailing the naming scheme for the tests.
-This document is based on a [[http://wiki.netbsd.org/tutorials/atf/|similar tutorial in NetBSD]] and a lot of text has been copied form there.
+Before reading this page, make sure to read ''%%tests(7)%%'' and [[http://svnweb.freebsd.org/base/head/share/mk/bsd.README?view=markup&pathrev=HEAD|src/share/mk/bsd.README]] for details.
+This document is based on a [[http://wiki.netbsd.org/tutorials/atf/|similar tutorial in NetBSD]] and a lot of text has been copied from there.
 ====== Adding a new test case ======
-To add a new **test case** to the source tree, look for any test program in the tree that can assimilate it.  If you find such a program, just add the test case to it: no other changes are required so your life is easy.  Otherwise, you will have to create a new test program.
+To add a new **test case** to the source tree, check if any existing test program can assimilate your test case.  What you are looking for here is a test program with a generic-enough name that can encapsulate the behavior of the new test you want to add.  If test programs use the right naming scheme, and if the code you are modifying already has tests, it is very likely that you will find a match.
-If you need to add a new **test program** to the source tree, you first need to consider the type of test program you will be creating.  It is recommended that you create an ATF-based test program, as these are more versatile.  Restrict yourself to using plain test programs only as a transitional step or, especiallly, if you do not own the code of the test (e.g. a test from a third-party package bundled in the base system).
+If you find such a program, bingo!  Just add the test case to it: no other changes are required.  Otherwise, you will have to create a new test program.
-Test programs should be named ''%%foo_test%%'' where ''%%foo%%'' is the name of the module or utility you are testing.
+====== Adding a new test program ======
-The procedure to add a new test program:
+If you need to add a new **test program** to the source tree, you first need to consider the type of test program you will be creating.  The recommendation is that you create an ATF-based test program; the rationale being that these are more versatile both from a development perspective and the user's experience.  Only resort to adding a plain test program as a transitional step or, especially, if you do not own the code of the test (e.g. you are just hooking a test from a third-party package bundled in the base system).
-  - Locate the appropriate subdirectory in which to put your test program.  In general this just means the ''%%tests%%'' subdirectory located where the code you are interested in testing lives.  It is OK and //expected// to have multiple test programs into the same directory.  **Restrain yourself from creating one directory per test program!**
+The procedure to add a new test program is the following:
-If the subdirectory exists:
+  - Locate the appropriate subdirectory in which to put your test program.  In general this just means the ''%%tests%%'' subdirectory located where the code you are interested in testing lives.  It is OK and //expected// to have multiple test programs into the same directory.  **Avoid creating a directory per test program!**
-  - Choose a sane name for the test program; the name must not be so specific that it restricts the addition of future test cases into it.
+If the ''%%tests%%'' subdirectory exists:
-  - Create the test program source file using one of the templates below.  E.g. ''%%src/lib/libc/tests/write_test.c%%''.
+  - Choose a sane name for the test program.  **Avoid naming the test program the same as your test case!**  If you find yourself in this situation, it probably means that you are choosing a bad name for either of them.
-  - Add the new test program to the ''%%Makefile%%''.
+  - Create the test program source file.  Refer to ''%%share/examples/tests/%%'' for test program sample code.
-If the subdirectory does not exist:
+  - Add the new test program to the existing ''%%Makefile%%''.
+If the ''%%tests%%'' subdirectory does not exist:
+  - Create an empty ''%%tests%%'' subdirectory.
   - Do the same as above.
-  - Create the ''%%Makefile%%'' for the directory using the templates below.
+  - Create the ''%%Makefile%%'' for the directory.  Refer to ''%%share/examples/tests/%%'' for ''%%Makefile%%'' sample code.
   - Edit the parent ''%%Makefile%%'' to recurse into the new subdirectory.
-  - Edit ''%%src/etc/mtree/BSD.tests.dist%%'' to register the new subdirectory.
+  - Edit ''%%etc/mtree/BSD.tests.dist%%'' to register the new subdirectory.  Note that if you are adding tests to, say, ''%%usr.bin/du/tests/%%'', the directory you register in the mtree file should be ''%%/usr/tests/usr.bin/du/%%''; i.e. the layout under ''%%/usr/tests/%%'' must match the layout of ''%%/usr/src%%''.
-===== Makefile template =====
+====== Adding a new top-level directory to /usr/tests/ ======
-Follow this template to create your ''%%Makefile%%''.  This assumes you are adding an ATF-based test program:
+If you need to add a test program to a top-level directory within ''%%/usr/tests/%%'' that does not exist yet, you will have to populate such directory first with a corresponding ''%%Kyuafile%%''.
-<code>
+<callout type="info">
-.include <bsd.own.mk>
+If you are looking to add kernel-level tests and don't find a ''%%sys/tests%%'' directory, that's expected.  Kernel-level tests are supposed to live in ''%%tests/sys/%%'' due to build system oddities.
+</callout>
-# This must always be defined.
-TESTSDIR= ${TESTSBASE}/bin/ls
-# These correspond to the test programs you have in the directory.
-ATF_TESTS_C+= c1_test c2_test  # Correspond to c1_test.c and c2_test.c.
-ATF_TESTS_SH+= sh1_test sh2_test  # Correspond to sh1_test.c and sh2_test.c
-# Define only if your tests need any data files.
-FILESDIR= ${TESTSDIR}
-FILES= testdata1.txt testdata2.bin  # Any necessary data files.
-.include <atf.test.mk>
-</code>
-====== Writing ATF-based test programs ======
-This section provides generic documentation on how to write ATF-based test programs both using the C interface and the shell interface.
+To do this, follow these steps (which assume you would like to register a ''%%tests%%'' subdirectory for ''%%usr.bin%%''):
-===== C test programs =====
+  - Edit ''%%etc/mtree/BSD.tests.dist%%'' to register the new top-level directory.
-The API to write C test programs is detailed in ''%%atf-c-api(3)%%''.  Please refer to that manual page for questions.
+  - Edit ''%%usr.bin/Makefile%%'' and:
-==== Template ====
+    - Add ''%%src.opts.mk%%'' (after r265419), ''%%bsd.init.mk%%'' or ''%%bsd.own.mk%%'' at the top of the Makefile if needed.  Make sure that the relevant .mk file is included **//after//** any override variables (LIBDIR, etc), otherwise including the .mk file might break the install path, e.g. [[https://svnweb.freebsd.org/base?view=revision&revision=271241|r271241]].
-The following code snippet provides a C test program with two test cases.  The specific details as to how this works follow later:
+    - Add the contents below at the end of the file (before any file inclusions).  Note that this may not work for tricky Makefiles that mess around with ''%%SUBDIR%%'' in strange ways.  All of these Makefiles already deal with subdirectories, so check how they do so and register ''%%tests%%'' using the existing format.  The standard way, however, is:
 <code>
-#include <atf-c.h>
+.if ${MK_TESTS} != "no"
+SUBDIR+= tests
-ATF_TC(tc, my_test_case);
+.endif
-ATF_TC_HEAD(tc, my_test_case)
-{
-    atf_tc_set_md_var(tc, "require.machine", "amd64");
-}
-ATF_TC_BODY(tc, my_test_case)
-{
-    ATF_CHECK(true); /* Success; continue execution. */
-    ATF_CHECK(false); /* Failure; continue execution. */
-    ATF_CHECK_EQ(5, 2 + 2); /* Failure; continue execution. */
-    ATF_REQUIRE_EQ(5, 2 + 2); /* Failure; abort execution. */
-    if (!condition)
-        atf_tc_fail("Condition not met!"); /* Abort execution. */
-}
-ATF_TC(tc, another_test_case);
-ATF_TC_HEAD(tc, another_test_case)
-{
-    atf_tc_set_md_var(tc, "require.progs", "gpart");
-}
-ATF_TC_BODY(tc, another_test_case)
-{
-    /* Do more tests here... */
-}
-ATF_TP_ADD_TCS(tp)
-{
-    ATF_TP_ADD_TC(tp, my_test_case);
-    ATF_TP_ADD_TC(tp, another_test_case);
-}
 </code>
-This program needs to be built with the ''%%Makefile%%'' shown below.  Once built, the program automatically gains a ''%%main()%%'' method that provides a consistent user interface to all test programs.  You are simply not intended to provide your own main method, nor to deal with the command-line of the invocation.
+Some complete examples can be found here:
-==== How to build ====
+  * [[https://svnweb.freebsd.org/base/head/lib/libnv/Makefile?revision=271241&view=markup|lib/libnv]]
+  * [[https://svnweb.freebsd.org/base/head/usr.bin/tr/Makefile?revision=265420&view=markup|usr.bin/tr]]
-To build a C test program, append the name of the test program (without the ''%%.c%%'' extension) to the ''%%ATF_TESTS_C%%'' variable in the ''%%Makefile%%''.
+  - Create ''%%usr.bin/tests/Makefile%%'' and add the lines below to it.  Make sure to replace ''%%usr.bin%%'' with the name of your new directory.  These lines cause the generic ''%%Kyuafile%%'' from ''%%tests/Kyuafile%%'' to be installed in your new directory so that tests living in subdirectories can automatically be discovered.
-For example:
 <code>
-.include <bsd.own.mk>
+# $FreeBSD$
-TESTSDIR= ${TESTSBASE}/bin/ls
-ATF_TESTS_C+= fs_test ui_test
-.include <atf.test.mk>
-</code>
-==== Common functions ====
-The following functions are commonly used from within a test case body:
-  * ''%%ATF_REQUIRE(boolean_expression)%%'': Checks if the given boolean expression is true and, if not, aborts execution and marks the test as failed.  Similarly ''%%ATF_CHECK%%'' performs the same test but does not abort execution: it records the failure but keeps processing the test case.  For an explanation on when to use which, refer to the FAQ question below.
-  * ''%%ATF_REQUIRE_EQ(expected_expression, actual_expression)%%'': Checks if the two expressions match and, if not, aborts marking the test as failed.  Similarly, ''%%ATF_CHECK_EQ%%'' records the error but does not abort execution.
-  * ''%%ATF_REQUIRE_STREQ(expected_string, actual_string)%%'': Same as ''%%ATF_REQUIRE_EQ%%'' but performs string comparisons with strcmp.
-  * ''%%atf_tc_skip(const char *format, ...)%%'': Marks the test case as skipped with the provided reason and exits.
-  * ''%%atf_tc_fail(const char *format, ...)%%'': Marks the test case as failed with the provided reason and exits.
-  * ''%%atf_tc_pass(void)%%'': Explicitly marks the test case as passed.  This is **implied** when the test case function ends, so you should not use this in general.
-  * ''%%atf_expect_fail(const char *format, ...)%%'': Tells the atf runtime that the code following this call is expected to raise one or more failures (be it with ''%%atf_tc_fail%%'', ''%%ATF_REQUIRE_*%%'', etc.).  Use this to mark a block of code that is known to be broken (e.g. a test that reproduces a known bug).  Use the string parameter to provide an explanation about why the code is broken; if possible, provide a PR number.  Lastly, to terminate the "expected failure" code block and reset the runtime to the default functionality, use the ''%%atf_expect_pass()%%'' function.
-  * ''%%atf_expect_death(const char *format, ...)%%'': Same as atf_expect_fail but expects an abrupt termination of the test case, be it due to a call to ''%%exit()%%'' or to the reception of a signal.
-  * ''%%atf_expect_exit(int exitcode, const char *format, ...)%%'': Same as ''%%atf_expect_fail%%'' but expects the test case to exit with a specific exitcode.  Provide -1 to indicate any exit code.
-  * ''%%atf_expect_signal(int signo, const char *format, ...)%%'': Same as atf_expect_fail but expects the test case to receive a specific signal.  Provide -1 to indicate any signal.
-  * ''%%atf_expect_timeout(const char *reason, ...)%%'': Same as atf_expect_fail but expects the test case to get stuck and time out.
-  * ''%%atf_tc_get_config_var("srcdir")%%'': Returns the path to the directory containing the test program binary.  This must be used to locate any data/auxiliary files stored alongside the binary.
-  * ''%%RL(integer_expression, integer)%%'': Used to evaluate a call to a libc function that updates errno when it returns an error and to provide correct error reporting.  The integer expression is the call to such function, and the literal integer provides the expected return value when there is an error.  For example: ''%%RL(open("foo", O_RDONLY), -1)%%''.  This would fail the test case if open returns -1, and would record the correct error message returned by libc.
-===== Shell test programs =====
-The API to write shell test programs is detailed in ''%%atf-sh-api(3)%%''.  Please refer to that manual page for questions.
-==== Template ====
-The following code snippet provides a shell test program with two test cases.  The details on how this works are provided later:
-<code>
-atf_test_case my_test_case
-my_test_case_head() {
-    atf_set "descr" "require.machine" "amd64"
-}
-my_test_case_body() {
-    touch file1 file2
-    cat >expout <<EOF
-file1
-file2
-EOF
-    # The following call validates that the 'ls' command returns an
-    # exit code of 0, that its stdout matches exactly the contents
-    # previously stored in the 'expout' file and that its stderr is
-    # completely empty.  See atf-check(1) for details, which is the
-    # auxiliary tool invoked by the atf_check wrapper function.
-    atf_check -s eq:0 -o file:expout -e empty 'ls'
-    atf_check_equal 4 $((2 + 2))
-    if [ 'a' != 'b' ]; then
-        atf_fail "Condition not met!"  # Explicit failure.
-    fi
-}
-atf_test_case another_test_case
-another_test_case_head() {
-    atf_set "require.progs" "gpart"
-}
-another_test_case_body() {
-    # Do more tests...
-}
-atf_init_test_cases() {
-    atf_add_test_case my_test_case
-    atf_add_test_case another_test_case
-}
-</code>
-This program needs to be built with the ''%%Makefile%%'' shown below.  The program automatically gains an entry point that provides a consistent user interface to all test programs.  You are simply not intended to provide your own "main method", nor to deal with the command-line of the invocation.
-==== How to build ====
-To build a shell test program, append the name of the test program (without the .sh extension) to the ''%%ATF_TESTS_SH%%'' variable in the ''%%Makefile%%''.
-For example:
-<code>
 .include <bsd.own.mk>
-TESTSDIR= ${TESTSBASE}/bin/ls
+TESTSDIR=       ${TESTSBASE}/usr.bin
-ATF_TESTS_SH+= integration_test something_else_test
+.PATH:          ${SRCTOP}/tests # This is required for tests/Kyuafile
+KYUAFILE=       yes
-.include <atf.test.mk>
+.include <bsd.test.mk>
 </code>
-If you want to run the test program yourself, you should know that shell-based test programs are processed with the atf-sh interpreter.  atf-sh is just a thin wrapper over ''%%/bin/sh%%'' that loads the shared atf code and then delegates execution to your source file.
+  - Add your tests within ''%%usr.bin/tests/%%'' as usual.
-==== Common functions ====
+====== Adding other intermediate directories ======
-The following functions are commonly used from within a test case body:
+If you are trying to add tests to subdirectories that are more than two levels deep from the ''%%/usr/tests/%%'' root, then the above instructions are insufficient and you will need to add more intermediate ''%%Kyuafile%%''s.  Kyua requires every directory in a path to have a single ''%%Kyuafile%%'' that tells it how to recurse into the next directory, so the source tree must provide all these.  For any given subdirectory that lacks a ''%%Kyuafile%%'', you will have to provide it.
-  * ''%%atf_check%%'': This is probably the most useful function for shell-based tests.  It may need some experience to get it right, but it allows, in one line, to check the execution of a command.  Where check means: validate exit code, stdout and stderr.  This is just a wrapper over atf-check, so please refer to ''%%atf-check(1)%%''.
+As an example, consider the case where we want to install ''%%/usr/tests/usr.bin/foo/bar/baz_test%%''.  If you follow the previous instructions, you would end up with ''%%Kyuafile%%''s in ''%%/usr/tests/usr.bin/%%'' and ''%%/usr/tests/usr.bin/foo/bar/%%'', but not in ''%%/usr/tests/usr.bin/foo/%%''.  To install the missing file:
-  * ''%%atf_check_equal value1 value2%%'': Check that the two values are equal and, if not, abort execution.
+  - Edit ''%%usr.bin/foo/Makefile%%'' and tell it to recurse into ''%%tests%%'' (same as above).
-  * ''%%atf_expect_*%%'': Same as their C counterparts; see above.
+  - Create ''%%tests/usr.bin/foo/tests/Makefile%%'' with the same procedure as above.  The only difference is that you will have to tweak the ''%%.PATH:%%'' stanza to add additional ''%%:H%%'' modifiers for every other subdirectory you are traversing and adjust ''%%TESTSDIR%%'' accordingly:
-  * ''%%atf_fail%%'' reason: Explicitly marks the test case as failed and aborts it.
-  * ''%%atf_skip%%'' reason: Explicitly marks the test case as skipped and exits.
-  * ''%%atf_pass%%'': Explicitly marks the test case as passed and exits.
-  * ''%%atf_get_srcdir%%'': Prints the path to the directory where the test case lives.  Use as ''%%$(atf_get_srcdir)/my-static-data-file%%''.
-====== FAQ ======
-===== How do I atfify a plain test program? =====
-Let's suppose you have a program to exercise a particular piece of code.  The code below, which does not use atf, implements what seem to be 4 different tests (see the calls to ''%%errx(3)%%'') bundled in one test case (the ''%%main()%%'' function):
 <code>
-#include <err.h>
+# $FreeBSD$
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-/* This test program exercises the snprintf function. */
+# This isn't required if the test program component is under usr.bin/foo/tests ; see
+# https://svnweb.freebsd.org/base?view=revision&revision=289158
+#TESTSDIR=       ${TESTSBASE}/usr.bin/foo
-int main(void)
+.PATH:          ${SRCTOP}/tests
-{
+KYUAFILE=       yes
-    char buf[1024];
-    printf("Testing integers");
+.include <bsd.test.mk>
-    snprintf(buf, sizeof(buf), "%d", 3);
-    if (strcmp(buf, "3") != 0)
-        errx(EXIT_FAILURE, "%d failed");
-    snprintf(buf, sizeof(buf), "a %d b", 5);
-    if (strcmp(buf, "a 5 b") != 0)
-        errx(EXIT_FAILURE, "%d failed");
-    printf("Testing strings");
-    snprintf(buf, sizeof(buf), "%s", "foo");
-    if (strcmp(buf, "foo") != 0)
-        errx(EXIT_FAILURE, "%s failed");
-    snprintf(buf, sizeof(buf), "a %s b", "bar");
-    if (strcmp(buf, "a bar b") != 0)
-        errx(EXIT_FAILURE, "%s failed");
-    return EXIT_SUCCESS;
-}
 </code>
-To convert this program into an atf test program, use the template above and keep this in mind:
+====== FAQ for ATF test programs ======
-  * Split the whole main function into separate test cases.  In this scenario, the calls to ''%%printf(3)%%'' delimit a good granularity for the test cases: one for the integer formatter, one for the string formatter, etc.
+===== Where can I find sample code? =====
-  * Use the ''%%ATF_CHECK*%%'' and/or ''%%atf_tc_fail%%'' functions to do the comparisons and report errors.  Neither ''%%errx%%'' nor any other error reporting and program termination functions (read: ''%%err%%'', ''%%errx%%'', ''%%warn%%'', ''%%warnx%%'', ''%%exit%%'' or, worse, ''%%abort%%'') are to be used in an ATF-based test program.
+Refer to [[https://cgit.freebsd.org/src/tree/share/examples/tests/tests/atf|share/examples/tests/tests/atf/]] for the sample C/shell test program code and the supporting build code.
+Refer to [[https://cgit.freebsd.org/src/tree/tests/examples|tests/examples/]] for the sample python code.
-The result would look like:
+===== Where are the APIs documented? =====
-<code>
+See [[https://www.freebsd.org/cgi/man.cgi?query=atf-c-api&apropos=0&sektion=3&arch=default&format=html|atf-c-api(3)]], [[https://www.freebsd.org/cgi/man.cgi?query=atf-c%2B%2B-api&apropos=0&sektion=3&arch=default&format=html|atf-c++-api]] and [[https://www.freebsd.org/cgi/man.cgi?query=atf-sh-api&apropos=0&sektion=3&arch=default&format=html|atf-sh-api(3)]] for details on the C, C++ and shell APIs respectively.  These manual pages are part of the base system.
-#include <atf-c.h>
-#include <stdio.h>
-ATF_TC_WITHOUT_HEAD(tc, integer_formatter);
+Python interface is not documented yet. Examples of the common use cases can be found in the [[https://cgit.freebsd.org/src/tree/tests/examples|tests/examples/]] directory. ATF pytest interface sources can be found in [[https://cgit.freebsd.org/src/tree/tests/atf_python|tests/atf_python]].
-ATF_TC_BODY(tc, integer_formatter)
-{
-    char buf[1024];
-    snprintf(buf, sizeof(1024), "%d", 3);
+===== Do I need to remove any temporary files I create? =====
-    ATF_CHECK_STREQ("3", buf);
-    snprintf(buf, sizeof(1024), "a %d b", 5);
+No, you don't if the files/directories are created relative to the starting directory.
-    ATF_CHECK_STREQ("a 5 b", buf);
-}
-ATF_TC_WITHOUT_HEAD(tc, string_formatter);
+"kyua test" creates a sandbox for test program when the test is executed, so each **test case** is executed as its own process with its own temporary subdirectory. Simply put, removing files relative to the test sandbox isn't required and is discouraged because it adds unnecessary complexity to test programs.
-ATF_TC_BODY(tc, string_formatter)
-{
-    char buf[1024];
-    snprintf(buf, sizeof(1024), "%s", "foo");
+===== When do I use ATF_CHECK and when ATF_REQUIRE? =====
-    ATF_CHECK_STREQ("foo", buf);
-    snprintf(buf, sizeof(1024), "a %s b", "bar");
+''%%ATF_CHECK%%'' logs errors but does not abort the execution of the test program.  ''%%ATF_REQUIRE%%'' logs errors in a similar way but immediately terminates the execution.
-    ATF_CHECK_STREQ("a bar b", buf);
-}
-ATF_TP_ADD_TCS(tp)
+You can use this distinction in the following way: use ''%%ATF_REQUIRE%%'' to check the code that "prepares" your test case.  Use ''%%ATF_CHECK%%'' to do the actual functionality tests once all the set up has been performed.
-{
-    ATF_TP_ADD_TC(tp, integer_formatter);
-    ATF_TP_ADD_TC(tp, string_formatter);
-}
-</code>
-Which can later be invoked as any of:
+====== FAQ for GoogleTest test programs ======
-<code>
+===== Where can I find sample code? =====
-$ kyua test snprintf_test  # Normal execution method.
-$ kyua debug snprintf_test:integer_formatter  # For debugging, with isolation.
-$ ./snprintf_test integer_formatter  # For debugging, without isolation.
-</code>
-===== Do I have to atfify the old tests? =====
+Refer to ''%%contrib/googletest/googletest/samples%%'' for the sample test programs' code; refer to ''%%share/examples/tests/tests/googletest%%'' for the supporting build code.
-Not really, but you should.
+====== FAQ for plain test programs ======
-Keep in mind that Kyua supports running these legacy test programs just fine, so you can technically keep them.  Use the ''%%plain.test.mk%%'' ''%%Makefile%%'' fragment to build them.
+===== Where can I find sample code? =====
-There are two cases in which you may want to keep plain test programs:
+Refer to ''%%share/examples/tests/tests/plain%%'' for sample test program code and the supporting build code.
-  - Legacy tests.  If you are converting existing tests, a first good step is just to move the code around to plug them into the test suite without actually changing their code.  Plugging tests into the test suite has a bigger impact than spending hours tweaking their code.
+====== FAQ for TAP test programs ======
-However once you are done with this first step, you should spend the time to actually convert the test program into the ATF format.  This will later allow the addition of new test cases into the program and will give you additional flexibility.
+===== Where can I find sample code? =====
-  - Third-party tests.  Any tests that come from source not maintained by %%FreeBSD%% should be kept unmodified and plugged into the test suite "as they are".
+Refer to ''%%share/examples/tests/tests/tap%%'' for sample test program code and the supporting build code.
-===== How do I write a test case for an unfixed PR? =====
+----
-Use the "expectations" mechanism to define part of the test case as faulty, crashy, etc.  This is for two reasons:
+{{tag>HowTo}}
-  * As long as the bug still exists, the test case will be reported as an "expected failure".  Such expected failures do not count towards the success or failure of the whole test suite.
-  * When the bug gets fixed, the bug will not trigger any more in the test case, and thus the expectation of failure will not be met any more.  At this point the test case will start raising a regular failure, which is usually addressed by just removing the ''%%expect_*%%'' calls (but add a comment with the PR number!).
-For example, suppose we have PR lib/1 that reports a condition in which ''%%snprintf()%%'' does the wrong formatting when using ''%%%s%%'', and PR lib/2 that mentions that another ''%%snprintf()%%'' call using %d with number 5 causes a segfault.  We could do:
-<code>
-#include <atf-c.h>
-#include <signal.h>
-#include <stdio.h>
-ATF_TC_WITHOUT_HEAD(tc, integer_formatter);
-ATF_TC_BODY(tc, integer_formatter)
-{
-    char buf[1024];
-    snprintf(buf, sizeof(buf), "Hello %d\n", 1);
-    ATF_CHECK_STREQ("Hello 1", buf);
-    atf_tc_expect_signal(SIGSEGV, "PR lib/2: %%d with 5 causes a crash");
-    snprintf(buf, sizeof(buf), "Hello %d\n", 5);
-    atf_tc_expect_pass();
-    ATF_CHECK_STREQ("Hello 5", buf);
-}
-ATF_TC_WITHOUT_HEAD(tc, string_formatter);
-ATF_TC_BODY(tc, string_formatter)
-{
-    char buf[1024];
-    snprintf(buf, sizeof(buf), "Hello %s\n", "world!");
-    atf_tc_expect_failure("PR lib/1: %%s does not work");
-    ATF_CHECK_STREQ("Hello world!", buf);
-    atf_tc_expect_pass();
-}
-ATF_TP_ADD_TCS(tp)
-{
-    ATF_TP_ADD_TC(tp, integer_formatter);
-    ATF_TP_ADD_TC(tp, string_formatter);
-}
-</code>
-===== Do I need to remove temporary files? =====
-No.  Kyua does this automatically for you because it runs every test program in its own temporary subdirectory.  Having code to remove files, or to perform any other kind of cleanup, is usually wrong and you should avoid it.  Keep things simple!
-===== When do I use ATF_CHECK and when ATF_REQUIRE? =====
-''%%ATF_CHECK%%'' logs errors but does not abort the execution of the test program.  ''%%ATF_REQUIRE%%'' logs errors in a similar way but immediately terminates the execution.
-You can use this distinction in the following way: use ''%%ATF_REQUIRE%%'' to check the code that "prepares" your test case.  Use ''%%ATF_CHECK%%'' to do the actual functionality tests once all the set up has been performed.  For example:
-<code>
-ATF_TC_BODY(getline) {
-    FILE *f;
-    char buf[1024];
-    /* Opening the file is not part of the functionality under test, but it
-     * must succeed before we actually test the relevant code. */
-    ATF_REQUIRE((f = fopen("foo")) != NULL);
-    ATF_CHECK(getline(f, buf, sizeof(buf)) > 0);
-    ATF_CHECK_STREQ("line 1", buf);
-    ATF_CHECK(getline(f, buf, sizeof(buf)) > 0);
-    ATF_CHECK_STREQ("line 2", buf);
-}
-</code>