Differences

This shows you the differences between two versions of the page.

--- testsuite:developerhowto [2013/11/24 18:57] – Remove sample code in favor of the in-tree examples and reword stuff. JulioMerino
+++ testsuite:developerhowto [2023/01/06 12:45] (current) – Add ATF-python details AlexanderChernikov
@@ Line 1: / Line 1: @@
 This page is part of the [[/TestSuite|TestSuite]] project and provides a tutorial on how to add new tests to the test suite and a reference guide for developers.  It is assumed that you have already gone over [[TestSuite/Structure]], especially the section detailing the naming scheme for the tests.
-This document is based on a [[http://wiki.netbsd.org/tutorials/atf/|similar tutorial in NetBSD]] and a lot of text has been copied form there.
+Before reading this page, make sure to read ''%%tests(7)%%'' and [[http://svnweb.freebsd.org/base/head/share/mk/bsd.README?view=markup&pathrev=HEAD|src/share/mk/bsd.README]] for details.
+This document is based on a [[http://wiki.netbsd.org/tutorials/atf/|similar tutorial in NetBSD]] and a lot of text has been copied from there.
 ====== Adding a new test case ======
@@ Line 21: / Line 23: @@
   - Choose a sane name for the test program.  **Avoid naming the test program the same as your test case!**  If you find yourself in this situation, it probably means that you are choosing a bad name for either of them.
-  - Create the test program source file.  Refer to ''%%src/share/examples/tests/%%'' for test program sample code.
+  - Create the test program source file.  Refer to ''%%share/examples/tests/%%'' for test program sample code.
   - Add the new test program to the existing ''%%Makefile%%''.
@@ Line 31: / Line 33: @@
   - Do the same as above.
-  - Create the ''%%Makefile%%'' for the directory.  Refer to ''%%src/share/examples/tests/%%'' for ''%%Makefile%%'' sample code.
+  - Create the ''%%Makefile%%'' for the directory.  Refer to ''%%share/examples/tests/%%'' for ''%%Makefile%%'' sample code.
   - Edit the parent ''%%Makefile%%'' to recurse into the new subdirectory.
-  - Edit ''%%src/etc/mtree/BSD.tests.dist%%'' to register the new subdirectory.  Note that if you are adding tests to, say, ''%%src/usr.bin/du/tests/%%'', the directory you register in the mtree file should be ''%%/usr/tests/usr.bin/du/%%''; i.e. the layout under ''%%/usr/tests/%%'' must match the layout of ''%%/usr/src%%''.
+  - Edit ''%%etc/mtree/BSD.tests.dist%%'' to register the new subdirectory.  Note that if you are adding tests to, say, ''%%usr.bin/du/tests/%%'', the directory you register in the mtree file should be ''%%/usr/tests/usr.bin/du/%%''; i.e. the layout under ''%%/usr/tests/%%'' must match the layout of ''%%/usr/src%%''.
+====== Adding a new top-level directory to /usr/tests/ ======
+If you need to add a test program to a top-level directory within ''%%/usr/tests/%%'' that does not exist yet, you will have to populate such directory first with a corresponding ''%%Kyuafile%%''.
+<callout type="info">
+If you are looking to add kernel-level tests and don't find a ''%%sys/tests%%'' directory, that's expected.  Kernel-level tests are supposed to live in ''%%tests/sys/%%'' due to build system oddities.
+</callout>
+To do this, follow these steps (which assume you would like to register a ''%%tests%%'' subdirectory for ''%%usr.bin%%''):
+  - Edit ''%%etc/mtree/BSD.tests.dist%%'' to register the new top-level directory.
+  - Edit ''%%usr.bin/Makefile%%'' and:
+    - Add ''%%src.opts.mk%%'' (after r265419), ''%%bsd.init.mk%%'' or ''%%bsd.own.mk%%'' at the top of the Makefile if needed.  Make sure that the relevant .mk file is included **//after//** any override variables (LIBDIR, etc), otherwise including the .mk file might break the install path, e.g. [[https://svnweb.freebsd.org/base?view=revision&revision=271241|r271241]].
+    - Add the contents below at the end of the file (before any file inclusions).  Note that this may not work for tricky Makefiles that mess around with ''%%SUBDIR%%'' in strange ways.  All of these Makefiles already deal with subdirectories, so check how they do so and register ''%%tests%%'' using the existing format.  The standard way, however, is:
+<code>
+.if ${MK_TESTS} != "no"
+SUBDIR+= tests
+.endif
+</code>
+Some complete examples can be found here:
+  * [[https://svnweb.freebsd.org/base/head/lib/libnv/Makefile?revision=271241&view=markup|lib/libnv]]
+  * [[https://svnweb.freebsd.org/base/head/usr.bin/tr/Makefile?revision=265420&view=markup|usr.bin/tr]]
+  - Create ''%%usr.bin/tests/Makefile%%'' and add the lines below to it.  Make sure to replace ''%%usr.bin%%'' with the name of your new directory.  These lines cause the generic ''%%Kyuafile%%'' from ''%%tests/Kyuafile%%'' to be installed in your new directory so that tests living in subdirectories can automatically be discovered.
+<code>
+# $FreeBSD$
+.include <bsd.own.mk>
+TESTSDIR=       ${TESTSBASE}/usr.bin
+.PATH:          ${SRCTOP}/tests # This is required for tests/Kyuafile
+KYUAFILE=       yes
+.include <bsd.test.mk>
+</code>
+  - Add your tests within ''%%usr.bin/tests/%%'' as usual.
+====== Adding other intermediate directories ======
+If you are trying to add tests to subdirectories that are more than two levels deep from the ''%%/usr/tests/%%'' root, then the above instructions are insufficient and you will need to add more intermediate ''%%Kyuafile%%''s.  Kyua requires every directory in a path to have a single ''%%Kyuafile%%'' that tells it how to recurse into the next directory, so the source tree must provide all these.  For any given subdirectory that lacks a ''%%Kyuafile%%'', you will have to provide it.
+As an example, consider the case where we want to install ''%%/usr/tests/usr.bin/foo/bar/baz_test%%''.  If you follow the previous instructions, you would end up with ''%%Kyuafile%%''s in ''%%/usr/tests/usr.bin/%%'' and ''%%/usr/tests/usr.bin/foo/bar/%%'', but not in ''%%/usr/tests/usr.bin/foo/%%''.  To install the missing file:
+  - Edit ''%%usr.bin/foo/Makefile%%'' and tell it to recurse into ''%%tests%%'' (same as above).
+  - Create ''%%tests/usr.bin/foo/tests/Makefile%%'' with the same procedure as above.  The only difference is that you will have to tweak the ''%%.PATH:%%'' stanza to add additional ''%%:H%%'' modifiers for every other subdirectory you are traversing and adjust ''%%TESTSDIR%%'' accordingly:
+<code>
+# $FreeBSD$
+# This isn't required if the test program component is under usr.bin/foo/tests ; see
+# https://svnweb.freebsd.org/base?view=revision&revision=289158
+#TESTSDIR=       ${TESTSBASE}/usr.bin/foo
+.PATH:          ${SRCTOP}/tests
+KYUAFILE=       yes
+.include <bsd.test.mk>
+</code>
 ====== FAQ for ATF test programs ======
@@ Line 41: / Line 112: @@
 ===== Where can I find sample code? =====
-Refer to ''%%src/share/examples/tests/tests/atf/%%'' for sample test program code and the supporting build code.
+Refer to [[https://cgit.freebsd.org/src/tree/share/examples/tests/tests/atf|share/examples/tests/tests/atf/]] for the sample C/shell test program code and the supporting build code.
+Refer to [[https://cgit.freebsd.org/src/tree/tests/examples|tests/examples/]] for the sample python code.
-This section provides generic documentation on how to write ATF-based test programs both using the C interface and the shell interface.
 ===== Where are the APIs documented? =====
-See ''%%atf-c-api(3)%%'', ''%%atf-c++-api(3)%%'' and ''%%atf-sh-api(3)%%'' for details on the C, C++ and shell APIs respectively.  These manual pages are part of the base system.
+See [[https://www.freebsd.org/cgi/man.cgi?query=atf-c-api&apropos=0&sektion=3&arch=default&format=html|atf-c-api(3)]], [[https://www.freebsd.org/cgi/man.cgi?query=atf-c%2B%2B-api&apropos=0&sektion=3&arch=default&format=html|atf-c++-api]] and [[https://www.freebsd.org/cgi/man.cgi?query=atf-sh-api&apropos=0&sektion=3&arch=default&format=html|atf-sh-api(3)]] for details on the C, C++ and shell APIs respectively.  These manual pages are part of the base system.
+Python interface is not documented yet. Examples of the common use cases can be found in the [[https://cgit.freebsd.org/src/tree/tests/examples|tests/examples/]] directory. ATF pytest interface sources can be found in [[https://cgit.freebsd.org/src/tree/tests/atf_python|tests/atf_python]].
 ===== Do I need to remove any temporary files I create? =====
-No!  Kyua does this automatically for you.  **Every test case** is executed in its process and its own temporary subdirectory.  Having code to remove files or to perform any other kind of cleanup is usually wrong and you should avoid it.  Keep things simple and let the infrastructure do the hard work for you!
+No, you don't if the files/directories are created relative to the starting directory.
+"kyua test" creates a sandbox for test program when the test is executed, so each **test case** is executed as its own process with its own temporary subdirectory. Simply put, removing files relative to the test sandbox isn't required and is discouraged because it adds unnecessary complexity to test programs.
 ===== When do I use ATF_CHECK and when ATF_REQUIRE? =====
@@ Line 58: / Line 132: @@
 You can use this distinction in the following way: use ''%%ATF_REQUIRE%%'' to check the code that "prepares" your test case.  Use ''%%ATF_CHECK%%'' to do the actual functionality tests once all the set up has been performed.
+====== FAQ for GoogleTest test programs ======
+===== Where can I find sample code? =====
+Refer to ''%%contrib/googletest/googletest/samples%%'' for the sample test programs' code; refer to ''%%share/examples/tests/tests/googletest%%'' for the supporting build code.
 ====== FAQ for plain test programs ======
@@ Line 63: / Line 143: @@
 ===== Where can I find sample code? =====
-Refer to ''%%src/share/examples/tests/tests/plain/%%'' for sample test program code and the supporting build code.
+Refer to ''%%share/examples/tests/tests/plain%%'' for sample test program code and the supporting build code.
+====== FAQ for TAP test programs ======
+===== Where can I find sample code? =====
+Refer to ''%%share/examples/tests/tests/tap%%'' for sample test program code and the supporting build code.
+----
+{{tag>HowTo}}