Integrating TET

[http://tetworks.opengroup.org/ TET] is a test execution framework from the Open Group.

• Support for local, remote and distributed testing. Briefly,
  • In local testing, the test controller and the test cases run on the same system.
  • In remote testing, the test controller runs on a central system while the test cases run on a remote system, possibly one with a different OS version or CPU architecture than the one the test controller is running on.
  • In distributed testing, a given test case can execute on multiple systems simultaneously, in a synchronized way.
• C, C++, Java, Perl, Python and Shell (KSH) APIs for writing test cases. These APIs are easy to use. (We'd probably only care about the C & C++ APIs.)
• Excellent documentation.
• Commercial support available in case someone building a product around FreeBSD wants to upgrade their test framework.
• Finally, a BSD compatible open-source license.

• It improves our ability to test FreeBSD. A standard way to invoke test cases and a standard way to analyse test logs helps when implementing a 'tinderbox'-like system for regular automated testing.
• TET offers a well-documented test API that reduces the effort needed to write and maintain test cases.
• We can leverage TET based test work being done by other groups.
  • The following “open-source” test suites from the Open Group are TET based. (I haven't yet studied the licenses that these are being released under though).
    • [http://www.opengroup.org/testing/downloads/vsx-pcts.html VSX-PCTS]: A verification suite for assessing conformance to ISO/IEC 9945-1 (POSIX®)/FIPS 151-2.
    • [http://www.opengroup.org/testing/downloads/vsclite.html VSC-Lite]: A test suite for 77 commands and utilities from POSIX 1003.2-1992.
    • [http://www.opengroup.org/testing/downloads/vsthlite.html VSTH-Lite]: A test suite for a subset of pthreads - POSIX 1003.1-1996 threads.
    • [http://www.opengroup.org/testing/downloads/vsw5.html VSW5]: A test suite for the X Window System.
  • [http://wiki.x.org/wiki/TestGroup Xorg's X Test Suite]: The test suite used by the XOrg Project.
  • The [http://freestandards.org/en/Download Linux Standards Base (LSB) test suites] use TET.

We are missing a capable test framework in our source tree.

We currently keep a few unit tests and regression test cases under /usr/src/tools/regression/. These tests use ad-hoc ways to build and execute test cases. Test case reporting is also not standardized.

TET makes a set of assumptions about directory layout that do not fit well in the BSD world.

• $TET_ROOT is assumed to contain binaries and control files for a TET run. We would probably want to keep TET and test suite binaries separate from the sources. For example, we might want to keep TET sources under /usr/src/contrib/ and have TET built in the way contributed code is usually built in FreeBSD – controlled by Makefiles under /usr/bin/ or /usr/sbin/.
• We need to investigate whether test suites can be made to create their binaries under /usr/obj/${ARCH} as part of a make world.
• Where would we 'install' test suites, under /usr/test/?

[To be filled in]