Integrating TET

[http://tetworks.opengroup.org/ TET] is a test execution framework from the Open Group.

• Support for local, remote and distributed testing. Briefly,
  • In local testing, the test controller and the test cases run on the same system.
  • In remote testing, the test controller runs on a central system while the test cases run on a remote system, possibly one with a different OS version or CPU architecture than the one the test controller is running on.
  • In distributed testing, a given test case can execute on multiple systems simultaneously, in a synchronized way.
• C, C++, Java, Perl, Python and Shell (KSH) APIs for writing test cases. We'd probably only care about the C & C++ APIs.
• Excellent documentation.
• Commercial support available in case someone building a product around FreeBSD wants to upgrade their test framework.
• Finally, a BSD compatible open-source license.

• TET would improve our ability to test FreeBSD.
• A standard and well-documented test API reduces the effort needed to write test cases.
• A standard way to build and execute test cases and standard ways to compare test run reports would help analyze test run results.
• We can use the following open-source TET based test suites relatively easily:
  • [http://www.opengroup.org/testing/downloads/vsx-pcts.html VSX-PCTS]: A verification suite for assessing conformance to ISO/IEC 9945-1 (POSIX®)/FIPS 151-2.
  • [http://www.opengroup.org/testing/downloads/vsclite.html VSC-Lite]: A test suite for 77 commands and utilities from POSIX 1003.2-1992.
  • [http://www.opengroup.org/testing/downloads/vsthlite.html VSTH-Lite]: A test suite for a subset of pthreads - POSIX 1003.1-1996 threads.
  • [http://www.opengroup.org/testing/downloads/vsw5.html VSW5]: A test suite for the X Window System.
  • [http://wiki.x.org/wiki/TestGroup Xorg's X Test Suite]: The test suite used by the XOrg Project.

We are missing a capable test framework in our source tree.

We currently keep a few unit tests and regression test cases under /usr/src/tools/regression/. These tests use ad-hoc ways to build and execute test cases. Test case reporting is also not standardized.

[To be filled in]