Integrating TET

We are missing a capable test framework that is part of our main source tree.

This means that building-in testing when working on FreeBSD's base system requires extra steps, and so is harder than should be.

We currently keep our unit tests and regression test cases under /usr/src/tools/regression/. These tests use ad-hoc ways to build and execute their test cases. Test case reporting is also not standardized (though some tests use the Perl 'Test Anything Protocol', see below). Running these tests in an automated way (a test 'tinderbox') is not always possible.

At this point of time we do not archive test logs. Even if we did so, analysis of historical test data would be tedious due to the ad-hoc nature of the test reports.

• The ability to write tests that cover all the functionality of the base system.
• The ability to manage multi-machine tests (i.e., distributed testing).
• A small, C-based test writing API that is easy to learn.
• Be capable of testing parts of the system that use threads.
• Integration with <bsd.*.mk>.
• Test logs should be easy to parse.
• Be available as open-source, and under a BSD-compatible license.

TET is a test execution framework from the Open Group. Its features include:

• Support for local, remote and distributed testing. Briefly,
  • In local testing, the test controller and the test cases run on the same system.
  • In remote testing, the test controller runs on a central system while the test cases run on a remote system, possibly one with a different OS version or CPU architecture than the one the test controller is running on.
  • In distributed testing, a given test case can execute on multiple systems simultaneously, in a synchronized way.
• C, C++, Java, Perl, Python and Shell (KSH) APIs for writing test cases.
• The C APIs are easy to use.
• Excellent documentation.
• A text-based log format.
• The ability to re-run portions of a prior test run.
• Commercial support available in case someone building a product around FreeBSD wants to upgrade their test framework.
• Finally, a BSD compatible open-source license.

• Having TET in the base system would improves our ability to test FreeBSD in an automated way. Having a 'standard' way to invoke test cases and a standard way to analyse test logs helps when implementing a 'tinderbox'-like system for the management of automated testing.
• TET's test API is well documented and easy to learn. A standard API reduces the effort needed to write and maintain test cases. The effort involved in writing new test cases can be further reduced by creating Makefile glue in <bsd.test.mk>.
• Plain text test logs can be archived for subsequent analysis.
• We can leverage the TET-based test work being done by other groups in the open-source world:
  • The following “open-source” test suites from the Open Group are TET based.
    • VSX-PCTS: A verification suite for assessing conformance to ISO/IEC 9945-1 (POSIX®)/FIPS 151-2.
    • VSC-Lite: A test suite for 77 commands and utilities from POSIX 1003.2-1992.
    • VSTH-Lite: A test suite for a subset of pthreads - POSIX 1003.1-1996 threads.
    • VSW5: A test suite for the X Window System.
  • Xorg's X Test Suite: The test suite used by the XOrg Project.
  • The Linux Standards Base (LSB) test suites use TET.
• We can keep test newer versions of FreeBSD against test suite binaries compiled on older versions of FreeBSD (i.e., testing for ABI compatibility breakage).

TET makes a set of assumptions about its execution environment that make integrating it in a BSD-compatible way into the base system complex.

1. Source layout:
   1. We would want to keep imported TET and TET-based test suites under /usr/src/contrib/, as hier(7) dictates.
   2. We would want to control test suite builds using make-based infrastructure under say /usr/src/test/, in a manner similar to the way other contributed software is built in FreeBSD, see src.conf(5), and build(7). Regular TET assumes that tcc (the test case controller) would control builds.
2. Cross-architecture builds:
   1. We would want our test suites to be cross-buildable, like the rest of /usr/src/.
   2. This implies that test suite binaries would get built under /usr/obj/${ARCH}/.
3. Test suite 'Installation':
   1. The TET API's support files (tcm.o, libapi.a, etc.) would need to be installed under some standard location (say, ${DESTDIR}/usr/test/tet3/?).
   2. We would want to 'install' test suite binaries under a location like ${DESTDIR}/usr/test/. This installation process should be controllable by the knobs in src.conf(5).
   3. The TET configuration files under ${DESTDIR}/usr/test/ would need to be configured to only run tests from this location (like a binary test suite).
4. TET Configuration:
   1. TET itself can be built in many flavours, e.g., supporting distributed testing or in non-distributed form, supporting thread-aware APIs or not, or built with shared-libraries for the API. We'll need to handle test suites that require one or the other form appropriately.
   2. 'Distributed TET' supports multi-machine operation, but requires additional configuration on the base system (entries in /etc/services, a startup script for daemons in /etc/rc.d/ etc.).
5. Running the tests:
   1. We would want the test journal from a test run to be placed under /var/ by default (say /var/tet/journals/?). This would allow the /usr/test/ hierarchy to be read-only and sharable across multiple systems.
   2. Handling tcc -c (clean) is tricky. For test suites under ${DESTDIR}/usr/test/ we only want temporary files to be removed.

Stock TET assumes that the directory named by environment variable ${TET_ROOT} contains all the control files needed for a TET run. Test suites are assumed be located under ${TET_ROOT}, and tcc, (the test case controller program) takes charge of all steps of test building, execution and cleaning. Some, but not all, of these assumptions are configurable using the command-line and configuration files.

We'll need FreeBSD-specific documentation to cover the places where we diverge from TET's documented defaults.

STAF/STAX: STAF (Software Testing Automation Framework) IBM is a test automation framework built over XML, Python(Jython) and other technologies. STAF/STAX offer a way to distribute tests (to multiple machines) and to collect and present the results.
STAF/STAX require a large number of dependencies. TET is much smaller, and yet, appears to offer equivalent functionality.

TAP: TAP is a text based protocol for reporting test results. Originally developed by the Perl community, there are now language bindings from many languages, including C, Python, PHP and Javascript. TAP is not a full-fledged test framework.

• P4 //depot/user/jkoshy/projects/tet/: main source tree for integration experiments.
• P4 //depot/user/jkoshy/projects/libbsdelf/ts/: LibElf's test suite is TET based.

• WP3 - Testing Framework and Quality Assurance Portal: This Debian-centric portal offers a collection of links to other open-source test frameworks.
• Nik Clayton's libtap library.
• Open Source Testing: a resource site for open-source test tools.
• Peter Holm's kernel stress test suite http://people.freebsd.org/~pho/stress/.
• STAF/STAX: a complex test framework for (very) large scale test automation.
• Test::Harness: implements PERL's 'Test Anything Protocol'.