tetintegration

This is an old revision of the document!

Integrating TET

FreeBSD Testing Today

We are missing a capable test framework that is part of our main source tree.

This means that building-in testing when working on FreeBSD's base system requires extra steps, and so is harder than should be.

We currently keep our unit tests and regression test cases under /usr/src/tools/regression/. These tests use ad-hoc ways to build and execute their test cases. Test case reporting is also not standardized (though some tests use the Perl 'Test Anything Protocol', see below). Running these tests in an automated way (a test 'tinderbox') is not always possible.

At this point of time we do not archive test logs. Even if we did so, analysis of historical test data would be tedious due to the ad-hoc nature of the test reports.

The desirables from a FreeBSD test framework

  • The ability to write tests that cover all the functionality of the base system.
  • The ability to manage multi-machine tests (i.e., distributed testing).
  • A small, C-based test writing API that is easy to learn.
  • Be capable of testing parts of the system that use threads.
  • Integration with <bsd.*.mk>.
  • Test logs should be easy to parse.
  • Be available as open-source, and under a BSD-compatible license.

What is TET

[http://tetworks.opengroup.org/ TET] is a test execution framework from the Open Group. Its features include:

  • Support for local, remote and distributed testing. Briefly,
    • In local testing, the test controller and the test cases run on the same system.
    • In remote testing, the test controller runs on a central system while the test cases run on a remote system, possibly one with a different OS version or CPU architecture than the one the test controller is running on.
    • In distributed testing, a given test case can execute on multiple systems simultaneously, in a synchronized way.
  • C, C++, Java, Perl, Python and Shell (KSH) APIs for writing test cases.
  • The C APIs are easy to use.
  • Excellent documentation.
  • A text-based log format.
  • The ability to re-run portions of a prior test run.
  • Commercial support available in case someone building a product around FreeBSD wants to upgrade their test framework.
  • Finally, a BSD compatible open-source license.

TET: Pros

  • Having TET in the base system would improves our ability to test FreeBSD in an automated way. Having a 'standard' way to invoke test cases and a standard way to analyse test logs helps when implementing a 'tinderbox'-like system for the management of automated testing.
  • TET's test API is well documented and easy to learn. A standard API reduces the effort needed to write and maintain test cases. The effort involved in writing new test cases can be further reduced by creating Makefile glue in <bsd.test.mk>.
  • Plain text test logs can be archived for subsequent analysis.
  • We can leverage the TET-based test work being done by other groups in the open-source world:
  • We can keep test newer versions of FreeBSD against test suite binaries compiled on older versions of FreeBSD (i.e., testing for ABI compatibility breakage).

TET: Cons

TET makes a set of assumptions about its execution environment that make integrating it in a BSD-compatible way into the base system complex.

  1. Source layout:
    1. We would want to keep imported TET and TET-based test suites under /usr/src/contrib/, as [http://www.freebsd.org/cgi/man.cgi?query=hier&apropos=0&sektion=7&manpath=FreeBSD+7.0-current&format=html hier(7)] dictates.
    2. We would want to control test suite builds using make-based infrastructure under say /usr/src/test/, in a manner similar to the way other contributed software is built in FreeBSD, see [http://www.freebsd.org/cgi/man.cgi?query=src.conf&apropos=0&sektion=0&manpath=FreeBSD+7.0-current&format=html src.conf(5)], and [http://www.freebsd.org/cgi/man.cgi?query=build&apropos=0&sektion=7&manpath=FreeBSD+7.0-current&format=html build(7)]. Regular TET assumes that tcc (the test case controller) would control builds.
  2. Cross-architecture builds:
    1. We would want our test suites to be cross-buildable, like the rest of /usr/src/.
    2. This implies that test suite binaries would get built under /usr/obj/${ARCH}/.
  3. Test suite 'Installation':
    1. The TET API's support files (tcm.o, libapi.a, etc.) would need to be installed under some standard location (say, ${DESTDIR}/usr/test/tet3/?).
    2. We would want to 'install' test suite binaries under a location like ${DESTDIR}/usr/test/. This installation process should be controllable by the knobs in src.conf(5).
    3. The TET configuration files under ${DESTDIR}/usr/test/ would need to be configured to only run tests from this location (like a binary test suite).
  4. TET Configuration:
    1. TET itself can be built in many flavours, e.g., supporting distributed testing or in non-distributed form, supporting thread-aware APIs or not, or built with shared-libraries for the API. We'll need to handle test suites that require one or the other form appropriately.
    2. 'Distributed TET' supports multi-machine operation, but requires additional configuration on the base system (entries in /etc/services, a startup script for daemons in /etc/rc.d/ etc.).
  5. Running the tests:
    1. We would want the test journal from a test run to be placed under /var/ by default (say /var/tet/journals/?). This would allow the /usr/test/ hierarchy to be read-only and sharable across multiple systems.
    2. Handling tcc -c (clean) is tricky. For test suites under ${DESTDIR}/usr/test/ we only want temporary files to be removed.

Stock TET assumes that the directory named by environment variable ${TET_ROOT} contains all the control files needed for a TET run. Test suites are assumed be located under ${TET_ROOT}, and tcc, (the test case controller program) takes charge of all steps of test building, execution and cleaning. Some, but not all, of these assumptions are configurable using the command-line and configuration files.

We'll need FreeBSD-specific documentation to cover the places where we diverge from TET's documented defaults.

Experimental Source Tree

In Perforce: P4 [http://perforce.freebsd.org/depotTreeBrowser.cgi?FSPC=//depot/user/jkoshy/projects/tet //depot/user/jkoshy/projects/tet/].

References & Related Projects

  • tetintegration.1151763836.txt.gz
  • Last modified: 2006/07/01 14:23
  • by JosephKoshy