• Student: Shivansh Rai (shivansh@freebsd.org)

• Mentors:
  • Alan Somers (asomers@freebsd.org)

• Brooks Davis (brooks@freebsd.org)

• Ngie Cooper (ngie@freebsd.org)

Smoke testing is a set of light tests which are done for checking basic functionalities of a software to ascertain if the crucial functions work correctly.
FreeBSD currently has a set of tests which are run using the kyua framework. These tests need to be first installed individually before they can be used for testing. This proves to be problematic in cases when direct testing of some newly installed or updated utility has to be performed. It makes testing changes to utilities and libraries difficult as one would like to perform tests (e.g. to ensure a proper build environment) before proceeding for installation.

This project aims to develop a test infrastructure and automation tool along with basic tests to verify if all the base utilities in FreeBSD are linked properly. The testing framework is supposed to ease the process of writing “trivial” test cases which will be run in a completely automated and developer friendly manner without need for any prior installation. Once integrated, the tool will also facilitate further development of tests.

The following steps will be performed in order to understand the problem at hand and the approaches which can be used -

• Note how testing has been traditionally done in FreeBSD.

• Study the kyua framework and the approaches currently being used for testing.

• Study the current problems faced as a developer or tester when writing new test cases, and the problems faced as a release engineer or tester when executing existing test cases.

The file functional_test.c is a simple test file which checks whether the ls utility is properly linked by running trivial commands. The options supported by ls are stored in short_options[] and long_options[] present in functional_test.h.

It should be noted that finally the tests will be automated with appropriately passed options. The above commands will not have to be run for testing individual programs. The test file uses getopt() and getopt_long() for testing the validity of the passed options. If a valid option is passed, the command <utility> --<option(i)> is executed. In case the command fails to execute for a valid option, this will imply that the utility under test is not properly linked.

short_options[] and long_options[] need to be initially populated with a few supported options for all the base utilities. This can be done by using either one or both of the following available approaches -

• Parse man pages for each utility to get the supported options.

• Pass an unsupported option to the utility. This might generate a usage message which can then be parsed. The unsupported option will be chosen experimentally.

An automation script will be written which will populate short_options[] and long_options[] by following the above mentioned approaches and will generate relevant test files. The initial version of the generated test scripts will only test trivial functionalities.

Although the test plan does not check the entire state of application under test, it suffices for the initial purposes of smoke testing.

• An automated tooling to generate test scripts which confirm minimal set of functionalities of a utility to check whether it is properly linked.

• Set of test programs generated by the tool to verify basic functionality of all the base utilities in FreeBSD.

Table design adapted from NonBSMtoBSMConversionTools#Milestones

• Add atf-sh test programs for a subset of base utilities which are currently without any test coverage. The maximum number of utilities possible in the duration will be covered.

• Prepare a basic design of the automation tool. Cover generation of a limited number of test scripts.

• Add functionality to enable users in defining which testcases to ignore while generating tests.

• Generate tests for all section 1 utilities and try to identify cases where the tool behaves unexpectedly. Fix such cases.

• Finish adding functionality to halt test generation during runtime.

• Clean up the code and prepare it for merge into the src tree.

• Add a diagram briefly depicting the architecture.

Note: While visiting links in the following reports to https://shivrai.github.io, they should be renamed to https://shivansh.github.io.