TestingFreeBSDStatus [FreeBSD WIKI]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Introduction ======

This page notes what things are currently completed/in-progress for testing in %%FreeBSD%%.

Eventually this page will also note coverage in source code as a meaningful rubric for progress.

====== ATF import status ======

  * ATF 0.16 has been imported into %%FreeBSD%% CURRENT and apart from some minor patches, all of the needed support has been added to %%FreeBSD%% (gcooper, marcel).
  * ATF and Kyua ports have been created (asomers, bdrewery).
  * MFCing ATF/bmake support (bmake is required by ATF) has been requested by developers (bapt, et al) to help regression test components on 9-STABLE; MFCing to 8.x was considered a bad idea, but all involved parties (bapt, bdrewery, davide, gcooper) agreed that having ports of ATF/bmake for 8.x was a good idea.

====== Build infrastructure status ======

  * Major build infrastructure pieces (atf.test.mk, bsd.test.mk) have been imported into %%FreeBSD%% in order to provide minimal overhead for developers when writing testcases, testing in-tree and out-of-tree. Minor usability improvements are queued for commit to %%FreeBSD%% current.
  * A Test::Harness testcase shim (prove.test.mk) is queued for commit into %%FreeBSD%%.
  * Official documentation (BSD.readme) is queued for commit.
  * Automatic test "discovery"/handling logic is queued for commit.
  * buildworld/installworld integration is queued for commit.
  * Building a test release package and being able to install it via bsdinstall is incomplete, but work has been done on GitHub to make this come true.
  * Getting the test infrastructure to run on stable/8 and stable/9 is possible, and all of the necessary bits have been pushed back to stable/8 and stable/9.

====== Test Infrastructure Status ======

===== ATF =====

  * Test fixtures (global cleanup/setup, per-testcase setup) need to be added to ATF to achieve feature parity with other test infrastructures (GNU libcheck, Python unittest testcases, Junit, etc).
  * As JulioMerino has noted on his GoogleCodeIn page for [[http://code.google.com/p/kyua|ATF/Kyua]], ATF was just the starting point, and Kyua is the evolution of ATF. So, with that in mind, the work that's being done here is just to provide some degree of feature parity with %%NetBSD%%, etc, and as time progresses, %%FreeBSD%% will catch up with Kyua (which is more evolved than ATF) and will use the test infrastructure [[http://code.google.com/p/kyua/wiki/ATF#Status_of_the_ATF_project|layering]] that Julio Merino has proposed.

===== RUMP =====

[[http://www.netbsd.org/docs/rump/|Rump]] is an extension to the %%NetBSD%% kernel that runs kernel code as a standalone user process for development and testing purposes. It is a great technology for isolating code and testing it using a whitebox method of testing, as opposed to having to use fuzzing techniques, failpoints, blackbox/graybox testing, etc, which ultimately allows developers and testers to write more code that isolates API calls and helps ensure that KPI contracts are sane without panicing a system or corrupting system state.

====== Testcase Status ======

===== ATF Testcases =====

| Directory | Repository | Owner | Status | Notes |
| bin/cat | [[#yatrc|1]] | gcooper | complete | Ported from %%NetBSD%%. |
| bin/date | [[#yatrc|1]] | gcooper | complete | Ported from tools/regression. Research adding coverage from %%NetBSD%%. |
| bin/dd | [[#yatrc|1]] | gcooper | complete | Ported from %%NetBSD%%. Needs greater coverage/porting. Throws false positives. |
| bin/expr | [[#yatrc|1]] | gcooper | complete | Ported from %%NetBSD%%. |
| bin/sh | [[#yatrc|1]] | gcooper | complete | Ported from tools/regression. Need to get jilles@ to stop committing files to tools/regression :P. |
| bin/sleep | [[#yatrc|1]] | gcooper | complete | Ported from %%NetBSD%%. |
| gnu/usr.bin/diff | [[#yatrc|1]] | gcooper | complete | Ported from %%NetBSD%%. |
| lib/libc | [[#yatrc|1]] | gcooper | in-progress | Ported from %%NetBSD%%. Needs greater coverage/porting. Throws false positives. |
| lib/libcrypt | [[#yatrc|1]] | gcooper | complete? |  |
| lib/librt | [[#yatrc|1]] | gcooper | in-progress | Ported from %%NetBSD%%. Needs greater coverage/porting. Throws false positives. |
| lib/libthr | [[#yatrc|1]] | gcooper | in-progress | Ported from %%NetBSD%%. Needs greater coverage/porting. Throws false positives. |
| lib/msun | [[#yatrc|1]] | gcooper | in-progress | Ported from %%NetBSD%%. Needs greater coverage/porting. Throws false positives. |
| sbin/ifconfig | [[#yatrc|1]] | gcooper | complete | Ported from %%NetBSD%%. Needs greater coverage/porting. Throws false positives. |
| usr.bin/apply | [[#yatrc|1]] | gcooper | complete | Ported from tools/regression. Research adding coverage from %%NetBSD%%. |
| usr.bin/basename | [[#yatrc|1]] | gcooper | complete | Ported from %%NetBSD%%. |
| usr.bin/calendar | [[#yatrc|1]] | gcooper | complete | Ported from tools/regression. |
| usr.bin/comm | [[#yatrc|1]] | gcooper | complete | Ported from tools/regression. |
| usr.bin/cut | [[#yatrc|1]] | gcooper | complete | Ported from tools/regression. |
| usr.bin/dirname | [[#yatrc|1]] | gcooper | complete | Ported from %%NetBSD%%. |
| usr.bin/file2c | [[#yatrc|1]] | gcooper | complete | Ported from tools/regression. |
| usr.bin/grep | [[#yatrc|1]] | gcooper | complete | Ported from %%NetBSD%%. |
| usr.bin/gzip | [[#yatrc|1]] | gcooper | complete | Ported from %%NetBSD%%. |
| usr.bin/join | [[#yatrc|1]] | gcooper | complete | Ported from tools/regression. |
| usr.bin/jot | [[#yatrc|1]] | gcooper | complete | Ported from tools/regression. |
| usr.bin/m4 | [[#yatrc|1]] | gcooper | complete | Ported from tools/regression. |
| usr.bin/ncal | [[#yatrc|1]] | gcooper | complete | Ported from tools/regression. |
| usr.bin/printf | [[#yatrc|1]] | gcooper | complete | Ported from tools/regression. |
| usr.bin/sed | [[#yatrc|1]] | gcooper | complete | Ported from tools/regression. |
| usr.bin/tr | [[#yatrc|1]] | gcooper | complete | Ported from tools/regression. |
| usr.bin/uudecode | [[#yatrc|1]] | gcooper | complete | Ported from tools/regression. |
| usr.bin/uuencode | [[#yatrc|1]] | gcooper | complete | Ported from tools/regression. |
| usr.bin/xargs | [[#yatrc|1]] | gcooper | complete | Ported from tools/regression. |
| usr.bin/yacc | [[#yatrc|1]] | gcooper | complete | Ported from tools/regression. |
| usr.sbin/nmtree | [[#yatrc|1]] | gcooper | complete | Ported from %%NetBSD%%. Adjusted slightly to match sha256sum string. |
| usr.sbin/tcpdump | [[#yatrc|1]] | gcooper | complete | Ported from %%NetBSD%%. Needs greater coverage/porting. Throws false positives. |
| tests/contrib/atf | [[#yatrc|1]] | gcooper | complete | Ported from %%NetBSD%%. All test issues have been pushed back via GoogleCodeIn page for the contrib/atf issues; the build issues and adjustments are all remaining local to %%FreeBSD%% for now.. |

===== Test::Harness Testcases =====

| Directory | Repository | Owner | Status | Notes |
| bin/mv | [[#yatrc|1]] | gcooper | in-progress | From tools/regression. Convert to ATF. |
| bin/pax | [[#yatrc|1]] | gcooper | in-progress | From tools/regression. Convert to ATF. |
| bin/pkill | [[#yatrc|1]] | gcooper | in-progress | From tools/regression. Convert to ATF. |
| bin/test | [[#yatrc|1]] | gcooper | in-progress | From tools/regression. Convert to ATF. |
| sbin/geom | [[#yatrc|1]] | gcooper | in-progress | From tools/regression. Global tests, mirror, and raid3 testcases have been ported over. Convert to ATF. |
| tests/sys/pjdfstest | [[#yatrc|1]] | gcooper | in-progress | From tools/regression. Need to shim in support for ATF. |
| usr.bin/lastcomm | [[#yatrc|1]] | gcooper | in-progress | From tools/regression. Convert to ATF. |
| usr.sbin/newsyslog | [[#yatrc|1]] | gcooper | in-progress | From tools/regression. Convert to ATF. Need to investigate failures. |
| usr.sbin/sa | [[#yatrc|1]] | gcooper | in-progress | From tools/regression. Convert to ATF. Need to investigate failures. |