Test all the things! http://goo.gl/Xcocrp

Last Working Group: Testing

• CI system:
  • TODO
  • DTrace test pipeline: https://ci.freebsd.org/job/FreeBSD-head-amd64-dtrace_test/

• user space utilities
  • SmokeTestingOfBaseUtilities
• kernel features
  • in src/tests/sys, many skipped due to test environment (ex: no zfs)
• performances
  • network
  • filesystem
• fuzzing
• non-x86 architectures
  • use qemu user-mode to test cross platform
    • slow, some tests are racy
  • there seems a arm-cloud provider willing to sponsor
• Tracing tools
• Possibly finding a way to test regressions while deploying security mitigations?
• How others do?
  • NetBSD
    • http://releng.netbsd.org/test-results.html
  • OpenBSD
    • http://bluhm.genua.de
      • has test for sys/
      • has performance tests
      • https://github.com/bluhm/regress-all
  • Microsoft, FreeBSD on Azure
    • https://github.com/clovertrail/rocket/tree/master/shell/performance

• Automated analysis of fuzzing results?
• Ideas to generate an analysis data structure (Crash Graph ideas)
• QEMU fuzzing?
• real hardware testing (arm/mips/… boards)
• unit testing of kernel functions/modules/units (requires framework)?
• user-space versions of kernel code (examples: SCTP library, TCP library, etc.)?
• Create “culture of testing”?
  • Should reviewers ask for tests for big features just like they ask for man pages?

• criteria of “good to ship”
  • no regression
  • feature
    • experimental feature should be marked explicitly

• go through cases in test suite
  • need list of “skipped”, “know to fail” cases
    • can jenkins list them?
• integrate more test suites
  • DTrace
    • needs WITH_DTRACE_TESTS=yes in src.conf
    • can be enabled after fixing all bugs (in DTrace or test cases)
  • Capsicum
    • https://github.com/google/capsicum-test
  • pf
    • https://lists.freebsd.org/pipermail/freebsd-hackers/2017-July/051261.html
  • TCP
    • Using Packet Drill tests from Michael Tuexen (gnn@)
      • Test suite is available here https://github.com/freebsd-net/tcp-testsuite
  • ZFS Test suite (brd@)

• Have we converted all things under tools/regression?
• setup performance testing cluster
• location? NYI
• hardware requirements? (brd@ / gnn@)
• generalize freebsd-ci codes for downstream adoption
• a package to setup everything?
• Some form of automated testing for security mitigation deployment (W^X, ASLR, PAN, …) (brooks@, Domagoj)
• Talk to Backtrace (gnn@)