[[https://hackmd.io/KYJhGMA4CMFYBMC0BDAbARgMyICwAZ1IV151FSRJUBOScWCHIA==?both]]

====== Test all the things! http://goo.gl/Xcocrp ======

Last Working Group: [[DevSummit/201706/Testing]]

===== Current Status =====

  * CI system:
    * [[Jenkins/TODO]]
    * DTrace test pipeline: https://ci.freebsd.org/job/FreeBSD-head-amd64-dtrace_test/

===== What to test =====

  * user space utilities
    * [[SummerOfCode2017/SmokeTestingOfBaseUtilities]]
  * kernel features
    * in ''%%src/tests/sys%%'', many skipped due to test environment (ex: no zfs)
  * performances
    * network
    * filesystem
  * fuzzing
  * non-x86 architectures
    * use qemu user-mode to test cross platform
      * slow, some tests are racy
    * there seems a arm-cloud provider willing to sponsor
  * Tracing tools
  * Possibly finding a way to test regressions while deploying security mitigations?
  * How others do?
    * %%NetBSD%%
      * http://releng.netbsd.org/test-results.html
    * %%OpenBSD%%
      * http://bluhm.genua.de
        * has test for sys/
        * has performance tests
        * https://github.com/bluhm/regress-all
    * Microsoft, %%FreeBSD%% on Azure
      * https://github.com/clovertrail/rocket/tree/master/shell/performance

===== How to test =====

  * Automated analysis of fuzzing results?
  * Ideas to generate an analysis data structure (Crash Graph ideas)
  * QEMU fuzzing?
  * real hardware testing (arm/mips/... boards)
  * unit testing of kernel functions/modules/units (requires framework)?
  * user-space versions of kernel code (examples: SCTP library, TCP library, etc.)?
  * Create "culture of testing"?
    * Should reviewers ask for tests for big features just like they ask for man pages?

===== Quality Assurance of FreeBSD =====
  * criteria of "good to ship"
    * no regression
    * feature
      * experimental feature should be marked explicitly

===== TODO list =====

  * go through cases in test suite
    * need list of "skipped", "know to fail" cases
      * can jenkins list them?
  * integrate more test suites
    * DTrace
      * needs ''%%WITH_DTRACE_TESTS=yes%%'' in src.conf
      * can be enabled after fixing all bugs (in DTrace or test cases)
    * Capsicum
      * https://github.com/google/capsicum-test
    * pf
      * https://lists.freebsd.org/pipermail/freebsd-hackers/2017-July/051261.html
    * TCP
      * Using Packet Drill tests from Michael Tuexen (gnn@)
        * ''%%Test suite%%'' is available here https://github.com/freebsd-net/tcp-testsuite
    * ZFS Test suite (brd@)

    * Have we converted all things under ''%%tools/regression%%''?
  * setup performance testing cluster
    * location? NYI
    * hardware requirements? (brd@ / gnn@)
  * generalize freebsd-ci codes for downstream adoption
    * a package to setup everything?
  * Some form of automated testing for security mitigation deployment (W^X, ASLR, PAN, ...) (brooks@, Domagoj)
  * Talk to Backtrace (gnn@)