====== Pf Testing ====== * Student: [[/PanagiotesMousikides|PanagiotesMousikides]] (paggas@) * Mentor: [[/KristofProvost|KristofProvost]] (kp@) ===== Project description ===== The aim of the project is to create a framework for testing the network packet filter pf in %%FreeBSD%%. Testing would cover both rule parsing (in pfctl, the parser and loader, user space) and actual packet processing (netpfil/pf, kernel space). The aim is to also put a system in place for automatically and regularly running these tests, and integrate it with appropriate build and release tools. It shall also be easy to extend the framework and add new test cases. Such a tool exists already for %%OpenBSD%%, and part of the effort would be in investigating the possibility of porting that to %%FreeBSD%% as much as possible, to share the effort and make use of existing tests. Such a framework would be good for %%FreeBSD%% since there currently is no such testing system in place. That would help with regression testing among others, and generally increase the productivity of the %%FreeBSD%% project by quickly notifying developers of new errors in the codebase. ===== Approach to solving the problem ===== An initial way to do rule parsing will be to create a testable version of ​pfctl​ , one that reports the commands to be sent to the kernel instead of actually sending them. Then, different rulesets will be created and the testable version of ​pfctl​ will be run on them. An initial way to test the packet processing will be to run the test target as a guest in a virtual machine (using ​bhyve​ or similar) with multiple network interfaces, and load different rulesets with ​pfctl​, after it has itself been tested for correct parsing and loading of rulesets. The possibility of loading rulesets via a separate testing method will also be investigated. Other necessary configuration files, such as network configuration, will also be loaded. The host will then generate different types of traffic for the guest, or create more guests that act as network traffic producers and consumers, and monitor the guest behaviour and output. Different kinds of behaviour can be recorded, such as correctness of packet processing, and performance and stability of the test target. Moreover, further testing can be done with cases of multiple interacting test targets, or targets running different network services and generating their own network traffic. The test cases will ideally be in the form of network traffic-generating shell scripts or small compiled programs. The possibility of creating or adopting a special scripting language for network testing will also be investigated. The test system will be designed to make it easy to load different targets with different versions, rulesets and network topologies. ===== Deliverables ===== ==== Testable version of ​pfctl ==== Investigate testability of pfctl​, perhaps using the ​-n​ flag. Investigate how %%OpenBSD%% does it, and port or adapt as necessary. ==== Parser tests for ​pfctl​ using testable version ==== Create parser tests and run them using the testable version of pfctl​. Investigate what types of tests are important and in demand. Borrow test cases from %%OpenBSD%% when appropriate. Investigate other possibilities for testing, e.g. by looking at other similar projects. ==== Test setup allowing network tests (using​ bhyve​ or similar) ==== To test the actual packet processing, the main approach will be not to make any changes in the kernel code, but to create an appropriate setup (in a virtual machine using bhyve​, or similar) that easily allows testing. Different ways to make this as simple and well-performing as possible will be investigated. ==== Network tests (using test setup) ==== Create network tests, borrowing from %%OpenBSD%% and other projects when appropriate. Investigate what types of tests are important and in demand. Investigate different ways of loading and logging in the test setup, which will probably be a host and guest virtual machine setup. ==== Integration with build and release system ==== Investigate how to best integrate the framework with existing %%FreeBSD%% build and release tools. Create a target for make​ or hooks for the version control system as appropriate. ==== System to run tests regularly ==== Install a system that runs tests regularly and reports results to the developers. That can be a virtual machine installed and running somewhere, or a module for an existing system. Investigate ways to make it as productive as possible, possibly by communicating with interested groups and mailing lists of the %%FreeBSD%% project. ==== Integration with Jenkins/CI/​ATF ==== Jenkins is a continuous integration (CI) tool used by %%FreeBSD%%. Investigate if it would be appropriate to integrate the framework there, and communicate with the Jenkins people of %%FreeBSD%% about it. Integrate with ​ ATF​ as appropriate. ==== Documentation ==== Produce documentation to make it easy for other to use and extend the framework with new tests. Documentation will be continuously delivered with a refactoring and cleanup phase at the end. ===== Milestones ===== ==== May 15-30 ==== * Preparatory work * Community bonding * Create wiki page * Setup accounts and working environment * Read %%FreeBSD%% coding styleguide * Get familiar with tools to be used * pf * bhyve * %%OpenBSD%% pf testing framework * Kyua, ATF * Jenkins, CI ==== May 30-June 20 ==== * **May 30: Start of coding** * Deliver pfctl testable version and parser tests * Documentation ==== June 20-July 28 ==== * **June 30: Phase 1 evaluation deadline** * Deliver network test setup and network tests * Documentation * **July 28: Phase 2 evaluation deadline** ==== July 28-August 21 ==== * Deliver running test system * Integrate with build system * Integrate with Jenkins, CI * Documentation * **August 21: End of coding (soft)** * **August 21: Start of final week** ==== August 21-29 ==== * Coding review, refactoring, cleanup * Documentation refactoring and cleanup * **August 29: End of coding (hard)** * **August 29: Final evaluation deadline** * **August 29: End of final week** ==== August 29-September 15 ==== * **September 5: Final evaluation deadline for mentor** * Followup with mentor and community * Update ideas list in wiki ===== Test Plan ===== Since this is a testing tool, a separate test plan would be redundant. The test plan for the test tool is to run the test against what we’re testing :) ===== Results ===== (Current as of 20170829.) We have managed to deliver almost all of our deliverables. The pfctl tests have been included in %%FreeBSD%%, while the kernel pf tests are on the way. The deliverable that was skipped was **System to run tests regularly**, as focus went on integrating the tests into the regular test suite. Such a system can still be built, and there is room for optimization for such an installation. ===== The Code ===== The code is on [[/GitHub|GitHub]] at https://github.com/paggas/freebsd, under the 'paggas' branch, forked off of %%FreeBSD%%'s [[/GitHub|GitHub]] mirror at https://github.com/freebsd/freebsd. Most, if not all, of the work will be under tests/. For easier review, we have gathered the work in two patches: * https://reviews.freebsd.org/D11322 - for the pfctl tests * https://reviews.freebsd.org/D11401 - for the kernel pf tests Patch D11322 is already included in %%FreeBSD%%, while D11401 is on the way (as of 20170829). ===== Useful links ===== This project is part of [[/SummerOfCode2017|SummerOfCode2017]].