Pf Testing

• Student: PanagiotesMousikides (paggas@)
• Mentor: KristofProvost (kp@)

The aim of the project is to create a framework for testing the network packet filter pf in FreeBSD. Testing would cover both rule parsing (in pfctl, the parser and loader, user space) and actual packet processing (netpfil/pf, kernel space). The aim is to also put a system in place for automatically and regularly running these tests, and integrate it with appropriate build and release tools. It shall also be easy to extend the framework and add new test cases.

Such a tool exists already for OpenBSD, and part of the effort would be in investigating the possibility of porting that to FreeBSD as much as possible, to share the effort and make use of existing tests.

Such a framework would be good for FreeBSD since there currently is no such testing system in place. That would help with regression testing among others, and generally increase the productivity of the FreeBSD project by quickly notifying developers of new errors in the codebase.

An initial way to do rule parsing will be to create a testable version of ​pfctl​ , one that reports the commands to be sent to the kernel instead of actually sending them. Then, different rulesets will be created and the testable version of ​pfctl​ will be run on them.

An initial way to test the packet processing will be to run the test target as a guest in a virtual machine (using ​bhyve​ or similar) with multiple network interfaces, and load different rulesets with ​pfctl​, after it has itself been tested for correct parsing and loading of rulesets. The possibility of loading rulesets via a separate testing method will also be investigated. Other necessary configuration files, such as network configuration, will also be loaded. The host will then generate different types of traffic for the guest, or create more guests that act as network traffic producers and consumers, and monitor the guest behaviour and output. Different kinds of behaviour can be recorded, such as correctness of packet processing, and performance and stability of the test target.

Moreover, further testing can be done with cases of multiple interacting test targets, or targets running different network services and generating their own network traffic. The test cases will ideally be in the form of network traffic-generating shell scripts or small compiled programs. The possibility of creating or adopting a special scripting language for network testing will also be investigated. The test system will be designed to make it easy to load different targets with different versions, rulesets and network topologies.

Investigate testability of pfctl​, perhaps using the ​-n​ flag. Investigate how OpenBSD does it, and port or adapt as necessary.

Create parser tests and run them using the testable version of pfctl​. Investigate what types of tests are important and in demand. Borrow test cases from OpenBSD when appropriate. Investigate other possibilities for testing, e.g. by looking at other similar projects.

To test the actual packet processing, the main approach will be not to make any changes in the kernel code, but to create an appropriate setup (in a virtual machine using bhyve​, or similar) that easily allows testing. Different ways to make this as simple and well-performing as possible will be investigated.

Create network tests, borrowing from OpenBSD and other projects when appropriate. Investigate what types of tests are important and in demand. Investigate different ways of loading and logging in the test setup, which will probably be a host and guest virtual machine setup.

Investigate how to best integrate the framework with existing FreeBSD build and release tools. Create a target for make​ or hooks for the version control system as appropriate.

Install a system that runs tests regularly and reports results to the developers. That can be a virtual machine installed and running somewhere, or a module for an existing system. Investigate ways to make it as productive as possible, possibly by communicating with interested groups and mailing lists of the FreeBSD project.

Jenkins is a continuous integration (CI) tool used by FreeBSD. Investigate if it would be appropriate to integrate the framework there, and communicate with the Jenkins people of FreeBSD about it. Integrate with ​ ATF​ as appropriate.

Produce documentation to make it easy for other to use and extend the framework with new tests. Documentation will be continuously delivered with a refactoring and cleanup phase at the end.

• Preparatory work
• Community bonding
• Create wiki page
• Setup accounts and working environment
• Read FreeBSD coding styleguide
• Get familiar with tools to be used
  • pf
  • bhyve
  • OpenBSD pf testing framework
  • Kyua, ATF
  • Jenkins, CI

• May 30: Start of coding
• Deliver pfctl testable version and parser tests
• Documentation

• June 30: Phase 1 evaluation deadline
• Deliver network test setup and network tests
• Documentation
• July 28: Phase 2 evaluation deadline

• Deliver running test system
• Integrate with build system
• Integrate with Jenkins, CI
• Documentation
• August 21: End of coding (soft)
• August 21: Start of final week

• Coding review, refactoring, cleanup
• Documentation refactoring and cleanup
• August 29: End of coding (hard)
• August 29: Final evaluation deadline
• August 29: End of final week

• September 5: Final evaluation deadline for mentor
• Followup with mentor and community
• Update ideas list in wiki

Since this is a testing tool, a separate test plan would be redundant. The test plan for the test tool is to run the test against what we’re testing :)

(Current as of 20170829.)

We have managed to deliver almost all of our deliverables. The pfctl tests have been included in FreeBSD, while the kernel pf tests are on the way.

The deliverable that was skipped was System to run tests regularly, as focus went on integrating the tests into the regular test suite. Such a system can still be built, and there is room for optimization for such an installation.

The code is on GitHub at https://github.com/paggas/freebsd, under the 'paggas' branch, forked off of FreeBSD's GitHub mirror at https://github.com/freebsd/freebsd. Most, if not all, of the work will be under tests/.

For easier review, we have gathered the work in two patches:

• https://reviews.freebsd.org/D11322 - for the pfctl tests
• https://reviews.freebsd.org/D11401 - for the kernel pf tests

Patch D11322 is already included in FreeBSD, while D11401 is on the way (as of 20170829).

This project is part of SummerOfCode2017.