NetFPGA SUME reference NIC device driver

This is an old revision of the document!

Student: Denis Salopek (denis.sale@gmail.com)
Mentors: Marko Zec, Bjoern A. Zeeb (zec@fer.hr, bz@freebsd.org)

Project description

NetFPGA is an open-source project aimed to foster networking hardware prototyping. Their latest and de-facto standard development board is NetFPGA SUME with a Virtex-7 FPGA and 4 SFP+ (10 Gbps) ports. The board’s open-source repository includes 14 designs: 'reference' projects to be used as a first step to building customized NICs, switches, routers, etc., as well as community member contributed projects. The NetFPGA software ecosystem, including the device drivers, is unfortunately Linux-centric, with no support for FreeBSD.

This project will enable using NetFPGA SUME as a 4×10 Gbps NIC on hosts running FreeBSD. There are already FreeBSD drivers for older (now obsolete) versions of NetFPGA 10G (if_nf10bmac(4)) and NetFPGA 1G (https://github.com/wkoszek/freebsd_netfpga), both incompatible with the SUME board but could provide some valuable insight.

By researching iflib(9) framework and by consulting with my mentors, I have come to the conclusion that iflib is not the right tool for this task. NetFPGA SUME is a network device with 4 physical 10G interfaces connected to the host with only one PCI endpoint. The existing SUME Linux driver allocates four netdevs, essentially creating four different 10G 'virtual' interfaces in the OS. It then multiplexes packets based on the header meta-data generated by hardware. It seems that iflib cannot provide neccessary functionalities needed for NetFPGA SUME hardware without additional hacking, so I will completely ditch the idea and create the device driver without iflib.

Using the slow RIFFA DMA engine with multiple interrupts for one packet over one DMA transaction yields low performance for the NetFPGA SUME. Benchmarks are done using pkt-gen (UDP sending / receiving 60 B or 1500 B packets), netperf (sending / receiving TCP with default options) and iperf (sending / receiving TCP - using option -l to change buffer size, otherwise getting inconsistent results) on Intel(R) Core(TM) i7-4771 CPU @ 3.50GHz (Ubuntu 18.04.4 with kernel 4.15.0-106-generic, FreeBSD 11.3-STABLE and FreeBSD 12-1-p7). 'TX' and 'RX' benchmarks are done by connecting one NetFPGA SUME port to a different computer (FreeBSD 12.1) to an Intel X520 NIC (ix) and 'loopback' is performed by connecting two ports of the same NetFPGA SUME, using one for TX and other for RX.

LINUX	Test	TX only	RX only	loopback
	iperf	189 Mb/s (-l 24)	2.52 Gb/s	-
	netperf	77.34 Mb/s	2.51 Gb/s	-
	pkt-gen - 60 B	314 Kpps (150 Mb/s)	242 Kpps (116 Mb/s)	200 Kpps
	pkt-gen - 1500 B	215 Kpps (2.6 Gb/s)	234 Kpps (2.8 Gb/s)	180 Kpps
FREEBSD 11.3
	iperf	2.15 Gb/s	606 Mb/s (-l 162)	-
	netperf	2.15 Gb/s	397 Mb/s	-
	pkt-gen - 60 B	303 Kpps (145 Mb/s)	246 Kpps (118 Mb/s)	152 Kpps
	pkt-gen - 1500 B	207 Kpps (2.5 Gb/s)	235 Kpps (2.9 Gb/s)	138 Kpps
FREEBSD 11.3
rx_queue branch	iperf	2.12 Gb/s	448 Mb/s (-l 120)	-
	netperf	2.12 Gb/s	-	-
	pkt-gen - 60 B	301 Kpps (144 Mb/s)	238 Kpps (114 Mb/s)	151 Kpps
	pkt-gen - 1500 B	208 Kpps (2.5 Gb/s)	237 Kpps (2.9 Gb/s)	125 Kpps
FREEBSD 12.1-p7
	iperf	2.2 Gb/s	610 Mb/s (-l 163)	-
	netperf	2.2 Gb/s	201 Mb/s	-
	pkt-gen - 60 B	279 Kpps (134 Mb/s)	220 Kpps (111 Mb/s)	164 Kpps
	pkt-gen - 1500 B	260 Kpps (3.1 Gb/s)	215 Kpps (2.7 Gb/s)	133 Kpps
FREEBSD 12.1-p7
rx_queue branch	iperf	2.22 Gb/s	607 Mb/s	-
	netperf	2.22 Gb/s	521 Mb/s	-
	pkt-gen - 60 B	278 Kpps (133 Mb/s)	238 Kpps (114 Mb/s)	125 Kpps
	pkt-gen - 1500 B	261 Kpps (3.1 Gb/s)	238 Kpps (2.9 Gb/s)	150 Kpps

Writing a device driver on top of the iflib would inherently provide support for netmap(4), but by writing a standalone driver, adding Netmap support will have to be done differently. Netmap support will enable high-speed traffic exchange between (custom) NetFPGA designs and userspace network applications, paving the way for experimentation with new hardware / software network processing paradigms.

NetFPGA SUME driver skeleton with basic PIO register communication
NetFPGA SUME reference NIC device driver for FreeBSD with basic NIC functionalities (sending/receiving packets via DMA)
netmap(4) integration with the developed device driver

June 1st: Start of coding
- Start writing a generic driver for basic communication between the OS and NetFPGA, mostly to learn more about the process of kernel driver programming and inner workings of host-NetFPGA communication (device interrupts, PCIe PIO and DMA). This will serve as a base for the next step. Improve the driver communication with FPGA registers, enable reading/writing to/from specific memory locations, gain control over DMA transfers. NetFPGA register communication is enabled.
July 3rd: Mid-term Evaluations #1
- Enable sending/receiving network packets to/from NetFPGA/OS. Implement interrupt handlers. Start integrating Netmap specifics to the driver.
July 31st: Mid-term Evaluations #2
- Full Netmap integration. Testing with different Netmap applications (for sending/receiving traffic). This will almost certainly reveal bugs in the driver so fixing them will inevitably take away some time. Driver optimizations for better traffic throughput. Last minute minor fixes and finishing touches. Documentation finishing up.
August 31st: Final Evaluations