This is an old revision of the document!
Tuning FreeBSD for routing
(Adapted to a generic FreeBSD from BSDRP website and Tuning FreeBSD for routing and firewalling, AsiaBSDCon 2018 paper)
How to correctly bench a router
On a router we measure the throughput (maximum packet-per-second with smallest packet size) and not the bandwidth (Gb/s).
A good full router benchmark should do all RFC2544 (Benchmarking Methodology for Network Interconnect Devices) and RFC3222 (Terminology for Forwarding Information Base (FIB) based Router Performance) tests. But this kind of full-tests are very time consuming: Starting by a more simple bench using a wire-speed (or line-rate) packet-generator using smallest packet size is a good start.
A wire-speed packet generator should be able to generate the maximum number of smallest packet per second on the tested media, this mean about 14.88Mpps on a TenGigabit media, 59.52Mpps on a 40Gigabit media. Benchmarking smallest packet size permit to measure the “worse” case you can encounter.
But using Packet-per-second (pps) as unique number isn't common for standard users that are waiting value in Gb/s.
We can estimated the bandwidth from the throughput using the standard Internet Mix packet size distribution.
The formula became: Bandwidth = PPS * ( 7*(40+14) + 4*(576+14) + (1500+14) )/12*8
Bench lab setup
netmap pkt-gen
Netmap pkt-gen with an optional checksum & IPv6 patch is a great tool for benching router. It's a hight throughput packet generator using range of src/dst IP addresses or/and src/dst UDP ports mandatory for using the multi-queue feature of 10G NIC.
Switch configuration
When netmap pkt-gen runs as a packet receiver it will NEVER generate a frame:
Then during your bench test, the generator will generate line-rate of traffic to the packet-receiver… but the MAC address of the receiver will age out on the switch table, and the switch will broadcast ALL traffic from the generator to ALL ports belonging to this VLAN 
This is why it's always wise to statically configure MAC address of packet receiver on a switch before this kind of bench.
If you want to check the drivers statistics against the switch stats, you should disable ALL advanced feature on the switch's ports used for the bench too, like:
- spanning-tree
- CDP/LLDP
- DTP (Cisco)
- Keep-alive
Tuning feature
Re-enabling fastforwarding
The first step is to disable ICMP redirect forwarding to enable tryforward.
echo net.inet.ip.redirect=0 >> /etc/sysctl.conf echo net.inet6.ip6.redirect=0 >> /etc/sysctl.conf service sysctl restart
This first tuning impact is massive:
x net.inet.ip.redirect=1 (default): Number of inet4 packets-per-second forwarded
+ net.inet.ip.redirect=0: Number of inet4 pps forwarded
+--------------------------------------------------------------------------+
|x +|
|xx ++|
|xx ++|
|MA |
| |A|
+--------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 5 2199966.5 2309062 2210097 2230250.6 45711.484
+ 5 8211578 8259515.5 8244041 8235045.2 20946.73
Difference at 95.0% confidence
6.00479e+06 +/- 51854.8
269.243% +/- 7.86461%
(Student's t, pooled s = 35554.9)
From 2.2Mpps with redirect enabled, it jumps to 8.2Mpps with redirect disabled.
Checking optimum usage of all cores by the NIC
Some drivers are not using all available core by default, here an example with default Chelsio settings: On a 32 cpus (16 cores * 2HT) the drivers is using a maximum of 8 RX queues and 16 TX queues:
[root@pxetest2]~# sysctl kern.smp.cores kern.smp.cores: 16 [root@pxetest2]~# sysctl hw.ncpu hw.ncpu: 32 [root@pxetest2]~# sysctl hw.cxgbe.nrxq hw.cxgbe.nrxq: 8 [root@pxetest2]~# sysctl hw.cxgbe.ntxq hw.cxgbe.ntxq: 16
Increasing the number of RX queue to match number of core or thread need to be done into /boot/loader.conf:
echo hw.cxgbe.nrxq=32 >> /boot/loader.conf echo hw.cxgbe.ntxq=32 >> /boot/loader.conf
And the benefit will be huge:
x 8 RX queues (default): Number of inet4 packets-per-second forwarded
+ 16 RX queues (=ncores): Number of inet4 pps forwarded
* 32 RX queues (=nthreads): Number of inet4 pps forwarded
+--------------------------------------------------------------------------+
|x + *|
|xx ++ **|
|xx ++ **|
|A| |
| |A |
| AM|
+--------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 5 8160234 8261012 8230285 8221687.8 42278.4
+ 5 10787172 10913225 10820346 10830279 49477.458
Difference at 95.0% confidence
2.60859e+06 +/- 67115.9
31.7282% +/- 0.934431%
(Student's t, pooled s = 46018.9)
* 5 19034716 19176514 19107114 19097753 56388.197
Difference at 95.0% confidence
1.08761e+07 +/- 72681.8
132.285% +/- 1.42045%
(Student's t, pooled s = 49835.2)
From 8.1Mpps with default value to 19Mpps using 32 RX queues (=all threads)
Allow interrupts on HTT logical CPUs
Another very interesting feature added into 13-head: allow interrupts on hyperthreaded cores.
echo machdep.hyperthreading_intr_allowed=1 >> /boot/loader.conf
On a 16cores/32thread server with a NIC configured with 32 RX queues, the benefit are very noticeable:
x machdep.hyperthreading_intr_allowed=0 (default): Number of inet4 packets-per-second forwarded
+ machdep.hyperthreading_intr_allowed=1: Number of pps
+--------------------------------------------------------------------------+
| x +|
| x +|
|xx +++|
||A |
| AM|
+--------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 5 19028337 19126428 19099721 19090535 37168.444
+ 5 24396932 24508797 24494912 24476810 45506.933
Difference at 95.0% confidence
5.38628e+06 +/- 60594.5
28.2144% +/- 0.355956%
(Student's t, pooled s = 41547.4)
With this feature, from 19Mpps to 24Mpps forwarded.
Disabling Ethernet Flow-control
Using Ethernet flow-control is a very bad idea on a router (and on a server): If your NIC is overloaded, it can ask an Ethernet “pause” to the direct peer (the switch). But this kind of mechanism already exist in TCP, and TCP have a better management than this very-basy Ethernet Flow-control. More information about this on When Flow Control is not a Good Thing
Disabling LRO and TSO
All modern NIC support LRO and TSO features that needs to be disabled on a router:
- By waiting to store multiple packets at the NIC level before to hand them up to the stack: This add latency, and because all packets need to be sending out again, the stack have to split in different packets again before to hand them down to the NIC. Intel drivers readme include this note “The result of not disabling LRO when combined with ip forwarding or bridging can be low throughput or even a kernel panic.”
- This break the End-to-end principle
There is no real impact of disabling these feature on PPS.
x tso.lro.enabled
+ tso.lro.disabled
+--------------------------------------------------------------------------+
| + + x+ * x+ x x|
| |___________________________A_M_________________________| |
||____________M___A________________| |
+--------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 5 1724046 1860817 1798145 1793343 61865.164
+ 5 1702496 1798998 1725396 1734863.2 38178.905
No difference proven at 95.0% confidence
Reducing entropy harvest
Lot's of tuning guide indicate to disable these for getting best forwarding performance:
- kern.random.sys.harvest.ethernet
- kern.random.sys.harvest.point_to_point
- kern.random.sys.harvest.interrupt.
But here is the impact on a router (value in pps):
x harvest DISABLED
+ harvest ENABLED (default)
+--------------------------------------------------------------------------------+
|+ x x x x+ + + + x|
| |_______________M_____A______________________| |
| |_________________________A_________M______________| |
+--------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 5 1918159 2036665 1950208 1963257 44988.621
+ 5 1878893 2005333 1988952 1967850.8 51378.188
No difference proven at 95.0% confidence
⇒ No difference.