Tuning FreeBSD for routing

This wiki page resume all information from BSDRP website and Tuning FreeBSD for routing and firewalling, AsiaBSDCon 2018 paper).

FreeBSD 13.0-CURRENT r354862 on a Xeon E5 2697A 16 cores (32 threads) with a 40G Chelsio T580 and a Mellanox ConnectX-4 is used here for the bench results.

The two main functions of a router are:

• Forwarding packets between its interfaces;
• Maintaining routing table using some routing protocols.

This page focuses only on optimizing the forwarding rate: Maintaining the routing table belongs to the userland daemons.

The only metric measured here will be the packet forwarding speed using packets-per-second (pps) unit.

RFC 2544 , Benchmarking Methodology for Network Interconnect Devices is a well-known reference to correctly bench routers.

But here we will not follow all recommendations given by this RFC for a simplest and faster methodology.

Here are some main divergences:

• Multiple frame size: In this paper, only the worst case matters, which is using the smallest Ethernet frame size. In this document one frame = one packet and unit fps=pps.
• Throughput is defined as the maximum frame rate supported by the DUT (device under test) without any drop: In this document the throughput is the outgoing forwarded frame rate when receiving at the maximum line rate.
• Bidirectional traffic: To simplify methodology, the bench labs described here generates only unidirectional traffic.

The first reference to know is the maximum Ethernet line rate (implying smallest frame size, like UDP packet with 1 byte of payload):

• Gigabit: 1.48 Mfps (frame-per-second)
• 10 Gigabit: 14.8 Mfps
• 40 Gigabit: 59.2 Mfps

With these first values and the fact that Ethernet is a full-duplex media, able to receive and transmit at the same time,

this means a line-rate router must be able to forward at:

• 3 Mpps = Gigabit bidirectional line-rate router
• 30 Mpps = 10 Gigabit bidirectional line-rate router
• 118 Mpps = 40 Gigabit bidirectional line-rate router

In real use cases there is no need of these line-rate routers because Internet traffic is not comprise of only small size packets but a mix of multiple sizes.

This packet size distribution evolves with time but there is a fixed-in-time reference, called Simple Internet Mix (IMIX), which uses this distribution:

• 1 large (1500 Bytes) packet: 37%
• 4 medium (576 Bytes) packets: 56%
• 7 small (40 Bytes) packets: 7%

Using Simple IMIX distribution it’s now possible to convert the reference packets-per-second to a more common value which is the bandwidth in bits per second (bps).

bps at the IP layer=PPS * ( 7 * 40 + 4 * 576 + 1500 ) / 12 * 8

Or the bandwidth at the Ethernet layer (need to add 14 Bytes for Ethernet headers), as seen by switch counters:

bps at the Ethernet layer = PPS * ( 7 * 54 + 4 * 590 + 1514 ) / 12 * 8

For real life use cases, the interesting question is now:

“Using a simple IMIX distribution size, what is the corresponding throughput for filling link capacity?”

These are the values that will be used for a new definition of a Full-duplex IMIX link-speed router and the minimum objectives to reach are:

• 700 Kpps = Gigabit IMIX router
• 7 Mpps = 10 Gigabit IMIX router
• 28 Mpps = 40 Gigabit IMIX router

A simple benchmarking lab can be set up with only 2 servers like here:

+---------+     +---------+
|         |-->--|         |
| pkt-gen |     |  DUT    |
|         |--<--|         |
+---------+     +---------+

• The first server with dual port Network Interface Card (NIC) is used as a packet generator and receiver (using net/pkt-gen ).
• The second server is the Device Under Test (DUT) running FreeBSD that will be tuned.

The purpose is to measure throughput (number of packets per second) forwarded by the DUT under the worst case: Receiving only smallest packet size at media line rate on one interface and forward to the packet receiver using its other interface.

The throughput is measured at the packet receiver side: Using a switch, with advanced monitoring counters for each port, can be useful to double cross-check its counters versus pkt-gen and Ethernet drivers counters

When netmap pkt-gen runs as a packet receiver it will NEVER generate a frame:

Then during your bench test, the generator will generate line-rate of traffic to the packet-receiver… but the MAC address of the receiver will age out on the switch table, and the switch will broadcast ALL traffic from the generator to ALL ports belonging to this VLAN

This is why it's always wise to statically configure MAC address of packet receiver on a switch before this kind of bench.

If you want to check the drivers statistics against the switch stats, you should disable ALL advanced feature on the switch's ports used for the bench too, like:

• spanning-tree
• CDP/LLDP
• DTP (Cisco)
• Keep-alive

Current NIC chipset & drivers behaviour:

1. NIC’s drivers create one queue per direction (transmit and receive) and per core detected with a maximum number of queues which is drivers dependant: 16 receiving (RX) queues for mlx4en, 8 RX queues for cxgbe and ixgbe as examples.
2. NIC’s chipsets use a Toeplitz hash to balance received packets across each RX queues: All 4 tuples of the packets (source IP, destination IP, source port and destination port) are used.

To being able to load-balance IP flows between cores, IP traffic must include multiple flows for being hashed: Using tunnelling features like IPSec, GRE or PPPoE prevents this distribution

The first step is to disable ICMP redirect forwarding to enable tryforward.

echo net.inet.ip.redirect=0 >> /etc/sysctl.conf
echo net.inet6.ip6.redirect=0 >> /etc/sysctl.conf
service sysctl restart

This first tuning impact is massive:

x net.inet.ip.redirect=1 (default): Number of inet4 packets-per-second forwarded
+ net.inet.ip.redirect=0: Number of inet4 pps forwarded
+--------------------------------------------------------------------------+
|x                                                                        +|
|xx                                                                      ++|
|xx                                                                      ++|
|MA                                                                        |
|                                                                        |A|
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5     2199966.5       2309062       2210097     2230250.6     45711.484
+   5       8211578     8259515.5       8244041     8235045.2      20946.73
Difference at 95.0% confidence
        6.00479e+06 +/- 51854.8
        269.243% +/- 7.86461%
        (Student's t, pooled s = 35554.9)

From 2.2Mpps with redirect enabled, it jumps to 8.2Mpps with redirect disabled.

Some drivers are not using all available cores or threads by default:

• cxgbe: “The default is 8 or the number of CPU cores in the system, whichever is less.”
• ixgbe seems to use same ase cxgbe
• mlx4en uses same as number of threads

As an example with default Chelsio settings on a 32 cpus (16 cores * 2HT) the drivers is using a maximum of 8 RX queues and 16 TX queues only:

[root@pxetest2]~# sysctl kern.smp.cores
kern.smp.cores: 16
[root@pxetest2]~# sysctl hw.ncpu
hw.ncpu: 32
[root@pxetest2]~# sysctl hw.cxgbe.nrxq
hw.cxgbe.nrxq: 8
[root@pxetest2]~# sysctl hw.cxgbe.ntxq
hw.cxgbe.ntxq: 16

Increasing the number of RX queue to match number of core or thread need to be done into /boot/loader.conf:

echo hw.cxgbe.nrxq=32 >> /boot/loader.conf
echo hw.cxgbe.ntxq=32 >> /boot/loader.conf

With Mellanox the variable to use is dev.mce.X.conf.channels=32 and with Intel dev.X.Y.iflib.override_nrxds=32

And the benefit will be huge:

x 8 RX queues (default): Number of inet4 packets-per-second forwarded
+ 16 RX queues (=ncores): Number of inet4 pps forwarded
* 32 RX queues (=nthreads): Number of inet4 pps forwarded
+--------------------------------------------------------------------------+
|x                 +                                                      *|
|xx               ++                                                     **|
|xx               ++                                                     **|
|A|                                                                        |
|                 |A                                                       |
|                                                                        AM|
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5       8160234       8261012       8230285     8221687.8       42278.4
+   5      10787172      10913225      10820346      10830279     49477.458
Difference at 95.0% confidence
        2.60859e+06 +/- 67115.9
        31.7282% +/- 0.934431%
        (Student's t, pooled s = 46018.9)
*   5      19034716      19176514      19107114      19097753     56388.197
Difference at 95.0% confidence
        1.08761e+07 +/- 72681.8
        132.285% +/- 1.42045%
        (Student's t, pooled s = 49835.2)

From 8.1Mpps with default value to 19Mpps using 32 RX queues (=all threads)

Another very interesting feature added into 13-head: allow interrupts on hyperthreaded cores.

echo machdep.hyperthreading_intr_allowed=1 >> /boot/loader.conf

On a 16cores/32thread server with a NIC configured with 32 RX queues, the benefit are very noticeable:

x machdep.hyperthreading_intr_allowed=0 (default): Number of inet4 packets-per-second forwarded
+ machdep.hyperthreading_intr_allowed=1: Number of pps
+--------------------------------------------------------------------------+
| x                                                                       +|
| x                                                                       +|
|xx                                                                     +++|
||A                                                                        |
|                                                                        AM|
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5      19028337      19126428      19099721      19090535     37168.444
+   5      24396932      24508797      24494912      24476810     45506.933
Difference at 95.0% confidence
        5.38628e+06 +/- 60594.5
        28.2144% +/- 0.355956%
        (Student's t, pooled s = 41547.4)

With this feature, from 19Mpps to 24Mpps forwarded.

By default, the Chelsio reserve hardware ressource for accelerated hardware features (RDMA, ISCSI, FCOE, TOE) that are useless on a router.

Disabling these features give more ressource to the chip:

cat <<EOF >>/boot/loader.conf
hw.cxgbe.toecaps_allowed="0"
hw.cxgbe.rdmacaps_allowed="0"
hw.cxgbe.iscsicaps_allowed="0"
hw.cxgbe.fcoecaps_allowed="0"
EOF

The benefit can be huge:

x hw.cxgbe.toecaps|rdmacaps|iscsicaps|fcoecaps_allowed=1 (default): Number of inet4 packets-per-second forwarded
+ hw.cxgbe.toecaps|rdmacaps|iscsicaps|fcoecaps_allowed=0: Number of inet4 pps
+--------------------------------------------------------------------------+
|                                                                         +|
|x                                                                        +|
|x                                                                        +|
|x                                                                        +|
|xx                                                                       +|
|A                                                                         |
|                                                                         A|
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5      12747220      12830462      12779478      12785165     31206.467
+   5      24485838      24535582      24491950      24500818     20931.042
Difference at 95.0% confidence
        1.17157e+07 +/- 38751.1
        91.6348% +/- 0.51107%
        (Student's t, pooled s = 26570.2)

From 12.7Mpps to 24.5Mpps (+91%).

Using Ethernet flow-control is a very bad idea on a router (and on a server): If your NIC is overloaded, it can ask an Ethernet “pause” to the direct peer (the switch). But this kind of mechanism already exist in TCP, and TCP have a better management than this very-basy Ethernet Flow-control. More information about this on When Flow Control is not a Good Thing

Disabling this feature is drivers dependent:

• Chelsio: Need to add hw.cxgbe.pause_settings="0" into /boot/loader.conf
• Mellanox: Need to add -mediaopt rxpause,txpause to the ifconfig_XX line of /etc/rc.conf
• Intel: Need to add dev.[ix|igb|em].X.fc=0 into /etc/sysctl.conf

There is not real impact of disabling feature on PPS:

x hw.cxgbe.pause_settings=0 (default): Number of inet4 packets-per-second forwarded
+ hw.cxgbe.pause_settings=1: Number of inet4 pps forwarded
+--------------------------------------------------------------------------+
|x                           +  x   +  +    + x         +                 x|
|         |_____________________M____A_________________________|           |
|                              |_______M_A_________|                       |
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5      24440254      24536843      24481016      24487790      35015.44
+   5      24477223      24513195      24490633      24493013     13385.716
No difference proven at 95.0% confidence

All modern NIC support LRO and TSO features that needs to be disabled on a router:

• By waiting to store multiple packets at the NIC level before to hand them up to the stack: This add latency, and because all packets need to be sending out again, the stack have to split in different packets again before to hand them down to the NIC. Intel drivers readme include this note “The result of not disabling LRO when combined with ip forwarding or bridging can be low throughput or even a kernel panic.”
• This break the End-to-end principle

To disable these features, you need to add -tso4 -tso6 -lro -vlanhwtso at the end of the the ifconfig_xxx line into /etc/rc.conf.

There is no real impact of disabling these feature on PPS:

x enabled (default): Number of inet4 packets-per-second forwarded
+ disabled (-tso4 -tso6 -lro -vlanhwtso): Number of inet4 pps forwarded
+--------------------------------------------------------------------------+
|x     x x       x                                                + +   +++|
|  |_____MA______|                                                         |
|                                                                  |__A_M_||
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5      24474825      24514420      24494558      24497564      16705.05
+   5      24636711      24656634      24651352      24648128     8776.9884
Difference at 95.0% confidence
        150564 +/- 19460.6
        0.614609% +/- 0.0798219%
        (Student's t, pooled s = 13343.4)