Router [FreeBSD WIKI]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Tuning FreeBSD for routing ======

This wiki page merge and update data from [[https://bsdrp.net|BSDRP website]] and [[https://people.freebsd.org/~olivier/talks/2018_AsiaBSDCon_Tuning_FreeBSD_for_routing_and_firewalling-Paper.pdf|Tuning FreeBSD for routing and firewalling, AsiaBSDCon 2018 paper]].

%%FreeBSD%% 13.0-CURRENT on a Xeon E5 2697A 16 cores (32 threads) with a 40G Chelsio T580 and a 100G Mellanox %%ConnectX%%-4 is used here for the bench results.

===== Basic concepts =====

==== How to correctly bench a router ====

The two main functions of a router are:
  * Forwarding packets between its interfaces;
  * Maintaining routing table using some routing protocols.

This page focuses only on optimizing the forwarding rate: Maintaining the routing table belongs to the userland daemons.

The only metric measured here will be the packet forwarding speed using packets-per-second (pps) unit.

==== Differences with RFC 2544 ====

[[https://tools.ietf.org/html/rfc2544|RFC 2544 , Benchmarking Methodology for Network Interconnect Devices]] is a well-known reference to correctly bench routers.

But here we will not follow all recommendations given by this RFC for a simplest and faster methodology.

Here are some main divergences:
  * Multiple frame size: In this paper, only the worst case matters, which is using the smallest Ethernet frame size. In this document one frame = one packet and unit fps=pps.
  * Throughput is defined as the maximum frame rate supported by the DUT (device under test) without any drop: In this document the throughput is the outgoing forwarded frame rate when receiving at the maximum line rate.
  * Bidirectional traffic: To simplify methodology, the bench labs described here generates only unidirectional traffic. 

==== Ethernet line rate references ====

The first reference to know is the maximum [[https://tools.cisco.com/security/center/resources/network_performance_metrics|Ethernet line rate]] (implying smallest frame size, like UDP packet with 1 byte of payload):
  * Gigabit: 1.48 Mfps (frame-per-second)
  * 10 Gigabit: 14.8 Mfps
  * 40 Gigabit: 59.2 Mfps
From these values and the fact that Ethernet is a full-duplex (FDX) media (able to receive and transmit at the same time), this means a FDX line-rate router must be able to forward at:
  * 3 Mpps = Gigabit FDX line-rate router
  * 30 Mpps = 10 Gigabit FDX line-rate router
  * 118 Mpps = 40 Gigabit FDX line-rate router

==== Throughput to bandwidth ====
In real use cases there is no need of these line-rate routers because Internet traffic is not comprise of only small size packets but a mix of multiple sizes.

This packet size distribution evolves with time but there is a fixed-in-time reference, called [[https://en.wikipedia.org/wiki/Internet_Mix|Simple Internet Mix (IMIX)]] which uses this distribution:
  * 1 large (1500 Bytes) packet: 37%
  * 4 medium (576 Bytes) packets: 56%
  * 7 small (40 Bytes) packets: 7%
Using this Simple IMIX distribution it’s now possible to convert the packets-per-second to a more common value which is the bandwidth in bits per second (bps).

bps at the IP layer=PPS * ( 7 * 40 + 4 * 576 + 1500 ) / 12 * 8

Or the bandwidth at the Ethernet layer (need to add 14 Bytes for Ethernet headers), as seen by switch counters:

bps at the Ethernet layer = PPS * ( 7 * 54 + 4 * 590 + 1514 ) / 12 * 8

For real life use cases, the interesting ratio to use is the one when using a simple IMIX distribution size that fill the link capacity.

And here are the minimum PPS for a "FDX IMIX link-speed router":
  * 700 Kpps = Gigabit IMIX router
  * 7 Mpps = 10 Gigabit IMIX router
  * 28 Mpps = 40 Gigabit IMIX router

==== Setting a benchmarking lab ====

A very simple benchmarking lab can be set up with only 2 servers like here:

<code>
+---------+   +-------+
|         |->-|       |
| pkt-gen |   |  DUT  |
|         |-<-|       |
+---------+   +-------+
</code>

  * The first server with dual port Network Interface Card (NIC) is used as a packet generator and receiver (using [[https://github.com/luigirizzo/netmap|net/pkt-gen]] ).
  * The second server is the Device Under Test (DUT) running %%FreeBSD%% that will be tuned.
The purpose is to measure throughput (number of packets per second) forwarded by the DUT under the worst case: Receiving only smallest packet size at media line rate on one interface and forward to the
packet receiver using its other interface.

The throughput is measured at the packet receiver side: Using a switch, with advanced monitoring counters for each port, can be useful to double cross-check its counters versus pkt-gen and Ethernet drivers counters.

A more complex setup, like using 40Giga NIC and range of IP addresses with pkt-gen (consuming too much CPU for using one server), could use 3 nodes:

<code>
+---------+   +-------+   +--------+
|         |   |       |   |        |
| pkt-gen |->-|  DUT  |->-|pkt-gen |
|generator|   |       |   |receiver|
+---------+   +-------+   +--------+
</code>

=== Switch configuration ===

When netmap pkt-gen runs as a packet receiver it will NEVER generate a frame:

Then during your bench test, the generator will generate line-rate of traffic to the packet-receiver… but the MAC address of the receiver will age out on the switch table, and the switch will broadcast ALL traffic from the generator to ALL ports belonging to this VLAN ;-)

This is why it's always wise to statically configure MAC address of packet receiver on a switch before this kind of bench.

If you want to check the drivers statistics against the switch stats, you should disable ALL advanced feature on the switch's ports used for the bench too, like:
  * spanning-tree
  * CDP/LLDP
  * DTP (Cisco)
  * Keep-alive

==== Multi-flows to benefit from multi-queue RSS ====

Current NIC chipset & drivers behaviour:
  - NIC’s drivers create one queue per direction (transmit and receive) and per core detected with a maximum number of queues which is drivers dependant: 16 receiving (RX) queues for mlx4en, 8 RX queues for cxgbe and ixgbe as examples.
  - NIC’s chipsets use a Toeplitz hash to balance received packets across each RX queues: All 4 tuples of the packets (source IP, destination IP, source port and destination port) are used.

To being able to load-balance IP flows between cores, IP traffic must include multiple flows for being hashed: Using tunnelling features like IPSec, GRE or %%PPPoE%% prevents this distribution

===== Minimum hardware requirements =====

Before tuning, to be able to reach the minimum requirement of a 10Gb/s router, the CPU must have 8 cores minimum, Intel Xeon or equivalent class.

NIC manufacturers known to take care of their %%FreeBSD%% drivers: Chelsio, Intel and Mellanox.

===== Tuning =====

==== Re-enabling fastforwarding ====

//Should be useless since r367628 (Nov 12, 2020) that "..adds redirects to the fastforward path" that fixed it.//

The first step is to disable sending IP redirects to enable tryforward.

<code>
echo net.inet.ip.redirect=0 >> /etc/sysctl.conf
echo net.inet6.ip6.redirect=0 >> /etc/sysctl.conf
service sysctl restart
</code>

This first tuning impact is massive:

<code>
Number of inet4 packets-per-second forwarded:
x net.inet.ip.redirect=1 (default)
+ net.inet.ip.redirect=0
+--------------------------------------------------------------------------+
|x                                                                        +|
|xx                                                                      ++|
|xx                                                                      ++|
|MA                                                                        |
|                                                                        |A|
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5     2199966.5       2309062       2210097     2230250.6     45711.484
+   5       8211578     8259515.5       8244041     8235045.2      20946.73
Difference at 95.0% confidence
        6.00479e+06 +/- 51854.8
        269.243% +/- 7.86461%
        (Student's t, pooled s = 35554.9)
</code>

From 2.2Mpps with redirect enabled, it jumps to 8.2Mpps with redirect disabled.

==== Configure the NIC to use all cores/threads ====

Some drivers are not using all available cores or threads by default:
  * [[https://www.freebsd.org/cgi/man.cgi?query=cxgbe&apropos=0&sektion=0&manpath=FreeBSD+12.1-RELEASE+and+Ports&arch=default&format=html|cxgbe]]: "The default is 8 or the number of CPU cores in the system, whichever is less."
  * ixgbe seems to use same ase cxgbe
  * mlx4en|mlx5en use same as number of threads

As an example with default Chelsio settings on a 32 cpus (16 cores * 2HT) the drivers is using a maximum of 8 RX queues and 16 TX queues only:

<code>
[root@pxetest2]~# sysctl kern.smp.cores
kern.smp.cores: 16
[root@pxetest2]~# sysctl hw.ncpu
hw.ncpu: 32
[root@pxetest2]~# sysctl hw.cxgbe.nrxq
hw.cxgbe.nrxq: 8
[root@pxetest2]~# sysctl hw.cxgbe.ntxq
hw.cxgbe.ntxq: 16
</code>

Increasing the number of RX queue to match number of core or thread need to be done into /boot/loader.conf:

<code>
echo hw.cxgbe.nrxq=32 >> /boot/loader.conf
echo hw.cxgbe.ntxq=32 >> /boot/loader.conf
</code>

With Mellanox the variable to use is ''%%dev.mce.X.conf.channels=32%%'' and with Intel ''%%dev.X.Y.iflib.override_nrxds=32%%''

And the benefit will be huge:

<code>
Number of inet4 packets-per-second forwarded
x 8 RX queues (default)
+ 16 RX queues (= all cores on this setup)
* 32 RX queues (= all threads on this setup)
+--------------------------------------------------------------------------+
|x                 +                                                      *|
|xx               ++                                                     **|
|xx               ++                                                     **|
|A|                                                                        |
|                 |A                                                       |
|                                                                        AM|
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5       8160234       8261012       8230285     8221687.8       42278.4
+   5      10787172      10913225      10820346      10830279     49477.458
Difference at 95.0% confidence
        2.60859e+06 +/- 67115.9
        31.7282% +/- 0.934431%
        (Student's t, pooled s = 46018.9)
*   5      19034716      19176514      19107114      19097753     56388.197
Difference at 95.0% confidence
        1.08761e+07 +/- 72681.8
        132.285% +/- 1.42045%
        (Student's t, pooled s = 49835.2)
</code>

From 8.1Mpps with default value to 19Mpps using 32 RX queues (=all threads)

==== Allow interrupts on HTT logical CPUs ====

Another very interesting feature added into 13-head: [[https://svnweb.freebsd.org/base?view=revision&revision=354338|allow interrupts on hyperthreaded cores]].

<code>
echo machdep.hyperthreading_intr_allowed=1 >> /boot/loader.conf
</code>

On a Intel Xeon CPU E5-2697A v4 2.60GHz (16 cores / 32 threads) server with a Chelsio T580-LP-CR configured with 32 RX queues (incoming traffic) and a Mellanox %%ConnectX%%-4 (outgoing traffic), the benefit is noticeable:

<code>
Number of inet4 packets-per-second forwarded:
x machdep.hyperthreading_intr_allowed=0 (default) and hw.cxgbe.nXxq=32
+ machdep.hyperthreading_intr_allowed=1 and hw.cxgbe.nXxq=32
+--------------------------------------------------------------------------+
| x                                                                       +|
| x                                                                       +|
|xx                                                                     +++|
||A                                                                        |
|                                                                        AM|
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5      19028337      19126428      19099721      19090535     37168.444
+   5      24396932      24508797      24494912      24476810     45506.933
Difference at 95.0% confidence
        5.38628e+06 +/- 60594.5
        28.2144% +/- 0.355956%
        (Student's t, pooled s = 41547.4)
</code>

With this feature, from 19Mpps to 24Mpps forwarded.

Notice this setup reach such value, because on the DUT the generated flow wasn't using the same ASIC for RX and TX, avoiding to overload the ASIC: All trafic was incoming to the Chelsio and outgoing toward the Mellanox. In case of traffic entering and exiting the other Chelsio port connected to the same ASIC, the performance stuck at about 19Mpps.

On a different setup, an Xeon E5 2650L V2 1.70GHz (10 Cores/20 Threads) with a Chelsio_T540-CR, result are less impressive but still an improvement:

<code>
Number of inet4 packets-per-second forwarded:
x machdep.hyperthreading_intr_allowed=0 (default) and hw.cxgbe.nXxq=16
+ machdep.hyperthreading_intr_allowed=1 and hw.cxgbe.nXxq=16
+--------------------------------------------------------------------------+
|        x                                                              ++ |
|        xx                                                   x   +   x ++ |
||________M_____________________A______________________________|           |
|                                                                   |__AM_||
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5     7586270.5       8644626       7616992     7990242.7     537190.89
+   5       8583402       8701804       8689267     8669742.3     48818.605
Difference at 95.0% confidence
        679500 +/- 556274
        8.50412% +/- 7.54931%
        (Student's t, pooled s = 381417)
</code>

==== Chelsio drivers tuning ====

By default, the Chelsio reserve hardware ressource for accelerated hardware features (RDMA, ISCSI, FCOE, TOE) that are useless on a router.

Disabling these features give more ressource to the chip:

<code>
cat <<EOF >>/boot/loader.conf
hw.cxgbe.toecaps_allowed="0"
hw.cxgbe.rdmacaps_allowed="0"
hw.cxgbe.iscsicaps_allowed="0"
hw.cxgbe.fcoecaps_allowed="0"
EOF
</code>

The benefit can be huge:

<code>
Number of inet4 packets-per-second forwarded:
x hw.cxgbe.toecaps|rdmacaps|iscsicaps|fcoecaps_allowed=1 (default)
+ hw.cxgbe.toecaps|rdmacaps|iscsicaps|fcoecaps_allowed=0
+--------------------------------------------------------------------------+
|                                                                         +|
|x                                                                        +|
|x                                                                        +|
|x                                                                        +|
|xx                                                                       +|
|A                                                                         |
|                                                                         A|
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5      12747220      12830462      12779478      12785165     31206.467
+   5      24485838      24535582      24491950      24500818     20931.042
Difference at 95.0% confidence
        1.17157e+07 +/- 38751.1
        91.6348% +/- 0.51107%
        (Student's t, pooled s = 26570.2)
</code>

From 12.7Mpps to 24.5Mpps (+91%).

==== Disabling Ethernet Flow-control ====

Using Ethernet flow-control is not a good idea on a router (and on a server): If your NIC is overloaded it will ask an Ethernet "pause" to the direct peer (the switch), that will temporary stop emitting ALL flows toward the server.

But this kind of mechanism already exist in TCP, and TCP have a better management than this very-basic Ethernet Flow-control.

More information about this on [[http://www.smallnetbuilder.com/lanwan/lanwan-features/30212-when-flow-control-is-not-a-good-thing|When Flow Control is not a Good Thing]]

Disabling this feature is drivers dependent:
  * Chelsio: Need to add ''%%hw.cxgbe.pause_settings="0"%%'' into /boot/loader.conf
  * Mellanox: Need to add ''%%dev.mce.X.rx_pauseframe_control=0%%'' into /boot/loader.con or adding ''%%-mediaopt rxpause,txpause%%'' to the ifconfig_XX line of /etc/rc.conf
  * Intel: Need to add ''%%dev.[ix|igb|em].X.fc=0%%'' into /etc/sysctl.conf

And there is not real impact of disabling feature on forwarding performance:

<code>
Number of inet4 packets-per-second forwarded:
x hw.cxgbe.pause_settings=0 (default)
+ hw.cxgbe.pause_settings=1
+--------------------------------------------------------------------------+
|x                           +  x   +  +    + x         +                 x|
|         |_____________________M____A_________________________|           |
|                              |_______M_A_________|                       |
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5      24440254      24536843      24481016      24487790      35015.44
+   5      24477223      24513195      24490633      24493013     13385.716
No difference proven at 95.0% confidence
</code>

==== Disabling LRO and TSO ====

All modern NIC support [[http://en.wikipedia.org/wiki/Large_receive_offload|LRO]] and [[http://en.wikipedia.org/wiki/Large_segment_offload|TSO]] features that needs to be disabled on a router:
  * By waiting to store multiple packets at the NIC level before to hand them up to the stack: This add latency, and because all packets need to be sending out again, the stack have to split in different packets again before to hand them down to the NIC. Intel drivers readme include this note "The result of not disabling LRO when combined with ip forwarding or bridging can be low throughput or even a kernel panic."
  * This break the [[http://en.wikipedia.org/wiki/End-to-end_principle|End-to-end principle]]

To disable these features, you need to add ''%%-tso4 -tso6 -lro -vlanhwtso%%'' at the end of the ifconfig_xxx line into /etc/rc.conf.

There is no negative impact of disabling these features on forwarding performance, and on some hardware setup it can even brings some good improvement:

<code>
Number of inet4 packets-per-second forwarded
x enabled (default)
+ disabled (-tso4 -tso6 -lro -vlanhwtso)
+--------------------------------------------------------------------------+
|                                                                        ++|
|xxxxx                                                                  +++|
||_A_|                                                                     |
|                                                                        A||
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5      27463046      27724202      27568318      27599181     109278.63
+   5      32355798      32463268      32416758      32414376     47287.862
Difference at 95.0% confidence
        4.8152e+06 +/- 122795
        17.4469% +/- 0.511089%
        (Student's t, pooled s = 84196.1)
</code>