• http://www.supermicro.com/products/accessories/addon/AOC-CTG-i2S.cfm
• 6 core, single socket sandy/ivybridge CPU

• hw.ix.tx_process_limit: 256 (512?)
• hw.ix.rx_process_limit: 256 (512?)
• hw.ix.rxd: 2048 (4096?) - In the past we've found 2048 gives the best perf, but with different cards
• hw.ix.txd: 2048 (4096?)
• net.inet.tcp.tso=0
• kern.sched.slice=12
• hw.ix.enable_aim: 1
• dev.ix.0.fc=3
• hw.ix.num_queues=6 (4/5?)
• kern.random.sys.harvest.ethernet=1
• kern.random.sys.harvest.point_to_point=1
• kern.random.sys.harvest.interrupt=1
• kern.ipc.maxsockbuf=524288 (16777216?) - This is tuned for multiple 1Gs, I don't see us hitting a limit, but we may need to increase in the future.

netperf can be used here to easily see throughput, and netstat -I ix0 -hw 1 will show us a running pps.

The processing limit, harvests, and TSO/LRO settings are ones that could ease CPU usage, though we are very light on inbound so TSO is likely more of a possibility. The rest are performance related.

• kern.random.sys.harvest.ethernet=0
• kern.random.sys.harvest.point_to_point=0
• kern.random.sys.harvest.interrupt=0

Specific tuning for obtaining the best PPS performance (smallest packet size).

Benchs done on quad cores Intel Xeon L5630 2.13GHz with hyper-threading disabled (IBM System x3550 M3 ) with an Intel Ethernet Controller 10-Gigabit X540-AT2:

ix0@pci0:21:0:0:        class=0x020000 card=0x00018086 chip=0x15288086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller 10-Gigabit X540-AT2'
    class      = network
    subclass   = ethernet
    bar   [10] = type Prefetchable Memory, range 64, base 0xfbc00000, size 2097152, enabled
    bar   [20] = type Prefetchable Memory, range 64, base 0xfbe04000, size 16384, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks 
    cap 11[70] = MSI-X supports 64 messages, enabled
                 Table in map 0x20[0x0], PBA in map 0x20[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR link x8(x8)
                 speed 5.0(5.0) ASPM disabled(L0s/L1)
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1 a0369fffff1e2814
    ecap 000e[150] = ARI 1
    ecap 0010[160] = SRIOV 1
    ecap 000d[1d0] = ACS 1

Disabling rx_process_limit and tx_process_limit give the best performance (value in pps):

x rx|tx_process_limit=256(default)
+ rx|tx_process_limit=512
* rx|tx_process_limit=-1(nolimit)
+--------------------------------------------------------------------------------+
|*   ++ * x+                x                          *    * *   *             *|
||______M__A_________|                                                           |
|  |__A___|                                                                      |
|                                                      |______M__A________|      |
+--------------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5       1656772       1786725       1690620     1704710.4     48677.407
+   5       1657620       1703418       1679787     1681153.8     17459.489
No difference proven at 95.0% confidence
*   5       1918159       2036665       1950208       1963257     44988.621
Difference at 95.0% confidence
        258547 +/- 68356.2
        15.1666% +/- 4.00984%
        (Student's t, pooled s = 46869.3)

Lot's of tuning guide indicate to disable kern.random.sys.harvest.ethernet, kern.random.sys.harvest.point_to_point and kern.random.sys.harvest.interrupt.

But what about the REAL impact on a router (value in pps):

x harvest DISABLED
+ harvest ENABLED (default)
+--------------------------------------------------------------------------------+
|+                   x          x    x        x+        +   +   +               x|
|                    |_______________M_____A______________________|              |
|                   |_________________________A_________M______________|         |
+--------------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5       1918159       2036665       1950208       1963257     44988.621
+   5       1878893       2005333       1988952     1967850.8     51378.188
No difference proven at 95.0% confidence