This is an old revision of the document!
Test setup:
- The test consisted in running the netperf tcp stream test through the wrap board, to see how much traffic it can sustain/forward/nat with different software setup.
- The network was setup to make all the requests and replies pass through the wrap board, while netserver and netperf where running on the desktop.
- Socket sizes: -s = -S = 57344.
- Desktop: PentiumD 920 (2×2.8Ghz), 2Gb RAM - FreeBSD-HEAD/amd64.
- Wrap: AMD Geode (486 clone 233Mhz), 128MB RAM, 10/100 sis nics (DP83816 PCI) [http://www.pcengines.ch/wrap.htm details here].
- World and kernel for the wrap board were compiled with -O1 (except in the gcc optimization tests).
- A tarball containing test results, gnuplot files and plots is available [http://people.freebsd.org/~piso/wrap-exp/wrap-exps.tgz here].
- Below the images are my comments.
http://people.freebsd.org/~piso/wrap-exp/wrap-exp.png
Legend:
- 4.x: FreeBSD 4.11/i386
- 7.x: FreeBSD HEAD/i386 (30/12/2006)
- fwd: net.inet.ip.forwarding=1
- ffwd: net.inet.ip.fastforwarding=1
- poll-1x: device polling activated for sis0
- poll-2x: device polling activated for sis0 and sis1
- nod: net.isr.direct=0 (7.x default: net.isr.direct=1)
- natd: ipfw divert socket + natd
- ipfwnat: ipfw nat
- O0: -O0 (no optimization)
- O1: -O1
- O2: -O2 -fno-strict-aliasing
- Os: -Os (implies -fno-strict-aliasing)
http://people.freebsd.org/~piso/wrap-exp/4vs7.png
- In the 4.x-ffwd case, looks like the 100Mbit nics are the bottleneck.
http://people.freebsd.org/~piso/wrap-exp/4vs7-forwarding-polling.png
- Dunno what's the cause for the drop in 4.x-fwd-poll2x.
- In all the tests without polling enabled, the wrap board was unresponsive (livelock?!?!).
- There's something definitely wrong with polling in 7.x (maybe since 5.x?).
http://people.freebsd.org/~piso/wrap-exp/4vs7-fastforwarding-polling.png
- All the 4.x plots are overlayed on top of each other.
- Turning on polling with fast forward enabled (contrary to what happened in the previous experiment), didn't give any performance boost: perhaps with Gb nics we would see an improvement?
http://people.freebsd.org/~piso/wrap-exp/4vs7-polling-complete.png
- Merge of the 2 previous experiments: quite messy, but useful to directly compare different combinations.
http://people.freebsd.org/~piso/wrap-exp/7-directvsnodirect.png
- As per rwatson request, i did this test (more from him? :).
http://people.freebsd.org/~piso/wrap-exp/4vs7-nat.png
- In 4.x the firewall (ipfw) and fast forwarding are not compatible, while in 7.x (in the fast forward path) we traverse the pfil hooks so i turned it on in that case.
- Turning on polling didn't improve the 4.x cases (while it improved simple forwarding - see above).
- In any case, ipfw nat looks faster than 4.x.
http://people.freebsd.org/~piso/wrap-exp/7-opt-complete.png
- Being the curious person i am, i wanted to see how much impact different gcc optimization levels can have on packets forwarding.