Differences

This shows you the differences between two versions of the page.

--- devsummit:201309:networking [2013/09/26 13:15] – LawrenceStewart
+++ devsummit:201309:networking [2021/04/25 10:00] (current) – Move to DevSummit namespace. JethroNederhof
@@ Line 12: / Line 12: @@
       * MPTCP (Lawrence)
       * Deferring mbuf drop on ACK (Scott)
-      * UI for controlling TCP dynamics (Lawrence, Andre)
+      * UI for controlling TCP/stack dynamics (Lawrence, Andre)
       * Reworking SACK + integrating with reassembly (Andre)
       * Pacing (Lawrence)
@@ Line 37: / Line 37: @@
     * MBUFS
-      * Ref counting (Gleb, Adrian) \\  Problem: uint32_t's coming from single zone non contiguous in memory with the mbuf \\  Possible solutions: \\
+      * Ref counting (Gleb, Adrian) \\  Problem: uint32_t's coming from single zone non contiguous in memory with the mbuf \\
       * Batching
-    * Multi-(process|thread) accept (Gleb) Problem: \\  Possible solutions: \\
+    * Multi-(process|thread) accept (Gleb) Problem: Wakeup causes all threads to race and only 1 wins \\
-    *
+    * Light weight ref counter based on count(9) (Gleb)
-    * Light weight ref counter based on count(9) (Gleb) Problem: \\
-  * Testing
+  * Testing/development
-    * NSC userpsace stack (Lawrence) [[http://caia.swin.edu.au/urp/incast/tools.html]]
+    * NSC userspace stack (Lawrence) [[http://caia.swin.edu.au/urp/incast/tools.html]]
     * Testing @ Netflix (Lawrence)
     * Testing @ Orange (Olivier)
@@ Line 53: / Line 52: @@
 //(Add a list or attach slides detailing the achieved results here.)//
+===== Stack/driver interface - Andre Oppermann =====
+To come:
+Formal documentation of stack driver boundary
+Split of ifnet structure into stack owned and driver owned (evaluate Juniper proposal)
+Adjust all drivers to new world order
+-- request for feedback to mailinglist messages
+Version shims or will all vendors be expected to switch?
+-- Shouldn't be too hard to change
+-- Clear documentation
+Will open projects branch for this.
+Technical monitor is Ed Maaste from FF
+Can change not be one large commit? #ifdef conditionalized, and convert most important drivers sequentially.
+One aim is to separate L2 and L3 but that's outside scope here
+Changing the way drivers announce capabilities to stack -- idea is to have an area of struct declared by driver
+like in setsockopt() -- list of items that the stack will walk.
+It's like newbus.  So why not newbus? "Newbus is not just buses, it's also not new (phk)"
+(Has been reinvented twice, not well since).  To be investigated.
+There can be additional failure possibilities if functions exported as pointers in list have to allocate memory otherwise will be the same as now.
+Vendor capabilities, route to integrate advanced features of hardware.
+TCP offload? Is ifnet the right level of abstraction for modern network hardware?
+Just in time compilation to improve performance on cache-deep processes?
+Not a problem now, but it will be tomorrow.  There will be cards that offload the entire network stack.
+Automagic export of C code at compile time a la
+IFQ interface: pain point
+different types of workload need different types of parallelism.  No one true method
+like bind one queue to one core.
+Need flexibility to configure via ifconfig or some other method.
+ALTQ not working.
+Get rid of IFQ and associated softqueue -- most HW has hardware support.
+The stack will call down completely lockless, (no ifnet lock or driver lock).  Drive can
+implement optimal approach
+Pretty much everything uses iftransmit now, except altq.  %%QoS%%? No explicit API for timing.
+Intermediary queue discipline?
+Stack iftransmit function pointer replaced by altq and will call down into driver iftransmit.
+Multiple layers of queue disipline? nterface only supports one layer.
+Work is to prepare proposal, documentation and run performance tests so we can make informed decision
+Can we pass in other structures than -- eg writing disk blocks directly into network packet?
+This is another construction site. Whois looking at busdma?
+Once documentation etc. looked at, can update drivers.
+sometime in the next 4 months.
+Common API for common functions for drivers to use, but not stack.
+Driver can benefit automatically from inprovements
+Vendor feedback: can we move bufferring up into the stack?  Control queue lengths and multilayer queue disciplines.
+Adaptive behaviour depending on how many packets waiting
+Copy-pasting in drivers is not a good thing.  Copy code should be centralised into the stack available for driver to use.
+Infrastructure almost in place to remove that (expected 10.1)
+Start thinking further into the future.  Driver consists of about 20% boiler plate.  Write condensed driver spec file and process
+into .[ch] files at compile time.
+===== Interface dangling pointers -- Gleb =====
+Implement some lightweight ref counting.  Need an efficient way to fetch current value gathered from all cpus.
+===== Routing performance =====
+Mutex contention kills performance on routing lookup.   Problems when egress interface disappears.  Cached rtentry leads to crashes
+Look at killing the need for rtentry locking. Ref counting as proposed by Gleb.
+===== Testing / Development =====
+Developers should communicate with companies able to test with different workloads.
+Orange (Olivier) and Yandex (Alexander) have interest in routing/forwarding performance. Orange have Ixia test hardware.
+Netflix (Lawrence, Adrian, Scott) have access to TCP-heavy production workload.
+Netflix looking to host focused mini devsummit events in Los Gatos, CA on a semi-regular basis.
+===== Mbuf refcounting =====
+- cache contention from tightly packed slab
+Gleb to continue testing prototype patch and post to list
+- cpu stall due to external memory read
+Evaluate further after Gleb's patch (which may address this issue)
+===== multi-(process|thread) accept =====
+"Thundering herd" problem.
+- SO_REUSEPORT molested as in Linux/DFLYBSD
+Pros: Currently used in Linux/DFLY, good for small apps
+Cons: listen queue migration, annoying to support for larger apps, less flexibility
+- Go our own way with e.g. SO_SHAREDQUEUE
+Seems to be a leaning towards this direction but a clear proposal should be articularted to a mailing list and a pro/con done later
+===== Routing subsystem =====
+- Multiple cache misses per radix lookup
+- Radix is too generic. Could have a super compact v4 only trie along side
+- Luigi/Marko's netmap + DXR work needs to be looked into
+- Alexander needs to solve API fro RIB->FIB callbacks to deal with MPLS which can be used to support per-protocol specific lookup structures
+===== Ownership =====
+OFED - get Netapp, Isilon, iX, RDMA NIC vendors to talk
+===== Miscellaneous =====
+Andre is carving ipsec out into a pfil based kmod
+Andre is working on the final bits of the TCP-AO
+TSO doesn't work with NAT because the copy out buffer is too small and truncates