Differences

This shows you the differences between two versions of the page.

--- summerofcode2018projects:extensionstonetmap [2018/07/19 13:51] – StefanoDuo
+++ summerofcode2018projects:extensionstonetmap [2018/10/27 11:20] (current) – StefanoDuo
@@ Line 22: / Line 22: @@
 ==== Per-ring fixed offsets ====
-Given the fixed nature of this type of offsets, the approach I have been followed is to let netmap internally handle offset math, thus giving directly the "real" buffer starting address, and then let the application "go backwards" if it needs the offset space.
+Given the fixed nature of this type of offsets, the approach I have followed is to let netmap internally handle offset math, thus giving directly the "real" buffer starting address, and then let the application "go backwards" if it needs the offset space.
 From the userland perspective a buffer is obtained through the ''%%NETMAP_BUF()%%'' macro (defined in netmap_user.h), which receives a ''%%struct netmap_ring%%'' and the index of the requested buffer. Therefore a new field has been created inside the ''%%struct netmap_ring%%'' to store the offset for that ring. To maintain the kernel-userland ABI the ''%%sem%%'' field, which is basically a placeholder for new fields, has been resized. After those changes ''%%NETMAP_BUF()%%'' has been modified to handle the offset internally, thus applications only needs to be recompiled to be able to process offsets passively.
-From the kernel perspective a buffer logical (and physical) address is obtained through ''%%NMB()%%'' and ''%%PNMB()%%'' (defined in netmap_kern.h). They receive a ''%%struct netmap_adapter%%'' and a few other things. They will need to be modified to receive the ''%%struct netmap_kring%%'' which contains the buffer instead of the adapter (it can be retrieved from the kring). A new field to store the offset value need to be added to the ''%%struct netmap_kring%%'' as well.
+From the kernel perspective a buffer logical (and physical) address is obtained through ''%%NMB()%%'' and ''%%PNMB()%%'' (defined in netmap_kern.h). They receive a ''%%struct netmap_adapter%%'' and a few other things. They have been modified to receive the ''%%struct netmap_kring%%'' which contains the buffer instead of the adapter (it can be retrieved from the kring). A new field to store the offset value has been added to the ''%%struct netmap_kring%%'' as well. Moreover ''%%NETMAP_BUF_SIZE()%%'' now accepts a new parameter: ''%%uint64_t offset%%'', which is subtracted to the buffer real size. This way the caller directly receives the size of the space left after applying the offset. By changing the function signature we're sure that all the code which requires a buffer size has been "updated", otherwise netmap would not compile.
-Finally a new option must be created to let the user specify if he or she wants to open an interface with offsets.
+Finally a new option has been added to let the user specify if he or she wants to open an interface with offsets.
 ==== Per-buffer variable offsets ====
-It follows the same scheme of per-adapter fixed buffer offsets, but now the various functions need a way to retrieve the ''%%struct netmap_slot%%'' of the buffer, probably by passing it as an additional argument.
+It follows the same idea of per-ring fixed buffer offsets and its stored inside the ''%%struct netmap_slot%%''. Because the ownership over the slots is well defined, this type of offset can be changed any time the kernel (or the userspace applications) owns the slot.
+Kernel side ''%%NMB()%%'' and ''%%PNMB()%%'' already receives the ''%%struct netmap_slot%%'' so we only need to the the offset field into account. Buffer overflow protection is accomplished through the offset parameter of ''%%NETMAP_BUF_SIZE()%%'', just like per-ring fixed offsets.
+From the userspace perspective this type of offset needs to be handled explicitly by the application. It cannot be handled transparently by the ''%%NETMAP_BUF()%%'' macro, because it doesn't receive a ''%%struct netmap_slot%%'' as one of its parameters, and the macro cannot be changed otherwise backwards compatibility cannot be obtained. Therefore problems could arise during zero-copy (or even while receiving slots from the kernel, if netmap starts using this type of offset itself). An option has been added to let the application specify that it won't handle per-slot variable offsets. Accordingly to this option the memory allocator gets flagged, so that every buffer from that allocator are used either by application aware of per-slot buffer offsets or by aware application.
 ==== Testing framework ====
-functional.c (from the netmap repository) is a CLI tool already capable of sending and receiving UDP packets with specified length and a few of other possible options. The general idea is to extend it with basic functionalities that can be used as building blocks for building tests, e.g. send/expect packet with a specific payload and/or a specific MAC address, etc.
+functional.c (from the netmap repository) is a CLI tool already capable of sending and receiving UDP packets with specified length and a few of other possible options. It has been extended it with basic functionalities that can be used as building blocks for writing tests (e.g. send/expect packet with a specific payload and/or a specific MAC address etc.).
-For more complex behaviour multiple calls to functional.c might be needed, and between those calls an interface may need to stay up, thus functional.c should have another process acting as a daemon which holds netmap interfaces file descriptors and lends them on request.
+For more complex behavior multiple calls to functional.c are needed, and between those calls an interface may need to stay up, thus functional.c can start and stop a daemon. This daemon acts like a file descriptor server and can hold netmap interfaces file descriptors and lend them on request.
+Furthermore i wrote a bash library which eases and shortens the time required to write a test script. It provides wrappers around the most common operations executing during a test, reducing the amount of boilerplate code needed. The wrappers provided to create VALE persistent ports and VETH interfaces and attach/detach interfaces to VALE bridges automatically handle the cleanup after the test ends (implemented through traps).
+===== Notes =====
+==== Per-ring fixed offset behavior ====
+This type of offset is usually applied to the rings specified by the register request but there are some exceptions.
+  * netmap pipes: the offset is applied to every ring of both ends of the pipe. This is needed because the exchange buffers through zero-copy.
+  * netmap monitors: the offset cannot be specified by the application, but it is retrieved from the monitored rings. This is needed because having a different offset from the monitored rings might cause a packet to require more slots inside the monitor ring.
 ===== Deliverables =====
-  * Implement an external kernel module which extends VALE switches to support virtual lans (IEEE 802.1Q standard) and a CLI tool to easily manage vlan configurations.
+  * Implement an external kernel module which extends VALE switches to support virtual lans (IEEE 802.1Q standard) and a CLI tool to easily manage vlan configurations. ✔
-  * Implement the testing framework as a CLI tool with a set of basic functionalities which will be used to write a suite of tests for netmap.
+  * Implement the testing framework as a CLI tool with a set of basic functionalities which will be used to write a suite of tests for netmap. ✔
-  * Extend netmap to support per-adapter buffer offsets and make use of the inside the vlan external module; then add support for per-buffer offsets.
+  * Extend netmap to support per-ring buffer offsets and make use of the inside the vlan external module; then add support for per-buffer offsets. ✔
 ===== Milestones =====
   * May 14th: Start of coding
-    * Write a kernel module which implements an hardcoded vlan configuration.
+    * Write a kernel module which implements an hardcoded vlan configuration. ✔
   * May 21st: Week 2
-    * Allow configurations to be managed through system calls instead of being hardcoded.
+    * Allow configurations to be managed through system calls instead of being hardcoded. ✔
   * May 28th: Week 3
-    * Write a CLI tool to manage configurations.
+    * Write a CLI tool to manage configurations. ✔
   * June 4th: Week 4
-    * Bug fixing and documentation.
+    * Bug fixing and documentation. ✔
   * June 11th - June 15th: Mid-term Evaluations #1
   * June 18th: Week 6
-    * Extend functional.c to manage file descriptors of interfaces.
+    * Extend functional.c to manage file descriptors of interfaces. ✔
   * June 25th: Week 7
-    * Extend functional.c with functionalities needed to compose tests.
+    * Extend functional.c with functionalities needed to compose tests. ✔
   * July 2nd: Week 8
-    * Write a suite of tests through the new framework.
+    * Write a suite of tests through the new framework. ✔
   * July 9th - July 13th: Mid-term Evaluations #2
   * July 9th: Week 9
-    * Extend netmap to support per-adapter buffer offsets.
+    * Extend netmap to support per-ring buffer offsets. ✔
   * July 16th: Week 10
-    * Extend the VALE vlan kernel module to make use of buffer offsets.
+    * Extend the VALE vlan kernel module to make use of buffer offsets. ✔
   * July 23rd: Week 11
-    * Extend netmap to support per-buffer offsets.
+    * Extend netmap to support per-buffer offsets. ✔
   * August 6th: End of coding (soft)
-    * Clean up code and documentation.
+    * Clean up code and documentation. ✔
   * August 14th: End of coding (hard)
@@ Line 77: / Line 92: @@
 https://svnweb.freebsd.org/socsvn/soc2018/sduo/
-===== Notes =====
+===== Future Contributions =====
+Update netmap specific NIC drivers to be offset aware, as they currently do not support any type of offset.
-Nothing at the moment.
-===== Useful links =====
+===== Useful Links =====
   * [[http://info.iet.unipi.it/~luigi/netmap/|netmap]].
   * [[http://info.iet.unipi.it/~luigi/vale/|VALE]].