Differences

This shows you the differences between two versions of the page.

--- wifi:txnotes [2012/12/09 04:00] – AdrianChadd
+++ wifi:txnotes [2018/04/05 23:31] (current) – Put this in a more useful place in the wiki hierarchy. Renamed from WifiTxNotes MateuszPiotrowski
@@ Line 1: / Line 1: @@
-====== net80211 TX path notes ======
+====== 802.11 driver/stack TX path notes ======
-The net80211 TX path is quirky. Here's some notes about it.
+The 802.11 TX path is quirky. Here's some notes about it.
 ===== TX path =====
 There's a few TX path entry points:
@@ Line 71: / Line 72: @@
 ===== The problem with taskqueues =====
+I've tried the TX taskqueue thing with ath(4), out of curiousity. (The real solution involves doing it with net80211 too, and either having a separate taskqueue for ath(4), or having the net80211 TX taskqueue call the driver direct and have the driver TX _only_ ever called from that context.)
+The initial problem is "dramatic TX throughput reduction."
+After tracing things a bit, I discovered the cause: when TX isn't direct dispatch, there is significant delays between TXing a frame and scheduling/running the taskqueue. Yes, even if it's a separate taskqueue (rather than a task in the same taskqueue as the RX task.)
+The normal case:
+  * you RX a frame
+  * it's an ACK and it causes a TX to occur;
+  * the network layer calls TX, and that pushes a frame up through the stack to be transmitted;
+  * it will call if_transmit / if_start, and that'll cause ath_start() to run and start transmitting;
+  * .. if there's more RX frames that occur, then they do occur and further TX frames are buffered if the TX queue is filled;
+  * TX completion gets scheduled;
+  * RX frame tasklet finishes and it yields to the TX completion thread;
+  * that completes frames, which calls ath_start() and/or the TX software queue task again to transmit more frames.
+The shared RX/TX/TX completion taskqueue case:
+  * You RX a frame
+  * it's an ACK and it causes a TX to occur;
+  * a frame is pushed onto the queue and the TX task is woken up;
+  * .. but it can't run until the RX taskqueue has completed;
+  * So more frames are received and processed; the TX task stays there waiting to be run;
+  * RX completes, the TX task wakes up and bursts out frames;
+  * TX completion tasklet runs and completes frames;
+  * etc, etc.
+Now, there's added latency between the TX occuring and the TX tasklet running because they run in the same shared taskqueue context. That extra latency causes an noticable performance dip. Very noticable.
+The shared RX/TX taskqueue case:
+  * You RX a frame;
+  * it's an ACK and causes a TX to occur;
+  * a frame is pushed onto the queue and the TX task is woken up;
+  * .. since it's the same priority as the RX task (say), it won't run until the RX task finishes.
+  * if the TX task is higher priority then sure, it'll preempt the RX taskqueue and run until the TX task has completed.
+This is a similar problem to the shared TX/RX taskqueue. The higher priority means that the RX tasklet (and TX completion tasklet) will get preempted.
+However, it's not a magic bullet: Specifically, the TX taskqueue will get woken up on every packet and if it can't do anything (eg the hardware TXQ is filled) it'll just software queue the frame and go to sleep. This is a lot of scheduler thrashing.
+Now, my ideal solution could be:
+  * The driver TX path would decide which node and TID queue the frame needs to be pushed onto;
+  * The driver TX path would just grab that lock, queue the frame and unlock;
+  * Then if the underlying hardware queue wasn't busy, it would schedule the TX taskqueue, otherwise it would not;
+  * The TX taskqueue would then be utterly responsible for the encapsulation, encryption stuff, etc, which it would do serialised.
+This is still suboptimal:
+  * The TX taskqueue would need to know which software queues need to be handled when it ran;
+  * There's still a race window where the TX taskqueue is scheduled by various driver threads because they don't (yet) see a busy hardware queue.
+The last point could be addressed by tracking how deep the hardware queue was about to be, rather than how deep the hardware queue _is_. That's just playing with fire though (if you get all of that wrong, you may screw up the queue depth figure and never handle things.)
+.. And this is all just for the ath(4) TX serialisation. This doesn't at all address the net80211 TX serialisation, which includes many of the same issues here.