net80211 TX path notes

The net80211 TX path is quirky. Here's some notes about it.

There's a few TX path entry points:

• if_start / if_transmit - goes in via a vap, gets queued as appropriate, 802.11 state gets handled (eg sequence number assignment, padding, etc)
• .. then frames go via the driver input method;
• the driver then finishes driver-specific encapsulation and adds whatever crypto state is needed - this includes the CCMP IV.
• then the frame gets pushed onto the driver hardware or software queue as appropriate.

This is the dirty bit. It occurs via ieee80211_fragment(). This breaks the mbuf into a chain of mbufs, linked via m→m_nextpkt.

However the moment one calls if_start or if_transmit with this chain of mbuf fragments, the m_nextpkt pointer gets cleared and only the first part of the frame is transmitted. The rest of the fragments aren't transmitted; the non-first fragments themselves are silently leaked (as m_nextpkt gets cleared rather than freeing the mbuf.)

In the past, the driver would get the 802.3 frame and it would then do the net80211 encapsulation, fragmenting and encryption. So it could loop over a list of fragments and transmit those.

This changed as part of the work to enable VAP transmission, where frames go into the VAP first and encapsulation happens at this layer. This includes fast-frames handling and fragmentation.

This path is rather annoying - if it's a “raw” frame, it goes straight to the driver. Otherwise it goes via if_transmit() to get encapsulated.

There's also an ic_raw_xmit() function; it gets used directly by some management frame transmission routines.

Power save queue packets get pushed onto the queue here from the net80211 TX path. It gets drained (normally) by the receive path - either on a ps-poll or a data frame with the power management bit cleared.

This happens via ff_transmit(); which can be called by ieee80211_ff_check() and ff_flush().

This gets called by a variety of contexts - the TX path, the TX completion path in ath(4) calls the FF flush routine; and the ath(4) RX path also does this.

The big thing to realise here is that if_transmit() and if_start() do not have any kind of serialisation guarantees. Even if_start(), which uses the ifnet send queue, can and will have multiple threads calling the stack and driver start method at the same time. Overlapping _and_ preempting. Keep that in mind.

The first problem here is the lack of TX serialisation. Specifically, TX can occur from multiple thread contexts and can be preempted. This includes multiple TX sending threads (eg multiple iperf processes), the RX context (eg an RX of an ACK kick-starts the TX of the next frame), various timers/callouts (eg BAR retransmission, etc.)

There's also the raw versus non-raw TX path. Some raw frames (eg EAPOL frames, as part of WPA/WPA2 handling) have sequence numbers assigned to them and are encrypted. These need to be pushed into the TX path in the “right” order; any out of order handling here will cause the receiver to drop frames.

The next problem is the disconnect between net80211 TX handling and encryption. The encryption is done in the driver, not the net80211 TX path. Thus the frames pushed into the driver has to be handled in order by the driver, otherwise there may be a slight disconnect between the 802.11 sequence numbers and the CCMP IV sequence number.

The linux solution is “hold a lock in the TX code, so that frames get handled in-order from the point where it's grabbed by mac80211 and processed (sequence number, 802.11 state) all the way through the driver, including encryption handling.

The “better” solution is to serialise both the net80211 TX path and the driver TX path so there's a TX queue that frames get pushed into, then either:

• A single thread is allowed to continue transmitting, the others just queue and quit;
• There's a TX taskqueue that is woken up whenever TX needs to occur.

The side-issues:

• Fragments still need to be handled somehow;
• The power save code can call the driver vap method direct (if the frames are already encapsulated);
• The raw / management frame API pushes frames in via the side, which should also queue frames.

There are significant issues that creep up when you do this:

• A single queue may cause issues with higher priority traffic (eg management frames, voice frames, etc.)
• When trying to implement a “single thread only can progress” model, there's a subtle race condition where the other threads think that it can't continue so it just queues frames, and the sending thread is in the process of actively wrapping up and quitting, so the work stalls until the next TX occurs. (This happens with buf_ring and some of the 10GE drivers, fwiw.)
• When trying to implement a “TX taskqueue entry” model, you have issues with increased latency decreasing TX throughput.