Transport [FreeBSD WIKI]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
===== FreeBSD Developer Summit: Transport Working Group =====

June 7, 2017 (Wednesday), 13:00-16:00

DMS 1140

==== Overview ====

We will discuss ongoing work, and ideas for improvements to, the transport protocols in the %%FreeBSD%% kernel.

There is a group that meets regularly to discuss transport work in the kernel. Notes of the group's work can be found in [[/TransportProtocols|TransportProtocols]]. The face-to-face time will allow us to whiteboard and discuss complex topics in an extended time. It will also allow us to include participants who are not able to make the regular meetings.

If you would like to participate, contact the working group chairs below and CC ''%%devsummit@%%''. You will be then added to this page. Please include a list of things you want to talk about or the areas you are interested in. This helps us in planning the session and to bring people together with common interests.

It //may// be possible to bring in people who cannot attend in person via video conference or chat tools. Notes during the session will be published later on for the whole community to see what we discussed.

==== Goals ====

In general, there are two areas we would like to cover:
  - Discussions of ongoing work that is complex enough, requires coordination, or requires architectural decisions that would benefit from face-to-face discussion among a larger group.
  - Exchange of ideas for upcoming work to gauge community interest, solicit feedback, look for conflicts/overlap, and generally keep everyone informed.

In particular, we may (or may not) cover the following suggested topics. This is not an exhaustive list and if you feel there is something missing that you want to talk about, contact one of the session chairs and we will include your topic here. Note that the numbering of the topics does not represent an ordering or importance indication of any kind, but rather a reference to the second table with the "topic of interest" column.

The final agenda will be guided by the interest the attendees express, so we may not even talk about any topics listed below if it appears there is little to no interest in the topic among the attendees. Therefore, if you feel strongly that we //should// discuss a topic, please communicate that to the chairs.

==== Topics ====

^ # ^ Topic Description ^
| 1 | RACK, BBR ([[/RandallStewart|RandallStewart]]) |
| 2 | Alternate stacks: How do we maintain them? What is the support expectation? How do we minimize code duplication? Etc. |
| 3 | Testing alternate TCP stacks: testing multiple module versions ([[/JonathanLooney|JonathanLooney]]) |
| 4 | Packet Pacing (Netflix learnings) |
| 5 | <del>iflib status??</del> |
| 6 | <del>Network stack as a module ([[/SteveKiernan|SteveKiernan]])</del> |
| 7 | Callout reworking ([[/HansPetterSelasky|HansPetterSelasky]] notes one possible option: [[https://svnweb.freebsd.org/base/projects/hps_head|hps_head branch]]) |
| 8 | Listen sockets ([[/GlebSmirnoff|GlebSmirnoff]]) |
| 9 | RSS update (see also [[TransportProtocols/TCPCalloutRedesign|this]])? |
| 10 | More efficient mbuf design for sendfile and friends ([[/DrewGallatin|DrewGallatin]]) |
| 11 | Drivers providing backpressure information to higher layers |
| 12 | Drivers providing packet timestamps to higher layers |
| 13 | TCP troubleshooting |
| 14 | LLE callout problem ([[https://reviews.freebsd.org/D4605|D4605]]) |

=== Suggested Agenda ===

  - Updates on What People Are Doing

    - (#1) RACK, BBR ([[/RandallStewart|RandallStewart]])

    - (#3) Testing alternate TCP stacks: testing multiple module versions ([[/JonathanLooney|JonathanLooney]])

    - (#8) Listen sockets ([[/GlebSmirnoff|GlebSmirnoff]])

    - (#10) More efficient mbuf design for sendfile and friends ([[/DrewGallatin|DrewGallatin]])

  - Interesting Information

    - (#4) Packet Pacing (Netflix learnings) ([[/RandallStewart|RandallStewart]])

  - Items to Discuss

    - (#2) Alternate stacks: How do we maintain them? What is the support expectation? How do we minimize code duplication? Etc.

    - (#13) TCP troubleshooting

    - (#11) Drivers providing backpressure information to higher layers

    - (#12) Drivers providing packet timestamps to higher layers

    - (#9) RSS work?

    - (#7) Callout reworking 

    - (#14) LLE callout problem ([[https://reviews.freebsd.org/D4605|D4605]]) ([[/HansPetterSelasky|HansPetterSelasky]])

//Note: General presentations about work you have done that does not require further discussions should be submitted for the %%FreeBSD%% Developer Summit track at BSDCan (see the [[DevSummit/201706|general developer summit]] page).//

==== Attending ====

In order to attend you need register for the [[DevSummit/201706|developer summit]] as well as by email for the session and be confirmed by the working group organizers.  Follow the guidelines described on the [[DevSummit/201706|main page]] or what you received by email.  For questions or if in doubt ask the session chairs.

**Please do NOT add yourself here.** Your name will appear automatically once you received the confirmation email. You need to put your name on the general [[DevSummit/201706|developer summit]] attendees list though.

^ # ^ Name ^ Username / Affiliation ^ Topics of Interest ^ Notes ^
| 1 | [[/JonathanLooney|JonathanLooney]] | jtl@ |  | Session co-chair |
| 2 | [[/DrewGallatin|DrewGallatin]] | gallatin@ | 10, 11, 12 |  |
| 3 | [[/RandallStewart|RandallStewart]] | rrs@ |  |  |
| 4 | [[/SepherosaZiehau|SepherosaZiehau]] | sephe@ |  |  |
| 5 | [[/MichaelTuexen|MichaelTuexen]] | tuexen@ |  |  |
| 6 | [[/BjoernZeeb|BjoernZeeb]] | bz@ |  |  |
| 7 | [[/EricvanGyzen|EricvanGyzen]] | vangyzen@ |  |  |
| 8 | [[/SteveWills|SteveWills]] | swills@ |  |  |
| 9 | [[/NavdeepParhar|NavdeepParhar]] | np@ | 2, 4, 10 |  |
| 10 | Steve Wahl | Dell EMC |  |  |
| 11 | Jason Eggleston | %%LimeLight%% Networks |  |  |
| 12 | [[/LawrenceStewart|LawrenceStewart]] | lstewart@ |  |  |
| 13 | [[/RyanStone|RyanStone]] | rstone@ |  |  |
| 14 | Ming Qiao | Juniper Networks |  |  |
| 15 | [[/HansPetterSelasky|HansPetterSelasky]] | hselasky@ | 14 |  |
| 16 | [[/MikeKarels|MikeKarels]] | karels@ |  |  |
| 17 | [[/MikeSilbersack|MikeSilbersack]] | silby@ | 2, 6 |  |
| 18 | Siva Mahadevan | %%FreeBSD%% Foundation/emaste@ |  |  |
| 19 | Charlie Yang | %%FreeBSD%% Foundation/emaste@ |  |  |
| 20 | [[/DavidSomayajulu|DavidSomayajulu]] | davidcs@ |  |  |
| 21 | Kevin Bowling | %%LimeLight%% Networks |  |  |

==== Results ====

We kick off with a discussion of BBR and Rack where rrs@ explains what these are and why we have them.  A [[https://www.ietf.org/proceedings/97/slides/slides-97-iccrg-bbr-congestion-control-02.pdf|presentation]] on BBR can be found beneath the link.  kbowling brings up small objects and says that %%LimeLight%% have a patch for issues there.  rrs@ says he tried that but that he did not find it to be helpful.  The issue seems to come down to datacenter workloads.  rrs@ says that it's all about pacing and that the congestion window is not as much the issue.

jtl@ brings up the idea of pushing the base stack into its own module so that with module versioning we can upgrade stacks without a reboot.  This is something that we hope to get to soon.

The listen sockets discussion doesn't really go anywhere as glebius is not present, but there are rumors that the current rework is ready to go into the tree.

gallatin@ explains a recent rework of the mbuf code to handle large file sends via sendfile.  The idea is to have the mbuf reference a lot more data than we can now and to free all those pages when the single mbuf is freed.  The code is not quite ready to commit but is getting closer to review.

rstone@ has changes for low RTT networks which need to use a new option but are otherwise ready to go in.  This will also be an IETF draft written with other folks who do IETFy things.

The NF folks have been looking at hardware vs. software pacing and have found, that for their workload, that they do not get benefits in CPU utilization but do get QOE (Quality of Experience) on the client side.

The room discusses the direction in which we take the TCP stack code.  There is general agreement, with qualifications from rrs@, about trying to re-unify the the TCP code.  A discussion then ensues on whether or not the coalesced stack would coalesce around a Rack based stack with changes to address connections that are non-Rack.  We'd like to have Rack be default in HEAD but not in STABLE to get more testing.  Michael asks about how well we can pull the pacer our of the Rack/BBR code and use it generally.  rrs@ thinks that is already done.  rrs@ says there are 3 steps to getting this in

  - Get in the Black Box
  - Get in the Pacer which no one uses until Rack/BBR is in.
  - Place Rack and BBR into the tree.

rrs@ and jtl@ give a black box demo

jtl@ will make a Phabricator review for the black box code

We then move into a discussion of driver back pressure.  hps@ has a mechanism in the %%ConnectX%% drivers to do this.  The tag mechanism goes from the TCP layer down through the driver.  hps@, npn@, lstewart@, gallatin@ to take this on.

gnn@ to look at the packet timestamps

----

{{tag>Historical}}