Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
| devsummit:201806:transport [2018/06/06 13:29] – Add marius JonathanLooney | devsummit:201806:transport [2018/06/08 18:59] (current) – add xref to e1000 big-endian PR MarkLinimon | ||
|---|---|---|---|
| Line 46: | Line 46: | ||
| === Suggested Agenda === | === Suggested Agenda === | ||
| - | //TBD// | + | * TCP Stacks |
| + | * RACK, BBR ([[/RandallStewart|RandallStewart]]) | ||
| + | * Alternate stacks: How do we maintain them? What is the support expectation? | ||
| + | * Alternate stacks: Building the " | ||
| + | * UDP option support | ||
| + | * iflib status?? | ||
| + | * High-precision timestamps [[https://reviews.freebsd.org/ | ||
| + | * 12.0 Preparation | ||
| + | * ABI padding check. | ||
| + | * [[https:// | ||
| + | * Software packet pacing / burst mitigation | ||
| + | * Unified API for hardware/ | ||
| + | * TLS-related stuff: | ||
| + | * Kernel TLS | ||
| + | * In-line TLS API | ||
| + | * More efficient mbuf design for sendfile and friends | ||
| ==== Attending ==== | ==== Attending ==== | ||
| Line 56: | Line 71: | ||
| ^ # ^ Name ^ Username / Affiliation ^ Topics of Interest ^ Notes ^ | ^ # ^ Name ^ Username / Affiliation ^ Topics of Interest ^ Notes ^ | ||
| | 1 | [[/ | | 1 | [[/ | ||
| - | | 2 | [[/DrewGallatin|DrewGallatin]] | gallatin@ | | | | + | | 2 | [[/RandallStewart|RandallStewart]] | rrs@ | | | |
| - | | 3 | [[/RandallStewart|RandallStewart]] | rrs@ | | | | + | | 3 | [[/MichaelTuexen|MichaelTuexen]] | tuexen@ | | | |
| - | | 4 | [[/MichaelTuexen|MichaelTuexen]] | tuexen@ | | | | + | | 4 | [[/LawrenceStewart|LawrenceStewart]] | lstewart@ | | | |
| - | | 5 | [[/LawrenceStewart|LawrenceStewart]] | lstewart@ | | | | + | | 5 | [[/MikeKarels|MikeKarels]] | karels@ | | | |
| - | | 6 | [[/MikeKarels|MikeKarels]] | karels@ | | | | + | | 6 | [[/NavdeepParhar|NavdeepParhar]] | np@ | | | |
| - | | 7 | [[/NavdeepParhar|NavdeepParhar]] | np@ | | | | + | | 7 | [[/MariusStrobl|MariusStrobl]] | marius@ | | | |
| - | | 8 | [[/MariusStrobl|MariusStrobl]] | marius@ | | | | + | | 8 | [[/JohnBaldwin|JohnBaldwin]] | jhb@ | | Second half only | |
| ==== Results ==== | ==== Results ==== | ||
| + | === Alternate TCP Stacks === | ||
| - | //To be completed during/after the session.// | + | * RACK to be committed soon unless traffic on review |
| + | * + Tail Loss Probe, | ||
| + | * Used in prod. by Netflix | ||
| + | * RACK committed during session | ||
| + | * BBR has open issue that is being worked with Google | ||
| + | * Netflix: Bandwidth measurements need tweaking (RACK outperforms it for them) | ||
| + | * Not going to make 12.0-RELEASE | ||
| + | * setsockopt() can choose TCP stack for an individual connection | ||
| + | * Can be changed on an in-flight socket | ||
| + | * example stack to be ripped out after RACK is in | ||
| + | * Pluggable Congestion Control vs. Separate Stack | ||
| + | * Ideally RACK and BBR would be pluggable cc modules | ||
| + | * In practice, existing abstractions are insufficient | ||
| + | * Unclear what abstractions are correct | ||
| + | * RACK would really require pluggable loss recovery | ||
| + | * Goal is to eventually get back to one TCP stack + plugins for RACK, BBR, etc | ||
| + | * Alternate stacks aren't being built by default | ||
| + | * Do it in Jenkins? | ||
| + | * LINT already does it? | ||
| + | * Building Default Stack as a kld module | ||
| + | * Significant code movement, will cause merge headaches | ||
| + | * Allows on-the-fly TCP upgrades via loading a new module | ||
| + | * " | ||
| + | * Currently require at least one TCP stack | ||
| + | * jtl hopes to fix this for 13-CURRENT | ||
| + | * Need to fix the module penalty (lock inlining, etc) | ||
| + | * jtl: current inline atomic primitives might be slower than function call (due to memory barrier) | ||
| + | |||
| + | === UDP Options === | ||
| + | |||
| + | * Tom Jones: new committer, working on IETF draft for UDP options | ||
| + | * UDP processing path to change (ip_len - sizeof(struct udphdr) != payload length) | ||
| + | * Review imminent for notification purposes | ||
| + | * Won't be pushed into tree until IETF draft finalized | ||
| + | |||
| + | === iflib Status === | ||
| + | |||
| + | * ixl(9) iflib driver almost ready to be committed to head | ||
| + | * Benno working on Aquantia driver | ||
| + | * Some netmap problems still | ||
| + | * Invaluable for vxlan development | ||
| + | * Need feedback from driver maintainers on KPI + documentation | ||
| + | * erj feedback: Documentation is okay | ||
| + | * shurd: Got up and running pretty easily | ||
| + | * LOR between netmap and iflib | ||
| + | * Needs somebody to look into and fix | ||
| + | * e1000 regressions (CURRENT) | ||
| + | * em: errata workarounds missing for TSO + TSO on by default | ||
| + | * Probably causing significant fraction of MAC hang complaints | ||
| + | * Disable TSO by default in em? Workarounds have a significant perf penalty anyway | ||
| + | * 10-STABLE, 11-STABLE seem fine | ||
| + | * Endian bugs on big-endian machines | ||
| + | * POWER9 bringup might shake these out? | ||
| + | * jtl: Regression in a major driver, probably would block 12.0-RELEASE | ||
| + | * See [[https://bugs.freebsd.org/ | ||
| + | |||
| + | === High Precision Timestamps === | ||
| + | |||
| + | * Two separate issues: higher precision internal timers, and on-wire TCP timestamp format | ||
| + | * Changes to on-wire format require standardization (ideally) | ||
| + | * RACK calculates RTT based off of timer values maintained internally. | ||
| + | * Currently uses millisecond granularity (BBR goes up to microsecond) | ||
| + | * Easiest path forward would be to increase the granularity and use the RACK stack. | ||
| + | |||
| + | === 12.0 Preparation === | ||
| + | * xtcpcb, x* compat broken last year | ||
| + | * brooks has a review to add spares, remove dependencies on internal kernel data structures | ||
| + | * netstat -M broken; should really fix it for RELEASE | ||
| + | |||
| + | === lstewart === | ||
| + | |||
| + | * Reschedule the transport call? | ||
| + | |||
| + | === Packet Pacing | ||
| + | |||
| + | * Can be done in s/w, can be quite expensive | ||
| + | * At least two NIC vendors support h/w pacing | ||
| + | * Need a unified API | ||
| + | * RACK | ||
| + | * Attempts to not send " | ||
| + | * If RACK tries to exceed this, it queues and delays for "a little while" | ||
| + | * Burst mitigation is not done in at-clock mode | ||
| + | * BBR | ||
| + | * Tries to find the optimal TSO size based on the measures bandwidth | ||
| + | * Will pace its output rate to the measured bandwidth | ||
| + | * Chelsio h/w pacing | ||
| + | * Pacing is configured per-flow | ||
| + | * The NIC will accept as many packets as you send and pace them out | ||
| + | * Leads to buffer bloat, needs backpressure | ||
| + | * MLX h/w pacing | ||
| + | * CX-4: can only pace to a certain set of rates (12-14 rates), pacing is done per-interface | ||
| + | * CX-5: Can pace to larger set of rates (10k rates w/ configurable ), pacing can be done per-flow | ||
| + | * s/w pacing is limited to 50us granularity | ||
| + | * There is currently a unified API, but there is no way to query capabilities | ||
| + | * Especially rate values supported | ||
| + | * Two use cases: automatic tuning of rate values, or rates specified by user and stack sw paces with hw pace taken into account | ||
| + | * Decision of how much sw pacing to do is a policy decision; need to offer users some level of control here | ||
| + | * it may be better to fail to pace than burn CPU on sw pacing | ||
| + | * MLX: expect that with future NICs, h/w pacing should have enough granularity in its configuration to make h/w-only pacing possible | ||
| + | * eg in the future you may be either h/w paced or s/w based, but not a combination | ||
| + | * What about pacing on a middlebox, rather than an endpoint? | ||
| + | * Requires a pacing API to work without an associated pcb | ||
| + | * e.g. dummynet would tag packets with a pacing queue id rather than doing it in ip_output() | ||
| + | * Would allow dummynet to do per-flow pacing | ||
| + | * " | ||
| + | * Allow to choose worst implementation you'll accept (ie HW-only, HW-SW combo, SW only) | ||
| + | * Defer discussion of how to handle routing changes | ||
| + | * It gets really bad if you fail over to a NIC that has a worse implementation that what you currently have | ||
| + | |||
| + | === NIC Configuration === | ||
| + | |||
| + | * MLX has 2000 sysctl nodes for configuration | ||
| + | * f/w version, stats, steering | ||
| + | |||
| + | === TLS === | ||
| + | |||
| + | * Netflix has in-kernel TLS | ||
| + | * Requires encryption to be in the kernel | ||
| + | * Key negotiation is done in userland | ||
| + | * Framing is done in-kernel | ||
| + | * Gets openssl out of the rx/tx path | ||
| + | * Netflix to upstream " | ||
| + | * " | ||
| + | * Encrypt from an unmapped mbuf, or into an unmapped mbuf | ||
| + | * Unmapped mbuf is independent of kernel TLS, but kernel TLS depends on it | ||
| + | * Special extra step where plaintext from sendfile is encrypted and then enqueued for tx | ||
| + | * h/w crypto offload: | ||
| + | * Needs a way to locate session key | ||
| + | * TCP needs special knowledge about encrypted packets (i.e. don't copy them and lose the encryption metadata!) | ||
| + | * mbufs can share TLS frames | ||
| + | * For rexmits, the cipher algorithms are much simpler if you start from the beginning of the TLS frame | ||
| + | * Chelsio can re-encrypt the start of the frame but not re-transmit the ACK'ed data (MSS granularity) | ||
| + | * A bunch of rough edges to sand down before committing | ||
| + | * re-keying not supported for in-kernel, but doesn' | ||
| + | |||
| + | === Actions === | ||
| + | |||
| + | * sbruno: ensure alternate stacks are built in Jenkins | ||