Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
| projectsroutingproposal [2015/09/23 21:52] – AlexanderChernikov | projectsroutingproposal [2016/01/18 19:23] (current) – glebius | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | This article outlines some routing-related problems in our network stack and describes | + | This article outlines some generic |
| ==== Current problems ==== | ==== Current problems ==== | ||
| === Performance issues === | === Performance issues === | ||
| - | Many people considered routing lookups slow (and created various mechanisms of avoiding such lookups, like route caching in tunnels (or even protocols), or having per-flow hash record (FLOWTABLE%%))%%. | + | Many people considered routing lookups slow (and created various mechanisms of avoiding such lookups, like route caching in tunnels (or even protocols), |
| + | or having per-flow hash record (FLOWTABLE%%))%%. | ||
| Typical packet processing (forwarding for router, or output for web server) path consists of: | Typical packet processing (forwarding for router, or output for web server) path consists of: | ||
| - | * doing routing lookup (radix read rwlock + rte mutex lock) | + | * doing routing lookup (radix read rwlock + routing entry (rte) mutex lock) |
| * (optionally) interface address (ifa) atomic refcount acquire/ | * (optionally) interface address (ifa) atomic refcount acquire/ | ||
| - | * doing LLE lookup (afdata read rwlock + llentry read (or write) lock) | + | * doing link level entry (lle, llentry) |
| - | Actually, given number of locks is roughly like having single Giant lock: single-cpu performance (600-1mpps) is very similar to the results you can get using modern dual-cpu system with 16/32 cores. The most expensive operation is acquiring //struct rtentry// mutex (think about default route delivering HUGE contention on single lock, which is also hold for plenty of time). However, other locks do heavily impact processing path. | + | Actually, given number of locks is roughly like having single Giant lock: single-cpu performance (600-1mpps) is very similar to the results you can get |
| + | using modern dual-cpu system with 16/32 cores. The most expensive operation is acquiring //struct rtentry// mutex (think about default route delivering | ||
| + | HUGE contention on single lock, which is also hold for plenty of time). However, other locks do heavily impact processing path. | ||
| === Routing code itself === | === Routing code itself === | ||
| Dealing with performance issues revealed number of problems which also have to be dealt with: | Dealing with performance issues revealed number of problems which also have to be dealt with: | ||
| Line 26: | Line 30: | ||
| We managed to remove rwlocks/ | We managed to remove rwlocks/ | ||
| In fact, we were able to reach close to 12MPPS for IPv4 forwarding (2xE2660, 8 cores) which is 5-10 times faster than current benchmarks. | In fact, we were able to reach close to 12MPPS for IPv4 forwarding (2xE2660, 8 cores) which is 5-10 times faster than current benchmarks. | ||
| - | It should also be noted, that, in general, (current) changes available in project/ | ||
| + | ==== Some benchmarks ==== | ||
| + | === Forwarding benchmark === | ||
| + | {{projectsroutingproposal/ | ||
| + | * " | ||
| + | * " | ||
| + | * " | ||
| + | * " | ||
| + | === Forwarding/ | ||
| + | {{projectsroutingproposal/ | ||
| + | * " | ||
| + | * " | ||
| + | * " | ||
| + | * " | ||
| + | * " | ||
| ==== High-level overview ==== | ==== High-level overview ==== | ||
| The key concept is hiding all routing internals and providing pre-computed easy-to-use result to all consumers with minimal locking. | The key concept is hiding all routing internals and providing pre-computed easy-to-use result to all consumers with minimal locking. | ||
| - | We provide separate functions for each address family (fib4/rib4/fib6/rib6). | + | We provide separate functions for each address family (fib4/fib6/rib). |
| We also provide several different (in terms of returned info and relative overhead) methods of retrieving routing data. | We also provide several different (in terms of returned info and relative overhead) methods of retrieving routing data. | ||
| - | Typically, lookup result is pre-computed next hop data, copied to caller-provided 64-bytes nhop_* structure, with all necessary data _including_ the data that needs to be prepended to mbuf (currently - pre-computed L2 header). | + | Typically, lookup result is pre-computed next hop data, copied to caller-provided 64-bytes nhop_* structure, with (optionally) |
| - | + | ||
| - | New api/ | + | |
| - | [[https:// | + | |
| - | [[https:// | + | |
| - | [[https:// | + | |
| - | [[https:// | + | |
| - | + | ||
| - | It is worth noting that all API/ | + | |
| - | While it might have seen as a good " | + | |
| - | + | ||
| - | Another related change is avoiding IPv6 addresses with embedded scope: you could not be sure when the IP address has its scope embedded. Additionally, | + | |
| === Locking changes === | === Locking changes === | ||
| - | + | | |
| - | | + | |
| * rte mutexes are mostly unused leading to single rmlock necessary for L3 lookup | * rte mutexes are mostly unused leading to single rmlock necessary for L3 lookup | ||
| * LLE if_afdata lock uses the same dual-locking approach. (there are plans to move lltable lock inside lltable) | * LLE if_afdata lock uses the same dual-locking approach. (there are plans to move lltable lock inside lltable) | ||
| * Additionally, | * Additionally, | ||
| + | |||
| + | New api/ | ||
| === More details on routing === | === More details on routing === | ||
| Line 58: | Line 65: | ||
| - Forwarding code (_output() routines) Here again we need very basic info, so we can use optimised fib table lookups. | - Forwarding code (_output() routines) Here again we need very basic info, so we can use optimised fib table lookups. | ||
| - Control plane code (rtsock, address routines) Use slower rib-based procedures returning individual rtentries | - Control plane code (rtsock, address routines) Use slower rib-based procedures returning individual rtentries | ||
| - | |||
| - | //struct route// was converted to //struct nhop_info// which is used for similar purpose: | ||
| - | * pass recently-looked up next top | ||
| - | * pass (properly referenced, maybe multipath) next hop | ||
| - | * pass address family (loopback case) | ||
| === More details on LLE === | === More details on LLE === | ||
| Line 76: | Line 78: | ||
| ==== Merge plan ==== | ==== Merge plan ==== | ||
| Basic plan looks like the following: | Basic plan looks like the following: | ||
| - | - Merge most of LLE changes not related to locking/ | + | - <del>Merge most of LLE changes not related to locking/ |
| - | - Discuss/ | + | - <del>Discuss/ |
| - | - Discuss/merge new routing lookup API | + | - Perform one-by-one conversion of non-critical |
| - | - Perform one-by-one conversion of route(4) consumers | + | - Discuss/ |
| + | - Discuss/ | ||
| - Discuss/ | - Discuss/ | ||
| - | === Considerations/ | + | ==== Status ==== |
| + | Current conversion status is available [[ProjectsRoutingProposal/ | ||
| + | ==== Considerations/ | ||
| - Wider use of rmlock in kernel. Rmlock has side effect of " | - Wider use of rmlock in kernel. Rmlock has side effect of " | ||
| - Could we teach PFIL to start looking at mbuf starting from certain offset? (eliminate 2x LLE header copy) | - Could we teach PFIL to start looking at mbuf starting from certain offset? (eliminate 2x LLE header copy) | ||