Ethng: layer 2 interface improvements

• Multiple Tx queue support
  • Move queue management out of ifnet into the driver
  • add queue label into pkthdr
• Inpcb cached routes

• toe_usrreqs send, rcvd, disconnect, and abort
• listen notification
• route update notification
• VM support for fast temporary holding of user pages that doesn't conflict with other parts of the system
• syncache hooks for offloaded sockets

[http://www.fsmware.com/freebsd/ethng/ethng_net.diff sys/net:]

• Important changes:
  • Change radix tree lock to be a rwlock, this allows, lookups, the most common case to be shared
  • add IFF_MULTIQ flag
  • add if_mq_start routine and entry point to ifnet which is akin to if_start but leaves queue management up to the driver
  • add multi-queue IFQ_HANDOFF macros
  • mark route as timed out when it is deleted

• Secondary changes:
  • move error handling in ether_input out of the fast patch for better i-cache packing
  • add ARP type for IB
  • add rtalloc2 which returns the route unlocked and eliminates the need in common case to lock the radix tree exclusively

[http://www.fsmware.com/freebsd/ethng/ethng_netinet.diff sys/netinet:]

• Important changes:
  • add rss_hash field to inpcb for random queue distribution
  • add route to inpcb to avoid doing a route lookup (with three highly contended lock acquisitions) on every call to ip_output

[http://www.fsmware.com/freebsd/ethng/ethng_kern.diff sys/kern:] [http://www.fsmware.com/freebsd/ethng/ethng_sys.diff sys/sys:] [http://www.fsmware.com/freebsd/ethng/ethng_amd64.diff sys/amd64:] [http://www.fsmware.com/freebsd/ethng/ethng_vm.diff sys/vm:]

• Important changes:
  • add cpu interrupt binding
  • add support for physically contiguous jumbo frames
  • re-add lazy ref counting
  • inline critical path of critical_{enter, exit}
  • remove packet zone in favor of embedded mbuf (not pushing for approval for this)

• Disposable change:
  • special case rwlock callout