====== What Is Infiniband? ====== InfiniBand (IB) is a high-speed (10-300Gb/s) low-latency (140–2600 ns) switched-fabric interconnect, developed primarily for HPC, but as of now widely adopted where its properties are in demand. InfiniBand is supported on most major operating systems: Windows, Linux, (Open)Solaris, AIX, HP-UX, z/OS, among others. %%OpenFabrics%% Alliance (founded in 2004 as %%OpenIB%% Alliance) is developing a cross-platform IB/RDMA stack for Linux and Windows. The stack is released as OFED (%%OpenFabrics%% Enterprise Distribution) under both GPL and BSD licenses. ====== Infiniband in FreeBSD ====== The OFED stack was merged in March 2011 ([[http://jeffr-tech.livejournal.com/23467.html]]) and is available in %%FreeBSD%% 9.0 and later. Proprietary IB stacks are known to exist outside of %%FreeBSD%% proper. In particular, three major storage appliance vendors -- [[http://www.emc.com/en-us/storage/isilon/index.htm|Dell EMC Isilon]], %%NetApp%%, and Panasas -- ship products based on %%FreeBSD%% and use InfiniBand for high-speed interconnects. ====== Infiniband hardware support by release ====== ^ Card series ^ Driver ^ %%FreeBSD%% 9.x ^ %%FreeBSD%% 10.x ^ %%FreeBSD%% 11.x ^ Comments ^ | %%InfiniHost%% | mthca | Yes | Yes | Yes | Not maintained | | %%ConnectX%% | mlx4 | Yes | Yes | Yes | | | %%ConnectX%%-2 | mlx4 | Yes | Yes | Yes | | | %%ConnectX%%-3/Pro | mlx4 | Yes | Yes | Yes | | | %%ConnectX%%-4 (Eth) | mlx5 | Yes | Yes | Yes | Available in 9.4+/10.2+(?)/11.0+ | | %%ConnectX%%-4 (IB) | mlx5 | Yes | Yes | Yes | Available in 9.4+/10.2+(?)/11.0+ | | Connect-IB | mlx5 | Yes | Yes | Yes | Available in 9.4+/10.3+(?)/11.0+ | ====== Building OFED in FreeBSD ====== ===== Kernel ===== - Add ''%%WITH_OFED='yes'%%'' to /etc/src.conf. This will build all of the OFED kernel modules, e.g. ibcore, alongside the static kernel image. - Add the following options to your KERNCONF to compile the necessary protocols/protocol options into your kernel: options OFED # Infiniband protocol stack and support options SDP # Sockets Direct Protocol for infiniband options IPOIB_CM # Use connected mode IPoIB * Add the OFED drivers to your KERNCONF (alternately you could use the MODULES_OVERRIDE variable in /etc/src.conf): device ipoib # IP over IB devices device mlx4ib # ConnectX Infiniband support device mlxen # ConnectX Ethernet support device mthca # Infinihost cards * Run buildkernel/installkernel as per the usual process. Note: If your /usr/src tree is newer than your kernel, you must do a buildkernel/installkernel along with buildworld/installworld. See the %%FreeBSD%% handbook for full details. ===== Userland ===== * This is no longer necessary as OFED support is on by default now, but is still documented here for those who need to do a buildworld anyway. - Add ''%%WITH_OFED='yes'%%'' to /etc/src.conf. - Run buildworld/installworld per usual process. - Set the "sys.device.mlx4_coreX.mlx4_portY" sysctl to either "eth" or "ib" depending on how you want the device ports to be configured. ===== Kernel Modules ===== It is now possible to enable and upgrade Infiniband support on %%FreeBSD%% in a few minutes. As an alternative to building a new kernel, recent releases provide IB drivers as kernel modules, and userland now supports Infiniband by default, so there is no need to perform a buildworld. The script below will build, install, and load the appropriate drivers: #!/bin/sh -e usage() { printf "Usage: $0 4|5 datagram|connected\n" printf "* Use 4 for ConnectX 3 and earlier, 5 for later HCAs.\n" printf "* Specify connected if you want to use connected mode ipoib.\n" exit 1 } if [ $# != 2 ]; then usage fi driver_version=$1 ipoib_mode=$2 case $ipoib_mode in datagram) ;; connected) export CFLAGS="-DIPOIB_CM" ;; *) usage ;; esac cd /usr/src make clean # Rebuild all objs in case src was updated cd sys/modules for module in mlx$driver_version ibcore mlx${driver_version}ib ipoib; do rm -f /boot/modules/$module.ko cd $module make clean make DEBUG_FLAGS="-DVIMAGE=1" make install # Minimize damage if kldload triggers a panic. Unlikely, but has happened # with experimental drivers. sync kldload $module || true auto-append-line ${module}'_load="YES"' /boot/loader.conf $0 cd .. done kldstat cat << EOM To disable Infiniband support, remove or comment out the following device loads in /boot/loader.conf: mlx$driver_version ibcore mlx${driver_version}ib ipoib EOM Following updates to /usr/src, the IB kernel modules can be upgraded using the script below: #!/bin/sh -e usage() { printf "Usage: $0 4|5 datagram|connected\n" printf "* Use 4 for ConnectX 3 and earlier, 5 for later HCAs.\n" printf "* Specify connected if you want to use connected mode ipoib.\n" exit 1 } if [ $# != 2 ]; then usage fi driver_version=$1 ipoib_mode=$2 for interface in $(ifconfig | awk -F : '$1 ~ "^ib" { print $1 }'); do ifconfig $interface down done kldunload ipoib mlx${driver_version}ib ibcore mlx$driver_version || true auto-build-infiniband-modules $1 $2 for interface in $(ifconfig | awk -F : '$1 ~ "^ib" { print $1 }'); do ifconfig $interface up done Both of these scripts will be included in sysutils/autoadmin in the near future. ===== OpenSM ===== To start %%OpenSM%% automatically when the system boots, add opensm_enable="YES" to /etc/rc.conf, e.g. # sysrc opensm_enable="YES" ====== Status ====== Mellanox engineers, et. al., have made substantial improvements to the IB stack in 2015/2016 to support the latest %%ConnectX%% HCAs. The [[http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband|freebsd-infiniband]] email archives contain discussions of progress and issues. There are some outstanding work items for Infiniband/OFED on %%FreeBSD%%. ===== Wishlist ===== * WITH_OFED doesn't cover all OFED stack utilities; a number of them need to be ported from Linux and contributed back to OFED (preferred) or rewritten to work with %%FreeBSD%%. ====== Feedback ====== There's an [[http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband|infiniband mailing list]]. If you're having issues or would like to provide feedback about the Infiniband stack on %%FreeBSD%%, you should subscribe and poke the mailing list. ====== References ====== * [[http://en.wikipedia.org/wiki/InfiniBand]] * [[FreeBSD9]] * [[GlusterFS]] * [[http://www.openfabrics.org/]] ---- Everything below is "work in progress". It will be moved into the correct place on this page when it's ready B) ====== Where are the sources? ====== * contrib/ofed - all of the userspace libraries and tools, e.g. opensm(8). * sys/ofed - all of the kernel code and drivers, e.g. mlx4(4). ====== Configuration / runtime settings ====== TODO: We should explain the relevant sysctl.conf, loader.conf, and rc.conf settings. ===== /etc/sysctl.conf ===== NOTE (ngie): these items should be documented in the device driver/network stack specific manpages to reduce duplication. * net.inet.sdp.pcblist * mlx4 (this : * Mode switching * sys.device.mlx4_core0.mlx4_portX (valid values "ib" or "eth") ===== /boot/loader.conf ===== See driver-specific manpages for more details, e.g. "man 4 mlx4en". ===== /etc/rc.conf ===== * IP configuration settings (ifconfig_ibX, ifconfig_mlxenX) * opensm_enable ====== Operating modes ====== We should add information here about the 2 main operating modes for the Mellanox Infiniband adapters: - Native infiniband mode - Native ethernet mode It would be useful to clarify that in Infiniband mode, IP can be encapsulated and sent using %%IPoIB%%. ====== Cables & connectors ====== We should add info and photo's here for the different cables & connector types * Explain passive vs active * Copper & optical cable * CX4 connectors ← JustinClift has some of these. Can take photos * SFP * QSFP connectors ← JustinClift has some of these. Can take photos ====== Maximum cable length ====== It would also be useful to explain the maximum length of cables, according to type, connector, data rate (and whatever else).