InfiniBand [FreeBSD WIKI]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== What Is Infiniband? ======

InfiniBand (IB) is a high-speed (10-300Gb/s) low-latency (140–2600 ns) switched-fabric interconnect, developed primarily for HPC, but as of now widely adopted where its properties are in demand.

InfiniBand is supported on most major operating systems: Windows, Linux, (Open)Solaris, AIX, HP-UX, z/OS, among others. %%OpenFabrics%% Alliance (founded in 2004 as %%OpenIB%% Alliance) is developing a cross-platform IB/RDMA stack for Linux and Windows. The stack is released as OFED (%%OpenFabrics%% Enterprise Distribution) under both GPL and BSD licenses.

====== Infiniband in FreeBSD ======

The OFED stack was merged in March 2011 ([[http://jeffr-tech.livejournal.com/23467.html]]) and is available in %%FreeBSD%% 9.0 and later.

Proprietary IB stacks are known to exist outside of %%FreeBSD%% proper. In particular, three major storage appliance vendors -- [[http://www.emc.com/en-us/storage/isilon/index.htm|Dell EMC Isilon]], %%NetApp%%, and Panasas -- ship products based on %%FreeBSD%% and use InfiniBand for high-speed interconnects.

====== Infiniband hardware support by release ======

^ Card series ^ Driver ^ %%FreeBSD%% 9.x ^ %%FreeBSD%% 10.x ^ %%FreeBSD%% 11.x ^ Comments ^
| %%InfiniHost%% | mthca | Yes | Yes | Yes | Not maintained |
| %%ConnectX%% | mlx4 | Yes | Yes | Yes |  |
| %%ConnectX%%-2 | mlx4 | Yes | Yes | Yes |  |
| %%ConnectX%%-3/Pro | mlx4 | Yes | Yes | Yes |  |
| %%ConnectX%%-4 (Eth) | mlx5 | Yes | Yes | Yes | Available in 9.4+/10.2+(?)/11.0+ |
| %%ConnectX%%-4 (IB) | mlx5 | Yes | Yes | Yes | Available in 9.4+/10.2+(?)/11.0+ |
| Connect-IB | mlx5 | Yes | Yes | Yes | Available in 9.4+/10.3+(?)/11.0+ |

====== Building OFED in FreeBSD ======

===== Kernel =====

  - Add ''%%WITH_OFED='yes'%%'' to /etc/src.conf. This will build all of the OFED kernel modules, e.g. ibcore, alongside the static kernel image.
  - Add the following options to your KERNCONF to compile the necessary protocols/protocol options into your kernel:

<code>
options        OFED            # Infiniband protocol stack and support
options        SDP             # Sockets Direct Protocol for infiniband
options        IPOIB_CM        # Use connected mode IPoIB
</code>

  * Add the OFED drivers to your KERNCONF (alternately you could use the MODULES_OVERRIDE variable in /etc/src.conf):

<code>
device         ipoib           # IP over IB devices
device         mlx4ib          # ConnectX Infiniband support
device         mlxen           # ConnectX Ethernet support
device         mthca           # Infinihost cards
</code>

  * Run buildkernel/installkernel as per the usual process.

Note: If your /usr/src tree is newer than your kernel, you must do a buildkernel/installkernel along with buildworld/installworld.  See the %%FreeBSD%% handbook for full details.

===== Userland =====

* This is no longer necessary as OFED support is on by default now, but is still documented here for those who need to do a buildworld anyway.

  - Add ''%%WITH_OFED='yes'%%'' to /etc/src.conf.
  - Run buildworld/installworld per usual process.
  - Set the "sys.device.mlx4_coreX.mlx4_portY" sysctl to either "eth" or "ib" depending on how you want the device ports to be configured.

===== Kernel Modules =====

It is now possible to enable and upgrade Infiniband support on %%FreeBSD%% in a few minutes.

As an alternative to building a new kernel, recent releases provide IB drivers as kernel modules, and userland now supports Infiniband by default, so there is no need to perform a buildworld.

The script below will build, install, and load the appropriate drivers:

<code>
 #!/bin/sh -e

usage()
{
    printf "Usage: $0 4|5 datagram|connected\n"
    printf "* Use 4 for ConnectX 3 and earlier, 5 for later HCAs.\n"
    printf "* Specify connected if you want to use connected mode ipoib.\n"
    exit 1
}

if [ $# != 2 ]; then
    usage
fi

driver_version=$1
ipoib_mode=$2

case $ipoib_mode in
datagram)
    ;;

connected)
    export CFLAGS="-DIPOIB_CM"
    ;;

*)
    usage
    ;;

esac
    
cd /usr/src
make clean  # Rebuild all objs in case src was updated

cd sys/modules
for module in mlx$driver_version ibcore mlx${driver_version}ib ipoib; do
    rm -f /boot/modules/$module.ko
    cd $module
    make clean
    make DEBUG_FLAGS="-DVIMAGE=1"
    make install
    # Minimize damage if kldload triggers a panic.  Unlikely, but has happened
    # with experimental drivers.
    sync
    kldload $module || true
    auto-append-line ${module}'_load="YES"' /boot/loader.conf $0
    cd ..
done
kldstat
cat << EOM

To disable Infiniband support, remove or comment out the following device
loads in /boot/loader.conf:

mlx$driver_version ibcore mlx${driver_version}ib ipoib

EOM
</code>

Following updates to /usr/src, the IB kernel modules can be upgraded using the script below:

<code>
 #!/bin/sh -e

usage()
{
    printf "Usage: $0 4|5 datagram|connected\n"
    printf "* Use 4 for ConnectX 3 and earlier, 5 for later HCAs.\n"
    printf "* Specify connected if you want to use connected mode ipoib.\n"
    exit 1
}

if [ $# != 2 ]; then
    usage
fi

driver_version=$1
ipoib_mode=$2

for interface in $(ifconfig | awk -F : '$1 ~ "^ib" { print $1 }'); do
    ifconfig $interface down
done

kldunload ipoib mlx${driver_version}ib ibcore mlx$driver_version || true

auto-build-infiniband-modules $1 $2

for interface in $(ifconfig | awk -F : '$1 ~ "^ib" { print $1 }'); do
    ifconfig $interface up
done
</code>

Both of these scripts will be included in sysutils/autoadmin in the near future.

===== OpenSM =====

To start %%OpenSM%% automatically when the system boots, add opensm_enable="YES" to /etc/rc.conf, e.g.

<code>
# sysrc opensm_enable="YES"
</code>

====== Status ======

Mellanox engineers, et. al., have made substantial improvements to the IB stack in 2015/2016 to support the latest %%ConnectX%% HCAs.  The [[http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband|freebsd-infiniband]] email archives contain discussions of progress and issues.

There are some outstanding work items for Infiniband/OFED on %%FreeBSD%%.

===== Wishlist =====

  * WITH_OFED doesn't cover all OFED stack utilities; a number of them need to be ported from Linux and contributed back to OFED (preferred) or rewritten to work with %%FreeBSD%%.

====== Feedback ======

There's an [[http://lists.freebsd.org/mailman/listinfo/freebsd-infiniband|infiniband mailing list]]. If you're having issues or would like to provide feedback about the Infiniband stack on %%FreeBSD%%, you should subscribe and poke the mailing list.

====== References ======

  * [[http://en.wikipedia.org/wiki/InfiniBand]]
  * [[FreeBSD9]]
  * [[GlusterFS]]
  * [[http://www.openfabrics.org/]]

----

Everything below is "work in progress".  It will be moved into the correct place on this page when it's ready B)

====== Where are the sources? ======

  * contrib/ofed - all of the userspace libraries and tools, e.g. opensm(8).
  * sys/ofed - all of the kernel code and drivers, e.g. mlx4(4).

====== Configuration / runtime settings ======

TODO: We should explain the relevant sysctl.conf, loader.conf, and rc.conf settings.

===== /etc/sysctl.conf =====

NOTE (ngie): these items should be documented in the device driver/network stack specific manpages to reduce duplication.

  * net.inet.sdp.pcblist
  * mlx4 (this :
  * Mode switching
    * sys.device.mlx4_core0.mlx4_portX (valid values "ib" or "eth")

===== /boot/loader.conf =====

See driver-specific manpages for more details, e.g. "man 4 mlx4en".

===== /etc/rc.conf =====

  * IP configuration settings (ifconfig_ibX, ifconfig_mlxenX)
  * opensm_enable

====== Operating modes ======

We should add information here about the 2 main operating modes for the Mellanox Infiniband adapters:

  - Native infiniband mode
  - Native ethernet mode

It would be useful to clarify that in Infiniband mode, IP can be encapsulated and sent using %%IPoIB%%.

====== Cables & connectors ======

We should add info and photo's here for the different cables & connector types

  * Explain passive vs active
  * Copper & optical cable

  * CX4 connectors  ← JustinClift has some of these.  Can take photos
  * SFP
  * QSFP connectors  ← JustinClift has some of these.  Can take photos

====== Maximum cable length ======

It would also be useful to explain the maximum length of cables, according to type, connector, data rate (and whatever else).