Differences

This shows you the differences between two versions of the page.

--- infiniband [2017/01/19 01:23] – EnjiCooper
+++ infiniband [2019/04/16 22:23] (current) – Update script for building IB modules, add script for upgrading JasonBacon
@@ Line 3: / Line 3: @@
 InfiniBand (IB) is a high-speed (10-300Gb/s) low-latency (140–2600 ns) switched-fabric interconnect, developed primarily for HPC, but as of now widely adopted where its properties are in demand.
-InfiniBand is supported on most major operating systems: Windows, Linux, (Open)Solaris, AIX, HP-UX, z/OS, among others. OpenFabrics Alliance (founded in 2004 as %%OpenIB%% Alliance) is developing a cross-platform IB/RDMA stack for Linux and Windows. The stack is released as OFED (OpenFabrics Enterprise Distribution) under both GPL and BSD licenses.
+InfiniBand is supported on most major operating systems: Windows, Linux, (Open)Solaris, AIX, HP-UX, z/OS, among others. %%OpenFabrics%% Alliance (founded in 2004 as %%OpenIB%% Alliance) is developing a cross-platform IB/RDMA stack for Linux and Windows. The stack is released as OFED (%%OpenFabrics%% Enterprise Distribution) under both GPL and BSD licenses.
 ====== Infiniband in FreeBSD ======
@@ Line 9: / Line 9: @@
 The OFED stack was merged in March 2011 ([[http://jeffr-tech.livejournal.com/23467.html]]) and is available in %%FreeBSD%% 9.0 and later.
-Proprietary IB stacks are known to exist outside of %%FreeBSD%% proper. In particular, three major storage appliance vendors -- EMC/Isilon, NetApp, and Panasas -- ship products based on %%FreeBSD%% and use InfiniBand for high-speed interconnects.
+Proprietary IB stacks are known to exist outside of %%FreeBSD%% proper. In particular, three major storage appliance vendors -- [[http://www.emc.com/en-us/storage/isilon/index.htm|Dell EMC Isilon]], %%NetApp%%, and Panasas -- ship products based on %%FreeBSD%% and use InfiniBand for high-speed interconnects.
 ====== Infiniband hardware support by release ======
@@ Line 18: / Line 18: @@
 | %%ConnectX%%-2 | mlx4 | Yes | Yes | Yes |  |
 | %%ConnectX%%-3/Pro | mlx4 | Yes | Yes | Yes |  |
-| %%ConnectX%%-4 (Eth) | mlx5 | No | Yes | Yes |  |
+| %%ConnectX%%-4 (Eth) | mlx5 | Yes | Yes | Yes | Available in 9.4+/10.2+(?)/11.0+ |
-| %%ConnectX%%-4 (IB) | mlx5 | No | Not Yet | Not Yet | In development |
+| %%ConnectX%%-4 (IB) | mlx5 | Yes | Yes | Yes | Available in 9.4+/10.2+(?)/11.0+ |
-| Connect-IB | mlx5 | No | Not Yet | Not Yet | In development |
+| Connect-IB | mlx5 | Yes | Yes | Yes | Available in 9.4+/10.3+(?)/11.0+ |
 ====== Building OFED in FreeBSD ======
@@ Line 45: / Line 45: @@
   * Run buildkernel/installkernel as per the usual process.
+Note: If your /usr/src tree is newer than your kernel, you must do a buildkernel/installkernel along with buildworld/installworld.  See the %%FreeBSD%% handbook for full details.
+===== Userland =====
+* This is no longer necessary as OFED support is on by default now, but is still documented here for those who need to do a buildworld anyway.
+  - Add ''%%WITH_OFED='yes'%%'' to /etc/src.conf.
+  - Run buildworld/installworld per usual process.
+  - Set the "sys.device.mlx4_coreX.mlx4_portY" sysctl to either "eth" or "ib" depending on how you want the device ports to be configured.
 ===== Kernel Modules =====
-As an alternative to building a new kernel, recent releases provide IB drivers as kernel modules.
+It is now possible to enable and upgrade Infiniband support on %%FreeBSD%% in a few minutes.
-Note: If your /usr/src tree is newer than your kernel, you must do a buildkernel/installkernel along with buildworld/installworld.  See the %%FreeBSD%% handbook for full details.
+As an alternative to building a new kernel, recent releases provide IB drivers as kernel modules, and userland now supports Infiniband by default, so there is no need to perform a buildworld.
 The script below will build, install, and load the appropriate drivers:
@@ Line 57: / Line 67: @@
  #!/bin/sh -e
-cd /usr/src/sys/modules
+usage()
-for module in mlx4 ibcore mlx4ib ipoib; do
+{
+    printf "Usage: $0 4|5 datagram|connected\n"
+    printf "* Use 4 for ConnectX 3 and earlier, 5 for later HCAs.\n"
+    printf "* Specify connected if you want to use connected mode ipoib.\n"
+    exit 1
+}
+if [ $# != 2 ]; then
+    usage
+fi
+driver_version=$1
+ipoib_mode=$2
+case $ipoib_mode in
+datagram)
+    ;;
+connected)
+    export CFLAGS="-DIPOIB_CM"
+    ;;
+*)
+    usage
+    ;;
+esac
+cd /usr/src
+make clean  # Rebuild all objs in case src was updated
+cd sys/modules
+for module in mlx$driver_version ibcore mlx${driver_version}ib ipoib; do
+    rm -f /boot/modules/$module.ko
     cd $module
-    make
+    make clean
+    make DEBUG_FLAGS="-DVIMAGE=1"
     make install
+    # Minimize damage if kldload triggers a panic.  Unlikely, but has happened
+    # with experimental drivers.
     sync
     kldload $module || true
-    printf "${module}_load=\"YES\"\n" >> /boot/loader.conf
+    auto-append-line ${module}'_load="YES"' /boot/loader.conf $0
     cd ..
 done
 kldstat
+cat << EOM
+To disable Infiniband support, remove or comment out the following device
+loads in /boot/loader.conf:
+mlx$driver_version ibcore mlx${driver_version}ib ipoib
+EOM
 </code>
-===== Userland =====
+Following updates to /usr/src, the IB kernel modules can be upgraded using the script below:
-  - Add ''%%WITH_OFED='yes'%%'' to /etc/src.conf.
+<code>
-  - Run buildworld/installworld per usual process.
+ #!/bin/sh -e
-  - Set the "sys.device.mlx4_coreX.mlx4_portY" sysctl to either "eth" or "ib" depending on how you want the device ports to be configured.
+usage()
+{
+    printf "Usage: $0 4|5 datagram|connected\n"
+    printf "* Use 4 for ConnectX 3 and earlier, 5 for later HCAs.\n"
+    printf "* Specify connected if you want to use connected mode ipoib.\n"
+    exit 1
+}
+if [ $# != 2 ]; then
+    usage
+fi
+driver_version=$1
+ipoib_mode=$2
+for interface in $(ifconfig | awk -F : '$1 ~ "^ib" { print $1 }'); do
+    ifconfig $interface down
+done
+kldunload ipoib mlx${driver_version}ib ibcore mlx$driver_version || true
+auto-build-infiniband-modules $1 $2
+for interface in $(ifconfig | awk -F : '$1 ~ "^ib" { print $1 }'); do
+    ifconfig $interface up
+done
+</code>
+Both of these scripts will be included in sysutils/autoadmin in the near future.
 ===== OpenSM =====