NIC driver set up and clarify Infinite Band (IB) device in Linux

After we get two IB devices (in this example, we get 2 pcs with Broadcom ethernet controller BCM57416 with RDMA feature) and we would like to do the RDMA performance test, we need to prepare the RDMA environment.

First, we should prepare the RDMA driver. For Broadcom RDMA driver, we could download from here: https://www.broadcom.com/products/ethernet-connectivity/network-adapters/p210tp

1) Install the package required and reboot

sudo apt-get update
sudo apt install -y automake autoconf libtool libibverbs-dev ibverbs-utils infiniband-diags perftest

2) Install the Linux L2 and RoCE drivers (in "Linux_Driver" folder)

Unzip the package downloaded from Broadcom website and then follow the step to install driver.

# Download the Driver and untar first and then go to the below driver folder to compile the driver
cd Downloads/
tar zxvf bcm_229.1.123.0b.tar.gz
cd bcm_229.1.123.0b/Linux/Linux_Driver
tar zxvf netxtreme-bnxt_en-1.10.3-229.0.139.0.tar

After untar the package, we should change parameter value "BNXT_RE_ABI_VERSION" from 6 to 1 in two different file since the driver not support the value "6". (bcm_229.1.123.0b is the Linux driver package)

File 1: bcm_229.1.123.0b/Linux/Linux_Driver/netxtreme-bnxt_en-1.10.3-229.0.139.0/bnxt_re-229.0.139.0/bnxt_re-abi.h

File 2: bcm_229.1.123.0b/Linux/KMP-RoCE-Lib/libbnxt_re-229.0.139.0/src/abi.h

cd netxtreme-bnxt_en-1.10.3-229.0.139.0.tar
sudo make
sudo make install

# if you could not use "make", please follow the installation
sudo apt-get update
sudo apt-get install make


# if you are not using gcc compiler, this installation might encounter error
# please follow below command to install gcc compiler:
sudo apt update 
sudo apt install gcc

# and then you can go back to the installation ("make", "make install")

3) Install the Linux RoCE User Library (in "RoCE_Lib" folder)

# $REL_DIR is the untar Linux driver package download from broadcom web
cd $REL_DIR/Linux/KMP-RoCE-Lib
tar xf libbnxt_re-x.y.z.tar.gz
cd libbnxt_re-x.y.z
sudo sh autogen.sh
sudo ./configure --sysconfdir=/etc/
sudo make
sudo make install all
sudo sh -c "echo /usr/local/lib >> /etc/ld.so.conf"
sudo ldconfig
sudo cp bnxt_re.driver /etc/libibverbs.d/

4) Reboot system and run below command to install RoCE device

sudo modprobe -r bnxt_en //L2 driver
sudo modprobe bnxt_re //RDMA driver

Run below command to check if the driver is set up finished

lsmod|grep bnxt

You should find "bnxt_re", "bnxt_en", "ib_core", "ib_uverbs"

5) Check if we could clarify IB device with below command

ibv_devices //check the device name

ibv_devinfo //check device info

you could check link status and device info

As for more information, please check here:

https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/adapters.html