Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

N/A

Hardware Requirements

N/A

Software Prerequisites

  • MLNX_OFED
  • Note

    • Use automatic FM update during MLNX_OFED installation ONLY if the current BlueField FW version is 18.24.1000 (GA) or above. In case its older, manually burn the FW after MLNX_OFED installation.
    • In case you plan to run DPDK on the host itself, install MLNX_OFED with "--dpdk --upstream-libs" flags
    • The "bluefield" flag is not required in some of the MLNX_OFED versions and OS releases

Database Prerequisites

...

N/A

Other Installation Requirements

...

N/A

Installation High-Level Overview      

SmartNIC Server Preparations

  • Install useful packages:

    yum install -y screen minicom net-tools tcpdump usbutils net-utils sshpass iptables-services fio epel-release iperf wget git unzip perl pciutils lsof tcl libxml2-python tk unbound gtk2 atk cairo gcc-gfortran tcsh nvme-cli

      

  • Install MLNX_OFED with the mentioned flags below.

    tar -xzvf MLNX_OFED_LINUX-4.4-2.5.11.0-rhel7.5-x86_64.tgz
    cd MLNX_OFED_LINUX-4.4-2.5.11.0-rhel7.5-x86_64
    ./mlnxofedinstall --with-nvmf --bluefield --without-fw-update


      Note

  • Use automatic FM update during MLNX_OFED installation ONLY if the current BlueField FW version is 18.24.1000 (GA) or above. In case its older, manually burn the FW after MLNX_OFED installation.
  • In case you plan to run DPDK on the host itself, install MLNX_OFED with "--dpdk --upstream-libs" flags
  • The "bluefield" flag is not required in some of the MLNX_OFED versions and OS releases

     Enable SR-IOV on FW (Optional):

    mlxconfig -y -d /dev/mst/mt41682_pciconf0 s SRIOV_EN=1
    mlxconfig -y -d /dev/mst/mt41682_pciconf0 s NUM_OF_VFS=8


  • Uncomment the following line in /etc/modprobe.d/rshim.conf to set the default RShim access method to the SmartNIC:

     options rshim backend_driver=rshim_usb


    Note

    1. When this line is commented, the first available driver will be used.
    2. When the mini USB cable is not used, the driver should be rshim_pcie


  • Create a file named ifcfg-tmfifo_net0 under /etc/sysconfig/network-scripts/ directory, with the following content:

NAME="tmfifo_net0"
HWADDR="00:1a:ca:ff:ff:02"
ONBOOT=yes
NETBOOT=yes
IPV6INIT=yes
BOOTPROTO=static
TYPE=Ethernet
IPADDR=192.168.100.1
NETMASK=255.255.255.0

  

  • ifcfg-tmfifo_net0 is the network interface used for IP communication between the x86 host and the SmartNIC with rshim_net driver.

  • Enable ip forwarding and set a NAT rule to allow the SmartNIC OS to communicate with the outer world via the host management interface:

    sysctl -w net.ipv4.ip_forward=1
    systemctl enable iptables
    systemctl start iptables
    iptables -F
    iptables -t nat -A POSTROUTING -o <host_mgmt_interfac> -j MASQUERADE
    service iptables save


  • Disable Firewall and Network Manager services

    systemctl disable firewalld.service
    systemctl disable NetworkManager


  • Reboot the server. Once server is up again, run some configuration sanity tests:

    To verify BlueField FW version:

    mlxfwmanager --query


     To verify RShim modules are loaded:

    lsmod | grep -i rshim


    To identify the backend active RShim driver:

     cat /dev/rshim0/misc


    To verify the NAT rule for SmartNIC communication with the outer world:

    iptables -L -n -t nat



Place the bfb image on the host and load it on the BlueField SmartNIC using the following command:

...