...

Introduction

BlueField's Ethernet SmartNIC solutions combine Arm® processing power with advanced network offloads to accelerate a multitude of security, networking and storage applications.
Based on the BlueField system-on-chip (SoC) programmable network engine, BlueField SmartNICs integrate Mellanox ConnectX®-5 network controller with Arm multicore processors and cryptology options to deliver an innovative and high-performance programmable platform.Through this article you will be able to quickly set up a simple test lab which will allow you to experiment and explore BlueField SmartNIC capabilities. The setup includes a bare metal server hosting BlueField 25GbE SmartNIC connected back-to-back to another bare metal server with ConnectX-5 NIC.

License

...

N/A

How to use this document

                     NAN/A

Deployment Architecture

The following diagram shows the physical configuration between the BlueField SmartNIC and the ConnectX-5 NIC. 

Image Modified

Pre-Installation Requirements

Minimum Hardware Requirements

1. Insert ConnectX-5 NIC into Server A
2. Insert BlueField SmartNIC into Server B
3. Connect the SmartNIC UART cable from SmartNIC to its hosting Server B USB port #1
4. Connect Mini USB cable from SmartNIC to its hosting Server B USB port #2
5. Insert QSA28 Adapter into Server B 100GB port
6. Connect one of the SmartNIC ports to the ConnectX-5 using 25GB cable
7. Connect both servers to a management LAN

Recommended Hardware Requirements

NA

• Schema scriptsNA

• Jump Host RequirementsNA

• Network RequirementsNA

• Bare Metal Node RequirementsNA

• Execution Requirements (Bare Metal Only)NA

• • Hardware RequirementsNA

  • Software Prerequisites

    • MLNX_OFED

    • Note

      • Use automatic FM update during MLNX_OFED installation ONLY if the current BlueField FW version is 18.24.1000 (GA) or above. In case its older, manually burn the FW after MLNX_OFED installation.
      • In case you plan to run DPDK on the host itself, install MLNX_OFED with "--dpdk --upstream-libs" flags
      • The "bluefield" flag is not required in some of the MLNX_OFED versions and OS releases

  • Database PrerequisitesNA

  • Other Installation RequirementsNA

  •    Installation High-Level Overview      

    • SmartNIC Server Preparations

      • Install useful packages:

        yum install -y screen minicom net-tools tcpdump usbutils net-utils sshpass iptables-services fio epel-release iperf wget git unzip perl pciutils lsof tcl libxml2-python tk unbound gtk2 atk cairo gcc-gfortran tcsh nvme-cli

          

      • Install MLNX_OFED with the mentioned flags below.

        tar -xzvf MLNX_OFED_LINUX-4.4-2.5.11.0-rhel7.5-x86_64.tgz cd MLNX_OFED_LINUX-4.4-2.5.11.0-rhel7.5-x86_64 ./mlnxofedinstall --with-nvmf --bluefield --without-fw-update

        
        

            Note

      • Use automatic FM update during MLNX_OFED installation ONLY if the current BlueField FW version is 18.24.1000 (GA) or above. In case its older, manually burn the FW after MLNX_OFED installation.
      • In case you plan to run DPDK on the host itself, install MLNX_OFED with "--dpdk --upstream-libs" flags
      • The "bluefield" flag is not required in some of the MLNX_OFED versions and OS releases

      • 
        

         Enable SR-IOV on FW (Optional):

        mlxconfig -y -d /dev/mst/mt41682_pciconf0 s SRIOV_EN=1 mlxconfig -y -d /dev/mst/mt41682_pciconf0 s NUM_OF_VFS=8

        
        

      • Uncomment the following line in /etc/modprobe.d/rshim.conf to set the default RShim access method to the SmartNIC:

        options rshim backend_driver=rshim_usb

        
        

        Note

        1. When this line is commented, the first available driver will be used.
        2. When the mini USB cable is not used, the driver should be rshim_pcie

        
        

      • Create a file named ifcfg-tmfifo_net0 under /etc/sysconfig/network-scripts/ directory, with the following content:

        NAME="tmfifo_net0" HWADDR="00:1a:ca:ff:ff:02" ONBOOT=yes NETBOOT=yes IPV6INIT=yes BOOTPROTO=static TYPE=Ethernet IPADDR=192.168.100.1 NETMASK=255.255.255.0

          

      • ifcfg-tmfifo_net0 is the network interface used for IP communication between the x86 host and the SmartNIC with rshim_net driver.

      • Enable ip forwarding and set a NAT rule to allow the SmartNIC OS to communicate with the outer world via the host management interface:

        sysctl -w net.ipv4.ip_forward=1 systemctl enable iptables systemctl start iptables iptables -F iptables -t nat -A POSTROUTING -o <host_mgmt_interfac> -j MASQUERADE service iptables save

        
        

      • Disable Firewall and Network Manager services

        systemctl disable firewalld.service systemctl disable NetworkManager

        
        

      • Reboot the server. Once server is up again, run some configuration sanity tests:
        

        To verify BlueField FW version:

        mlxfwmanager --query

        
        

         To verify RShim modules are loaded:

        lsmod | grep -i rshim

        
        

        To identify the backend active RShim driver:

        cat /dev/rshim0/misc

        
        

        To verify the NAT rule for SmartNIC communication with the outer world:

        iptables -L -n -t nat

        
        

...