Versions Compared

Old Version 9

changes.mady.by.user Chao Han

Saved on

compared with

New Version 10

changes.mady.by.user Chao Han

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

yum install pciutils-devel libusbx-devel fuse-devel

rpm -Uvh rshim-<version-number>.x86_64.rpm -force

       Note: before Before start the driver, guarantee that old version of kernel modules are unloaded if there has.

rmmod rshim_net
rmmod rshim_pcie
rmmod rshim_usb
rmmod rshim

systemctl start rshim


  • Create a file named ifcfg-tmfifo_net0 under /etc/sysconfig/network-scripts/ directory, with the following content:

...

Console over USB-Serial / Serial Port properties: Rate 115200, HW Flow Control OFF.

minicom -D /dev/ttyUSB0

Console over USB (rshim_usb) / Serial Port properties: Rate 115200, HW Flow Control OFF

...

ssh root@192.168.100.2

Note
      : SmartNIC OS is configured to obtain internet access via its hosting server.

...

minicom -D /dev/ttyUSB0

  

Note: This is not a power cycle of the SmartNIC.


 How-to load BlueField2 SmartNIC OS image from the hosting server:

...

x86_host #mst start
x86_host #mlxconfig -d /dev/mst/mt41686_pciconf0 s INTERNAL_CPU_MODEL=1


 RebootReboot.


Verify:

x86_host#mst start
x86_host#mlxconfig -d /dev/mst/mt41686_pciconf0 q | grep -i model

...

Issue the following commands to enable ECPF parameters (applied per port):

Note
: The ECPF parameters will be available for setting only after completing the previous steps.

...

 
On Arm: following devices should be listed:

link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev pf0hpf
link mlx5_0/2 state ACTIVE physical_state LINK_UP netdev pf0sf0
link mlx5_0/19 state ACTIVE physical_state LINK_UP netdev p0
link mlx5_1/1 state ACTIVE physical_state LINK_UP netdev pf1hpf
link mlx5_1/2 state ACTIVE physical_state LINK_UP netdev pf1sf0
link mlx5_1/19 state DOWN physical_state DISABLED netdev p1
link mlx5_2/1 state ACTIVE physical_state LINK_UP netdev p0m0
link mlx5_3/1 state ACTIVE physical_state LINK_UP netdev p1m0


Issue the following commands on the Arm to see the correlation between the PCI device and its representors:  

mst start
mst status -v
 
 
PCI devices:
------------
DEVICE_TYPE MST PCI RDMA NET NUMA
BlueField2(rev:0) NA 03:00.1 mlx5_1 net-p1 -1
BlueField2(rev:0) NA 03:00.0 mlx5_0 net-p0 -1


Note
: The pf0hpf / pf1hpf are the PF representors facing the x86 host.

...

Testing Traffic in EMBEDDED Mode using DPDK

In Embedded mode, traffic from the x86 server hosting the SmartNIC to the remote x86 server hosting the ConnectX-6 Dx is going via the SmartNIC Arm.

DPDK application is used to forward the traffic and bypass Arm OS kernel. Follow the configuration steps below to test it.

...

Identify the SmartNIC PCI device ID:

lspci | grep -i mellanox

Start testPMD application  while probing the representors  (pci device is 03:00.0):


Configure OVS-DPDK:

ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true

ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem=4096

ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-extra="-w 03:00.0,representor=[0,65535]"

ovs-vsctl add-br dpdkbr0 -- set bridge dpdkbr0 datapath_type=netdev 

-- set bridge dpdkbr0 fail-mode=standalone

ovs-vsctl add-port dpdkbr0 p0 -- set Interface p0 type=dpdk options:dpdk-devargs=0000:03:00.0

ovs-vsctl add-port dpdkbr0 pf0hpf -- set Interface pf0hpf type=dpdk options:dpdk-devargs="0000:

cd /usr/bin/
./testpmd -w

03:00.0,representor=[

0,

65535]

 – -i

"

systemctl restart openvswitch


testpmd> start


Set IPs on both x86 host and ConnectX-6 Dx host and test connectivity with ping (it will be forwarded by the Arm DPDK)

...

On the x86 host, start iperf client:

iperf -c 172.16.0.2 -P12

While the test is running, you can check traffic statistics on Arm testPMD:

testpmd> show port stats all


Bare Metal Deployment Guide

...