...
rpm -Uvh rshim-<version-number>.x86_64.rpm -force |
Note: before Before start the driver, guarantee that old version of kernel modules are unloaded if there has.
rmmod rshim_net systemctl start rshim |
- Create a file named ifcfg-tmfifo_net0 under /etc/sysconfig/network-scripts/ directory, with the following content:
...
- ifcfg-tmfifo_net0 is the network interface used for IP communication between the x86 host and the SmartNIC with rshim driver.
- Enable ip forwarding and set a NAT rule to allow the SmartNIC OS to communicate with the outer world via the host management interface:
...
sysctl -w net.ipv4.ip_forward=1
systemctl enable iptables
systemctl start iptables
iptables -F
iptables -t nat -A POSTROUTING -o <host_mgmt_interfac> -j MASQUERADE
service iptables save
- The oob_net0 interface on BlueField2 ARM can be used for management interface and a connection to outside. The configuration for it can be as below:
NAME="oob_net0" |
- Disable Firewall and Network Manager Disable Firewall and Network Manager services
|
...
Console over USB-Serial / Serial Port properties: Rate 115200, HW Flow Control OFF.
|
Console over USB (rshim_usb) / Serial Port properties: Rate 115200, HW Flow Control OFF
...
|
Note
: SmartNIC OS is configured to obtain internet access via its hosting server.
...
|
Note: This is not a power cycle of the SmartNIC.
How-to load BlueField2 SmartNIC OS image from the hosting server:
...
|
RebootReboot.
Verify:
|
...
Issue the following commands to enable ECPF parameters (applied per port):
Note
: The ECPF parameters will be available for setting only after completing the previous steps.
...
On Arm: following devices should be listed:
|
Issue the following commands on the Arm to see the correlation between the PCI device and its representors:
|
Note
: The pf0hpf / pf1hpf are the PF representors facing the x86 host.
...
Testing Traffic in EMBEDDED Mode using DPDK
In Embedded mode, traffic from the x86 server hosting the SmartNIC to the remote x86 server hosting the ConnectX-6 Dx is going via the SmartNIC Arm.
DPDK application is used to forward the traffic and bypass Arm OS kernel. Follow the configuration steps below to test it.
...
Identify the SmartNIC PCI device ID:
|
Start testPMD application while probing the representors (pci device is 03:00.0):
Configure OVS-DPDK:
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem=4096 ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-extra="-w 03:00.0,representor=[0,65535]" ovs-vsctl add-br dpdkbr0 -- set bridge dpdkbr0 datapath_type=netdev -- set bridge dpdkbr0 fail-mode=standalone ovs-vsctl add-port dpdkbr0 p0 -- set Interface p0 type=dpdk options:dpdk-devargs=0000:03:00.0 ovs-vsctl add-port dpdkbr0 pf0hpf -- set Interface pf0hpf type=dpdk options:dpdk-devargs="0000: |
cd /usr/bin/
./testpmd -w
03:00.0,representor=[ |
65535] |
" systemctl restart openvswitch |
testpmd> start
Set IPs on both x86 host and ConnectX-6 Dx host and test connectivity with ping (it will be forwarded by the Arm DPDK)
...
On the x86 host, start iperf client:
|
While the test is running, you can check traffic statistics on Arm testPMD:
|
Bare Metal Deployment Guide
...