Table of Contents
Introduction
The R3 R4 release will evaluation the throughput and packet forwarding performance of the Mellanox BlueField SmartNIC cardevaluate the conntrack performance of Open vSwitch (OVS).
A DPDK based Open vSwitch (OVS-DPDK) is used as the virtual switch, and the network traffic is virtualized with the VXLAN encapsulation.
Currently, the community version OVS-DPDK is considered experimental and not mature enough, it only supports "partial offload" which cannot
utilize the full performance advantage of Mellanox NICs. Thus the OVS-DPDK we used is a fork of the community Open vSwitch. We develop
our own offload code which enables the full hardware offload with DPDK rte_flow APIs.
We have plans to open-source this OVS-DPDK. More details will be provided in future documentation.
We will compare the performance of an optimized software based OVS with the performance of the OVS running on a MLNX SmartNIC with hardware offload enabled.
Akraino Test Group Information
Overall Test Architecture
SmartNIC deployed Architecture
We reuse the test architecture for smart NIC in R3 release. The below description is the same with the R3 release test documents.
To deploy the Test architecture for smartNIC, we use a private Jenkins and an Intel a server equipped with a BlueField v1 v2 SmartNIC.
We use Ansible to automatically setup the filesystem image and install the OVS-DPDK in the SmartNICs.
...
according to the ansible-playbook.
Image | download link |
---|---|
BlueField-23.5.10.1121311563.tar.xz | https http://www.mellanox.com/downloads/ products/software/bluefieldBlueField/BlueField-3.5.0.11563/BlueField-3.5.0.11563.tar.xz |
core-image-full-dev-BlueField-23.5.10.1121311563.2.7.4.tar.xz | |
mft-4.16.0-105-x86_64-rpm.tgz | https://www.mellanox.com/productsdownloads/softwareMFT/bluefieldmft-4.1416.0-105-x86_64-debrpm.tgz |
MLNX_OFED_LINUX-5.2-1.0.4.0-rhel7.6-x86_64.tgz | http://www.mellanox.com/page/mlnx_ofed_eula?mtag=linux_sw_drivers&mrequest=downloads&mtype=ofed&mver=MLNX_OFED-5.2-1.0.4.0&mname=MLNX_OFED_LINUX-5.2-1.80.4.0-debian8rhel7.116-x86_64.tgz |
Test Bed
OVS-DPDK Test Architecture
OVS-DPDK on BlueField Test Architecture
The testbed setup is shown in the above diagram. DUT stands for Device Under Test
...
Type | Description |
---|---|
SmartNICs | BlueField v1v2, 25Gbps |
DPDK | version 1920.11 |
vSwitch | OVS-DPDK 2.12 with VXLAN DECAP/ENCAP offload enabled. |
Code Block |
---|
root@bluefield:/home/ovs-dpdk# ovs-vsctl show 2dccd148-526c-44a5-9351-67b04c5e2da4 Bridge br-phy0 Bridge br-intfail_mode: standalone datapath_type: netdev Port vxlanbr-vtpphy0 Interface vxlanbr-vtpphy0 type: vxlaninternal Port p0 options: {dst_port="4789", key=flow, local_ip="192.168.1.1", remote_ip=flow, tos=inherit} Interface p0 Port br-int type: dpdk Interface br-int options: {dpdk-devargs="0000:03:00.0"} type: internalBridge br-int0 Port pf1hpffail_mode: standalone datapath_type: netdev Interface pf1hpf Port br-int0 type: dpdkInterface br-int0 options: {dpdk-devargs="class=eth,mac=ae:d8:8a:c5:22:fb"}type: internal Bridge br-ex Port vxlan0 datapath_type: netdev Interface vxlan0 Port br-ex Interface br-extype: vxlan type: internaloptions: {flags="0", key="101", local_ip="1.1.1.1", remote_ip="1.1.1.3"} Port p1pf0vf0 Interface p1pf0vf0 type: dpdk options: {dpdk-devargs="class=eth,mac=980000:03:9b:af:7b:0b"}00.0,representor=[0]"} ovs_version: "2.14.1" |
Code Block |
---|
root@bluefieldroot:/home/ovs-dpdk# ovs-vsctl --format=csv --data=bare --no-headings --column=other_config list open_vswitch "dpdk-extra=-w [PCIE] -l 70 dpdk-init=true dpdk-socket-mem=2048,2048 emc-insert-inv-prob=0 n-handler-threads=1 n-revalidator-threads=4 neigh-notifier-enable=true pmd-cpu-mask=0xc00000000000c00000 pmd-pause=false pmd-rxq-assign=roundrobin smc-enable=true tx-flush-interval=0 userspace-tso-enable=true" |
Code Block |
---|
root:/home/ovs-dpdk# ovs-vsctl --format=csv --data=bare --no-headings --column=other_config list open_vswitch
dpdk-extra="-w 0000:03:00.0,representor=[0-1],dv_xmeta_en=1,sys_mem_en=1", dpdk-init="true", dpdk-socket-mem="4096", hw-offload="true", max-idle="120000" |
Traffic Generator
We will use DPDK pktgen as the Traffic Generator.
CT Ruleset
Code Block | ||
---|---|---|
| ||
Br-ex rules
table=0 priority=300 in_port=eth2,tcp,tp_dst=32768/0x8000 actions=output:br-int-patch
table=0 priority=300 in_port=eth2,udp,tp_dst=32768/0x8000 actions=output:br-int-patch
table=0 priority=200 in_port=eth2 actions=output:LOCAL
table=0 priority=200 in_port=LOCAL actions=eth2
table=0 priority=200 in_port=br-int-patch actions=eth2 |
Code Block | ||
---|---|---|
| ||
Br-int rules
table=0,arp,action=normal
table=1,priority=1,ip,ct_state=+trk+new,action=ct(commit),normal
table=0,ip,ct_state=-trk,action=ct(table=1)
table=1,priority=1,ip,ct_state=+trk+est,action=normal
|
Nginx configuration
Code Block | ||
---|---|---|
| ||
user nginx; worker_processes auto; error_log /var/log/nginx/error.log warn; pid_uuid : 2dccd148-526c-44a5-9351-67b04c5e2da4 bridges : [22334686-733a-445e-9130-a42009a3586e, 38af610d-01f7-497d-878b-c6b6a44abf6a] cur_cfg : 10 datapath_types : [netdev, system] datapaths : {} db_version/var/run/nginx.pid; events { worker_connections : [] dpdk_initialized2000000; } http { :include true dpdk_version /etc/nginx/mime.types; : "DPDK 19.11.0" external_ids default_type application/octet-stream; : {} iface_typeslog_format main '$remote_addr - : [dpdk, dpdkr, dpdkvhostuser, dpdkvhostuserclient, erspan, geneve, gre, internal, ip6erspan, ip6gre, lisp, patch, stt, system, tap, vxlan] manager_options$remote_user [$time_local] "$request" ' : [] next_cfg '$status $body_bytes_sent "$http_referer" ' : 10 other_config : {dpdk-extra="-w 03:00.1,representor=[0,65535] --legacy-mem ", dpdk-init="true", hw-offload="true"} ovs_version : [] ssl '"$http_user_agent" "$http_x_forwarded_for"'; access_log off; sendfile on; keepalive_timeout : []65; statistics include /etc/nginx/conf.d/*.conf; } |
Code Block | ||
---|---|---|
| ||
server { listen : {cpu="16", file_systems="/,13521220,3918464 /data,243823,2064 /boot,357176,61104", load_average="1.41,1.37,1.36", memory="16337652,5589928,1707640,0,0", process_ovs-vswitchd="5959388,256352,11515170,0,11484792,11484792,7", process_ovsdb-server="12620,6260,13190,0,68233832,68233832,6"} system_type : [] system_version : [] |
Traffic Generator
We will use DPDK pktgen as the Traffic Generator.
80 backlog=8192 reuseport;
server_name localhost;
location / {
return 200 "hello";
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}
|
WRK configuration
connection setup rates:
Code Block |
---|
./wrk -t 32 -c 640 -d30s http://10.0.1.127/ -H "Connection: Close" |
Performance Results
For the optimized software-based OVS, we tested on a 48C24q VM, running NGINX as a server, CT is enabled on OVS-DPDK.( upstream version / our version )
We also run 50 2c2q VMs as clients, generating HTTP GET traffic to evaluate the performance.
pps (not closing connection after each query) | pps (closing connection after each query) | connection initial rates (closing connection after each query) | QPS (not closing connection after each query) |
---|---|---|---|
1.66Mpps/2Mpps | 1.66Mpps/2Mpps | 140Kcps/200Kcps | 889Kqps/1.14Mqps |
For the OVS-DPDK running on SmartNIC with CT function enabled, we tested on 17C8G VM running testpmd (where 1 core for lcore and 16 cores for nb cores) as the traffic forwarder and reached the performance below.
Note: the result also depends on the traffic generator side.
Frame size | 114 bytes |
---|---|
Packets per second | ~23Mpps |
...
The test is to evaluate the performance of SmartNIC offloading.
Test API description
Thus we currently don't have any Test APIs provided.
...
25 cases are skipped due to the configuration.
Performance Tests
Single PF
...
Directly forwarding without the match
"in_port=vxlan-vtp, actions=output: pf1hpf"
...
Single PF, Single VF
...
Directly forwarding without the match
"in_port=vxlan-vtp, actions=output: pf1vf0"
...
match and forwarding
100 rules
"in_port=vxlan-vtp,ip,nw_dst=10.1.1.[1-100] actions=pf1vf0"
...
21.6Mpps
(100 megaflows
offloaded)
...
match and forwarding
1000 rules
"in_port=vxlan-vtp,ip,nw_dst=10.1.[1-10].[1-100] actions=pf1vf0"
...
1000 flows with nw_dst=10.1.[1-10].[1-100]
891 flows
(due to pktgen limit, only 891 flows are loaded)
...
23.3Mpps
(891 megaflows
offloaded)
...
Single PF, 4 VFs (only test with offloaded)
...
match and forwarding
1000 rules
"in_port=vxlan-vtp,ip,nw_dst=10.1.[1-10].[1-25] actions=pf1vf0"
"in_port=vxlan-vtp,ip,nw_dst=10.1.[1-10].[26-50] actions=pf1vf1"
"in_port=vxlan-vtp,ip,nw_dst=10.1.[1-10].[51-75] actions=pf1vf2"
"in_port=vxlan-vtp,ip,nw_dst=10.1.[1-10].[75-100] actions=pf1vf3"
...
1000 flows with nw_dst=10.1.[1-10].[1-100]
891 flows
(due to pktgen limit, only 891 flows are loaded)
...
23258808
...
Additional Testing
n/a
Bottlenecks/Errata
...