Introduction

The R3 R4 release will evaluation the throughput and packet forwarding performance of the Mellanox BlueField SmartNIC cardevaluate the conntrack performance of Open vSwitch (OVS).

A DPDK based Open vSwitch (OVS-DPDK) is used as the virtual switch, and the network traffic is virtualized with the VXLAN encapsulation.

Currently, the community version OVS-DPDK is considered experimental and not mature enough, it only supports "partial offload" which cannot

utilize the full performance advantage of Mellanox NICs. Thus the OVS-DPDK we used is a fork of the community Open vSwitch. We develop

our own offload code which enables the full hardware offload with DPDK rte_flow APIs.

We have plans to open-source this OVS-DPDK. More details will be provided in future documentation.

We will compare the performance of an optimized software based OVS with the performance of the OVS running on a MLNX SmartNIC with hardware offload enabled.

Akraino Test Group Information

Overall Test Architecture

SmartNIC deployed Architecture

We reuse the test architecture for smart NIC in R3 release. The below description is the same with the R3 release test documents.

To deploy the Test architecture for smartNIC, we use a private Jenkins and an Intel a server equipped with a BlueField v1 v2 SmartNIC.

We use Ansible to automatically setup the filesystem image and install the OVS-DPDK in the SmartNICs.

Image RemovedImage Added

The File Server is a simple Nginx based web server where stores the BF drivers, FS image.

...

according to the ansible-playbook.

Image	download link
BlueField-23.5.10.1121311563.tar.xz	https http://www.mellanox.com/downloads/ products/software/bluefield BlueField/BlueField-3.5.0.11563/BlueField-3.5.0.11563.tar.xz
core-image-full-dev-BlueField-23.5.10.1121311563.2.7.4.tar.xz	http://www.mellanox.com/downloads/BlueField/BlueField-3.5.0.11563/core-image-full-BlueField-3.5.0.11563.2.7.4.tar.xz
mft-4.16.0-105-x86_64-rpm.tgz	https://www.mellanox.com/productsdownloads/softwareMFT/bluefieldmft-4.1416.0-105-x86_64-debrpm.tgz
MLNX_OFED_LINUX-5.2-1.0.4.0-rhel7.6-x86_64.tgz	http://www.mellanox.com/page/mlnx_ofed_eula?mtag=linux_sw_drivers&mrequest=downloads&mtype=ofed&mver=MLNX_OFED-5.2-1.0.4.0&mname=MLNX_OFED_LINUX-5.2-1.0.84.0-debian8rhel7.116-x86_64.tgz

Test Bed

OVS-DPDK Test Architecture

Image Added

OVS-DPDK on BlueField Test Architecture

Image AddedImage Removed

The testbed setup is shown in the above diagram. DUT stands for Device Under Test

Test Framework

The software used and the OVS-DPDK setup is shown below.

...

Type

Description

SmartNICs

BlueField v1v2, 25Gbps

DPDK

version 1920.11

vSwitch

OVS-DPDK 2.12 with VXLAN DECAP/ENCAP offload enabled.

https://github.com/bytedance/ovs-dpdk

Code Block

root@bluefield:/home/ovs-dpdk# ovs-vsctl show
2dccd148-526c-44a5-9351-67b04c5e2da4
	Bridge br-phy0
        Bridge br-intfail_mode: standalone
        datapath_type: netdev
        Port vxlanbr-vtpphy0
            Interface vxlanbr-vtpphy0
                type: vxlaninternal
        Port p0
       options: {dst_port="4789", key=flow, local_ip="192.168.1.1", remote_ip=flow, tos=inherit}     Interface p0
        Port br-int
       type: dpdk
    Interface br-int
           options: {dpdk-devargs="0000:03:00.0"}
    type: internalBridge br-int0
        Port pf1hpffail_mode: standalone
        datapath_type: netdev
    Interface pf1hpf
   Port br-int0
            type: dpdkInterface br-int0
                options: {dpdk-devargs="class=eth,mac=ae:d8:8a:c5:22:fb"}type: internal
    Bridge br-ex    Port vxlan0
        datapath_type: netdev
    Interface vxlan0
   Port br-ex
            Interface br-extype: vxlan
                type: internaloptions: {flags="0", key="101", local_ip="1.1.1.1", remote_ip="1.1.1.3"}
        Port p1pf0vf0
            Interface p1pf0vf0
                type: dpdk
                options: {dpdk-devargs="class=eth,mac=980000:03:9b:af:7b:0b"}00.0,representor=[0]"}
    ovs_version: "2.14.1"

Code Block

root@bluefieldroot:/home/ovs-dpdk# ovs-vsctl --format=csv --data=bare --no-headings --column=other_config list open_vswitch
"dpdk-extra=-w [PCIE]  -l 70 dpdk-init=true dpdk-socket-mem=2048,2048 emc-insert-inv-prob=0 n-handler-threads=1 n-revalidator-threads=4 neigh-notifier-enable=true pmd-cpu-mask=0xc00000000000c00000 pmd-pause=false pmd-rxq-assign=roundrobin smc-enable=true tx-flush-interval=0 userspace-tso-enable=true"

Code Block

root:/home/ovs-dpdk# ovs-vsctl --format=csv --data=bare --no-headings --column=other_config list open_vswitch
dpdk-extra="-w 0000:03:00.0,representor=[0-1],dv_xmeta_en=1,sys_mem_en=1", dpdk-init="true", dpdk-socket-mem="4096", hw-offload="true", max-idle="120000"

Traffic Generator

We will use DPDK pktgen as the Traffic Generator.

CT Ruleset

Code Block

language	bash

Br-ex rules
table=0 priority=300 in_port=eth2,tcp,tp_dst=32768/0x8000 actions=output:br-int-patch
table=0 priority=300 in_port=eth2,udp,tp_dst=32768/0x8000 actions=output:br-int-patch
table=0 priority=200 in_port=eth2 actions=output:LOCAL
table=0 priority=200 in_port=LOCAL actions=eth2
table=0 priority=200 in_port=br-int-patch actions=eth2

Code Block

language	bash

Br-int rules
table=0,arp,action=normal
table=1,priority=1,ip,ct_state=+trk+new,action=ct(commit),normal
table=0,ip,ct_state=-trk,action=ct(table=1)
table=1,priority=1,ip,ct_state=+trk+est,action=normal

Nginx configuration

Code Block

language	bash

user  nginx;
worker_processes  auto;
error_log  /var/log/nginx/error.log warn;
pid_uuid               : 2dccd148-526c-44a5-9351-67b04c5e2da4
bridges             : [22334686-733a-445e-9130-a42009a3586e, 38af610d-01f7-497d-878b-c6b6a44abf6a]
cur_cfg             : 10
datapath_types      : [netdev, system]
datapaths           : {}
db_version/var/run/nginx.pid;
events {
    worker_connections      : []
dpdk_initialized2000000;
}
http {
    :include true
dpdk_version      /etc/nginx/mime.types;
  : "DPDK 19.11.0"
external_ids   default_type  application/octet-stream;
      : {}
iface_typeslog_format  main  '$remote_addr -    : [dpdk, dpdkr, dpdkvhostuser, dpdkvhostuserclient, erspan, geneve, gre, internal, ip6erspan, ip6gre, lisp, patch, stt, system, tap, vxlan]
manager_options$remote_user [$time_local] "$request" '
                  : []
next_cfg   '$status $body_bytes_sent "$http_referer" '
         : 10
other_config        : {dpdk-extra="-w 03:00.1,representor=[0,65535] --legacy-mem ", dpdk-init="true", hw-offload="true"}
ovs_version         : []
ssl    '"$http_user_agent" "$http_x_forwarded_for"';
    access_log off;
    sendfile        on;
    keepalive_timeout : []65;
statistics    include /etc/nginx/conf.d/*.conf;
}

Code Block

language	bash

server {
listen    : {cpu="16", file_systems="/,13521220,3918464 /data,243823,2064 /boot,357176,61104", load_average="1.41,1.37,1.36", memory="16337652,5589928,1707640,0,0", process_ovs-vswitchd="5959388,256352,11515170,0,11484792,11484792,7", process_ovsdb-server="12620,6260,13190,0,68233832,68233832,6"}
system_type         : []
system_version      : []

Traffic Generator

We will use DPDK pktgen as the Traffic Generator.

80 backlog=8192 reuseport;
server_name localhost;

location / {
return 200 "hello";
}

error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}

WRK configuration

connection setup rates:

Code Block
./wrk -t 32 -c 640 -d30s http://10.0.1.127/ -H "Connection: Close"

Performance Results

For the optimized software-based OVS, we tested on a 48C24q VM, running NGINX as a server, CT is enabled on OVS-DPDK.( upstream version / our version )

We also run 50 2c2q VMs as clients, generating HTTP GET traffic to evaluate the performance.

pps

(not closing connection after each query)

pps

(closing connection after each query)

connection initial rates

(closing connection after each query)

QPS

(not closing connection after each query)

1.66Mpps/2Mpps

140Kcps/200Kcps

889Kqps/1.14Mqps

For the OVS-DPDK running on SmartNIC with CT function enabled, we tested on 17C8G VM running testpmd (where 1 core for lcore and 16 cores for nb cores) as the traffic forwarder and reached the performance below.

Note: the result also depends on the traffic generator side.

Frame size	114 bytes
Packets per second	~23Mpps

...

The test is to evaluate the performance of SmartNIC offloading.
Test API description

Thus we currently don't have any Test APIs provided.

Test Dashboards

Functional Tests

Open vSwitch itself contains a test suite for functional test, the link is http://docs.openvswitch.org/en/latest/topics/testing/

...

25 cases are skipped due to the configuration.

Performance Tests

Single PF

...

Directly forwarding without the match

"in_port=vxlan-vtp, actions=output: pf1hpf"

...

Single PF, Single VF

...

Directly forwarding without the match

"in_port=vxlan-vtp, actions=output: pf1vf0"

...

match and forwarding

100 rules

"in_port=vxlan-vtp,ip,nw_dst=10.1.1.[1-100] actions=pf1vf0"

...

21.6Mpps

(100 megaflows

offloaded)

...

match and forwarding

1000 rules

"in_port=vxlan-vtp,ip,nw_dst=10.1.[1-10].[1-100] actions=pf1vf0"

...

1000 flows with nw_dst=10.1.[1-10].[1-100]

891 flows

(due to pktgen limit, only 891 flows are loaded)

...

23.3Mpps

(891 megaflows

offloaded)

...

Single PF, 4 VFs (only test with offloaded)

...

match and forwarding

1000 rules

"in_port=vxlan-vtp,ip,nw_dst=10.1.[1-10].[1-25] actions=pf1vf0"

"in_port=vxlan-vtp,ip,nw_dst=10.1.[1-10].[26-50] actions=pf1vf1"

"in_port=vxlan-vtp,ip,nw_dst=10.1.[1-10].[51-75] actions=pf1vf2"

"in_port=vxlan-vtp,ip,nw_dst=10.1.[1-10].[75-100] actions=pf1vf3"

...

1000 flows with nw_dst=10.1.[1-10].[1-100]

891 flows

(due to pktgen limit, only 891 flows are loaded)

...

23258808

...

Additional Testing

n/a

Bottlenecks/Errata

n/a

Space shortcuts

Page tree

Versions Compared

Old Version 1

New Version Current

Key

Table of Contents

Introduction

Akraino Test Group Information

Overall Test Architecture

SmartNIC deployed Architecture

Test Bed

OVS-DPDK Test Architecture

OVS-DPDK on BlueField Test Architecture

Test Framework

Traffic Generator

CT Ruleset

Nginx configuration

Traffic Generator

Performance Results

The test is to evaluate the performance of SmartNIC offloading.
Test API description

Test Dashboards

Functional Tests

Performance Tests

Single PF

Single PF, Single VF

Single PF, 4 VFs (only test with offloaded)

Additional Testing

Bottlenecks/Errata

Space shortcuts

Page tree

Page History

Versions Compared

Old Version 1

New Version Current

Key

Table of Contents

Introduction

Akraino Test Group Information

Overall Test Architecture

SmartNIC deployed Architecture

Test Bed

OVS-DPDK Test Architecture

OVS-DPDK on BlueField Test Architecture

Test Framework

Traffic Generator

CT Ruleset

Nginx configuration

Traffic Generator

Performance Results

The test is to evaluate the performance of SmartNIC offloading.Test API description

Test Dashboards

Functional Tests

Performance Tests

Single PF

Single PF, Single VF

Single PF, 4 VFs (only test with offloaded)

Additional Testing

Bottlenecks/Errata

The test is to evaluate the performance of SmartNIC offloading.
Test API description