...
OVS-DPDK Test Architecture
Test Bed
OVS-DPDK on BlueField Test Architecture
The testbed setup is shown in the above diagram. DUT stands for Device Under Test
...
| Type | Description |
|---|---|
| SmartNICs | BlueField v1, 25Gbps |
| DPDK | version 19.11 |
| vSwitch | OVS-DPDK 2.12 with VXLAN DECAP/ENCAP offload enabled. |
| Code Block |
|---|
root@bluefield:/home/ovs-dpdk# ovs-vsctl show 2dccd148-526c-44a5-9351-67b04c5e2da4 Bridge br-int datapath_type: netdev Port br-int Bridge Interface br-int datapath_type: netdevinternal Port vxlanvhost-user-vtp01 Interface vxlanvhost-user-vtp01 type: vxlandpdkvhostuserclient options: {dst_port="4789", key=flow, local_ip="192.168.1.1", remote_ip=flow, tos=inheritvhost-server-path="/var/run/openvswitch/dpdkvhostclient01"} Port brvxlan-intvtp Interface brvxlan-intvtp type: internal vxlan options: {dst_port="4789", key=flow, local_ip="YOU IP", remote_ip=flow, tos=inherit} Port pf1hpfbr-ex-patch Interface pf1hpfbr-ex-patch type: dpdkpatch options: {dpdk-devargs="class=eth,mac=ae:d8:8a:c5:22:fb"peer=br-int-patch} Bridge br-ex datapath_type: netdev Port br-int-expatch Interface br-int-expatch type: internalpatch options: {peer=br-ex-patch} Port p1eth2 Interface p1eth2 type: dpdk options: {dpdk-devargs="PCIE"class=eth,mac=98:03:9b:af:7b:0b"} |
| Code Block |
root@bluefield:/home/ovs-dpdk# ovs-vsctl list open_vswitch _uuid , n_rxq="4"} Port br-ex :Interface 2dccd148-526c-44a5-9351-67b04c5e2da4 bridgesbr-ex type: internal : [22334686-733a-445e-9130-a42009a3586e, 38af610d-01f7-497d-878b-c6b6a44abf6a] cur_cfg : 10 datapath_types : [netdev, system] datapaths : {} db_version ovs_version: "2.13.90" |
| Code Block |
|---|
root:/home/ovs-dpdk# ovs-vsctl --format=csv --data=bare --no-headings --column=other_config list open_vswitch
"dpdk-extra=-w [PCIE] -l 70 dpdk-init=true dpdk-socket-mem=2048,2048 emc-insert-inv-prob=0 n-handler-threads=1 n-revalidator-threads=4 neigh-notifier-enable=true pmd-cpu-mask=0xc00000000000c00000 pmd-pause=false pmd-rxq-assign=roundrobin smc-enable=true tx-flush-interval=0 userspace-tso-enable=true"
|
Traffic Generator
We will use DPDK pktgen as the Traffic Generator.
CT Ruleset
| Code Block | ||
|---|---|---|
| ||
Br-ex rules
table=0 priority=300 in_port=eth2,tcp,tp_dst=32768/0x8000 actions=output:br-int-patch
table=0 priority=300 in_port=eth2,udp,tp_dst=32768/0x8000 actions=output:br-int-patch
table=0 priority=200 in_port=eth2 actions=output:LOCAL
table=0 priority=200 in_port=LOCAL actions=eth2
table=0 priority=200 in_port=br-int-patch actions=eth2 |
| Code Block | ||
|---|---|---|
| ||
Br-int rules
|
Nginx configuration
| Code Block | ||
|---|---|---|
| ||
user nginx; worker_processes auto; error_log /var/log/nginx/error.log warn; pid : [] dpdk_initialized /var/run/nginx.pid; events { : true dpdk_version worker_connections 2000000; } http { include : "DPDK 19.11.0" external_ids /etc/nginx/mime.types; default_type : {} iface_types application/octet-stream; log_format main '$remote_addr - :$remote_user [dpdk, dpdkr, dpdkvhostuser, dpdkvhostuserclient, erspan, geneve, gre, internal, ip6erspan, ip6gre, lisp, patch, stt, system, tap, vxlan] manager_options$time_local] "$request" ' : [] next_cfg '$status $body_bytes_sent "$http_referer" ' : 10 other_config : {dpdk-extra="-w 03:00.1,representor=[0,65535] --legacy-mem ", dpdk-init="true", hw-offload="true"} ovs_version : [] ssl '"$http_user_agent" "$http_x_forwarded_for"'; access_log off; sendfile on; keepalive_timeout : []65; statistics include /etc/nginx/conf.d/*.conf; } |
| Code Block | ||
|---|---|---|
| ||
server { listen : {cpu="16", file_systems="/,13521220,3918464 /data,243823,2064 /boot,357176,61104", load_average="1.41,1.37,1.36", memory="16337652,5589928,1707640,0,0", process_ovs-vswitchd="5959388,256352,11515170,0,11484792,11484792,7", process_ovsdb-server="12620,6260,13190,0,68233832,68233832,6"} system_type : [] system_version : [] |
Traffic Generator
...
80 backlog=8192 reuseport;
server_name localhost;
location / {
return 200 "hello";
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}
|
WRK configuration
connection setup rates:
| Code Block |
|---|
./wrk -t 32 -c 640 -d30s http://10.0.1.127/ -H "Connection: Close" |
Test API description
The test is to evaluate the performance of SmartNIC offloading.
...
25 cases are skipped due to the configuration.
Performance Tests
Single PF
...
Directly forwarding without the match
"in_port=vxlan-vtp, actions=output: pf1hpf"
...
Single PF, Single VF
...
Directly forwarding without the match
"in_port=vxlan-vtp, actions=output: pf1vf0"
...
match and forwarding
100 rules
"in_port=vxlan-vtp,ip,nw_dst=10.1.1.[1-100] actions=pf1vf0"
...
21.6Mpps
(100 megaflows
offloaded)
...
match and forwarding
1000 rules
"in_port=vxlan-vtp,ip,nw_dst=10.1.[1-10].[1-100] actions=pf1vf0"
...
1000 flows with nw_dst=10.1.[1-10].[1-100]
891 flows
(due to pktgen limit, only 891 flows are loaded)
...
23.3Mpps
(891 megaflows
offloaded)
...
Single PF, 4 VFs (only test with offloaded)
...
match and forwarding
1000 rules
"in_port=vxlan-vtp,ip,nw_dst=10.1.[1-10].[1-25] actions=pf1vf0"
"in_port=vxlan-vtp,ip,nw_dst=10.1.[1-10].[26-50] actions=pf1vf1"
"in_port=vxlan-vtp,ip,nw_dst=10.1.[1-10].[51-75] actions=pf1vf2"
"in_port=vxlan-vtp,ip,nw_dst=10.1.[1-10].[75-100] actions=pf1vf3"
...
1000 flows with nw_dst=10.1.[1-10].[1-100]
891 flows
(due to pktgen limit, only 891 flows are loaded)
...
23258808
...
Additional Testing
n/a
Bottlenecks/Errata
...