Mellanox CX5/CX6 DX with DPDK rte_flow (dv_flow_en=1) performance problem


DPDK version: 18.11/19.11/20.5/20.8/20.11

OFED & mlx5 drv version: 4.6/4.7-

FW version: 16.27.2008

Memory: 16G*8

KERNEL verision: 4.14.13/5.4.56

CPU: INTEL broadwell microarch 2696v4 core*22 NUMA * 2

Test config:

IXIA 100G, IXIA-> switch → DPDK testpmd;

DPDK testpmd:

app_opt="-i --burst=32 --txd=1024 --rxd=1024 --mbcache=512 "

app_opt+=" --no-numa --enable-rx-cksum --mlockall "

app_opt+=" --rxq=16 --txq=16 --nb-cores=16 --coremask=0xffff0 "

app_opt+=" --pkt-filter-mode=perfect "

app_opt+=" --forward-mode icmpecho -a"

dev_opt=" -w 54:00.0,rx_vec_en=1,dv_flow_en=1 "

./testpmd --log-level=8 -c 0xffffffff --socket-mem=5120,5120 -n 2 -r 2 $dev_opt – $app_opt

testpmd> start

DPDK rte_flow

flow create 0 ingress pattern eth / ipv4 / udp / end actions count / drop / end

IXIA send 99G 64bytes traffic, but flow only count 14G+ traffic

mlx_fs_dump :

FT: 0x40009 (level: 0x1, type: NIC_RX)

±- FG: 0x20 (OUT_HDR: |ip_ver|cvlan_tag|ip_prot|)

±- FTE: 0x0 (FWD FLOW_COUNT) to (TIR:0x57) ip_ver:0x4 ip_prot:UDP

FT: 0x40008 (level: 0x20, type: NIC_RX)

|-- FG: 0x1a (OUT_HDR: |dmac_47_16|dmac_15_0|ip_ver|cvlan_tag|)

| |-- FTE: 0x0 (FWD) to (TIR:0x54) dmac:0xffffffffffff ip_ver:0x6

| ±- FTE: 0x1 (FWD) to (TIR:0x55) dmac:0xffffffffffff ip_ver:0x4

|-- FG: 0x1b (OUT_HDR: |dmac_47_16|dmac_15_0|)

| ±- FTE: 0x2492 (FWD) to (TIR:0x56) dmac:0xffffffffffff

|-- FG: 0x1c (OUT_HDR: |dmac_47_16|ip_ver|cvlan_tag|)

| |-- FTE: 0x4924 (FWD) to (TIR:0x54) dmac:0x33330000 ip_ver:0x6

| ±- FTE: 0x4925 (FWD) to (TIR:0x55) dmac:0x33330000 ip_ver:0x4

±- FG: 0x1d (OUT_HDR: |dmac_47_16|)

±- FTE: 0x6db6 (FWD) to (TIR:0x56) dmac:0x33330000

FT: 0x40007 (level: 0x1d, type: NIC_RX)

|-- FG: 0x19 (NO_MATCH)

| ±- FTE: 0x0 (FWD) to (TIR:0x56)

±- FG: 0x1f (OUT_HDR: |dmac_47_16|dmac_15_0|)

±- FTE: 0x2492 (FWD) to (TIR:0x56) dmac:0x98039b975076

FT: 0x40006 (level: 0x1b, type: NIC_RX)

|-- FG: 0x18 (OUT_HDR: |ip_ver|cvlan_tag|)

| |-- FTE: 0x0 (FWD) to (TIR:0x54) ip_ver:0x6

| ±- FTE: 0x1 (FWD) to (TIR:0x55) ip_ver:0x4

±- FG: 0x1e (OUT_HDR: |dmac_47_16|dmac_15_0|ip_ver|cvlan_tag|)

|-- FTE: 0x2492 (FWD) to (TIR:0x54) dmac:0x98039b975076 ip_ver:0x6

±- FTE: 0x2493 (FWD) to (TIR:0x55) dmac:0x98039b975076 ip_ver:0x4

but we set dv_flow_en=0 use mlx5 verbs flow API with rte_flow

IXIA send 99G 64bytes traffic, flow can count 99G traffic

Maybe DV flow API has BUG ?

Hi Arthas,

Which traffic is generated via IXIA? TCP?

What happens if you’re deleting the rule and keeping dv flow enabled?