# Bluefield-2 DOCA FLOW Sample wont run on DPU?

**URL:** <https://forums.developer.nvidia.com/t/bluefield-2-doca-flow-sample-wont-run-on-dpu/257664>\
**Category:** BlueField\
**Created:** [June 24, 2023, 12:59am UTC](https://forums.developer.nvidia.com/t/bluefield-2-doca-flow-sample-wont-run-on-dpu/257664 "2023-06-24T00:59:17Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![IamAries](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/iamaries/32/261634_2.png) [@IamAries](https://forums.developer.nvidia.com/u/IamAries)\
**Post date:** [June 24, 2023, 12:59am UTC](https://forums.developer.nvidia.com/t/bluefield-2-doca-flow-sample-wont-run-on-dpu/257664/1 "2023-06-24T00:59:17Z")

</div>

Just getting started with BlueField-2 & DOCA, trying to run one of the sample apps just to “see it work” and get a baseline before really digging into things.

The sample built fine but won’t run. The sample is “flow\_hairpin”.

Perhaps someone can point me in the right direction?

Here are the details and error message:

The card has been setup thusly:

root@localhost:/opt/mellanox/doca/samples/doca\_flow/flow\_hairpin# mlxconfig -d 0000:03:00.0 s PF\_BAR2\_ENABLE=0 PER\_PF\_NUM\_SF=1 PF\_TOTAL\_SF=236

## Device #1:

Device type: BlueField2  
Name: MBF2H332A-AEEO\_Ax\_Bx  
Description: BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; PCIe Gen4 x8; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; HHHL  
Device: 0000:03:00.0

Configurations: Next Boot New  
PF\_BAR2\_ENABLE False(0) False(0)  
PER\_PF\_NUM\_SF True(1) True(1)  
PF\_TOTAL\_SF

The sample is being run with this command line:

./build/doca\_flow\_hairpin -a auxiliary:mlx5\_core.sf.2,dv\_flow\_en=2 -a auxiliary:mlx5\_core.sf.3,dv\_flow\_en=2 – -l 60

The error message that appears to be at the root of the issue is:

“mlx5\_net: [mlx5dr\_action\_create\_generic]: Cannot create HWS action since HWS is not supported”

And the full run output is:

root@localhost:/opt/mellanox/doca/samples/doca\_flow/flow\_hairpin# ./build/doca\_flow\_hairpin -a auxiliary:mlx5\_core.sf.2,dv\_flow\_en=2 -a auxiliary:mlx5\_core.sf.3,dv\_flow\_en=2 – -l 60  
EAL: Detected 8 lcore(s)  
EAL: Detected 1 NUMA nodes  
EAL: Detected shared linkage of DPDK  
EAL: Multi-process socket /var/run/dpdk/rte/mp\_socket  
EAL: Selected IOVA mode ‘PA’  
EAL: Probing VFIO support…  
EAL: VFIO support initialized  
EAL: No legacy callbacks, legacy socket not created  
[00:54:05:578715][DOCA][DBG][NUTILS:507]: Port 0 MAC: 02 7a db 5b 5e 7c  
[00:54:05:651509][DOCA][DBG][NUTILS:507]: Port 1 MAC: 02 ab 52 2b 30 a9  
[00:54:05:653878][DOCA][INF][engine\_model:73]: engine model defined with mode=vnf  
[00:54:05:653909][DOCA][INF][engine\_model:75]: engine model defined with nr\_pipe\_queues=8  
[00:54:05:653929][DOCA][INF][engine\_model:76]: engine model defined with pipe\_queue\_depth=0  
[00:54:05:654180][DOCA][INF][engine\_field\_mapping:96]: Engine field mapping initialized with 3 focus 12 protocols  
[00:54:05:654215][DOCA][INF][engine\_shared\_resources:94]: Engine shared resources initialized successfully  
[00:54:05:654250][DOCA][INF][dpdk\_engine:437]: queue depth is zero, set it to default 128.  
[00:54:05:654307][DOCA][INF][encap\_table:119]: encap table created  
[00:54:05:654417][DOCA][DBG][dpdk\_table\_hws:870]: Initialized dpdk table work module to be HW steering  
[00:54:05:654443][DOCA][INF][dpdk\_table:70]: Initializing dpdk table successfully  
[00:54:05:654463][DOCA][DBG][dpdk\_flow\_hws:33]: Initialized dpdk flow work module to be HW steering  
[00:54:05:654487][DOCA][INF][dpdk\_flow:82]: Initializing dpdk flow successfully  
[00:54:05:654513][DOCA][INF][engine\_shared\_resources:133]: Allocated 16 shared resources of type 2  
[00:54:05:654533][DOCA][INF][dpdk\_resource\_manager:184]: Dpdk resource manager register completed  
[00:54:05:654585][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.outer.eth.dst\_mac, offset=0)  
[00:54:05:654614][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.outer.eth.src\_mac, offset=6)  
[00:54:05:654634][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.outer.eth.type, offset=12)  
[00:54:05:654659][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.inner.eth.dst\_mac, offset=0)  
[00:54:05:654680][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.inner.eth.src\_mac, offset=6)  
[00:54:05:654700][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.inner.eth.type, offset=12)  
[00:54:05:654720][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.outer.eth\_vlan.tci, offset=0)  
[00:54:05:654745][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.inner.eth\_vlan.tci, offset=0)  
[00:54:05:654766][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.outer.ipv4.src\_ip, offset=12)  
[00:54:05:654782][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.outer.ipv4.dst\_ip, offset=16)  
[00:54:05:654804][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.outer.ipv4.next\_proto, offset=9)  
[00:54:05:654824][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.inner.ipv4.src\_ip, offset=12)  
[00:54:05:654849][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.inner.ipv4.dst\_ip, offset=16)  
[00:54:05:654871][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.inner.ipv4.next\_proto, offset=9)  
[00:54:05:654891][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.outer.ipv6.src\_ip, offset=8)  
[00:54:05:654910][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.outer.ipv6.dst\_ip, offset=24)  
[00:54:05:654935][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.outer.ipv6.next\_proto, offset=6)  
[00:54:05:654955][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.inner.ipv6.src\_ip, offset=8)  
[00:54:05:654975][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.inner.ipv6.dst\_ip, offset=24)  
[00:54:05:654999][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.inner.ipv6.next\_proto, offset=6)  
[00:54:05:655020][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.outer.udp.src\_port, offset=0)  
[00:54:05:655040][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.outer.udp.dst\_port, offset=2)  
[00:54:05:655059][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.inner.udp.src\_port, offset=0)  
[00:54:05:655083][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.inner.udp.dst\_port, offset=2)  
[00:54:05:655109][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.outer.tcp.src\_port, offset=0)  
[00:54:05:655129][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.outer.tcp.dst\_port, offset=2)  
[00:54:05:655154][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.outer.tcp.flags, offset=13)  
[00:54:05:655173][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.inner.tcp.src\_port, offset=0)  
[00:54:05:655193][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.inner.tcp.dst\_port, offset=2)  
[00:54:05:655218][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.inner.tcp.flags, offset=13)  
[00:54:05:655239][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.tunnel.vxlan.vni, offset=4)  
[00:54:05:655263][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.tunnel.gre.key, offset=0)  
[00:54:05:655288][DOCA][DBG][dpdk\_layer:58]: Registered dpdk field opcode=match.packet.tunnel.gtp.teid, offset=4)  
[00:54:05:655307][DOCA][INF][dpdk\_layer:260]: Dpdk layer register completed  
[00:54:05:655331][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.outer.eth.dst\_mac, offset=42, len=6)  
[00:54:05:655356][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.outer.eth.src\_mac, offset=36, len=6)  
[00:54:05:655381][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.outer.eth.type, offset=48, len=2)  
[00:54:05:655403][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.inner.eth.dst\_mac, offset=150, len=6)  
[00:54:05:655423][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.inner.eth.src\_mac, offset=144, len=6)  
[00:54:05:655448][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.inner.eth.type, offset=156, len=2)  
[00:54:05:655470][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.outer.eth\_vlan.tci, offset=50, len=2)  
[00:54:05:655491][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.inner.eth\_vlan.tci, offset=158, len=2)  
[00:54:05:655515][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.outer.ipv4.src\_ip, offset=56, len=4)  
[00:54:05:655536][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.outer.ipv4.dst\_ip, offset=76, len=4)  
[00:54:05:655555][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.outer.ipv4.next\_proto, offset=92, len=1)  
[00:54:05:655576][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.inner.ipv4.src\_ip, offset=164, len=4)  
[00:54:05:655600][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.inner.ipv4.dst\_ip, offset=184, len=4)  
[00:54:05:655621][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.inner.ipv4.next\_proto, offset=200, len=1)  
[00:54:05:655645][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.outer.ipv6.src\_ip, offset=56, len=16)  
[00:54:05:655667][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.outer.ipv6.dst\_ip, offset=76, len=16)  
[00:54:05:655689][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.outer.ipv6.next\_proto, offset=92, len=1)  
[00:54:05:655713][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.inner.ipv6.src\_ip, offset=164, len=16)  
[00:54:05:655734][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.inner.ipv6.dst\_ip, offset=184, len=16)  
[00:54:05:655755][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.inner.ipv6.next\_proto, offset=200, len=1)  
[00:54:05:655779][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.outer.udp.src\_port, offset=94, len=2)  
[00:54:05:655800][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.outer.udp.dst\_port, offset=96, len=2)  
[00:54:05:655820][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.inner.udp.src\_port, offset=202, len=2)  
[00:54:05:655846][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.inner.udp.dst\_port, offset=204, len=2)  
[00:54:05:655867][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.outer.tcp.src\_port, offset=94, len=2)  
[00:54:05:655887][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.outer.tcp.dst\_port, offset=96, len=2)  
[00:54:05:655911][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.outer.tcp.flags, offset=93, len=1)  
[00:54:05:655934][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.inner.tcp.src\_port, offset=202, len=2)  
[00:54:05:655954][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.inner.tcp.dst\_port, offset=204, len=2)  
[00:54:05:655978][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.inner.tcp.flags, offset=201, len=1)  
[00:54:05:655998][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.tunnel.vxlan.vni, offset=104, len=3)  
[00:54:05:656018][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.tunnel.gre.key, offset=108, len=4)  
[00:54:05:656038][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.tunnel.gre.protocol, offset=106, len=2)  
[00:54:05:656063][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.tunnel.gtp.teid, offset=104, len=4)  
[00:54:05:656083][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.tunnel.nisp.hdr, offset=104, len=40)  
[00:54:05:656108][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.tunnel.audp.hdr, offset=104, len=24)  
[00:54:05:656128][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.tunnel.esp.spi, offset=104, len=4)  
[00:54:05:656149][DOCA][DBG][doca\_flow\_layer:66]: Registered field opcode=match.packet.tunnel.esp.sn, offset=108, len=4)  
[00:54:05:656213][DOCA][INF][doca\_flow\_layer:466]: Doca flow layer initialized  
[00:54:05:656234][DOCA][INF][doca\_flow:526]: Doca flow initialized successfully  
[00:54:05:657054][DOCA][INF][utils\_hash\_table:123]: hash table a\_tmplt\_t port 0 created  
[00:54:05:657130][DOCA][INF][utils\_hash\_table:123]: hash table p\_tmplt\_t port 0 created  
[00:54:05:657208][DOCA][INF][utils\_hash\_table:123]: hash table dpdk\_tbl\_mgr port 0 created  
[00:54:05:657335][DOCA][INF][utils\_hash\_table:123]: hash table grp\_fwd port 0 created  
[00:54:05:657358][DOCA][INF][dpdk\_port:167]: Dpdk port 0 initialized successfully with 9 queues  
mlx5\_net: [mlx5dr\_action\_create\_generic]: Cannot create HWS action since HWS is not supported  
[00:54:05:714997][DOCA][ERR][dpdk\_flow\_hws\_legacy:143]: failed to configure flow hws port 0 - rte flow configure, type 1 message: fail to configure port  
[00:54:05:715066][DOCA][ERR][dpdk\_engine:1694]: failed to start port 0 - init port, ret=-1  
[00:54:05:715146][DOCA][INF][utils\_hash\_table:151]: hash table destroyed  
[00:54:05:715581][DOCA][INF][utils\_hash\_table:151]: hash table destroyed  
[00:54:05:715639][DOCA][INF][utils\_hash\_table:151]: hash table destroyed  
[00:54:05:715673][DOCA][INF][utils\_hash\_table:151]: hash table destroyed  
[00:54:05:715692][DOCA][INF][dpdk\_port:230]: Dpdk port 0 destroyed successfully with 9 queues  
[00:54:05:715723][DOCA][ERR][flow\_common:82]: Failed to start port - dpdk port start failed (0)  
[00:54:05:715747][DOCA][ERR][FLOW\_HAIRPIN:139]: Failed to init DOCA ports  
[00:54:05:715774][DOCA][INF][doca\_flow\_layer:478]: Doca flow layer destroyed  
[00:54:05:715794][DOCA][INF][dpdk\_resource\_manager:191]: Dpdk resource manager unregister completed  
[00:54:05:715814][DOCA][INF][dpdk\_flow:205]: Cleanup dpdk flow  
[00:54:05:715833][DOCA][DBG][dpdk\_flow\_hws:69]: Cleanup dpdk flow HW steering module  
[00:54:05:715852][DOCA][INF][dpdk\_table:77]: Cleanup dpdk table  
[00:54:05:715871][DOCA][DBG][dpdk\_table\_hws:877]: Cleanup dpdk table HW steering module  
[00:54:05:715890][DOCA][INF][dpdk\_layer:272]: Dpdk layer unregister completed  
[00:54:05:715917][DOCA][INF][dpdk\_resource\_manager:191]: Dpdk resource manager unregister completed  
[00:54:05:715936][DOCA][INF][dpdk\_flow:205]: Cleanup dpdk flow  
[00:54:05:715954][DOCA][DBG][dpdk\_flow\_hws:69]: Cleanup dpdk flow HW steering module  
[00:54:05:715972][DOCA][INF][dpdk\_table:77]: Cleanup dpdk table  
[00:54:05:715990][DOCA][DBG][dpdk\_table\_hws:877]: Cleanup dpdk table HW steering module  
[00:54:05:716011][DOCA][INF][dpdk\_layer:272]: Dpdk layer unregister completed  
[00:54:05:716037][DOCA][INF][encap\_table:136]: encap table destroyed  
[00:54:05:716060][DOCA][INF][engine\_shared\_resources:243]: Cleanup 16 shared resources of type 2 completed  
[00:54:05:716082][DOCA][INF][engine\_field\_mapping:104]: Engine field mapping destroyed  
[00:54:05:716105][DOCA][INF][engine\_model:150]: engine model destroyed  
[00:54:05:716125][DOCA][INF][doca\_flow:542]: Doca flow destroyed  
[00:54:05:716144][DOCA][ERR][FLOW\_HAIRPIN::MAIN:72]: flow\_hairpin sample encountered errors  
Tx port 0 is already stopped  
[00:54:05:716219][DOCA][ERR][NUTILS:104]: Failed to bind hairpin queues (-16)  
[00:54:05:716242][DOCA][ERR][NUTILS:191]: Disabling hairpin queues failed: err=21, port=0  
Tx port 0 is already stopped  
[00:54:05:716294][DOCA][ERR][NUTILS:117]: Failed to bind hairpin queues (-16)  
[00:54:05:716316][DOCA][ERR][NUTILS:191]: Disabling hairpin queues failed: err=21, port=1  
Device with port\_id=0 already stopped  
Segmentation fault (core dumped)  
root@localhost:/opt/mellanox/doca/samples/doca\_flow/flow\_hairpin#

Hopefully I am just missing something simple…!

Thanks!  
-J

---

<div class="post-metadata">

**Author:** ![yavtuk](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@yavtuk](https://forums.developer.nvidia.com/u/yavtuk)\
**Post date:** [June 24, 2023, 9:22pm UTC](https://forums.developer.nvidia.com/t/bluefield-2-doca-flow-sample-wont-run-on-dpu/257664/2 "2023-06-24T21:22:54Z")

</div>

Hi,

I’ve got the same issue, I just updated a firmware using mlxofedinstall and It helps to me. Can you share firmware version?

---

<div class="post-metadata">

**Author:** ![IamAries](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/iamaries/32/261634_2.png) [@IamAries](https://forums.developer.nvidia.com/u/IamAries)\
**Post date:** [June 25, 2023, 4:07am UTC](https://forums.developer.nvidia.com/t/bluefield-2-doca-flow-sample-wont-run-on-dpu/257664/3 "2023-06-25T04:07:38Z")

</div>

Hi Yavtuk -

Thanks for the reply. I believe our firmware is current. Here are the version details…

root@localhost:/opt/mellanox/doca/samples/doca\_flow/flow\_hairpin# flint -d /dev/mst/mt41686\_pciconf0 query  
Image type: FS4  
FW Version: 24.37.1300  
FW Version(Running): 24.35.2000  
FW Release Date: 11.5.2023  
Product Version: 24.35.2000  
Rom Info: type=UEFI Virtio net version=21.4.10 cpu=AMD64,AARCH64  
type=UEFI Virtio blk version=22.4.10 cpu=AMD64,AARCH64  
type=UEFI version=14.28.16 cpu=AMD64,AARCH64  
type=PXE version=3.6.805 cpu=AMD64  
Description: UID GuidsNumber  
Base GUID: b8cef60300677860 14  
Base MAC: b8cef6677860 14  
Image VSD: N/A  
Device VSD: N/A  
PSID: MT\_0000000540  
Security Attributes: N/A  
root@localhost:/opt/mellanox/doca/samples/doca\_flow/flow\_hairpin#

-J

---

<div class="post-metadata">

**Author:** ![yavtuk](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@yavtuk](https://forums.developer.nvidia.com/u/yavtuk)\
**Post date:** [June 25, 2023, 10:51am UTC](https://forums.developer.nvidia.com/t/bluefield-2-doca-flow-sample-wont-run-on-dpu/257664/4 "2023-06-25T10:51:52Z")

</div>

Hi @IamAries, I check the sample with your options and It works on my environment.

let me describe how I setup env step by step:

1. /opt/mellanox/dpdk/bin/dpdk-hugepages.py --pagesize 2M --setup 4G
2. ovs configuration:  
[ovs.log](https://forums.developer.nvidia.com/uploads/short-url/ii1m6d08CZZQBkudFznh4mYOEM8.log) (2.1 KB)
3. app log  
[flow\_hairpin.log](https://forums.developer.nvidia.com/uploads/short-url/nKryVYoXlTsNDC4clHtnFQAzywc.log) (41.2 KB)

---

<div class="post-metadata">

**Author:** ![yavtuk](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@yavtuk](https://forums.developer.nvidia.com/u/yavtuk)\
**Post date:** [June 25, 2023, 10:53am UTC](https://forums.developer.nvidia.com/t/bluefield-2-doca-flow-sample-wont-run-on-dpu/257664/5 "2023-06-25T10:53:32Z")

</div>

Also, you should perform power cycle because

FW Version: 24.37.1300  
FW Version(Running): 24.35.2000

---

<div class="post-metadata">

**Author:** ![IamAries](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/iamaries/32/261634_2.png) [@IamAries](https://forums.developer.nvidia.com/u/IamAries)\
**Post date:** [June 25, 2023, 10:15pm UTC](https://forums.developer.nvidia.com/t/bluefield-2-doca-flow-sample-wont-run-on-dpu/257664/6 "2023-06-25T22:15:32Z")

</div>

Hey Yavtuk -

Thank you for taking the time to run the sample and send your logs. It turns out issue was I did not specify the proper sub-function in the command line arg. Your log showed it right away. I now have things working properly.

Thanks again  
-J

---

<div class="post-metadata">

**Author:** ![IamAries](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/iamaries/32/261634_2.png) [@IamAries](https://forums.developer.nvidia.com/u/IamAries)\
**Post date:** [June 28, 2023, 9:17am UTC](https://forums.developer.nvidia.com/t/bluefield-2-doca-flow-sample-wont-run-on-dpu/257664/7 "2023-06-28T09:17:10Z")

</div>

Hey Yavtuk -

Well, seems I spoke too soon. If you don’t mind, I could use a little more assistance. I can now get the samples to run without errors but I just cannot see it actually work!

The test scenario I have is running a ping on the network segment connected to port 1. I run either the flow\_hairpin or flow\_drop samples which run with no error but do not seem to have any effect.

Running tcpdump on the host machine on my ens2f1 port I can see the ping traffic coming in. Looking at the ovs switch flows I can see the packet counters increasing. It would seem the DPU is passing the traffic through ovs and up to the host – ok. I then run the flow\_drop sample application. I would expect, at least, the traffic going up to the host to stop as presumably the flow\_drop app should be passing all traffic now to the second port on the DPU (and not up to the host). This does not happen.

Looking at the code for this sample it appears that its intent is to pass all traffic between the ports except for the traffic that it attempts to block (appears to be traffic destined to 8.8.8.8 on port 80). Not seeing traffic emanating from my port 0, and at the same time seeing the traffic continue to appear at host would seem to suggest something isn’t working.

I also note that my starting condition does not appear to matching yours. Your log file output seems to snow no sf’s existing before you added two for use by the sample (your output of “/opt/mellanox/iproute2/sbin/mlxdevm port show” was blank). On my system there are already two existing sf’s that I did not create. I am not sure what accounts for this difference. I just did a reinstall of the SDK & re-flash of the card to attempt to clear out anything that may have been pre-existing somehow but result is still the same.

Here is what my system looks like:

/opt/mellanox/iproute2/sbin/mlxdevm port show  
pci/0000:03:00.0/229376: type eth netdev en3f0pf0sf0 flavour pcisf controller 0 pfnum 0 sfnum 0  
function:  
hw\_addr 02:f4:65:74:e3:69 state active opstate attached roce true max\_uc\_macs 128 trust off  
pci/0000:03:00.0/229377: type eth netdev en3f0pf0sf4 flavour pcisf controller 0 pfnum 0 sfnum 4  
function:  
hw\_addr 00:00:00:00:10:00 state active opstate attached roce true max\_uc\_macs 128 trust on  
pci/0000:03:00.1/294912: type eth netdev en3f1pf1sf0 flavour pcisf controller 0 pfnum 1 sfnum 0  
function:  
hw\_addr 02:f5:45:34:a0:38 state active opstate attached roce true max\_uc\_macs 128 trust off  
pci/0000:03:00.1/294913: type eth netdev en3f1pf1sf5 flavour pcisf controller 0 pfnum 1 sfnum 5  
function:  
hw\_addr 00:00:00:00:20:00 state active opstate attached roce true max\_uc\_macs 128 trust on

(sf4 and sf5 are the ones I added. the two sf0’s are the ones that were already existing)

And my command line to run the flow\_drop example:

./build/doca\_flow\_drop -a auxiliary:mlx5\_core.sf.5,dv\_flow\_en=2 -a auxiliary:mlx5\_core.sf.4,dv\_flow\_en=2 – -l 60

And to run the hairpin:

./build/doca\_flow\_hairpin -a auxiliary:mlx5\_core.sf.5,dv\_flow\_en=2 -a auxiliary:mlx5\_core.sf.4,dv\_flow\_en=2 – -l 60

Might you have any thoughts that might help me here?

Thanks again for your time  
-J  
[config.log](https://forums.developer.nvidia.com/uploads/short-url/bWdYDbuHMs6lvVSeqfW99Otlmd3.log) (2.7 KB)  
[flow\_drop.log](https://forums.developer.nvidia.com/uploads/short-url/eBNZeyolQyU0IYDryY1EOx7ImVk.log) (25.9 KB)

---

<div class="post-metadata">

**Author:** ![yavtuk](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@yavtuk](https://forums.developer.nvidia.com/u/yavtuk)\
**Post date:** [June 28, 2023, 2:18pm UTC](https://forums.developer.nvidia.com/t/bluefield-2-doca-flow-sample-wont-run-on-dpu/257664/8 "2023-06-28T14:18:17Z")

</div>

Hi @IamAries, I don’t have much experience with openvswitch.  
It’s hard to say but I can try to find time to check it locally.

Can you share me the rules for ovs bridges?

sudo ovs-ofctl dump-flows ovsbr1  
sudo ovs-ofctl dump-flows ovsbr2

BTW, you can delete SFs manually  
/opt/mellanox/iproute2/sbin/mlxdevm port del pci/0000:03:00.0/xxxxxx

---

<div class="post-metadata">

**Author:** ![yavtuk](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@yavtuk](https://forums.developer.nvidia.com/u/yavtuk)\
**Post date:** [June 28, 2023, 2:24pm UTC](https://forums.developer.nvidia.com/t/bluefield-2-doca-flow-sample-wont-run-on-dpu/257664/9 "2023-06-28T14:24:12Z")

</div>

based on config log, it seems wrong ovs configuration

b8a82ad1-cabf-4a6f-829b-2b1c05c5e06b  
Port en3f1pf1sf0  
Interface en3f1pf1sf0

```
    Port en3f0pf0sf0
        Interface en3f0pf0sf0

```

this ports aren’t trusted in your setup

pci/0000:03:00.0/229376: type eth netdev en3f0pf0sf0 flavour pcisf controller 0 pfnum 0 sfnum 0  
function:  
hw\_addr 02:f4:65:74:e3:69 state active opstate attached roce true max\_uc\_macs 128 trust off

---

<div class="post-metadata">

**Author:** ![IamAries](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/iamaries/32/261634_2.png) [@IamAries](https://forums.developer.nvidia.com/u/IamAries)\
**Post date:** [June 28, 2023, 3:10pm UTC](https://forums.developer.nvidia.com/t/bluefield-2-doca-flow-sample-wont-run-on-dpu/257664/10 "2023-06-28T15:10:41Z")

</div>

Here are the OVS flows:

root@localhost:/opt/mellanox/doca/samples/doca\_flow/flow\_drop# ovs-ofctl dump-flows ovsbr1  
cookie=0x0, duration=22826.342s, table=0, n\_packets=380, n\_bytes=118022, priority=0 actions=NORMAL  
root@localhost:/opt/mellanox/doca/samples/doca\_flow/flow\_drop# ovs-ofctl dump-flows ovsbr2  
cookie=0x0, duration=22829.571s, table=0, n\_packets=15970, n\_bytes=1427430, priority=0 actions=NORMAL  
root@localhost:/opt/mellanox/doca/samples/doca\_flow/flow\_drop#

My assumption had been, though, that if traffic was being hairpin in the DPU that it would never come up to OVS because this is offloaded, is it not so?

The untrusted SF’s are the first two that were not created by me. I did not specify either of these in my CLI args so presumably the sample is not attempting to use these.

I will try to remove them shortly and see if it makes any difference.

If you do find time to try it on yours I would most appreciate it. Again, thank you for taking the time to help me get started here; I really appreciate it.

-J

---

<div class="post-metadata">

**Author:** ![yavtuk](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@yavtuk](https://forums.developer.nvidia.com/u/yavtuk)\
**Post date:** [June 28, 2023, 3:40pm UTC](https://forums.developer.nvidia.com/t/bluefield-2-doca-flow-sample-wont-run-on-dpu/257664/11 "2023-06-28T15:40:47Z")

</div>

I add some rules to openvswitch for ingress traffic:

ingress packet → p0 → ovsbr1 → en3f0pf0sf2 → app → en3f0pf0sf1 → ovsbr2 → pf0hpf → host  
sudo ovs-ofctl add-flow ovsbr1 in\_port=p0,actions=output: en3f0pf0sf2  
sudo ovs-ofctl add-flow ovsbr2 in\_port= en3f0pf0sf1,actions=output: pf0hpf

I can see the packets inside my app. You can try to add the same rules for P1

---

<div class="post-metadata">

**Author:** ![IamAries](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/iamaries/32/261634_2.png) [@IamAries](https://forums.developer.nvidia.com/u/IamAries)\
**Post date:** [June 28, 2023, 5:13pm UTC](https://forums.developer.nvidia.com/t/bluefield-2-doca-flow-sample-wont-run-on-dpu/257664/12 "2023-06-28T17:13:21Z")

</div>

This appear to maybe be where my issue was.

Setting up rules similar to yours in ovs now seems to allow the examples to pass traffic in a way that I would expect.

What is curious is that ovs does not show the flows as hardware offloaded. Should it? Or is it implied that they are offloaded because we reference the representers?

Here are the output of dump-flow on my ovs – traffic is passing successfully, but how to verify that this traffic is actually staying in the chip, not coming into the ARM cores?

(this is a run of the flow\_drop example… so all traffic should either drop in hardware, or pass to other physical port in hardware… nothing up to the ARM cores)

root@localhost:/home/ubuntu# ovs-appctl dpctl/dump-flows  
recirc\_id(0),in\_port(5),eth(src=e8:eb:d3:8c:08:6a,dst=00:00:00:00:fe:00),eth\_type(0x0800),ipv4(frag=no), packets:1518, bytes:127512, used:0.010s, actions:6  
recirc\_id(0),in\_port(4),eth(src=02:f5:45:34:a0:38,dst=ff:ff:ff:ff:ff:ff),eth\_type(0x0800),ipv4(frag=no), packets:0, bytes:0, used:9.440s, actions:2,3  
recirc\_id(0),in\_port(4),eth(src=e8:eb:d3:8c:08:6a,dst=00:00:00:00:fe:00),eth\_type(0x0800),ipv4(frag=no), packets:28, bytes:2352, used:0.010s, actions:2,3  
root@localhost:/home/ubuntu#  
root@localhost:/home/ubuntu# ovs-ofctl dump-flows br0  
cookie=0x0, duration=164.480s, table=0, n\_packets=1576, n\_bytes=132398, in\_port=p0 actions=output:en3f1pf1sf4  
cookie=0x0, duration=1704.893s, table=0, n\_packets=4251, n\_bytes=356274, priority=0 actions=NORMAL  
root@localhost:/home/ubuntu# ovs-ofctl dump-flows br1  
cookie=0x0, duration=1460.190s, table=0, n\_packets=9106, n\_bytes=928786, in\_port=p1 actions=output:en3f1pf1sf0  
cookie=0x0, duration=1705.415s, table=0, n\_packets=1423, n\_bytes=126236, priority=0 actions=NORMAL  
root@localhost:/home/ubuntu#

-J

---

<div class="post-metadata">

**Author:** ![yavtuk](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@yavtuk](https://forums.developer.nvidia.com/u/yavtuk)\
**Post date:** [June 28, 2023, 6:41pm UTC](https://forums.developer.nvidia.com/t/bluefield-2-doca-flow-sample-wont-run-on-dpu/257664/13 "2023-06-28T18:41:20Z")

</div>

"What is curious is that ovs does not show the flows as hardware offloaded. Should it? Or is it implied that they are offloaded because we reference the representers?

Here are the output of dump-flow on my ovs – traffic is passing successfully, but how to verify that this traffic is actually staying in the chip, not coming into the ARM cores?"

I am thinking about the same issue and I am going to investigate it. let me know if you find something about it

---

<div class="post-metadata">

**Author:** ![IamAries](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/iamaries/32/261634_2.png) [@IamAries](https://forums.developer.nvidia.com/u/IamAries)\
**Post date:** [June 29, 2023, 3:34am UTC](https://forums.developer.nvidia.com/t/bluefield-2-doca-flow-sample-wont-run-on-dpu/257664/14 "2023-06-29T03:34:00Z")

</div>

Okay, yes I will do so! Thanks for all of your help; it sure sped up my getting things going!

-J

---

<div class="post-metadata">

**Author:** ![yavtuk](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@yavtuk](https://forums.developer.nvidia.com/u/yavtuk)\
**Post date:** [June 29, 2023, 6:12am UTC](https://forums.developer.nvidia.com/t/bluefield-2-doca-flow-sample-wont-run-on-dpu/257664/15 "2023-06-29T06:12:40Z")

</div>

You are very welcome 🙂

---

<div class="post-metadata">

**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\
**Post date:** [July 13, 2023, 6:13am UTC](https://forums.developer.nvidia.com/t/bluefield-2-doca-flow-sample-wont-run-on-dpu/257664/16 "2023-07-13T06:13:11Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.