Bluefield3 DPU (B3210E) loopback to both ports and ping failed

james.hsu2 · March 6, 2026, 2:30am

Hi NV,

I have one Bluefield3 DPU (B3210E) card and one QSFP28 connector , I use QSFP28 connector to loopback to both ports , and use port1 to ping port2 , below is my setting,

I use mst command to check BF3 card status,

$ sudo mst status -v

MST modules:

MST PCI module is not loaded
MST PCI configuration module is not loaded

PCI devices:

DEVICE_TYPE MST PCI RDMA NET NUMA VFIO
BlueField3(rev:1) NA 0005:01:00.0 mlx5_0 net-enP5p1s0f0np0 -1

GB100(rev:0) NA 0000:01:00.0 -1

BlueField3(rev:1) NA 0005:01:00.1 mlx5_1 net-enP5p1s0f1np1 -1

$ sudo mlxlink -d 0005:01:00.0 -m

Operational Info

State : Active
Physical state : LinkUp
Speed : 100G
Width : 4x
FEC : Standard RS-FEC - RS(528,514)
Loopback Mode : No Loopback
Auto Negotiation : ON

Supported Info

Enabled Link Speed (Ext.) : 0x00000ff2 (100G_1X,100G_2X,100G_4X,50G_1X,50G_2X,40G,25G,10G,1G)
Supported Cable Speed (Ext.) : 0x000002f2 (100G_4X,50G_2X,40G,25G,10G,1G)

Troubleshooting Info

Status Opcode : 0
Group Opcode : N/A
Recommendation : No issue was observed

Tool Information

Firmware Version : 32.40.1000
amBER Version : 5.75
MFT Version : 4.34.1-10

Module Info

Temperature [C] : 0 [0..0]
Voltage [mV] : 0 [0..0]
Bias Current [mA] : 0,0,0,0 [0..0]
Rx Power Current [dBm] : 0,0,0,0 [0..0]
Tx Power Current [dBm] : 0,0,0,0 [0..0]
Identifier : QSFP+
Compliance : 40GBASE-CR4,100GBASE-CR4, 25GBASE-CR CA-25G-L or 50GBASE-CR2 with RS (Clause91) FEC
Cable Technology : Copper cable, passive, unequalized
Cable Type : Passive copper cable
OUI : Other
Vendor Name : HanElectronic
Vendor Part Number : HE-P2E30-3.0M
Vendor Serial Number : 20201300002
Rev : 1
Wavelength [nm] : N/A
Transfer Distance [m] : 3
Attenuation (5g,7g,12g)[dB] : 4,5,9
FW Version : N/A
Digital Diagnostic Monitoring : Yes
Power Class : N/A
MAX Power : N/A
CDR RX : N/A
CDR TX : N/A
LOS Alarm : N/A
SNR Media Lanes [dB] : N/A
SNR Host Lanes [dB] : N/A
IB Cable Width : 1x,2x,4x
Memory Map Revision : 5
Linear Direct Drive : 0
Cable Breakout : Channels implemented [1,2,3,4]/Far end is unspecified
SMF Length : N/A
Cable Rx AMP : N/A
Cable Rx Emphasis : N/A
Cable Rx Post Emphasis : N/A
Cable Tx Equalization : N/A
Wavelength Tolerance : N/A
Module State : N/A
DataPath state [per lane] : N/A,N/A,N/A,N/A
Rx Output Valid [per lane] : 0,0,0,0
Nominal bit rate : 25.750Gb/s
Rx Power Type : OMA
Manufacturing Date : 11_08_19
Active Set Host Compliance Code : N/A
Active Set Media Compliance Code : N/A
Error Code Response : N/A
Module FW Fault : N/A
DataPath FW Fault : N/A
Tx Fault [per lane] : 0,0,0,0
Tx LOS [per lane] : 0,0,0,0
Tx CDR LOL [per lane] : 0,0,0,0
Rx LOS [per lane] : 0,0,0,0
Rx CDR LOL [per lane] : 0,0,0,0
Tx Adaptive EQ Fault [per lane] : 0,0,0,0

→set ip address to enP5p1s0f0np0 : 192.168.1.1

→set ip address to enP5p1s0f1np1 : 192.168.1.2

$ ifconfig

enP5p1s0f0np0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet 192.168.1.1 netmask 255.255.255.0 broadcast 0.0.0.0
inet6 fe80::1d65:9cb4:9391:a20b prefixlen 64 scopeid 0x20
ether 9c:63:c0:35:d0:74 txqueuelen 1000 (Ethernet)
RX packets 432288 bytes 28567863 (28.5 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 7592298 bytes 68420957219 (68.4 GB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enP5p1s0f1np1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet 192.168.1.2 netmask 255.255.255.0 broadcast 0.0.0.0
ether 9c:63:c0:35:d0:75 txqueuelen 1000 (Ethernet)
RX packets 7591968 bytes 68420923709 (68.4 GB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 432708 bytes 28615843 (28.6 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lan1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether 20:46:a1:00:00:00 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0x30000000-300fffff

lan2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.50.87 netmask 255.255.255.0 broadcast 192.168.50.255
inet6 fe80::17ec:218f:5191:2c4f prefixlen 64 scopeid 0x20
ether 20:46:a1:0e:fb:58 txqueuelen 1000 (Ethernet)
RX packets 6080 bytes 524176 (524.1 KB)
RX errors 0 dropped 2818 overruns 0 frame 0
TX packets 2017 bytes 351130 (351.1 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0x30300000-303fffff

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1000 (Local Loopback)
RX packets 386 bytes 40759 (40.7 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 386 bytes 40759 (40.7 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

$ ping -I enP5p1s0f0np0 192.168.1.2

ping: Warning: source address might be selected on device other than: enP5p1s0f0np0
PING 192.168.1.2 (192.168.1.2) from 192.168.50.87 enP5p1s0f0np0: 56(84) bytes of data.
^C
— 192.168.1.2 ping statistics —
20 packets transmitted, 0 received, 100% packet loss, time 19447ms

$ ping -I enP5p1s0f1np1 192.168.1.1

ping: Warning: source address might be selected on device other than: enP5p1s0f1np1
PING 192.168.1.1 (192.168.1.1) from 192.168.50.87 enP5p1s0f1np1: 56(84) bytes of data.
^C
— 192.168.1.1 ping statistics —
26 packets transmitted, 0 received, 100% packet loss, time 25585ms

How should I configure my BF3 card to ensure it can ping successfully?

Thanks

James

jtal · March 11, 2026, 7:39am

Hi,

This does not look like a BF3 issue.
Seems like linux is still choosing lan2 to ping from, instead of the BF3 ports:
PING 192.168.1.2 (192.168.1.2) from 192.168.50.87 enP5p1s0f0np0: 56(84) bytes of data.

Please try running ping like so:

ping -I 192.168.1.1 192.168.1.2
ping -I 192.168.1.2 192.168.1.1

Thanks,

Jonathan.

ping -I 192.168.1.1 192.168.1.2
ping -I 192.168.1.2 192.168.1.1

james.hsu2 · April 16, 2026, 9:49am

Hi @jtal / Nvidia,

We use ‘ib_send_bw’ and ‘ib_write_wb’ to test ,

ib_send_bw :

server :

$ ib_send_bw -d mlx5_0 -x 3 -q 4 -t 256

client :

$ ib_send_bw -d mlx5_0 -F -x 3 -q 4 -t 256 --duration 30 192.168.1.2

test result :

------- WARNING: BW peak won't be measured in this run.
-------------------------------------------------------------------------------
                    Send BW Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 4		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 RX depth        : 512
 CQ Moderation   : 1
 Mtu             : 4096[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 0[B]
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
-------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0063 PSN 0x1d3e8f
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 local address: LID 0000 QPN 0x0065 PSN 0x4386cd
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 local address: LID 0000 QPN 0x0067 PSN 0x4a1493
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 local address: LID 0000 QPN 0x0069 PSN 0xb173e6
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x0062 PSN 0xb06439
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x0064 PSN 0xefbf66
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x0066 PSN 0xd3082
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
 remote address: LID 0000 QPN 0x0068 PSN 0xba9f72
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:01
-------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 65536      4000             0.00               13147.49		   0.210360

--------------------------------------------------------------------------

ib_write_bw