We are using 2 Bluefield 3 DPUs directly connected to each other.
Below is the output of lspci -xxxvvv | grep Mellanox at each server.
0000:ac:00.0 Ethernet controller: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller (rev 01)
Subsystem: Mellanox Technologies Device 0013
0000:ac:00.1 Ethernet controller: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller (rev 01)
Subsystem: Mellanox Technologies Device 0013
0000:ac:00.2 DMA controller: Mellanox Technologies MT43244 BlueField-3 SoC Management Interface (rev 01) (prog-if 00 [8237])
Subsystem: Mellanox Technologies Device 0013
0000:ac:00.0 Ethernet controller: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller (rev 01)
Subsystem: Mellanox Technologies Device 0013
0000:ac:00.1 Ethernet controller: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller (rev 01)
Subsystem: Mellanox Technologies Device 0013
0000:ac:00.2 DMA controller: Mellanox Technologies MT43244 BlueField-3 SoC Management Interface (rev 01) (prog-if 00 [8237])
Subsystem: Mellanox Technologies Device 0013
We set the link type to ethernet by using mlxconfig.
We are using QSFP112 Passive Copper Cable at 400G QSFP112 PCC Passive Copper Cable 2m | FiberMall
The problem is that the link is active (<UP,BROADCAST,RUNNING,MULTICAST> in ifconfig output), and the led is green (not blinking.), but the IP packets are not transferred between servers.
The ethtool out is as below.
sudo ethtool ens5f0np0
Settings for ens5f0np0:
Supported ports: [ Backplane ]
Supported link modes: 1000baseT/Full
1000baseKX/Full
10000baseT/Full
10000baseKR/Full
10000baseR_FEC
40000baseKR4/Full
40000baseCR4/Full
40000baseSR4/Full
40000baseLR4/Full
25000baseCR/Full
25000baseKR/Full
25000baseSR/Full
50000baseCR2/Full
50000baseKR2/Full
100000baseKR4/Full
100000baseSR4/Full
100000baseCR4/Full
100000baseLR4_ER4/Full
50000baseSR2/Full
1000baseX/Full
10000baseCR/Full
10000baseSR/Full
10000baseLR/Full
10000baseER/Full
50000baseKR/Full
50000baseSR/Full
50000baseCR/Full
50000baseLR_ER_FR/Full
50000baseDR/Full
100000baseKR2/Full
100000baseSR2/Full
100000baseCR2/Full
100000baseLR2_ER2_FR2/Full
100000baseDR2/Full
200000baseKR4/Full
200000baseSR4/Full
200000baseLR4_ER4_FR4/Full
200000baseDR4/Full
200000baseCR4/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: None BaseR RS
Advertised link modes: 1000baseT/Full
1000baseKX/Full
10000baseT/Full
10000baseKR/Full
10000baseR_FEC
40000baseKR4/Full
40000baseCR4/Full
40000baseSR4/Full
40000baseLR4/Full
25000baseCR/Full
25000baseKR/Full
25000baseSR/Full
50000baseCR2/Full
50000baseKR2/Full
100000baseKR4/Full
100000baseSR4/Full
100000baseCR4/Full
100000baseLR4_ER4/Full
50000baseSR2/Full
1000baseX/Full
10000baseCR/Full
10000baseSR/Full
10000baseLR/Full
10000baseER/Full
50000baseKR/Full
50000baseSR/Full
50000baseCR/Full
50000baseLR_ER_FR/Full
50000baseDR/Full
100000baseKR2/Full
100000baseSR2/Full
100000baseCR2/Full
100000baseLR2_ER2_FR2/Full
100000baseDR2/Full
200000baseKR4/Full
200000baseSR4/Full
200000baseLR4_ER4_FR4/Full
200000baseDR4/Full
200000baseCR4/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: RS
Link partner advertised link modes: Not reported
Link partner advertised pause frame use: No
Link partner advertised auto-negotiation: Yes
Link partner advertised FEC modes: Not reported
Speed: 400000Mb/s
Duplex: Full
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000004 (4)
link
Link detected: yes