Connect QSFP to dual SFP

Hello,

I have two Ubuntu machines, A and B. Machine A is equipped with a BlueField-2 card (100Gbps Ethernet, dual-port), while machine B has a ConnectX-6 network card (25Gbps Ethernet, dual-port).
I am using a QSFP28 to SFP28 DAC breakout cable to connect one port of the BlueField-2 card to both ports of the CX6 network card. However, strangely, only one port on machine B is working (the one connected by the first cable).

Here is output of ibstat on machine B

CA ‘mlx5_2’
CA type: MT4127
Number of ports: 1
Firmware version: 26.35.3502
Hardware version: 0
Node GUID: 0xb83fd20300192696
System image GUID: 0xb83fd20300192696
Port 1:
State: Active
Physical state: LinkUp
Rate: 25
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x00010000
Port GUID: 0xba3fd2fffe192696
Link layer: Ethernet
CA ‘mlx5_3’
CA type: MT4127
Number of ports: 1
Firmware version: 26.35.3502
Hardware version: 0
Node GUID: 0xb83fd20300192697
System image GUID: 0xb83fd20300192696
Port 1:
State: Down
Physical state: Disabled
Rate: 40
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x00010000
Port GUID: 0xba3fd2fffe192697
Link layer: Ethernet

Here is output of ifconfig on machine B,port ens14f0np0 and ens14f1np1 are on the CX6 network adapter.

docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
ether 02:42:f0:68:6d:f5 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ens11f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.44 netmask 255.255.255.0 broadcast 192.168.1.255
inet6 fe80::a242:3fff:fe34:8000 prefixlen 64 scopeid 0x20
ether a0:42:3f:34:80:00 txqueuelen 1000 (Ethernet)
RX packets 75133 bytes 5098311 (5.0 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 40586 bytes 3860875 (3.8 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0xc7120000-c713ffff

ens11f1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether a0:42:3f:34:80:01 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0xc7100000-c711ffff

ens14f0np0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.111.44 netmask 255.255.255.0 broadcast 192.168.111.255
inet6 fe80::ba3f:d2ff:fe19:2696 prefixlen 64 scopeid 0x20
ether b8:3f:d2:19:26:96 txqueuelen 1000 (Ethernet)
RX packets 313 bytes 101412 (101.4 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 20 bytes 1496 (1.4 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ens14f1np1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.111.45 netmask 255.255.255.128 broadcast 192.168.111.127
ether b8:3f:d2:19:26:97 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1000 (Local Loopback)
RX packets 10826 bytes 981774 (981.7 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 10826 bytes 981774 (981.7 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Here is output of ibstat on machine A

CA ‘mlx5_0’
CA type: MT41686
Number of ports: 1
Firmware version: 24.38.1002
Hardware version: 1
Node GUID: 0xb83fd2030054c116
System image GUID: 0xb83fd2030054c116
Port 1:
State: Down
Physical state: Disabled
Rate: 40
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x00010000
Port GUID: 0xba3fd2fffe54c116
Link layer: Ethernet
CA ‘mlx5_1’
CA type: MT41686
Number of ports: 1
Firmware version: 24.38.1002
Hardware version: 1
Node GUID: 0xb83fd2030054c117
System image GUID: 0xb83fd2030054c116
Port 1:
State: Active
Physical state: LinkUp
Rate: 25
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x00010000
Port GUID: 0xba3fd2fffe54c117
Link layer: Ethernet

Here is output of ifconfig on machine A:

docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
ether 02:42:e5:c2:34:fa txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ens11f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.43 netmask 255.255.255.0 broadcast 192.168.1.255
inet6 fe80::a242:3fff:fe34:809c prefixlen 64 scopeid 0x20
ether a0:42:3f:34:80:9c txqueuelen 1000 (Ethernet)
RX packets 441558 bytes 39350842 (39.3 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 287031 bytes 34946941 (34.9 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0xc7220000-c723ffff

ens11f1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether a0:42:3f:34:80:9d txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0xc7200000-c721ffff

ens14f1np1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet 192.168.111.43 netmask 255.255.255.0 broadcast 192.168.111.255
inet6 fe80::ba3f:d2ff:fe54:c117 prefixlen 64 scopeid 0x20
ether b8:3f:d2:54:c1:17 txqueuelen 1000 (Ethernet)
RX packets 111848467 bytes 483459752210 (483.4 GB)
RX errors 0 dropped 1966 overruns 0 frame 0
TX packets 157996435 bytes 701246606387 (701.2 GB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1000 (Local Loopback)
RX packets 42460 bytes 7151796 (7.1 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 42460 bytes 7151796 (7.1 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

tmfifo_net0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.100.1 netmask 255.255.255.0 broadcast 192.168.100.255
inet6 fe80::21a:caff:feff:ff02 prefixlen 64 scopeid 0x20
ether 00:1a:ca:ff:ff:02 txqueuelen 1000 (Ethernet)
RX packets 50601 bytes 4369920 (4.3 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 37211 bytes 3887597 (3.8 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Can you help me with this?

Breakout(split) cable only support switch to NIC port, can’t link NIC to NIC.

You can check FW RN select tested cable,

https://docs.nvidia.com/networking/display/connectx6firmwarev20392048lts/validated+and+supported+cables+and+switches

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.