Dual-port MCX653106A - running OpenSM on server, only one port goes active

Infrastructure & Networking Adapters and Cables InfiniBand/VPI Adapter Cards
1

Hi all,

Installed a MCX653106A-ECAT-SP into two servers, have them connected “back-to-back” on both ports. Running OpenSM thru OFED 23.07 install, but I see only one of the two connections goes State: Active (port 0, CA = mlx5_0) but the other (mlx5_1) is stuck in State: Initializing… When I list the single card in lspci, I see two instances as so:

root@server1:~# lspci | grep Mella
98:00.0 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]
98:00.1 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]

I found this article saying there must be an OpenSM running for each port:
Second ConnectX-5 IB adapter port remain in INITIALIZING state - how to change to ACTIVE?

But even though I followed the instructions to run opensm --create-config opensm.conf in /etc/opensm, and then I changed the guid attribute to have the two Port GUIDs, upon an opensm restart, still only port 0 goes Active… What must I do to get the seconf port active?

2

Hi Will,
This is because you only have subnet in port 1, and when you connect B2B, port1 and port 2 in differnet subnet.
You can use
opensm -g
opensm -g

Thanks,
Suo