I am running different perftests like ib_send_bw on two servers with ConnectX-7s over an Infiniband network. The lmc is set to 3.
The following are the commands I use:
Receiver:./perftest/ib_send_bw -d mlx5_0
Sender: ./perftest/ib_send_bw -d mlx5_0 --dlid 48 [RECEIVER_IP]
(48 is the base LID for mlx5_0 on the receiver)
This works perfectly. However, if I keep the server command the same, but change the client command to
Sender: ./perftest/ib_send_bw -d mlx5_0 --dlid 49 [RECEIVER_IP]
it fails, despite 49 also being a valid LID for the receiver.
Why is this the case, and how can I fix this issue?