How do I set the subnet rate?

I have a Mellanox MSB-7890 externally managed switch and I have five nodes now that all have a Mellanox ConnectX-4 dual port 4x EDR 100 Gbps card (MCX456A-ECAT).

One of the nodes runs the subnet manager.

All of the nodes are running CentOS 7.6.1810 with ‘Infiniband Support’ package group installed.

When I go to run ibdiagnet, it shows:

I---------------------------------------------------

I- IPoIB Subnets Check

I---------------------------------------------------

I- Subnet: IPv4 PKey:0x7fff QKey:0x00000b1b MTU:2048Byte rate:10Gbps SL:0x00

W- Suboptimal rate for group. Lowest member rate:40Gbps > group-rate:10Gbps

However, when I run iblinkinfo, this is what it shows:

CA: aes0 mlx5_0:

0x7cfe9003004431f8 1 1 ==( 4X 25.78125 Gbps Active/ LinkUp)==> 3 5 "SwitchIB Mellanox Technol

ogies" ( )

CA: aes2 mlx5_0:

0x248a0703002b1ec6 4 1 ==( 4X 25.78125 Gbps Active/ LinkUp)==> 3 3 "SwitchIB Mellanox Technol

ogies" ( )

CA: aes3 mlx5_0:

0x248a0703002b1eca 2 1 ==( 4X 25.78125 Gbps Active/ LinkUp)==> 3 2 "SwitchIB Mellanox Technol

ogies" ( )

CA: aes4 mlx5_0:

0x248a0703002b1ece 6 1 ==( 4X 25.78125 Gbps Active/ LinkUp)==> 3 1 "SwitchIB Mellanox Technol

ogies" ( )

Switch: 0xec0d9a0300224e70 SwitchIB Mellanox Technologies:

3 1 ==( 4X 25.78125 Gbps Active/ LinkUp)==> 6 1 “aes4 mlx5_0” ( )

3 2 ==( 4X 25.78125 Gbps Active/ LinkUp)==> 2 1 “aes3 mlx5_0” ( )

3 3 ==( 4X 25.78125 Gbps Active/ LinkUp)==> 4 1 “aes2 mlx5_0” ( )

3 4 ==( 4X 25.78125 Gbps Active/ LinkUp)==> 5 1 “aes1 mlx5_0” ( )

3 5 ==( 4X 25.78125 Gbps Active/ LinkUp)==> 1 1 “aes0 mlx5_0” ( )

How do I set the subnet manager speed rate to be something higher than 10 Gbps?

I tried looking through the documentation and doing research online and it doesn’t appear that anybody else has encountered this before?

(I found this out while I was trying to find out what the MTU was on the switch. All of the cards are set to ‘connected mode’ with a MTU of 4092 (/etc/sysconfig/network-scripts/ifcfg-ib0).)

Your help is greatly appreciated.

Thank you.

Also, for /etc/rdma/partitions.conf, where would I be able to find what the valid options are for mtu= and rate=?

I’m trying to look for it and RedHat published it, but I have no way of checking or confirming that those options are valid for /etc/rdma/partitions.conf and none of the Mellanox documentation says it as well.

(And I also couldn’t find an OpenSM mailing list to join either where I can ask this question to the developers.)

Thank you.

So I literally copied and pasted this into /etc/rdma/partitions.conf and it is still showing me that the subnet rate is only 10 Gbps.

(I found the IBTA specification that says what the valid values are for the rates and mtus.)

Hello Ewen,

Many thanks for posting your question on the Mellanox Community.

Also in this post you are mentioning that you are using the CentOS INBOX driver, for which we do not provide support. Support needs to be obtained from the Linux Distro.

As you mentioned that you are using an example partition.conf file from the RHEL distro, please do the following.

Empty the ‘partition.conf’ file, and add only the following line and restart OpenSM.

Default=0x7fff, ipoib, mtu=5, defmember=full : ALL=full, ALL_SWITCHES=full,SELF=full;

This will remove all rate settings on the partitions.

Many thanks,

~Mellanox Technical Support

Yeah, I tried that as well.

Didn’t work either.

When I run ibdiagnet, it still showed that the link speed for the subnet manager is 10 Gbps as opposed to 100 Gbps.

From other documentation online, it states that rate=3 is the default, although in reading the IBTA “Infiniband Architecture Release 1.3 - Volume I - General Specification”, Table 224 PathRecord, p. 984, it doesn’t explicitly state that (at least not that table).

So I’m not really sure why the /etc/rdma/partitions.conf file isn’t taking despite the changes.

Thank you.

Did you figure this out? I set rate=7 (40gb according to that same table you found) on my QDR cards and the warning/error went away. That is,

Default=0x7fff, ipoib, mtu=5, rate=7, defmember=full : ALL=full, ALL_SWITCHES=full,SELF=full;

I’d guess that can be set according to any rate in that chart, copied here:

Component: Rate

Length(bits): 6

Offset(bits): 442

Description: Enumeration of the rate:

2: 2.5 Gb/sec.

3: 10 Gb/sec.

4: 30 Gb/sec.

5: 5 Gb/sec.

6: 20 Gb/sec.

7: 40 Gb/sec.

8: 60 Gb/sec.

9: 80 Gb/sec.

10: 120 Gb/sec.

When SA ClassPortInfo:CapabilityMask2.IsExtended-SpeedsSupported,

11: 14 Gb/sec. (FDR 1x)

12: 56 Gb/sec. (FDR 4x)

13: 112 Gb/sec. (FDR 8x)

14: 168 Gb/sec. (FDR 12x)

15: 25 Gb/sec. (EDR 1x)

16: 100 Gb/sec. (EDR 4x)

17: 200 Gb/sec. (EDR 8x)

18: 300 Gb/sec. (EDR 12x)

0, 1, 19-63: reserved

Otherwise:

0, 1, 11-63: reserved

In a query request, if RateSelector is not specified

(i.e., the ComponentMaskBit for RateSelector is 0) or

if it is specified and has a value of 3, this component

is ignored.