I’m trying to setup partitions and PKEYS between two Linux hosts and a Mellanox SX6036 switch. I’ve defined a partition on the switch using a 0x0001 PKEY but all my Linux interfaces are being created as ib1.0x8001 (using the directions below). How do I get the PKEYS the same?
Following the Linux kernel documentation for Pkeys and partitions here: https://www.kernel.org/doc/Documentation/infiniband/ipoib.txt
Thank you for posting your inquiry on the Mellanox Community.
Based on your information provided, we want to share the following information.
There are special requirements for the names of IPoIB P_Key interfaces.
All IPoIB P_Keys range from 0x0000 to 0x7fff, and the high bit, 0x8000, denotes that membership in a P_Key is full membership instead of partial membership.
The Linux kernel’s IPoIB driver only supports full membership in P_Key subnets, so for any subnet that Linux can connect to, the high bit of the P_Key number will always be set. That means that if a Linux computer joins P_Key 0x0001, its actual P_Key number once joined will be 0x8001, denoting that we are full members of P_Key 0x0001.
For this reason, when creating a P_Key definition on a switch or in an opensm partitions.conf file on a node which is running OpenSM, it is required to specify a P_Key value without 0x8000, but when defining the P_Key IPoIB interfaces on the Linux clients, add the 0x8000 value to the base P_Key value.
Thanks and regards,
~Mellanox Technical Support