Assign non-default pkey without using child adapter

I’m trying to partition my IB network using ipoib.
I was able to do this by creating a child on my ib0 adapter with the pkey of another partition.
For example:
ib0 (default pkey 0xfff)
ib0.0a00 (0xa0 pkey)

The problem is that I don’t want the nodes to be able to talk over the default network. I want only nodes on 0xa0 to talk to themselves and to the head node adapter on 0xa0.

Is there a way to either change the pkey of the ib0 adapter from the default or disable that adapter?
Everything I have tried has given me permission denied.
I kinda suspect that I just need to do away with a default ALL=full if I want to do this and manually assign GUIDS to partitions in opensm.

Maybe there’s a value for pkey I can put in the ifcfg file that assigns it a pkey at boot?

Hello,

Hope you are doing well

You can change the pkey of the ib0 adapter from the default by setting the index0 in the partitions.conf. When the IPoIB driver is loaded, it creates one interface for each port using the P_Key at index 0.

https://docs.kernel.org/infiniband/ipoib.html

"You can create sub-interfaces for a primary IPoIB interface to provide traffic isolation. Each such sub-interface (also called a child interface) has a different IP and network addresses from the primary (parent) interface. The default Partition Key (PKey), ff:ff, applies to the primary (parent) interface. Disabling the parent interface would result in your sub-interfaces losing connectivity. "

https://docs.nvidia.com/networking/display/mlnxofedv23100541/ip+over+infiniband+(ipoib)#src-144707070_safe-id-SVBvdmVySW5maW5pQmFuZChJUG9JQiktU3ViLWludGVyZmFjZXM

As my visibility into your environment is limited, therefore the guidance provided is of a high-level overview. Would having multiple limited pkeys be feasible for your use case? (members in the partitions can only talk to other members in the same partition)(limited members can talk to SM (full members)).

If you have a valid support contract, please open a support ticket with us, that way we can assist you further in detail

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.