How to configure Partition Key (PKey) to isolate different tenants' traffic on Mellanox M8790 switches in an IB network?

Infrastructure & Networking Switches and Gateways InfiniBand/VPI Switch Systems
1

Hello ,The current IB network environment consists of 2 M8700 switches and 10 M8790 switches, forming 4 spines and 8 leaf networks. Is there any official documentation or configuration examples for Partition Key that I can refer to for learning?

I have made the relevant configurations, but the isolation is not working as expected.

cat /etc/opensm/partitions.conf

Storage=0x8020,indx0,ipoib,defmember=full:0xe41d2d0300cac9f6,0xe41d2d0300cac9f7 ;
hpc=0x8021,indx1,ipoib,defmember=limited:0x946dae03009cc5b4,0x946dae03009cf751 ;

smpquery PkeyTable -D 0

0: 0x8020 0xffff 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
8: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
16: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
24: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
32: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
40: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
48: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
56: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
64: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
72: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
80: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
88: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
96: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
104: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
112: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
120: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
128 pkeys capacity for this port

2

https://docs.nvidia.com/networking/display/mlnxofedv23100550/nvidia+sm#src-2396915000_NVIDIASM-FileFormat

https://docs.nvidia.com/networking/display/mlnxofedv23100550/ip+over+infiniband+(ipoib)#src-2396915069_safe-id-SVBvdmVySW5maW5pQmFuZChJUG9JQiktU3ViLWludGVyZmFjZXM

1 Like
3

Thank you very much for your assistance. I am now able to configure the SM to achieve isolation between different tenants. I would like to know how to make a host running opensm become the SM in an IB network. Also, if the current SM host in the IB network fails, how can I make another host in the IB network become the SM host? I look forward to your response. Thank you!

4

Hi，I have same question,2 HPC to be isolated but both of them can access the STORAGE，I assigned 2 different pkey at index0 to HPC to make them isolated,it works but they can’t ping STORAGE too.

What’s the difference between full and limited?Is some thing like PVLAN(community or isolate)?

Thanks a lot.