I want build a IPoIB VLAN virtual network via VST.
I’m also made a partition and each vSwitch setting was completed.
But just only VLAN 0, 21 works fine.
vSphere 4.x was greate works!
But vSphere 5.1 U1 wasn’t…
Do you have any solutions?
Is there any godd news…
I’m tried.
Same results…
Partition10=0x8010
and
Below is vSphere vSwitch’s configuration
vSphere 5.1U1 doesn’t accept VLAN ID 8010.
I was configured VLAN ID to 10.
Below is result command that sm-info show
Frankly speak, I’m prefer uplink option like vSphere 4.x OFED driver.
But current vSphere 5.x OFED driver can’t support now…
What’s wrong?
Partition16=0x8016 works greatly…
P.S
If I configure partition partition10=0x10 then my Solaris accept that partition is 0x8010.
What does mean?
- Here it is…
~ # esxcfg-vswitch -l
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 128 6 128 1500 vmnic4,vmnic5
PortGroup Name VLAN ID Used Ports Uplinks
Management Network 2934 1 vmnic4,vmnic5
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch1 128 6 128 9000 vmnic0,vmnic1
PortGroup Name VLAN ID Used Ports Uplinks
NFS 0 1 vmnic0,vmnic1
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch2 128 17 128 1500 vmnic2,vmnic3
PortGroup Name VLAN ID Used Ports Uplinks
Simulation VM Network 2936 0 vmnic2,vmnic3
VPN VM Network 2931 0 vmnic2,vmnic3
Practice VM Network 2935 2 vmnic2,vmnic3
External VM Network 0 2 vmnic2,vmnic3
DHCP VM Network 2934 8 vmnic2,vmnic3
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch3 128 5 128 4092 vmnic_ib0
PortGroup Name VLAN ID Used Ports Uplinks
iSCSI02 VM Network 36 0 vmnic_ib0
iSCSI01 VM Network 35 0 vmnic_ib0
Heartbeat VM Network 34 0 vmnic_ib0
LM VM Network 33 0 vmnic_ib0
CSV VM Network 32 0 vmnic_ib0
FT 17 1 vmnic_ib0
vMotion 16 1 vmnic_ib0
~ #
- Below is my 4036’s partition.conf
Default= 0xffff , ipoib , mtu=5 : ALL=full ;
Partition10=0x10 , ipoib , mtu=5, defmember=full : ALL=full;
Partition11=0x11 , ipoib , mtu=5, defmember=full : ALL=full;
Partition12=0x12 , ipoib , mtu=5, defmember=full : ALL=full;
Partition13=0x13 , ipoib , mtu=5, defmember=full : ALL=full;
Partition14=0x14 , ipoib , mtu=5, defmember=full : ALL=full;
Partition15=0x15 , ipoib , mtu=5, defmember=full : ALL=full;
Partition16=0x16 , ipoib , mtu=5, defmember=full : ALL=full;
Partition17=0x17 , ipoib , mtu=5, defmember=full : ALL=full;
Partition18=0x18 , ipoib , mtu=5, defmember=full : ALL=full;
Partition19=0x19 , ipoib , mtu=5, defmember=full : ALL=full;
Partition20=0x20 , ipoib , mtu=5, defmember=full : ALL=full;
Partition21=0x21 , ipoib , mtu=5, defmember=full : ALL=full;
Partition22=0x22 , ipoib , mtu=5, defmember=full : ALL=full;
Partition23=0x23 , ipoib , mtu=5, defmember=full : ALL=full;
Partition24=0x24 , ipoib , mtu=5, defmember=full : ALL=full;
Partition25=0x25 , ipoib , mtu=5, defmember=full : ALL=full;
Partition26=0x26 , ipoib , mtu=5, defmember=full : ALL=full;
Partition27=0x27 , ipoib , mtu=5, defmember=full : ALL=full;
Partition28=0x28 , ipoib , mtu=5, defmember=full : ALL=full;
Partition29=0x29 , ipoib , mtu=5, defmember=full : ALL=full;
Partition30=0x30 , ipoib , mtu=5, defmember=full : ALL=full;
Partition31=0x31 , ipoib , mtu=5, defmember=full : ALL=full;
Partition32=0x32 , ipoib , mtu=5, defmember=full : ALL=full;
Partition33=0x33 , ipoib , mtu=5, defmember=full : ALL=full;
Partition34=0x34 , ipoib , mtu=5, defmember=full : ALL=full;
Partition35=0x35 , ipoib , mtu=5, defmember=full : ALL=full;
Partition36=0x36 , ipoib , mtu=5, defmember=full : ALL=full;
Partition37=0x37 , ipoib , mtu=5, defmember=full : ALL=full;
Partition38=0x38 , ipoib , mtu=5, defmember=full : ALL=full;
Partition39=0x39 , ipoib , mtu=5, defmember=full : ALL=full;
Partition40=0x40 , ipoib , mtu=5, defmember=full : ALL=full;
Partition41=0x41 , ipoib , mtu=5, defmember=full : ALL=full;
Partition42=0x42 , ipoib , mtu=5, defmember=full : ALL=full;
Partition43=0x43 , ipoib , mtu=5, defmember=full : ALL=full;
Partition44=0x44 , ipoib , mtu=5, defmember=full : ALL=full;
Partition45=0x45 , ipoib , mtu=5, defmember=full : ALL=full;
Partition46=0x46 , ipoib , mtu=5, defmember=full : ALL=full;
Partition47=0x47 , ipoib , mtu=5, defmember=full : ALL=full;
Partition48=0x48 , ipoib , mtu=5, defmember=full : ALL=full;
Partition49=0x49 , ipoib , mtu=5, defmember=full : ALL=full;
- Below is result that response vmkping and ping each hosts
(* checked partition in only works… )
- Default= 0xffff , ipoib , mtu=5 : ALL=full ;
Partition10=0x10 , ipoib , mtu=5, defmember=full : ALL=full;
Partition11=0x11 , ipoib , mtu=5, defmember=full : ALL=full;
Partition12=0x12 , ipoib , mtu=5, defmember=full : ALL=full;
Partition13=0x13 , ipoib , mtu=5, defmember=full : ALL=full;
Partition14=0x14 , ipoib , mtu=5, defmember=full : ALL=full;
Partition15=0x15 , ipoib , mtu=5, defmember=full : ALL=full;
-
Partition16=0x16 , ipoib , mtu=5, defmember=full : ALL=full; vMotion
-
Partition17=0x17 , ipoib , mtu=5, defmember=full : ALL=full; FT
-
Partition18=0x18 , ipoib , mtu=5, defmember=full : ALL=full; IPoIBSAN01
-
Partition19=0x19 , ipoib , mtu=5, defmember=full : ALL=full; IPoIBSAN02
-
Partition20=0x20 , ipoib , mtu=5, defmember=full : ALL=full;
-
Partition21=0x21 , ipoib , mtu=5, defmember=full : ALL=full;
-
Partition22=0x22 , ipoib , mtu=5, defmember=full : ALL=full;
-
Partition23=0x23 , ipoib , mtu=5, defmember=full : ALL=full;
-
Partition24=0x24 , ipoib , mtu=5, defmember=full : ALL=full;
-
Partition25=0x25 , ipoib , mtu=5, defmember=full : ALL=full;
Partition26=0x26 , ipoib , mtu=5, defmember=full : ALL=full;
Partition27=0x27 , ipoib , mtu=5, defmember=full : ALL=full;
Partition28=0x28 , ipoib , mtu=5, defmember=full : ALL=full;
Partition29=0x29 , ipoib , mtu=5, defmember=full : ALL=full;
Partition30=0x30 , ipoib , mtu=5, defmember=full : ALL=full;
Partition31=0x31 , ipoib , mtu=5, defmember=full : ALL=full;
-
Partition32=0x32 , ipoib , mtu=5, defmember=full : ALL=full; CSV VM
-
Partition33=0x33 , ipoib , mtu=5, defmember=full : ALL=full; LM VM
-
Partition34=0x34 , ipoib , mtu=5, defmember=full : ALL=full; Heartbeat VM
-
Partition35=0x35 , ipoib , mtu=5, defmember=full : ALL=full; iSCSI01 VM
-
Partition36=0x36 , ipoib , mtu=5, defmember=full : ALL=full; iSCSI02 VM
-
Partition37=0x37 , ipoib , mtu=5, defmember=full : ALL=full;
-
Partition38=0x38 , ipoib , mtu=5, defmember=full : ALL=full;
-
Partition39=0x39 , ipoib , mtu=5, defmember=full : ALL=full;
-
Partition40=0x40 , ipoib , mtu=5, defmember=full : ALL=full;
-
Partition41=0x41 , ipoib , mtu=5, defmember=full : ALL=full;
Partition42=0x42 , ipoib , mtu=5, defmember=full : ALL=full;
Partition43=0x43 , ipoib , mtu=5, defmember=full : ALL=full;
Partition44=0x44 , ipoib , mtu=5, defmember=full : ALL=full;
Partition45=0x45 , ipoib , mtu=5, defmember=full : ALL=full;
Partition46=0x46 , ipoib , mtu=5, defmember=full : ALL=full;
Partition47=0x47 , ipoib , mtu=5, defmember=full : ALL=full;
-
Partition48=0x48 , ipoib , mtu=5, defmember=full : ALL=full;
-
Partition49=0x49 , ipoib , mtu=5, defmember=full : ALL=full;
Default partition is always works.
But next 6 partitions are not works and next 10 partitions are works and …
What’s the my mistake?
Nice that You could use the information, and that You could verify also in your environment.
I agree that this information is essential, and should be in the documents.
Regards, Ingvar
I’m so sorry for my late reply.
That’s a vSwitch’s VLAN ID for nested virtualization test environment.
I prefer to test all of VMware and Microsoft virtuaization technologies.
Hi!
If I change to default MTU=2044 can help to solve this problem?
Well… when we configured our vSphere 5.1 just recently to run over IB, I found out that the decimal VLAN ID (1-4096) are converted to their hexadecimal equivalence when leaving the ESXiVLAN ID Pkey
1 0x0001
2 0x0002
9 0x0009
10 0x000A etc.
and the ones I use
230 0x00e6
223 0x00df
partition.conf on the subnetmanager host
vmotion223=0xdf, ipoib, sl=0, defmember=full : 0x2590ffff07a3a9, 0x2590ffff07a3aa, 0x2590ffff07cb9d, 0x2590ffff07cb9e;
pk0x0026=0xe6, ipoib, sl=0, defmember=full : SELF;
NFS230=0xe6, ipoib, sl=0, defmember=full : 0x2c90300a7cff2, 0x2590ffff07a3a9, 0x2590ffff07a3aa, 0x2590ffff07cb9d, 0x2590ffff07cb9e;
On the ESXi host, after configuring the VLAN 230 and 223 via the vCenter dSwitch
grep -v “0000” /proc/infiniband/mlx4_0/ports/1/pkeys/*
/proc/infiniband/mlx4_0/ports/1/pkeys/0:0xffff
/proc/infiniband/mlx4_0/ports/1/pkeys/2:0x80e6
/proc/infiniband/mlx4_0/ports/1/pkeys/3:0x80df
So the VLAN ID 230 gets translated to PKEY 0x00E6, which in “member=FULL” mode = 0x80E6
I hope this info will help. I could not find anything in the current documentation nor in previous versions. I did not test this on vSphere 4.1, so no idea how it was handled there.
Regards, Ingvar
Hi inbusiness,
The way of VLAN configuration for IPoIB on vSphere 4.x/5.x didn’t change.
I suspect that you may have the wrong configuration on 5.x.
Can you please provide more information.
-
Subnet Manager PKEY configuration file
-
vmnic configuration (esxcfg-vswitch -l on ESX, etc…)
Thank you for your kind reply~!
I’ll try it!
That’s a good information.
I’m also know that SRPT reside in default partition (0x7fff)
Thank you again…
that’s odd,
can you try to set the full membership bit (MSB #16) in the pkey value for all the entries.
For example, instead of:
Partition10=0x10, use:
Partition10=0x8010
By the way, I found the information in
https://git.ipxe.org/mirror/scst/.git/blob_plain?f=srpt/README https://git.ipxe.org/mirror/scst/.git/blob_plain?f=srpt/README
“”
SCSI RDMA Protocol (SRP) Target driver for Linux
=================================================
…
InfiniBand Partitions
Just like a VLAN allows to segment traffic on an Ethernet network partitions
allow to segment traffic on an InfiniBand network. Each InfiniBand partition
is identified by a partition key which is a 16-bit number. During fabric
initialization the subnet manager assigns one or more partition keys to
each InfiniBand port. For opensm partitions are defined in
/etc/opensm/partitions.conf. ib_srpt uses the partition with index 0. Which
partition key corresponds to index 0 can be found out by querying sysfs:
$ head /sys/class/infiniband//ports//pkeys/0
==> /sys/class/infiniband/mlx4_0/ports/1/pkeys/0 <==
0xffff
==> /sys/class/infiniband/mlx4_0/ports/2/pkeys/0 <==
0xffff
So in Linux it is stored under /sys/class/infiniband/ ,
and in ESXi under /proc/infiniband
The index file “0” holds the default 7fff pkey, and the following index entries 1,2,3 etc holds the created ones.
If the contetns of and index is 0x0000 it means that its unused. (i.e. grep -v "0000 " will list the indexes holding configured values)
You are correct!
I’m test 0x80e7.
~ # grep -v “0000” /proc/infiniband/mlx4_0/ports/1/pkeys/*
/proc/infiniband/mlx4_0/ports/1/pkeys/0:0xffff
/proc/infiniband/mlx4_0/ports/1/pkeys/1:0x8001
/proc/infiniband/mlx4_0/ports/1/pkeys/2:0x8002
/proc/infiniband/mlx4_0/ports/1/pkeys/3:0x8003
/proc/infiniband/mlx4_0/ports/1/pkeys/4:0x8004
/proc/infiniband/mlx4_0/ports/1/pkeys/5:0x8005
/proc/infiniband/mlx4_0/ports/1/pkeys/6:0x8006
/proc/infiniband/mlx4_0/ports/1/pkeys/7:0x8007
/proc/infiniband/mlx4_0/ports/1/pkeys/8:0x8040
/proc/infiniband/mlx4_0/ports/1/pkeys/9:0x80e7
9 means partition order by created in partitions.conf.
1 means HCA port number.
0x80e7 means real partition number.
This case VLAN ID 9 wasn’t worked, but 231 was working correctly…
(e7 is hexadecimal number of 231)
Default partition number is 0x7ffff.
Therefore user partition number starts with 0x8.
Thank you for your help.
P.S
to Mellanox Community steps.
Can you describe this procedure in next vSphere OFED user guide?
It’s a very important for your partner and admins.
In Windows environment, there is a limit with one partition/port. Just wondering how you were having mutiple partitions since I saw you list CSV/LM/HB/iSCSI?