vSphere 5.1 U1 IPoIB VLAN support

I want build a IPoIB VLAN virtual network via VST.

I’m also made a partition and each vSwitch setting was completed.

But just only VLAN 0, 21 works fine.

vSphere 4.x was greate works!

But vSphere 5.1 U1 wasn’t…

Do you have any solutions?

Is there any godd news…

I’m tried.

Same results…

Partition10=0x8010

and

Below is vSphere vSwitch’s configuration

vSphere 5.1U1 doesn’t accept VLAN ID 8010.

I was configured VLAN ID to 10.

Below is result command that sm-info show

Frankly speak, I’m prefer uplink option like vSphere 4.x OFED driver.

But current vSphere 5.x OFED driver can’t support now…

What’s wrong?

Partition16=0x8016 works greatly…

P.S

If I configure partition partition10=0x10 then my Solaris accept that partition is 0x8010.

What does mean?

  • Here it is…

~ # esxcfg-vswitch -l

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch0 128 6 128 1500 vmnic4,vmnic5

PortGroup Name VLAN ID Used Ports Uplinks

Management Network 2934 1 vmnic4,vmnic5

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch1 128 6 128 9000 vmnic0,vmnic1

PortGroup Name VLAN ID Used Ports Uplinks

NFS 0 1 vmnic0,vmnic1

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch2 128 17 128 1500 vmnic2,vmnic3

PortGroup Name VLAN ID Used Ports Uplinks

Simulation VM Network 2936 0 vmnic2,vmnic3

VPN VM Network 2931 0 vmnic2,vmnic3

Practice VM Network 2935 2 vmnic2,vmnic3

External VM Network 0 2 vmnic2,vmnic3

DHCP VM Network 2934 8 vmnic2,vmnic3

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch3 128 5 128 4092 vmnic_ib0

PortGroup Name VLAN ID Used Ports Uplinks

iSCSI02 VM Network 36 0 vmnic_ib0

iSCSI01 VM Network 35 0 vmnic_ib0

Heartbeat VM Network 34 0 vmnic_ib0

LM VM Network 33 0 vmnic_ib0

CSV VM Network 32 0 vmnic_ib0

FT 17 1 vmnic_ib0

vMotion 16 1 vmnic_ib0

~ #


  • Below is my 4036’s partition.conf

Default= 0xffff , ipoib , mtu=5 : ALL=full ;

Partition10=0x10 , ipoib , mtu=5, defmember=full : ALL=full;

Partition11=0x11 , ipoib , mtu=5, defmember=full : ALL=full;

Partition12=0x12 , ipoib , mtu=5, defmember=full : ALL=full;

Partition13=0x13 , ipoib , mtu=5, defmember=full : ALL=full;

Partition14=0x14 , ipoib , mtu=5, defmember=full : ALL=full;

Partition15=0x15 , ipoib , mtu=5, defmember=full : ALL=full;

Partition16=0x16 , ipoib , mtu=5, defmember=full : ALL=full;

Partition17=0x17 , ipoib , mtu=5, defmember=full : ALL=full;

Partition18=0x18 , ipoib , mtu=5, defmember=full : ALL=full;

Partition19=0x19 , ipoib , mtu=5, defmember=full : ALL=full;

Partition20=0x20 , ipoib , mtu=5, defmember=full : ALL=full;

Partition21=0x21 , ipoib , mtu=5, defmember=full : ALL=full;

Partition22=0x22 , ipoib , mtu=5, defmember=full : ALL=full;

Partition23=0x23 , ipoib , mtu=5, defmember=full : ALL=full;

Partition24=0x24 , ipoib , mtu=5, defmember=full : ALL=full;

Partition25=0x25 , ipoib , mtu=5, defmember=full : ALL=full;

Partition26=0x26 , ipoib , mtu=5, defmember=full : ALL=full;

Partition27=0x27 , ipoib , mtu=5, defmember=full : ALL=full;

Partition28=0x28 , ipoib , mtu=5, defmember=full : ALL=full;

Partition29=0x29 , ipoib , mtu=5, defmember=full : ALL=full;

Partition30=0x30 , ipoib , mtu=5, defmember=full : ALL=full;

Partition31=0x31 , ipoib , mtu=5, defmember=full : ALL=full;

Partition32=0x32 , ipoib , mtu=5, defmember=full : ALL=full;

Partition33=0x33 , ipoib , mtu=5, defmember=full : ALL=full;

Partition34=0x34 , ipoib , mtu=5, defmember=full : ALL=full;

Partition35=0x35 , ipoib , mtu=5, defmember=full : ALL=full;

Partition36=0x36 , ipoib , mtu=5, defmember=full : ALL=full;

Partition37=0x37 , ipoib , mtu=5, defmember=full : ALL=full;

Partition38=0x38 , ipoib , mtu=5, defmember=full : ALL=full;

Partition39=0x39 , ipoib , mtu=5, defmember=full : ALL=full;

Partition40=0x40 , ipoib , mtu=5, defmember=full : ALL=full;

Partition41=0x41 , ipoib , mtu=5, defmember=full : ALL=full;

Partition42=0x42 , ipoib , mtu=5, defmember=full : ALL=full;

Partition43=0x43 , ipoib , mtu=5, defmember=full : ALL=full;

Partition44=0x44 , ipoib , mtu=5, defmember=full : ALL=full;

Partition45=0x45 , ipoib , mtu=5, defmember=full : ALL=full;

Partition46=0x46 , ipoib , mtu=5, defmember=full : ALL=full;

Partition47=0x47 , ipoib , mtu=5, defmember=full : ALL=full;

Partition48=0x48 , ipoib , mtu=5, defmember=full : ALL=full;

Partition49=0x49 , ipoib , mtu=5, defmember=full : ALL=full;


  • Below is result that response vmkping and ping each hosts

(* checked partition in only works… )


  • Default= 0xffff , ipoib , mtu=5 : ALL=full ;

Partition10=0x10 , ipoib , mtu=5, defmember=full : ALL=full;

Partition11=0x11 , ipoib , mtu=5, defmember=full : ALL=full;

Partition12=0x12 , ipoib , mtu=5, defmember=full : ALL=full;

Partition13=0x13 , ipoib , mtu=5, defmember=full : ALL=full;

Partition14=0x14 , ipoib , mtu=5, defmember=full : ALL=full;

Partition15=0x15 , ipoib , mtu=5, defmember=full : ALL=full;

  • Partition16=0x16 , ipoib , mtu=5, defmember=full : ALL=full; vMotion

  • Partition17=0x17 , ipoib , mtu=5, defmember=full : ALL=full; FT

  • Partition18=0x18 , ipoib , mtu=5, defmember=full : ALL=full; IPoIBSAN01

  • Partition19=0x19 , ipoib , mtu=5, defmember=full : ALL=full; IPoIBSAN02

  • Partition20=0x20 , ipoib , mtu=5, defmember=full : ALL=full;

  • Partition21=0x21 , ipoib , mtu=5, defmember=full : ALL=full;

  • Partition22=0x22 , ipoib , mtu=5, defmember=full : ALL=full;

  • Partition23=0x23 , ipoib , mtu=5, defmember=full : ALL=full;

  • Partition24=0x24 , ipoib , mtu=5, defmember=full : ALL=full;

  • Partition25=0x25 , ipoib , mtu=5, defmember=full : ALL=full;

Partition26=0x26 , ipoib , mtu=5, defmember=full : ALL=full;

Partition27=0x27 , ipoib , mtu=5, defmember=full : ALL=full;

Partition28=0x28 , ipoib , mtu=5, defmember=full : ALL=full;

Partition29=0x29 , ipoib , mtu=5, defmember=full : ALL=full;

Partition30=0x30 , ipoib , mtu=5, defmember=full : ALL=full;

Partition31=0x31 , ipoib , mtu=5, defmember=full : ALL=full;

  • Partition32=0x32 , ipoib , mtu=5, defmember=full : ALL=full; CSV VM

  • Partition33=0x33 , ipoib , mtu=5, defmember=full : ALL=full; LM VM

  • Partition34=0x34 , ipoib , mtu=5, defmember=full : ALL=full; Heartbeat VM

  • Partition35=0x35 , ipoib , mtu=5, defmember=full : ALL=full; iSCSI01 VM

  • Partition36=0x36 , ipoib , mtu=5, defmember=full : ALL=full; iSCSI02 VM

  • Partition37=0x37 , ipoib , mtu=5, defmember=full : ALL=full;

  • Partition38=0x38 , ipoib , mtu=5, defmember=full : ALL=full;

  • Partition39=0x39 , ipoib , mtu=5, defmember=full : ALL=full;

  • Partition40=0x40 , ipoib , mtu=5, defmember=full : ALL=full;

  • Partition41=0x41 , ipoib , mtu=5, defmember=full : ALL=full;

Partition42=0x42 , ipoib , mtu=5, defmember=full : ALL=full;

Partition43=0x43 , ipoib , mtu=5, defmember=full : ALL=full;

Partition44=0x44 , ipoib , mtu=5, defmember=full : ALL=full;

Partition45=0x45 , ipoib , mtu=5, defmember=full : ALL=full;

Partition46=0x46 , ipoib , mtu=5, defmember=full : ALL=full;

Partition47=0x47 , ipoib , mtu=5, defmember=full : ALL=full;

  • Partition48=0x48 , ipoib , mtu=5, defmember=full : ALL=full;

  • Partition49=0x49 , ipoib , mtu=5, defmember=full : ALL=full;


Default partition is always works.

But next 6 partitions are not works and next 10 partitions are works and …

What’s the my mistake?

Nice that You could use the information, and that You could verify also in your environment.

I agree that this information is essential, and should be in the documents.

Regards, Ingvar

I’m so sorry for my late reply.

That’s a vSwitch’s VLAN ID for nested virtualization test environment.

I prefer to test all of VMware and Microsoft virtuaization technologies.

Hi!

If I change to default MTU=2044 can help to solve this problem?

Well… when we configured our vSphere 5.1 just recently to run over IB, I found out that the decimal VLAN ID (1-4096) are converted to their hexadecimal equivalence when leaving the ESXiVLAN ID Pkey

1 0x0001

2 0x0002

9 0x0009

10 0x000A etc.

and the ones I use

230 0x00e6

223 0x00df

partition.conf on the subnetmanager host

vmotion223=0xdf, ipoib, sl=0, defmember=full : 0x2590ffff07a3a9, 0x2590ffff07a3aa, 0x2590ffff07cb9d, 0x2590ffff07cb9e;

pk0x0026=0xe6, ipoib, sl=0, defmember=full : SELF;

NFS230=0xe6, ipoib, sl=0, defmember=full : 0x2c90300a7cff2, 0x2590ffff07a3a9, 0x2590ffff07a3aa, 0x2590ffff07cb9d, 0x2590ffff07cb9e;

On the ESXi host, after configuring the VLAN 230 and 223 via the vCenter dSwitch

grep -v “0000” /proc/infiniband/mlx4_0/ports/1/pkeys/*

/proc/infiniband/mlx4_0/ports/1/pkeys/0:0xffff

/proc/infiniband/mlx4_0/ports/1/pkeys/2:0x80e6

/proc/infiniband/mlx4_0/ports/1/pkeys/3:0x80df

So the VLAN ID 230 gets translated to PKEY 0x00E6, which in “member=FULL” mode = 0x80E6

I hope this info will help. I could not find anything in the current documentation nor in previous versions. I did not test this on vSphere 4.1, so no idea how it was handled there.

Regards, Ingvar

Hi inbusiness,

The way of VLAN configuration for IPoIB on vSphere 4.x/5.x didn’t change.

I suspect that you may have the wrong configuration on 5.x.

Can you please provide more information.

  • Subnet Manager PKEY configuration file

  • vmnic configuration (esxcfg-vswitch -l on ESX, etc…)

Thank you for your kind reply~!

I’ll try it!

That’s a good information.

I’m also know that SRPT reside in default partition (0x7fff)

Thank you again…

that’s odd,

can you try to set the full membership bit (MSB #16) in the pkey value for all the entries.

For example, instead of:

Partition10=0x10, use:

Partition10=0x8010

By the way, I found the information in

https://git.ipxe.org/mirror/scst/.git/blob_plain?f=srpt/README https://git.ipxe.org/mirror/scst/.git/blob_plain?f=srpt/README

“”

SCSI RDMA Protocol (SRP) Target driver for Linux

=================================================

InfiniBand Partitions


Just like a VLAN allows to segment traffic on an Ethernet network partitions

allow to segment traffic on an InfiniBand network. Each InfiniBand partition

is identified by a partition key which is a 16-bit number. During fabric

initialization the subnet manager assigns one or more partition keys to

each InfiniBand port. For opensm partitions are defined in

/etc/opensm/partitions.conf. ib_srpt uses the partition with index 0. Which

partition key corresponds to index 0 can be found out by querying sysfs:

$ head /sys/class/infiniband//ports//pkeys/0

==> /sys/class/infiniband/mlx4_0/ports/1/pkeys/0 <==

0xffff

==> /sys/class/infiniband/mlx4_0/ports/2/pkeys/0 <==

0xffff


So in Linux it is stored under /sys/class/infiniband/ ,

and in ESXi under /proc/infiniband

The index file “0” holds the default 7fff pkey, and the following index entries 1,2,3 etc holds the created ones.

If the contetns of and index is 0x0000 it means that its unused. (i.e. grep -v "0000 " will list the indexes holding configured values)

You are correct!

I’m test 0x80e7.

~ # grep -v “0000” /proc/infiniband/mlx4_0/ports/1/pkeys/*

/proc/infiniband/mlx4_0/ports/1/pkeys/0:0xffff

/proc/infiniband/mlx4_0/ports/1/pkeys/1:0x8001

/proc/infiniband/mlx4_0/ports/1/pkeys/2:0x8002

/proc/infiniband/mlx4_0/ports/1/pkeys/3:0x8003

/proc/infiniband/mlx4_0/ports/1/pkeys/4:0x8004

/proc/infiniband/mlx4_0/ports/1/pkeys/5:0x8005

/proc/infiniband/mlx4_0/ports/1/pkeys/6:0x8006

/proc/infiniband/mlx4_0/ports/1/pkeys/7:0x8007

/proc/infiniband/mlx4_0/ports/1/pkeys/8:0x8040

/proc/infiniband/mlx4_0/ports/1/pkeys/9:0x80e7

9 means partition order by created in partitions.conf.

1 means HCA port number.

0x80e7 means real partition number.

This case VLAN ID 9 wasn’t worked, but 231 was working correctly…

(e7 is hexadecimal number of 231)

Default partition number is 0x7ffff.

Therefore user partition number starts with 0x8.

Thank you for your help.

P.S

to Mellanox Community steps.

Can you describe this procedure in next vSphere OFED user guide?

It’s a very important for your partner and admins.

In Windows environment, there is a limit with one partition/port. Just wondering how you were having mutiple partitions since I saw you list CSV/LM/HB/iSCSI?