EMC VNX5200 10G iSCSI with Mellanox SX1012

Infrastructure & Networking Switches and Gateways Ethernet Switches
1

Hi

We are testing the VNX5200 10G iSCSI storage solution with 2 x VMware 5.5, on Supermicro server with ConnectX3 Pro EN 10G HBA cards in them, and it is all connected with Mellanox Cables to the SX1012.

The communication from the ConnectX3 Pro EN 10G card to the Mellanox SX1012 is working fine, but the Mellanox cable 40G-10G Hybrid cables with SFP+ - QSFP is not connecting.

I am trying to find out if there is some setting in the switch or on the cable it self that needs to be set to get it to work, but i cannot find any guides from Mellanox to get this t work.

Do you guys have any idea what we need to do?

Best Regards

Petter

2

Hi Petter,

From what I know - EMC are doing very strict certification of it’s solution including the cables selected for each solution.

Is there a document (by EMC) specifying the supported cables/switches for the VNX solution?

3

Hi

We have now tested all possible configurations, the POC setup here on SEC Datacom is:

2 x Supermicro servers with 1 x Mellanox ConnectX3 Pro 10-EN

1 x Mellanox SX1012

1 x EMC VNX 5200 with 2 contollers with 2x10G iSCSI each

We had all the Mellanox ports set @ 10G with no LAG. We used the same port and other ports, and we tried both VNX controllers. We tried 2 meter cables and 3 meter cables.

Tested cables:

Mellanox QSFP → SFP+ = VNX got and error that the cable was not supported, we could ping.

Mellanox Splitter → 4xSFP+ = VNX got and error that the cable was not supported, we could ping.

Mellanox SFP+ module → EMC SFP+ - SFP+ = VNX had no error, we could ping.

Mellanox SFP+ module → Mellanox SFP+ - SFP+ = VNX got and error that the cable was not supported, we could ping.

Mellanox SFP+ module → Noname SFP+ - SFP+ = VNX had no error, we could ping!! (Noname = low cost china cable)

The conclusion must be from a simple users point of view, that EMC 10G iSCSI port will not reconize a cable that is not a EMC cable or a cable that says it is a Mellanox cable, it works you can get data from storage to the servers, but you have a error in the EMC system.

Maybe it would be great that Mellanox would include a software that EMC reconize, or Mellanox could get in touch with EMC and put Mellanox cables into the 10G iSCSI firmware.

We are selling alot of VNX systems in Denmark, and i think that EMC VNX systems are the most sold storage system in the EMC world.

4

Also - I have a feeling the end customer in this case has a support contract with Mellanox. can you ask him to open a support case with Mellanox to handle this issue not via a technical forum.

5

HI Petter,

some questions:

"The communication from the ConnectX3 Pro EN 10G card to the Mellanox SX1012 is working fine, "

Not sure what it means, which cable is used in the above scnario

'but the Mellanox cable

40G-10G Hybrid cables with SFP+ - QSFP is not connecting."

This type of cable requires configuring the SX1012 switch port with the below command(s):

To use all 4 SFP+ ends of the cable:

switch(config)# interface ethernet 1/x module-type qsfp-split-4 force

To use all 2 SFP+ ends of the cable:

switch(config)# interface ethernet 1/x module-type qsfp-split-2 force

It’s explained in the MLNX-OS UM http://www.mellanox.com/related-docs/prod_management_software/MLNX-OS_VPI_v3_4_3002_UM.pdf section 5.1.1 (configuration example - 5.1.1.1)

The SX1012 supports to have all of it’s 12 ports to be split.

6

Hi

he link from the servers with MLNX HBA is working fine. But the link from the EMV VNX iSCSI 10G controller is not nonnecting.

We have tried both a splitter cable QSFP → 4x10G, and we have tried the QSFP ->10G.

But it seems that the EMC 10G port is not reconized by the Mellanox switch.

But we will try the split-4 force and see if that helps.

7

We have done the split4, but the VNX cannot be seen. Any ideas?

8

Hi Petter,

I’m sure it’s not first deployment of an EMC storage device with a Mellanox switch, is there some compatibility list of EMC and it’s certified switches?

is the issue with the physical link or some logical connection between the storage to the servers?

Can you check with the storage NIC vendor?

9

Hi Peter,

I’ve read through the post and have the following comments. Please note that Mellanox does not “lock” out any cable vendors. In other words, the SX1012 switch is open for use with any cable vendor and we do a fair amount of interoperability testing. EMC on the other had does approve specific cables for specific products such as the VNX5200 from my understanding. So, it is likely that when the VNX system reads the EEPROM in the cables you’re using, it doesn’t recognize it as an approved cable and hence doesn’t bring up the link. So, I’m guessing it is a cable issue and not a switch issue. When you purchased the VNX system, did it specify a list of approved cable vendors for this particular product line? Can you please contact your EMC supplier and ask them to provide this? I did a quick web search but couldn’t locate this information. Please keep us posted.

Thanks,

Branko.