I just got my MCX313A-BCBT HCAs and MC2207310-020 optical cable.
I installed and connected them without a hitch in Win2008R2 using WinOF-3_2_0 but when I ran a quick test to measure the bandwidth I got a very disappointing 1.37GB/s
The way I set up the benchmark is by creating a StarWind Ram Disk on my file server, sharing the directory, and then using Iometer to run a 1MB sequential read test with a 16 queue depth.
I tried varying the size of the read, and the queue depth along with a couple of other parameters but the results were always around 1.3GB/s.
My questions are:
Is my testing methodology flawed? If so, what’s the best way to test max bandwidth in Win2008R2?
What specific performance optimizations can I perform in the drivers to get closer to 40Gbps?
Someone pointed me the chelsio website which has white papers for 40GbE over SMB between Windows 2012 servers setup just like mine getting a whopping 36Gbps… Another poster confirmed that he’s getting 31Gbps over SMB using Mellanox ConnectX-2 QDR cards with IPoIB.
I’ve ordered myself a Mellanox 40GbE passive copper cable to see if my Fiber Optic FDR cable is what’s causing the problem.
Hi. I don’t know if you’re asking Bmac or me (the OP) but no, I’m not using iSCSI.
As I already posted, a forum member on ServeTheHome was able to get 2GB/s using a 20Gbps ConnectX IB card from one windows machine to another using plain network shares and a StarWind RAM disk. I’m simply trying to understand why my more advanced 40GbE ConnectX-3 won’t give me more than 1.3GB/s in the same environment.
I checked with the guy and he said he tried it both direct connect like I have mine, and through a 40Gbps switch and got the same 2GB/s results.
4322MB/s using Starwind RAMDisk as a network share!!!
As it turns out the Startech PCIE riser I was using in my render nodes was limiting the HCA to PCIe2 x8 which was having a disproportional effect on performance.
I had to jerry-rig the mobo out of the chassis so that I could install the 40GbE card upright temporarily for testing. Once it was running at PCIE3-x8 (confirmed using SIV) all was good.
My Mellanox card isn’t IPoIB though. It’s an actual 40Gbps Ethernet HCA. I’m sure the silicone is the same as the IB cards but it’s hard-wired to function as a pure Ethernet controller with the proper header and MAC address of an ethernet controller.
I appreciate the test suggestion but I’m really only interested in getting my 40GbE cards working as efficiently as they can in Windows using TCP/IP. 1.3GB/s is pretty crap considering 40GbE is theoretically 5GB/s and PCIe3 has max throughput of 6.5GB/s.
My rendering software uses mapped network drives so that’s what I’m testing against.
I do plan on migrating to CentOS in the future but for now Windows will have to do.
I’ll definitely try datagram mode, though I tried larger MTUs and it didn’t change anything.
I updated the firmware on both cards and installed Windows 2012 on my file server as the 4.2 drivers have a lot more options than the 3.2 ones do in Win2008R2.
There are 3 performance tuning options in the 4.2 drivers: Single Port, Multicast, and Single Stream. Which would be best suited for my type of application? I’m basically accessing large (500MB+) Binary scene files and lots of textures (5-200MB images) off of a RAID array.
have you tried turning TCP offload on the driver to OFF. that used to make some difference for me with 20Gb cards using Ethernet. they used to achieve 1800MB/s so you should get up to 3600MB/s or 3.6GB/s on 40Gb
Im guesing your not using a 40Gb Mellanox managed switch either, as i believe there are many benefits to doing so, like collision management etc.
well not multicast, either single port or single stream. But considering your connecting via windows mapped network, which uses CIFS, then thats your problem right there. CIFS just cant carry that much data.
You need a SCSI transport like iSCSI and then connect windows using iscsi initiator over your IB IP network. iSCSI on 20Gb Infiniband cards using firmware 2.7 and later get 1800MB/s read write over iSCSI on SCST Target.
Again id seriously suggest if you want real performance forget windows of any flavour as a San Target. It is very bad at target mode.
Use ubuntu to setup SCST. takes 30 mins. no need to recompile teh kernel it works just as good without that.
have you tried turning TCP offload on the driver to OFF.
No, I’ll give that a try.
But considering your connecting via windows mapped network, which uses CIFS, then thats your problem right there. CIFS just cant carry that much data.
A member of the Servethehome forums has a ConnectX 20Gb IB card that does 2000MB/s using the mapped starwind drive method in Windows2008R2, with IPoIB. So I figured my brand new ConnectX-3 40GbE card would get at least that…
Im guesing your not using a 40Gb Mellanox managed switch either, as i believe there are many benefits to doing so, like collision management etc.
That might have something to do with it. I believe he had his card connected to a switch.
You need a SCSI transport like iSCSI and then connect windows using iscsi initiator over your IB IP network. iSCSI on 20Gb Infiniband cards using firmware 2.7 and later get 1800MB/s read write over iSCSI on SCST Target.
My limited understanding of iSCSI targets is that they can only be accessed by one machine; is that correct? I’m a 3D Artist, not an IT Pro so I really have limited knowledge of these things. I use Maya for my work and it requires a project directory where it organizes the scene assets. Each scene references textures and geometry that have to be accessible in the same directory structure to all of the render slaves that will be working on the frames. Directory mapping is very convenient for this purpose. SCST may be fast but if it doesn’t accomplish what my specific job requires then it’s just not suitable.
Use ubuntu to setup SCST. takes 30 mins. no need to recompile teh kernel it works just as good without that.
I’m game to give linux a try, but Ubuntu has very limited support in 3D. I mostly use Autodesk and theFoundry products which are all RHL. It was always my intention to one day switch to CentOS.
Also make sure you use a LSI raid card or something better than mobo raid.