Memory bandwidth

Have you looked at the PCI tree (/sbin/lspci -tv) to see if there is any clue?

Massimiliano,

Yes, that’s with both x16 slots occupied (actually, both x16 slots and one x8 slot, all three occupied).

I’ll have to do the second-slot-only experiment and get back to you with those numbers. With luck I

ought to be able to get that done today.

Cheers,

John Stone

I just re-ran my bandwidth tests with only a single card installed in the second x16 slot, and I get exactly the same result, so having multiple cards in the machine isn’t affecting the results for that slot.

Here’s the PCI tree for the 3-GPU machine:

johns@toledo{26} /sbin/lspci -tv

-[0000:00]-±00.0 nVidia Corporation C55 Host Bridge

       +-00.1  nVidia Corporation C55 Memory Controller

       +-00.2  nVidia Corporation C55 Memory Controller

       +-00.3  nVidia Corporation C55 Memory Controller

       +-00.4  nVidia Corporation C55 Memory Controller

       +-00.5  nVidia Corporation C55 Memory Controller

       +-00.6  nVidia Corporation C55 Memory Controller

       +-00.7  nVidia Corporation C55 Memory Controller

       +-01.0  nVidia Corporation C55 Memory Controller

       +-01.1  nVidia Corporation C55 Memory Controller

       +-01.2  nVidia Corporation C55 Memory Controller

       +-01.3  nVidia Corporation C55 Memory Controller

       +-01.4  nVidia Corporation C55 Memory Controller

       +-01.5  nVidia Corporation C55 Memory Controller

       +-01.6  nVidia Corporation C55 Memory Controller

       +-02.0  nVidia Corporation C55 Memory Controller

       +-02.1  nVidia Corporation: Unknown device 03bc

       +-02.2  nVidia Corporation C55 Memory Controller

       +-03.0-[0000:01]----00.0  nVidia Corporation: Unknown device 0191

       +-09.0  nVidia Corporation MCP55 Memory Controller

       +-0a.0  nVidia Corporation MCP55 LPC Bridge

       +-0a.1  nVidia Corporation MCP55 SMBus

       +-0a.2  nVidia Corporation MCP55 Memory Controller

       +-0b.0  nVidia Corporation MCP55 USB Controller

       +-0b.1  nVidia Corporation MCP55 USB Controller

       +-0d.0  nVidia Corporation MCP55 IDE

       +-0e.0  nVidia Corporation MCP55 SATA Controller

       +-0e.1  nVidia Corporation MCP55 SATA Controller

       +-0e.2  nVidia Corporation MCP55 SATA Controller

       +-0f.0-[0000:02]----0b.0  VIA Technologies, Inc. IEEE 1394 Host Controller

       +-0f.1  nVidia Corporation MCP55 High Definition Audio

       +-11.0  nVidia Corporation MCP55 Ethernet

       +-12.0  nVidia Corporation MCP55 Ethernet

       +-13.0-[0000:03]----00.0  nVidia Corporation: Unknown device 0191

       +-14.0-[0000:04]--

       +-15.0-[0000:05]--

       +-16.0-[0000:06]--

       +-17.0-[0000:07]--

       \-18.0-[0000:08]----00.0  nVidia Corporation: Unknown device 0191

Here’s the PCI tree for the 1-GPU configuration in the second x16 slot:

johns@proteus{31} /sbin/lspci -tv

-[0000:00]-±00.0 nVidia Corporation C55 Host Bridge

       +-00.1  nVidia Corporation C55 Memory Controller

       +-00.2  nVidia Corporation C55 Memory Controller

       +-00.3  nVidia Corporation C55 Memory Controller

       +-00.4  nVidia Corporation C55 Memory Controller

       +-00.5  nVidia Corporation C55 Memory Controller

       +-00.6  nVidia Corporation C55 Memory Controller

       +-00.7  nVidia Corporation C55 Memory Controller

       +-01.0  nVidia Corporation C55 Memory Controller

       +-01.1  nVidia Corporation C55 Memory Controller

       +-01.2  nVidia Corporation C55 Memory Controller

       +-01.3  nVidia Corporation C55 Memory Controller

       +-01.4  nVidia Corporation C55 Memory Controller

       +-01.5  nVidia Corporation C55 Memory Controller

       +-01.6  nVidia Corporation C55 Memory Controller

       +-02.0  nVidia Corporation C55 Memory Controller

       +-02.1  nVidia Corporation: Unknown device 03bc

       +-02.2  nVidia Corporation C55 Memory Controller

       +-03.0-[0000:01]--

       +-06.0-[0000:02]--

       +-07.0-[0000:03]--

       +-09.0  nVidia Corporation MCP55 Memory Controller

       +-0a.0  nVidia Corporation MCP55 LPC Bridge

       +-0a.1  nVidia Corporation MCP55 SMBus

       +-0a.2  nVidia Corporation MCP55 Memory Controller

       +-0b.0  nVidia Corporation MCP55 USB Controller

       +-0b.1  nVidia Corporation MCP55 USB Controller

       +-0d.0  nVidia Corporation MCP55 IDE

       +-0e.0  nVidia Corporation MCP55 SATA Controller

       +-0e.1  nVidia Corporation MCP55 SATA Controller

       +-0e.2  nVidia Corporation MCP55 SATA Controller

       +-0f.0-[0000:04]----0b.0  VIA Technologies, Inc. IEEE 1394 Host Controller

       +-0f.1  nVidia Corporation MCP55 High Definition Audio

       +-11.0  nVidia Corporation MCP55 Ethernet

       +-12.0  nVidia Corporation MCP55 Ethernet

       +-13.0-[0000:05]--

       +-14.0-[0000:06]--

       +-15.0-[0000:07]--

       +-16.0-[0000:08]--

       +-17.0-[0000:09]--

       \-18.0-[0000:0a]----00.0  nVidia Corporation: Unknown device 0191

Both machines are running RHEL4u4 (kernel: 2.6.9-42.ELsmp) with the 1.0-9751 CUDA drivers etc.

The motherboard is an Asus P5N32-E SLI.

Cheers,

John Stone

Thanks for trying that for me, John. I’m planning to build a 3-card setup in the near future, so it’s good to know the bandwidth numbers as I design the basic code.

Jim

Jim,
Hopefully Massimilano or one of the other NVIDIA gurus can tell us if the bandwidths I’m getting on the second x16 slot are a result of the Linux kernel PCI mappings, a driver issue, or if this is simply due to the motherboard hardware implementation. I suspect that down the road it will become easy to build a machine that runs all three cards flat out. Until I benchmarked the machine using the page-locked memory allocations, I didn’t see any difference in the performance of the three slots. It’s only since I began using the page-locked memory buffers for GPU I/O (earlier this week) that I even began delving into this at this level.

Cheers,
John Stone

John,
I need to confirm it with the chipset folks, but the first 16x PCIe slot should hang out from the SPP (Northbridge) and the second one from the MPP (Southbridge).
The SPP and MPP are connected by an HT bus, on some motherboards it is possible to change the HT clock between the two in the BIOS.

Could you see if this option is available in your BIOS?

Massimiliano

I had access to a HP xw9400, that has 2 16x PCI-e slots.
The performance is the same in both slots (around 3.1 GB/s in pinned mode), so I think your slowdown is hardware related. Still trying to dig more info from the chipset group.

I did the test under Windows, I was using a Quadro FX 5600 and Linux in 32 bit mode had some issues with PCI mapping in the second slot. The HP workstation is using the NVIDIA Professional chipset.

Massimiliano,

Sorry for the delayed reply, I was travelling and without regular net access for the last several days. I don’'t think I have seen any BIOS options that sound like what you’re describing, but I’ll look at this closely when I’m at the lab tomorrow. Let me know if there’s a typical name for the particular chipset setting I should be trying to find on our test machines.

John

tachyon_john’s results confused me at first, because I know that we have about a 15% packet overhead on PCI-e transfers. This means the max you can expect to achieve on a standard 4GB/s PCI-e 16x connection is about 3.4 GB/s.

But John is getting almost 4 GB/s! Well, after checking around, someone pointed me to this:
[url=“http://www.nvidia.com/content/nforce5/TB-02423-001_v01_LinkBoost.pdf”]http://www.nvidia.com/content/nforce5/TB-0...1_LinkBoost.pdf[/url]

Turns out John has an ASUS P5N32-E motherboard, which features an NVIDIA nForce 680i chipset. This chipset has a feature known as “LinkBoost”, which basically automatically overclock’s PCI-e and HyperTransport buses.

So the max theoretical B/W (depending on how fast the chipset decides it can go) is 5 GB/s on this motherboard. With the packet overhead, that comes down to 4.25 GB/s. John is at about 94% of this number. Not bad.

So matching GeForce/Quadro boards with nForce motherboards has advantages. It’s nice to see them showing up in practice! :)

Edit: More info on 680i motherboards with multiple PCI-e slots.
[url=“http://www.nvidia.com/object/designed_by_nvidia_partners.html#680isli”]http://www.nvidia.com/object/designed_by_n...rs.html#680isli[/url]

Mark

More info on this. On your motherboard, the first x16 and the x8 are attached to the northbridge, and the second x16 is on the southbridge. It is limited by an HT8X link to about 2GB/s (probably slightly faster due to the LinkBoost overclocking, as your numbers demonstrate).

Apparently HP has an nForce Pro motherboard in their xw9400 workstation that has dual x16 slots that both get 3.2GB/s in our internal measurements.

Mark

Hi John, thanks for posting your benchmark figures. I’ve found in my benchmarks that some motherboards may produce good DMA performance when a DMA test is run on each card one at a time. But when two or three cards have DMA running at the same time, performance can drop quite dramatically.

Did you obtain the above performance figures when running DMA simultaneously to all three cards? Thanks.

Hi,

No, those tests were done individually to each board…

I’ve got some code I’m working on that will make it easy to benchmark aggregate multi-GPU PCIe bandwidth, GPU memory bandwidth, etc. I’ll probably add it into VMD as a “benchmark” command of some sort for convenience. I recently tested an HP opteron system that performed pretty well with 4 GPUs (two QuadroPlexes), I’d be curious what motherboard suggestions the NVIDIA folks might have since they undoubtably have a better grasp of the tradeoffs in chipset designs and what vendors are using chipsets that work well with their GPUs.

Cheers,

John