On another thread, it developed that the bandwidth that I am getting between host and device is not really what it should be.
First: I have a phenom II 940 cpu running at 3.0 GHz installed on a mid-range Gigabyte 790X motherboard. My graphics card is a GTX 260 and it is plugged into the X16 slot of the motherboard. There is only one card installed. My linux system is stock Suse 10.3, kept up to date with the patches. The present kernel version is 2.6.22.19-0.2. I’ve installed the latest 180.29 Nvidia driver and CUDA 2.1. (The 180.20 driver made no difference).
Executing bandwidthTest from the SDK gives me:
[codebox]./bandwidthTest --memory=pinned
Running on…
device 0:GT200
Quick Mode
Host to Device Bandwidth for Pinned memory
.
Transfer Size (Bytes) Bandwidth(MB/s)
33554432 2584.6
Quick Mode
Device to Host Bandwidth for Pinned memory
.
Transfer Size (Bytes) Bandwidth(MB/s)
33554432 3131.5
Quick Mode
Device to Device Bandwidth
.
Transfer Size (Bytes) Bandwidth(MB/s)
33554432 94451.0
&&&& Test PASSED
[/codebox]
I gather that PCIe 2.0 is meant to have about twice that bandwidth. I seem to only have PCIe 1.1 bandwidth.
I’ve check for BIOS settings, but found nothing regarding PCIe other than the frequency, which is 100 Hz by default, apparently (I have it set to “Auto”).
Note that “bandwidthTest” reports the device as generic “device 0:GT200”. I gather the windows 181.20 driver reports the specific model.
Can anyone suggest what I might do to probe the issue or change to bring my bandwidth to its full potential? I’ve google searched and looked over the system logs and found nothing. I suspect the problem is an NVIDIA or kernel-level PCIe driver issue.
With that disparity between HtoD and DtoH, the BIOS is probably setting the wrong PCIe credits. You could look at GPU-Z or something to confirm that it is 16x (I don’t know of any better method than that, sorry), but I feel like it’s a BIOS problem.
Thanks for the push… I poked around the Gigabyte webpage and learned that on this motherboard (a GA-MA790X-DS4) if I hit Cntrl-F1 when in the BIOS I can get at some “advanced” bios settings. There were a number of new options for PCIe - they were all “disabled” and I set them all to “auto”. So now I get:
[codebox] ./bandwidthTest --memory=pinned
Running on…
device 0:GeForce GTX 260
Quick Mode
Host to Device Bandwidth for Pinned memory
.
Transfer Size (Bytes) Bandwidth(MB/s)
33554432 5280.0
Quick Mode
Device to Host Bandwidth for Pinned memory
.
Transfer Size (Bytes) Bandwidth(MB/s)
33554432 5290.8
Quick Mode
Device to Device Bandwidth
.
Transfer Size (Bytes) Bandwidth(MB/s)
33554432 94576.6
&&&& Test PASSED
[/codebox]
In addition, the motherboard had a power socket specifically for PCIe. The idea is apparently that if one installs two graphics cards, the board needs some additional power. I soldered up an extension cable (its an IDE hard disk plug) and plugged in, on the theory that perhaps the GTX 260 might need the additional power. It looks like that might have made some difference (“bandwidthTest” properly identified the 260 after I did that, but perhaps that’s just voodoo…). But the main effect was the secret BIOS setting…
Alternatively, I could have burnt a small goat at midnight during a new moon while drinking frog’s blood, hopping on one leg and patting my stomach…(note jaded cynicism).
Oh, it keeps PCIe 2.0 turned off by default? Huh. That’s pretty terrible. Good to know, though. Wonder if that’s a 790FX/GX/whatever thing, because the only other place I’ve seen it is the MSI K9A2 790FX board.
I did upgrade the BIOS to prepare the motherboard for the Phenom II - perhaps it is a BIOS upgrade thing. I’m pretty sure immediately afterward I loaded the “Optimized Defaults” and rebooted before doing anything.
I can confirm that the 260 definitely needs to have the extra power plugged into the motherboard, particularly with these new settings. It seems to have been just unstable otherwise. That seems to suggest reasons for the occasional odd behavior with X11 before now. The GTX 200’s are the Dick Cheney’s of the graphics card world…power hungry…