I’m writing a CFD simulation program for GPUs. The program schedules the simulations, and can run paralelly 2 different simulations on the GTX295 I have. The problem is, that it’s not as fast as it should (at least I think).
Computer:
CPU: Intel Extreme 975
Motherboard: Asus P6T SE
Memory: 12 GB DDR3 (now on 1066MHz).
GPU: GTX295 (Asus)
OS: Windows Vista x64 Business
NVIDIA Control Panel Settings: PhysX OFF, Multi-GPU Support: ON
If I run only one simulation with a specific settings, then it finishes in 21m:40s, if I run 2 of them, they finish in 33m:22s and 29m:20s, so the total performance is only 33% bigger. The program does not send more than 200 MB during the total run-time between the GPU and the CPU, so the PCI Express cannot cause the difference.
I also tested the n-body from the SDK. If run only on one GPU, it measures 304 GFlops, if run on 2 (I have forced to run on two different devices), then 303 and 120.
Is it normal? Does anybody know what causes it?
Thanks in advance!
Yours sincerely:
Laszlo Daroczy
I don’t have any experience with MultiGPU, but, it seems out of these possibilities,
shared resource bottleneck
limited ||ism - no, Monte Carlo is embarassingly ||
Inefficient algorithm (not likely because Mark Harris & friends @NVIDIA have done very good, albeit sometimes arcane optimizations).
#1 is most likely because I assume the GPUs are also being used for display. I’m using a Tesla 1060 and have a Quadro 295 for the display. I noticed using the Quadro, results in lower than expected speed, which is probably due to driving the display (1024 * 768 * 4 bytes/pixel * 60Hz = 180 MiB/s bandwidth).
The Tesla card also has to be mapped to an extended part of the screen for CUDA to recognize it. This is a requirement for the regular display driver. The specialized Tesla driver doesn’t require you to map to extended desktop, but due to Windows Display Driver Model 1.0, you can only use 1 driver, and you need to use the regular driver along with another NVIDIA board if you want to have a display. Windows 7’s WDDM 1.1 does support multiple drivers. Mapping the Tesla to the desktop doesn’t affect performance because Tesla doesn’t have display circuitry to begin with.
I suggest you try the following to reduce the sharing between CUDA and the display driver functions:
choose the lowest display resolution and bit depth
disable compositing (Aero for Vista, Compiz or anything that uses OpenGL)
try mapping the 2nd GPU to a different part of the screen (maybe you can find way to disable display all together)
Usually Aero is disabled for me, and the second GPU is mapped as a PhysX device. The bigger problem is, that in this case, sometimes random mistakes happen in the data, and it causes problems for my program. External Media
I simply have no idea, why is this happening. Last time e.g. after some secs, in the 66,273 element of my mesh was simply modified which caused divergence in the algorithm… External Media
The 295 consumes <= 290W according to Wikipedia and 200W according eXtreme Power Supply Calculator. Maybe the cards aren’t getting enough power. Maybe you can find a tool to display the voltage of the 12V supply? I know some BIOSes show it, but I don’t know of any other programs - maybe GPU-Z?
You can also try underclocking using RivaTuner and see if that fixes the error. I’m using Tesla 1060 and never had a stability problem. I’ve asked what’s so good about Tesla in terms of reliability and didn’t get any convincing answer, but did see that Tesla significantly under clocks the memory.
