Performance degradation when running on multiple GPUs

Accelerated Computing CUDA CUDA Programming and Performance
1

I have a OpengGL, GLSL application with CUDA to simulate particle system and rendering in stereo, which runs fine on my Laptop with Quadro K5000M. However, the frame rate is significantly degraded when I run it on a desktop with three K5000 running with a sync signal to project to 10 different displays.

Any suggestions on why I am seeing performance loss with three GPUs and how do I debug and eventually fix it? Thank you.

2

Not enough detail to come up with an explanation.
Many factors can be involved. For example, you’re simulating on one board, need to transfer the data over to the other devices, have less memory available because of the heavy display, and add additional syncing of some kind.

  • Please quantify your "significant frame rate degradation" in absolute numbers.
  • What's the exact system setup of both systems? (OS version, display driver version, resolution of these 10 displays, mirrored display or a huge desktop, running in mosaic mode, with "to project to 10 displays" do you mean actual projector hardware, what model, etc.)
  • What kind of sync signal do you mean? Are you running a dedicated Quadro Sync hardware board?
  • Are you running the CUDA simulation on one board and all devices need the data for display?
  • Do you use OpenGL interop on the single board system?
  • What's the size of the simulation data?
  • How does it behave with one K5000 and one display? How with two? Without the sync?
    • 3

    These are all excellent points, and I agree with Detlef that without this information it is impossible to diagnose the observed behavior.

    The last item strikes me as especially important in trying to figure out what may be going on here, as it provides the most controlled experiment possible. Using the exact same hardware platform and software environment, examine multi-GPU scaling by using first one, then two, finally three K5000 GPUs (while making no other changes). The repeat this sequence with different monitor counts. The result should be some sort of two-dimensional “shmoo” plot, with #GPUs and #monitors as the axis dimensions.