A computer has 6 Tesla GPU’s. It has two hard disks with Windows 7/Cuda 4 on one and Fedora 12 Linux/Cuda 3.2 on the other. I use the Cuda command SetDevice(0…5) to select which GPU the program uses. SetDevice is the only command that I alter for this test. When running Linux the application program runs just fine on all of the GPU’s 0…5. When running Windows the application program runs just fine on GPU 0. It produces a frizzed out image for the other GPU’s 1…5. The images is distorted and the normally black/white color comes out purple. Obviously something is wrong. When I installed the Tesla device driver on Windows I don’t remember doing any special setup for GPU 0 that I did not do for the other GPU’s. Any idea why the GPU’s are not acting consistently the same on Windows? Is there a stress test that one can run to make sure a GPU is functioning properly?
Mystery solved. After the port I moved the SetDevice(0) command to a new location. The program still worked fine so I thought nothing of it. Here’s the deal though: it would have worked fine no matter where I would have put the SetDevice(0) command. Why? Because 0 is the default device. Moving the SetDevice(0) command from one location to another is what messed me up later when I set the device to another number, say for example 3. Several cuda commands were run before the SetDevice was run. They would always run on gpu 0 where as the commands after set device would run on the new gpu. See examples below. It’s a silly boneheaded error in retrospect. If you have the same behavior, check for this.
examples:
cuda command 1 (runs on device 0)
cuda command 2 (runs on device 0)
cuda command 3 (runs on device 0)
SetDevice(0)
cuda command 4 (runs on device 0)
cuda command 5 (runs on device 0)
cuda command 6 (runs on device 0)
cuda command 1 (runs on device 0)
cuda command 2 (runs on device 0)
cuda command 3 (runs on device 0)
SetDevice(3)
cuda command 4 (runs on device 3)
cuda command 5 (runs on device 3)
cuda command 6 (runs on device 3)