Unable to set exclusive compute mode using nvidia-smi

Accelerated Computing CUDA CUDA Programming and Performance
1

I just installed the new CUDA Toolkit 2.2 along with the latest (185.18.08) nvidia driver. We have a Tesla S1070 1U box that is part of a batch system, and for this reason would really like the exclusive compute mode. However, I can’t seem to get nvidia-smi to set it. Here is how I’m running it:

nvidia-smi -g 0 -c 1

and I’ve also tried:
nvidia-smi --gpu=0 --compute-mode-rules=1

When I run it, I then check the compute mode in my program using cudaGetDeviceProperties(), but it never comes back with the exclusive mode. Is there something I’m missing here? The new syntax for compute modes seems to be completely useless. Any insights here woudl be greatly appreciated.

Thanks,
Dan Hansen

2

You need to have nvidia-smi running in the background:

nvidia-smi --loop-continuously --interval=60 --filename=/var/log/nvidia-smi.log &

[root@compute-0-0 ~]# nvidia-smi -g 1 -c 1
[root@compute-0-0 ~]# nvidia-smi -g 0 -c 1
[root@compute-0-0 ~]# nvidia-smi -g 1 -s
Compute-mode rules for GPU=0x1: 0x1
[root@compute-0-0 ~]# nvidia-smi -g 0 -s
Compute-mode rules for GPU=0x0: 0x1

3

Hi,

Can you be a little clearer on what this is doing and why it’s necessary?

When you set the compute mode for a given GPU, where is that information stored? Does it “go away” after some time interval, unless the “–loop-continuously” command continually revives it? This doesn’t make sense to me.

Thanks in advance,

Jim

4

Part of the kernel module will unload itself when no client of the driver is running. nvidia-smi is a client, so it ensures that the configuration data doesn’t get reset.

We’re going to fix this in a future driver release.

5

Thanks, that makes sense. It’s working for me now.

6

For those who wants to set the exlusive compute mode on each card of the TESLA S1070, don’t forget to type these commands also for card 2 and 3

[root@compute-0-0 ~]# nvidia-smi -g 3 -c 1

[root@compute-0-0 ~]# nvidia-smi -g 2 -c 1

[root@compute-0-0 ~]# nvidia-smi -g 1 -c 1

[root@compute-0-0 ~]# nvidia-smi -g 0 -c 1

[root@compute-0-0 ~]# nvidia-smi -g 3 -s

Compute-mode rules for GPU=0x3: 0x1

[root@compute-0-0 ~]# nvidia-smi -g 2 -s

Compute-mode rules for GPU=0x2: 0x1

[root@compute-0-0 ~]# nvidia-smi -g 1 -s

Compute-mode rules for GPU=0x1: 0x1

[root@compute-0-0 ~]# nvidia-smi -g 0 -s

Compute-mode rules for GPU=0x0: 0x1