I have 2 A100 cards, when switching between them with CUDA_VISIBLE_DEVICES on is ~ 2x slower than the other

Accelerated Computing CUDA CUDA Programming and Performance
1

I have 2 if the same A100 cards on my box. When I run with
CUDA_VISIBLE_DEVICES=0 or CUDA_VISIBLE_DEVICES=0 ,1 my program runs 2x faster than when I run it with CUDA_VISIBLE_DEVICES=1. How do I diagnose the issue? Is it hardware related (bus, something else) or software related?

Thanks in advance!

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.199.02   Driver Version: 470.199.02   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA A100 80G...  Off  | 00000000:17:00.0 Off |                    0 |
| N/A   36C    P0    64W / 300W |   7238MiB / 80994MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA A100 80G...  Off  | 00000000:CA:00.0 Off |                    0 |
| N/A   35C    P0    48W / 300W |      2MiB / 80994MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    719773      C   ../Release/myprocess          7236MiB |
+-----------------------------------------------------------------------------+
2

The output from nvidia-smi would be more interesting if it were captured with the application actively running on GPU0 and GPU1, respectively. This might reveal issues with power or thermals. The GPU utilization shown as 0% for both GPUs indicates that both were idling / not in use when this snapshot was captured.

3

to look for configuration differences between the 2 GPUs, the expanded output from nvidia-smi may be useful as well:

nvidia-smi -a

4

CARD 0 -
Tx Throughput : 9000 KB/s
Rx Throughput : 14000 KB/s

CARD 1 -
Tx Throughput : 8000 KB/s
Rx Throughput : 22000 KB/s

Temperature differs by 1 degree, power draw by .5 W

5

To check for a HW-related issue, you would want to extract power draw and temperature data from nvidia-smi while the application is actively using the GPU (that is, GPU utilization near 100%). Can you show that data?

6

Here’s the same job running at the same time on each card:

I’ve made %age into 0-100 and power is in W

image
image1538×720 87.3 KB

7

I wonder if the PCIe bus for card 0 is in Gen3 mode rather than Gen4, or only x8 instead of x16.

Again from nvidia-smi -a while the task is running, is the “PCI” section the same between the two?

    GPU Link Info
        PCIe Generation
            Max                 : N/A
            Current             : N/A
        Link Width
            Max                 : N/A
            Current             : N/A
8

From the power plot one plausible hypothesis is that PCIe auxiliary power is not being supplied to the slower GPU. The PCIe slot itself can supply at most 75W, and it seems that GPU is pretty much stuck below that limit. Check the auxiliary power connectors. [Later:] Looking at a picture of an A100 80 GB, it does not seem to have traditional auxiliary power connectors? If so, no idea how to check that it is properly being supplied with power. Maybe ask the system integrator you acquired this system from for assistance.

Check all instances of Slowdown and SW Power Cap in the output of nvidia-smi -q while the app is using the GPU. Are any of them shown as Active? You may also want to compare Performance State and the various Clocks in the output from nvidia-smi, while the app is actively using each GPU.

9

The behavior is very sporadic. I cannot reproduce it now… I thought that it may be due to a “first” time run where my compile/link settings weren’t correct but I deleted my Release folder and cmake/make from scratch and can’t reproduce. I will do the -a while the programs are running and get back to you

Thanks!

10

This is great advice! Thanks I will pursue all of these ideas