The p2pBandwidthLatencyTest only run on 1 GPU

Accelerated Computing CUDA CUDA Programming and Performance
1

Hi, I’m trying to check the GPUs on SYS-4029GP-TRT3 and I ran into this error.

My system:
cuda: 10.0.130
driver: 410.57
8x RTX 2080 ti

As the result below, I ran the p2pBandwidthLatencyTest build from the this repo: https://github.com/NVIDIA/cuda-samples/tree/master/Samples/p2pBandwidthLatencyTest. I got the result of only 1 GPU. While running the nvidia-smi command, I can get all the GPUs.

[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, GeForce RTX 2080 Ti, pciBusID: 24, pciDeviceID: 0, pciDomainID:0

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0
     0	     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0 
     0 528.25 
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0 
     0 530.74 
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0 
     0 532.91 
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0 
     0 531.46 
P2P=Disabled Latency Matrix (us)
   GPU     0 
     0   2.07 

   CPU     0 
     0   8.67 
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0 
     0   2.07 

   CPU     0 
     0   8.70 

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

Edit: I checked on the nvlink status using

nvidia-smi nvlink -s

and found that all the links are inactive. Is this the reason the test cannot work on all the GPUs?

Please let me know if there is any problem with my system.
Thank you,
Chung

2

This might a result of having the CUDA_VISIBLE_DEVICES environment variable set.