Error with running CUPTI PC Sampling sample

Accelerated Computing CUDA CUDA Programming and Performance
1

Hi,
I am trying to experiment with PC sampling sample provided in :

/usr/local/cuda-10.0/extras/CUPTI/sample/pc_sampling

I am able to build but while running the executable, I am getting the following error:

Device Name: GeForce RTX 2080 Ti
Device compute capability: 7.5
pc_sampling.cu:212: error: function cuptiActivityConfigurePCSampling(cuCtx, &configPC) failed with error CUPTI_ERROR_NOT_SUPPORTED.

Can some help me understand what is the issue here?

Couple of things we tried at our end that didn’t give any results were:

  1. Sampled at default sampling rate
  2. Increased the ARRAY_SIZE to 512 MB.

These were done to see if we can capture any events but there were none.

Any help is appreciated.

Thanks
SK

2

This is expected behavior.

[url]https://developer.nvidia.com/CUPTI-1_0[/url]

“Variations from the CUPTI found in the CUDA Toolkit 10.0
The CUPTI version bundled with the CUDA 10.0 Toolkit does not include the new metric APIs.
The CUPTI version bundled with the CUDA 10.0 Toolkit does not support compute capability 7.5 (Turing GPU architecture).”

You may wish to try the latest CUDA toolkit 10.1 U1 (10.1.168):

[url]https://developer.nvidia.com/CUPTI-CTK10_1u1[/url]

3

I tried to install the latest CUDA toolkit 10.1 U1 (10.1.168) from the link you provided.

In the requirements section, it says to use the 425.25 drivers provided in this link:

I selected the correct OS etc and followed the instruction provided:

sudo dpkg -i cuda-repo-ubuntu1804-10-1-local-10.1.168-418.67_1.0-1_amd64.deb
sudo apt-key add /var/cuda-repo-<version>/7fa2af80.pub
sudo apt-get update
sudo apt-get install cuda

Installation was complete. Checked the nvidia-smi to see if the driver version got updated. Unfortunately, When I checked the nvidia-smi I got this error:

Failed to initialize NVML: Driver/library version mismatch

I rebooted the machine and then all hell breaks loose. I am not able to boot my OS and the screen keeps flickering. I am not able to go to my serial console anymore since the screen freezes.

Any idea how to debug and roll back to my previous driver?

SK

4

If you are installing the CUDA toolkit, you need to follow the instructions in the linux install guide:

[url]Installation Guide Linux :: CUDA Toolkit Documentation

I won’t be able to walk you through that entire guide, line by line, asking if you have followed every step. For example if you didn’t follow the instructions carefully in section 2.7, that would potentially be problematic. If you’re installing this on a Optimus laptop, there may be additional challenges.

Beyond that, there are many questions about this in the CUDA setup forum.

5

The latest version of CUDA that you pointed gives me this message when I run the pc sample code.

Device Name: GeForce RTX 2080 Ti
Device compute capability: 7.5
sample is waived on this device, pc sampling is supported on devices with compute capability 5.2 or 6.0 and CC <= 7.2 i.e supported till volta

Is this something planned for a future release?
SK