I have both a 2080 Ti and a 50 HX graphics card installed on my machine. I attempted to remotely connect to this machine using the NVIDIA system tools to analyze a program running on it. This program runs on the 2080 Ti and does not utilize the 50 HX. However, after profiling the target, I received the following two errors:
ActivityFlushAll returned 42: CUPTI_ERROR_CMP_DEVICE_NOT_SUPPORTED
ActivityFlushAll returned 39: CUPTI_ERROR_MULTIPLE_SUBSCRIBERS_NOT_SUPPORTED
Additionally, I am unable to see the CUDA HW subplot in the timeline view. nvidia-bug-report.log.gz (606.3 KB)
To clarify, you were connecting to the remote machine and launching a application there and then you saw these messages? (Rather than attempting to connect to an already running process?)
When Nsight Systems collects CUDA api and kernel information we use CUPTI, which is another NVIDIA tool. Unfortunately only one collector can use a CUPTI instance. It looks like what happened was that something else was already using CUPTI system wide and Nsys therefore could not use it.
My guess on likely candidate would be DCGM if you were running that.
Yes, I was connecting to the remote machine and launching a application there.
I didn’t install DCGM there. The OS is Ubuntu 22.04.4 LTS. I just installed CUDA 12.6 on it. Nothing else.
When I removed the 50HX GPU from the server, everything returned to normal.
The NVIDIA 50HX is a Crypto Mining Processor (CMP), and NVIDIA developer tools, such as profilers and debuggers, do not support these devices. As a result, CUPTI returns the error code CUPTI_ERROR_CMP_DEVICE_NOT_SUPPORTED when attempting to interact with them.
Ref: NVIDIA Development Tools Solutions - ERR_NVCMPGPU: | NVIDIA Developer