Nvvp: Internal profiling error 3959:999

cuda ver: 11.6 , and on Linux
==3554001== Error: Internal profiling error 3959:999.
We know that 999 is: CUPTI_ERROR_UNKNOWN = 999 , so not really helpful. And I do call “cudaProfilerStop()” before calling ‘exit(0)’ as mentioned somewhere but doesn’t help. And this error, unlike some other irritating warning/timestamp errors that happen with nvprof, is not benign: you get NO output if you are running nvvp. (while with some other errors you still get some analysis/vizualization)
Now how does one debug an issue like this? Error message is as unhelpful as it can get, and there’s really not much to delve into. The program itself has no issue when ran without profiler, and even with profiler the outputs are ok, so it is running fine but somehow the profiler has an issue with it.
P.S. And no, I don’t want to use nsys et. al, I want nvvp (& thus nvprof)
another mentions of similar phenomena: (there are more, but some are not relevant: like version mismatch issues leading to profiler error)

P.S.2: I do get “======== Profiling result:” with time and other info of different calls. But I really want to view the stuff in nvvp, which doesn’t work. So: profiling result printed in nvprof, but also that error is printed, and when ran from nvvp it doesnt show the results and just terminates.

Hi,

Agree, error code 999 i.e. CUPTI_ERROR_UNKNOWN doesn’t provide any insight about the issue. Can you pls tell whether you are collecting tracing (timing) information, or are you doing events/metrics profiling?

Can you pls provide these details:

  • nvprof command line
  • GPU information
  • CUDA driver version
  • minimal reproducer (if possible)
    nvidia-smi output would help to know some of these details.

nvprof output can be imported in nvvp. Use below command to generate the nvprof output
$ nvprof --export-profile output.prof <app> <app args>
And import it in nvvp using the option File → Import → nvprof

nvprof ./a.out
SMI output:
±----------------------------------------------------------------------------+
| NVIDIA-SMI 510.39.01 Driver Version: 510.39.01 CUDA Version: 11.6 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce … On | 00000000:65:00.0 On | N/A |
| 0% 51C P8 12W / 120W | 930MiB / 3072MiB | 1% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
(its a GTX 1060)
output.prof (512 KB)