Internal profiling error 4087:35.

Developer Tools Other Tools Visual Profiler and nvprof
1

Hello,
At the end of profiling with visual porfiler after execution of the program finishes I get the following error:
======== Error: CUDA profiling error.
==20424== Error: Internal profiling error 4087:35.

Program itself runs fine , cuda-memcheck doesn’t complain (though kernel instrumentation mode doesn’t seem to work)

Any info on what that error might mean and how to workaround it ?

Thanks.

Edit: I’m using latest sdk and latest drivers and windows10 64bit.

Update: I’m trying to isolate what function call is causing the problem and I discovered that it happens to choke on specific cuda calls, for instance, if don’t call cudaGetMemInfo function, then it goes further without an error until it complains about cudaCreateStream call (which I cannot workaround unfortunately).

2

Hi sergeyn,

Can you please provide more information like,

Exact version of CUDA toolkit
Which GPU are using

Are able to profile other apps successfully?
Or you are getting same error for any app?
Can you profile the apps successfully if you run as an administrator?

–
Thanks,
Ramesh

3

Hi Ramesh,

I was able to identify the problem. When I simplified the program to a few cuda calls, in between those 2 lines with the original error message a new line appeared saying something about non-admin user access. Then I checked that the setting which enables profiling for non-admin users and discovered it was reset to disabled, probably by a windows or driver update.

Basically the actual problem was that I didn’t get a proper error message. It would be nice if nvprof would provide more context in addition to internal error code number.

Thanks,
Sergey.

4

Hi Sergey,

Good to know that you solved the problem.

We have improved related messages in “CUDA toolkit 10.1 Update 2”. Which CUDA toolkit version you are using?

–
Thanks,
Ramesh

5

I’m using cuda 10.1.243, which seem to be the latest you have available for downlad.

Regards,
Sergey.

6

Hi，I also have the same error in a docker container. I use cuda10.0, and when I use nvprof to analyze the TensorFlow example, its output is as follows.
==24928== Error: Internal profiling error 4055:34.
======== Error: CUDA profiling error.
Is any detailed log where I can get?

7

==24928== Error: Internal profiling error 4055:34.
This error code indicates that application needs to be recompiled nvcc version 9.0 or later. Can you please recheck after recompiling the application?

8

I got the same problem too, I got 4142:999. error code, and I can’t find any way to remove it, is there any documents contain code detail? or what’s the meaning of 4142:999.

9

Second number in the duplet is the CUPTI error code, which can be found in the header cupti_result.h shipped as part of the CUPTI SDK with the CUDA toolkit. Error code 999 stands for CUPTI_ERROR_UNKNOWN which makes it difficult to decipher what went wrong.

Few error codes for quick ref:
CUPTI_ERROR_CUDA_COMPILER_NOT_COMPATIBLE = 34
CUPTI_ERROR_INSUFFICIENT_PRIVILEGES = 35
CUPTI_ERROR_OLD_PROFILER_API_INITIALIZED = 36
CUPTI_ERROR_UNKNOWN = 999