nsight compute ui and cli can't profiling any cuda application

Developer Tools Nsight Compute
1

I think I found a bug of injection/hook module in nsight target package.
target application build by cuda toolkit 10.1, visual studio 2017.
nsight compute ui reports such an error in “Profile” Mode.
“profiler returned an error code: 3765269247”
but the app only does a simple check and return.

gpuErrchk(cudaSetDevice(0));

    gpuErrchk(cudaGetDeviceProperties(&deviceProp, 0));
    printf("GPU Device %d: \"%s\" with compute capability %d.%d\n\n", 0,
           deviceProp.name, deviceProp.major, deviceProp.minor);

    return true;

in interactive profile mode after resume first cuda api break, profiling ui disappears immediately.

test with Cuda 10.1 sample “simplePrintf”, launch from cli.bat gave same result:
“E:\Program Files\NVIDIA Corporation\Nsight Compute 2019.3.1\nv-nsight-cu-cli.bat” -o profile simplePrintf.exe
==ERROR== The application returned an error code (3765269347)
==WARNING== No kernels were profiled
==WARNING== Profiling kernels launched by child processes requires the --target-processes all option
and demo suite in 10.1 toolkit have same error after first cuda api call.
profiling on normal windows without cuda api call doesnt cause such a 3765269347 error (maybe hook mechanism havn’t trigged).

2019.3.0 and 2019.3.1 gives same error.
but these application working fine without profiler.
and I can profiling them in nsight system (with cuda options checked).

my os: windows 10 enterprise LTSC 2019, x64
cpu: E5-1650v4
ram: 128GB, with ECC
vga card: zotec RTX2070Super 8G pgf OC

2

I am unable to reproduce the issues you are seeing e.g. with simplePrintf on a comparable setup on my side (Windows 10, Turing GPU, Nsight Compute 2019.3.0 and nwer). Could you please let us know the exact Nvidia driver version you are using? As for the application you listed, would it be possible for you to provide the source to us, at least in a minimal form that reproduces the problem for you? Thank you.

3

here is the source.

I don’t think its caused by my code.
same source with different target (linux-x64 on ubuntu, RTX 2080 Ti) can profiling fine remotely from same windows host.
from google 3765269247 is an generic exception code from msc library functions.
I tried to check if there a LoadLibrary Error during cli injection, I used Api monitor, but failed to capture such error.

I updated driver from 431.36 to 431.60, but can’t resolve profiling problem.
Just introduced a new bug to my system: bios bootup/setup screen keeps black/blank after driver update. my monitor only have DVI plug so I used hdmi to DVI convertor, but the problem is another story I need to report to production but not develop forum.

4

Thank you for the details, We are working on repro’ing the problem internally. In the meantime, can you please double-check if performance counter collection is enabled for your driver in the Nvidia Control Panel? You would need to select “Desktop”, “Enable Developer Settings”, “Manage GPU Performance Counters”.

5

yes, it keeps enabled during nsight test, because I am used visual profiler in past which need same trick.
toggle between user/administrator, same error code.
my system volume is not C drive (pre-assigned by winntsetup), and my os is LTSC 2019, not cbb branch. but I think LTSC is more stable for a workstation usage.
I even tested studio driver 431.70 but have no luck.

6

The incoming cuda 10.1 Update 2 (bundle with nsight compute 2019.4) solved my problem, after upgrade, without any tweak, I can profiling cuda application now.

7

Glad to hear that, thanks for trying with the latest version.