Nsys cannot collect cuda information on Drive OS 5.1

@VickNV I removed the two other ports and redo the test, but the issue is the same.

Please try again with the exact steps (make sure all host&target components installed and flashed successfully) and use the host&target.

@VickNV Unfortunately except the " * on host system install DRIVE Software 10 successfully (both host and target components)." the target part is not installed (I cannot do much on this), I followed exactly the same steps, but the result is the same.
Maybe I can transfer the target stuff from pegasus to host?

Did you mean you haven’t run throught DRIVE Software 10 in SDK Manager again to make sure host components are all good and target is fresh?
We need you to help on that. I don’t know your problems is on host or target.
If you want to keep your current host and target, you can try on another host or target to narrow down further.

@VickNV as mentioned in previous post, the host is installed successfully but target is not installed successfully partially (on the host) and I did not flash the target but seems that the target files are intact

Could you create another topic for your target installation problem?
We can help with that at the same time.

just double-check. Are you using host of Ubuntu 18.04 on native x86?

@VickNV Sdkmanager install drive software 10.0 issue

@VickNV Hello Vick, thank you for your valuable support. In summary, I currently have a desktop which works with os5.2 with no error, work with 5.1 but no cuda api is collected due to injection failure. These are similar to using nsight 2021.3 which involves only the target system.
To start the issue again, what kind of information do you need?

Same desktop running nsight-sys 2019.3
target is 5.1.6

Analysis 00:00.000 Profiling has started.
Daemon 21328 00:00.001 Process was launched by the profiler, see /tmp/nvidia/nsight_systems/streams/pid_21328_stdout.log and stderr.log for program output
Daemon 21328 00:00.001 Profiler attached to the process.
Injection 21328 00:00.011 Common injection library initialized successfully.
Injection 21328 00:00.032 OS runtime libraries injection initialized successfully.
Injection 21328 00:00.040 Buffers holding CUDA trace data will be flushed on CudaProfilerStop() call.
Injection 21328 00:00.054 CUDA injection initialization failed.
Analysis 00:17.593 Profiling has stopped.
Analysis 21328 00:19.619 CUDA profiling stopped unexpectedly: Cannot initialize CUDA event collection.
Analysis 21328 00:19.619 No CUDA events collected. Does the process use CUDA?
Analysis 21328 00:19.619 Not all OS runtime libraries events might have been collected.
Analysis 21328 00:19.619 Number of OS runtime libraries events collected: 587.
Analysis 00:19.619 Number of GPU context switch events collected: 7.

target is 5.2: I can see the pretty cuda streaming profiling graph

||Analysis||00:00.000|Profiling has started.|
||Daemon|32582|00:00.000|Process was launched by the profiler, see /tmp/nvidia/nsight_systems/streams/pid_32582_stdout.log and stderr.log for program output|
||Daemon|32582|00:00.001|Profiler attached to the process.|
||Injection|32582|00:00.061|Common injection library initialized successfully.|
||Injection|32582|00:00.096|OS runtime libraries injection initialized successfully.|
||Injection|32582|00:00.108|Buffers holding CUDA trace data will be flushed on CudaProfilerStop() call.|
||Injection|32582|00:00.207|CUDA injection initialized successfully.|
||Analysis||00:20.644|Profiling has stopped.|
||Injection|32582|00:20.668|Number of CUPTI events produced: 42, CUPTI buffers: 20.|
||Analysis|32582|00:21.646|Not all CUDA events might have been collected.|
||Analysis|32582|00:21.646|Number of CUDA events collected: 10.|
||Analysis|32582|00:21.646|Not all OS runtime libraries events might have been collected.|
||Analysis|32582|00:21.646|Number of OS runtime libraries events collected: 880.|

Please list out the detailed steps and commands for my reproducing.
Please also share the output of “Diagnostics Summary”.
Thanks.

that is the diagnostic summary, probably not well formated:
for the os5.2

||Analysis||00:00.000|Profiling has started.|
||Daemon|32582|00:00.000|Process was launched by the profiler, see /tmp/nvidia/nsight_systems/streams/pid_32582_stdout.log and stderr.log for program output|
||Daemon|32582|00:00.001|Profiler attached to the process.|
||Injection|32582|00:00.061|Common injection library initialized successfully.|
||Injection|32582|00:00.096|OS runtime libraries injection initialized successfully.|
||Injection|32582|00:00.108|Buffers holding CUDA trace data will be flushed on CudaProfilerStop() call.|
||Injection|32582|00:00.207|CUDA injection initialized successfully.|
||Analysis||00:20.644|Profiling has stopped.|
||Injection|32582|00:20.668|Number of CUPTI events produced: 42, CUPTI buffers: 20.|
||Analysis|32582|00:21.646|Not all CUDA events might have been collected.|
||Analysis|32582|00:21.646|Number of CUDA events collected: 10.|
||Analysis|32582|00:21.646|Not all OS runtime libraries events might have been collected.|
||Analysis|32582|00:21.646|Number of OS runtime libraries events collected: 880.|

My steps:

I used a test program from nvidia called baseline, put under /home/nvidia

and use attach and launch:
command argument: ./baseline
command work directory: /home/nvidia

did this for all the tested pegasus.

Then connect to them and start profiling.
Hope it helps

@VickNV Any updates on this issue? Thanks

Sorry that I missed this topic. Where can I get the baseline program?

@VickNV any program using cudabaseline.tar.gz (580 KB)
is fine, but I can send to you if you want

I’ll install back to DRIVE Software 10 and have a try. Thanks.

@VickNV Thanks for the update

I have no problem running deviceQuery cuda sample application.
Below are the screenshots. FYI.


@VickNV Sorry for the late reply. I did not check DeviceQuery, but you did not check the cuda, so I am not sure if cuda stream will be collected.

My DRIVE Software 10 environment isn’t available now. Please try with DeviceQuery. Thanks.

@VickNV Unfortunately I cannot access the desktop which can work today. I will try it later, thanks

@VickNV I recently flashed a drive os 5.2.6. I tried nsight sys 2021.4.1 with CLI on the target, it still cannot profile cuda functions. What nsight version I shall use for 5.2.6? Thanks