Nsight Compute Error

[ashimabisla03]

Hello Team,
I have converted Pytorch model to TRT model and I could successfully profile it with nsight systems but while profiling with nsight compute, I am facing the following error:

[AastaLLL]

Hi,

The usage are different.

usage: ncu [options] [program] [program-arguments]

General Options:
  -h [ --help ]                         Print this help message.
  -v [ --version ]                      Print the version number.
  --mode arg (=launch-and-attach)       Select the mode of interaction with the target application:
                                          launch-and-attach
                                          (launch and attach for profiling)
                                          launch
                                          (launch and suspend for later attach)
                                          attach
                                          (attach to launched application)
  ...

For example:

$ sudo /opt/nvidia/nsight-compute/2022.2.1/ncu /usr/src/tensorrt/bin/trtexec --onnx=/usr/src/tensorrt/data/mnist/mnist.onnx

Thanks.

[ashimabisla03]

@AastaLLL

/usr/src/tensorrt/bin/trtexec --onnx=

Hi,
I tried the above example you shared and I am getting the following error :

Thanks

[AastaLLL]

Hi,

It looks like the TensorRT command is not working in your environment.
Could you share the complete log so we can get more information about the error?

Thanks.

[ashimabisla03]

Hi,
I have attached the log file :

error_log.log (13.8 KB)
Thanks

[AastaLLL]

Hi,

Could you try mnist.onnx to see if it works?

The command can work well in our environment.
We would like to know if the error is related to the environment or model first.

$ sudo /opt/nvidia/nsight-compute/2022.2.1/ncu /usr/src/tensorrt/bin/trtexec --onnx=/usr/src/tensorrt/data/mnist/mnist.onnx 
&&&& RUNNING TensorRT.trtexec [TensorRT v8401] # /usr/src/tensorrt/bin/trtexec --onnx=/usr/src/tensorrt/data/mnist/mnist.onnx
[11/21/2022-09:09:55] [I] === Model Options ===
...
[11/21/2022-09:10:01] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +137, GPU +134, now: CPU 2025, GPU 6461 (MiB)
[11/21/2022-09:10:01] [I] [TRT] Local timing cache in use. Profiling results in this builder pass will not be stored.
==PROF== Profiling "copyPackedKernel" - 0: 0%....50%....100% - 9 passes
==PROF== Profiling "copyPackedKernel" - 1: 0%....50%....100% - 9 passes
==PROF== Profiling "copyPackedKernel" - 2: 0%....50%....100% - 9 passes
==PROF== Profiling "copyPackedKernel" - 3: 0%....50%....100% - 9 passes
==PROF== Profiling "copyPackedKernel" - 4: 0%....50%....100% - 9 passes
==PROF== Profiling "copyPackedKernel" - 5: 0%....50%....100% - 9 passes
==PROF== Profiling "copyPackedKernel" - 6: 0%....50%....100% - 9 passes
==PROF== Profiling "copyPackedKernel" - 7: 0%....50%....100% - 9 passes
==PROF== Profiling "permutationKernelPLC3" - 8: 0%....50%....100% - 9 passes
...

Thanks.

[ashimabisla03]

Thanks for the response. Appreciated.

I have double checked that the issue what we have that is mostly with environment related.

Approach:
We have downloaded Nsight Compute Studio from the NVIDIA official sites and installed it in our local environment and we have Jetson AGX Orin as a Target Machine. Can you pls confirm that are we moving on right track?

Pls Suggest.

Thank you.

[AastaLLL]

Hi,

Did you use the package from the below link?
If not, please give it a try.

Or you should be able to install with apt directly.

$ sudo apt install nsight-compute-2022.2.1

Thanks.

[ashimabisla03]

Hi,

I guess you have not mentioned the link.
However, we tried with sudo apt for versions 2022.2.1 and 2021.2.2, but we are getting the following error :

Thanks

[AastaLLL]

Hi,

Could you share which JetPack version you use with us first?

$ cat /etc/nv_tegra_release

And the source list of the nvidia-l4t-apt-source.list file:

$ cat /etc/nv_tegra_release 

Thanks

[ashimabisla03]

Hi,
PFA screenshot of JetPack version and source list of nvidia-l4t-apt-source.list file:

Thanks

[AastaLLL]

Hi,

Please upgrade your environment to JetPack 5.0.2 GA, which should be r35.1.
Thanks.