I have a Jetson AGX Orin with Jetpack 6.0. I am new to Nsight tools and I am trying to use Nsight-systems to profile deepstream-app to see what Gstreamer plugins are running, on what hardware (like VIC/GPU etc.) they are running on and how long they are taking in the pipeline. I tried to run nsys on the Jetson Orin, but I am unable to install it. Then I tried to run nsys-ui remotely from my Ubuntu host, but I do not see any CUDA or plugin details. Can you please guide me? What am I missing?
Let me see if I am following here.
You have Jetpack with Nsight Systems and you tried to launch a collection on the Orin (cli or gui?), that failed, and so you tried to launch remote from an Ubuntu box (which is the more normal way to work with Orin). That collection worked, but you don’t see all the data you expect.
Can you attach the .nsys-rep file generated by the run? If not, can you show me what options you used?
Yes, you are following what I am saying correctly.
A few more details - I tried to profile deepstream-app that runs a custom Yolov8s model. Please see the attached text file exported from the .nsys-rep file. I am unable to upload the .nsys-rep file as it is.
deepstream-app_Yolov8s_2816*1024.txt (40.1 MB)
Yeah, I can’t work with that.
Can you give me a screen shot of the diagnositic summary (drop down):
Likewise if you could go to the analysis summary and scan down to the section that will give me exact options used?
Here they are.
Is profiling the Orin remotely the right way to use Nsight-systems on an Orin? Does it require installing anything on the Orin in particular?
When you call into the Orin from the host it will transfer the files it needs. And it is clever enough to only transfer the first time you run that device from that version of the host.
@liuyis can you help here?
Thank you for your quick response.
If the remote host transfers the files on the Orin, I should be able to run nsys on the Orin on CLI, right? I cannot run nsys on the Orin. Then, I did an “apt install nsight-systems” on the Orin and it installed 2021 version of Nsight without any errors. However, I still cannot get nsys to run on the Orin on CLI. What am I missing?
I think for Orin you can install Nsys from Jetpack through SDK Manager. What was the error you hit when trying to installing or running Nsys on Orin directly?
It just says “command not found”. I did not install from SDK manager, however. I just did a apt install.
The package from apt install is not specifically tested for Orin. Officially you’ll need to install Nsys through Jetpack for Jetson Linux (a.k.a. L4T), see JetPack SDK | NVIDIA Developer.
Regarding “command not found”, it is likely because the nsys executable wasn’t added into PATH. Usually it is installed at /usr/local or /opt/. I suggest searching for the nsys binary under these paths and use the absolute path. For example
$ find /usr/local -name "nsys"
$ find /opt -name "nsys"
Thank you, I found nsys was in /opt.
I am still wondering why I am not finding plugin level details when I profile deepstream-app remotely? Should I not profile remotely, but use nsys on the CLI on the Orin?
Whether you profile from CLI or remotely should not be related, although it won’t hurt trying the CLI.
I’m not very familiar with deepstream-app, could you ellaborate more regarding “plugin level details”? Is the “plugin” a separate process from deepstream?
Note that Nsys also has the concept of “plugin” itself, but I assume you are not referring to that from the context.
Thank you for your quick response, I really appreciate it.
I am talking about Gstreamer plugins like nvinfer, that are used in Deepstream. I want to see where the actual bottlenecks are, how data is moving from device-to-host and vice versa. I want to see if queues are building, and more profiling details like that.
I’m lacking the domain specific knowledges for Deepstream, I can only try to troubleshoot issues for Nsys itself.
Do you know how is the plugin running in deepstream? Is it a seperate process? Or is it running within the same process? If it’s the latter case I wonder if you have NVTX annotations to identify the regions on timeline.