I am running a ROS2 application inside docker on AGX Orin. I want to then use nsight system to do profiling of the ROS2 application that I am running.
Inside nsys-ui (local host), I am selecting “attach or launch” option as “target application”. Under command line with argument, I am writing the command that I write in terminal while running that ROS2 application. But when I do that, it is not able to find the path of the application. I use same command in terminal and I can run it without any error.
Command I use:
ros2 launch <package_name> <executable_name>
I also have a realsense camera and when I use “realsense-viewer” command, I can see it running the camera. So the installation looks fine.
I can see the PID of the process but in PID I am not getting any CUDA traces output. I want to get CUDA traces as well, so I want to run the process directly through nsys-ui. But when I do that it is not able to run the ROS2 application at all.
To clarify, you are running from the GUI outside of the docker attempting to analyze inside the docker?
Have you taken a look through https://developer.nvidia.com/blog/nvidia-nsight-systems-containers-cloud/
Honestly, my first suggestion would be to use the CLI inside the docker.
Thank you for your quick reply.
I am running the GUI from inside the docker. Actually, I am using “attach or launch” option so I have to run it from inside the docker, right?
For installing the nsight system, I tried using
apt-get install -y cuda-nsight-systems-11-4 nsight-systems-2022.3.3, (cude version of my orin is 11.4 and the nsight system I want to use is 2022.3.3) but it cant find the package. So I just copied the nsight system folder from outside of the docker to the inside the docker. The nsight system, outside the docker, was installed through jetpack.
I have gone through the article that you mentioned. I found the command mentioned above to install the nsight system from that article only. But I think that command is for the x86_64 system and not for Jetson boards.
I will try using CLI now. I will let you know about my findings.
When I use CLI, I get this error.
FATAL ERROR: /build/agent/work/323cb361ab84164c/QuadD/Common/GpuTraits/Src/GpuTicksConverter.cpp(376): Throw in function QuadDCommon::TimestampType GpuTraits::GpuTicksConverter::ConvertToCpuTime(const QuadDCommon::Uuid&, uint64_t&) const
Dynamic exception type: boost::wrapexcept<QuadDCommon::NotFoundException>
[QuadDCommon::tag_message*] = No GPU associated to the given UUID
So what I am doing here is that I go to
target-linux-tegra-armv8 folder and then use
nsys profile from that folder. Is that the right way to run nsys profile from docker?
Are you on a tegra system in a docker???
I suspect you have the wrong version of Nsys for your archetecture.
Are you on a tegra system in a docker???
I am using “nvcr.io/nvidia/isaac/ros:aarch64-humble-nav2_6dfaf7adbe190f1181c3a0a2f2418760” docker image. It is nvidia’s image which is specifically created for ROS2 Humble. I am building this image on AGX Orin and I want to then run some processes inside the docker. If I want to profile these processes, I must run nsight inside the docker right?
To have nsight inside the docker, I tried modifying the docker file as per the article you mentioned above (I added
apt-get install -y cuda-nsight-systems-11-4 nsight-systems-2022.3.3 and some other dependencies inside my docker file.) But I think those commands are for x86_64 architecture and not for aarch_64 architecture. So it didn’t work.
Then I just copied the nsight that was installed during the flashing process from Orin to the docker image. When I copied the nsight, I got this
target-linux-tegra-armv8 folder inside the docker. In that folder I saw an executable named
nsys. So I used this to run a ros2 process.
ros2 launch <package_name> <lunch_file.launch.py>, I got the error that I showed in the previous reply.
So what I want to know is:
- Is this also a right way to run nsight inside the docker? Or I must install nsight inside the docker image?
- Is it possible to install nsight inside aarch_64 docker image?
I suspect you have the wrong version of Nsys for your architecture.
- How can I know which version of Nsys is right for my architecture?
I hope I was able to explain my problem properly.
@Andrey_Trachenko can you help here.
Yes, Nsight Systems needs to run inside the container. There are two main ways to get Nsight Systems there:
- Mount the installation directory from the host system:
sudo docker run -it --rm -v /opt/nvidia/nsight-systems/2022.5.2:/nsys ubuntu:latest bash
map the nsys directory
/nsys/bin/nsys profile sleep 1
specify your application here
- Install Nsight Systems directly inside the container. You will need to get
nsight-systems-2022.5.2.171-3255900_arm64.deb from JetPack and install inside the container. I’m not sure though how to easily extract it from the JetPack installer.