Can we run ros2 through nsight system as "attach or launch"?

Developer Tools Nsight Systems Profiling Linux Targets
1

Hi,

I am running a ROS2 application inside docker on AGX Orin. I want to then use nsight system to do profiling of the ROS2 application that I am running.

Inside nsys-ui (local host), I am selecting “attach or launch” option as “target application”. Under command line with argument, I am writing the command that I write in terminal while running that ROS2 application. But when I do that, it is not able to find the path of the application. I use same command in terminal and I can run it without any error.

Command I use:
ros2 launch <package_name> <executable_name>

I also have a realsense camera and when I use “realsense-viewer” command, I can see it running the camera. So the installation looks fine.

I can see the PID of the process but in PID I am not getting any CUDA traces output. I want to get CUDA traces as well, so I want to run the process directly through nsys-ui. But when I do that it is not able to run the ROS2 application at all.

2

To clarify, you are running from the GUI outside of the docker attempting to analyze inside the docker?

Have you taken a look through https://developer.nvidia.com/blog/nvidia-nsight-systems-containers-cloud/

Honestly, my first suggestion would be to use the CLI inside the docker.

3

Hi @hwilper,

Thank you for your quick reply.

I am running the GUI from inside the docker. Actually, I am using “attach or launch” option so I have to run it from inside the docker, right?

For installing the nsight system, I tried using apt-get install -y cuda-nsight-systems-11-4 nsight-systems-2022.3.3, (cude version of my orin is 11.4 and the nsight system I want to use is 2022.3.3) but it cant find the package. So I just copied the nsight system folder from outside of the docker to the inside the docker. The nsight system, outside the docker, was installed through jetpack.

I have gone through the article that you mentioned. I found the command mentioned above to install the nsight system from that article only. But I think that command is for the x86_64 system and not for Jetson boards.

I will try using CLI now. I will let you know about my findings.

4

When I use CLI, I get this error.

FATAL ERROR: /build/agent/work/323cb361ab84164c/QuadD/Common/GpuTraits/Src/GpuTicksConverter.cpp(376): Throw in function QuadDCommon::TimestampType GpuTraits::GpuTicksConverter::ConvertToCpuTime(const QuadDCommon::Uuid&, uint64_t&) const
Dynamic exception type: boost::wrapexcept<QuadDCommon::NotFoundException>
std::exception::what: NotFoundException
[QuadDCommon::tag_message*] = No GPU associated to the given UUID

So what I am doing here is that I go to target-linux-tegra-armv8 folder and then use nsys profile from that folder. Is that the right way to run nsys profile from docker?

5

Are you on a tegra system in a docker???

I suspect you have the wrong version of Nsys for your archetecture.

6

Are you on a tegra system in a docker???

I am using “nvcr.io/nvidia/isaac/ros:aarch64-humble-nav2_6dfaf7adbe190f1181c3a0a2f2418760” docker image. It is nvidia’s image which is specifically created for ROS2 Humble. I am building this image on AGX Orin and I want to then run some processes inside the docker. If I want to profile these processes, I must run nsight inside the docker right?

To have nsight inside the docker, I tried modifying the docker file as per the article you mentioned above (I added apt-get install -y cuda-nsight-systems-11-4 nsight-systems-2022.3.3 and some other dependencies inside my docker file.) But I think those commands are for x86_64 architecture and not for aarch_64 architecture. So it didn’t work.

Then I just copied the nsight that was installed during the flashing process from Orin to the docker image. When I copied the nsight, I got this target-linux-tegra-armv8 folder inside the docker. In that folder I saw an executable named nsys. So I used this to run a ros2 process.

After running ros2 launch <package_name> <lunch_file.launch.py>, I got the error that I showed in the previous reply.

So what I want to know is:

  1. Is this also a right way to run nsight inside the docker? Or I must install nsight inside the docker image?
  2. Is it possible to install nsight inside aarch_64 docker image?

I suspect you have the wrong version of Nsys for your architecture.

  1. How can I know which version of Nsys is right for my architecture?

I hope I was able to explain my problem properly.

7

@Andrey_Trachenko can you help here.

8

Yes, Nsight Systems needs to run inside the container. There are two main ways to get Nsight Systems there:

  1. Mount the installation directory from the host system:
sudo docker run -it --rm -v /opt/nvidia/nsight-systems/2022.5.2:/nsys ubuntu:latest bash
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                    map the nsys directory

/nsys/bin/nsys profile sleep 1
                       ^^^^^^^
                       specify your application here
  1. Install Nsight Systems directly inside the container. You will need to get nsight-systems-2022.5.2.171-3255900_arm64.deb from JetPack and install inside the container. I’m not sure though how to easily extract it from the JetPack installer.