Isaac ROS Apriltag is slowing down

Autonomous Machines Robotics - Isaac Isaac ROS
1

Hi,
First of all, thank you for such a project.

Let’s get to the point.
My system:

  • Jetson Orin NX 16Gb and AverMedia D131 Carrier board.
  • Jetpack 5.1.1 (L4T 35.3.1) with MAXN NV Power and running Jetson Clocks.
  • DP 3.1 Release

image
image1219×452 228 KB

AprilTags with a static camera pose:

image
image1026×773 212 KB

SW Flow:

image
image712×190 12.3 KB

Main parameters:

image

I recorded a few bag files and visualized measurements, one of them:

image
image2073×915 96 KB

Here you can see how data publishing stably for 30 seconds at a frequency of about 10Hz, then you can observe a ~7-second pause, which is observed only in the topic of the Isaac Apriltag node. From time to time, freeze data can be more/less frequent and longer/shorter. On what it depends, I can not understand.

I also noticed that at this moment the consumed voltage drops. This is clearly related to the slowing down of the node, but it is difficult to say whether this is the cause of the hang or the consequence. You can see this behavior starting from the 5th second in the video below:

External Media

Even when I lowered the camera’s resolution to 1080p, I observed the same hangs. In addition, with the lower resolution, the tag orientation has too large deviations, which does not meet accuracy requirements.

So, in this regard, I have a question. Have you ever encountered a similar problem, and, in your opinion, what might be the problem that needs solving?

I appreciate your help!

2

Very strange indeed. The voltage drop is likely a key fault leading to the pause from the Apriltag node but not clear what could cause this issue or how exactly they are related. I’m following up with our Jetson experts and see if we can reproduce the issue locally with an Orin NX.

3

Thank you for the reply.
In the meantime, I’ll check the Apriltag detection performance by connecting a carrier board to another power supply.

Thank you!

4

I tested the performance with the various power supply sources. The result is the same.
Apparently, the power drops for some time were due to the lack of computations.

5

We have not yet been able to reproduce this voltage drop/stall in our nightly benchmarking on Orin NX unfortunately. Could you attach an Nsight profile of your workflow? We may be able to follow what’s stalling out the GPU that leads to the voltage drop perhaps.

6

I launch a whole bunch of ROS nodes through a single launch file inside a docker container.
How do I install nsys in a docker container and what command should I use to start profiling? Or is it possible to connect profiling on a docker container?

7

Take a look at the Profiling section of Isaac ROS Benchmark. nsys is already a part of Isaac ROS Docker containers.