Issue with FoundationPose and new object

[e0014667]

I am a student learning about computer vision and pose estimation.

I have installed FoundationPose on a Jetson Orin NX 16GB with an Intel realsense D435i depth camera

I have completed the tutorials for RT-DETR and FoundationPose

I am trying to run FoundationPose on a new object, and I have added the mesh .obj file (pinkcat in question in the following command) and the texture file to the folders, and I am trying to run this command:
ros2 launch isaac_ros_foundationpose isaac_ros_foundationpose.launch.py refine_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/refine_trt_engine.plan score_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/refine_trt_engine.plan mesh_file_path:=${ISAAC_ROS_WS}/mesh/pinkcat_scaled.obj texture_path:=${ISAAC_ROS_WS}/mesh/Texture_pinkcat.png launch_rviz:=true launch_bbox_to_mask:=true mask_height:=480mask_width:=640

I am experiencing this issue (shown in screenshot) where no bounding box and no camera live feed is present.

I am not sure how to begin debugging this, and would appreciate some guidance for this issue.

Thank you very much!

[e0014667]

just to add on:

even on isaac_ros_rtdetr tutorial:

I seem to be getting no image output from rqt_image_view

I’m not quite sure what might be the problem, as I have followed and double-checked the camera setup using the Isaac ROS Realsense setup.

Once again, any guidance would be greatly appreciated. Thank you!

[vchuang]

Hello @e0014667,

Welcome to the Isaac ROS forum and thanks for your post!

One thing to note is that the command you entered to launch FoundationPose should have two separate arguments (you missed a space): mask_height:=480 mask_width:=640

Also, score_engine_file_path should point to the score engine, not the refine engine, e.g.:

score_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/score_trt_engine.plan

You could reference here for running the command.

As for running the isaac_ros_rtdetr example, have you tried running with the rosbag to see if it works? ros2 bag play -l ${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_rtdetr/quickstart.bag

[e0014667]

very good day vchuang, apologies for the late reply and thank you very much for your response

I have started the foundationpose setup from scratch and attempted the rosbag example as you suggested:

rosbag seems to work fine, so I suspect the issue is somehow in my image stream from realsense camera to the rtdetr example. I’m not very sure where to begin debugging at the rtdeter side, unfortunately…

[e0014667]

quick update:

I found an error in the log when I attempted to rerun the realsense example:

[component_container_mt-1] [ERROR] [1771996054.503912747] [isaac_ros_examples.container]: Could not find requested resource in ament index
[ERROR] [launch_ros.actions.load_composable_nodes]: Failed to load node ‘realsense2_camera’ of type ‘realsense2_camera::RealSenseNodeFactory’ in container ‘/isaac_ros_examples/container’: Could not find requested resource in ament index

Edit:

I read some github issues from realsense-ros and one of the suggestions was to reverse the usb-c cable orientation and try again, this did not work

some other commands I have tried:
$ros2 launch realsense2_camera rs_camera.launch

Package ‘realsense2_camera’ not found: “package ‘realsense2_camera’ not found, searching: [‘/opt/ros/humble’]”

$ros2 launch realsense2_camera rs_camera.launch initial_reset:=true

Package ‘realsense2_camera’ not found: “package ‘realsense2_camera’ not found, searching: [‘/opt/ros/humble’]”

$lsusb

Bus 002 Device 005: ID 8086:0b3a Intel Corp. Intel(R) RealSense™ Depth Camera 435i
Bus 002 Device 002: ID 0bda:0489 Realtek Semiconductor Corp. 4-Port USB 3.0 Hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 0bda:c822 Realtek Semiconductor Corp. Bluetooth Radio

$realsense-viewer (inside the Docker container)

launches realsense viewer and color / stereo both work

[e0014667]

Upon further investigation, it seems the realsense2 package is not installed on my docker container, even though I went through the entire NVIDIA realsense ros setup tutorial on Isaac ROS RealSense Setup — isaac_ros_docs documentation

$ros2 pkg list | grep realsense returns empty
&
$dpkg -l | grep realsense also returns empty
in the Docker container.

I can confirm that this step in the realsense install tutorial:
cd ${ISAAC_ROS_WS}/src/isaac_ros_common/scripts &&
touch .isaac_ros_common-config &&
echo CONFIG_IMAGE_KEY=ros2_humble.realsense > .isaac_ros_common-config

was properly followed and so was this step:
cd ${ISAAC_ROS_WS} &&
colcon build --symlink-install --packages-up-to-regex realsense*

I do not understand why the realsense package just doesnt seem to work. any guidance or resources on how to troubleshoot this would be immensely appreciated…

Thank you!

[vchuang]

Hello @e0014667,

After you colcon build the workspace, did you source install/setup.bash?
If not, you need to do it after the build completes:

cd ${ISAAC_ROS_WS}
source install/setup.bash

Then verify that ROS sees the package:

ros2 pkg list | grep realsense

You need to do this every time you start a new terminal.

[e0014667]

Good day,

I have tried source install/setup.bash after colcon step for realsense, and my rtdetr and foundationpose tutorials work! (this source step was not mentioned in the realsense setup tutorial)

I have also fixed the initial command:
ros2 launch isaac_ros_foundationpose isaac_ros_foundationpose.launch.py refine_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/refine_trt_engine.plan score_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/score_trt_engine.plan mesh_file_path:=${ISAAC_ROS_WS}/mesh/pinkcat_scaled.obj texture_path:=${ISAAC_ROS_WS}/mesh/Texture_pinkcat.png launch_rviz:=true launch_bbox_to_mask:=true mask_height:=480 mask_width:=640

based on your pointers from earlier
unfortunately the camera still seems to not show up on rviz

my expectation is that some image would pop up and prompt me to click the bounding box due to launch_bbox_to_mask:=true but I do not observe that happening either. is there an issue with my ros2 command? or perhaps some steps I am missing to enable foundationpose for a custom object

here is the log file when I run the command mentioned above:

fp_log.txt (9.7 KB)

once again, much appreciated for your guidance, thank you for taking the time to look at my issue

[vchuang]

Your RViz screenshot shows Global Status: “Fixed Frame has no tf data”, which means the frame selected in Global Options → Fixed Frame is not present in /tf or /tf_static.
You need to either choose a frame that already exists in TF, or publish one yourself.

If you just want to test the feature and don’t have a suitable TF yet, you can simulate one in a separate terminal with:

ros2 run tf2_ros static_transform_publisher \
  0 0 0 0 0 0 1 \
  world camera_1_link

This will continuously publish a TF from world → camera_1_link.
Replace camera_1_link with whatever frame your camera driver is actually using (you can see it in /tf).

After that, in RViz, set Fixed Frame = world so RViz can use that frame without errors.

[e0014667]

Much appreciated for pointing out the fixed frame issue.

I have also noticed that I have made the foolish assumption that foundationpose / Rviz would launch the realsense camera (or at least run the driver) but that was not the case.

After launching the realsense driver in another terminal using

ros2 launch realsense2_camera rs_launch.py \
depth_module.profile:=640x480x30 \
rgb_camera.profile:=640x480x30

using a 3rd terminal to run

ros2 run tf2_ros static_transform_publisher \
0 0 0 0 0 0 1 \
world camera_link

and finally, changing Fixed Frame = world & Camera > Topic = /camera/color/image_raw

I finally get a camera output on Rviz!

However, I am aware there is a mask that has to be fed into foundationpose to enable the pose estimation. my understanding is that launch_bbox_to_mask:=true parameter launches some bounding box to mask converter, but it still seems like the Node is idle once Rviz is launched.

the pose estimation still doesnt seem to be working for me, so it seems I will be focusing on debugging the mask issue now

[vchuang]

It would be easier for us to assist if you could share the exact command you used to launch FoundationPose, and the full console log from that run.

Note that launch_bbox_to_mask:=true only starts the Detection2D → mask node, which converts existing 2D detections into a mono8 mask. It does not start a 2D detector by itself.

If you only ran the command you posted earlierros2 launch isaac_ros_foundationpose isaac_ros_foundationpose.launch.py ... launch_bbox_to_mask:=true ... you would still be missing a 2D detector (for example SyntheticaDETR).
In our reference example, this is provided via:
rt_detr_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.plan which brings up the RT-DETR detector that publishes the bounding boxes needed for the bbox→mask node and FoundationPose to work.

[e0014667]

Apologies for that confusion, I will list step-by-step the commands I have been running:

1. launch realsense camera
   ros2 launch realsense2_camera rs_launch.py \
depth_module.profile:=640x480x30 \
rgb_camera.profile:=640x480x30

2. simulate TF
   ros2 run tf2_ros static_transform_publisher
0 0 0 0 0 0 1
world camera_link

3. run foundationpose (I have adjusted the command with the DETR portion you mentioned above, but the bounding box still doesnt seem to work)
   ros2 launch isaac_ros_foundationpose isaac_ros_foundationpose.launch.py refine_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/refine_trt_engine.plan score_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/score_trt_engine.plan mesh_file_path:=${ISAAC_ROS_WS}/mesh/pinkcat_scaled.obj texture_path:=${ISAAC_ROS_WS}/mesh/Texture_pinkcat.png rt_detr_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.plan launch_rviz:=true launch_bbox_to_mask:=true mask_height:=480 mask_width:=640

here is the log output from the foundationpose terminal:

fp_log1.txt (10.2 KB)

and here is a screenshot of my RViz after changing Fixed Frame = world & Camera > Topic = /camera/color/image_raw

the little pink cat is what I am attempting to detect / pose-estimate using foundationpose

Once again, thank you so much for your guidance (and many apologies for my inadequate knowledge in this space)

[vchuang]

SyntheticaDETR is an exact-instance detector trained on a fixed set of objects, so when you swap in a new object like your pink cat without a matching trained RT-DETR model, the detector won’t be able to detect it.

To get correct results on a new object, you need a detector model trained specifically for it. You can follow this documentation to create a 3D mesh for your specific object, and refer to this guide here for model training details.

[e0014667]

Good day! Apologies for the late response.

Many thanks for your guidance regarding this issue. I have begun looking into training a detr model for the 2D detection. much appreciated for your patience and prompt responses to my issues.

I will be closing the issue now, as I believe this post has run its course.

Once again, my sincerest thanks for your guidance.

[2287278570]

Hello? have you succeed? I also meet this problem. Can you share your method?