An Three error occurred during Foundationpose Quickstart

hirship1234 · August 15, 2024, 1:33pm

Most of all, thank you for giving me the opportunity to use such a great package.

I ran into three issues while doing the FoundationPose Quickstart. Photos and videos for the related questions below have been attached to the drive link below.

https://drive.google.com/drive/folders/1TyX0iz8TCZ6glnp41dAGV4eYII4kl9ie?usp=sharing

First, I conducted the Isaac ROS RT-DETR tutorial, but the detection results are different from the actual object location. This is shown in the attached drive.

Second, when building Foundationpose from source rather than installing it as a binary, 'ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren’t any 'The error keeps appearing. I tried to find related information and tried to solve the problem, but I couldn’t solve it, so I’m leaving a question.

The third is a different dimension error between the depth image and the segmentation image when running Foundationpose. I am attaching a photo related to this.

The command is ‘ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=realsense_mono_rect_depth,foundationpose mesh_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/Mac_and_cheese_0_1/Mac_and_cheese_0_1.obj texture_path:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/Mac_and_cheese_0_1/materials/textures/baked_mesh_tex0.png score_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/score_trt_engine.plan refine_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/refine_trt_engine.plan rt_detr_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.plan’.

Raffaello · August 16, 2024, 5:19pm

Hi @hirship1234

Welcome to the Isaac ROS forum.
Let me help you

I think these two errors are related. It seems that a wrong setup of the image size between the foundation pose and the camera output is causing the issues. Fixing point three will also resolve point one.

To better help you to resolve the issue. How did you set up your camera input? Looking at the stream it seems you are not running our default demo

To fix this error, you can temporarily follow this workaround:

Best,
Raffaello

hirship1234 · August 19, 2024, 1:55pm

Hello, thank you for the quick reply.

In summary, all the errors I had previously were resolved thanks to your advice. I’m currently facing a problem where no output comes out when running FoundationPose, so I’d like to ask for additional help.

First, I found out that the abnormal output of RT-DETR and the dimension error that occurred in FoundationPose were problems with RealSense settings. Previously, I confirmed that the issue occurred by running the Realsense node directly. After reconfiguring following the guidelines in the Isaac ROS documentation, RTDETR worked normally, and the error in FoundationPose also disappeared.

Also, the ERROR: setuptools_scm._file_finders.git listing git files failed - pretending there aren’t any error was resolved by changing the version of setuptools_scm.

pip install setuptools_scm==5.0.0

As a result, all previously encountered errors have been resolved, and Foundation pose is now running, but there is no response to the /output topic. As you can see in the video above, rtdetr masking seems to be working.

I have attached related logs and ROS 2 topic list below. I would appreciate your advice.

(For .obj files and .png files, I created them by referring to [Try More Examples] - [Tutorial to create your own 3D object mesh in .obj].)

> ros2 topic list
/accel/imu_info
/accel/metadata
/accel/sample
/aligned_depth_to_color/camera_info
/aligned_depth_to_color/image_raw
/aligned_depth_to_color/image_raw/compressed
/aligned_depth_to_color/image_raw/compressedDepth
/aligned_depth_to_color/image_raw/nitros
/aligned_depth_to_color/image_raw/theora
/camera_info_rect
/camera_info_resized
/camera_info_resized/nitros
/camera_info_segmentation
/camera_info_segmentation/nitros
/clicked_point
/color/image_raw/compressed
/color/image_raw/compressedDepth
/color/image_raw/theora
/color/metadata
/color_image_resized
/color_image_resized/nitros
/depth
/depth/camera_info
/depth/image_rect_raw
/depth/image_rect_raw/compressed
/depth/image_rect_raw/compressedDepth
/depth/image_rect_raw/theora
/depth/metadata
/depth/nitros
/depth_image
/depth_image/nitros
/detection2_d
/detections_output
/extrinsics/depth_to_accel
/extrinsics/depth_to_color
/extrinsics/depth_to_gyro
/goal_pose
/gyro/imu_info
/gyro/metadata
/gyro/sample
/image_rect
/image_rect/nitros
/imu
/initialpose
/normalized_tensor
/normalized_tensor/nitros
/output
/output/nitros
/padded_image
/padded_image/nitros
/parameter_events
/planar_tensor
/planar_tensor/nitros
/pose_estimation/pose_matrix_output
/pose_estimation/pose_matrix_output/nitros
/reshaped_tensor
/reshaped_tensor/nitros
/rgb/camera_info
/rgb/camera_info/nitros
/rgb/camera_info_viz
/rgb/camera_info_viz/nitros
/rgb/image_rect_color
/rgb/image_rect_color/nitros
/rgb/image_rect_color_viz
/rgb/image_rect_color_viz/nitros
/rosout
/rt_detr_segmentation
/rt_detr_segmentation/nitros
/segmentation
/segmentation/nitros
/segmentation/nitros/nitros_image_mono8
/tensor_pub
/tensor_pub/nitros
/tensor_sub
/tensor_sub/nitros
/tf
/tf_static


> FoundationPose Terminal Log

admin@minipin-MS-7E01:/workspaces/isaac_ros-dev$ ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=realsense_mono_rect_depth,foundationpose mesh_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/banana/banana.obj texture_path:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/banana/materials/textures/baked_mesh_tex0.png score_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/score_trt_engine.plan refine_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/refine_trt_engine.plan rt_detr_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.plan

[INFO] [launch]: All log files can be found below /home/admin/.ros/log/2024-08-19-22-33-54-656144-minipin-MS-7E01-60259
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container_mt-1]: process started with pid [60269]
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/realsense2_camera' in container '/isaac_ros_examples/container'
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/convert_metric' in container '/isaac_ros_examples/container'
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/drop_node' in container '/isaac_ros_examples/container'
[component_container_mt-1] [INFO] [1724074434.723876986] [isaac_ros_examples.container]: Load Library: /opt/ros/humble/lib/libresize_node.so
[component_container_mt-1] [INFO] [1724074434.735921496] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::ResizeNode>
[component_container_mt-1] [INFO] [1724074434.735940736] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::ResizeNode>
[component_container_mt-1] [INFO] [1724074434.737744137] [resize_left_rt_detr_node]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1724074434.737939676] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/std/libgxf_std.so
[component_container_mt-1] [INFO] [1724074434.742347044] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_gxf_helpers.so
[component_container_mt-1] [INFO] [1724074434.744390109] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_sight.so
[component_container_mt-1] [INFO] [1724074434.746048054] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_atlas.so
[component_container_mt-1] 2024-08-19 22:33:54.751 WARN  gxf/std/program.cpp@532: No GXF scheduler specified.
[component_container_mt-1] [INFO] [1724074434.751393799] [resize_left_rt_detr_node]: [ResizeNode] Set output data format to: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1724074434.751428508] [resize_left_rt_detr_node]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1724074434.756309687] [resize_left_rt_detr_node]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1724074434.756393898] [resize_left_rt_detr_node]: [NitrosContext] Loading extension: gxf/lib/multimedia/libgxf_multimedia.so
[component_container_mt-1] [INFO] [1724074434.757822359] [resize_left_rt_detr_node]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_message_compositor.so
[component_container_mt-1] [INFO] [1724074434.758449076] [resize_left_rt_detr_node]: [NitrosContext] Loading extension: gxf/lib/cuda/libgxf_cuda.so
[component_container_mt-1] [INFO] [1724074434.759872337] [resize_left_rt_detr_node]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_tensorops.so
[component_container_mt-1] [INFO] [1724074434.761995393] [resize_left_rt_detr_node]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1724074434.763635506] [resize_left_rt_detr_node]: [NitrosNode] Running optimization
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/resize_left_rt_detr_node' in container '/isaac_ros_examples/container'
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/pad_node' in container '/isaac_ros_examples/container'
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/image_to_tensor_node' in container '/isaac_ros_examples/container'
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/interleaved_to_planar_node' in container '/isaac_ros_examples/container'
[component_container_mt-1] [INFO] [1724074434.835957476] [resize_left_rt_detr_node]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1724074434.837468392] [resize_left_rt_detr_node]: [NitrosPublisherSubscriberGroup] Pinning the component "image_sink/sink" (type="nvidia::isaac_ros::MessageRelay") to use its compatible format only: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1724074434.841862978] [resize_left_rt_detr_node]: [NitrosNode] Starting negotiation...
[component_container_mt-1] [INFO] [1724074434.842272840] [isaac_ros_examples.container]: Load Library: /opt/ros/humble/lib/libpad_node.so
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/reshape_node' in container '/isaac_ros_examples/container'
[component_container_mt-1] [INFO] [1724074434.852325724] [resize_left_rt_detr_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.852364045] [resize_left_rt_detr_node]: Could not negotiate
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/rtdetr_preprocessor' in container '/isaac_ros_examples/container'
[component_container_mt-1] [INFO] [1724074434.868580946] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::PadNode>
[component_container_mt-1] [INFO] [1724074434.868620345] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::PadNode>
[component_container_mt-1] [INFO] [1724074434.875099654] [pad_node.ManagedNitrosSubscriber]: Starting Managed Nitros Subscriber
[component_container_mt-1] [INFO] [1724074434.875184597] [resize_left_rt_detr_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.875237601] [resize_left_rt_detr_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074434.877100146] [pad_node.ManagedNitrosPublisher]: Starting Managed Nitros Publisher
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/tensor_rt' in container '/isaac_ros_examples/container'
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/rtdetr_decoder' in container '/isaac_ros_examples/container'
[ERROR] [launch_ros.actions.load_composable_nodes]: Failed to load node 'detection2_d_to_mask' of type 'nvidia::isaac_ros::foundationpose::Detection2DToMask' in container '/isaac_ros_examples/container': Could not find requested resource in ament index
[component_container_mt-1] [INFO] [1724074434.922535310] [isaac_ros_examples.container]: Load Library: /opt/ros/humble/lib/libimage_to_tensor_node.so
[component_container_mt-1] [INFO] [1724074434.922654786] [pad_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.923232641] [pad_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.923452087] [pad_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.924144860] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::ImageToTensorNode>
[component_container_mt-1] [INFO] [1724074434.924157381] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::ImageToTensorNode>
[component_container_mt-1] [INFO] [1724074434.927400375] [image_to_tensor_node.ManagedNitrosSubscriber]: Starting Managed Nitros Subscriber
[component_container_mt-1] [INFO] [1724074434.927421746] [pad_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.928040785] [image_to_tensor_node.ManagedNitrosPublisher]: Starting Managed Nitros Publisher
[component_container_mt-1] [INFO] [1724074434.928249100] [isaac_ros_examples.container]: Load Library: /opt/ros/humble/lib/libinterleaved_to_planar_node.so
[component_container_mt-1] [INFO] [1724074434.928803481] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::InterleavedToPlanarNode>
[component_container_mt-1] [INFO] [1724074434.928811379] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::InterleavedToPlanarNode>
[component_container_mt-1] [INFO] [1724074434.930988621] [interleaved_to_planar_node]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1724074434.931591788] [interleaved_to_planar_node]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1724074434.935368365] [interleaved_to_planar_node]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1724074434.935684486] [interleaved_to_planar_node]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1724074434.936617340] [interleaved_to_planar_node]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1724074434.938357751] [image_to_tensor_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.938404384] [image_to_tensor_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074434.938550644] [image_to_tensor_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.938575826] [image_to_tensor_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074434.938752934] [image_to_tensor_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.938782428] [image_to_tensor_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074434.938918407] [image_to_tensor_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.938944905] [image_to_tensor_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074434.939126893] [image_to_tensor_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.939160677] [image_to_tensor_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074434.939292928] [interleaved_to_planar_node]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1724074434.939340638] [image_to_tensor_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.939372203] [image_to_tensor_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074434.939536293] [image_to_tensor_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.939549969] [image_to_tensor_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074434.939667852] [image_to_tensor_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.939697399] [image_to_tensor_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074434.939844608] [image_to_tensor_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.939996976] [image_to_tensor_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074434.940134524] [image_to_tensor_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.940158967] [image_to_tensor_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074434.940882531] [interleaved_to_planar_node]: [NitrosNode] Starting negotiation...
[component_container_mt-1] [INFO] [1724074434.941054747] [image_to_tensor_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.941070475] [image_to_tensor_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074434.941579298] [isaac_ros_examples.container]: Load Library: /opt/ros/humble/lib/libreshape_node.so
[component_container_mt-1] [INFO] [1724074434.942277321] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::ReshapeNode>
[component_container_mt-1] [INFO] [1724074434.942287010] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::ReshapeNode>
[component_container_mt-1] [INFO] [1724074434.944584015] [reshape_node]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1724074434.944849864] [reshape_node]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1724074434.947188474] [reshape_node]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1724074434.947332469] [reshape_node]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1724074434.947945333] [reshape_node]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1724074434.950812109] [reshape_node]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1724074434.951774140] [interleaved_to_planar_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.951778656] [image_to_tensor_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.951810637] [image_to_tensor_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074434.951953273] [interleaved_to_planar_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.952023802] [image_to_tensor_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.952040456] [image_to_tensor_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074434.952270342] [image_to_tensor_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.952284061] [image_to_tensor_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074434.952275855] [interleaved_to_planar_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.952597421] [interleaved_to_planar_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.952602820] [image_to_tensor_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.952628015] [image_to_tensor_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074434.953190204] [reshape_node]: [NitrosNode] Starting negotiation...
[component_container_mt-1] [INFO] [1724074434.953680823] [interleaved_to_planar_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.953691843] [image_to_tensor_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.953714826] [image_to_tensor_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074434.953995283] [isaac_ros_examples.container]: Load Library: /workspaces/isaac_ros-dev/install/isaac_ros_rtdetr/lib/librtdetr_preprocessor_node.so
[component_container_mt-1] [INFO] [1724074434.954023023] [image_to_tensor_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.954031162] [image_to_tensor_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074434.954261403] [image_to_tensor_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.954275049] [image_to_tensor_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074434.955475135] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::rtdetr::RtDetrPreprocessorNode>
[component_container_mt-1] [INFO] [1724074434.955493309] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::rtdetr::RtDetrPreprocessorNode>
[component_container_mt-1] [INFO] [1724074434.961372186] [rtdetr_preprocessor.ManagedNitrosSubscriber]: Starting Managed Nitros Subscriber
[component_container_mt-1] [INFO] [1724074434.961454335] [reshape_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.961517867] [interleaved_to_planar_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.961539008] [image_to_tensor_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.961561089] [image_to_tensor_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074434.962005703] [image_to_tensor_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.962029040] [image_to_tensor_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074434.962168778] [image_to_tensor_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.962187420] [image_to_tensor_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074434.962414910] [rtdetr_preprocessor.ManagedNitrosPublisher]: Starting Managed Nitros Publisher
[component_container_mt-1] [INFO] [1724074434.963478402] [isaac_ros_examples.container]: Load Library: /opt/ros/humble/lib/libtensor_rt_node.so
[component_container_mt-1] [INFO] [1724074434.963925552] [reshape_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.964241255] [reshape_node]: Negotiating
[component_container_mt-1] [INFO] [1724074434.977245536] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::TensorRTNode>
[component_container_mt-1] [INFO] [1724074434.977272192] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::TensorRTNode>
[component_container_mt-1] [INFO] [1724074434.979307798] [rtdetr_preprocessor]: Negotiating
[component_container_mt-1] [INFO] [1724074434.981858464] [tensor_rt]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1724074434.983116029] [tensor_rt]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1724074434.985449607] [tensor_rt]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1724074434.985576109] [tensor_rt]: [NitrosContext] Loading extension: gxf/lib/serialization/libgxf_serialization.so
[component_container_mt-1] [INFO] [1724074434.987176083] [tensor_rt]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_tensor_rt.so
[component_container_mt-1] [INFO] [1724074434.987797231] [tensor_rt]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1724074434.988676634] [tensor_rt]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1724074434.996407311] [tensor_rt]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1724074434.996987103] [tensor_rt]: [NitrosPublisherSubscriberGroup] Pinning the component "inference/rx" (type="nvidia::gxf::DoubleBufferReceiver") to use its compatible format only: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1724074434.996999283] [tensor_rt]: [NitrosPublisherSubscriberGroup] Pinning the component "sink/sink" (type="nvidia::isaac_ros::MessageRelay") to use its compatible format only: "nitros_tensor_list_nhwc_rgb_f32"
[component_container_mt-1] [INFO] [1724074434.998261322] [tensor_rt]: [NitrosNode] Starting negotiation...
[component_container_mt-1] [INFO] [1724074434.998493802] [rtdetr_preprocessor]: Negotiating
[component_container_mt-1] [INFO] [1724074434.998664797] [isaac_ros_examples.container]: Load Library: /workspaces/isaac_ros-dev/install/isaac_ros_rtdetr/lib/librtdetr_decoder_node.so
[component_container_mt-1] [INFO] [1724074434.999607931] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::rtdetr::RtDetrDecoderNode>
[component_container_mt-1] [INFO] [1724074434.999624075] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::rtdetr::RtDetrDecoderNode>
[component_container_mt-1] [INFO] [1724074435.002820487] [rtdetr_decoder.ManagedNitrosSubscriber]: Starting Managed Nitros Subscriber
[component_container_mt-1] [INFO] [1724074435.002867424] [tensor_rt]: Negotiating
[component_container_mt-1] [INFO] [1724074435.002893561] [tensor_rt]: Could not negotiate
[component_container_mt-1] [INFO] [1724074435.005000477] [isaac_ros_examples.container]: Load Library: /workspaces/isaac_ros-dev/install/isaac_ros_foundationpose/lib/libdetection2_d_to_mask.so
[component_container_mt-1] [INFO] [1724074435.011735106] [tensor_rt]: Negotiating
[component_container_mt-1] [INFO] [1724074435.011765347] [tensor_rt]: Could not negotiate
[component_container_mt-1] [INFO] [1724074435.013899295] [tensor_rt]: Negotiating
[component_container_mt-1] [INFO] [1724074435.013929952] [tensor_rt]: Could not negotiate
[component_container_mt-1] [INFO] [1724074435.025334413] [resize_left_rt_detr_node]: Negotiating
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/resize_mask_node' in container '/isaac_ros_examples/container'
[component_container_mt-1] [INFO] [1724074435.025361766] [resize_left_rt_detr_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074435.040282543] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::foundationpose::Detection2DToMask>
[component_container_mt-1] [INFO] [1724074435.040319133] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::foundationpose::Detection2DToMask>
[component_container_mt-1] [INFO] [1724074435.050981611] [detection2_d_to_mask]: Mask Height: 480, Mask Width: 640
[component_container_mt-1] [INFO] [1724074435.051233056] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::ResizeNode>
[component_container_mt-1] [INFO] [1724074435.051250860] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::ResizeNode>
[component_container_mt-1] [INFO] [1724074435.054461128] [resize_mask_node]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1724074435.055016226] [resize_mask_node]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1724074435.060334801] [resize_mask_node]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1724074435.060516157] [resize_mask_node]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1724074435.062027107] [resize_mask_node]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1724074435.140433773] [resize_mask_node]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1724074435.145576915] [resize_mask_node]: [NitrosNode] Starting negotiation...
[component_container_mt-1] [INFO] [1724074435.146027950] [resize_left_rt_detr_node]: Negotiating
[component_container_mt-1] [INFO] [1724074435.146058552] [resize_left_rt_detr_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074435.146684199] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::ResizeNode>
[component_container_mt-1] [INFO] [1724074435.146697643] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::ResizeNode>
[component_container_mt-1] [INFO] [1724074435.148718631] [resize_left_viz]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1724074435.149070451] [resize_left_viz]: [ResizeNode] Set output data format to: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1724074435.149093337] [resize_left_viz]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1724074435.152981971] [resize_left_viz]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1724074435.153257210] [resize_left_viz]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1724074435.155040942] [resize_left_viz]: [NitrosNode] Running optimization
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/resize_left_viz' in container '/isaac_ros_examples/container'
[ERROR] [launch_ros.actions.load_composable_nodes]: Failed to load node 'foundationpose_node' of type 'nvidia::isaac_ros::foundationpose::FoundationPoseNode' in container '/isaac_ros_examples/container': Could not find requested resource in ament index
[component_container_mt-1] [INFO] [1724074435.237695281] [resize_left_viz]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1724074435.239809340] [resize_left_viz]: [NitrosPublisherSubscriberGroup] Pinning the component "image_sink/sink" (type="nvidia::isaac_ros::MessageRelay") to use its compatible format only: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1724074435.253632753] [resize_left_viz]: [NitrosNode] Starting negotiation...
[component_container_mt-1] [INFO] [1724074435.254643597] [isaac_ros_examples.container]: Load Library: /workspaces/isaac_ros-dev/install/isaac_ros_foundationpose/lib/libfoundationpose_node.so
[component_container_mt-1] [INFO] [1724074435.257298529] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::foundationpose::FoundationPoseNode>
[component_container_mt-1] [INFO] [1724074435.257317182] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::foundationpose::FoundationPoseNode>
[component_container_mt-1] [INFO] [1724074435.264095064] [foundationpose_node]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1724074435.265027429] [foundationpose_node]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1724074435.267388587] [foundationpose_node]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1724074435.267565098] [foundationpose_node]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_depth_image_proc.so
[component_container_mt-1] [INFO] [1724074435.268097009] [foundationpose_node]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_sgm.so
[component_container_mt-1] [INFO] [1724074435.268476359] [foundationpose_node]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_messages.so
[component_container_mt-1] [INFO] [1724074435.268885789] [foundationpose_node]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_ros_messages.so
[component_container_mt-1] [INFO] [1724074435.269240682] [foundationpose_node]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_foundationpose.so
[component_container_mt-1] [INFO] [1724074435.271908305] [foundationpose_node]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1724074435.280106788] [foundationpose_node]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1724074435.597716255] [foundationpose_node]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1724074435.603804321] [foundationpose_node]: [NitrosNode] Starting negotiation...
[component_container_mt-1] [INFO] [1724074435.604305325] [resize_mask_node]: Negotiating
[component_container_mt-1] [INFO] [1724074435.604562126] [resize_left_rt_detr_node]: Negotiating
[component_container_mt-1] [INFO] [1724074435.604586637] [resize_left_rt_detr_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074435.604994092] [resize_left_rt_detr_node]: Negotiating
[component_container_mt-1] [INFO] [1724074435.605004201] [foundationpose_node]: Negotiating
[component_container_mt-1] [INFO] [1724074435.605017117] [foundationpose_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074435.605006285] [resize_left_rt_detr_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074435.605022477] [foundationpose_node]: Negotiating
[component_container_mt-1] [INFO] [1724074435.605028773] [foundationpose_node]: Could not negotiate
[component_container_mt-1] [INFO] [1724074435.843138374] [resize_left_rt_detr_node]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1724074435.843220966] [resize_left_rt_detr_node]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1724074435.843247668] [resize_left_rt_detr_node]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1724074435.843260777] [resize_left_rt_detr_node]: [NitrosPublisher] Use only the compatible publisher: topic_name="/color_image_resized", data_format="nitros_image_rgb8"
[component_container_mt-1] [INFO] [1724074435.843270188] [resize_left_rt_detr_node]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1724074435.843277726] [resize_left_rt_detr_node]: [NitrosPublisher] Use only the compatible publisher: topic_name="/camera_info_resized", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1724074435.843286669] [resize_left_rt_detr_node]: [NitrosSubscriber] Use the negotiated data format: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1724074435.843295235] [resize_left_rt_detr_node]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1724074435.843302206] [resize_left_rt_detr_node]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/rgb/camera_info", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1724074435.843639744] [resize_left_rt_detr_node]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1724074435.848904448] [resize_left_rt_detr_node]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/PHIICFDJLX/PHIICFDJLX.yaml"
[component_container_mt-1] [INFO] [1724074435.848945577] [resize_left_rt_detr_node]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1724074435.851855972] [resize_left_rt_detr_node]: [ResizeNode] postLoadGraphCallback().
[component_container_mt-1] [INFO] [1724074435.851959164] [resize_left_rt_detr_node]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1724074435.882670043] [resize_left_rt_detr_node]: [NitrosNode] Node was started
[component_container_mt-1] [INFO] [1724074435.942234124] [interleaved_to_planar_node]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1724074435.942280748] [interleaved_to_planar_node]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1724074435.942290861] [interleaved_to_planar_node]: [NitrosPublisher] Use the negotiated data format: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1724074435.942301249] [interleaved_to_planar_node]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1724074435.942308676] [interleaved_to_planar_node]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/normalized_tensor", data_format="nitros_tensor_list_nhwc_rgb_f32"
[component_container_mt-1] [INFO] [1724074435.942415836] [interleaved_to_planar_node]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1724074435.944756427] [interleaved_to_planar_node]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/FCPSEZHHSX/FCPSEZHHSX.yaml"
[component_container_mt-1] [INFO] [1724074435.944777132] [interleaved_to_planar_node]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1724074435.946447071] [interleaved_to_planar_node]: In InterleavedToPlanarNode postLoadGraphCallback().
[component_container_mt-1] [INFO] [1724074435.946507471] [interleaved_to_planar_node]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1724074435.950387955] [interleaved_to_planar_node]: [NitrosNode] Node was started
[component_container_mt-1] [INFO] [1724074435.953876737] [reshape_node]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1724074435.953925930] [reshape_node]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1724074435.953938616] [reshape_node]: [NitrosPublisher] Use the negotiated data format: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1724074435.953952517] [reshape_node]: [NitrosSubscriber] Use the negotiated data format: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1724074435.954075844] [reshape_node]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1724074435.956198946] [reshape_node]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/APDVQIGPEM/APDVQIGPEM.yaml"
[component_container_mt-1] [INFO] [1724074435.956237038] [reshape_node]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1724074435.957891401] [reshape_node]: In ReshapeNode postLoadGraphCallback().
[component_container_mt-1] [INFO] [1724074435.957993743] [reshape_node]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1724074435.961656501] [reshape_node]: [NitrosNode] Node was started
[component_container_mt-1] [INFO] [1724074435.999003208] [tensor_rt]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1724074435.999046219] [tensor_rt]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1724074435.999050667] [tensor_rt]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1724074435.999056357] [tensor_rt]: [NitrosPublisher] Use only the compatible publisher: topic_name="/tensor_sub", data_format="nitros_tensor_list_nhwc_rgb_f32"
[component_container_mt-1] [INFO] [1724074435.999062266] [tensor_rt]: [NitrosSubscriber] Use the negotiated data format: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1724074435.999145734] [tensor_rt]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1724074436.002282026] [tensor_rt]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/WCCGVDZRHN/WCCGVDZRHN.yaml"
[component_container_mt-1] [INFO] [1724074436.002328368] [tensor_rt]: [NitrosNode] Loading application
[component_container_mt-1] 2024-08-19 22:33:56.003 WARN  gxf/std/yaml_file_loader.cpp@1077: Using unregistered parameter 'dev_id' in component 'stream'.
[component_container_mt-1] [INFO] [1724074436.003725244] [tensor_rt]: In TensorRTNode postLoadGraphCallback().
[component_container_mt-1] [INFO] [1724074436.112704521] [tensor_rt]: Read tensor shape information from TRT Model Engine: /workspaces/isaac_ros-dev/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.plan
[component_container_mt-1] [INFO] [1724074436.112896601] [tensor_rt]: Tensors 4800 bytes, num outputs 40 x tensors per output 3 = 120 blocks
[component_container_mt-1] [INFO] [1724074436.113082778] [tensor_rt]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1724074436.116970976] [tensor_rt]: [NitrosNode] Node was started
[component_container_mt-1] [INFO] [1724074436.146017112] [resize_mask_node]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1724074436.146069176] [resize_mask_node]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1724074436.146076015] [resize_mask_node]: [NitrosPublisher] Use the negotiated data format: "nitros_image_mono8"
[component_container_mt-1] [INFO] [1724074436.146083682] [resize_mask_node]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1724074436.146088530] [resize_mask_node]: [NitrosPublisher] Use only the compatible publisher: topic_name="/camera_info_segmentation", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1724074436.146094420] [resize_mask_node]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1724074436.146099126] [resize_mask_node]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/rt_detr_segmentation", data_format="nitros_image_bgr8"
[component_container_mt-1] [INFO] [1724074436.146104490] [resize_mask_node]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1724074436.146108515] [resize_mask_node]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/camera_info_resized", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1724074436.146125826] [resize_mask_node]: [NitrosPublisherSubscriberGroup] Adjusted the compatible format of the component "sync/image_in" (type="nvidia::gxf::DoubleBufferReceiver") from "nitros_image_bgr8" to "nitros_image_mono8"
[component_container_mt-1] [INFO] [1724074436.146277225] [resize_mask_node]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1724074436.155690276] [resize_mask_node]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/ZLUFXDMGUE/ZLUFXDMGUE.yaml"
[component_container_mt-1] [INFO] [1724074436.155743619] [resize_mask_node]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1724074436.159175601] [resize_mask_node]: [ResizeNode] postLoadGraphCallback().
[component_container_mt-1] [INFO] [1724074436.159273494] [resize_mask_node]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1724074436.174677041] [resize_mask_node]: [NitrosNode] Node was started
[component_container_mt-1] [INFO] [1724074436.254498218] [resize_left_viz]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1724074436.254532772] [resize_left_viz]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1724074436.254536857] [resize_left_viz]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1724074436.254540590] [resize_left_viz]: [NitrosPublisher] Use only the compatible publisher: topic_name="/rgb/image_rect_color_viz", data_format="nitros_image_rgb8"
[component_container_mt-1] [INFO] [1724074436.254544516] [resize_left_viz]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1724074436.254546820] [resize_left_viz]: [NitrosPublisher] Use only the compatible publisher: topic_name="/rgb/camera_info_viz", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1724074436.254550715] [resize_left_viz]: [NitrosSubscriber] Use the negotiated data format: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1724074436.254553903] [resize_left_viz]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1724074436.254556077] [resize_left_viz]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/rgb/camera_info", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1724074436.254635257] [resize_left_viz]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1724074436.259355701] [resize_left_viz]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/KTTIXXVMIN/KTTIXXVMIN.yaml"
[component_container_mt-1] [INFO] [1724074436.259385651] [resize_left_viz]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1724074436.262261801] [resize_left_viz]: [ResizeNode] postLoadGraphCallback().
[component_container_mt-1] [INFO] [1724074436.262347668] [resize_left_viz]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1724074436.270146819] [resize_left_viz]: [NitrosNode] Node was started
[component_container_mt-1] [INFO] [1724074436.604512200] [foundationpose_node]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1724074436.604595437] [foundationpose_node]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1724074436.604606728] [foundationpose_node]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1724074436.604616799] [foundationpose_node]: [NitrosPublisher] Use only the compatible publisher: topic_name="/output", data_format="nitros_detection3_d_array"
[component_container_mt-1] [INFO] [1724074436.604626008] [foundationpose_node]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1724074436.604632930] [foundationpose_node]: [NitrosPublisher] Use only the compatible publisher: topic_name="/pose_estimation/pose_matrix_output", data_format="nitros_tensor_list_nchw"
[component_container_mt-1] [INFO] [1724074436.604644338] [foundationpose_node]: [NitrosSubscriber] Use the negotiated data format: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1724074436.604653170] [foundationpose_node]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1724074436.604659852] [foundationpose_node]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/rgb/camera_info", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1724074436.604668175] [foundationpose_node]: [NitrosSubscriber] Use the negotiated data format: "nitros_image_mono8"
[component_container_mt-1] [INFO] [1724074436.604675093] [foundationpose_node]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1724074436.604681659] [foundationpose_node]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/depth_image", data_format="nitros_image_32FC1"
[component_container_mt-1] [INFO] [1724074436.604828398] [foundationpose_node]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1724074438.703970659] [foundationpose_node]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/SUFHNGHDPY/SUFHNGHDPY.yaml"
[component_container_mt-1] [INFO] [1724074438.704023870] [foundationpose_node]: [NitrosNode] Loading application
[component_container_mt-1] 2024-08-19 22:33:58.715 WARN  gxf/std/yaml_file_loader.cpp@1077: Using unregistered parameter 'dummy_rx' in component ''.
[component_container_mt-1] 2024-08-19 22:33:58.715 WARN  gxf/std/yaml_file_loader.cpp@1077: Using unregistered parameter 'dummy_rx' in component ''.
[component_container_mt-1] 2024-08-19 22:33:58.716 WARN  gxf/std/yaml_file_loader.cpp@1077: Using unregistered parameter 'dev_id' in component 'stream'.
[component_container_mt-1] [INFO] [1724074438.717427789] [foundationpose_node]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] 2024-08-19 22:33:58.721 WARN  gxf/std/scheduling_terms.cpp@333: 'min_size' parameter in MultiMessageAvailableSchedulingTerm is deprecated. Use 'min_sum' with SumOfAll sampling mode instead
[component_container_mt-1] 2024-08-19 22:33:58.721 WARN  gxf/std/scheduling_terms.cpp@333: 'min_size' parameter in MultiMessageAvailableSchedulingTerm is deprecated. Use 'min_sum' with SumOfAll sampling mode instead
[component_container_mt-1] 2024-08-19 22:33:58.721 WARN  gxf/std/scheduling_terms.cpp@333: 'min_size' parameter in MultiMessageAvailableSchedulingTerm is deprecated. Use 'min_sum' with SumOfAll sampling mode instead
[component_container_mt-1] 2024-08-19 22:33:58.721 WARN  gxf/std/scheduling_terms.cpp@333: 'min_size' parameter in MultiMessageAvailableSchedulingTerm is deprecated. Use 'min_sum' with SumOfAll sampling mode instead
[component_container_mt-1] 2024-08-19 22:33:58.721 WARN  gxf/std/scheduling_terms.cpp@333: 'min_size' parameter in MultiMessageAvailableSchedulingTerm is deprecated. Use 'min_sum' with SumOfAll sampling mode instead
[component_container_mt-1] [INFO] [1724074438.723003345] [foundationpose_node]: [NitrosNode] Node was started
[component_container_mt-1] [INFO] [1724074446.891331961] [rtdetr_preprocessor]: Negotiating
[component_container_mt-1] [WARN] [1724074451.516996823] [resize_mask_node.NitrosSubscriber]: 10.0s passed while waiting to push a message entity (eid=1901) to the receiver image_in
[component_container_mt-1] [WARN] [1724074466.822987938] [resize_mask_node.NitrosSubscriber]: 20.0s passed while waiting to push a message entity (eid=1901) to the receiver image_in
[component_container_mt-1] [WARN] [1724074482.119674235] [resize_mask_node.NitrosSubscriber]: 30.0s passed while waiting to push a message entity (eid=1901) to the receiver image_in
[component_container_mt-1] [WARN] [1724074497.403901845] [resize_mask_node.NitrosSubscriber]: 40.0s passed while waiting to push a message entity (eid=1901) to the receiver image_in
[component_container_mt-1] [WARN] [1724074512.678219366] [resize_mask_node.NitrosSubscriber]: 50.0s passed while waiting to push a message entity (eid=1901) to the receiver image_in
[component_container_mt-1] [WARN] [1724074527.973691996] [resize_mask_node.NitrosSubscriber]: 60.0s passed while waiting to push a message entity (eid=1901) to the receiver image_in
[component_container_mt-1] [WARN] [1724074543.280735195] [resize_mask_node.NitrosSubscriber]: 70.0s passed while waiting to push a message entity (eid=1901) to the receiver image_in
[component_container_mt-1] [WARN] [1724074558.545521391] [resize_mask_node.NitrosSubscriber]: 80.0s passed while waiting to push a message entity (eid=1901) to the receiver image_in

hirship1234 · August 22, 2024, 10:33am

After several attempts, it was confirmed that a dimension error occurred again between the depth image and the segmentation image.

Information on each topic is below.

$ ros2 topic echo /depth_image
header:
  stamp:
    sec: 1724322674
    nanosec: 147919360
  frame_id: camera_color_optical_frame
height: 720
width: 1280
encoding: 32FC1
is_bigendian: 0
step: 5120

$ ros2 topic echo /segmentation
header:
  stamp:
    sec: 1724322674
    nanosec: 147919360
  frame_id: camera_color_optical_frame
height: 480
width: 640
encoding: mono8
is_bigendian: 0
step: 640

So, I specified depth width and depth height in the realsense_mono_ depth.yaml file and ran launch. However, an error like the picture below appears.

After seeing that the RTX 4060 ti was working in the NGC Catalog, I purchased the 4070 ti super. Is it possible that foundationpose cannot be run on 16g? If it can’t be done, is there any other alternative? Since I recently purchased it, I think it will be difficult to exchange it right away.

Raffaello · August 23, 2024, 5:38pm

Thank you for your reply. We are still investigating the cause of this bug, and there is another thread related to the same error.

Could you please add the debug log on your demo?

--- a/isaac_ros_realsense/launch/isaac_ros_realsense_mono_rect_depth_core.launch.py
+++ b/isaac_ros_realsense/launch/isaac_ros_realsense_mono_rect_depth_core.launch.py
@@ -84,7 +84,8 @@ def generate_launch_description():
         composable_node_descriptions=(
             IsaacROSRealSenseMonoRectDepthLaunchFragment.get_composable_nodes()
         ),
-        output='screen'
+        output='screen',
+        arguments=['--ros-args', '--log-level', 'debug']
     )

hirship1234 · August 25, 2024, 9:41am

As you mentioned, I added a part to modify the log level in the launch file. The log results are as follows.

Even though I added the following to isaac_ros_examples/isaac_ros_realsense/config/realsense_mono_depth.yaml to modify the depth resolution, the dimension error still occurs.

/isaac_ros_realsense/config/realsense_mono_depth.yaml
+  depth_width: 640
+ depth_height: 480

Total Log (while running foundationpose)
Foundationpose Log.txt (109.1 KB)

bren1 · August 26, 2024, 11:46pm

I think you did the config wrong. my config looks like:

enable_infra1: false
enable_infra2: false
align_depth.enable: true
depth_module.depth_profile: 640x480x15
rgb_camera:
  profile: '640x480x15'
  color_qos: "SYSTEM_DEFAULT"

Which should sole the wrong images and there are some errors with the topics being wrong. I haven’t solved all the problems getting the realsense to properly work with foundation_pose.

hirship1234 · August 27, 2024, 1:06pm

Thank you for your answer. I have completed modifying realsense_mono_depth.yaml as you have mentioned. I am currently reviewing the internal code one by one, and if i find out anything, I will leave a comment here. Thank you.

bren1 · August 27, 2024, 4:38pm

I also had to modify the realsense launch file to have the remapping working properly:

        return {
            'camera_node': ComposableNode(
                package='realsense2_camera',
                plugin='realsense2_camera::RealSenseNodeFactory',
                name='realsense2_camera',
                namespace='',
                parameters=[
                    realsense_config_file_path
                ],
                remappings=[('/realsense2_camera/color/image_raw', '/image_rect'),
                            ('/realsense2_camera/color/camera_info', '/camera_info_rect'),
                            ('/realsense2_camera/depth/image_rect_raw', '/image_raw'),
                            ]
            ),
            # Realsense depth is in uint16 and millimeters. Convert to float32 and meters
            'convert_metric_node': ComposableNode(
                package='isaac_ros_depth_image_proc',
                plugin='nvidia::isaac_ros::depth_image_proc::ConvertMetricNode',
                name='convert_metric',
                remappings=[
                #     ('/realsense2_camera/aligned_depth_to_color/image_raw', '/aligned_depth_to_color/image_raw'),
                     ('/image', '/depth')
                ]
            ),
        }

As the name spacing wasn’t working as default. Not sure what over changes I had to get to no error msgs. But still not 100% working yet.

hirship1234 · August 27, 2024, 4:52pm

Thank you for reply.

In my situation, even though the rgb and depth profiles were clearly changed in the .yaml file, the actual ros2 topic does not seem to reflect the changed values. So, I’m looking at the realsense node factory code in this regard.

In my case, the topics seemed to be fine, but why did you modify the remapping? Or, if there is something I’m missing, I would appreciate it if you could give me some advice.

bren1 · August 27, 2024, 6:03pm

I’m using humble, I had to change the mapping as they where wrong. I looked at the rosbag examples and then realsense where publishing the wrong values. As it didn’t take into consideration the /realsense2_camera namespacing of the cameras. Also the depth map convertion wasn’t publishing ‘/depth’ that the pose module was expecting.

dsonmez · August 27, 2024, 6:45pm

I was having the same issue involving different output resolutions between /segmentation and /depth_image topics on the RealSense D455.

When I applied @bren1’s solution, the application was able to estimate the pose of my “Drill” object. However, the estimation result is too bad for the Drill, and it doesn’t work with some other scanned, custom objects as described here: Tutorial to create your own 3D object mesh for FoundationPose — isaac_ros_docs documentation

I have applied the steps to place the object on X,Y,Z axes and translation to center using the Meshlab. Here is the picture of the estimation result for the Drill object:

hirship1234 · August 28, 2024, 1:19am

I understand. In my case, I use Realsense d455, so if I use isaac ros realsense, the topic name comes out accurately, so that comments does not apply to me, but thank you for your comment.

hirship1234 · August 28, 2024, 1:28am

Thank you for replying.

I also used Realsense D455 and modified the .yaml file as @bren1 mentioned above. After that, by following instruction I created a banana custom 3d object mesh and moved the center point with meshlab. However, in my case, I keep getting an error that the depth dimension and segmentation dimension are different. How to solve that?

Also, I have two questions. First, I am running it in the order below, but is there something missing in the execution arguments?

cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
   ./scripts/run_dev.sh

pip install setuptools_scm==5.0.0
cd ${ISAAC_ROS_WS} && \
colcon build --symlink-install --packages-up-to-regex realsense*

rosdep install --from-paths ${ISAAC_ROS_WS}/src/isaac_ros_pose_estimation/isaac_ros_foundationpose --ignore-src -y
cd ${ISAAC_ROS_WS}/ && \
   colcon build --symlink-install --packages-up-to isaac_ros_foundationpose
   
rosdep install --from-paths ${ISAAC_ROS_WS}/src/isaac_ros_object_detection --ignore-src -y
cd ${ISAAC_ROS_WS} && \
   colcon build --symlink-install --packages-up-to isaac_ros_rtdetr

cd ${ISAAC_ROS_WS} && \
   colcon build --symlink-install --packages-up-to isaac_ros_examples

source install/setup.bash

sudo apt-get install -y ros-humble-isaac-ros-realsense

ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=realsense_mono_rect_depth,foundationpose mesh_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/banana/banana.obj texture_path:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/banana/materials/textures/baked_mesh_tex0.png score_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/score_trt_engine.plan refine_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/refine_trt_engine.plan rt_detr_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.plan

Second, I am currently using an rtx 4070 ti super vram 16g. What GPU are you using? I am worried that the GPU ram is insufficient.

hirship1234 · August 28, 2024, 3:43pm

To fix dimension error, I started over from basic realsense settings. So I checked and found out that the firmware version was different.

So I modified the firmware version as explained in that link.

However, the dimension error still persists. I modified the realsense_mono_depth.yaml file as below as mentioned by bren1, but I don’t know why the size of aligned_depth_to_color/image_raw is 1280x720.


> isaac_ros_examples/isaac_ros_realsense/config/realsense_mono_depth.yaml

enable_infra1: false
enable_infra2: false
align_depth.enable: true
depth_module.depth_profile: 640x480x15
rgb_camera:
  profile: '640x480x15'
  color_qos: "SYSTEM_DEFAULT"

@Raffaello Is there any way to solve this? And is it possible to operate it with rtx 4070 ti super (vram 16g)?

bren1 · August 28, 2024, 5:58pm

I’m not sure this is needed. As its not in the rosbag example and also in rqt_graph not used.

hirship1234 · August 28, 2024, 11:51pm

Thank you for reply. @bren1

If you look at isaac_ros_realsense_mono_rect_depth_core.launch.py in isaac_ros_exaples, you will see that ConvertMtericNode exists as shown below. In that node, aligned_depth_to_color/image_raw is received, the depth content is modified, and the depth is exported again.

So first of all, I thought that the width and height of aligned_depth_to_color/image_raw, which is the depth topic published by the realsense camera, should be changed.

/isaac_ros_examples/isaac_ros_realsense/launch/isaac_ros_realsense_mono_rect_depth_core.launch.py

# Realsense depth is in uint16 and millimeters. Convert to float32 and meters
'convert_metric_node': ComposableNode(
   package='isaac_ros_depth_image_proc',
   plugin='nvidia::isaac_ros::depth_image_proc::ConvertMetricNode',
   name='convert_metric',
   remappings=[
              ('image_raw', 'aligned_depth_to_color/image_raw'),
              ('image', 'depth
   ]
 ),

hirship1234 · September 1, 2024, 1:49am

Thank you all for your kind participation in the answers. The dimension error has been completely resolved. However, there is still a problem where the actual output result shakes randomly when moving the object a little. I am still thinking about this part and I am not sure.

system · September 15, 2024, 1:49am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Isaac_ros_foundationpose tracking takes approximately 20 seconds to show pose detection results upon starting rviz Isaac ROS camera , mesh , isaac-ros-object-detection , isaac-ros-pose-estimation	10	153	March 24, 2025
FoundationPose tutorial error: Missing GXF preset 'isaac_ros_foundationpose' Isaac ROS isaac-ros-pose-estimation	2	40	July 1, 2025
Isaac_ros_centerpose inference with Intel RealSense D435 stuck at "Unable to get power limit for GPU 0" Isaac ROS	4	85	February 7, 2025
/Centerpose/image-visualized no inference image display Isaac ROS	1	34	February 11, 2025
Isaac_ros_centerpose on a trained model for custom object usin TAO toolkit and data from isaac sim yields no inference Isaac ROS camera , ros , docker , isaac-ros-pose-estimation , ros-2-humble	7	61	February 12, 2025
Launch isaac-ros-foundationpose node failed Isaac ROS isaac-ros-pose-estimation	4	126	December 15, 2024
Can't complete isaac_ros_foundationpose quickstart Isaac ROS	5	610	July 1, 2024
Foundationpose package using orbbec camera Isaac ROS isaac-ros-pose-estimation	0	88	November 23, 2024
No inference results for custom object using isaac_ros_centerpose - Compression requires single-channel 32bit-floating point or 16bit raw depth images Isaac ROS	2	43	February 12, 2025
Isacc_ros_centerpose Isaac ROS	9	99	January 10, 2025

An Three error occurred during Foundationpose Quickstart

Related topics