Foundation pose with own .obj not working

[tina525241]

Hello,

I have run the foundation pose rosbag demo and rtdetr realsense demo successfully.
However, when I tried to detect my object, no mask was shown as the picture.

The cmd I used:
ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=realsense_mono_rect_depth,foundationpose mesh_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/dlink/dlink.obj texture_path:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_foundationpose/dlink/baked_mesh_tex0.png score_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/score_trt_engine.plan refine_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/foundationpose/refine_trt_engine.plan rt_detr_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.plan

dlink is my object and I created the .obj by following the turtorial

The camera I’m using is RealSense D435.

Here’s the log:
fp_log.txt (46.0 KB)

Could anyone tell me how to solve the problem? Thank you.

[Raffaello]

Hi @tina525241 welcome to the Isaac ROS forum,

Based on your log, all Isaac ROS packages are working correctly without any errors.

The only thing that catches my attention is this message:

[component_container_mt-1]  22/08 16:45:09,163 WARNING [140598363285056] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  22/08 16:45:09,174 WARNING [140598363285056] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  22/08 16:45:09,184 WARNING [140598363285056] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  22/08 16:45:09,194 WARNING [140598363285056] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11

This error is related to the Realsense camera and can be related to a wrong USB cable used. I found a similar issue related: error control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11 while trying slam · Issue #1663 · IntelRealSense/realsense-ros · GitHub

I suggest to check the cable connection and run the RealSenseViewer to verify that your camera works properly

Best,
Raffaello

[tina525241]

Hi @Raffaello thanks for the reply,

I modified isaac_ros_realsense/config/realsense_mono_depth.yaml by adding initial:=true, and filter:=pointcloud to try to solve the control_transfer error. Unfortunately, none of this work.

The camera works fine in realsense-viewer, and the foundation pose can detect the scissors properly with the command I used (/wo making a scissors mesh). I doubt it was the camera’s problem that I can’t detect my own object.

I found this post and modified the isaac_ros_realsense_mono_rect_depth_core.launch.py. Here is the log: log.txt (47.1 KB). Hope this could help figuring out what’s wrong.

Thank you.

[bren1]

Hello, I’m having the a similar problem. when I looked through the launch files, it seems that some are wrongly configure with regard to remapping. I had to add to the realsense_mono_depth.yaml → depth_module.depth_profile: 640x480x15 And also it looks like the launch files wrong for the remapping:

   @staticmethod
    def get_composable_nodes(interface_specs: Dict[str, Any]) -> Dict[str, ComposableNode]:
        realsense_config_file_path = os.path.join(
            get_package_share_directory('isaac_ros_realsense'),
            'config', 'realsense_mono_depth.yaml'
        )

        return {
            'camera_node': ComposableNode(
                package='realsense2_camera',
                plugin='realsense2_camera::RealSenseNodeFactory',
                name='realsense2_camera',
                namespace='',
                parameters=[
                    realsense_config_file_path
                ],
                remappings=[('/realsense2_camera/color/image_raw', '/image_rect'),
                            ('/realsense2_camera/color/camera_info', '/camera_info_rect'),
                            ('/realsense2_camera/depth/image_rect_raw', '/image_raw'),
                            ]
            ),
            # Realsense depth is in uint16 and millimeters. Convert to float32 and meters
            'convert_metric_node': ComposableNode(
                package='isaac_ros_depth_image_proc',
                plugin='nvidia::isaac_ros::depth_image_proc::ConvertMetricNode',
                name='convert_metric',
                remappings=[
                #     ('/realsense2_camera/aligned_depth_to_color/image_raw', '/aligned_depth_to_color/image_raw'),
                     ('/image', '/depth')
                ]
            ),
        }

But I’m still struggling to get it to work on a realsense or another camera.

[tina525241]

Hi @bren1 , have you got this whole thing working? Thanks.