Hi @Raffaello,
I’m having a problem with integrating RealSense D457 (GMSL). I know this specific camera model is not listed under the following guide but I can actually get the video stream using realsense-viewer and listen to ros messages of the ros2 launch realsense2_camera rs_launch.py command with the following order:
git clone https://github.com/IntelRealSense/realsense-ros.git -b 4.51.1
sudo apt install -y ros-humble-realsense2-camera
sudo pip3 uninstall -y setuptools-scm
sudo pip3 install setuptools==58.2.0
cd ${ISAAC_ROS_WS} && \
colcon build --symlink-install --packages-up-to-regex realsense*
realsense-viewer
# or
ros2 launch realsense2_camera rs_launch.py
ros2 launch realsense2_camera rs_launch.py:
However, when I run the Bi3D ros package with the following command, even though it is not throwing any errors, I can not get the visualization or listen to the topics it’s publishing. It just hangs:
ros2 launch isaac_ros_examples isaac_ros_examples.launch.py \
launch_fragments:=realsense_stereo_rect,bi3d \
interface_specs_file:=${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_bi3d/realsense_quickstart_interface_specs.json \
featnet_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/bi3d_proximity_segmentation/featnet.plan \
segnet_engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/bi3d_proximity_segmentation/segnet.plan \
max_disparity_values:=10
Here is the output of the ros launch:
[INFO] [launch]: All log files can be found below /home/admin/.ros/log/2024-08-23-17-54-32-327844-nvidia-desktop-263880
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container_mt-1]: process started with pid [263896]
[component_container_mt-1] [INFO] [1724450072.722712687] [isaac_ros_examples.container]: Load Library: /opt/ros/humble/lib/librealsense2_camera.so
[component_container_mt-1] [INFO] [1724450072.825729896] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<realsense2_camera::RealSenseNodeFactory>
[component_container_mt-1] [INFO] [1724450072.826628739] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<realsense2_camera::RealSenseNodeFactory>
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/realsense2_camera' in container '/isaac_ros_examples/container'
[component_container_mt-1] [INFO] [1724450072.842051175] [realsense2_camera]: RealSense ROS v4.55.1
[component_container_mt-1] [INFO] [1724450072.842116647] [realsense2_camera]: Built with LibRealSense v2.55.1
[component_container_mt-1] [INFO] [1724450072.842131782] [realsense2_camera]: Running with LibRealSense v2.55.1
[component_container_mt-1] [INFO] [1724450072.846151054] [isaac_ros_examples.container]: Load Library: /opt/ros/humble/lib/libimage_format_converter_node.so
[component_container_mt-1] [INFO] [1724450072.887462488] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::ImageFormatConverterNode>
[component_container_mt-1] [INFO] [1724450072.887556568] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::ImageFormatConverterNode>
[component_container_mt-1] [INFO] [1724450072.892827288] [image_format_left]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1724450072.893261686] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/std/libgxf_std.so
[component_container_mt-1] [INFO] [1724450072.897294205] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_gxf_helpers.so
[component_container_mt-1] [INFO] [1724450072.902195008] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_sight.so
[component_container_mt-1] [INFO] [1724450072.907653440] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_atlas.so
[component_container_mt-1] 2024-08-23 17:54:32.921 WARN gxf/std/program.cpp@532: No GXF scheduler specified.
[component_container_mt-1] [INFO] [1724450072.922104714] [image_format_left]: [ImageFormatConverterNode] Set output data format to: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1724450072.922232393] [image_format_left]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1724450072.931574129] [image_format_left]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1724450072.931691376] [image_format_left]: [NitrosContext] Loading extension: gxf/lib/multimedia/libgxf_multimedia.so
[component_container_mt-1] [INFO] [1724450072.935464410] [image_format_left]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_message_compositor.so
[component_container_mt-1] [INFO] [1724450072.936926545] [image_format_left]: [NitrosContext] Loading extension: gxf/lib/cuda/libgxf_cuda.so
[component_container_mt-1] [INFO] [1724450072.940657819] [image_format_left]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_tensorops.so
[component_container_mt-1] [INFO] [1724450072.947439378] [image_format_left]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1724450072.949242600] [image_format_left]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1724450073.003305957] [realsense2_camera]: Device with serial number 220422301857 was found.
[component_container_mt-1]
[component_container_mt-1] [INFO] [1724450073.003566468] [realsense2_camera]: Device with physical ID /sys/devices/platform/tegra-capture-vi/video4linux/video0 was found.
[component_container_mt-1] [INFO] [1724450073.003603396] [realsense2_camera]: Device with name Intel RealSense D457 was found.
[component_container_mt-1] [WARN] [1724450073.003887522] [realsense2_camera]: Error extracting usb port from device with physical ID: /sys/devices/platform/tegra-capture-vi/video4linux/video0
[component_container_mt-1] Please report on github issue at https://github.com/IntelRealSense/realsense-ros
[component_container_mt-1] [INFO] [1724450073.003999393] [realsense2_camera]: getParameters...
[component_container_mt-1] [INFO] [1724450073.004435135] [realsense2_camera]: JSON file is not provided
[component_container_mt-1] [INFO] [1724450073.004459838] [realsense2_camera]: Device Name: Intel RealSense D457
[component_container_mt-1] [INFO] [1724450073.004473534] [realsense2_camera]: Device Serial No: REDACTED
[component_container_mt-1] [INFO] [1724450073.004482718] [realsense2_camera]: Device physical port: /sys/devices/platform/tegra-capture-vi/video4linux/video0
[component_container_mt-1] [INFO] [1724450073.004503934] [realsense2_camera]: Device FW version: 5.16.0.1
[component_container_mt-1] [INFO] [1724450073.004513822] [realsense2_camera]: Device Product ID: 0xABCD
[component_container_mt-1] [INFO] [1724450073.004522558] [realsense2_camera]: Sync Mode: Off
[component_container_mt-1] [INFO] [1724450073.027612372] [image_format_left]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1724450073.036253440] [image_format_left]: [NitrosPublisherSubscriberGroup] Pinning the component "sink/sink" (type="nvidia::isaac_ros::MessageRelay") to use its compatible format only: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1724450073.040898564] [image_format_left]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/image_format_left' in container '/isaac_ros_examples/container'
[component_container_mt-1] [INFO] [1724450073.043745043] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::ImageFormatConverterNode>
[component_container_mt-1] [INFO] [1724450073.043824691] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::ImageFormatConverterNode>
[component_container_mt-1] [INFO] [1724450073.049084948] [image_format_right]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1724450073.049492369] [image_format_right]: [ImageFormatConverterNode] Set output data format to: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1724450073.049564145] [image_format_right]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1724450073.059953234] [image_format_right]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1724450073.060196081] [image_format_right]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1724450073.061630024] [image_format_right]: [NitrosNode] Running optimization
[component_container_mt-1] [WARN] [1724450073.100177698] [realsense2_camera]: Could not set param: depth_module.emitter_enabled with 1 Range: [0, 1]: parameter 'depth_module.emitter_enabled' has invalid type: expected [bool] got [integer]
[component_container_mt-1] [INFO] [1724450073.141610282] [image_format_right]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1724450073.154257854] [image_format_right]: [NitrosPublisherSubscriberGroup] Pinning the component "sink/sink" (type="nvidia::isaac_ros::MessageRelay") to use its compatible format only: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1724450073.157215789] [image_format_right]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/image_format_right' in container '/isaac_ros_examples/container'
[component_container_mt-1] [INFO] [1724450073.160135067] [isaac_ros_examples.container]: Load Library: /opt/ros/humble/lib/libbi3d_node.so
[component_container_mt-1] [INFO] [1724450073.166576789] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::bi3d::Bi3DNode>
[component_container_mt-1] [INFO] [1724450073.166658708] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::bi3d::Bi3DNode>
[component_container_mt-1] [INFO] [1724450073.171294360] [bi3d_node]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1724450073.171980948] [bi3d_node]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1724450073.176034300] [bi3d_node]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1724450073.176346586] [bi3d_node]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_sgm.so
[component_container_mt-1] [INFO] [1724450073.179387144] [bi3d_node]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_bi3d.so
[component_container_mt-1] [INFO] [1724450073.212189732] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/serialization/libgxf_serialization.so
[component_container_mt-1] [INFO] [1724450073.217924994] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_messages.so
[component_container_mt-1] [INFO] [1724450073.219600696] [bi3d_node]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1724450073.228926944] [bi3d_node]: [NitrosNode] Running optimization
[component_container_mt-1] [WARN] [1724450073.234161056] [realsense2_camera]: Could not set param: rgb_camera.exposure with nan Range: [nan, nan]: parameter 'rgb_camera.exposure' could not be set: Parameter {rgb_camera.exposure} doesn't comply with floating point range.
[component_container_mt-1] [INFO] [1724450073.251679192] [realsense2_camera]: No default profile found. Setting the first available profile as the default one.
[component_container_mt-1] [INFO] [1724450073.256252956] [realsense2_camera]: Stopping Sensor: Depth Module
[component_container_mt-1] [INFO] [1724450073.434335411] [realsense2_camera]: Starting Sensor: Depth Module
[component_container_mt-1] [INFO] [1724450073.465839063] [bi3d_node]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1724450073.491015520] [bi3d_node]: [NitrosPublisherSubscriberGroup] Expanding "any" data formats to all registered data formats
[component_container_mt-1] [INFO] [1724450073.491109247] [bi3d_node]: [NitrosNode] Starting negotiation...
[component_container_mt-1] [INFO] [1724450073.491769691] [image_format_left]: Negotiating
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/bi3d_node' in container '/isaac_ros_examples/container'
[component_container_mt-1] [INFO] [1724450073.491774011] [image_format_right]: Negotiating
[component_container_mt-1] [INFO] [1724450073.737054240] [realsense2_camera]: Open profile: stream_type: Infra(1), Format: Y8, Width: 640, Height: 480, FPS: 30
[component_container_mt-1] [INFO] [1724450073.737199327] [realsense2_camera]: Open profile: stream_type: Infra(2), Format: Y8, Width: 640, Height: 480, FPS: 30
[component_container_mt-1] [INFO] [1724450073.739880911] [realsense2_camera]: RealSense Node Is Up!
[component_container_mt-1] [WARN] [1724450073.764871258] [realsense2_camera]: Frame metadata isn't available! (frame_timestamp_domain = RS2_TIMESTAMP_DOMAIN_SYSTEM_TIME)
[component_container_mt-1] [INFO] [1724450074.041520066] [image_format_left]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1724450074.041624994] [image_format_left]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1724450074.041654178] [image_format_left]: [NitrosPublisher] Use the negotiated data format: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1724450074.041668961] [image_format_left]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1724450074.041677601] [image_format_left]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/infra1/image_rect_raw_mono", data_format="nitros_image_rgb8"
[component_container_mt-1] [INFO] [1724450074.041987424] [image_format_left]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1724450074.055048497] [image_format_left]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/QOSABFGFMA/QOSABFGFMA.yaml"
[component_container_mt-1] [INFO] [1724450074.055143121] [image_format_left]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1724450074.063964444] [image_format_left]: [ImageFormatConverterNode] postLoadGraphCallback().
[component_container_mt-1] [INFO] [1724450074.064087547] [image_format_left]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1724450074.157898664] [image_format_right]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1724450074.158020615] [image_format_right]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1724450074.158042983] [image_format_right]: [NitrosPublisher] Use the negotiated data format: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1724450074.158058759] [image_format_right]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1724450074.158069031] [image_format_right]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/infra2/image_rect_raw_mono", data_format="nitros_image_rgb8"
[component_container_mt-1] [INFO] [1724450074.158317125] [image_format_right]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1724450074.175503550] [image_format_right]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/GAYLVFINGA/GAYLVFINGA.yaml"
[component_container_mt-1] [INFO] [1724450074.175610782] [image_format_right]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1724450074.226252814] [image_format_left]: [NitrosNode] Node was started
[component_container_mt-1] [INFO] [1724450074.227866948] [image_format_right]: [ImageFormatConverterNode] postLoadGraphCallback().
[component_container_mt-1] [INFO] [1724450074.227948900] [image_format_right]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1724450074.256000027] [image_format_right]: [NitrosNode] Node was started
[component_container_mt-1] [INFO] [1724450074.491683957] [bi3d_node]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1724450074.491774069] [bi3d_node]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1724450074.491789621] [bi3d_node]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1724450074.491799508] [bi3d_node]: [NitrosPublisher] Use only the compatible publisher: topic_name="/bi3d_node/bi3d_output", data_format="nitros_disparity_image_32FC1"
[component_container_mt-1] [INFO] [1724450074.491810004] [bi3d_node]: [NitrosSubscriber] Use the negotiated data format: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1724450074.491821940] [bi3d_node]: [NitrosSubscriber] Use the negotiated data format: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1724450074.491829268] [bi3d_node]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1724450074.491835508] [bi3d_node]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/left/camera_info_rect", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1724450074.491843316] [bi3d_node]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1724450074.491849588] [bi3d_node]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/right/camera_info_rect", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1724450074.491969811] [bi3d_node]: [NitrosPublisherSubscriberGroup] Adjusted the compatible format of the component "disparity_roundrobin/data_receiver" (type="nvidia::gxf::DoubleBufferReceiver") from "" to "nitros_bi3d_inference_param_array"
[component_container_mt-1] [INFO] [1724450074.492142450] [bi3d_node]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1724450074.775343119] [bi3d_node]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/ZFRDBKMIUE/ZFRDBKMIUE.yaml"
[component_container_mt-1] [INFO] [1724450074.775463854] [bi3d_node]: In Bi3D Node preLoadGraphCallback().
[component_container_mt-1] [INFO] [1724450074.775517454] [bi3d_node]: [NitrosNode] Loading application
[component_container_mt-1] 2024-08-23 17:54:34.783 WARN gxf/std/yaml_file_loader.cpp@1077: Using unregistered parameter 'dev_id' in component 'stream'.
[component_container_mt-1] 2024-08-23 17:54:34.783 WARN gxf/std/yaml_file_loader.cpp@1077: Using unregistered parameter 'forward_pool' in component ''.
[component_container_mt-1] 2024-08-23 17:54:34.783 WARN gxf/std/yaml_file_loader.cpp@1077: Using unregistered parameter 'dev_id' in component 'stream'.
[component_container_mt-1] 2024-08-23 17:54:34.784 WARN gxf/std/yaml_file_loader.cpp@1077: Using unregistered parameter 'forward_pool' in component ''.
[component_container_mt-1] 2024-08-23 17:54:34.784 WARN gxf/std/yaml_file_loader.cpp@1077: Using unregistered parameter 'disparity_tensor_name' in component ''.
[component_container_mt-1] 2024-08-23 17:54:34.784 WARN gxf/std/yaml_file_loader.cpp@1077: Using unregistered parameter 'disparity_values_tensor_name' in component ''.
[component_container_mt-1] [INFO] [1724450074.785752849] [bi3d_node]: [Bi3DNode] Setting featnet_engine_file_path: /workspaces/isaac_ros-dev/isaac_ros_assets/models/bi3d_proximity_segmentation/featnet.plan.
[component_container_mt-1] [INFO] [1724450074.785848496] [bi3d_node]: [Bi3DNode] Setting segnet_engine_file_path: /workspaces/isaac_ros-dev/isaac_ros_assets/models/bi3d_proximity_segmentation/segnet.plan.
[component_container_mt-1] [INFO] [1724450074.785863312] [bi3d_node]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] 2024-08-23 17:54:34.791 WARN gxf/std/scheduling_terms.cpp@333: 'min_size' parameter in MultiMessageAvailableSchedulingTerm is deprecated. Use 'min_sum' with SumOfAll sampling mode instead
[component_container_mt-1] [INFO] [1724450075.003403927] [bi3d_node]: [NitrosNode] Node was started
I also tried the solution presented in the topic below, but ros2-development branch requires librealsense version 2.56.0 which doesn’t exist. There is only 2.56.1 might be logical but when I install that, it throws D457 not supported error.
What would be the reason for ROS packages to hang even though we can get video stream from both the viewer and plain realsense ros package? Thanks in advance.
