Isaac ROS Argus Camera on Non-GMSL Stereocamera

Robotics & Edge Computing Robotics - Isaac Isaac ROS
1

Hi folks,

We are trying to run ESS Stereo Disparity Node from Isaac Ros on our custom Stereocamera based on Orin NX.

SETUP
We have modified the Devicetree and created Kernel drivers to include two IMX264 image sensors which are each connected to a 4x CSI port on the Orin NX, respectively. The two image sensors are synchronized in hardware.

Two video ports /dev/video0 and /dev/video1 appear in the filesystem as expected.

We have verified that both image sensors are working fine with the following Gstreamer pipeline(s):

On the camera:

gst-launch-1.0 \
nvarguscamerasrc sensor-id=0 aelock=true ! 'video/x-raw(memory:NVMM), width=(int)2448, height=(int)2048, format=(string)NV12, framerate=(fraction)30/1' ! cammux.sink_1 \
nvarguscamerasrc sensor-id=1 aelock=true ! 'video/x-raw(memory:NVMM), width=(int)2448, height=(int)2048, format=(string)NV12, framerate=(fraction)30/1' ! cammux.sink_0 \
\
nvstreammux name=cammux width=2448 height=2048 batch-size=2 batched-push-timeout=20000 ! \
nvmultistreamtiler rows=1 columns=2 width=4896 height=2048 ! \
nvvideoconvert ! "video/x-raw(memory:NVMM), format=(string)I420" ! \
nvv4l2h265enc preset-level=1 control-rate=1 bitrate=5000000 ! \
h265parse ! rtph265pay mtu=1400 config-interval=1 pt=96 ! udpsink host=$CLIENT_IP port=5000 sync=false async=false

On the host:

gst-launch-1.0 udpsrc port=5000 ! application/x-rtp,encoding-name=H265,payload=96 ! \
rtph265depay ! h265parse ! queue ! avdec_h265 ! xvimagesink sync=false async=false

Isaac ROS
I can successfully setup the Isaac ROS environment with run_dev.sh and start an nvidia::isaac_ros::argus::ArgusMonoNode by using “ros2 launch isaac_ros_argus_camera isaac_ros_argus_camera_mono.launch.py”. The stream of /dev/video0 is successfully published as /left/image_raw.

However, when I try to launch the stereo node I get ERROR extensions/hawk/argus_camera.cpp@229: No synchronized stereo camera modules found, but camera type was specified as stereo:

admin@tegra-ubuntu:/workspaces/isaac_ros-dev$ ros2 launch isaac_ros_argus_camera isaac_ros_argus_camera_stereo.launch.py
[INFO] [launch]: All log files can be found below /home/admin/.ros/log/2023-07-07-11-45-17-369569-tegra-ubuntu-86846
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container_mt-1]: process started with pid [86857]
[component_container_mt-1] [INFO] [1688730317.835126181] [argus_stereo_container]: Load Library: /workspaces/isaac_ros-dev/install/isaac_ros_argus_camera/lib/libstereo_node.so
[component_container_mt-1] [INFO] [1688730317.942722387] [argus_stereo_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::argus::ArgusStereoNode>
[component_container_mt-1] [INFO] [1688730317.942849942] [argus_stereo_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::argus::ArgusStereoNode>
[component_container_mt-1] [INFO] [1688730317.950296375] [NitrosContext]: [NitrosContext] Creating a new shared context
[component_container_mt-1] [INFO] [1688730317.950484861] [argus_stereo]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1688730317.951512732] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/std/libgxf_std.so
[component_container_mt-1] [INFO] [1688730317.955899136] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_gxf_helpers.so
[component_container_mt-1] [INFO] [1688730317.958706197] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_sight.so
[component_container_mt-1] [INFO] [1688730317.961664878] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_atlas.so
[component_container_mt-1] [INFO] [1688730317.965711688] [NitrosContext]: [NitrosContext] Loading application: '/workspaces/isaac_ros-dev/install/isaac_ros_nitros/share/isaac_ros_nitros/config/type_adapter_nitros_context_graph.yaml'
[component_container_mt-1] [INFO] [1688730317.966797097] [NitrosContext]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1688730317.968824550] [NitrosContext]: [NitrosContext] Running application...
[component_container_mt-1] 2023-07-07 11:45:17.968 WARN  gxf/std/program.cpp@456: No system specified. Nothing to do
[component_container_mt-1] [INFO] [1688730317.973645271] [argus_stereo]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1688730317.973703769] [argus_stereo]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1688730317.976605073] [argus_stereo]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1688730317.976664434] [argus_stereo]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1688730317.976696531] [argus_stereo]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1688730317.977926040] [argus_stereo]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1688730317.978215233] [argus_stereo]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1688730317.978652718] [argus_stereo]: [NitrosContext] Loading extension: gxf/lib/libgxf_message_compositor.so
[component_container_mt-1] [INFO] [1688730317.980093786] [argus_stereo]: [NitrosContext] Loading extension: gxf/lib/cuda/libgxf_cuda.so
[component_container_mt-1] [INFO] [1688730317.982628550] [argus_stereo]: [NitrosContext] Loading extension: gxf/lib/serialization/libgxf_serialization.so
[component_container_mt-1] [INFO] [1688730317.986505723] [argus_stereo]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_messages.so
[component_container_mt-1] [INFO] [1688730317.990701690] [argus_stereo]: [NitrosContext] Loading extension: gxf/lib/multimedia/libgxf_multimedia.so
[component_container_mt-1] [INFO] [1688730317.991336397] [argus_stereo]: [NitrosContext] Loading extension: gxf/lib/image_proc/libgxf_tensorops.so
[component_container_mt-1] [INFO] [1688730317.995330821] [argus_stereo]: [NitrosContext] Loading extension: gxf/lib/image_proc/libgxf_rectify_params_generator.so
[component_container_mt-1] [INFO] [1688730318.001426877] [argus_stereo]: [NitrosContext] Loading extension: gxf/lib/libgxf_argus.so
[component_container_mt-1] [INFO] [1688730318.152117438] [argus_stereo]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1688730318.158777735] [argus_stereo]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1688730318.241585352] [argus_stereo]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1688730318.243492258] [argus_stereo]: [NitrosNode] Creating negotiated publishers/subscribers
[component_container_mt-1] [INFO] [1688730318.257407430] [argus_stereo]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/argus_stereo' in container '/argus_stereo_container'
[component_container_mt-1] [INFO] [1688730319.260168136] [argus_stereo]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1688730319.260541011] [argus_stereo]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1688730319.260638358] [argus_stereo]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1688730319.260722489] [argus_stereo]: [NitrosPublisher] Use only the compatible publisher: topic_name="/left/image_raw", data_format="nitros_image_rgb8"
[component_container_mt-1] [INFO] [1688730319.260779771] [argus_stereo]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1688730319.260815804] [argus_stereo]: [NitrosPublisher] Use only the compatible publisher: topic_name="/right/image_raw", data_format="nitros_image_rgb8"
[component_container_mt-1] [INFO] [1688730319.261268617] [argus_stereo]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1688730319.261310251] [argus_stereo]: [NitrosPublisher] Use only the compatible publisher: topic_name="/left/camerainfo", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1688730319.261338443] [argus_stereo]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1688730319.261363436] [argus_stereo]: [NitrosPublisher] Use only the compatible publisher: topic_name="/right/camerainfo", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1688730319.261677686] [argus_stereo]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1688730319.295222505] [argus_stereo]: [NitrosNode] Wrote the final top level YAML graph to "/workspaces/isaac_ros-dev/install/isaac_ros_argus_camera/share/isaac_ros_argus_camera/IDKJHCUHFS.yaml"
[component_container_mt-1] [INFO] [1688730319.295752953] [argus_stereo]: [NitrosNode] Calling user's pre-load-graph callback
[component_container_mt-1] [INFO] [1688730319.295811515] [argus_stereo]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1688730319.295866813] [argus_stereo]: [NitrosContext] Loading application: '/workspaces/isaac_ros-dev/install/isaac_ros_argus_camera/share/isaac_ros_argus_camera/IDKJHCUHFS.yaml'
[component_container_mt-1] 2023-07-07 11:45:19.305 WARN  gxf/std/yaml_file_loader.cpp@952: Using unregistered parameter 'left_camera_frame' in component ''.
[component_container_mt-1] 2023-07-07 11:45:19.306 WARN  gxf/std/yaml_file_loader.cpp@952: Using unregistered parameter 'right_camera_frame' in component ''.
[component_container_mt-1] 2023-07-07 11:45:19.306 WARN  gxf/std/yaml_file_loader.cpp@952: Using unregistered parameter 'transmitter' in component ''.
[component_container_mt-1] 2023-07-07 11:45:19.306 WARN  gxf/std/yaml_file_loader.cpp@952: Using unregistered parameter 'transmitter' in component ''.
[component_container_mt-1] [INFO] [1688730319.309733183] [argus_stereo]: [NitrosNode] Linking Nitros pub/sub to the loaded application
[component_container_mt-1] [INFO] [1688730319.310490774] [argus_stereo]: [NitrosNode] Calling user's post-load-graph callback
[component_container_mt-1] [INFO] [1688730319.310691548] [argus_stereo]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1688730319.764105894] [argus_stereo]: [NitrosContext] Running application...
[component_container_mt-1] [INFO] [1688730319.764334476] [argus_stereo]: [NitrosNode] Starting a heartbeat timer (eid=72)
[component_container_mt-1] 2023-07-07 11:45:19.824 ERROR extensions/hawk/argus_camera.cpp@229: No synchronized stereo camera modules found, but camera type was specified as stereo
[component_container_mt-1] 2023-07-07 11:45:19.824 ERROR gxf/std/entity_executor.cpp@540: Entity [IDKJHCUHFS_argus_camera] must be in Lifecycle::kStarted or Lifecycle::kIdle stage before stopping. Current state is StartPending
[component_container_mt-1] 2023-07-07 11:45:19.824 WARN  gxf/std/greedy_scheduler.cpp@241: Error while executing entity 21 named 'IDKJHCUHFS_argus_camera': GXF_FAILURE
[component_container_mt-1] 2023-07-07 11:45:20.764 ERROR gxf/std/entity_executor.cpp@203: Entity with eid 72 not found!
[component_container_mt-1] [WARN] [1688730320.764935178] [argus_stereo]: [NitrosNode] The heartbeat entity (eid=72) was stopped. The graph may have been terminated.
[component_container_mt-1] [INFO] [1688730320.779522402] [argus_stereo]: [NitrosNode] Terminating the running application
[component_container_mt-1] [INFO] [1688730320.779573155] [argus_stereo]: [NitrosContext] Interrupting GXF...
[component_container_mt-1] [INFO] [1688730320.779634181] [argus_stereo]: [NitrosContext] Waiting on GXF...
[component_container_mt-1] 2023-07-07 11:45:20.779 ERROR gxf/std/program.cpp@497: wait failed. Deactivating...
[component_container_mt-1] 2023-07-07 11:45:20.832 ERROR gxf/core/runtime.cpp@1251: Graph wait failed with error: GXF_FAILURE
[component_container_mt-1] [ERROR] [1688730320.832320282] [argus_stereo]: [NitrosContext] GxfGraphWait Error: GXF_FAILURE
[component_container_mt-1] [INFO] [1688730320.832358619] [argus_stereo]: [NitrosNode] Application termination done
[component_container_mt-1] |==================================================================================================================================================================|
[component_container_mt-1] |                                           Job Statistics Report (regular)                                                                                        |
[component_container_mt-1] |==================================================================================================================================================================|
[component_container_mt-1] | Name                                               |   Count | Time (Median - 90% - Max) [ms] | Load (%) | Exec(ms) | Variation (Median - 90% - Max) [ns]        |
[component_container_mt-1] |------------------------------------------------------------------------------------------------------------------------------------------------------------------|
[component_container_mt-1] |==================================================================================================================================================================|
[component_container_mt-1] |==================================================================================================================================================================|
[component_container_mt-1] |                                           Entity Statistics Report (regular)                                                                                     |
[component_container_mt-1] |==================================================================================================================================================================|
[component_container_mt-1] | Entity Name             | Entity State             |   Count | Time (Median - 90% - Max) [ms]                                                                    |
[component_container_mt-1] |------------------------------------------------------------------------------------------------------------------------------------------------------------------|
[component_container_mt-1] |==================================================================================================================================================================|
[INFO] [component_container_mt-1]: process has finished cleanly [pid 86857]

I also tried to launch two separate ArgusMonoNodes with appropriate settings like so:

def generate_launch_description():
    argus_mono_node_right = ComposableNode(
        name='argus_right',
        namespace='right',
        package='isaac_ros_argus_camera',
        plugin='nvidia::isaac_ros::argus::ArgusMonoNode',
        remappings=[
            ('left/image_raw', 'image_raw'),
            ('left/camerainfo', 'camera_info'),
            ('left/image_raw/nitros', 'image_raw/nitros'),
            ('left/camerainfo/nitros', 'camerainfo/nitros'),
        ],
        parameters=[{
            'camera_id': 0,
            'module_id': 0,
            'mode': 0,
            'camera_type': 0,
            'camera_info_url': '',
            'optical_frame_name': 'right_sensor',
        }],
    )
    argus_mono_node_left = ComposableNode(
        name='argus_left',
        namespace='left',
        package='isaac_ros_argus_camera',
        plugin='nvidia::isaac_ros::argus::ArgusMonoNode',
        remappings=[
            ('left/image_raw', 'image_raw'),
            ('left/camerainfo', 'camerainfo'),
            ('left/image_raw/nitros', 'image_raw/nitros'),
            ('left/camerainfo/nitros', 'camerainfo/nitros'),
        ],
        parameters=[{
            'camera_id': 1,
            'module_id': 1,
            'mode': 0,
            'camera_type': 0,
            'camera_info_url': '',
            'optical_frame_name': 'left_sensor',
        }],
    )
    argus_stereo_container = ComposableNodeContainer(
            name='argus_stereo_container',
            package='rclcpp_components',
            executable='component_container_mt',
            composable_node_descriptions=[argus_mono_node_right, argus_mono_node_left],
            namespace='',
            output='screen',
            arguments=['--ros-args', '--log-level', 'info'],
        )
    return launch.LaunchDescription([argus_stereo_container])

The above launch script led to ERROR extensions/hawk/argus_camera.cpp@156: Error while listing camera devices: Failed to get ICameraProvider interface:

[INFO] [launch]: All log files can be found below /home/admin/.ros/log/2023-07-07-11-57-47-870758-tegra-ubuntu-87019
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container_mt-1]: process started with pid [87030]
[component_container_mt-1] [INFO] [1688731068.339234983] [argus_stereo_container]: Load Library: /workspaces/isaac_ros-dev/install/isaac_ros_argus_camera/lib/libmono_node.so
[component_container_mt-1] [INFO] [1688731068.437806879] [argus_stereo_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::argus::ArgusMonoNode>
[component_container_mt-1] [INFO] [1688731068.437952675] [argus_stereo_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::argus::ArgusMonoNode>
[component_container_mt-1] [INFO] [1688731068.444795403] [NitrosContext]: [NitrosContext] Creating a new shared context
[component_container_mt-1] [INFO] [1688731068.444990640] [right.argus_right]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1688731068.445846633] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/std/libgxf_std.so
[component_container_mt-1] [INFO] [1688731068.450006338] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_gxf_helpers.so
[component_container_mt-1] [INFO] [1688731068.452834165] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_sight.so
[component_container_mt-1] [INFO] [1688731068.455742314] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_atlas.so
[component_container_mt-1] [INFO] [1688731068.459826689] [NitrosContext]: [NitrosContext] Loading application: '/workspaces/isaac_ros-dev/install/isaac_ros_nitros/share/isaac_ros_nitros/config/type_adapter_nitros_context_graph.yaml'
[component_container_mt-1] [INFO] [1688731068.460871391] [NitrosContext]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1688731068.462788279] [NitrosContext]: [NitrosContext] Running application...
[component_container_mt-1] 2023-07-07 11:57:48.462 WARN  gxf/std/program.cpp@456: No system specified. Nothing to do
[component_container_mt-1] [INFO] [1688731068.467539233] [right.argus_right]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1688731068.467596067] [right.argus_right]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1688731068.470496439] [right.argus_right]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1688731068.470557305] [right.argus_right]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1688731068.471396946] [right.argus_right]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1688731068.471430867] [right.argus_right]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1688731068.471712955] [right.argus_right]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1688731068.472105894] [right.argus_right]: [NitrosContext] Loading extension: gxf/lib/libgxf_message_compositor.so
[component_container_mt-1] [INFO] [1688731068.473465358] [right.argus_right]: [NitrosContext] Loading extension: gxf/lib/cuda/libgxf_cuda.so
[component_container_mt-1] [INFO] [1688731068.475910293] [right.argus_right]: [NitrosContext] Loading extension: gxf/lib/serialization/libgxf_serialization.so
[component_container_mt-1] [INFO] [1688731068.479727300] [right.argus_right]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_messages.so
[component_container_mt-1] [INFO] [1688731068.483764762] [right.argus_right]: [NitrosContext] Loading extension: gxf/lib/multimedia/libgxf_multimedia.so
[component_container_mt-1] [INFO] [1688731068.484354667] [right.argus_right]: [NitrosContext] Loading extension: gxf/lib/image_proc/libgxf_tensorops.so
[component_container_mt-1] [INFO] [1688731068.488157594] [right.argus_right]: [NitrosContext] Loading extension: gxf/lib/image_proc/libgxf_rectify_params_generator.so
[component_container_mt-1] [INFO] [1688731068.494068294] [right.argus_right]: [NitrosContext] Loading extension: gxf/lib/libgxf_argus.so
[component_container_mt-1] [INFO] [1688731068.643716750] [right.argus_right]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1688731068.647526941] [right.argus_right]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1688731068.664278213] [right.argus_right]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1688731068.665528874] [right.argus_right]: [NitrosNode] Creating negotiated publishers/subscribers
[component_container_mt-1] [INFO] [1688731068.672673082] [right.argus_right]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/right/argus_right' in container '/argus_stereo_container'
[component_container_mt-1] [INFO] [1688731068.677571049] [argus_stereo_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::argus::ArgusMonoNode>
[component_container_mt-1] [INFO] [1688731068.677637707] [argus_stereo_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::argus::ArgusMonoNode>
[component_container_mt-1] [INFO] [1688731068.683186252] [left.argus_left]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1688731068.684850525] [left.argus_left]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1688731068.684889342] [left.argus_left]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1688731068.687058269] [left.argus_left]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1688731068.687113919] [left.argus_left]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1688731068.687959479] [left.argus_left]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1688731068.687994712] [left.argus_left]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1688731068.688247392] [left.argus_left]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1688731068.691923659] [left.argus_left]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1688731068.694816895] [left.argus_left]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1688731068.710935445] [left.argus_left]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1688731068.711967283] [left.argus_left]: [NitrosNode] Creating negotiated publishers/subscribers
[component_container_mt-1] [INFO] [1688731068.715716096] [left.argus_left]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/left/argus_left' in container '/argus_stereo_container'
[component_container_mt-1] [INFO] [1688731069.674389499] [right.argus_right]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1688731069.674699204] [right.argus_right]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1688731069.674808168] [right.argus_right]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1688731069.674854825] [right.argus_right]: [NitrosPublisher] Use only the compatible publisher: topic_name="/right/image_raw", data_format="nitros_image_rgb8"
[component_container_mt-1] [INFO] [1688731069.674894250] [right.argus_right]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1688731069.674927659] [right.argus_right]: [NitrosPublisher] Use only the compatible publisher: topic_name="/right/camera_info", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1688731069.675212147] [right.argus_right]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1688731069.695225370] [right.argus_right]: [NitrosNode] Wrote the final top level YAML graph to "/workspaces/isaac_ros-dev/install/isaac_ros_argus_camera/share/isaac_ros_argus_camera/KDKDGAXCVF.yaml"
[component_container_mt-1] [INFO] [1688731069.695535555] [right.argus_right]: [NitrosNode] Calling user's pre-load-graph callback
[component_container_mt-1] [INFO] [1688731069.695590469] [right.argus_right]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1688731069.695649255] [right.argus_right]: [NitrosContext] Loading application: '/workspaces/isaac_ros-dev/install/isaac_ros_argus_camera/share/isaac_ros_argus_camera/KDKDGAXCVF.yaml'
[component_container_mt-1] 2023-07-07 11:57:49.703 WARN  gxf/std/yaml_file_loader.cpp@952: Using unregistered parameter 'left_camera_frame' in component ''.
[component_container_mt-1] 2023-07-07 11:57:49.704 WARN  gxf/std/yaml_file_loader.cpp@952: Using unregistered parameter 'right_camera_frame' in component ''.
[component_container_mt-1] 2023-07-07 11:57:49.704 WARN  gxf/std/yaml_file_loader.cpp@952: Using unregistered parameter 'transmitter' in component ''.
[component_container_mt-1] [INFO] [1688731069.707632900] [right.argus_right]: [NitrosNode] Linking Nitros pub/sub to the loaded application
[component_container_mt-1] [INFO] [1688731069.708322168] [right.argus_right]: [NitrosNode] Calling user's post-load-graph callback
[component_container_mt-1] [INFO] [1688731069.708543422] [right.argus_right]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1688731069.717151129] [left.argus_left]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1688731069.717521284] [left.argus_left]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1688731069.717579142] [left.argus_left]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1688731069.717654440] [left.argus_left]: [NitrosPublisher] Use only the compatible publisher: topic_name="/left/image_raw", data_format="nitros_image_rgb8"
[component_container_mt-1] [INFO] [1688731069.717700777] [left.argus_left]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1688731069.717724842] [left.argus_left]: [NitrosPublisher] Use only the compatible publisher: topic_name="/left/camerainfo", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1688731069.717850766] [left.argus_left]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1688731069.731795716] [left.argus_left]: [NitrosNode] Wrote the final top level YAML graph to "/workspaces/isaac_ros-dev/install/isaac_ros_argus_camera/share/isaac_ros_argus_camera/CSFZZNAAYT.yaml"
[component_container_mt-1] [INFO] [1688731069.731954441] [left.argus_left]: [NitrosNode] Calling user's pre-load-graph callback
[component_container_mt-1] [INFO] [1688731069.731983497] [left.argus_left]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1688731070.054268855] [right.argus_right]: [NitrosContext] Running application...
[component_container_mt-1] [INFO] [1688731070.054520223] [right.argus_right]: [NitrosNode] Starting a heartbeat timer (eid=44)
[component_container_mt-1] [INFO] [1688731070.054536191] [left.argus_left]: [NitrosContext] Loading application: '/workspaces/isaac_ros-dev/install/isaac_ros_argus_camera/share/isaac_ros_argus_camera/CSFZZNAAYT.yaml'
[component_container_mt-1] 2023-07-07 11:57:50.057 WARN  gxf/std/yaml_file_loader.cpp@952: Using unregistered parameter 'left_camera_frame' in component ''.
[component_container_mt-1] 2023-07-07 11:57:50.057 WARN  gxf/std/yaml_file_loader.cpp@952: Using unregistered parameter 'right_camera_frame' in component ''.
[component_container_mt-1] 2023-07-07 11:57:50.057 WARN  gxf/std/yaml_file_loader.cpp@952: Using unregistered parameter 'transmitter' in component ''.
[component_container_mt-1] [INFO] [1688731070.058491538] [left.argus_left]: [NitrosNode] Linking Nitros pub/sub to the loaded application
[component_container_mt-1] [INFO] [1688731070.058692536] [left.argus_left]: [NitrosNode] Calling user's post-load-graph callback
[component_container_mt-1] [INFO] [1688731070.058746618] [left.argus_left]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1688731070.062374883] [left.argus_left]: [NitrosContext] Running application...
[component_container_mt-1] [INFO] [1688731070.062561129] [left.argus_left]: [NitrosNode] Starting a heartbeat timer (eid=96)
[component_container_mt-1] (NvCameraUtils) Error InvalidState: Mutex already initialized (in Mutex.cpp, function initialize(), line 41)
[component_container_mt-1] (Argus) Error InvalidState:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function open(), line 54)
[component_container_mt-1] (Argus) Error InvalidState:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function openSocketConnection(), line 262)
[component_container_mt-1] (Argus) Error InvalidState: Cannot create camera provider (in src/rpc/socket/client/SocketClientDispatch.cpp, function createCameraProvider(), line 106)
[component_container_mt-1] 2023-07-07 11:57:50.063 ERROR extensions/hawk/argus_camera.cpp@156: Error while listing camera devices: Failed to get ICameraProvider interface
[ERROR] [component_container_mt-1]: process has died [pid 87030, exit code -11, cmd '/opt/ros/humble/install/lib/rclcpp_components/component_container_mt --ros-args --log-level info --ros-args -r __node:=argus_stereo_container -r __ns:=/'].

Is it correct to assume that both left and right image are expected to be streamed via /dev/video0? Which would require GSML with virtual channels?

Is there any workaround for this? So that the ROS ArgusStereoNode can be launched with two video ports /dev/video0 and /dev/video1 (so two physical CSI ports)?

Any help is highly appreciated, thank you!

2

If I understood correctly, you’re trying to develop a stereo camera from a rigid enclosure with two IMX264s with a hardware trigger. libArgus relies on drivers developed by vendors that ship with Jetpack for specific, supported cameras over GMSL2. The IMX264s will have a monocular driver, but a pair of them wouldn’t appear as a stereo camera to Argus which considers a stereo camera as a single “camera” with two imagers. A stereo camera driver that knows that the two imagers are part of the same camera, for example, could also provide stereo rectification parameters and extrinsics of the right imager relative to the left.

You can take a look at the Argus docs to consider writing your own driver. You might find it helpful to write your own ROS node that manages the two streams and publishes the image pairs yourself than going through Argus too.

3

@hemals

Thank you for the quick reply.

Argus

Ok, this explains why ArgusStereoNode is not working. However, since we can run nvarguscamerasrc with sensor-id 0 and 1, I thought it would be possible to run two ArgusMonoNodes with the different sensor-ids and appropriate camera_info_urls for the calibration data?

Can you confirm that this is not possible?

We developed the driver according to the Camera Development Section in the Docs and it it is working fine with below Gstreamer pipeline

(Note: We developed the camera for a different deepstream based application which is working fine. Now, we would like to evaluate Isaac ROS ESS on the same hardware if possible)

I could not find any information in the Docs as to how the image sensor kernel module would have to be modified to be “stereo-compatible”.

Can you let me know if our hardware setup (see picture below) would even allow us to modify the sensor kernel driver for stereo-compatability (non-monocular) so that we can run Isaac ROS ArgusStereoNode?

hw_schematic

If this would be possible, can you provide any ressources on how to modify the driver so that it becomes a “stereo-compatible” driver?

Workaround

As a workaround I created some ROS nodes which publish [left | right]/[image_raw | camera_info] as you suggested. Then I launched two Isaac ROS rectify nodes + Isaac ROS Ess and Isaac ROS pointcloudnode like so:

import os
from ament_index_python.packages import get_package_share_directory

import launch
from launch.actions import DeclareLaunchArgument, IncludeLaunchDescription
from launch.substitutions import LaunchConfiguration
from launch_ros.actions import ComposableNodeContainer
from launch_ros.descriptions import ComposableNode
from launch.launch_description_sources import PythonLaunchDescriptionSource


def generate_launch_description():
    launch_args = [
        DeclareLaunchArgument(
            'engine_file_path',
            default_value='',
            description='The absolute path to the ESS engine plan.'),
    ]
    engine_file_path = LaunchConfiguration('engine_file_path')

    left_rectify_node = ComposableNode(
        name='left_rectify_node',
        package='isaac_ros_image_proc',
        plugin='nvidia::isaac_ros::image_proc::RectifyNode',
        parameters=[{
            'output_width': 1920,
            'output_height': 1200,
        }],
        remappings=[
            ('image_raw', 'left/image_raw'),
            ('camera_info', 'left/camerainfo'),
            ('image_rect', 'left/image_rect'),
            ('camera_info_rect', 'left/camera_info_rect')
        ]
    )

    right_rectify_node = ComposableNode(
        name='right_rectify_node',
        package='isaac_ros_image_proc',
        plugin='nvidia::isaac_ros::image_proc::RectifyNode',
        parameters=[{
            'output_width': 1920,
            'output_height': 1200,
        }],
        remappings=[
            ('image_raw', 'right/image_raw'),
            ('camera_info', 'right/camerainfo'),
            ('image_rect', 'right/image_rect'),
            ('camera_info_rect', 'right/camera_info_rect')
        ]
    )

    disparity_node = ComposableNode(
        name='disparity',
        package='isaac_ros_ess',
        plugin='nvidia::isaac_ros::dnn_stereo_disparity::ESSDisparityNode',
        parameters=[{'engine_file_path': engine_file_path}],
        remappings=[
            ('left/camera_info', 'left/camera_info_rect'),
            ('right/camera_info', 'right/camera_info_rect')
        ]
    )

    point_cloud_node = ComposableNode(
        name='point_cloud_node',
        package='isaac_ros_stereo_image_proc',
        plugin='nvidia::isaac_ros::stereo_image_proc::PointCloudNode',
        parameters=[{
                'approximate_sync': False,
                'use_color': False,
                'use_system_default_qos': True,
        }],
        remappings=[
            ('left/image_rect_color', 'left/image_rect'),
            ('left/camera_info', 'left/camera_info_rect'),
            ('right/camera_info', 'right/camera_info_rect')
        ]
    )

    container = ComposableNodeContainer(
        name='disparity_container',
        namespace='disparity',
        package='rclcpp_components',
        executable='component_container_mt',
        composable_node_descriptions=[
            left_rectify_node, right_rectify_node,
            disparity_node, point_cloud_node
        ],
        output='screen'
    )

    return (launch.LaunchDescription(launch_args + [container]))

The image_raw and camerainfo topics were published and according to rqt_graph everything seemed to be linked just fine. However, this failed with some gxf related errors, see logs below.

Do I have to update some gxf configuration files when running without ArgusStereoNode?

If so, which files and how?

admin@tegra-ubuntu:/workspaces/isaac_ros-dev$ ros2 launch isaac_ros_ess isaac_ros_rect_ess.launch.py 
[INFO] [launch]: All log files can be found below /home/admin/.ros/log/2023-07-10-11-21-55-437458-tegra-ubuntu-18328
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container_mt-1]: process started with pid [18339]
[component_container_mt-1] [INFO] [1688988115.918243687] [disparity.disparity_container]: Load Library: /workspaces/isaac_ros-dev/install/isaac_ros_image_proc/lib/librectify_node.so
[component_container_mt-1] [INFO] [1688988115.999291280] [disparity.disparity_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::RectifyNode>
[component_container_mt-1] [INFO] [1688988115.999399571] [disparity.disparity_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::RectifyNode>
[component_container_mt-1] [INFO] [1688988116.005331110] [NitrosContext]: [NitrosContext] Creating a new shared context
[component_container_mt-1] [INFO] [1688988116.005582030] [left_rectify_node]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1688988116.006433415] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/std/libgxf_std.so
[component_container_mt-1] [INFO] [1688988116.010960944] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_gxf_helpers.so
[component_container_mt-1] [INFO] [1688988116.013911208] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_sight.so
[component_container_mt-1] [INFO] [1688988116.016902466] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_atlas.so
[component_container_mt-1] [INFO] [1688988116.021309607] [NitrosContext]: [NitrosContext] Loading application: '/workspaces/isaac_ros-dev/install/isaac_ros_nitros/share/isaac_ros_nitros/config/type_adapter_nitros_context_graph.yaml'
[component_container_mt-1] [INFO] [1688988116.022312549] [NitrosContext]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1688988116.024183966] [NitrosContext]: [NitrosContext] Running application...
[component_container_mt-1] 2023-07-10 11:21:56.024 WARN  gxf/std/program.cpp@456: No system specified. Nothing to do
[component_container_mt-1] [INFO] [1688988116.025652650] [left_rectify_node]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1688988116.025710860] [left_rectify_node]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1688988116.028661925] [left_rectify_node]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1688988116.028750055] [left_rectify_node]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1688988116.033223758] [left_rectify_node]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1688988116.033336657] [left_rectify_node]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1688988116.033764990] [left_rectify_node]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1688988116.034229868] [left_rectify_node]: [NitrosContext] Loading extension: gxf/lib/libgxf_message_compositor.so
[component_container_mt-1] [INFO] [1688988116.035954208] [left_rectify_node]: [NitrosContext] Loading extension: gxf/lib/cuda/libgxf_cuda.so
[component_container_mt-1] [INFO] [1688988116.038877336] [left_rectify_node]: [NitrosContext] Loading extension: gxf/lib/image_proc/libgxf_tensorops.so
[component_container_mt-1] [INFO] [1688988116.043264572] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/multimedia/libgxf_multimedia.so
[component_container_mt-1] [INFO] [1688988116.043900495] [left_rectify_node]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1688988116.048163216] [left_rectify_node]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1688988116.098447386] [left_rectify_node]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1688988116.101732893] [left_rectify_node]: [NitrosNode] Creating negotiated publishers/subscribers
[component_container_mt-1] [INFO] [1688988116.114962667] [left_rectify_node]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/left_rectify_node' in container '/disparity/disparity_container'
[component_container_mt-1] [INFO] [1688988116.119253965] [disparity.disparity_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::RectifyNode>
[component_container_mt-1] [INFO] [1688988116.119326991] [disparity.disparity_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::RectifyNode>
[component_container_mt-1] [INFO] [1688988116.124065726] [right_rectify_node]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1688988116.124867766] [right_rectify_node]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1688988116.124898839] [right_rectify_node]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1688988116.126878706] [right_rectify_node]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1688988116.126924820] [right_rectify_node]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1688988116.130402236] [right_rectify_node]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1688988116.130491967] [right_rectify_node]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1688988116.130823497] [right_rectify_node]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1688988116.132624287] [right_rectify_node]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1688988116.136046246] [right_rectify_node]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1688988116.186783134] [right_rectify_node]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1688988116.188554964] [right_rectify_node]: [NitrosNode] Creating negotiated publishers/subscribers
[component_container_mt-1] [INFO] [1688988116.194867666] [right_rectify_node]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/right_rectify_node' in container '/disparity/disparity_container'
[component_container_mt-1] [INFO] [1688988116.201697791] [disparity.disparity_container]: Load Library: /workspaces/isaac_ros-dev/install/isaac_ros_ess/lib/libisaac_ros_ess_node.so
[component_container_mt-1] [INFO] [1688988116.208063519] [disparity.disparity_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_stereo_disparity::ESSDisparityNode>
[component_container_mt-1] [INFO] [1688988116.208166722] [disparity.disparity_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_stereo_disparity::ESSDisparityNode>
[component_container_mt-1] [INFO] [1688988116.214104885] [disparity]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1688988116.215153045] [disparity]: [NitrosNode] Terminating the running application
[component_container_mt-1] [INFO] [1688988116.215190006] [disparity]: [NitrosContext] Interrupting GXF...
[component_container_mt-1] 2023-07-10 11:21:56.215 ERROR gxf/std/program.cpp@475: Attempted interrupting when not running (state=0d).
[component_container_mt-1] 2023-07-10 11:21:56.215 ERROR gxf/core/runtime.cpp@1243: Graph interrupt failed with error: GXF_INVALID_EXECUTION_SEQUENCE
[component_container_mt-1] [ERROR] [1688988116.215244855] [disparity]: [NitrosContext] GxfGraphInterrupt Error: GXF_INVALID_EXECUTION_SEQUENCE
[component_container_mt-1] [INFO] [1688988116.215258552] [disparity]: [NitrosContext] Waiting on GXF...
[component_container_mt-1] [INFO] [1688988116.215277560] [disparity]: [NitrosContext] Deinitializing...
[component_container_mt-1] [INFO] [1688988116.215296985] [disparity]: [NitrosContext] Destroying context
[component_container_mt-1] [INFO] [1688988116.215322202] [disparity]: [NitrosNode] Application termination done
[component_container_mt-1] [ERROR] [1688988116.219797025] [disparity.disparity_container]: Component constructor threw an exception: [ESSDisparityNode] Empty engine_file_path
[ERROR] [launch_ros.actions.load_composable_nodes]: Failed to load node 'disparity' of type 'nvidia::isaac_ros::dnn_stereo_disparity::ESSDisparityNode' in container '/disparity/disparity_container': Component constructor threw an exception: [ESSDisparityNode] Empty engine_file_path
[component_container_mt-1] [INFO] [1688988116.227860627] [disparity.disparity_container]: Load Library: /workspaces/isaac_ros-dev/install/isaac_ros_stereo_image_proc/lib/libpoint_cloud_node.so
[component_container_mt-1] [INFO] [1688988116.244498920] [disparity.disparity_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::stereo_image_proc::PointCloudNode>
[component_container_mt-1] [INFO] [1688988116.244809554] [disparity.disparity_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::stereo_image_proc::PointCloudNode>
[component_container_mt-1] [INFO] [1688988116.251150417] [point_cloud_node]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1688988116.252650206] [point_cloud_node]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1688988116.252698815] [point_cloud_node]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1688988116.255226539] [point_cloud_node]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1688988116.255308078] [point_cloud_node]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1688988116.255987458] [point_cloud_node]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1688988116.256069349] [point_cloud_node]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1688988116.256085509] [point_cloud_node]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1688988116.257486640] [point_cloud_node]: [NitrosContext] Loading extension: gxf/lib/serialization/libgxf_serialization.so
[component_container_mt-1] [INFO] [1688988116.260770066] [point_cloud_node]: [NitrosContext] Loading extension: gxf/lib/libgxf_synchronization.so
[component_container_mt-1] [INFO] [1688988116.261721359] [point_cloud_node]: [NitrosContext] Loading extension: gxf/lib/libgxf_point_cloud.so
[component_container_mt-1] [INFO] [1688988116.329190783] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_messages.so
[component_container_mt-1] [INFO] [1688988116.330528167] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_ros_messages.so
[component_container_mt-1] [INFO] [1688988116.331896528] [point_cloud_node]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1688988116.335502333] [point_cloud_node]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1688988116.376148581] [point_cloud_node]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1688988116.377114114] [point_cloud_node]: [NitrosNode] Creating negotiated publishers/subscribers
[component_container_mt-1] [INFO] [1688988116.386802374] [point_cloud_node]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/point_cloud_node' in container '/disparity/disparity_container'
[component_container_mt-1] [INFO] [1688988116.387592222] [right_rectify_node]: Negotiating
[component_container_mt-1] [INFO] [1688988116.387861926] [left_rectify_node]: Negotiating
[component_container_mt-1] [INFO] [1688988116.388287155] [left_rectify_node]: Negotiating
[component_container_mt-1] [INFO] [1688988116.800966174] [right_rectify_node]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1688988116.801951068] [right_rectify_node]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1688988116.802877688] [right_rectify_node]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1688988116.803583117] [right_rectify_node]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1688988116.804250721] [right_rectify_node]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1688988116.818174404] [right_rectify_node]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1688988116.851244104] [right_rectify_node]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1688988116.884603637] [right_rectify_node]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1688988116.917273549] [right_rectify_node]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1688988116.950433363] [right_rectify_node]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1688988116.983485943] [right_rectify_node]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1688988117.016360917] [right_rectify_node]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1688988117.049544252] [right_rectify_node]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1688988117.082517853] [right_rectify_node]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1688988117.091714674] [left_rectify_node]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1688988117.092521290] [left_rectify_node]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1688988117.093365348] [left_rectify_node]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1688988117.094077337] [left_rectify_node]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1688988117.100901287] [left_rectify_node]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1688988117.115269207] [right_rectify_node]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1688988117.116128017] [left_rectify_node]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1688988117.116222612] [left_rectify_node]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1688988117.116254933] [left_rectify_node]: [NitrosPublisher] Use the negotiated data format: "nitros_image_bgr8"
[component_container_mt-1] [INFO] [1688988117.116275030] [left_rectify_node]: [NitrosPublisher] Use the negotiated data format: "nitros_camera_info"
[component_container_mt-1] [INFO] [1688988117.116287158] [left_rectify_node]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1688988117.116298422] [left_rectify_node]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/left/image_raw", data_format="nitros_image_bgr8"
[component_container_mt-1] [INFO] [1688988117.116311831] [left_rectify_node]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1688988117.116320471] [left_rectify_node]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/left/camerainfo", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1688988117.116488028] [left_rectify_node]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1688988117.127185598] [left_rectify_node]: [NitrosNode] Wrote the final top level YAML graph to "/workspaces/isaac_ros-dev/install/isaac_ros_image_proc/share/isaac_ros_image_proc/IQGDEKPGYD.yaml"
[component_container_mt-1] [INFO] [1688988117.127326275] [left_rectify_node]: [NitrosNode] Calling user's pre-load-graph callback
[component_container_mt-1] [INFO] [1688988117.127364900] [left_rectify_node]: [RectifyNode] preLoadGraphCallback().
[component_container_mt-1] [INFO] [1688988117.127424614] [left_rectify_node]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1688988117.127456230] [left_rectify_node]: [NitrosContext] Loading application: '/workspaces/isaac_ros-dev/install/isaac_ros_image_proc/share/isaac_ros_image_proc/IQGDEKPGYD.yaml'
[component_container_mt-1] [INFO] [1688988117.133928841] [left_rectify_node]: [NitrosNode] Linking Nitros pub/sub to the loaded application
[component_container_mt-1] [INFO] [1688988117.134265748] [left_rectify_node]: [NitrosNode] Calling user's post-load-graph callback
[component_container_mt-1] [INFO] [1688988117.134319541] [left_rectify_node]: [RectifyNode] postLoadGraphCallback().
[component_container_mt-1] [INFO] [1688988117.134436729] [left_rectify_node]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1688988117.148577955] [right_rectify_node]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1688988117.181640358] [right_rectify_node]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set.
[component_container_mt-1] [INFO] [1688988117.195909076] [right_rectify_node]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1688988117.196038872] [right_rectify_node]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1688988117.196068761] [right_rectify_node]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1688988117.196090073] [right_rectify_node]: [NitrosPublisher] Use only the compatible publisher: topic_name="/right/image_rect", data_format="nitros_image_bgr8"
[component_container_mt-1] [INFO] [1688988117.196116474] [right_rectify_node]: [NitrosPublisher] Use the negotiated data format: "nitros_camera_info"
[component_container_mt-1] [INFO] [1688988117.196137211] [right_rectify_node]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1688988117.196153051] [right_rectify_node]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/right/image_raw", data_format="nitros_image_bgr8"
[component_container_mt-1] [INFO] [1688988117.196169500] [right_rectify_node]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1688988117.196182940] [right_rectify_node]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/right/camerainfo", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1688988117.196251774] [right_rectify_node]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1688988117.210673041] [right_rectify_node]: [NitrosNode] Wrote the final top level YAML graph to "/workspaces/isaac_ros-dev/install/isaac_ros_image_proc/share/isaac_ros_image_proc/EIRSUMOTYU.yaml"
[component_container_mt-1] [INFO] [1688988117.210828757] [right_rectify_node]: [NitrosNode] Calling user's pre-load-graph callback
[component_container_mt-1] [INFO] [1688988117.210865590] [right_rectify_node]: [RectifyNode] preLoadGraphCallback().
[component_container_mt-1] [INFO] [1688988117.210906776] [right_rectify_node]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1688988117.387454172] [point_cloud_node]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1688988117.387567200] [point_cloud_node]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1688988117.387588192] [point_cloud_node]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1688988117.387603265] [point_cloud_node]: [NitrosPublisher] Use only the compatible publisher: topic_name="/points2", data_format="nitros_point_cloud"
[component_container_mt-1] [INFO] [1688988117.387615969] [point_cloud_node]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1688988117.387625474] [point_cloud_node]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/disparity", data_format="nitros_disparity_image_32FC1"
[component_container_mt-1] [INFO] [1688988117.387638786] [point_cloud_node]: [NitrosSubscriber] Use the negotiated data format: "nitros_image_bgr8"
[component_container_mt-1] [INFO] [1688988117.387735525] [point_cloud_node]: [NitrosSubscriber] Use the negotiated data format: "nitros_camera_info"
[component_container_mt-1] [INFO] [1688988117.387750533] [point_cloud_node]: [NitrosSubscriber] Use the negotiated data format: "nitros_camera_info"
[component_container_mt-1] [INFO] [1688988117.387802343] [point_cloud_node]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1688988117.439652512] [left_rectify_node]: [NitrosContext] Running application...
[component_container_mt-1] [INFO] [1688988117.440344149] [left_rectify_node]: [NitrosNode] Starting a heartbeat timer (eid=134)
[component_container_mt-1] [INFO] [1688988117.440354326] [right_rectify_node]: [NitrosContext] Loading application: '/workspaces/isaac_ros-dev/install/isaac_ros_image_proc/share/isaac_ros_image_proc/EIRSUMOTYU.yaml'
[component_container_mt-1] [INFO] [1688988117.443484212] [right_rectify_node]: [NitrosNode] Linking Nitros pub/sub to the loaded application
[component_container_mt-1] [INFO] [1688988117.443829662] [right_rectify_node]: [NitrosNode] Calling user's post-load-graph callback
[component_container_mt-1] [INFO] [1688988117.443895968] [right_rectify_node]: [RectifyNode] postLoadGraphCallback().
[component_container_mt-1] [INFO] [1688988117.443984291] [right_rectify_node]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1688988117.458769696] [point_cloud_node]: [NitrosNode] Wrote the final top level YAML graph to "/workspaces/isaac_ros-dev/install/isaac_ros_stereo_image_proc/share/isaac_ros_stereo_image_proc/OWJUSLZKTI.yaml"
[component_container_mt-1] [INFO] [1688988117.458963878] [point_cloud_node]: [NitrosNode] Calling user's pre-load-graph callback
[component_container_mt-1] [INFO] [1688988117.459076745] [point_cloud_node]: [PointCloudNode] preLoadGraphCallback().
[component_container_mt-1] [INFO] [1688988117.459131115] [point_cloud_node]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1688988117.518922100] [right_rectify_node]: [NitrosContext] Running application...
[component_container_mt-1] [INFO] [1688988117.519369729] [right_rectify_node]: [NitrosNode] Starting a heartbeat timer (eid=203)
[component_container_mt-1] [INFO] [1688988117.519381441] [point_cloud_node]: [NitrosContext] Loading application: '/workspaces/isaac_ros-dev/install/isaac_ros_stereo_image_proc/share/isaac_ros_stereo_image_proc/OWJUSLZKTI.yaml'
[component_container_mt-1] 2023-07-10 11:21:57.522 WARN  gxf/std/yaml_file_loader.cpp@952: Using unregistered parameter 'disparity_image_input' in component ''.
[component_container_mt-1] 2023-07-10 11:21:57.522 WARN  gxf/std/yaml_file_loader.cpp@952: Using unregistered parameter 'rgb_image_input' in component ''.
[component_container_mt-1] 2023-07-10 11:21:57.522 WARN  gxf/std/yaml_file_loader.cpp@952: Using unregistered parameter 'left_cam_input' in component ''.
[component_container_mt-1] 2023-07-10 11:21:57.522 WARN  gxf/std/yaml_file_loader.cpp@952: Using unregistered parameter 'right_cam_input' in component ''.
[component_container_mt-1] 2023-07-10 11:21:57.522 WARN  gxf/std/yaml_file_loader.cpp@952: Using unregistered parameter 'disparity_image_output' in component ''.
[component_container_mt-1] 2023-07-10 11:21:57.522 WARN  gxf/std/yaml_file_loader.cpp@952: Using unregistered parameter 'rgb_image_output' in component ''.
[component_container_mt-1] 2023-07-10 11:21:57.522 WARN  gxf/std/yaml_file_loader.cpp@952: Using unregistered parameter 'left_cam_output' in component ''.
[component_container_mt-1] 2023-07-10 11:21:57.522 WARN  gxf/std/yaml_file_loader.cpp@952: Using unregistered parameter 'right_cam_output' in component ''.
[component_container_mt-1] [INFO] [1688988117.523651842] [point_cloud_node]: [NitrosNode] Linking Nitros pub/sub to the loaded application
[component_container_mt-1] [INFO] [1688988117.523889545] [point_cloud_node]: [NitrosNode] Calling user's post-load-graph callback
[component_container_mt-1] [INFO] [1688988117.523963371] [point_cloud_node]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1688988117.836585426] [point_cloud_node]: [NitrosContext] Running application...
[component_container_mt-1] [INFO] [1688988117.837012799] [point_cloud_node]: [NitrosNode] Starting a heartbeat timer (eid=320)
[component_container_mt-1] 2023-07-10 11:21:57.923 ERROR /workspaces/isaac_ros-dev/src/isaac_ros_image_pipeline/isaac_ros_image_proc/gxf/tensorops/extensions/tensorops/components/Undistort.cpp@330: invalid input/output type for image undistort.
[component_container_mt-1] 2023-07-10 11:21:57.923 ERROR /workspaces/isaac_ros-dev/src/isaac_ros_image_pipeline/isaac_ros_image_proc/gxf/tensorops/extensions/tensorops/components/TensorOperator.cpp@233: operation failed.
[component_container_mt-1] 2023-07-10 11:21:57.924 ERROR /workspaces/isaac_ros-dev/src/isaac_ros_image_pipeline/isaac_ros_image_proc/gxf/tensorops/extensions/tensorops/components/Undistort.cpp@330: invalid input/output type for image undistort.
[component_container_mt-1] 2023-07-10 11:21:57.924 ERROR /workspaces/isaac_ros-dev/src/isaac_ros_image_pipeline/isaac_ros_image_proc/gxf/tensorops/extensions/tensorops/components/TensorOperator.cpp@233: operation failed.
[component_container_mt-1] 2023-07-10 11:21:57.926 ERROR gxf/std/entity_executor.cpp@509: Failed to tick codelet undistort_algo in entity: IQGDEKPGYD_rectifier code: GXF_FAILURE
[component_container_mt-1] 2023-07-10 11:21:57.926 ERROR gxf/std/entity_executor.cpp@540: Entity [IQGDEKPGYD_rectifier] must be in Lifecycle::kStarted or Lifecycle::kIdle stage before stopping. Current state is Ticking
[component_container_mt-1] 2023-07-10 11:21:57.926 WARN  gxf/std/multi_thread_scheduler.cpp@235: Error while executing entity E142 named 'IQGDEKPGYD_rectifier': GXF_FAILURE
[component_container_mt-1] 2023-07-10 11:21:57.928 ERROR gxf/std/entity_executor.cpp@509: Failed to tick codelet undistort_algo in entity: EIRSUMOTYU_rectifier code: GXF_FAILURE
[component_container_mt-1] 2023-07-10 11:21:57.928 ERROR gxf/std/entity_executor.cpp@540: Entity [EIRSUMOTYU_rectifier] must be in Lifecycle::kStarted or Lifecycle::kIdle stage before stopping. Current state is Ticking
[component_container_mt-1] 2023-07-10 11:21:57.928 WARN  gxf/std/multi_thread_scheduler.cpp@235: Error while executing entity E211 named 'EIRSUMOTYU_rectifier': GXF_FAILURE

Thank you in advance!

4

@hemals

Any update on this, please?

Thank you!

5

You may be missing the following property in your camera device tree:
sync_sensor_index = <1>;

Try adding this and verify with argus_syncstereo directly.
You can refer to the device tree for HAWK (AR0234) to see how we did this here:
t234-common-modules/tegra234-camera-p3762-a00.dtsi

6

@hoermandinger did you ever figure this out? I would like to do the same.

7

Hi @dakejahl

For now I ended up creating a custom ROS node. The Node internally creates a Gstreamer Pipeline with two muxed nvarguscamerasrc elements and two instances of a custom Gstreamer plugin for rectification (based on OpenCV and VPI). The Rectification plugins read some calibration files which are created using stereoRectify from openCV. The ROS Node receives the rectified images via Gstreamer appsink callback and then publishes both rectified images and both calibration messages at once. Subsequently, this data is directly fed into the isaac_ros_ess node etc.

Obviously, the aforementioned method introduces a performance bottleneck since the images are brought to CPU before going to isaac_ros_ess.

In the future I still want to try out the following 2 approaches to optimize:

a) Use the stereo_node from isaac_ros_argus_camera package but replace the ArgusCamera inside the gxf application (argus_camera_stereo_node.yaml file) with something that can obtain the images from two different sensor IDs 0 and 1. Maybe nvarguscamerasrc elements but then I don’t know yet how I could link this to the rest of the gxf application.

b) Try to mux both of the 4-lane CSI ports (A+B and C+D) with virtual channels inside Jetson via the device tree. We run two IMX264 sensors on our stereocamera, where we convert the Sub-LVDS streams to MIPI via FPGAs. Therefore, we have enough control to write virtual channels into the MIPI stream. However, our hardware setup differs from the classical SerDes GMSL setup, where two GMSL Streams are already muxed into a single MIPI stream via the Deserializer prior to entering only one physical CSI port. I’m unsure if 2 physical CSI ports can be muxed in Jetson internally with virtual-channels.

1 Like
8

Thanks for the quick reply! It’s unfortunate that this isn’t supported, this seems like it would be a really common use case.

Use the stereo_node from isaac_ros_argus_camera package but replace the ArgusCamera inside the gxf application (argus_camera_stereo_node.yaml file) with something that can obtain the images from two different sensor IDs 0 and 1. Maybe nvarguscamerasrc elements but then I don’t know yet how I could link this to the rest of the gxf application.

I arrived at this conclusion too, but I don’t know how you would go about modifying the gfx applicaton since it’s just shared object files. The syncSensor and syncStereo examples make sense but it’s unclear how those could be brought into isaac_ros and leverage the optimized image processing.

@hemals Are there any plans to support this configuration in isaac_ros_argus_camera? Namely hardware synchronized mipi csi cameras? It would be awesome if it could also support software synchronized cameras too so very low cost sensors like the imx219 could be used.

9

Does running two instances of ArgusMonoCamera and then managing time synchronization not work well enough here? The changes to the device tree above should allow libArgus to recognize your setup as a stereo camera, though you’ll need to provide stereo rectification parameters as well.

There are no plans to support arbitrary synchronized imagers as stereo cameras yet. Only stereo cameras qualified for libArgus should work well with Isaac ROS Argus Camera.