Problem using isaac camera with isaac rectification node

Autonomous Machines Robotics - Isaac Isaac ROS
,
1

Hi all,

I am currently trying to move our ROS2 image processing pipeline to ISAAC ROS nodes. However I get an error when using the ArgusMonoNode from isaac_ros_argus_camera together with the RectifyNode from isaac_ros_image_pipeline.

When running the following minimal example:

from launch import LaunchDescription
from launch_ros.substitutions import FindPackageShare
from launch.substitutions import PathJoinSubstitution
from launch_ros.actions import ComposableNodeContainer
from launch_ros.descriptions import ComposableNode
from launch.actions import OpaqueFunction

def launch_setup(context, *args, **kwargs):
    camera_ini = PathJoinSubstitution([FindPackageShare('common'), 'resource', 'camera', 'calibration_isaac.ini'])

    ########## CAMERA ##############
    isaac_camera = ComposableNode(
        name=f"camera_left",
        package='isaac_ros_argus_camera',
        plugin='nvidia::isaac_ros::argus::ArgusMonoNode',
        parameters=[{'camera_id': 2},
                    {'camera_info_url': f"file://{camera_ini.perform(context)}"}
                    ],
        remappings=[
            ('left/image_raw', f'/camera/roofleft/image_color'),
            ('left/camerainfo', f'/camera/roofleft/camera_info')
        ],
    )

    ########## RECTIFY ##############
    isaac_rectify = ComposableNode(
        name=f'rectify_left',
        package='isaac_ros_image_proc',
        plugin='nvidia::isaac_ros::image_proc::RectifyNode',
        parameters=[{
            'output_width': 2048,
            'output_height': 1280,
        }],
        remappings=[
            ('/image_raw', f'/camera/roofleft/image_color'),
            ('/camera_info', f'/camera/roofleft/camera_info'),
            ('/image_rect', f'/camera/roofleft/image_color_rect'),
            ('/camera_info_rect', f'/camera/roofleft/camera_info_rect')
        ],
    )    

    nitros_container = ComposableNodeContainer(
        name=f'nitros_container_left',
        package='rclcpp_components',
        executable='component_container_mt',
        composable_node_descriptions=[isaac_camera, isaac_rectify],
        namespace='',
        output='screen',
    )

    return [
        nitros_container,
    ]

def generate_launch_description():
    launch_description = LaunchDescription()
    launch_description.add_action(OpaqueFunction(function=launch_setup))
    return launch_description

I get the following error:

[INFO] [launch]: All log files can be found below /home/root/.ros/log/2022-05-05-15-37-30-721133-jetson-xavier-kraken-emmc-3755
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container_mt-1]: process started with pid [3768]
[component_container_mt-1] [INFO] [1651765051.410829568] [nitros_container_left]: Load Library: /workspaces/isaac_ros-dev/install/isaac_ros_argus_camera/lib/libmono_node.so
[component_container_mt-1] [INFO] [1651765051.584342176] [nitros_container_left]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::argus::ArgusMonoNode>
[component_container_mt-1] [INFO] [1651765051.584518016] [nitros_container_left]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::argus::ArgusMonoNode>
[component_container_mt-1] [INFO] [1651765051.596898560] [NitrosContext]: [NitrosContext] Creating a new shared context
[component_container_mt-1] [INFO] [1651765051.597423296] [camera_left]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1651765051.602225952] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/std/libgxf_std.so
[component_container_mt-1] [INFO] [1651765051.611003424] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_gxf_helpers.so
[component_container_mt-1] [INFO] [1651765051.617513312] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_sight.so
[component_container_mt-1] [INFO] [1651765051.624306432] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_atlas.so
[component_container_mt-1] [INFO] [1651765051.631286208] [NitrosContext]: [NitrosContext] Loading application: '/workspaces/isaac_ros-dev/install/isaac_ros_nitros/share/isaac_ros_nitros/config/type_adapter_nitros_context_graph.yaml'
[component_container_mt-1] [INFO] [1651765051.634018144] [NitrosContext]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1651765051.641429024] [NitrosContext]: [NitrosContext] Running application...
[component_container_mt-1] 2022-05-05 15:37:31.641 WARN  gxf/std/program.cpp@456: No system specified. Nothing to do
[component_container_mt-1] [INFO] [1651765051.660817312] [camera_left]: camera calibration URL: file:///home/root/workspace/install_aarch64/common/share/common/resource/camera/calibration_isaac.ini
[component_container_mt-1] [WARN] [1651765051.661306624] [camera_left]: [calibration_isaac] does not match camera in file /home/root/workspace/install_aarch64/common/share/common/resource/camera/calibration_isaac.ini
[component_container_mt-1] [INFO] [1651765051.662320000] [camera_left]: [ArgusMonoNode] Loaded camera info from "file:///home/root/workspace/install_aarch64/common/share/common/resource/camera/calibration_isaac.ini"
[component_container_mt-1] [INFO] [1651765051.662445856] [camera_left]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1651765051.662501088] [camera_left]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1651765051.672118880] [camera_left]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1651765051.672259456] [camera_left]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1651765051.674536032] [camera_left]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1651765051.674654912] [camera_left]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1651765051.675460224] [camera_left]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1651765051.677864384] [camera_left]: [NitrosContext] Loading extension: gxf/lib/libgxf_message_compositor.so
[component_container_mt-1] [INFO] [1651765051.683911552] [camera_left]: [NitrosContext] Loading extension: gxf/lib/cuda/libgxf_cuda.so
[component_container_mt-1] [INFO] [1651765051.689537408] [camera_left]: [NitrosContext] Loading extension: gxf/lib/serialization/libgxf_serialization.so
[component_container_mt-1] [INFO] [1651765051.702644416] [camera_left]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_messages.so
[component_container_mt-1] [INFO] [1651765051.710519456] [camera_left]: [NitrosContext] Loading extension: gxf/lib/multimedia/libgxf_multimedia.so
[component_container_mt-1] [INFO] [1651765051.713370720] [camera_left]: [NitrosContext] Loading extension: gxf/lib/image_proc/libgxf_tensorops.so
[component_container_mt-1] [INFO] [1651765051.725667136] [camera_left]: [NitrosContext] Loading extension: gxf/lib/image_proc/libgxf_rectify_params_generator.so
[component_container_mt-1] [INFO] [1651765051.753444160] [camera_left]: [NitrosContext] Loading extension: gxf/lib/libgxf_argus.so
[component_container_mt-1] [INFO] [1651765052.253110208] [camera_left]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1651765052.265980960] [camera_left]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1651765052.309232736] [camera_left]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1651765052.312026976] [camera_left]: [NitrosNode] Creating negotiated publishers/subscribers
[component_container_mt-1] [INFO] [1651765052.327310592] [camera_left]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/camera_left' in container '/nitros_container_left'
[component_container_mt-1] [INFO] [1651765052.342128352] [nitros_container_left]: Load Library: /workspaces/isaac_ros-dev/install/isaac_ros_image_proc/lib/librectify_node.so
[component_container_mt-1] [INFO] [1651765052.349532960] [nitros_container_left]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::RectifyNode>
[component_container_mt-1] [INFO] [1651765052.349684768] [nitros_container_left]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::RectifyNode>
[component_container_mt-1] [INFO] [1651765052.366164704] [rectify_left]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1651765052.371687232] [rectify_left]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1651765052.371807392] [rectify_left]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1651765052.381277280] [rectify_left]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1651765052.381411008] [rectify_left]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1651765052.393890464] [rectify_left]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1651765052.394023744] [rectify_left]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1651765052.394697312] [rectify_left]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1651765052.410246016] [rectify_left]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1651765052.423553760] [rectify_left]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1651765052.530288768] [rectify_left]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1651765052.535054464] [rectify_left]: [NitrosNode] Creating negotiated publishers/subscribers
[component_container_mt-1] [INFO] [1651765052.553246880] [rectify_left]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/rectify_left' in container '/nitros_container_left'
[component_container_mt-1] [INFO] [1651765052.556815616] [camera_left]: Negotiating
[component_container_mt-1] [INFO] [1651765052.558309728] [camera_left]: Negotiating
[component_container_mt-1] [INFO] [1651765052.562331328] [rectify_left]: Negotiating
[component_container_mt-1] [INFO] [1651765052.562470720] [rectify_left]: Could not negotiate
[component_container_mt-1] [INFO] [1651765052.562567904] [rectify_left]: Negotiating
[component_container_mt-1] [INFO] [1651765052.562627360] [rectify_left]: Could not negotiate
[component_container_mt-1] [INFO] [1651765053.332325344] [camera_left]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1651765053.332592640] [camera_left]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1651765053.332738304] [camera_left]: [NitrosPublisher] Use the negotiated data format: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1651765053.332862464] [camera_left]: [NitrosPublisher] Use the negotiated data format: "nitros_camera_info"
[component_container_mt-1] [INFO] [1651765053.333094080] [camera_left]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1651765053.363389568] [camera_left]: [NitrosNode] Wrote the final top level YAML graph to "/workspaces/isaac_ros-dev/install/isaac_ros_argus_camera/share/isaac_ros_argus_camera/CFQNXEFOTG.yaml"
[component_container_mt-1] [INFO] [1651765053.363712512] [camera_left]: [NitrosNode] Calling user's pre-load-graph callback
[component_container_mt-1] [INFO] [1651765053.363798848] [camera_left]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1651765053.363879360] [camera_left]: [NitrosContext] Loading application: '/workspaces/isaac_ros-dev/install/isaac_ros_argus_camera/share/isaac_ros_argus_camera/CFQNXEFOTG.yaml'
[component_container_mt-1] 2022-05-05 15:37:33.374 WARN  gxf/std/yaml_file_loader.cpp@952: Using unregistered parameter 'left_camera_frame' in component ''.
[component_container_mt-1] 2022-05-05 15:37:33.375 WARN  gxf/std/yaml_file_loader.cpp@952: Using unregistered parameter 'right_camera_frame' in component ''.
[component_container_mt-1] 2022-05-05 15:37:33.375 WARN  gxf/std/yaml_file_loader.cpp@952: Using unregistered parameter 'transmitter' in component ''.
[component_container_mt-1] [INFO] [1651765053.377413248] [camera_left]: [NitrosNode] Linking Nitros pub/sub to the loaded application
[component_container_mt-1] [INFO] [1651765053.377794784] [camera_left]: [NitrosNode] Calling user's post-load-graph callback
[component_container_mt-1] [INFO] [1651765053.377983168] [camera_left]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1651765053.556031808] [rectify_left]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1651765053.556250080] [rectify_left]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1651765053.556315904] [rectify_left]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1651765053.556390176] [rectify_left]: [NitrosPublisher] Use only the compatible publisher: topic_name="/camera/roofleft/image_color_rect", data_format="nitros_image_bgr8"
[component_container_mt-1] [INFO] [1651765053.556447360] [rectify_left]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1651765053.556524416] [rectify_left]: [NitrosPublisher] Use only the compatible publisher: topic_name="/camera/roofleft/camera_info_rect", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1651765053.557755264] [rectify_left]: [NitrosSubscriber] Use the negotiated data format: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1651765053.558006944] [rectify_left]: [NitrosSubscriber] Use the negotiated data format: "nitros_camera_info"
[component_container_mt-1] [INFO] [1651765053.558124864] [rectify_left]: [NitrosPublisherSubscriberGroup] Adjusted the compatible format of the component "image_sink/sink" (type="nvidia::isaac_ros::MessageRelay") from "nitros_image_bgr8" to "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1651765053.558234976] [rectify_left]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1651765053.580899328] [rectify_left]: [NitrosNode] Wrote the final top level YAML graph to "/workspaces/isaac_ros-dev/install/isaac_ros_image_proc/share/isaac_ros_image_proc/GPEGAECXLO.yaml"
[component_container_mt-1] [INFO] [1651765053.581124800] [rectify_left]: [NitrosNode] Calling user's pre-load-graph callback
[component_container_mt-1] [INFO] [1651765053.581205536] [rectify_left]: [RectifyNode] preLoadGraphCallback().
[component_container_mt-1] [INFO] [1651765053.581307392] [rectify_left]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1651765054.110590112] [camera_left]: [NitrosContext] Running application...
[component_container_mt-1] [INFO] [1651765054.111031680] [camera_left]: [NitrosNode] Starting a heartbeat timer (eid=44)
[component_container_mt-1] [INFO] [1651765054.111078144] [rectify_left]: [NitrosContext] Loading application: '/workspaces/isaac_ros-dev/install/isaac_ros_image_proc/share/isaac_ros_image_proc/GPEGAECXLO.yaml'
[component_container_mt-1] [INFO] [1651765054.123016352] [rectify_left]: [NitrosNode] Linking Nitros pub/sub to the loaded application
[component_container_mt-1] [INFO] [1651765054.123437952] [rectify_left]: [NitrosNode] Calling user's post-load-graph callback
[component_container_mt-1] [INFO] [1651765054.123520896] [rectify_left]: [RectifyNode] postLoadGraphCallback().
[component_container_mt-1] [INFO] [1651765054.123666240] [rectify_left]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1651765054.146405536] [rectify_left]: [NitrosContext] Running application...
[component_container_mt-1] [INFO] [1651765054.147701664] [rectify_left]: [NitrosNode] Starting a heartbeat timer (eid=66)
[component_container_mt-1] 2022-05-05 15:37:34.490 WARN  extensions/hawk/argus_camera.cpp@580: Failed to get calibration data from Argus!
[component_container_mt-1] 2022-05-05 15:37:36.853 ERROR /workspaces/isaac_ros-dev/src/isaac_ros_image_pipeline/isaac_ros_image_proc/gxf/tensorops/extensions/tensorops/components/ImageUtils.cpp@117: invalid distortion type.
[component_container_mt-1] 2022-05-05 15:37:36.853 ERROR /workspaces/isaac_ros-dev/src/isaac_ros_image_pipeline/isaac_ros_image_proc/gxf/tensorops/extensions/tensorops/components/TensorOperator.cpp@233: operation failed.
[component_container_mt-1] 2022-05-05 15:37:36.854 ERROR gxf/std/entity_executor.cpp@509: Failed to tick codelet undistort_algo in entity: GPEGAECXLO_rectifier code: GXF_FAILURE
[component_container_mt-1] 2022-05-05 15:37:36.854 ERROR gxf/std/entity_executor.cpp@540: Entity [GPEGAECXLO_rectifier] must be in Lifecycle::kStarted or Lifecycle::kIdle stage before stopping. Current state is Ticking
[component_container_mt-1] 2022-05-05 15:37:36.854 WARN  gxf/std/multi_thread_scheduler.cpp@235: Error while executing entity E74 named 'GPEGAECXLO_rectifier': GXF_FAILURE

What I have observed:

  • The error occurs only sometimes (50 %'ish), and occurs immidiately after startup. If it does not occur, everything runs fine.
  • I can run both nodes seperately without any issues.

Initially I suspected, that something was wrong with the camera_info/distortion_type from the ArgusMonoNode. However, I’m pretty sure, that it is not the issue since:

  • I have tried having a seperate node to publish the my own camera_info. Gets exact same error.
  • If I record data from the camera node and plays it with the RectifyNode i don’t get any errors.

Additional Information
I am running it on a Jetson Xavier NX, that has a custom yocto build which is based on JetPack 5.1.1. Due to company stuff, I have to make it work in this environment. I am aware that this is not the recommended way to run it, however I still find the behavior a bit odd.

Any help, suggestions or information about stability when using JetPack 5.1.1 is appreciated.

2

Thanks for the note on the custom Yocto build and yes, it is quite odd. What camera exactly are you using, by the way? Are there any other Argus cameras attached to the same Xavier NX?

The line “Failed to get calibration data from Argus!” is when libArgus is unable to get an ISyncSensorCalibrationData interface on the monocam. This could happen because of a timing issue I suppose when reading from the EEPROM on the camera. Likely, it is trying to read the data from the camera and then ignores it by overriding it with the information at the URL, and the error should not interrupt this flow. Could it be that the ini file parameters are not correct? Could you post the contents of this file perhaps?

3

Thank you for the reply!

I am using the LI-AR0233-GMSL2-065H. It is the only camera connected to the Xavier. I am not an expert about cameras, but as far as I am aware, we cannot get the ISyncSensorCalibrationData directly for this camera, thus I am using the following .ini file:

# oST version 5.0 parameters

[image]

width
2048

height
1280

[camera]

camera matrix
815.173806 0.000000 1023.957456
0.000000 815.939003 634.330074
0.000000 0.000000 1.000000

distortion
7.554281 2.749951 0.000454 0.000133 0.075021 7.893061 5.153860 0.472870

rectification
1.000000 0.000000 0.000000
0.000000 1.000000 0.000000
0.000000 0.000000 1.000000

projection
380.983887 0.000000 1027.890058 0.000000
0.000000 639.504639 632.306959 0.000000
0.000000 0.000000 1.000000 0.000000

As you suggest, I was also suspecting that something was wrong with it. Therefore, I have tried the two following things:

  • Using 0 for all distortion coefficients, to attempt just passing an image through the node, with no correction.
  • Additionally, I have tried having a seperate camera_info_publisher to publisher the camera_info message instead of having the camera node do it.

However, I get the same behavior: Sometimes it works and the output looks fine, sometimes it crashes.