Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8)

mona.jalal · February 10, 2025, 6:53pm

I ran same command on 2.7.25 and was able to see the live visualization for shoes using Intel RealSense D435 camera. Now, when I run it, I get this error. Could you please help how to fix it? When I visualize it with rviz, the visualized item is grey. I am able to see the feed inside the docker using realsense-viewer.


admin@DOS:/workspaces/isaac_ros-dev/isaac_ros_assets/models$  ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=realsense_mono_rect,centerpose,centerpose_visualizer model_name:=centerpose_shoe model_repository_paths:=[${ISAAC_ROS_WS}/isaac_ros_assets/models]
[INFO] [launch]: All log files can be found below /home/admin/.ros/log/2025-02-10-13-38-13-419625-DOS-29388
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container_mt-1]: process started with pid [29401]
[component_container_mt-1] [INFO] [1739212693.735613082] [isaac_ros_examples.container]: Load Library: /opt/ros/humble/lib/libresize_node.so
[component_container_mt-1] [INFO] [1739212693.749205254] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::ResizeNode>
[component_container_mt-1] [INFO] [1739212693.749237422] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::ResizeNode>
[component_container_mt-1] [INFO] [1739212695.917159649] [centerpose_encoder.resize_node]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1739212695.917324174] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/std/libgxf_std.so
[component_container_mt-1] [INFO] [1739212695.918121220] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_gxf_helpers.so
[component_container_mt-1] [INFO] [1739212695.920184320] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_sight.so
[component_container_mt-1] [INFO] [1739212695.922202224] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_atlas.so
[component_container_mt-1] 2025-02-10 13:38:15.927 WARN  gxf/std/program.cpp@538: No GXF scheduler specified.
[component_container_mt-1] [INFO] [1739212695.927456997] [centerpose_encoder.resize_node]: [ResizeNode] Set output data format to: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1739212695.927500623] [centerpose_encoder.resize_node]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1739212695.933532206] [centerpose_encoder.resize_node]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1739212695.933589496] [centerpose_encoder.resize_node]: [NitrosContext] Loading extension: gxf/lib/multimedia/libgxf_multimedia.so
[component_container_mt-1] [INFO] [1739212695.935319318] [centerpose_encoder.resize_node]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_message_compositor.so
[component_container_mt-1] [INFO] [1739212695.935634669] [centerpose_encoder.resize_node]: [NitrosContext] Loading extension: gxf/lib/cuda/libgxf_cuda.so
[component_container_mt-1] [INFO] [1739212695.937118719] [centerpose_encoder.resize_node]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_tensorops.so
[component_container_mt-1] [INFO] [1739212695.939234647] [centerpose_encoder.resize_node]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1739212695.941406218] [centerpose_encoder.resize_node]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1739212696.044679950] [centerpose_encoder.resize_node]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1739212696.046598301] [centerpose_encoder.resize_node]: [NitrosPublisherSubscriberGroup] Pinning the component "image_sink/sink" (type="nvidia::isaac_ros::MessageRelay") to use its compatible format only: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1739212696.051143353] [centerpose_encoder.resize_node]: [NitrosNode] Starting negotiation...
[component_container_mt-1] [INFO] [1739212696.051607809] [isaac_ros_examples.container]: Load Library: /opt/ros/humble/lib/librealsense2_camera.so
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/centerpose_encoder/resize_node' in container '/isaac_ros_examples/container'
[component_container_mt-1] [INFO] [1739212696.149185045] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<realsense2_camera::RealSenseNodeFactory>
[component_container_mt-1] [INFO] [1739212696.149244401] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<realsense2_camera::RealSenseNodeFactory>
[component_container_mt-1] [INFO] [1739212696.162936478] [realsense2_camera]: RealSense ROS v4.51.1
[component_container_mt-1] [INFO] [1739212696.162969420] [realsense2_camera]: Built with LibRealSense v2.55.1
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/realsense2_camera' in container '/isaac_ros_examples/container'
[component_container_mt-1] [INFO] [1739212696.162978549] [realsense2_camera]: Running with LibRealSense v2.55.1
[component_container_mt-1] [INFO] [1739212696.163761316] [isaac_ros_examples.container]: Load Library: /opt/ros/humble/lib/libimage_format_converter_node.so
[component_container_mt-1] [INFO] [1739212696.165803961] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::ImageFormatConverterNode>
[component_container_mt-1] [INFO] [1739212696.165827863] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::ImageFormatConverterNode>
[component_container_mt-1] [INFO] [1739212696.175663324] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1739212696.175929958] [centerpose_encoder.image_format_converter_node]: [ImageFormatConverterNode] Set output data format to: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1739212696.175997403] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1739212696.207153941] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1739212696.209134034] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1739212696.213703442] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1739212696.236083091] [realsense2_camera]: Device with serial number 819112070232 was found.
[component_container_mt-1]
[component_container_mt-1] [INFO] [1739212696.236148670] [realsense2_camera]: Device with physical ID 3-6.1-21 was found.
[component_container_mt-1] [INFO] [1739212696.236154226] [realsense2_camera]: Device with name Intel RealSense D435 was found.
[component_container_mt-1] [INFO] [1739212696.236288030] [realsense2_camera]: Device with port number 3-6.1 was found.
[component_container_mt-1] [INFO] [1739212696.236295352] [realsense2_camera]: Device USB type: 2.1
[component_container_mt-1] [WARN] [1739212696.236300935] [realsense2_camera]: Device 819112070232 is connected using a 2.1 port. Reduced performance is expected.
[component_container_mt-1] [INFO] [1739212696.237418661] [realsense2_camera]: getParameters...
[component_container_mt-1] [INFO] [1739212696.237691951] [realsense2_camera]: JSON file is not provided
[component_container_mt-1] [INFO] [1739212696.237707821] [realsense2_camera]: Device Name: Intel RealSense D435
[component_container_mt-1] [INFO] [1739212696.237713258] [realsense2_camera]: Device Serial No: 819112070232
[component_container_mt-1] [INFO] [1739212696.237717928] [realsense2_camera]: Device physical port: 3-6.1-21
[component_container_mt-1] [INFO] [1739212696.237722526] [realsense2_camera]: Device FW version: 5.16.0.1
[component_container_mt-1] [INFO] [1739212696.237726636] [realsense2_camera]: Device Product ID: 0x0B07
[component_container_mt-1] [INFO] [1739212696.237730378] [realsense2_camera]: Sync Mode: Off
[component_container_mt-1] [INFO] [1739212696.258314492] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1739212696.263383278] [centerpose_encoder.image_format_converter_node]: [NitrosPublisherSubscriberGroup] Pinning the component "sink/sink" (type="nvidia::isaac_ros::MessageRelay") to use its compatible format only: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1739212696.264643232] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Starting negotiation...
[component_container_mt-1] [INFO] [1739212696.264970527] [centerpose_encoder.resize_node]: Negotiating
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/centerpose_encoder/image_format_converter_node' in container '/isaac_ros_examples/container'
[component_container_mt-1] [INFO] [1739212696.265025979] [isaac_ros_examples.container]: Load Library: /opt/ros/humble/lib/libtriton_node.so
[component_container_mt-1] [INFO] [1739212696.268721479] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::TritonNode>
[component_container_mt-1] [INFO] [1739212696.268747913] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::TritonNode>
[component_container_mt-1] [INFO] [1739212696.270919277] [centerpose_inference]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1739212696.271151794] [centerpose_inference]: [TritonNode] Set input data format to: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1739212696.271165273] [centerpose_inference]: [TritonNode] Set output data format to: "nitros_tensor_list_nhwc_rgb_f32"
[component_container_mt-1] [INFO] [1739212696.271196114] [centerpose_inference]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1739212696.274260436] [centerpose_inference]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1739212696.274324055] [centerpose_inference]: [NitrosContext] Loading extension: gxf/lib/serialization/libgxf_serialization.so
[component_container_mt-1] [INFO] [1739212696.274707311] [centerpose_inference]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_triton.so
[component_container_mt-1] [INFO] [1739212696.295497870] [centerpose_inference]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1739212696.296627889] [centerpose_inference]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1739212696.373320260] [centerpose_inference]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1739212696.383614970] [centerpose_inference]: [NitrosPublisherSubscriberGroup] Pinning the component "triton_request/input" (type="nvidia::gxf::DoubleBufferReceiver") to use its compatible format only: "nitros_tensor_list_nchw_rgb_f32"

[component_container_mt-1] [INFO] [1739212697.058363996] [centerpose_encoder.resize_node]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1739212697.060443278] [centerpose_encoder.resize_node]: [ResizeNode] postLoadGraphCallback().
[component_container_mt-1] [INFO] [1739212697.060507382] [centerpose_encoder.resize_node]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1739212697.070745005] [centerpose_encoder.resize_node]: [NitrosNode] Node was started
[component_container_mt-1]  10/02 13:38:17,153 WARNING [132291546318400] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Success, number: 0
[component_container_mt-1] [INFO] [1739212697.265383016] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1739212697.265436037] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1739212697.265442955] [centerpose_encoder.image_format_converter_node]: [NitrosPublisher] Use the negotiated data format: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1739212697.265449446] [centerpose_encoder.image_format_converter_node]: [NitrosSubscriber] Use the negotiated data format: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1739212697.265558828] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1739212697.273633736] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/JOPJNHWKZH/JOPJNHWKZH.yaml"
[component_container_mt-1] [INFO] [1739212697.273683326] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1739212697.274444811] [centerpose_encoder.image_format_converter_node]: [ImageFormatConverterNode] postLoadGraphCallback().
[component_container_mt-1] [INFO] [1739212697.274492048] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1739212697.275361104] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Node was started
[component_container_mt-1] [INFO] [1739212697.387314503] [centerpose_inference]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1739212697.387377104] [centerpose_inference]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1739212697.387387826] [centerpose_inference]: [NitrosSubscriber] Use the negotiated data format: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1739212697.387406067] [centerpose_inference]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1739212697.387413365] [centerpose_inference]: [NitrosPublisher] Use only the compatible publisher: topic_name="/tensor_sub", data_format="nitros_tensor_list_nhwc_rgb_f32"
[component_container_mt-1] [INFO] [1739212697.387541496] [centerpose_inference]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1739212697.399714519] [centerpose_inference]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/ORDCNIZZKN/ORDCNIZZKN.yaml"
[component_container_mt-1] [INFO] [1739212697.399762911] [centerpose_inference]: [NitrosNode] Loading application
[component_container_mt-1] 2025-02-10 13:38:17.400 WARN  gxf/std/yaml_file_loader.cpp@1076: Using unregistered parameter 'dummy_rx' in component 'requester'.
[component_container_mt-1] [INFO] [1739212697.401040523] [centerpose_inference]: In TritonNode postLoadGraphCallback().
[component_container_mt-1] [INFO] [1739212697.401129255] [centerpose_inference]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1739212697.436553209] [centerpose_encoder.crop_node]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1739212697.436625901] [centerpose_encoder.crop_node]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1739212697.436636735] [centerpose_encoder.crop_node]: [NitrosPublisher] Use the negotiated data format: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1739212697.436647841] [centerpose_encoder.crop_node]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1739212697.436655865] [centerpose_encoder.crop_node]: [NitrosPublisher] Use only the compatible publisher: topic_name="/centerpose_encoder/crop/camera_info", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1739212697.436665467] [centerpose_encoder.crop_node]: [NitrosSubscriber] Use the negotiated data format: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1739212697.436706550] [centerpose_encoder.crop_node]: [NitrosSubscriber] Use the negotiated data format: "nitros_camera_info"
[component_container_mt-1] [INFO] [1739212697.436856607] [centerpose_encoder.crop_node]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1739212697.457844843] [centerpose_decoder_node]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1739212697.457909299] [centerpose_decoder_node]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1739212697.457915222] [centerpose_decoder_node]: [NitrosPublisher] Use the negotiated data format: "nitros_detection3_d_array"
[component_container_mt-1] [INFO] [1739212697.457923008] [centerpose_decoder_node]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1739212697.457927271] [centerpose_decoder_node]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/tensor_sub", data_format="nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1739212697.457933103] [centerpose_decoder_node]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1739212697.457936705] [centerpose_decoder_node]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/camera_info_rect", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1739212697.458023042] [centerpose_decoder_node]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1739212697.460655188] [centerpose_decoder_node]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/ZLPFAWCXOK/ZLPFAWCXOK.yaml"
[component_container_mt-1] [INFO] [1739212697.460707282] [centerpose_decoder_node]: [NitrosNode] Loading application
[component_container_mt-1] WARNING: infer_trtis_server.cpp:1217 NvDsTritonServerInit suggest to set model_control_mode:none. otherwise may cause unknow issues.
[component_container_mt-1] I0210 18:38:17.486437 29401 pinned_memory_manager.cc:277] "Pinned memory pool is created at '0x784fa6000000' with size 268435456"
[component_container_mt-1] [WARN] [1739212697.486886954] [realsense2_camera]:
[component_container_mt-1] I0210 18:38:17.489881 29401 cuda_memory_manager.cc:107] "CUDA memory pool is created on device 0 with size 67108864"
[component_container_mt-1] I0210 18:38:17.497415 29401 server.cc:604]
[component_container_mt-1] +------------------+------+
[component_container_mt-1] | Repository Agent | Path |
[component_container_mt-1] +------------------+------+
[component_container_mt-1] +------------------+------+
[component_container_mt-1]
[component_container_mt-1] I0210 18:38:17.497446 29401 server.cc:631]
[component_container_mt-1] +---------+------+--------+
[component_container_mt-1] | Backend | Path | Config |
[component_container_mt-1] +---------+------+--------+
[component_container_mt-1] +---------+------+--------+
[component_container_mt-1]
[component_container_mt-1] I0210 18:38:17.497459 29401 server.cc:674]
[component_container_mt-1] +-------+---------+--------+
[component_container_mt-1] | Model | Version | Status |
[component_container_mt-1] +-------+---------+--------+
[component_container_mt-1] +-------+---------+--------+
[component_container_mt-1]
[component_container_mt-1] [INFO] [1739212697.509113670] [centerpose_encoder.crop_node]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/RHYRWAAAXF/RHYRWAAAXF.yaml"
[component_container_mt-1] [INFO] [1739212697.509158882] [centerpose_encoder.crop_node]: [NitrosNode] Loading application
[component_container_mt-1] I0210 18:38:17.526851 29401 metrics.cc:877] "Collecting metrics for GPU 0: NVIDIA GeForce RTX 3080 Laptop GPU"
[component_container_mt-1] I0210 18:38:17.529872 29401 metrics.cc:770] "Collecting CPU metrics"
[component_container_mt-1] I0210 18:38:17.529954 29401 tritonserver.cc:2598]
[component_container_mt-1] +----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[component_container_mt-1] | Option                           | Value                                                                                                                                                                                                           |
[component_container_mt-1] +----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[component_container_mt-1] | server_id                        | triton                                                                                                                                                                                                          |
[component_container_mt-1] | server_version                   | 2.49.0                                                                                                                                                                                                          |
[component_container_mt-1] | server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data parameters statistics trace logging |
[component_container_mt-1] | model_repository_path[0]         | /workspaces/isaac_ros-dev/isaac_ros_assets/models                                                                                                                                                               |
[component_container_mt-1] | model_control_mode               | MODE_EXPLICIT                                                                                                                                                                                                   |
[component_container_mt-1] | strict_model_config              | 1                                                                                                                                                                                                               |
[component_container_mt-1] | model_config_name                |                                                                                                                                                                                                                 |
[component_container_mt-1] | rate_limit                       | OFF                                                                                                                                                                                                             |
[component_container_mt-1] | pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                                       |
[component_container_mt-1] | cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                                        |
[component_container_mt-1] | min_supported_compute_capability | 6.0                                                                                                                                                                                                             |
[component_container_mt-1] | strict_readiness                 | 1                                                                                                                                                                                                               |
[component_container_mt-1] | exit_timeout                     | 30                                                                                                                                                                                                              |
[component_container_mt-1] | cache_enabled                    | 0                                                                                                                                                                                                               |
[component_container_mt-1] +----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[component_container_mt-1]
[component_container_mt-1] [INFO] [1739212697.530268721] [centerpose_inference]: [NitrosNode] Node was started
[component_container_mt-1] I0210 18:38:17.532135 29401 model_lifecycle.cc:472] "loading: centerpose_shoe:1"
[component_container_mt-1] [INFO] [1739212697.533632105] [centerpose_encoder.crop_node]: [CropNode] postLoadGraphCallback().
[component_container_mt-1] [INFO] [1739212697.533676373] [centerpose_decoder_node]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1739212697.533720044] [centerpose_encoder.crop_node]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1739212697.534157706] [centerpose_decoder_node]: [NitrosNode] Node was started
[component_container_mt-1] [INFO] [1739212697.536461852] [centerpose_encoder.crop_node]: [NitrosNode] Node was started
[component_container_mt-1] I0210 18:38:17.537370 29401 tensorrt.cc:65] "TRITONBACKEND_Initialize: tensorrt"
[component_container_mt-1] I0210 18:38:17.537383 29401 tensorrt.cc:75] "Triton TRITONBACKEND API version: 1.19"
[component_container_mt-1] I0210 18:38:17.537386 29401 tensorrt.cc:81] "'tensorrt' TRITONBACKEND API version: 1.19"
[component_container_mt-1] I0210 18:38:17.537389 29401 tensorrt.cc:105] "backend configuration:\n{\"cmdline\":{\"auto-complete-config\":\"false\",\"backend-directory\":\"/opt/tritonserver/backends\",\"min-compute-capability\":\"6.000000\",\"default-max-batch-size\":\"4\"}}"
[component_container_mt-1] I0210 18:38:17.540469 29401 tensorrt.cc:231] "TRITONBACKEND_ModelInitialize: centerpose_shoe (version 1)"
[component_container_mt-1] I0210 18:38:17.548504 29401 tensorrt.cc:297] "TRITONBACKEND_ModelInstanceInitialize: centerpose_shoe_0 (GPU device 0)"
[component_container_mt-1] [INFO] [1739212697.600741546] [centerpose_visualizer]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1739212697.600795814] [centerpose_visualizer]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1739212697.600803406] [centerpose_visualizer]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1739212697.600809674] [centerpose_visualizer]: [NitrosPublisher] Use only the compatible publisher: topic_name="/centerpose/image_visualized", data_format="nitros_image_bgr8"
[component_container_mt-1] [INFO] [1739212697.600815736] [centerpose_visualizer]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1739212697.600820081] [centerpose_visualizer]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/image_rect", data_format="nitros_image_bgr8"
[component_container_mt-1] [INFO] [1739212697.600825245] [centerpose_visualizer]: [NitrosSubscriber] Use the negotiated data format: "nitros_detection3_d_array"
[component_container_mt-1] [INFO] [1739212697.600829686] [centerpose_visualizer]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1739212697.600833111] [centerpose_visualizer]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/camera_info_rect", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1739212697.600921834] [centerpose_visualizer]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1739212697.602212595] [centerpose_visualizer]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/ZQYENQMAZE/ZQYENQMAZE.yaml"
[component_container_mt-1] [INFO] [1739212697.602240046] [centerpose_visualizer]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1739212697.603061536] [centerpose_visualizer]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1739212697.603487593] [centerpose_visualizer]: [NitrosNode] Node was started
[component_container_mt-1] [INFO] [1739212697.624937185] [centerpose_encoder.interleaved_to_planar_node]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1739212697.624988026] [centerpose_encoder.interleaved_to_planar_node]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1739212697.624995752] [centerpose_encoder.interleaved_to_planar_node]: [NitrosPublisher] Use the negotiated data format: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1739212697.625003554] [centerpose_encoder.interleaved_to_planar_node]: [NitrosSubscriber] Use the negotiated data format: "nitros_tensor_list_nhwc_rgb_f32"
[component_container_mt-1] [INFO] [1739212697.625110365] [centerpose_encoder.interleaved_to_planar_node]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1739212697.628383614] [centerpose_encoder.interleaved_to_planar_node]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/GCHRSDFUCE/GCHRSDFUCE.yaml"
[component_container_mt-1] [INFO] [1739212697.628431007] [centerpose_encoder.interleaved_to_planar_node]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1739212697.629692589] [centerpose_encoder.interleaved_to_planar_node]: In InterleavedToPlanarNode postLoadGraphCallback().
[component_container_mt-1] [INFO] [1739212697.629750673] [centerpose_encoder.interleaved_to_planar_node]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1739212697.633695677] [centerpose_encoder.interleaved_to_planar_node]: [NitrosNode] Node was started
[component_container_mt-1] [INFO] [1739212697.641069901] [centerpose_encoder.reshape_node]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1739212697.641174576] [centerpose_encoder.reshape_node]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1739212697.641187430] [centerpose_encoder.reshape_node]: [NitrosPublisher] Use the negotiated data format: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1739212697.641210814] [centerpose_encoder.reshape_node]: [NitrosSubscriber] Use the negotiated data format: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1739212697.641364253] [centerpose_encoder.reshape_node]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1739212697.643968500] [centerpose_encoder.reshape_node]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/LVGBMKXGRM/LVGBMKXGRM.yaml"
[component_container_mt-1] [INFO] [1739212697.644001324] [centerpose_encoder.reshape_node]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1739212697.645647803] [centerpose_encoder.reshape_node]: In ReshapeNode postLoadGraphCallback().
[component_container_mt-1] [INFO] [1739212697.645731848] [centerpose_encoder.reshape_node]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1739212697.656862495] [centerpose_encoder.reshape_node]: [NitrosNode] Node was started
[component_container_mt-1] I0210 18:38:17.710883 29401 logging.cc:46] "Loaded engine size: 126 MiB"
[component_container_mt-1] I0210 18:38:17.760320 29401 logging.cc:46] "[MS] Running engine with multi stream info"
[component_container_mt-1] I0210 18:38:17.760345 29401 logging.cc:46] "[MS] Number of aux streams is 6"
[component_container_mt-1] I0210 18:38:17.760350 29401 logging.cc:46] "[MS] Number of total worker streams is 7"
[component_container_mt-1] I0210 18:38:17.760355 29401 logging.cc:46] "[MS] The main stream provided by execute/enqueue calls is the first worker stream"
[component_container_mt-1] I0210 18:38:17.774897 29401 logging.cc:46] "[MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +246, now: CPU 0, GPU 369 (MiB)"
[component_container_mt-1] I0210 18:38:17.775340 29401 instance_state.cc:186] "Created instance centerpose_shoe_0 on GPU 0 with stream priority 0 and optimization profile default[0];"
[component_container_mt-1] I0210 18:38:17.775530 29401 model_lifecycle.cc:839] "successfully loaded 'centerpose_shoe'"
[component_container_mt-1] INFO: infer_simple_runtime.cpp:71 TrtISBackend id:118 initialized model: centerpose_shoe
[component_container_mt-1] 2025-02-10 13:38:17.864 WARN  extensions/triton/inferencers/triton_inferencer_impl.cpp@508: Incomplete inference; response appeared out of order; invalid: index = 3
[component_container_mt-1] 2025-02-10 13:38:17.864 WARN  extensions/triton/inferencers/triton_inferencer_impl.cpp@510: Inference appeared out of order
[component_container_mt-1] W0210 18:38:18.531485 29401 metrics.cc:631] "Unable to get power limit for GPU 0. Status:Success, value:0.000000"
[component_container_mt-1] W0210 18:38:19.536522 29401 metrics.cc:631] "Unable to get power limit for GPU 0. Status:Success, value:0.000000"
[component_container_mt-1] W0210 18:38:20.536873 29401 metrics.cc:631] "Unable to get power limit for GPU 0. Status:Success, value:0.000000"
[component_container_mt-1] [ERROR] [1739212775.626219961] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).
[component_container_mt-1] [ERROR] [1739212775.692867423] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).
[component_container_mt-1] [ERROR] [1739212775.759632533] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).
[component_container_mt-1] [ERROR] [1739212775.826358586] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).
[component_container_mt-1] [ERROR] [1739212775.892954017] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).
[component_container_mt-1] [ERROR] [1739212775.959695004] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).
[component_container_mt-1] [ERROR] [1739212776.026363553] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).
[component_container_mt-1] [ERROR] [1739212776.093130349] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).
[component_container_mt-1] [ERROR] [1739212776.159979521] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).
[component_container_mt-1] [ERROR] [1739212776.226647413] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).
[component_container_mt-1] [ERROR] [1739212776.293286375] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).
[component_container_mt-1] [ERROR] [1739212776.360026547] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).
[component_container_mt-1] [ERROR] [1739212776.426745982] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).
[component_container_mt-1] [ERROR] [1739212776.493587008] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).
[component_container_mt-1] [ERROR] [1739212776.560120372] [compressed_depth_image_transport]: Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8).

^C[WARNING] [launch]: user interrupted with ctrl-c (SIGINT)
[component_container_mt-1] [INFO] [1739212793.492127184] [rclcpp]: signal_handler(signum=2)
[component_container_mt-1] [INFO] [1739212793.495194642] [centerpose_encoder.reshape_node]: [NitrosNode] Terminating the running application
[component_container_mt-1] [INFO] [1739212793.495206783] [centerpose_encoder.reshape_node]: [NitrosContext] Interrupting GXF...
[component_container_mt-1] [INFO] [1739212793.495366781] [centerpose_encoder.reshape_node]: [NitrosContext] Waiting on GXF...
[component_container_mt-1] [INFO] [1739212793.495375298] [centerpose_encoder.reshape_node]: [NitrosContext] Deinitializing...
[component_container_mt-1] [INFO] [1739212793.495780055] [centerpose_encoder.reshape_node]: [NitrosContext] Destroying context
[component_container_mt-1] [INFO] [1739212793.495869127] [centerpose_encoder.reshape_node]: [NitrosNode] Application termination done
[component_container_mt-1] [INFO] [1739212793.501997551] [centerpose_encoder.interleaved_to_planar_node]: [NitrosNode] Terminating the running application
[component_container_mt-1] [INFO] [1739212793.502004426] [centerpose_encoder.interleaved_to_planar_node]: [NitrosContext] Interrupting GXF...
[component_container_mt-1] [INFO] [1739212793.502146189] [centerpose_encoder.interleaved_to_planar_node]: [NitrosContext] Waiting on GXF...
[component_container_mt-1] [INFO] [1739212793.502149242] [centerpose_encoder.interleaved_to_planar_node]: [NitrosContext] Deinitializing...
[component_container_mt-1] [INFO] [1739212793.502411655] [centerpose_encoder.interleaved_to_planar_node]: [NitrosContext] Destroying context
[component_container_mt-1] [INFO] [1739212793.502495447] [centerpose_encoder.interleaved_to_planar_node]: [NitrosNode] Application termination done
[component_container_mt-1] [INFO] [1739212793.509900329] [centerpose_visualizer]: [NitrosNode] Terminating the running application
[component_container_mt-1] [INFO] [1739212793.509908738] [centerpose_visualizer]: [NitrosContext] Interrupting GXF...
[component_container_mt-1] [INFO] [1739212793.510056412] [centerpose_visualizer]: [NitrosContext] Waiting on GXF...
[component_container_mt-1] [INFO] [1739212793.510059485] [centerpose_visualizer]: [NitrosContext] Deinitializing...
[component_container_mt-1] [INFO] [1739212793.511012787] [centerpose_visualizer]: [NitrosContext] Destroying context
[component_container_mt-1] [INFO] [1739212793.511055780] [centerpose_visualizer]: [NitrosNode] Application termination done
[component_container_mt-1] [INFO] [1739212793.517515921] [centerpose_decoder_node]: [NitrosNode] Terminating the running application
[component_container_mt-1] [INFO] [1739212793.517522450] [centerpose_decoder_node]: [NitrosContext] Interrupting GXF...
[component_container_mt-1] |==================================================================================================================================================================|
[component_container_mt-1] |                                           Job Statistics Report (regular)                                                                                        |
[component_container_mt-1] |==================================================================================================================================================================|
[component_container_mt-1] | Name                                               |   Count | Time (Median - 90% - Max) [ms] | Load (%) | Exec(ms) | Variation (Median - 90% - Max) [ns]        |
[component_container_mt-1] |------------------------------------------------------------------------------------------------------------------------------------------------------------------|
[component_container_mt-1] |                                    LVGBMKXGRM_sink |    1438 |     0.02 |     0.03 |     0.33 |    0.0 % |     30.7 |         7855 |        14548 |       125375 |
[component_container_mt-1] |                                LVGBMKXGRM_reshaper |    1438 |     0.07 |     0.09 |     2.31 |    0.1 % |    106.4 |        18551 |        31399 |       415994 |
[component_container_mt-1] |==================================================================================================================================================================|
[component_container_mt-1] |==================================================================================================================================================================|
[component_container_mt-1] |                                           Entity Statistics Report (regular)                                                                                     |
[component_container_mt-1] |==================================================================================================================================================================|
[component_container_mt-1] | Entity Name             | Entity State             |   Count | Time (Median - 90% - Max) [ms]                                                                    |
[component_container_mt-1] |------------------------------------------------------------------------------------------------------------------------------------------------------------------|
[component_container_mt-1] |         LVGBMKXGRM_sink  |              StopPending |       1 |  0.00289 |  0.00289 |  0.00289                                                                   |
[component_container_mt-1] |         LVGBMKXGRM_sink  |                     Idle |    1438 | 66.82859 | 67.93623 | 90.22056                                                                   |
[component_container_mt-1] |         LVGBMKXGRM_sink  |                  Ticking |    1438 |  0.01030 |  0.01457 |  0.31507                                                                   |
[component_container_mt-1] |         LVGBMKXGRM_sink  |              TickPending |    1438 |  0.00479 |  0.00759 |  0.12732                                                                   |
[component_container_mt-1] |------------------------------------------------------------------------------------------------------------------------------------------------------------------|
[component_container_mt-1] |     LVGBMKXGRM_reshaper  |              StopPending |       1 |  0.00871 |  0.00871 |  0.00871                                                                   |
[component_container_mt-1] |     LVGBMKXGRM_reshaper  |                     Idle |    1438 | 66.73471 | 67.88522 | 87.86595                                                                   |
[component_container_mt-1] |     LVGBMKXGRM_reshaper  |                  Ticking |    1438 |  0.05575 |  0.07723 |  2.28366                                                                   |
[component_container_mt-1] |     LVGBMKXGRM_reshaper  |              TickPending |    1438 |  0.00755 |  0.00927 |  0.03082                                                                   |
[component_container_mt-1] |------------------------------------------------------------------------------------------------------------------------------------------------------------------|
[component_container_mt-1] |==================================================================================================================================================================|
[component_container_mt-1] |==================================================================================================================================================================|
[component_container_mt-1] |                                           Job Statistics Report (regular)                                                                                        |
[component_container_mt-1] |==================================================================================================================================================================|
[component_container_mt-1] | Name                                               |   Count | Time (Median - 90% - Max) [ms] | Load (%) | Exec(ms) | Variation (Median - 90% - Max) [ns]        |
[component_container_mt-1] |------------------------------------------------------------------------------------------------------------------------------------------------------------------|
[component_container_mt-1] |                                    GCHRSDFUCE_sink |    1438 |     0.02 |     0.03 |     0.38 |    0.0 % |     35.5 |        10698 |        30111 |       116491 |
[component_container_mt-1] |                   GCHRSDFUCE_interleaved_to_planar |    1438 |     0.08 |     0.10 |     4.62 |    0.1 % |    126.8 |        19128 |        36651 |       136618 |
[component_container_mt-1] |==================================================================================================================================================================|
[component_container_mt-1] |==================================================================================================================================================================|
[component_container_mt-1] |                                           Entity Statistics Report (regular)                                                                                     |
[component_container_mt-1] |==================================================================================================================================================================|
[component_container_mt-1] | Entity Name             | Entity State             |   Count | Time (Median - 90% - Max) [ms]                                                                    |
[component_container_mt-1] |------------------------------------------------------------------------------------------------------------------------------------------------------------------|
[component_container_mt-1] |         GCHRSDFUCE_sink  |              StopPending |       1 |  0.00303 |  0.00303 |  0.00303                                                                   |
[component_container_mt-1] |         GCHRSDFUCE_sink  |                     Idle |    1438 | 66.75263 | 67.87476 | 88.29906                                                                   |
[component_container_mt-1] |         GCHRSDFUCE_sink  |                  Ticking |    1438 |  0.01256 |  0.01735 |  0.37503                                                                   |
[component_container_mt-1] |         GCHRSDFUCE_sink  |              TickPending |    1438 |  0.00657 |  0.00887 |  0.02385                                                                   |
[component_container_mt-1] |------------------------------------------------------------------------------------------------------------------------------------------------------------------|
[component_container_mt-1] | GCHRSDFUCE_interleaved_  |              StopPending |       1 |  0.00627 |  0.00627 |  0.00627                                                                   |
[component_container_mt-1] | GCHRSDFUCE_interleaved_  |                     Idle |    1438 | 66.72014 | 67.79755 | 88.21439                                                                   |
[component_container_mt-1] | GCHRSDFUCE_interleaved_  |                  Ticking |    1438 |  0.06888 |  0.07848 |  4.58151                                                                   |
[component_container_mt-1] | GCHRSDFUCE_interleaved_  |              TickPending |    1438 |  0.00875 |  0.01226 |  0.03707                                                                   |
[component_container_mt-1] |------------------------------------------------------------------------------------------------------------------------------------------------------------------|
[component_container_mt-1] |==================================================================================================================================================================|
[component_container_mt-1] |==================================================================================================================================================================|
^C[WARNING] [launch]: user interrupted with ctrl-c (SIGINT) again, ignoring...
[component_container_mt-1] [INFO] [1739212793.967203117] [rclcpp]: signal_handler(signum=2)
[ERROR] [component_container_mt-1]: process[component_container_mt-1] failed to terminate '5' seconds after receiving 'SIGINT', escalating to 'SIGTERM'
[INFO] [component_container_mt-1]: sending signal 'SIGTERM' to process[component_container_mt-1]
[component_container_mt-1] [INFO] [1739212798.671138397] [rclcpp]: signal_handler(signum=15)
[ERROR] [component_container_mt-1]: process[component_container_mt-1] failed to terminate '10.0' seconds after receiving 'SIGTERM', escalating to 'SIGKILL'
[INFO] [component_container_mt-1]: sending signal 'SIGKILL' to process[component_container_mt-1]
[ERROR] [component_container_mt-1]: process has died [pid 29401, exit code -9, cmd '/opt/ros/humble/lib/rclcpp_components/component_container_mt --ros-args -r __node:=container -r __ns:=/isaac_ros_examples'].
[component_container_mt-1] |                                           Job Statistics Report (regular)                                
admin@DOS:/workspaces/isaac_ros-dev/isaac_ros_assets/models$

I have attached the whole log in txt file below.

centerpose_shoe.txt

This has been an inconsistent behavior (happens every couple runs).

Also, when it runs correctly, after a few seconds it freezes.

[component_container_mt-1] [INFO] [1739213959.220116271] [centerpose_encoder.reshape_node]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1739213959.223254699] [centerpose_encoder.reshape_node]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/AWNWRGXGEZ/AWNWRGXGEZ.yaml"
[component_container_mt-1] [INFO] [1739213959.223312935] [centerpose_encoder.reshape_node]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1739213959.224727513] [centerpose_encoder.reshape_node]: In ReshapeNode postLoadGraphCallback().
[component_container_mt-1] [INFO] [1739213959.224798491] [centerpose_encoder.reshape_node]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1739213959.227243103] [centerpose_encoder.reshape_node]: [NitrosNode] Node was started
[component_container_mt-1] I0210 18:59:19.267083 70643 logging.cc:46] "Loaded engine size: 126 MiB"
[component_container_mt-1] I0210 18:59:19.320839 70643 logging.cc:46] "[MS] Running engine with multi stream info"
[component_container_mt-1] I0210 18:59:19.320869 70643 logging.cc:46] "[MS] Number of aux streams is 6"
[component_container_mt-1] I0210 18:59:19.320874 70643 logging.cc:46] "[MS] Number of total worker streams is 7"
[component_container_mt-1] I0210 18:59:19.320879 70643 logging.cc:46] "[MS] The main stream provided by execute/enqueue calls is the first worker stream"
[component_container_mt-1] I0210 18:59:19.338148 70643 logging.cc:46] "[MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +246, now: CPU 0, GPU 369 (MiB)"
[component_container_mt-1] I0210 18:59:19.338522 70643 instance_state.cc:186] "Created instance centerpose_shoe_0 on GPU 0 with stream priority 0 and optimization profile default[0];"
[component_container_mt-1] I0210 18:59:19.338718 70643 model_lifecycle.cc:839] "successfully loaded 'centerpose_shoe'"
[component_container_mt-1] INFO: infer_simple_runtime.cpp:71 TrtISBackend id:118 initialized model: centerpose_shoe
[component_container_mt-1] 2025-02-10 13:59:19.437 WARN  extensions/triton/inferencers/triton_inferencer_impl.cpp@508: Incomplete inference; response appeared out of order; invalid: index = 3
[component_container_mt-1] 2025-02-10 13:59:19.437 WARN  extensions/triton/inferencers/triton_inferencer_impl.cpp@510: Inference appeared out of order
[component_container_mt-1] W0210 18:59:20.108134 70643 metrics.cc:631] "Unable to get power limit for GPU 0. Status:Success, value:0.000000"
[component_container_mt-1] W0210 18:59:21.115316 70643 metrics.cc:631] "Unable to get power limit for GPU 0. Status:Success, value:0.000000"
[component_container_mt-1] W0210 18:59:22.115675 70643 metrics.cc:631] "Unable to get power limit for GPU 0. Status:Success, value:0.000000"

^^ this is part of the log when it runs correctly, I do see the following two warnings in it:

[component_container_mt-1] 2025-02-10 13:59:19.437 WARN  extensions/triton/inferencers/triton_inferencer_impl.cpp@508: Incomplete inference; response appeared out of order; invalid: index = 3
[component_container_mt-1] 2025-02-10 13:59:19.437 WARN  extensions/triton/inferencers/triton_inferencer_impl.cpp@510: Inference appeared out of order

^^I attached above the screencast that showcases the inference freezing.

Raffaello · February 11, 2025, 4:45pm

Hi @mona.jalal

Apologies, but could you please clarify where the error is? After reviewing the logs and screenshot, I don’t see any mistakes.

I just saw you interrupt the execution of the ros 2 launch.

Check if your laptop has enough memory to run the demo; this issue often arises when the CPU is occupied with other tasks.

mona.jalal:

[component_container_mt-1] 2025-02-10 13:59:19.437 WARN  extensions/triton/inferencers/triton_inferencer_impl.cpp@508: Incomplete inference; response appeared out of order; invalid: index = 3
[component_container_mt-1] 2025-02-10 13:59:19.437 WARN  extensions/triton/inferencers/triton_inferencer_impl.cpp@510: Inference appeared out of order

You can skip these warnings; I’m asking engineer for a double-check.

Best,
Raffaello

mona.jalal · February 11, 2025, 4:49pm

@Raffaello the error is in freezing after like 30sec or a 1 min of execution.

You can also see the $ htop output when running the inference on shoes:

(I had to use CTRL+C since the visualization using rviz freezes)

Here’s my memory info:

2025-02-11 11:50:41 [mona@DOS ~]$ sudo cat /proc/meminfo
MemTotal:       65559092 kB
MemFree:         1827292 kB
MemAvailable:   44680460 kB
Buffers:         5752476 kB
Cached:         33545280 kB
SwapCached:         8772 kB
Active:         13563444 kB
Inactive:       39125432 kB
Active(anon):   11274364 kB
Inactive(anon):  3776372 kB
Active(file):    2289080 kB
Inactive(file): 35349060 kB
Unevictable:     3377240 kB
Mlocked:         2534620 kB
SwapTotal:        999420 kB
SwapFree:         907772 kB
Zswap:                 0 kB
Zswapped:              0 kB
Dirty:              1092 kB
Writeback:             0 kB
AnonPages:      16764664 kB
Mapped:          2316852 kB
Shmem:           1637268 kB
KReclaimable:    5976172 kB
Slab:            6857236 kB
SReclaimable:    5976172 kB
SUnreclaim:       881064 kB
KernelStack:       66176 kB
PageTables:       170776 kB
SecPageTables:         0 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    33778964 kB
Committed_AS:   43035268 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      185624 kB
VmallocChunk:          0 kB
Percpu:            29568 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:  1150976 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
Unaccepted:            0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:     2590144 kB
DirectMap2M:    49551360 kB
DirectMap1G:    14680064 kB
2025-02-11 11:50:47 [mona@DOS ~]$ sudo  cat /proc/meminfo | grep MemTotal 
MemTotal:       65559092 kB
2025-02-11 11:51:00 [mona@DOS ~]$ free -m
               total        used        free      shared  buff/cache   available
Mem:           64022       18022        1774        1611       44225       43623
Swap:            975          89         886

Raffaello · February 11, 2025, 5:21pm

Have you also check also with nvtop?

I’m escalating your issue within the team. I tested it on my laptop and did not encounter any problems.

Raffaello

mona.jalal · February 11, 2025, 6:07pm

Thanks a lot for looking into this.

I created a new recording that shows when it gets stuck.

also screenshot of htop+nvtop after hang.

Ahung · February 12, 2025, 8:54am

Hi @mona.jalal,

[component_container_mt-1] [INFO] [1739212696.237722526] [realsense2_camera]: Device FW version: 5.16.0.1

Could you try to use RealSense firmware version 5.13.0.50 and see if any improvement in error/warning log and freeze issue?

https://nvidia-isaac-ros.github.io/getting_started/hardware_setup/sensors/realsense_setup.html

Best,
Ahung

mona.jalal · February 12, 2025, 2:05pm

@Ahung

I followed the same exact instructions as in Isaac ROS RealSense Setup — isaac_ros_docs documentation

It installed Intel RealSense Viewer v2.55.1, do you know why it did install a newer firmware?

When I enter this command, it shows 5.13.0.55. I am not sure what is going on?

admin@DOS:/workspaces/isaac_ros-dev$ rs-fw-update -l

Connected devices:
1) [USB] Intel RealSense D435 s/n 250122070375, update serial number: 415743023395, firmware version: 5.13.0.55
admin@DOS:/workspaces/isaac_ros-dev$

mona.jalal · February 12, 2025, 2:24pm

The hanging didn’t happen. The only change was restarting my machine.

no_hang_centerpose_shoes.txt (99.6 KB)

[component_container_mt-1] [INFO] [1739369460.298326007] [realsense2_camera]: Device with serial number 250122070375 was found.
[component_container_mt-1]
[component_container_mt-1] [INFO] [1739369460.298521740] [realsense2_camera]: Device with physical ID 2-1.4.4.2-5 was found.
[component_container_mt-1] [INFO] [1739369460.298529425] [realsense2_camera]: Device with name Intel RealSense D435 was found.
[component_container_mt-1] [INFO] [1739369460.298763934] [realsense2_camera]: Device with port number 2-1.4.4.2 was found.
[component_container_mt-1] [INFO] [1739369460.298778196] [realsense2_camera]: Device USB type: 3.2
[component_container_mt-1] [INFO] [1739369460.301382454] [realsense2_camera]: getParameters...
[component_container_mt-1] [INFO] [1739369460.301777385] [realsense2_camera]: JSON file is not provided
[component_container_mt-1] [INFO] [1739369460.301792438] [realsense2_camera]: Device Name: Intel RealSense D435
[component_container_mt-1] [INFO] [1739369460.301819801] [realsense2_camera]: Device Serial No: 250122070375
[component_container_mt-1] [INFO] [1739369460.301829692] [realsense2_camera]: Device physical port: 2-1.4.4.2-5
[component_container_mt-1] [INFO] [1739369460.301836112] [realsense2_camera]: Device FW version: 5.13.0.55
[component_container_mt-1] [INFO] [1739369460.301847646] [realsense2_camera]: Device Product ID: 0x0B07
[component_container_mt-1] [INFO] [1739369460.301873729] [realsense2_camera]: Sync Mode: Off
[component_container_mt-1] [INFO] [1739369460.303995812] [centerpose_inference]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1739369460.304257427] [centerpose_inference]: [TritonNode] Set input data format to: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1739369460.304273290] [centerpose_inference]: [TritonNode] Set output data format to: "nitros_tensor_list_nhwc_rgb_f32"

I notice every time despite committing changes to the docker, I am still installing the followings every time.

Any thoughts how this could be fixed?

sudo apt-get install -y ros-humble-isaac-ros-centerpose

sudo apt-get install -y ros-humble-isaac-ros-examples ros-humble-isaac-ros-realsense

It’s not a big deal to enter two commands but I am not sure why docker commit is not working as expected. Probably an err on my side.

I’ve been using this command:

docker commit fe0b5a6fb5e1 isaac_ros_dev-x86_64:latest

Following:

2025-02-12 09:19:12 [mona@DOS ~]$ docker ps
CONTAINER ID   IMAGE                                                 COMMAND                  CREATED          STATUS                    PORTS                                                 NAMES
fe0b5a6fb5e1   isaac_ros_dev-x86_64                                  "/usr/local/bin/scri…"   24 minutes ago   Up 24 minutes                                                                   isaac_ros_dev-x86_64-container

mona.jalal · February 12, 2025, 2:32pm

@Ahung @Raffaello

Given the fact, I got it working with Intel RealSense D435 (I do see D435i, which is like mine but with an IMU), I think we can add it to this list here:

https://nvidia-isaac-ros.github.io/getting_started/hardware_setup/sensors/realsense_setup.html

Raffaello · February 19, 2025, 5:56pm

Hi @mona.jalal

I forward your suggestion to the engineering.

Thank you!

Raffaello

system · March 5, 2025, 5:57pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Jetson Orin Nano with Intel Realsense D455--isaac_ros_visual_slam problem Isaac ROS jetson , isaac-ros-vslam	3	119	November 8, 2024
Running Isaac ROS Packages with Intel RealSense D457 Isaac ROS	2	162	August 29, 2024
No inference results for custom object using isaac_ros_centerpose - Compression requires single-channel 32bit-floating point or 16bit raw depth images Isaac ROS	2	27	February 12, 2025
Isaac_ros_freespace_segmentation launch error on Docker container Jetson Orin Nano containers	2	731	April 26, 2023
Isaac_ros_visual_slam Quickstart - No PCL - ubuntu 22 x86, RS 435i Isaac ROS	1	168	June 24, 2024
/Centerpose/image-visualized no inference image display Isaac ROS	1	26	February 11, 2025
Isaac_ros_ess Visualization Problem Isaac ROS camera	10	175	November 1, 2024
Isaac_ros_centerpose inference with Intel RealSense D435 stuck at "Unable to get power limit for GPU 0" Isaac ROS	4	53	February 7, 2025
Isaac_ros_foundationpose tracking takes approximately 20 seconds to show pose detection results upon starting rviz Isaac ROS camera , mesh , isaac-ros-object-detection , isaac-ros-pose-estimation	10	96	March 24, 2025
Isaac ROS VSlam Examples No Realsense devices found (Tried all Troubleshooting mentioned in documentation) Isaac ROS camera , isaac-ros-vslam	5	202	July 15, 2024

Compressed Depth Image Transport - Compression requires single-channel 32bit-floating point or 16bit raw depth images (input format is: rgb8)

Related topics