Isaac_ros_centerpose inference with Intel RealSense D435 stuck at "Unable to get power limit for GPU 0"

mona.jalal · February 5, 2025, 8:09pm

Currently, this command is stuck at here:


admin@DOS:/workspaces/isaac_ros-dev$ ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=realsense_mono_rect,centerpose,centerpose_visualizer model_name:=centerpose_shoe model_repository_paths:=[${ISAAC_ROS_WS}/isaac_ros_assets/models]
[INFO] [launch]: All log files can be found below /home/admin/.ros/log/2025-02-05-15-02-12-500591-DOS-15739
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container_mt-1]: process started with pid [15752]
[component_container_mt-1] [INFO] [1738785732.813857183] [isaac_ros_examples.container]: Load Library: /opt/ros/humble/lib/libresize_node.so
[component_container_mt-1] [INFO] [1738785732.827032738] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::ResizeNode>
[component_container_mt-1] [INFO] [1738785732.827056585] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::ResizeNode>
[component_container_mt-1] [INFO] [1738785735.348455726] [centerpose_encoder.resize_node]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1738785735.348656678] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/std/libgxf_std.so
[component_container_mt-1] [INFO] [1738785735.349608794] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_gxf_helpers.so
[component_container_mt-1] [INFO] [1738785735.351550108] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_sight.so
[component_container_mt-1] [INFO] [1738785735.353611199] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_atlas.so
[component_container_mt-1] 2025-02-05 15:02:15.358 WARN  gxf/std/program.cpp@538: No GXF scheduler specified.
[component_container_mt-1] [INFO] [1738785735.359016937] [centerpose_encoder.resize_node]: [ResizeNode] Set output data format to: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1738785735.359103145] [centerpose_encoder.resize_node]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1738785735.365014286] [centerpose_encoder.resize_node]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1738785735.365063246] [centerpose_encoder.resize_node]: [NitrosContext] Loading extension: gxf/lib/multimedia/libgxf_multimedia.so
[component_container_mt-1] [INFO] [1738785735.366534825] [centerpose_encoder.resize_node]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_message_compositor.so
[component_container_mt-1] [INFO] [1738785735.366807747] [centerpose_encoder.resize_node]: [NitrosContext] Loading extension: gxf/lib/cuda/libgxf_cuda.so
[component_container_mt-1] [INFO] [1738785735.368373252] [centerpose_encoder.resize_node]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_tensorops.so
[component_container_mt-1] [INFO] [1738785735.370403984] [centerpose_encoder.resize_node]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1738785735.372631660] [centerpose_encoder.resize_node]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1738785735.480341516] [centerpose_encoder.resize_node]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1738785735.482505191] [centerpose_encoder.resize_node]: [NitrosPublisherSubscriberGroup] Pinning the component "image_sink/sink" (type="nvidia::isaac_ros::MessageRelay") to use its compatible format only: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1738785735.488608579] [centerpose_encoder.resize_node]: [NitrosNode] Starting negotiation...
[component_container_mt-1] [INFO] [1738785735.489255523] [isaac_ros_examples.container]: Load Library: /opt/ros/humble/lib/librealsense2_camera.so
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/centerpose_encoder/resize_node' in container '/isaac_ros_examples/container'
[component_container_mt-1] [INFO] [1738785735.559955594] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<realsense2_camera::RealSenseNodeFactory>
[component_container_mt-1] [INFO] [1738785735.560014323] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<realsense2_camera::RealSenseNodeFactory>
[component_container_mt-1] [INFO] [1738785735.574216809] [realsense2_camera]: RealSense ROS v4.51.1
[component_container_mt-1] [INFO] [1738785735.574592880] [realsense2_camera]: Built with LibRealSense v2.55.1
[component_container_mt-1] [INFO] [1738785735.574604803] [realsense2_camera]: Running with LibRealSense v2.55.1
[component_container_mt-1] [INFO] [1738785735.575161031] [isaac_ros_examples.container]: Load Library: /opt/ros/humble/lib/libimage_format_converter_node.so
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/realsense2_camera' in container '/isaac_ros_examples/container'
[component_container_mt-1] [INFO] [1738785735.576271082] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::ImageFormatConverterNode>
[component_container_mt-1] [INFO] [1738785735.576295046] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::ImageFormatConverterNode>
[component_container_mt-1] [INFO] [1738785735.583623185] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1738785735.583879479] [centerpose_encoder.image_format_converter_node]: [ImageFormatConverterNode] Set output data format to: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1738785735.583944306] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1738785735.606913634] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1738785735.608928260] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1738785735.611928192] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1738785735.663628175] [realsense2_camera]: Device with serial number 819112070232 was found.
[component_container_mt-1] 
[component_container_mt-1] [INFO] [1738785735.664320267] [realsense2_camera]: Device with physical ID 3-7.2-71 was found.
[component_container_mt-1] [INFO] [1738785735.664330537] [realsense2_camera]: Device with name Intel RealSense D435 was found.
[component_container_mt-1] [INFO] [1738785735.664472386] [realsense2_camera]: Device with port number 3-7.2 was found.
[component_container_mt-1] [INFO] [1738785735.664480336] [realsense2_camera]: Device USB type: 2.1
[component_container_mt-1] [WARN] [1738785735.664486124] [realsense2_camera]: Device 819112070232 is connected using a 2.1 port. Reduced performance is expected.
[component_container_mt-1] [INFO] [1738785735.665557935] [realsense2_camera]: getParameters...
[component_container_mt-1] [INFO] [1738785735.665758982] [realsense2_camera]: JSON file is not provided
[component_container_mt-1] [INFO] [1738785735.665768182] [realsense2_camera]: Device Name: Intel RealSense D435
[component_container_mt-1] [INFO] [1738785735.665773148] [realsense2_camera]: Device Serial No: 819112070232
[component_container_mt-1] [INFO] [1738785735.665777700] [realsense2_camera]: Device physical port: 3-7.2-71
[component_container_mt-1] [INFO] [1738785735.665782564] [realsense2_camera]: Device FW version: 5.16.0.1
[component_container_mt-1] [INFO] [1738785735.665786669] [realsense2_camera]: Device Product ID: 0x0B07
[component_container_mt-1] [INFO] [1738785735.665790873] [realsense2_camera]: Sync Mode: Off
[component_container_mt-1] [INFO] [1738785735.682216615] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1738785735.688518206] [centerpose_encoder.image_format_converter_node]: [NitrosPublisherSubscriberGroup] Pinning the component "sink/sink" (type="nvidia::isaac_ros::MessageRelay") to use its compatible format only: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1738785735.689820768] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Starting negotiation...
[component_container_mt-1] [INFO] [1738785735.690130881] [centerpose_encoder.resize_node]: Negotiating
[component_container_mt-1] [INFO] [1738785735.690157949] [isaac_ros_examples.container]: Load Library: /opt/ros/humble/lib/libtriton_node.so
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/centerpose_encoder/image_format_converter_node' in container '/isaac_ros_examples/container'
[component_container_mt-1] [INFO] [1738785735.693957170] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::TritonNode>
[component_container_mt-1] [INFO] [1738785735.693986982] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::TritonNode>
[component_container_mt-1] [INFO] [1738785735.696141529] [centerpose_inference]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1738785735.696391548] [centerpose_inference]: [TritonNode] Set input data format to: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1738785735.696402649] [centerpose_inference]: [TritonNode] Set output data format to: "nitros_tensor_list_nhwc_rgb_f32"
[component_container_mt-1] [INFO] [1738785735.696420690] [centerpose_inference]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1738785735.699378880] [centerpose_inference]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1738785735.699436757] [centerpose_inference]: [NitrosContext] Loading extension: gxf/lib/serialization/libgxf_serialization.so
[component_container_mt-1] [INFO] [1738785735.699894273] [centerpose_inference]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_triton.so
[component_container_mt-1] [INFO] [1738785735.721147343] [centerpose_inference]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1738785735.722439887] [centerpose_inference]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1738785735.804650052] [centerpose_inference]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1738785735.812539547] [centerpose_inference]: [NitrosPublisherSubscriberGroup] Pinning the component "triton_request/input" (type="nvidia::gxf::DoubleBufferReceiver") to use its compatible format only: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1738785735.814990672] [centerpose_inference]: [NitrosPublisherSubscriberGroup] Pinning the component "sink/sink" (type="nvidia::isaac_ros::MessageRelay") to use its compatible format only: "nitros_tensor_list_nhwc_rgb_f32"
[component_container_mt-1] [INFO] [1738785735.815287241] [centerpose_inference]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/centerpose_inference' in container '/isaac_ros_examples/container'
[component_container_mt-1] [INFO] [1738785735.815849090] [isaac_ros_examples.container]: Load Library: /opt/ros/humble/lib/libcrop_node.so
[component_container_mt-1] [INFO] [1738785735.817329153] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::CropNode>
[component_container_mt-1] [INFO] [1738785735.817350209] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::image_proc::CropNode>
[component_container_mt-1] [INFO] [1738785735.819674801] [centerpose_encoder.crop_node]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1738785735.820014277] [centerpose_encoder.crop_node]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1738785735.826460008] [centerpose_encoder.crop_node]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1738785735.826582101] [centerpose_encoder.crop_node]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1738785735.828734232] [centerpose_encoder.crop_node]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1738785735.859084633] [centerpose_encoder.crop_node]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1738785735.863224666] [centerpose_encoder.crop_node]: [NitrosNode] Starting negotiation...
[component_container_mt-1] [INFO] [1738785735.863646531] [centerpose_encoder.resize_node]: Negotiating
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/centerpose_encoder/crop_node' in container '/isaac_ros_examples/container'
[component_container_mt-1] [INFO] [1738785735.863695476] [centerpose_encoder.image_format_converter_node]: Negotiating
[component_container_mt-1] [INFO] [1738785735.863806759] [isaac_ros_examples.container]: Load Library: /opt/ros/humble/lib/libcenterpose_decoder_node.so
[component_container_mt-1] [INFO] [1738785735.863835532] [centerpose_encoder.resize_node]: Negotiating
[component_container_mt-1] [INFO] [1738785735.869952383] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::centerpose::CenterPoseDecoderNode>
[component_container_mt-1] [INFO] [1738785735.870007521] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::centerpose::CenterPoseDecoderNode>
[component_container_mt-1] [INFO] [1738785735.870377491] [centerpose_encoder.crop_node]: Negotiating
[component_container_mt-1] [INFO] [1738785735.870637829] [centerpose_encoder.crop_node]: Could not negotiate
[component_container_mt-1] [INFO] [1738785735.870652893] [centerpose_encoder.crop_node]: Negotiating
[component_container_mt-1] [INFO] [1738785735.870659053] [centerpose_encoder.crop_node]: Could not negotiate
[component_container_mt-1] [INFO] [1738785735.874349962] [centerpose_decoder_node]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1738785735.874618260] [centerpose_decoder_node]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1738785735.877098495] [centerpose_decoder_node]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1738785735.877180064] [centerpose_decoder_node]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_centerpose.so
[component_container_mt-1] [INFO] [1738785735.878584865] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_messages.so
[component_container_mt-1] [INFO] [1738785735.878955495] [centerpose_decoder_node]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1738785735.879960372] [centerpose_decoder_node]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1738785735.885926817] [centerpose_decoder_node]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1738785735.888933726] [centerpose_decoder_node]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/centerpose_decoder_node' in container '/isaac_ros_examples/container'
[component_container_mt-1] [INFO] [1738785735.889145804] [centerpose_inference]: Negotiating
[component_container_mt-1] [INFO] [1738785735.889176789] [centerpose_inference]: Could not negotiate
[component_container_mt-1] [INFO] [1738785735.889302102] [isaac_ros_examples.container]: Load Library: /opt/ros/humble/lib/libimage_to_tensor_node.so
[component_container_mt-1] [INFO] [1738785735.920820915] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::ImageToTensorNode>
[component_container_mt-1] [INFO] [1738785735.920878555] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::ImageToTensorNode>
[component_container_mt-1] [INFO] [1738785735.924378544] [centerpose_encoder.image_to_tensor.ManagedNitrosSubscriber]: Starting Managed Nitros Subscriber
[component_container_mt-1] [INFO] [1738785735.924485206] [centerpose_encoder.crop_node]: Negotiating
[component_container_mt-1] [INFO] [1738785735.924526665] [centerpose_encoder.resize_node]: Negotiating
[component_container_mt-1] [INFO] [1738785735.924550603] [centerpose_encoder.image_format_converter_node]: Negotiating
[component_container_mt-1] [INFO] [1738785735.925115558] [centerpose_encoder.image_to_tensor.ManagedNitrosPublisher]: Starting Managed Nitros Publisher
[component_container_mt-1] [INFO] [1738785736.015365116] [isaac_ros_examples.container]: Load Library: /opt/ros/humble/lib/libcenterpose_visualizer_node.so
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/centerpose_encoder/image_to_tensor' in container '/isaac_ros_examples/container'
[component_container_mt-1] [INFO] [1738785736.017027817] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::centerpose::CenterPoseVisualizerNode>
[component_container_mt-1] [INFO] [1738785736.017048141] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::centerpose::CenterPoseVisualizerNode>
[component_container_mt-1] [INFO] [1738785736.019796975] [centerpose_visualizer]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1738785736.020062071] [centerpose_visualizer]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1738785736.022067507] [centerpose_visualizer]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1738785736.022164318] [centerpose_visualizer]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1738785736.023062748] [centerpose_visualizer]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1738785736.026062197] [centerpose_visualizer]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1738785736.029085587] [centerpose_visualizer]: [NitrosNode] Starting negotiation...
[component_container_mt-1] [INFO] [1738785736.029473380] [centerpose_decoder_node]: Negotiating
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/centerpose_visualizer' in container '/isaac_ros_examples/container'
[component_container_mt-1] [INFO] [1738785736.029633276] [centerpose_inference]: Negotiating
[component_container_mt-1] [INFO] [1738785736.029657606] [centerpose_inference]: Could not negotiate
[component_container_mt-1] [INFO] [1738785736.029722264] [isaac_ros_examples.container]: Load Library: /opt/ros/humble/lib/libimage_tensor_normalize_node.so
[component_container_mt-1] [INFO] [1738785736.031056492] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::ImageTensorNormalizeNode>
[component_container_mt-1] [INFO] [1738785736.031076742] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::ImageTensorNormalizeNode>
[component_container_mt-1] [INFO] [1738785736.035115142] [centerpose_encoder.normalize_node.ManagedNitrosSubscriber]: Starting Managed Nitros Subscriber
[component_container_mt-1] [INFO] [1738785736.035298836] [centerpose_encoder.image_to_tensor]: Negotiating
[component_container_mt-1] [INFO] [1738785736.035319934] [centerpose_encoder.image_to_tensor]: Could not negotiate
[component_container_mt-1] [INFO] [1738785736.035626636] [centerpose_encoder.normalize_node.ManagedNitrosPublisher]: Starting Managed Nitros Publisher
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/centerpose_encoder/normalize_node' in container '/isaac_ros_examples/container'
[component_container_mt-1] [INFO] [1738785736.039090741] [isaac_ros_examples.container]: Load Library: /opt/ros/humble/lib/libinterleaved_to_planar_node.so
[component_container_mt-1] [INFO] [1738785736.040100027] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::InterleavedToPlanarNode>
[component_container_mt-1] [INFO] [1738785736.040129938] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::InterleavedToPlanarNode>
[component_container_mt-1] [INFO] [1738785736.043846879] [centerpose_encoder.interleaved_to_planar_node]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1738785736.044194307] [centerpose_encoder.interleaved_to_planar_node]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1738785736.047760240] [centerpose_encoder.interleaved_to_planar_node]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1738785736.047843639] [centerpose_encoder.interleaved_to_planar_node]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1738785736.048728549] [centerpose_encoder.interleaved_to_planar_node]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1738785736.051762286] [centerpose_encoder.interleaved_to_planar_node]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1738785736.053674966] [centerpose_encoder.interleaved_to_planar_node]: [NitrosNode] Starting negotiation...
[component_container_mt-1] [INFO] [1738785736.054110129] [centerpose_encoder.normalize_node]: Negotiating
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/centerpose_encoder/interleaved_to_planar_node' in container '/isaac_ros_examples/container'
[component_container_mt-1] [INFO] [1738785736.055850761] [isaac_ros_examples.container]: Load Library: /opt/ros/humble/lib/libreshape_node.so
[component_container_mt-1] [INFO] [1738785736.056767590] [isaac_ros_examples.container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::ReshapeNode>
[component_container_mt-1] [INFO] [1738785736.056792355] [isaac_ros_examples.container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::ReshapeNode>
[component_container_mt-1] [INFO] [1738785736.060127344] [centerpose_encoder.reshape_node]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1738785736.060437212] [centerpose_encoder.reshape_node]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1738785736.064043543] [centerpose_encoder.reshape_node]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1738785736.064106380] [centerpose_encoder.reshape_node]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1738785736.064973775] [centerpose_encoder.reshape_node]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1738785736.069785814] [centerpose_encoder.reshape_node]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1738785736.071857410] [centerpose_encoder.reshape_node]: [NitrosNode] Starting negotiation...
[component_container_mt-1] [INFO] [1738785736.072254515] [centerpose_encoder.interleaved_to_planar_node]: Negotiating
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/centerpose_encoder/reshape_node' in container '/isaac_ros_examples/container'
[component_container_mt-1] [INFO] [1738785736.072530956] [centerpose_encoder.normalize_node]: Negotiating
[component_container_mt-1] [INFO] [1738785736.142365334] [centerpose_encoder.reshape_node]: Negotiating
[component_container_mt-1] [INFO] [1738785736.142760505] [centerpose_encoder.interleaved_to_planar_node]: Negotiating
[component_container_mt-1] [INFO] [1738785736.142974975] [centerpose_encoder.normalize_node]: Negotiating
[component_container_mt-1] [INFO] [1738785736.215553853] [realsense2_camera]: Stopping Sensor: RGB Camera
[component_container_mt-1] [INFO] [1738785736.243613057] [realsense2_camera]: Starting Sensor: RGB Camera
[component_container_mt-1] [INFO] [1738785736.250909707] [realsense2_camera]: Open profile: stream_type: Color(0), Format: RGB8, Width: 640, Height: 480, FPS: 15
[component_container_mt-1] [INFO] [1738785736.251910778] [realsense2_camera]: RealSense Node Is Up!
[component_container_mt-1] [INFO] [1738785736.489519262] [centerpose_encoder.resize_node]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1738785736.489560925] [centerpose_encoder.resize_node]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1738785736.489571442] [centerpose_encoder.resize_node]: [NitrosPublisher] Use the negotiated data format: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1738785736.489579130] [centerpose_encoder.resize_node]: [NitrosPublisher] Use the negotiated data format: "nitros_camera_info"
[component_container_mt-1] [INFO] [1738785736.489583571] [centerpose_encoder.resize_node]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1738785736.489588597] [centerpose_encoder.resize_node]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/image_rect", data_format="nitros_image_rgb8"
[component_container_mt-1] [INFO] [1738785736.489593282] [centerpose_encoder.resize_node]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1738785736.489596600] [centerpose_encoder.resize_node]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/camera_info_rect", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1738785736.489723310] [centerpose_encoder.resize_node]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1738785736.496828313] [centerpose_encoder.resize_node]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/KRNWWPNPNO/KRNWWPNPNO.yaml"
[component_container_mt-1] [INFO] [1738785736.496885463] [centerpose_encoder.resize_node]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1738785736.499030373] [centerpose_encoder.resize_node]: [ResizeNode] postLoadGraphCallback().
[component_container_mt-1] [INFO] [1738785736.499103241] [centerpose_encoder.resize_node]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1738785736.508753777] [centerpose_encoder.resize_node]: [NitrosNode] Node was started
[component_container_mt-1] [INFO] [1738785736.690344583] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1738785736.690391894] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1738785736.690398475] [centerpose_encoder.image_format_converter_node]: [NitrosPublisher] Use the negotiated data format: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1738785736.690404989] [centerpose_encoder.image_format_converter_node]: [NitrosSubscriber] Use the negotiated data format: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1738785736.690478521] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1738785736.699641832] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/RQGWIZYAFR/RQGWIZYAFR.yaml"
[component_container_mt-1] [INFO] [1738785736.699710642] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1738785736.700790424] [centerpose_encoder.image_format_converter_node]: [ImageFormatConverterNode] postLoadGraphCallback().
[component_container_mt-1] [INFO] [1738785736.700907694] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1738785736.702002088] [centerpose_encoder.image_format_converter_node]: [NitrosNode] Node was started
[component_container_mt-1] [INFO] [1738785736.816037471] [centerpose_inference]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1738785736.816093360] [centerpose_inference]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1738785736.816103807] [centerpose_inference]: [NitrosSubscriber] Use the negotiated data format: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1738785736.816126084] [centerpose_inference]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1738785736.816132755] [centerpose_inference]: [NitrosPublisher] Use only the compatible publisher: topic_name="/tensor_sub", data_format="nitros_tensor_list_nhwc_rgb_f32"
[component_container_mt-1] [INFO] [1738785736.816226563] [centerpose_inference]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1738785736.829542586] [centerpose_inference]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/CPRCMKCRKV/CPRCMKCRKV.yaml"
[component_container_mt-1] [INFO] [1738785736.829592811] [centerpose_inference]: [NitrosNode] Loading application
[component_container_mt-1] 2025-02-05 15:02:16.830 WARN  gxf/std/yaml_file_loader.cpp@1076: Using unregistered parameter 'dummy_rx' in component 'requester'.
[component_container_mt-1] [INFO] [1738785736.830934337] [centerpose_inference]: In TritonNode postLoadGraphCallback().
[component_container_mt-1] [INFO] [1738785736.831075049] [centerpose_inference]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [WARN] [1738785736.831289423] [realsense2_camera]: 
[component_container_mt-1] [INFO] [1738785736.891855280] [centerpose_decoder_node]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1738785736.891910426] [centerpose_decoder_node]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1738785736.891902617] [centerpose_encoder.crop_node]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1738785736.891946140] [centerpose_encoder.crop_node]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1738785736.891919750] [centerpose_decoder_node]: [NitrosPublisher] Use the negotiated data format: "nitros_detection3_d_array"
[component_container_mt-1] [INFO] [1738785736.891952660] [centerpose_encoder.crop_node]: [NitrosPublisher] Use the negotiated data format: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1738785736.891969797] [centerpose_encoder.crop_node]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1738785736.891974719] [centerpose_encoder.crop_node]: [NitrosPublisher] Use only the compatible publisher: topic_name="/centerpose_encoder/crop/camera_info", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1738785736.891981127] [centerpose_encoder.crop_node]: [NitrosSubscriber] Use the negotiated data format: "nitros_image_rgb8"
[component_container_mt-1] [INFO] [1738785736.891963293] [centerpose_decoder_node]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1738785736.891993092] [centerpose_decoder_node]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/tensor_sub", data_format="nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1738785736.891997665] [centerpose_decoder_node]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1738785736.892001171] [centerpose_decoder_node]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/camera_info_rect", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1738785736.892007960] [centerpose_encoder.crop_node]: [NitrosSubscriber] Use the negotiated data format: "nitros_camera_info"
[component_container_mt-1] [INFO] [1738785736.892085471] [centerpose_decoder_node]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1738785736.892121890] [centerpose_encoder.crop_node]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1738785736.893945907] [centerpose_decoder_node]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/ZGFMSLTZUS/ZGFMSLTZUS.yaml"
[component_container_mt-1] [INFO] [1738785736.893972164] [centerpose_decoder_node]: [NitrosNode] Loading application
[component_container_mt-1] WARNING: infer_trtis_server.cpp:1217 NvDsTritonServerInit suggest to set model_control_mode:none. otherwise may cause unknow issues.
[component_container_mt-1] I0205 20:02:16.898656 15752 pinned_memory_manager.cc:277] "Pinned memory pool is created at '0x72e5ca000000' with size 268435456"
[component_container_mt-1] I0205 20:02:16.904303 15752 cuda_memory_manager.cc:107] "CUDA memory pool is created on device 0 with size 67108864"
[component_container_mt-1] I0205 20:02:16.908482 15752 server.cc:604] 
[component_container_mt-1] +------------------+------+
[component_container_mt-1] | Repository Agent | Path |
[component_container_mt-1] +------------------+------+
[component_container_mt-1] +------------------+------+
[component_container_mt-1] 
[component_container_mt-1] I0205 20:02:16.908510 15752 server.cc:631] 
[component_container_mt-1] +---------+------+--------+
[component_container_mt-1] | Backend | Path | Config |
[component_container_mt-1] +---------+------+--------+
[component_container_mt-1] +---------+------+--------+
[component_container_mt-1] 
[component_container_mt-1] I0205 20:02:16.908522 15752 server.cc:674] 
[component_container_mt-1] +-------+---------+--------+
[component_container_mt-1] | Model | Version | Status |
[component_container_mt-1] +-------+---------+--------+
[component_container_mt-1] +-------+---------+--------+
[component_container_mt-1] 
[component_container_mt-1] [INFO] [1738785736.918271209] [centerpose_encoder.crop_node]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/HIPQOHKPWZ/HIPQOHKPWZ.yaml"
[component_container_mt-1] [INFO] [1738785736.918316637] [centerpose_encoder.crop_node]: [NitrosNode] Loading application
[component_container_mt-1] I0205 20:02:16.935130 15752 metrics.cc:877] "Collecting metrics for GPU 0: NVIDIA GeForce RTX 3080 Laptop GPU"
[component_container_mt-1] I0205 20:02:16.938398 15752 metrics.cc:770] "Collecting CPU metrics"
[component_container_mt-1] I0205 20:02:16.938471 15752 tritonserver.cc:2598] 
[component_container_mt-1] +----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[component_container_mt-1] | Option                           | Value                                                                                                                                                                                                           |
[component_container_mt-1] +----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[component_container_mt-1] | server_id                        | triton                                                                                                                                                                                                          |
[component_container_mt-1] | server_version                   | 2.49.0                                                                                                                                                                                                          |
[component_container_mt-1] | server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data parameters statistics trace logging |
[component_container_mt-1] | model_repository_path[0]         | /workspaces/isaac_ros-dev/isaac_ros_assets/models                                                                                                                                                               |
[component_container_mt-1] | model_control_mode               | MODE_EXPLICIT                                                                                                                                                                                                   |
[component_container_mt-1] | strict_model_config              | 1                                                                                                                                                                                                               |
[component_container_mt-1] | model_config_name                |                                                                                                                                                                                                                 |
[component_container_mt-1] | rate_limit                       | OFF                                                                                                                                                                                                             |
[component_container_mt-1] | pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                                       |
[component_container_mt-1] | cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                                        |
[component_container_mt-1] | min_supported_compute_capability | 6.0                                                                                                                                                                                                             |
[component_container_mt-1] | strict_readiness                 | 1                                                                                                                                                                                                               |
[component_container_mt-1] | exit_timeout                     | 30                                                                                                                                                                                                              |
[component_container_mt-1] | cache_enabled                    | 0                                                                                                                                                                                                               |
[component_container_mt-1] +----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[component_container_mt-1] 
[component_container_mt-1] [INFO] [1738785736.938723305] [centerpose_inference]: [NitrosNode] Node was started
[component_container_mt-1] I0205 20:02:16.940221 15752 model_lifecycle.cc:472] "loading: centerpose_shoe:1"
[component_container_mt-1] [INFO] [1738785736.941523841] [centerpose_encoder.crop_node]: [CropNode] postLoadGraphCallback().
[component_container_mt-1] [INFO] [1738785736.941547518] [centerpose_decoder_node]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1738785736.943392350] [centerpose_decoder_node]: [NitrosNode] Node was started
[component_container_mt-1] [INFO] [1738785736.943421419] [centerpose_encoder.crop_node]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1738785736.944911060] [centerpose_encoder.crop_node]: [NitrosNode] Node was started
[component_container_mt-1] I0205 20:02:16.944909 15752 tensorrt.cc:65] "TRITONBACKEND_Initialize: tensorrt"
[component_container_mt-1] I0205 20:02:16.944927 15752 tensorrt.cc:75] "Triton TRITONBACKEND API version: 1.19"
[component_container_mt-1] I0205 20:02:16.944930 15752 tensorrt.cc:81] "'tensorrt' TRITONBACKEND API version: 1.19"
[component_container_mt-1] I0205 20:02:16.944933 15752 tensorrt.cc:105] "backend configuration:\n{\"cmdline\":{\"auto-complete-config\":\"false\",\"backend-directory\":\"/opt/tritonserver/backends\",\"min-compute-capability\":\"6.000000\",\"default-max-batch-size\":\"4\"}}"
[component_container_mt-1] I0205 20:02:16.947854 15752 tensorrt.cc:231] "TRITONBACKEND_ModelInitialize: centerpose_shoe (version 1)"
[component_container_mt-1] I0205 20:02:16.955743 15752 tensorrt.cc:297] "TRITONBACKEND_ModelInstanceInitialize: centerpose_shoe_0 (GPU device 0)"
[component_container_mt-1] [INFO] [1738785737.030098125] [centerpose_visualizer]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1738785737.030164715] [centerpose_visualizer]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1738785737.030174478] [centerpose_visualizer]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1738785737.030182663] [centerpose_visualizer]: [NitrosPublisher] Use only the compatible publisher: topic_name="/centerpose/image_visualized", data_format="nitros_image_bgr8"
[component_container_mt-1] [INFO] [1738785737.030192671] [centerpose_visualizer]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1738785737.030199148] [centerpose_visualizer]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/image_rect", data_format="nitros_image_bgr8"
[component_container_mt-1] [INFO] [1738785737.030207309] [centerpose_visualizer]: [NitrosSubscriber] Use the negotiated data format: "nitros_detection3_d_array"
[component_container_mt-1] [INFO] [1738785737.030214816] [centerpose_visualizer]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1738785737.030220635] [centerpose_visualizer]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/camera_info_rect", data_format="nitros_camera_info"
[component_container_mt-1] [INFO] [1738785737.030358905] [centerpose_visualizer]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1738785737.032079062] [centerpose_visualizer]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/WSACOOUNJK/WSACOOUNJK.yaml"
[component_container_mt-1] [INFO] [1738785737.032116444] [centerpose_visualizer]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1738785737.033311609] [centerpose_visualizer]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1738785737.033851228] [centerpose_visualizer]: [NitrosNode] Node was started
[component_container_mt-1] [INFO] [1738785737.054456564] [centerpose_encoder.interleaved_to_planar_node]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1738785737.054509004] [centerpose_encoder.interleaved_to_planar_node]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1738785737.054517074] [centerpose_encoder.interleaved_to_planar_node]: [NitrosPublisher] Use the negotiated data format: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1738785737.054525270] [centerpose_encoder.interleaved_to_planar_node]: [NitrosSubscriber] Use the negotiated data format: "nitros_tensor_list_nhwc_rgb_f32"
[component_container_mt-1] [INFO] [1738785737.054624360] [centerpose_encoder.interleaved_to_planar_node]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1738785737.056496701] [centerpose_encoder.interleaved_to_planar_node]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/IDHWWZPFGG/IDHWWZPFGG.yaml"
[component_container_mt-1] [INFO] [1738785737.056525893] [centerpose_encoder.interleaved_to_planar_node]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1738785737.057421803] [centerpose_encoder.interleaved_to_planar_node]: In InterleavedToPlanarNode postLoadGraphCallback().
[component_container_mt-1] [INFO] [1738785737.057461042] [centerpose_encoder.interleaved_to_planar_node]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1738785737.059002891] [centerpose_encoder.interleaved_to_planar_node]: [NitrosNode] Node was started
[component_container_mt-1] I0205 20:02:17.061536 15752 logging.cc:46] "Loaded engine size: 127 MiB"
[component_container_mt-1] [INFO] [1738785737.072468795] [centerpose_encoder.reshape_node]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1738785737.072506593] [centerpose_encoder.reshape_node]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1738785737.072513563] [centerpose_encoder.reshape_node]: [NitrosPublisher] Use the negotiated data format: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1738785737.072521362] [centerpose_encoder.reshape_node]: [NitrosSubscriber] Use the negotiated data format: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1738785737.072625028] [centerpose_encoder.reshape_node]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1738785737.075541581] [centerpose_encoder.reshape_node]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/GSANYZBSTL/GSANYZBSTL.yaml"
[component_container_mt-1] [INFO] [1738785737.075576365] [centerpose_encoder.reshape_node]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1738785737.076710805] [centerpose_encoder.reshape_node]: In ReshapeNode postLoadGraphCallback().
[component_container_mt-1] [INFO] [1738785737.076759566] [centerpose_encoder.reshape_node]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1738785737.096123051] [centerpose_encoder.reshape_node]: [NitrosNode] Node was started
[component_container_mt-1] I0205 20:02:17.132633 15752 logging.cc:46] "[MS] Running engine with multi stream info"
[component_container_mt-1] I0205 20:02:17.132657 15752 logging.cc:46] "[MS] Number of aux streams is 6"
[component_container_mt-1] I0205 20:02:17.132660 15752 logging.cc:46] "[MS] Number of total worker streams is 7"
[component_container_mt-1] I0205 20:02:17.132662 15752 logging.cc:46] "[MS] The main stream provided by execute/enqueue calls is the first worker stream"
[component_container_mt-1] I0205 20:02:17.149633 15752 logging.cc:46] "[MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +246, now: CPU 0, GPU 369 (MiB)"
[component_container_mt-1] I0205 20:02:17.149987 15752 instance_state.cc:186] "Created instance centerpose_shoe_0 on GPU 0 with stream priority 0 and optimization profile default[0];"
[component_container_mt-1] I0205 20:02:17.150137 15752 model_lifecycle.cc:839] "successfully loaded 'centerpose_shoe'"
[component_container_mt-1] INFO: infer_simple_runtime.cpp:71 TrtISBackend id:118 initialized model: centerpose_shoe
[component_container_mt-1] 2025-02-05 15:02:17.220 WARN  extensions/triton/inferencers/triton_inferencer_impl.cpp@508: Incomplete inference; response appeared out of order; invalid: index = 2
[component_container_mt-1] 2025-02-05 15:02:17.220 WARN  extensions/triton/inferencers/triton_inferencer_impl.cpp@510: Inference appeared out of order
[component_container_mt-1] W0205 20:02:17.939795 15752 metrics.cc:631] "Unable to get power limit for GPU 0. Status:Success, value:0.000000"
[component_container_mt-1] W0205 20:02:18.946881 15752 metrics.cc:631] "Unable to get power limit for GPU 0. Status:Success, value:0.000000"
[component_container_mt-1] W0205 20:02:19.947213 15752 metrics.cc:631] "Unable to get power limit for GPU 0. Status:Success, value:0.000000"

2025-02-05 15:04:30 [mona@DOS isaac_ros_common]$  nvidia-smi -q -d POWER

==============NVSMI LOG==============

Timestamp                                 : Wed Feb  5 15:04:31 2025
Driver Version                            : 570.86.15
CUDA Version                              : 12.8

Attached GPUs                             : 1
GPU 00000000:01:00.0
    GPU Power Readings
        Power Draw                        : 41.62 W
        Current Power Limit               : 90.00 W
        Requested Power Limit             : 90.00 W
        Default Power Limit               : 90.00 W
        Min Power Limit                   : 1.00 W
        Max Power Limit                   : 110.00 W
    Power Samples
        Duration                          : Not Found
        Number of Samples                 : Not Found
        Max                               : Not Found
        Min                               : Not Found
        Avg                               : Not Found
    GPU Memory Power Readings 
        Power Draw                        : N/A
    Module Power Readings
        Power Draw                        : N/A
        Current Power Limit               : N/A
        Requested Power Limit             : N/A
        Default Power Limit               : N/A
        Min Power Limit                   : N/A
        Max Power Limit                   : N/A

I am following this part of isaac_ros_centerpose tutorial:

https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_pose_estimation/isaac_ros_centerpose/index.html#quickstart

I also see this part of log above and I am not sure if that should be taken care of as well?

[component_container_mt-1] [INFO] [1738785736.029633276] [centerpose_inference]: Negotiating
[component_container_mt-1] [INFO] [1738785736.029657606] [centerpose_inference]: Could not negotiate

mona.jalal · February 5, 2025, 8:12pm

I may have been quick to judge since after being stuck at that message, now it errs wrt realsense:

[component_container_mt-1] W0205 20:02:17.939795 15752 metrics.cc:631] "Unable to get power limit for GPU 0. Status:Success, value:0.000000"
[component_container_mt-1] W0205 20:02:18.946881 15752 metrics.cc:631] "Unable to get power limit for GPU 0. Status:Success, value:0.000000"
[component_container_mt-1] W0205 20:02:19.947213 15752 metrics.cc:631] "Unable to get power limit for GPU 0. Status:Success, value:0.000000"
[component_container_mt-1]  05/02 15:06:15,897 WARNING [126339212379712] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/02 15:06:15,908 WARNING [126339212379712] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/02 15:06:15,918 WARNING [126339212379712] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/02 15:06:15,928 WARNING [126339212379712] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/02 15:06:15,938 WARNING [126339212379712] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/02 15:06:15,949 WARNING [126339212379712] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/02 15:06:15,959 WARNING [126339212379712] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/02 15:06:15,969 WARNING [126339212379712] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/02 15:06:15,980 WARNING [126339212379712] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/02 15:06:15,990 WARNING [126339212379712] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/02 15:06:16,000 WARNING [126339212379712] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/02 15:06:16,010 WARNING [126339212379712] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/02 15:06:16,021 WARNING [126339212379712] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/02 15:06:16,031 WARNING [126339212379712] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/02 15:06:16,041 WARNING [126339212379712] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/02 15:06:16,052 WARNING [126339212379712] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  05/02 15:06:16,062 WARNING [126334850303552] (messenger-libusb.cpp:92) usb_request_queue returned error, endpoint: 132 error: No such device, number: 19
[component_container_mt-1]  05/02 15:06:16,062 ERROR [126334850303552] (uvc-streamer.cpp:137) failed to submit UVC request, error: -13
[component_container_mt-1]  05/02 15:06:16,062 WARNING [126334850303552] (messenger-libusb.cpp:92) usb_request_queue returned error, endpoint: 132 error: No such device, number: 19
[component_container_mt-1]  05/02 15:06:16,062 ERROR [126334850303552] (uvc-streamer.cpp:137) failed to submit UVC request, error: -13
[component_container_mt-1]  05/02 15:06:16,062 WARNING [126339212379712] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No such device, number: 19
[component_container_mt-1]  05/02 15:06:16,062 WARNING [126339212379712] (messenger-libusb.cpp:92) usb_request_queue returned error, endpoint: 135 error: No such device, number: 19
[component_container_mt-1]  05/02 15:06:16,063 ERROR [126339371763264] (global_timestamp_reader.cpp:239) Error during time_diff_keeper polling: usb device disconnected
[component_container_mt-1] [ERROR] [1738785976.224379896] [realsense2_camera]: The device has been disconnected!
[component_container_mt-1] [INFO] [1738785976.224516129] [realsense2_camera]: Stop Sensor: RGB Camera
[component_container_mt-1] [INFO] [1738785976.224545084] [realsense2_camera]: Close Sensor. 
[component_container_mt-1]  05/02 15:06:16,702 ERROR [126334850303552] (uvc-streamer.cpp:105) uvc streamer watchdog triggered on endpoint: 132
[component_container_mt-1]  05/02 15:06:16,702 WARNING [126334850303552] (messenger-libusb.cpp:30) reset_endpoint returned error, index: 132, error: No such device, number: 19
[component_container_mt-1]  05/02 15:06:17,368 WARNING [126334850303552] (messenger-libusb.cpp:30) reset_endpoint returned error, index: 132, error: No such device, number: 19
[component_container_mt-1] [INFO] [1738785977.374540136] [realsense2_camera]: Close Sensor - Done. 
[component_container_mt-1] [INFO] [1738785979.381251147] [realsense2_camera]: Checking new devices...
[component_container_mt-1] [INFO] [1738785979.420696787] [realsense2_camera]: Device with serial number 819112070232 was found.
[component_container_mt-1] 
[component_container_mt-1] [INFO] [1738785979.420751169] [realsense2_camera]: Device with physical ID 3-7.2-72 was found.
[component_container_mt-1] [INFO] [1738785979.420756726] [realsense2_camera]: Device with name Intel RealSense D435 was found.
[component_container_mt-1] [INFO] [1738785979.420911951] [realsense2_camera]: Device with port number 3-7.2 was found.
[component_container_mt-1] [INFO] [1738785979.420920498] [realsense2_camera]: Device USB type: 2.1
[component_container_mt-1] [WARN] [1738785979.420925305] [realsense2_camera]: Device 819112070232 is connected using a 2.1 port. Reduced performance is expected.
[component_container_mt-1] [INFO] [1738785979.421435227] [realsense2_camera]: getParameters...
[component_container_mt-1] [INFO] [1738785979.421726078] [realsense2_camera]: JSON file is not provided
[component_container_mt-1] [INFO] [1738785979.421736849] [realsense2_camera]: Device Name: Intel RealSense D435
[component_container_mt-1] [INFO] [1738785979.421741688] [realsense2_camera]: Device Serial No: 819112070232
[component_container_mt-1] [INFO] [1738785979.421745604] [realsense2_camera]: Device physical port: 3-7.2-72
[component_container_mt-1] [INFO] [1738785979.421749646] [realsense2_camera]: Device FW version: 5.16.0.1
[component_container_mt-1] [INFO] [1738785979.421753445] [realsense2_camera]: Device Product ID: 0x0B07
[component_container_mt-1] [INFO] [1738785979.421757004] [realsense2_camera]: Sync Mode: Off
[component_container_mt-1] [INFO] [1738785979.916215554] [realsense2_camera]: Stopping Sensor: RGB Camera
[component_container_mt-1] [INFO] [1738785979.932368031] [realsense2_camera]: Starting Sensor: RGB Camera
[component_container_mt-1] [INFO] [1738785979.939069908] [realsense2_camera]: Open profile: stream_type: Color(0), Format: RGB8, Width: 640, Height: 480, FPS: 15
[component_container_mt-1] [INFO] [1738785979.939767284] [realsense2_camera]: RealSense Node Is Up!
[component_container_mt-1]  05/02 15:09:23,972 WARNING [126334850303552] (messenger-libusb.cpp:92) usb_request_queue returned error, endpoint: 132 error: No such device, number: 19
[component_container_mt-1]  05/02 15:09:23,972 ERROR [126334850303552] (uvc-streamer.cpp:137) failed to submit UVC request, error: -13
[component_container_mt-1]  05/02 15:09:23,972 WARNING [126334850303552] (messenger-libusb.cpp:92) usb_request_queue returned error, endpoint: 132 error: No such device, number: 19
[component_container_mt-1]  05/02 15:09:23,972 ERROR [126334850303552] (uvc-streamer.cpp:137) failed to submit UVC request, error: -13
[component_container_mt-1] [ERROR] [1738786164.443042708] [realsense2_camera]: The device has been disconnected!
[component_container_mt-1] [INFO] [1738786164.443153974] [realsense2_camera]: Stop Sensor: RGB Camera
[component_container_mt-1] [INFO] [1738786164.443166274] [realsense2_camera]: Close Sensor. 
[component_container_mt-1]  05/02 15:09:25,105 WARNING [126334850303552] (messenger-libusb.cpp:30) reset_endpoint returned error, index: 132, error: No such device, number: 19
[component_container_mt-1] [INFO] [1738786165.138490679] [realsense2_camera]: Close Sensor - Done. 
[component_container_mt-1] [INFO] [1738786165.139308642] [realsense2_camera]: Checking new devices...
[component_container_mt-1] [INFO] [1738786165.190225027] [realsense2_camera]: Device with serial number 819112070232 was found.
[component_container_mt-1] 
[component_container_mt-1] [INFO] [1738786165.190278798] [realsense2_camera]: Device with physical ID 3-7.2-73 was found.
[component_container_mt-1] [INFO] [1738786165.190283631] [realsense2_camera]: Device with name Intel RealSense D435 was found.
[component_container_mt-1] [INFO] [1738786165.190423924] [realsense2_camera]: Device with port number 3-7.2 was found.
[component_container_mt-1] [INFO] [1738786165.190444274] [realsense2_camera]: Device USB type: 2.1
[component_container_mt-1] [WARN] [1738786165.190449576] [realsense2_camera]: Device 819112070232 is connected using a 2.1 port. Reduced performance is expected.
[component_container_mt-1] [INFO] [1738786165.191008478] [realsense2_camera]: getParameters...
[component_container_mt-1] [INFO] [1738786165.191379479] [realsense2_camera]: JSON file is not provided
[component_container_mt-1] [INFO] [1738786165.191395648] [realsense2_camera]: Device Name: Intel RealSense D435
[component_container_mt-1] [INFO] [1738786165.191400540] [realsense2_camera]: Device Serial No: 819112070232
[component_container_mt-1] [INFO] [1738786165.191404660] [realsense2_camera]: Device physical port: 3-7.2-73
[component_container_mt-1] [INFO] [1738786165.191408514] [realsense2_camera]: Device FW version: 5.16.0.1
[component_container_mt-1] [INFO] [1738786165.191412431] [realsense2_camera]: Device Product ID: 0x0B07
[component_container_mt-1] [INFO] [1738786165.191416478] [realsense2_camera]: Sync Mode: Off
[component_container_mt-1] [INFO] [1738786165.738131577] [realsense2_camera]: Stopping Sensor: RGB Camera
[component_container_mt-1] [INFO] [1738786165.753555459] [realsense2_camera]: Starting Sensor: RGB Camera
[component_container_mt-1] [INFO] [1738786165.758999010] [realsense2_camera]: Open profile: stream_type: Color(0), Format: RGB8, Width: 640, Height: 480, FPS: 15
[component_container_mt-1] [INFO] [1738786165.760339843] [realsense2_camera]: RealSense Node Is Up!

mona.jalal · February 6, 2025, 9:25pm

@Raffaello Hi Raffaello, I followed your instructions and have cuda 12.8, and docker 27, ubuntu 22.04, not sure why this step didn’t work. Could you please have a look?

mona.jalal · February 7, 2025, 7:19pm

I redid all steps and it works.

system · February 21, 2025, 7:19pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
/Centerpose/image-visualized no inference image display Isaac ROS	1	24	February 11, 2025
Isaac_ros_centerpose on a trained model for custom object usin TAO toolkit and data from isaac sim yields no inference Isaac ROS camera , ros , docker , isaac-ros-pose-estimation , ros-2-humble	7	35	February 12, 2025
No inference results for custom object using isaac_ros_centerpose - Compression requires single-channel 32bit-floating point or 16bit raw depth images Isaac ROS	2	24	February 12, 2025
Isaac_ros_foundationpose tracking takes approximately 20 seconds to show pose detection results upon starting rviz Isaac ROS camera , mesh , isaac-ros-object-detection , isaac-ros-pose-estimation	3	74	March 18, 2025
Running Isaac ROS Packages with Intel RealSense D457 Isaac ROS	2	159	August 29, 2024
Isacc_ros_centerpose Isaac ROS	9	65	January 10, 2025
Isaac_ros_ess Visualization Problem Isaac ROS camera	10	156	November 1, 2024
An Three error occurred during Foundationpose Quickstart Isaac ROS	18	383	September 1, 2024
Isaac_ros_visual_slam Quickstart - No PCL - ubuntu 22 x86, RS 435i Isaac ROS	1	166	June 24, 2024
Jetson Orin Nano with Intel Realsense D455--isaac_ros_visual_slam problem Isaac ROS jetson , isaac-ros-vslam	3	101	November 8, 2024

Isaac_ros_centerpose inference with Intel RealSense D435 stuck at "Unable to get power limit for GPU 0"

Related topics