Freezes when using ESS(sgm) + yolov8

Hi, I’m trying to use the ess+yolov8n bundle, and the frame source is from the imx219 camera.

The problem arises at the stage when I try to subscribe to the rgb image topic, to the topic with the detection results and the disparity topic:

def callback(self, left_image_msg: Image, disp_msg: DisparityImage, detections_msg: Detection2DArray):

After processing, calculating the distance to the detected object, drawing the frame and labeling the frames, the output of this frame is not stable, freezes are observed. But freezes occur even when I subscribe only to the disparity topic, while detection also works, but if detection is disabled, freezes in disparity topics stop.

At the same time, it does not matter which method of calculating disparity ess or sgm (from isaac ros) freezes still remain.

I’ve tried this on Orin Nano (8 GB) and Orin NX(16GB) with no changes.

Video demonstration

What could it be and how to fix it?

Hi @Nikolay196

Could you please confirm if you are using the combined Isaac ROS packages to assist me in better understanding your testing environment?

If you are using only one imx219 camera, the output of Isaac ROS ess will be incorrect as it requires synchronized input from a stereo camera.

Best,
Raffaello

Hi, @Raffaello

That’s right, I use the above packages plus, I tried using isaac_ros_stereo_image_proc instead of Isaac ROS DNN Stereo Depth

I use an Imx219-83 stereo camera https://www.waveshare.com/wiki/IMX219-83_Stereo_Camera

They are not synchronized, but in order to work with Isaac ROS DNN Stereo Depth and isaac_ros_stereo_image_proc, I equated the message time from the left frame to the message from the right frame (in reality, there is a 20-30 ms difference between frames).

Freezes occur when I subscribe to the topics /disparity and /detections_output in one node, if separately, then there is no frieze in /disparity (DNN Stereo Depth or isaac_ros_stereo_image_proc)

Addition. @Raffaello

I also tried using a pre-recorded video as a frame source, the result is exactly the same

The images must be synchronized to make the ESS work properly.
The documentation report: “The Stereo disparity is calculated from a time-synchronized image pair sourced from a stereo camera and is used to produce a depth image or a point cloud for a scene.”

If this issue is fixed, the output will be stable again.

Let me know if fix your bug.

Best,
Raffaello

@Raffaello

I tried using a synchronized frame source (realsense D435i camera), the breaks in the stream are still the same.

I am attaching a launch file and a script that I use for visualization, maybe I am missing something…
https://drive.google.com/drive/folders/1L0o-DUbYC2_sNvxXqew-hzqLQdF4mwxt?usp=sharing

terminal 1:

ros2 launch <package_name> realsense_yolov8_sgm.launch.py model_file_path:=<path_to_model>.onnx engine_file_path:=<path_to_engine>.plan

terminal 2:
It will show a window with a demonstration

./demo_to_nvidia_full.py --source=realsense --engine=sgm

You’re pulling images from an IMX219-83 stereo camera, running just the left frame through an image-space object detection from YOLOv8 and the stereo pair through disparity where one (Python?) node listens to the detections and disparity output, correct? You can get the depth calculated for you from the disparity rather than consuming the raw disparity, by the way. If you’re looking for depth-to-object, though, an image segmentation model instead of object detection (per-pixel labels) may be more effective to correlate the depth at pixel, for example.

That aside, if you’re faking the timestamps to match between the left and right imagers of your IMX219-83 stereo camera to appear like they are synchronized, SGM and ESS should both output disparity but it will have severely degraded quality.

As far as the freezing, is it that you’re final node subscribing to the disparity and detection is never receiving anything or you start to see the disparity message fall behind and then effectively stall out? What frame rate are you running the camera at? Are you seeing the disparity topic always empty or there are a few messages but the lag between them keeps increasing until it looks like there isn’t any new disparity image being published? What does jtop read for system resource use at this time? If you’re using Realsense and its depth channel, disparity wouldn’t even be calculated on your Orin, so it indicates there could be something else going on if you are still seeing any sort of freezing still.