Memory overwritten between buffers in DeepStream 5.0 stream

• Hardware Platform: TX2
**• DeepStream 5 developer preview **
• JetPack Version 4.4

Since upgrading my system from Jetpack 4.3 DeepStream 4.0 to Jetpack 4.4 DeepStream 5.0 I am getting corrupt buffers. What I see is horizontal bands in many frames are replaced with sections of other frames. In other words, some number of rows of a buffer are overwritten by a few hundred rows of another buffer.

This only happens with deepstream elements in my pipeline, and only with Deepstream 5.0. It actually affects a tee that is BEFORE the deepstream elements. If I had to guess it seems that deepstream elements are overwriting data in gstreamer buffers allocated to adjacent addresses.

My pipeline is below. It works perfectly fine with DeepStream 4.0. If I remove the nvvideoconvert/nvinfer/etc elements there are no messed up buffers. With DeepStream 5.0 I get images with corrupted/partially ovewritten buffers in the first filesink tee’d before the deepstream elements, as well as the last filesink after.

The nvinfer element is using the Deepstream 5.0 YOLO implementation for YoloV3-Tiny. For Deepstream 4.0, I use the corresponding YOLO implementation from the DeepStream 4.0 source.

gst-launch-1.0 aravissrc ! bayer2rgb ! tee name=t ! \
queue leaky=downstream max-size-time=500000000 max-size-bytes=62914560 max-size-buffers=200 ! \
jpegenc ! multifilesink location="$IMAGE_DIR/%06d.jpg" t. ! videoflip method=rotate-180 ! \
nvvideoconvert ! "video/x-raw(memory:NVMM),width=(int)1296,height=(int)1024,format=NV12" ! \
m.sink_0 nvstreammux name=m batch-size=1 width=1296 height=1024 live-source=true ! \
nvinfer config-file-path=config_infer_primary_yolov3_tiny.txt batch-size=1 unique-id=1 ! \
nvvideoconvert ! nvdsosd ! nvvideoconvert ! \
videoconvert ! "video/x-raw, format=I420, width=1296, height=1024" ! videoscale ! \
"video/x-raw, width=648, height=512" ! queue leaky=downstream max-size-time=500000000 max-size-bytes=62914560 max-size-buffers=200 ! \
jpegenc quality=50 ! multifilesink location="/www/latest.jpg" max-files=1

Hi
I am trying to repro your issue, using below command,
gst-launch-1.0 videotestsrc ! ‘video/x-raw, format=(string)BGRx, width=(int)1280, height=(int)720’! tee name=t ! queue leaky=downstream max-size-time=500000000 max-size-bytes=62914560 max-size-buffers=200 ! jpegenc ! multifilesink location="/home/nvidia/tmp/%06d.jpg" t. ! videoflip method=rotate-180 ! nvvideoconvert ! “video/x-raw(memory:NVMM),width=(int)1296,height=(int)1024,format=NV12” ! m.sink_0 nvstreammux name=m batch-size=1 width=1296 height=1024 live-source=true ! nvinfer config-file-path=config_infer_primary.txt batch-size=1 unique-id=1 ! nvvideoconvert ! nvdsosd ! nvvideoconvert ! videoconvert ! “video/x-raw, format=I420, width=1296, height=1024” ! videoscale ! “video/x-raw, width=648, height=512” ! queue leaky=downstream max-size-time=500000000 max-size-bytes=62914560 max-size-buffers=200 ! jpegenc quality=50 ! multifilesink location="/home/nvidia/latest.jpg" max-files=1
difference is input source use videotestsrc, but trying to similar as yours, I see you convert bayer foramat to rgb format, so I use BGRx format for videotestsrc in order to have coexist caps with jpegenc and nvvideoconvert elements. but I can not repro your issue, pasted one picture from after first tee, and latest.jpg for the final file sink.
I am not sure if model implementation have effect on this, i use primary model builtin of deepstream, i will try with same model with you, meanwhile, can you use primary model to try to see if you have this issue?
000009.jpg

latest.jpg

I also can not repro your issue with yolov3tiny model
gst-launch-1.0 videotestsrc ! ‘video/x-raw, format=(string)BGRx, width=(int)1280, height=(int)720’! tee name=t ! queue leaky=downstream max-size-time=500000000 max-size-bytes=62914560 max-size-buffers=200 ! jpegenc ! multifilesink location="/home/nvidia/tmp/%06d.jpg" t. ! videoflip method=rotate-180 ! nvvideoconvert ! “video/x-raw(memory:NVMM),width=(int)1296,height=(int)1024,format=NV12” ! m.sink_0 nvstreammux name=m batch-size=1 width=1296 height=1024 live-source=true ! nvinfer config-file-path=config_infer_primary_yoloV3_tiny.txt batch-size=1 unique-id=1 ! nvvideoconvert ! nvdsosd ! nvvideoconvert ! videoconvert ! “video/x-raw, format=I420, width=1296, height=1024” ! videoscale ! “video/x-raw, width=648, height=512” ! queue leaky=downstream max-size-time=500000000 max-size-bytes=62914560 max-size-buffers=200 ! jpegenc quality=50 ! multifilesink location="/home/nvidia/latest.jpg" max-files=1

014242.jpg

latest.jpg

1 Like

Hi
Our Eng team is looking into it, but please start with a fresh setup to rule out any setup issues.

May be aravissrc related:

What happens is you use another source?

Confirmed the root cause is aravissrc. It is only happening with DeepStream 5.0 because my pipeline has an additional ~500 ms of latency with DS 5.0 compared to 4.0, and thus has more buffers in the pipeline… which exposes the bug that mdegans linked to above. I’ve managed to reproduce the aravissrc bug without DeepStream.

I think my stream is also affected by this bug, so it made diagnosing the problem difficult: Migrated my app to DS5.0 preview and recorded mp4 files behaving odd

Thanks all for the help!