Wrong frame order when decoding the stream on GPU

Accelerated Computing Intelligent Video Analytics DeepStream SDK
#1

We are getting wrong frame order on the decoding streams on GPU. On CPU the order seems to be OK.

Nvidia Driver: 418.87
Deepstream: 3.0
Video SDK: 9.0.20
Deepstreamer: 1.8.3

Command:
gst-launch-1.0 nvstreammux name=mux width=544 height=544 batch-size=16 ! fakesink
uridecodebin uri=‘file:///storage/ext_media/sochi_wedding_1.h264’ ! nvvidconv ! ‘video/x-raw(memory:NVMM)’ ! videorate ! ‘video/x-raw(memory:NVMM), format=NV12, framerate=10/1’ ! queue name=q0 ! mux.sink_0
uridecodebin uri=‘file:///storage/ext_media/sochi_wedding_1.h264’ ! nvvidconv ! ‘video/x-raw(memory:NVMM)’ ! videorate ! ‘video/x-raw(memory:NVMM), format=NV12, framerate=10/1’ ! queue ! tee name=t
t. ! queue ! mux.sink_1
t. ! queue ! nvvidconv ! ‘video/x-raw’ ! videorate ! ‘video/x-raw, framerate=10/1’ ! videoscale ! ‘video/x-raw, width=640, height=360’ ! jpegenc quality=23 ! multifilesink location=’/storage/ext_media/out_dir/image_%d.jpg’

In my opinion the problem is with the “tee” plugin - one of its consumers is really fast (jpegenc and multifilesink) and the other (nvstreammux) waits for 16 frames before consuming the data (this is controlled by the “batch-size” parameter of nvstreammux). Reducing this parameter to 4 or 2 solves the problem, but it is not a viable option for us - later modules work optimally with batch-size of 32. Note that for 2 streams (i.e. using the command above) the pipeline gets stuck in PREROLLING if we set batch-size to 32.

I checked the timestamps of the buffers arriving to the jpegenc element and they have the right pts.
So indeed it looks like buffers being rewritten while travelling through the pipeline.

According to my tests, nvidia’s elements are not using the ALLOCATION query to compute the number of decoder output buffers.
The decoder seems to always allocates 4 buffers, regardless of the pipeline topology and nvstreammux is not even responding to the allocation query:
0:00:00.495817443 1831 0x16eaa80 DEBUG tee gsttee.c:617:gst_tee_query_allocation: Allocation query failed on pad src_0, ignoring allocation

I generated a graph of your full pipeline for easier visualization see attached
nv-pipeline.png

Wrong frame order from rtsp source
#2

HI
Sorry for a late reply, can you use DS 4.0.2 for a try instead of 3.0?

#3

Hi amjad.akkawi,

Please try with DS 4.0.2, if still an issue, please open a new topic.

Thanks

#4

Hello kayccc,

I faced still the issue, with DS 4.0.2 please check it.

#5

I faced it also with DS 4.02.

#6

Can you specify details how you see frame order wrong on the decoding streams on GPU?

#7

In the original ticket i put the command i used and then manually viewed the images . Here is the command again:

Command:
gst-launch-1.0 nvstreammux name=mux width=544 height=544 batch-size=16 ! fakesink
uridecodebin uri=‘file:///storage/ext_media/sochi_wedding_1.h264’ ! nvvidconv ! ‘video/x-raw(memory:NVMM)’ ! videorate ! ‘video/x-raw(memory:NVMM), format=NV12, framerate=10/1’ ! queue name=q0 ! mux.sink_0
uridecodebin uri=‘file:///storage/ext_media/sochi_wedding_1.h264’ ! nvvidconv ! ‘video/x-raw(memory:NVMM)’ ! videorate ! ‘video/x-raw(memory:NVMM), format=NV12, framerate=10/1’ ! queue ! tee name=t
t. ! queue ! mux.sink_1
t. ! queue ! nvvidconv ! ‘video/x-raw’ ! videorate ! ‘video/x-raw, framerate=10/1’ ! videoscale ! ‘video/x-raw, width=640, height=360’ ! jpegenc quality=23 ! multifilesink location=’/storage/ext_media/out_dir/image_%d.jpg’

#8

@amjad.akkawi

Why are you saving the images before streammux ? Have you tried using demux ? I can verify that it will work if you try and save the images after demux.

Pro-tip: If you want to save images before streammux i recommend using interpipesrc/interpipesink. I can verify that will save the images in the correct order if you dont want to use demux.

#9

I see your commands, but how you verify frame order wrong?

#10