Migrated my app to DS5.0 preview and recorded mp4 files behaving odd

I have an app based on deepstream-test3. When a detection is made in pgie it dynamically adds a new branch to a tee that saves to a file.
This worked ok in DS 4.02 and saved high quality h265 muxed into an MP4 files.

I have just setup DS5.0 and got the app running - only had to change the DS version in the Makefile and some file paths.

For some reason however the recorded files are weird. They are jumpy and sometimes go backward in time. Why would this be?

I note that you now have the smart record feature but this doesn’t look to show the bounding boxes in the recorded files.

I have added attach-sys-ts = TRUE to the streammux component. This stabilises the recorded video quite a bit, however its still a little glitchy where it seems to repeat a frame twice or go backward a couple of frames during playback. Any other suggestions on what to look at and what could have changed since DS4.02 would be great.

Is it possible to add the smart record in the sink? I think the bbox should be available if it can work.
refer Smart Video Record section in https://docs.nvidia.com/metropolis/deepstream/plugin-manual/index.html#page/DeepStream%20Plugins%20Development%20Guide/deepstream_plugin_details.3.01.html#

In an try and get it to work after the demux but then main issue here is that recorded frames are not working properly. They are now jump whereas they were nice and smooth with DD4.02?

I don’t think smart record can work on DS4.0.2, are you using smart record to dump the frame?

This issue is all about my own program - not smart record. I have written my own program in DS4.02 that was based off deepstream-test3.

When a detection of an object is made I dynamically add a new branch to the pipeline to record a mp4 file for 15 seconds.
So very similar to Smart Record.

My problem is that I have migrated this to DS5.0 preview and the recorded files are choppy and the frames look like they okay in reverse sometimes.
This is what I want to solve.

Btw. I have separately tested smart record and it’s files are also chop and record a weird framerates dofferent to the camera. I have another thread in here for that…

OK. I see, so I think we can focus on the issue of Smart Record in DS 5.0 int this tread Smart Record issue - record files have strange grey blocks in the middle . And go back here if you still need your own pipeline, is that ok for you?

Yes - main issue is to get my pipeline recording files nicely like it was in DS4.02… Is there a way I can send you a sample mp4 file - the forum doesn’t allow me to attach them. If you watch the file you will see what I mean by it being choppy.

I have a feeling its something to do with timing. Adding attach-sys-ts = TRUE helped it a lot! Before I added that to the muxer the files were very weird and some parts of them even played in reverse! With attach-sys-ts = TRUE its still a little glitchy but much better. The overall quality for the same bitrate seems to be lower as well. Faces look a bit more blurred than they did in DS4.02 with exactly the same settings…

Thankyou for your help.

Is there a way I can send you a sample mp4 file - the forum doesn’t allow me to attach them. If you watch the file you will see what I mean by it being choppy.

Is it possible to send the file through a private message? Or upload the file to 3rd party cloud and share the link with me to download.

There are some difference in muxer between DS5.0 and 4.0.2, Let’s move to another thread to look into the issue and debug.

@bcao did you get the sample - send via private message?

Yes, we are checking it in Smart Record issue - record files have strange grey blocks in the middle

As additional information I have changed my app over to use smart record. I have added smart record AFTER the stream demuxer as I want to have bounding boxes from the pgie drawn. The mp4 files generated with smart record are also glitchy and jump around back in time.
It obviously a timing issue upstream…

I notice that in deepstream-test5 you have used an rtpsrc element whereas I purely used a decodebin as used in deepstream-test3 Could that be the issue? Smart record is not liking the decodebin do the rtpsrc parsing and queueing?
Maybe it has something to do with this new ntp feature??

Although it is counter-intuitive I have just set the muxer parameter live-sources to FALSE - even though I actually have live sources (RTSP cameras from Hikvision) and now recorded files play normally without jitter and reversing and all the weird things they were doing. They do still seem to pause every so often and go in slow motion for a little but sometimes, but thats far better than the weirdness of having live-sources set to 1.
Note: this all used to work fine with DS 4.02 so I will have to revert back to 4.02 unless we can solve this.

Can someone from nvidia please explain this as the documentation clearly states that we should set live-sources to TRUE for rtsp camera courses.?
Something must be wrong with the muxer component I think…

@bcao any news on this… By setting streammux live-source to 0 (even though I am using live sources) I can get decent files recorded by smart record but there are still occasional pauses and sections where it plays in slow motion.

Just a guess, have you tried setting qos=True in your sink elements? I found that upstream nvidia elements do not behave nicely if the sink doesn’t generate qos messages.

Hi @dorin.clisu.ntt - yes have tested all possible combinations of that. I cannot detect any difference in performance with QoS true or false in the pipeline. In fact one of the nvidia team actually said its best to have QoS as false on a resource constrained device like the jetson nano (I asked the question about QoS a while back). However inside the reference app (deepstream-app) there is a comment saying QoS should always be true.

I think it is something wrong with the stream muxer/demuxer components.
In DS4.02 the pipeline worked fine with dynamic recording of files. There were some memory leaks but we can deal with them.
Then with the same app running on DS5.0 dp I notice the recorded files behave weird and sometimes play sections in reverse and are very jumpy. The only way to stop this is to set live-sources FALSE on the streammux element.
But even then the recorded files are not perfect. They pause occasionally and run in slow-mo for a second or two. Note that none of these issues occurred with DS4.02.
I switched my pipeline from using my own dynamically added elements to using the new smart record bin - and it has exactly he same issues with recorded files. Tested on both nano and xavier nx.

Sry for the late reply, we had fixed the issue and please wait for next release which is coming soon.

1 Like

Thanks @bcao can you let us know what the issue was? Was it in the streammux component or smart record bin?

We have a improvement on both, pls wait for next release.