Stuttery smart record files when adding an extra camera

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

I have an app I have built which is based on deepstream-test3.
I use smart record and have it working in two modes.
Mode 1 taps off the stream before the streammuxer and adds the smart record bin. This is the same as the deepstream examples.
Mode 2 add smart record after muxing and inference then demux and re-encode. This way you can get the detections drawn on the recorded clips.

I have a few hikvision rtsp cameras that I test with and it all works great. All good good quality clips recorded with mode 1 and mode 2.

Now I have got hold of another camera. A hikvision colorvu camera which does better in low light situations.
When I run with this camera added - using mode 1 recording it works fine. But using mode 2 recording the recorded clips are all stuttery when I have a few cameras running.

If I run ONLY with this colorvu camera it seems fine in mode 1 and 2.
If I add a second camera it’s also ok. But with a third camera (or more) I get the stuttery recording - but only for the colorvu camera. Recording from the other cameras are fine.

I don’t see any strange log output about this.

Have you got any ideas on why this would be the case.

Note that I have set all cameras to the same settings of :
1080p
25fps
Vbr max bitrate 5Mbps
I-frame interval 50
H265

I don’t think this is a performance issue because i can run with 4 cameras just find in mode 1 and mode 2.

But when I use the colorvu camera and just two of my others (3 in total) I get the really stuttery clips recorded from the colorvu. It’s like the frame timing is out of whack or the frames are out of order after re-encoding.
Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) jetson nano
• DeepStream Version 5.0.1
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

3

For every ip camera connected with deepstream pipeline, deepstream apps treat them in the same way. It is hard to know what is the reason.

4

That is true but from your experience what could cause stuttery smart record files just for this camera. If I run with 5 cameras - only the hikvision colorvu camera has this issue. all the other record buttery smooth files.

Here is a link to a sample video clip to see what I mean. Some sort of timing issue.

External Media

Also note that this only occurs when I re-encode the stream and then tap in the SR bin.
I I tap in the SR bin before decoding this is not an issue and I get buttery smooth file recordings.

Something weird is going on with the encoder?!? Notice that timestamp at the top of the frames keeps glitching and going backwards. These are the sorts of issues I used to see back with Deepstream 4.0 when the muxer was set to live_source=true. Back then we had to use false to see good smooth output.

ps.you can ignore the yellow box shading. I have some logic that detects when an object is stationary and it just colours it yellow.

5

The playback of the IMG_3439.mp4 file in my PC looks OK. No glitching appears.

6

Did you watch it from the beginning. Try and ignore the bboxes and watch the head as it enters the room - its jittering all over the place. I can see it on my iPhone, iPad and desktop computer so its not something wrong with the video player.

7

I playback from the beginning, there is no jittering from the beginning to the end.

8

I dont understand. On an apple device its full of stuttering. Let me try and play this on a windows machine.

9

@Fiona.Chen turns out these clips play fine on windows.
They also play fine using VLC on Mac. However using the default video player on Mac (QuickTime player) shows lots of stuttering.

Also you see the stuttering on both iOS and android devices. In fact on android it seems to freeze on some of the frames for a few seconds.

10

@Fiona.Chen I’ve parsed the file with ffmpeg to check with errors using :

ffmpeg -v error -i IMG_3439.mp4 -f null - 2>error.log

And I get the below error:

[null @ 0x7ffac1019200] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 2 >= 1
[null @ 0x7ffac1019200] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 2 >= 2

So there is something wrong with these files. It seems if you use VLC player to watch them it is clever enough to somehow fix it on the fly. Where when watching on an android or iOS device it cant so we get a stuttering video.

11

Anyone know how the encoder could be creating frames with writing timestamps??

12

Can you dump the original stream from the special camera? Can you try to dump the timestamp before and after the HW encoder in front of smart recording?

13

Hi @Fiona.Chen ive found a few more things out while testing the colorvu cameras.

For some reason when the cameras outputs h265 the frame rate seems to be stuck at 29.97fps. I don’t know if this is a bug or not but it might be the cause of timing issues given the other connected cameras are all at 25fps.

When I set the camera to h264 it seems to behave properly and I can tell it to use whatever FPS I like. So I have set it to 25 fps like my other cameras.

So with 4 cameras connected : 1 of which is the colorvu camera I know see these following errros every so often only on the colorvu camera stream:

Apr 03 17:55:45 nano-1 python[3648]: ===== NVMEDIA: NVENC =====
Apr 03 17:55:45 nano-1 python[3648]: NvMMLiteBlockCreate : Block : BlockType = 8
Apr 03 17:55:49 nano-1 python[3648]: NvMMLiteOpen : Block : BlockType = 261
Apr 03 17:55:49 nano-1 python[3648]: NVMEDIA: Reading vendor.tegra.display-size : status: 6
Apr 03 17:55:49 nano-1 python[3648]: NvMMLiteBlockCreate : Block : BlockType = 261
Apr 03 17:55:49 nano-1 python[3648]: NvMMLiteOpen : Block : BlockType = 8

Apr 03 17:59:25 nano-1 python[3648]: ===== NVMEDIA: NVENC ===== [264/1913]
Apr 03 17:59:25 nano-1 python[3648]: NvMMLiteBlockCreate : Block : BlockType = 8
Apr 03 17:59:26 nano-1 python[3648]: NvRmChannelSubmit: NvError_IoctlFailed with error code 22
Apr 03 17:59:26 nano-1 python[3648]: NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 7, SyncPointValue = 0)
Apr 03 17:59:26 nano-1 python[3648]: NvRmChannelSubmit: NvError_IoctlFailed with error code 22
Apr 03 17:59:26 nano-1 python[3648]: NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 14, SyncPointValue = 0)
Apr 03 17:59:26 nano-1 python[3648]: fence_set_name ioctl failed with 22
Apr 03 17:59:26 nano-1 python[3648]: NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 14, SyncPointValue = 0)
Apr 03 17:59:26 nano-1 python[3648]: NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 14, SyncPointValue = 0)
Apr 03 17:59:26 nano-1 python[3648]: NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 14, SyncPointValue = 0)
Apr 03 17:59:26 nano-1 python[3648]: NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 14, SyncPointValue = 0)
Apr 03 17:59:26 nano-1 python[3648]: NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 14, SyncPointValue = 0)
Apr 03 17:59:26 nano-1 python[3648]: NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 14, SyncPointValue = 0)
Apr 03 17:59:26 nano-1 python[3648]: NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 14, SyncPointValue = 0)
Apr 03 17:59:26 nano-1 python[3648]: NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 14, SyncPointValue = 0)
Apr 03 17:59:26 nano-1 python[3648]: NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 14, SyncPointValue = 0)
Apr 03 17:59:26 nano-1 python[3648]: NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 14, SyncPointValue = 0)
Apr 03 17:59:26 nano-1 python[3648]: NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 14, SyncPointValue = 0)
Apr 03 17:59:27 nano-1 python[3648]: NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 14, SyncPointValue = 0)
Apr 03 17:59:27 nano-1 python[3648]: NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 14, SyncPointValue = 0)
Apr 03 17:59:27 nano-1 python[3648]: NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 14, SyncPointValue = 0)
Apr 03 17:59:27 nano-1 python[3648]: NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 14, SyncPointValue = 0)
Apr 03 17:59:27 nano-1 python[3648]: NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 14, SyncPointValue = 0)
Apr 03 17:59:27 nano-1 python[3648]: NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 14, SyncPointValue = 0)
Apr 03 17:59:27 nano-1 python[3648]: NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 14, SyncPointValue = 0)
Apr 03 17:59:27 nano-1 python[3648]: NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 14, SyncPointValue = 0)
Apr 03 17:59:27 nano-1 python[3648]: NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 14, SyncPointValue = 0)

Apr 03 17:59:29 nano-1 python[3648]: NvMMLiteOpen : Block : BlockType = 261
Apr 03 17:59:29 nano-1 python[3648]: NVMEDIA: Reading vendor.tegra.display-size : status: 6
Apr 03 17:59:29 nano-1 python[3648]: NvMMLiteBlockCreate : Block : BlockType = 261
Apr 03 17:59:37 nano-1 python[3648]: NvMMLiteOpen : Block : BlockType = 261
Apr 03 17:59:37 nano-1 python[3648]: NVMEDIA: Reading vendor.tegra.display-size : status: 6
Apr 03 17:59:37 nano-1 python[3648]: NvMMLiteBlockCreate : Block : BlockType = 261
Apr 03 17:59:41 nano-1 python[3648]: NvMMLiteOpen : Block : BlockType = 261
Apr 03 17:59:41 nano-1 python[3648]: NVMEDIA: Reading vendor.tegra.display-size : status: 6
Apr 03 17:59:41 nano-1 python[3648]: NvMMLiteBlockCreate : Block : BlockType = 261
Apr 03 17:59:49 nano-1 python[3648]: NvMMLiteOpen : Block : BlockType = 261
Apr 03 17:59:49 nano-1 python[3648]: NVMEDIA: Reading vendor.tegra.display-size : status: 6
Apr 03 17:59:49 nano-1 python[3648]: NvMMLiteBlockCreate : Block : BlockType = 261
Apr 03 17:59:53 nano-1 python[3648]: NvMMLiteOpen : Block : BlockType = 261
Apr 03 17:59:53 nano-1 python[3648]: NVMEDIA: Reading vendor.tegra.display-size : status: 6
Apr 03 17:59:53 nano-1 python[3648]: NvMMLiteBlockCreate : Block : BlockType = 261
Apr 03 18:00:01 nano-1 python[3648]: NvMMLiteOpen : Block : BlockType = 261
Apr 03 18:00:01 nano-1 python[3648]: NVMEDIA: Reading vendor.tegra.display-size : status: 6
Apr 03 18:00:01 nano-1 python[3648]: NvMMLiteBlockCreate : Block : BlockType = 261
Apr 03 18:00:05 nano-1 python[3648]: NvMMLiteOpen : Block : BlockType = 261

Apr 03 18:38:10 nano-1 python[3648]: ** ERROR: RunUserCallback:199: No video stream found
Apr 03 18:38:10 nano-1 python[3648]: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x55a137cf60] Invalid mdhd time scale 0, defaulting to 1
Apr 03 18:38:10 nano-1 python[3648]: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x55a137cf60] invalid STSD entries 0
Apr 03 18:38:10 nano-1 python[3648]: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x55a137cf60] error reading header

When these errors invalid files are written by smart record. The stream seems to get into a state such that no more smart records will work on this stream.

However the other camera streams continue to work just fine while this is going on and smart record will successfully record files for them.

I have read elsewhere that this error message:

NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 14, SyncPointValue = 0)

…shows there there is an error with the hardware block.

14

Can you dump the camera stream for us to analyze?

15

Will do @Fiona.Chen - what is the best way (format) to “dump” the camera stream?

16

Due to the format in the stream in RTP. It is better to dump the RTP payload as it is.

18

@jasonpgf2a Have you dumped the camera stream?

19

Hi @Fiona.Chen . No, not yet as I’ve been busy with some other things.

What is the best way to dump the stream into a format you can use? Can you provide a gstreamer cli pipeline?

20

Is the camera transferring H264 or H265 stream by RTSP? If so, the pipeline may like:

gst-launch-1.0 rtspsrc location=rtsp://… ! rtph264depay ! h264parse ! qtmux ! filesink location=…