DeepStream. Jittered output

mrbadger · April 19, 2023, 5:43pm

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GeForce GTX 1650 Ti Mobile
• DeepStream Version DeepStream5.1
• JetPack Version (valid for Jetson only)
• TensorRT Version tensorrt 21.02
• NVIDIA GPU Driver Version (valid for GPU only) 515.86.01
• Issue Type( questions, new requirements, bugs) bug
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
Hello,
I am using Dell G5 with specified Nvidia GPU and Nvidia i5 10th gen CPU.

I am running a pipeline with Yolov4 as primary gie and EfficientNet as secondary gie. Attached configuration files below.
config.txt (2.3 KB)
config_infer_primary.txt (604 Bytes)
config_infer_secondary.txt (957 Bytes)

Sources are RTSP streams of 3840x2160 resolution with 20 PFS.

I did optimized DeepStream configuration according to recommendations in Troubleshooting — DeepStream 6.2 Release documentation

However, DeepStream output frames are very jittered

Perhaps you have any suggestion to fix the issue?
Thanks in advance!

fanzh · April 20, 2023, 5:36am

Please refer to this topic delayed, can you measure the delay?

mrbadger · April 20, 2023, 10:16am

Attached a screenshot with latency measurements
crim0 is a custom plugin

fanzh · April 20, 2023, 2:48pm

please check why it takes about 500ms in crim0 plugin.

mrbadger · April 21, 2023, 3:44am

I am looking into it. Perhaps, there are instructions or protocol, how to inspect a custom plugin?
Also, maybe you have suggestions why nvv4l2decoder0 creates such amount of latency?

fanzh · April 21, 2023, 8:15am

to narrow down this issue, please do some tests:

set one source and one fakesink(type=1), can you reproduce this issue(nvv4l2decoder0 takes 5s )?
set one source and one eglsink(type=2), can you reproduce nvv4l2decoder0 issue?

mrbadger · April 25, 2023, 8:44am

It turns out, that bottleneck is the custom plugin.
When using streaming 12 3840x2160 videos: with plugin enabled: ~3FPS; with plugin disabled ~26FPS. Also CPU usage is way higher with enabled custom plugin

Tests:
RTSP 3840x2160 20FPS:
one source; one fakesink(type=1); custom plugin enabled

************BATCH-NUM = 251**************
Comp name = nvv4l2decoder0 in_system_timestamp = 1682411763713.623047 out_system_timestamp = 1682411764024.021973               component latency= 310.398926
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 251               in_system_timestamp = 1682411764024.301025 out_system_timestamp = 1682411764024.333008               component_latency = 0.031982
Comp name = primary_gie in_system_timestamp = 1682411764024.396973 out_system_timestamp = 1682411764024.419922               component latency= 0.022949
Comp name = secondary_gie_0 in_system_timestamp = 1682411764024.436035 out_system_timestamp = 1682411764024.449951               component latency= 0.013916
Comp name = crim0 in_system_timestamp = 1682411764025.520020 out_system_timestamp = 1682411764069.080078               component latency= 43.560059
Comp name = demuxer in_system_timestamp = 1682411764069.089111 out_system_timestamp = 1682411764069.127930               component latency= 0.038818
Source id = 0 Frame_num = 251 Frame latency = 355.626953 (ms) 


(deepstream-app:1): GLib-GObject-CRITICAL **: 08:36:04.070: g_object_get: assertion 'G_IS_OBJECT (object)' failed

************BATCH-NUM = 252**************
Comp name = nvv4l2decoder0 in_system_timestamp = 1682411763824.253906 out_system_timestamp = 1682411764065.006104               component latency= 240.752197
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 252               in_system_timestamp = 1682411764065.688965 out_system_timestamp = 1682411764065.847900               component_latency = 0.158936
Comp name = primary_gie in_system_timestamp = 1682411764066.041992 out_system_timestamp = 1682411764066.143066               component latency= 0.101074
Comp name = secondary_gie_0 in_system_timestamp = 1682411764066.211914 out_system_timestamp = 1682411764066.281006               component latency= 0.069092
Comp name = crim0 in_system_timestamp = 1682411764069.155029 out_system_timestamp = 1682411764108.593994               component latency= 39.438965
Comp name = demuxer in_system_timestamp = 1682411764108.613037 out_system_timestamp = 1682411764108.693115               component latency= 0.080078
Source id = 0 Frame_num = 252 Frame latency = 284.565186 (ms) 


(deepstream-app:1): GLib-GObject-CRITICAL **: 08:36:04.110: g_object_get: assertion 'G_IS_OBJECT (object)' failed


(deepstream-app:1): GLib-GObject-CRITICAL **: 08:36:04.150: g_object_get: assertion 'G_IS_OBJECT (object)' failed

************BATCH-NUM = 253**************
Comp name = nvv4l2decoder0 in_system_timestamp = 1682411763712.967041 out_system_timestamp = 1682411764121.940918               component latency= 408.973877
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 253               in_system_timestamp = 1682411764122.202881 out_system_timestamp = 1682411764122.228027               component_latency = 0.025146
Comp name = primary_gie in_system_timestamp = 1682411764122.254883 out_system_timestamp = 1682411764122.274902               component latency= 0.020020
Comp name = secondary_gie_0 in_system_timestamp = 1682411764122.368896 out_system_timestamp = 1682411764122.385986               component latency= 0.017090
Comp name = crim0 in_system_timestamp = 1682411764123.373047 out_system_timestamp = 1682411764166.558105               component latency= 43.185059
Comp name = demuxer in_system_timestamp = 1682411764166.584961 out_system_timestamp = 1682411764166.625000               component latency= 0.040039
Source id = 0 Frame_num = 253 Frame latency = 453.703857 (ms) 


(deepstream-app:1): GLib-GObject-CRITICAL **: 08:36:04.191: g_object_get: assertion 'G_IS_OBJECT (object)' failed

************BATCH-NUM = 254**************
Comp name = nvv4l2decoder0 in_system_timestamp = 1682411763925.260010 out_system_timestamp = 1682411764123.507080               component latency= 198.247070
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 254               in_system_timestamp = 1682411764162.775879 out_system_timestamp = 1682411764162.843018               component_latency = 0.067139
Comp name = primary_gie in_system_timestamp = 1682411764162.885010 out_system_timestamp = 1682411764162.916016               component latency= 0.031006
Comp name = secondary_gie_0 in_system_timestamp = 1682411764162.944092 out_system_timestamp = 1682411764162.979980               component latency= 0.035889
Comp name = crim0 in_system_timestamp = 1682411764166.681885 out_system_timestamp = 1682411764209.720947               component latency= 43.039062
Comp name = demuxer in_system_timestamp = 1682411764209.749023 out_system_timestamp = 1682411764209.822998               component latency= 0.073975
Source id = 0 Frame_num = 254 Frame latency = 284.671875 (ms)

one source; one eglsink(type=2); custom plugin enabled

***********BATCH-NUM = 251**************
Comp name = nvv4l2decoder0 in_system_timestamp = 1682412390432.306885 out_system_timestamp = 1682412390638.093018               component latency= 205.786133
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 251               in_system_timestamp = 1682412390638.695068 out_system_timestamp = 1682412390638.934082               component_latency = 0.239014
Comp name = primary_gie in_system_timestamp = 1682412390639.108887 out_system_timestamp = 1682412390639.222900               component latency= 0.114014
Comp name = secondary_gie_0 in_system_timestamp = 1682412390639.319092 out_system_timestamp = 1682412390639.383057               component latency= 0.063965
Comp name = crim0 in_system_timestamp = 1682412390688.700928 out_system_timestamp = 1682412390732.364014               component latency= 43.663086
Comp name = demuxer in_system_timestamp = 1682412390732.375977 out_system_timestamp = 1682412390732.428955               component latency= 0.052979
Source id = 0 Frame_num = 251 Frame latency = 300.176025 (ms) 


(deepstream-app:1): GLib-GObject-CRITICAL **: 08:46:30.735: g_object_get: assertion 'G_IS_OBJECT (object)' failed


(deepstream-app:1): GLib-GObject-CRITICAL **: 08:46:30.775: g_object_get: assertion 'G_IS_OBJECT (object)' failed

************BATCH-NUM = 252**************
Comp name = nvv4l2decoder0 in_system_timestamp = 1682412390323.545898 out_system_timestamp = 1682412390728.129883               component latency= 404.583984
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 252               in_system_timestamp = 1682412390728.945068 out_system_timestamp = 1682412390729.426025               component_latency = 0.480957
Comp name = primary_gie in_system_timestamp = 1682412390729.699951 out_system_timestamp = 1682412390729.936035               component latency= 0.236084
Comp name = secondary_gie_0 in_system_timestamp = 1682412390729.986084 out_system_timestamp = 1682412390730.060059               component latency= 0.073975
Comp name = crim0 in_system_timestamp = 1682412390732.496094 out_system_timestamp = 1682412390780.479004               component latency= 47.982910
Comp name = demuxer in_system_timestamp = 1682412390780.502930 out_system_timestamp = 1682412390780.581055               component latency= 0.078125
Source id = 0 Frame_num = 252 Frame latency = 457.104980 (ms) 

(deepstream-app:1): GLib-GObject-CRITICAL **: 08:46:30.815: g_object_get: assertion 'G_IS_OBJECT (object)' failed


************BATCH-NUM = 253**************
Comp name = nvv4l2decoder0 in_system_timestamp = 1682412390520.887939 out_system_timestamp = 1682412390728.644043               component latency= 207.756104
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 253               in_system_timestamp = 1682412390729.310059 out_system_timestamp = 1682412390729.510010               component_latency = 0.199951
Comp name = primary_gie in_system_timestamp = 1682412390729.743896 out_system_timestamp = 1682412390729.962891               component latency= 0.218994
Comp name = secondary_gie_0 in_system_timestamp = 1682412390730.029053 out_system_timestamp = 1682412390730.079102               component latency= 0.050049
Comp name = crim0 in_system_timestamp = 1682412390780.638916 out_system_timestamp = 1682412390825.968018               component latency= 45.329102
Comp name = demuxer in_system_timestamp = 1682412390825.988037 out_system_timestamp = 1682412390826.069092               component latency= 0.081055
Source id = 0 Frame_num = 253 Frame latency = 305.251953 (ms) 


(deepstream-app:1): GLib-GObject-CRITICAL **: 08:46:30.855: g_object_get: assertion 'G_IS_OBJECT (object)' failed

************BATCH-NUM = 254**************
Comp name = nvv4l2decoder0 in_system_timestamp = 1682412390522.653076 out_system_timestamp = 1682412390836.216064               component latency= 313.562988
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 254               in_system_timestamp = 1682412390836.662109 out_system_timestamp = 1682412390836.701904               component_latency = 0.039795
Comp name = primary_gie in_system_timestamp = 1682412390836.737061 out_system_timestamp = 1682412390836.768066               component latency= 0.031006
Comp name = secondary_gie_0 in_system_timestamp = 1682412390836.790039 out_system_timestamp = 1682412390836.810059               component latency= 0.020020
Comp name = crim0 in_system_timestamp = 1682412390837.852051 out_system_timestamp = 1682412390880.280029               component latency= 42.427979
Comp name = demuxer in_system_timestamp = 1682412390880.302002 out_system_timestamp = 1682412390880.371094               component latency= 0.069092
Source id = 0 Frame_num = 254 Frame latency = 357.826904 (ms)

one source; one fakesink(type=1); custom plugin disabled

************BATCH-NUM = 251**************
Comp name = nvv4l2decoder0 in_system_timestamp = 1682412131701.305908 out_system_timestamp = 1682412132001.195068               component latency= 299.889160
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 251               in_system_timestamp = 1682412132001.636963 out_system_timestamp = 1682412132001.696045               component_latency = 0.059082
Comp name = primary_gie in_system_timestamp = 1682412132001.813965 out_system_timestamp = 1682412132001.857910               component latency= 0.043945
Comp name = secondary_gie_0 in_system_timestamp = 1682412132001.875977 out_system_timestamp = 1682412132001.918945               component latency= 0.042969
Comp name = demuxer in_system_timestamp = 1682412132002.980957 out_system_timestamp = 1682412132003.027100               component latency= 0.046143
Source id = 0 Frame_num = 251 Frame latency = 301.750000 (ms) 

************BATCH-NUM = 252**************
Comp name = nvv4l2decoder0 in_system_timestamp = 1682412131796.216064 out_system_timestamp = 1682412132002.200928               component latency= 205.984863
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 252               in_system_timestamp = 1682412132002.461914 out_system_timestamp = 1682412132002.474121               component_latency = 0.012207
Comp name = primary_gie in_system_timestamp = 1682412132002.512939 out_system_timestamp = 1682412132002.523926               component latency= 0.010986
Comp name = secondary_gie_0 in_system_timestamp = 1682412132002.531982 out_system_timestamp = 1682412132002.541016               component latency= 0.009033
Comp name = demuxer in_system_timestamp = 1682412132003.042969 out_system_timestamp = 1682412132003.059082               component latency= 0.016113
Source id = 0 Frame_num = 252 Frame latency = 206.875000 (ms) 


(deepstream-app:1): GLib-GObject-CRITICAL **: 08:42:12.016: g_object_get: assertion 'G_IS_OBJECT (object)' failed


(deepstream-app:1): GLib-GObject-CRITICAL **: 08:42:12.056: g_object_get: assertion 'G_IS_OBJECT (object)' failed


(deepstream-app:1): GLib-GObject-CRITICAL **: 08:42:12.096: g_object_get: assertion 'G_IS_OBJECT (object)' failed

************BATCH-NUM = 253**************
Comp name = nvv4l2decoder0 in_system_timestamp = 1682412131797.468018 out_system_timestamp = 1682412132102.622070               component latency= 305.154053
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 253               in_system_timestamp = 1682412132102.934082 out_system_timestamp = 1682412132102.993896               component_latency = 0.059814
Comp name = primary_gie in_system_timestamp = 1682412132103.042969 out_system_timestamp = 1682412132103.072998               component latency= 0.030029
Comp name = secondary_gie_0 in_system_timestamp = 1682412132103.100098 out_system_timestamp = 1682412132103.125977               component latency= 0.025879
Comp name = demuxer in_system_timestamp = 1682412132104.225098 out_system_timestamp = 1682412132104.264893               component latency= 0.039795
Source id = 0 Frame_num = 253 Frame latency = 306.841064 (ms) 

************BATCH-NUM = 254**************
Comp name = nvv4l2decoder0 in_system_timestamp = 1682412131699.408936 out_system_timestamp = 1682412132104.739990               component latency= 405.331055
Comp name = src_bin_muxer source_id = 0 pad_index = 0 frame_num = 254               in_system_timestamp = 1682412132105.021973 out_system_timestamp = 1682412132105.053955               component_latency = 0.031982
Comp name = primary_gie in_system_timestamp = 1682412132105.090088 out_system_timestamp = 1682412132105.114990               component latency= 0.024902
Comp name = secondary_gie_0 in_system_timestamp = 1682412132105.135010 out_system_timestamp = 1682412132105.155029               component latency= 0.020020
Comp name = demuxer in_system_timestamp = 1682412132105.197998 out_system_timestamp = 1682412132105.232910               component latency= 0.034912
Source id = 0 Frame_num = 254 Frame latency = 405.852051 (ms)

fanzh · April 25, 2023, 10:07am

Is this still an DeepStream issue to support?

fanzh · April 25, 2023, 2:25pm

you can use nvmsgconv + nvmsgbroker to send message to redis, please refer to this topic: How to configure ds-app to save data to redis? - #10 by jiangyuzhe1992

mrbadger · April 25, 2023, 4:04pm

Is Redis a standard option to transfer frames from DeepStream to another service?
Maybe there is an even more efficient method for image transfer?

fanzh · April 27, 2023, 6:33am

no, deepstream supports many brokers, pelase refer to nvmsgbroker
please refer to this topic: How to save frame as jpg and send the filename to kafka in deepstream 5.0 deepstream_test5

mrbadger · May 2, 2023, 1:51am

Thanks!

system · May 16, 2023, 1:51am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.