Facing Glitches RTSP using my deepstream app

[aniketrohara]

• Hardware Platform: Jetson AGX Orin
• DeepStream Version: 7.0
• JetPack Version: (6.0+b106 and 6.0+b87 both are installed) L4T 36.3.0
• TensorRT Version: 8.6.2

hello i am trying to use my deepstream-app with a camera directly attached on my jetson agx orin board everything is working but as you can see in following images i am facing glitches but i am not able to understand why is that happening as for same FPS but with a mp4 as input and output i was not facing glithces.

[fanzh]

1. what is the fps of rtsp source? you can use ffmepg or gst-discoverer to get.
2. please try latency=4000 in source configurations. please find the explanation in this link.

[aniketrohara]

@fanzh

what is the fps of rtsp source? you can use ffmepg or gst-discoverer to get.

source is having 30fps but the output stream from my deepstream app is having varying fps from 8 to 15 and i am facing glitches in the output stream generated by my app only.

is it possible because my pipeline is kind of heavy? because the performance prints on terminal (at 5 sec interval) are varying from 6.3 to 18.9 and the avg final perf is 14.47

[aniketrohara]

@fanzh

please try latency=4000 in source configurations. please find the explanation in this link.

well i guess this will not work in my case because my source is giving 30 fps as i tested using ffmpeg.

[aniketrohara]

there is an update
main_app_config (1).txt (4.3 KB)
using this config i am getting an average fps of 20 using ffmpeg

but the stream on VLC was still lagging and having frame freezes so i set the network caching level in VLC to Highest Latency and then the stream improved visually as frame didn’t freeze but still stream is having too much lag and stickyness in frames

[fanzh]

please narrow down this issue by the following steps.

1. please use the following cmds to test if the source and the network are fine. for example, please check if the output fps is 30( fpsdisplaysink outputs fps). if the output fps is not 30, please check if “latency=2000” helps.

gst-launch-1.0 -v rtspsrc location=rtsp://10.19.226.223/media/video1 drop-on-latency=true latency=100 ! rtph264depay ! h264parse !  nvv4l2decoder ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=1 
gst-launch-1.0 -v rtspsrc location=rtsp://10.19.226.223/media/video1 drop-on-latency=true latency=2000 ! rtph264depay ! h264parse !  nvv4l2decoder ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=1 

2. if output fps is close to 30, the source and network should be fine. please check if it is performance issue. for example, you can use “jop” to check the GPU utilization. are you using own custom model? if only enable pgie and disable sgie ,tracker, osd, will the “PERF fps” increase?

[aniketrohara]

@fanzh

please use the following cmds to test if the source and the network are fine. for example, please check if the output fps is 30( fpsdisplaysink outputs fps). if the output fps is not 30, please check if “latency=2000” helps.

I have verified the rtsp source using your given command with latency=100 and it is giving 30fps

@fanzh

if output fps is close to 30, the source and network should be fine. please check if it is performance issue. for example, you can use “jop” to check the GPU utilization. are you using own custom model?

please tell me about “jop” and how to install this on my board as i cannot see it currently,

@fanzh

if only enable pgie and disable sgie ,tracker, osd, will the “PERF fps” increase?

yess if i disable my SGIEs then PERF fps increases

Also when running GST_DEBUG=3 i am getting following warnings:

0:01:31.429343368 237308 0xfffed0017aa0 WARN basesink gstbasesink.c:1249:gst_base_sink_query_latency: warning: Not enough buffering available for the processing deadline of 0:00:00.020000000, add enough queues to buffer 0:00:00.020000000 additional data. Shortening processing latency to 0:00:00.000000000.
0:01:31.429436359 237308 0xfffed0017aa0 WARN rtspmedia rtsp-media.c:3281:default_handle_message: 0xfffe6c0669a0: got warning Pipeline construction is invalid, please add queues. (…/libs/gst/base/gstbasesink.c(1249): gst_base_sink_query_latency (): /GstPipeline:media-pipeline/GstAppSink:appsink0:…
Bad file descriptor from nvv4l2decoder…
0:00:22.801507968 249114 0xaaab0c653000 WARN v4l2bufferpool gstv4l2bufferpool.c:1116:gst_v4l2_buffer_pool_start:nvv4l2decoder0:pool:src Uncertain or not enough buffers, enabling copy threshold

Also if i am using input and output type as file instead. of rtsp then i am getting 30fps and video is proper.

Also i am using few custom models in my pipeline.

Also i tried both of following commands but i am not able to get any component latency values
NVDS_ENABLE_LATENCY_MEASUREMENT=1
NVDS_ENABLE_COMPONENT_LATENCY_MEASUREMENT=1

[fanzh]

@aniketrohara

latency=100 and it is giving 30fps

seems the source and network are fine. Here are some steps to narrow down this issue.

1. please refer to this link for performance improvement. Especially Jetson clocks should be set high, and you need to use export to set latency value.
2. Please refer to this topic for how to use jtop. if the issue persisted, please share some logs after running “sudo tegrastats”.
3. if only disabling three sgie and secondary-pre-process1, is the “PERF fps” 30? are there still video glitches?

[aniketrohara]

@fanzh

please refer to this link for performance improvement.

i have already set the clock on high.

@fanzh

Especially Jetson clocks should be set high, and you need to use export to set latency value.

what do you mean by this?

@fanzh

Please refer to this topic for how to use jtop. if the issue persisted, please share some logs after running “sudo tegrastats”

actually i checked the GPU usage in jtop and it is 100% most of the times.

@fanzh

if only disabling three sgie and secondary-pre-process1, is the “PERF fps” 30? are there still video glitches?

yess the PERF fps is 30 and there are no glitches
this was also the same case with only secondary-pre-process1&swap SGIE disabled, rest models were enabled yet i was getting 30 fps

@aniketrohara

Also i tried both of following commands but i am not able to get any component latency values
NVDS_ENABLE_LATENCY_MEASUREMENT=1
NVDS_ENABLE_COMPONENT_LATENCY_MEASUREMENT=1

Can you please also answer this.

Also i have one more question i have enabled NvSORT tracker after PGIE and all SGIEs are after tracker even though the recognition model was called significantly less like almost 1/3rd times as of Swap model, so what is the reason for this?

Also can you tell that if there is any benefit or advantage of using triton instead of nvinfer?

One more thing actually as you can see in config file i am using one yolo model so i converted it to INT8 and all other models to fp16 except swapper (because it is giving wrong output in fp16) and even then i was not getting proper rtsp output so what are other possible options?

[fanzh]

@aniketrohara

what do you mean by this? / Can you please also answer this.

I mean, As the doc shown, please run “export NVDS_ENABLE_LATENCY_MEASUREMENT=1” and “export NVDS_ENABLE_COMPONENT_LATENCY_MEASUREMENT=1”. then you can see the latency in DeepStream plugin. please refer to this link. could you share some latency logs? Wondering if the latency time is big in secondary-pre-process1&swap SGIE.

@aniketrohara

it is 100% most of the times.

'100%" means queue in source plugin will dump data if the downstream plugins are not able to process data ASAP. packets loss will cause decoding error, namely glitch.

@aniketrohara

this was also the same case with only secondary-pre-process1&swap SGIE disabled, rest models were enabled yet i was getting 30 fps

please add log to check if the custom algorithm in preprocess cost too much time. you can set “Interval” property of sgie nvinfer to reduce the loading.

@aniketrohara

so what is the reason for this?

how do you know “was called significantly less like almost 1/3rd times as of Swap model”?

@aniketrohara

Also can you tell that if there is any benefit or advantage of using triton instead of nvinfer?

trtion is an inference sever. it can support many other backend besides TensorRT. it can support remote deployment by GRPC and on-prem deployment by CAPI. please refer to the doc for more details.

@aniketrohara

getting proper rtsp output so what are other possible options?

In short, it is because high load causes packet loss. please do the checking above first. could you share the scenario? what are models used to do respectively?

[aniketrohara]

@fanzh

I mean, As the doc shown, please run “export NVDS_ENABLE_LATENCY_MEASUREMENT=1” and “export NVDS_ENABLE_COMPONENT_LATENCY_MEASUREMENT=1”. then you can see the latency in DeepStream plugin. please refer to this link. could you share some latency logs? Wondering if the latency time is big in secondary-pre-process1&swap SGIE.

i already used the export and i raised the ticket after that only and as said i am not getting any latency prints just the PERF fps.

@fanzh

you can set “Interval” property of sgie nvinfer to reduce the loading.

actually i have already tried this but this is not working because the counter prints of custom parsers were same as without interval property.

@fanzh

how do you know “was called significantly less like almost 1/3rd times as of Swap model”?

because i added counter prints in the custom parsers of both models and observed that SWAP model is being called around 462 times whereas recognition model only around 100 times

@fanzh

In short, it is because high load causes packet loss. please do the checking above first. could you share the scenario? what are models used to do respectively?

i guess you have mistaken, i said i am NOT getting proper output with the above said condition.

[fanzh]

@aniketrohara

i am not getting any latency prints just the PERF fps.

please enable sink0, and set type=1.
did you add custom algorithm in preprocess1? if yes, please add log to check if the custom code cost too much time. one correction. you can set “secondary-reinfer-interval” property of sgie nvinfer to reduce the load. “Interval” is for pgie. BTW, nvinfer plugin and nvinfer low-level lib are opensource.

[aniketrohara]

@fanzh

please enable sink0, and set type=1.

okay thanks adding the fakesink worked out…but if possible can you explain why this helped?

@fanzh

did you add custom algorithm in preprocess1? if yes, please add log to check if the custom code cost too much time.

yess and this is the latency output of a frame where 3 faces are present for detection.
Comp name = nvv4l2decoder0 in_system_timestamp = 1733216469451.605957 out_system_timestamp = 1733216469473.147949 component latency= 21.541992

Comp name = nvstreammux-src_bin_muxer source_id = 0 pad_index = 0 frame_num = 532 in_system_timestamp = 1733216469555.176025 out_system_timestamp = 1733216469611.762939 component_latency = 56.586914

Comp name = primary_gie in_system_timestamp = 1733216469611.792969 out_system_timestamp = 1733216469625.304932 component latency= 13.511963

Comp name = secondary_preprocess_0 in_system_timestamp = 1733216470357.958984 out_system_timestamp = 1733216470462.867920 component latency= 104.908936

Comp name = secondary_gie_0 in_system_timestamp = 1733216470463.346924 out_system_timestamp = 1733216470625.464111 component latency= 162.117188

Comp name = demuxer in_system_timestamp = 1733216470626.097900 out_system_timestamp = 1733216470626.175049 component latency= 0.077148

Comp name = osd_conv in_system_timestamp = 1733216470626.260986 out_system_timestamp = 1733216470627.467041 component latency= 1.206055

Comp name = nvosd0 in_system_timestamp = 1733216470637.048096 out_system_timestamp = 1733216470638.138916 component latency= 1.090820

Source id = 0 Frame_num = 532 Frame latency = 1186.634033 (ms) Encode Latency = 4.401123

@fanzh

you can set “secondary-reinfer-interval” property of sgie nvinfer to reduce the load

can you provide any documentation link or any reference link as i am not able to find it in
https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinfer.html

[aniketrohara]

@fanzh

trtion is an inference sever. it can support many other backend besides TensorRT. it can support remote deployment by GRPC and on-prem deployment by CAPI. please refer to the doc for more details.

actually i wanted to know if triton will increase the performance or not

[fanzh]

@aniketrohara

if possible can you explain why this helped?

1. deepstream-app is opensource. please refer to is_sink_available_for_source_id in create_pipeline of opt\nvidia\deepstream\deepstream\sources\apps\sample_apps\deepstream-app\deepstream_app.c. in your configuration file, tiler is disabled. sink0 needs to be enabled for source0. Noticing there are two preprocess plugins, please check if the cfg is correct. you can set drop-frame-interval of source to control fps.
2. secondary-reinfer-interval is a configuration not a property. please find secondary-reinfer-interval in \opt\nvidia\deepstream\deepstream\sources\gst-plugins\gst-nvinfer.
3. both nvinfer and nvinferserver used TensorRT to do inference. In theory, the performance is similar.

[fanzh]

@aniketrohara

with a mp4 as input and output i was not facing glithces.

Is this still an DeepStream issue to support? Thanks! Noticing there is not glitch if using mp4 out. that means the issue is related to udpsink, rtspserver and the network.
can you play the output RTSP on the device running deepstream-app? wondering if it is related to the network. or you can save the stream by the following cmd then play.

gst-launch-1.0 rtspsrc location=XXX ! rtph264depay ! h264parse ! 'video/x-h264,stream-format=byte-stream' ! filesink location=test.h264