Identifying Latency Issues with GStreamer Latency Plugin in DeepStream 7.1

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) dGPU Nvidia L4
• DeepStream Version 7.1
• TensorRT Version Using Docker
• Issue Type( questions, new requirements, bugs) Bug
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

I am trying to identify the bottleneck in my pipeline using the GStreamer latency plugin. Here are the parameters I am using:

  • GST_DEBUG: GST_TRACER:7
  • GST_DEBUG_COLOR_MODE: off
  • GST_TRACERS: "latency(flags=pipeline+element)"

With 10 RTSP inputs for the deepstream-rtsp-in-rtsp-out sample of DeepStream 7.1, everything works fine when the latency plugin is disabled.

However, when it is enabled, I observe increasing latencies. Here are some example measurements:

,element,median (ms),avg (ms),min (ms),max (ms),std (ms),p99 (ms)
82,Stream-muxer,18446744073672.64,11710187839741.58,0.01,18446744073709.55,8881798178985.06,18446744073709.16
97,src(udpsrc10)->sink(udpsink),72255.96,71703.58,72.92,153062.67,46959.05,150923.95
96,src(udpsrc21)->sink(udpsink),71313.23,68929.24,75.84,145876.71,45281.57,143607.92
99,src(udpsrc6)->sink(udpsink),69206.08,67924.14,75.40,145856.94,45253.92,145123.38
94,src(udpsrc20)->sink(udpsink),66367.42,67499.10,90.03,145596.62,44478.15,144436.42
95,src(udpsrc8)->sink(udpsink),64724.46,64846.53,44.44,146929.71,43561.54,143523.52
92,src(udpsrc15)->sink(udpsink),63222.28,64812.92,22.42,145453.81,45323.78,144118.89
35,rtpjitterbuffer5,59538.92,62946.97,0.02,153611.28,48848.26,150253.73
93,src(udpsrc4)->sink(udpsink),58905.81,63200.47,91.56,145474.28,44799.85,144545.38
90,src(udpsrc9)->sink(udpsink),58796.49,65024.46,100.02,146859.89,45923.95,144278.60
98,src(udpsrc7)->sink(udpsink),58445.16,61396.73,25.96,144199.00,42497.88,140956.53
21,rtpjitterbuffer3,56033.67,59383.17,0.02,145630.94,46026.57,143853.73
28,rtpjitterbuffer4,54062.81,59014.21,0.02,147343.89,46639.20,145459.08
24,rtpjitterbuffer1,53988.03,59178.54,0.02,145030.65,46486.27,143261.90
32,rtpjitterbuffer8,53582.22,58536.05,0.01,146997.83,46165.86,144598.50
34,rtpjitterbuffer6,53337.61,58765.44,0.02,145474.32,47377.99,144193.92
27,rtpjitterbuffer0,52329.93,58250.83,0.02,144870.75,47270.11,143469.33
25,rtpjitterbuffer7,52219.04,58449.63,0.01,145198.16,45627.24,143848.73
20,rtpjitterbuffer2,50634.44,56819.73,0.01,143873.69,44534.34,141201.75
...

I also notice that the RTSP output is completely delayed.

When the plugin is disabled, with 10 input streams, the RTSP output is synchronized with the RTSP input, showing no latency.

Here is the command I use when plugin enabled :

GST_DEBUG_COLOR_MODE=off GST_TRACERS="latency(flags=pipeline+element)" GST_DEBUG=GST_TRACER:7 GST_DEBUG_FILE=app_traces.log python3 deepstream_test1_rtsp_in_rtsp_out.py -i rtsp://192.168.15.133:8554/criminal-gang-perf-15 rtsp://192.168.15.133:8554/criminal-gang-perf-15 rtsp://192.168.15.133:8554/criminal-gang-perf-15 rtsp://192.168.15.133:8554/criminal-gang-perf-15 rtsp://192.168.15.133:8554/criminal-gang-perf-15 rtsp://192.168.15.133:8554/criminal-gang-perf-15 rtsp://192.168.15.133:8554/criminal-gang-perf-15 rtsp://192.168.15.133:8554/criminal-gang-perf-15 rtsp://192.168.15.133:8554/criminal-gang-perf-15 -g nvinfer

I am looking to understand why enabling the latency plugin causes such an increase in latency.

Thank you

1 Like
3

If you use the gstreamer latency plugin, gstreamer will dynamically add tracker points to each element’s pad.
This will affect performance

For deepstream pipeline, please refer to this FAQ to measure latency