Pixel distortion in multiple RTSP sources with different frame rate using New Streammux

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

• Hardware Platform (Jetson / GPU) Jetson
• DeepStream Version 6.3 (docker image: nvcr.io/nvidia/deepstream:6.3-triton-multiarch)
• JetPack Version (valid for Jetson only) JetPack 5.1.2 L4T 35.4.1
• TensorRT Version TensorRT 8.5.2 Cuda 11.4
• Issue Type( questions, new requirements, bugs) Bugs

Hi,

I am using the deepstream reference application (apps/sample_apps/­deepstream-app) enabling the New Gst-nvstreammux ( export USE_NEW_NVSTREAMMUX=yes).
I am using the provided configuration file: samples/configs/deepstream-app/source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.yml.

The inputs are RTSP sources. When the sources have the same frame rate everything works fine. However, when the sources have different frame rate, the higher frame rate sources shows pixel distortion on the sink (type: 2).

Using the same configuration with the “old” Gst-nvstreammux (export USE_NEW_NVSTREAMMUX=no) everything works fine even when the sources have different frame rates.

I have experimented this issue with RTSP sources from both IP cameras and rtsp-simple-server.

I also attempted the same test on DeepStream Version 6.4 (docker image: nvcr.io/nvidia/deepstream:6.4-triton-multiarch), but encountered the same issue with RTSP sources of different frame rates.

Has anyone encountered this issue before?
Your assistance would be greatly appreciated.
Thank you.

1 Like
3

What kind of ditortion it is? Can you show us the image you captured?

Can you provide your deepstream-app configuration file?

4

The deepstream-app configuration file with the Old Streammux is:

application:
  enable-perf-measurement: 1
  perf-measurement-interval-sec: 5
  ##gie-kitti-output-dir=streamscl

tiled-display:
  enable: 1
  rows: 1
  columns: 2
  width: 1920
  height: 1080
  gpu-id: 0
  #(0): nvbuf-mem-default - Default memory allocated, specific to particular platform
  #(1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory, applicable for Tesla
  #(2): nvbuf-mem-cuda-device - Allocate Device cuda memory, applicable for Tesla
  #(3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory, applicable for Tesla
  #(4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson
  nvbuf-memory-type: 0

source:
  csv-file-path: sources_rtsp.csv

sink0:
  enable: 1
  #Type - 1=FakeSink 2=EglSink/nv3dsink (Jetson only) 3=File
  type: 2
  sync: 1
  source-id: 0
  gpu-id: 0
  nvbuf-memory-type: 0

streammux:
  gpu-id: 0
  ##Boolean property to inform muxer that sources are live
  live-source: 1
  buffer-pool-size: 2
  batch-size: 2
  ##time out in usec, to wait after the first buffer is available
  ##to push the batch even if the complete batch is not formed
  batched-push-timeout: 40000
  ## Set muxer output width and height
  width: 1920
  height: 1080
  ##Enable to maintain aspect ratio wrt source, and allow black borders, works
  ##along with width, height properties
  enable-padding: 0
  nvbuf-memory-type: 0

The deepstream-app configuration file with the New Streammux is:

application:
  enable-perf-measurement: 1
  perf-measurement-interval-sec: 5
  ##gie-kitti-output-dir=streamscl

tiled-display:
  enable: 1
  rows: 1
  columns: 2
  width: 1920
  height: 1080
  gpu-id: 0
  #(0): nvbuf-mem-default - Default memory allocated, specific to particular platform
  #(1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory, applicable for Tesla
  #(2): nvbuf-mem-cuda-device - Allocate Device cuda memory, applicable for Tesla
  #(3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory, applicable for Tesla
  #(4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson
  nvbuf-memory-type: 0

source:
  csv-file-path: sources_rtsp.csv

sink0:
  enable: 1
  #Type - 1=FakeSink 2=EglSink/nv3dsink (Jetson only) 3=File
  type: 2
  sync: 1
  source-id: 0
  gpu-id: 0
  nvbuf-memory-type: 0

streammux:
  gpu-id: 0
  config-file-path: config_new_streammux.txt

where config_new_streammux.txt contains:

[property]
batch-size=2
overall-min-fps-n=25
overall-min-fps-d=1

Tests conducted are as follows:

Test 1
Streammux. Old
Source Frame Rates: Left - 25 fps, Right - 25 fps
Video File: old_streammux_same_framerate.webm

Test 2
Streammux: New
Source Frame Rates: Left - 25 fps, Right - 25 fps
Video File:new_streammux_same_framerate.webm

Test 3
Streammux: Old
Source Frame Rates: Left - 15 fps, Right - 25 fps
Video File: old_streammux_different_framerate.webm

Test 4
Streammux: New
Source Frame Rates: Left - 15 fps, Right - 25 fps
Video File: new_streammux_different_framerate.webm

The distortion is only in Test4 on the video with higher frame rate.

All videos mentioned above have been uploaded in the zip file.
videos.zip (3.6 MB)

Thanks

5

The video corruption is caused by network packet loss. Can you show us the “sources_rtsp.csv” too?

For the cameras with different FPSs case, please follow DeepStream SDK FAQ - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums to configure the new nvstreammux.

6

The file source_rtsp.csv contains:

enable,type,uri,num-sources,gpu-id,cudadec-memtype
1,4,rtsp://10.10.11.121:554,1,0,0
1,4,rtsp://10.10.11.123:554,1,0,0

Where the first RTSP source operates at 15 fps and the second at 25 fps.

Following guidance from DeepStream SDK FAQ - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums to configure the new nvstreammux, the config_new_streammux.txt now contains:

[property]
adaptive-batching=1
batch-size=2
overall-min-fps-n=25
overall-min-fps-d=1
max-same-source-frames=2

[source-config-0]
## Max number of frames allowed to be muxed per output batch buffer; uint
max-num-frames-per-batch=1

[source-config-1]
## Max number of frames allowed to be muxed per output batch buffer; uint
max-num-frames-per-batch=2

I have conducted several tests, and the issue appears to be resolved. There is no longer any distortion in the video with the higher frame rate.

Thanks

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.