FPS Drop over time Deepstream Python Application

[socieboy]

• Hardware Platform (Jetson / GPU) Jetson AGX Xavier
• DeepStream Version 6.0.1
• JetPack Version (valid for Jetson only) 4.6.2
• Issue Type( questions, new requirements, bugs) Lost FPS over time

Hello, I have a custom python deepstream application in docker container that runs 3 cameras in the following pipeline.

sources → nvstreammux → nvinfer (Primary Infer) → nvtracker → nvinfer (Secondary Infer) → nvdsanalytics → nvvideoconvert → nvmultistreamtiler → nvvideoconvert → nvdsosd → nvoverlaysink

The application runs perfect, i’m able to attach a probe function a get all metadata i want, depending of the AI model we are able to get 120FPS, the problem is that after 2 or 3 hours the FPS start dropping and we have ended after 12hours having 10FPS which is not good.

We have seen many issues like this in other threads but for old versions of deepstream, we have tried different parameter tuning on the nvinfer, nvstreammux, nvtracker, and we haven’t been able to fix the problem, we also tried remplacing a gstbase file libgstbase-1.0.so.0.1405 and still we have the problem.

We would like to get some help or direction what to do with this problem

[snapeandcandy]

I’m also interested in this issue. I think some more info would be helpful:

• What is the source type? RTSP from cameras? What is the camera’s FPS?
• How did you measure the FPS?

[socieboy]

I’m using 3 GMSL cameras, the video is grabbed into the nvstreammux with the following plugins:

nvv4l2camerasrc -> nvvideoconvert ->  nvstreammux

# Mux
streammux = create_gst_element("nvstreammux", "muxer")  // Helper to create GST Elements
streammux.set_property('live-source', True)
streammux.set_property('sync-inputs', False)
streammux.set_property('width', 1280)
streammux.set_property('height', 720)
streammux.set_property('batch-size', 1) 
pipeline.add(streammux)

# Source
source = create_gst_element("nvv4l2camerasrc")
source.set_property('device',  "/dev/video0")
source.set_property('do-timestamp', True)
source.set_property('bufapi-version', True)
pipeline.add(source)

# Caps
caps = create_gst_element("capsfilter", "source-caps")
caps.set_property('caps', Gst.Caps.from_string(f"video/x-raw(memory:NVMM), width=(int)1280, height=(int)720 framerate=(fraction)120/1"))
pipeline.add(caps)
source.link(caps)

# Convertor
convertor = create_gst_element("nvvideoconvert", "source-convertor")
pipeline.add(convertor)
caps.link(convertor)

# Link source with Mux
sinkpad = streammux.get_request_pad("sink_0")
srcpad = convertor.get_static_pad("src")
srcpad.link(sinkpad)

# Primary Inferance
nvinfer = create_gst_element("nvinfer", "primary-inferance")
nvinfer.set_property("config-file-path", "...")
pipeline.add(nvinfer)
streammux.link(nvinfer)

The video looks great in hight quality the problem is after around 3 hours, the 30FPS become 9FPS

[socieboy]

@fanzh @zhliunycm2

Anyone can help with this issue.

[fanzh]

1. please check if the source can output 30fps after 3 hours, you can use fpsdisplaysink to test fps, like this: gst-launch-1.0 rtspsrc location=rtsp://xxx ! rtph264depay ! h264parse ! queue ! nvv4l2decoder ! fpsdisplaysink
2. please measure the latency of the pipeline comonents, please refer to The deepstream-test3 demo using rtsp webcam delayed - #5 by fanzh

[socieboy]

@fanzh

fpsdisplaysink

Hello I set the camera to run at 30FPS, the camera is capable to stream up to 120FPS.
Still getting drops.
The pipeline is this:
https://github.com/socieboy/video-test/blob/master/video.py

[Fiona.Chen]

There is no clue in your description. Please use “tegrastats” command to monitor the GPU performance and usage during you running the case for 12 hours.

[socieboy]

@Fiona.Chen @fanzh
Why would i wait 12hours when after 3 i’m getting almost 0FPS? GPU and CPU performance looks great at start, but after the 3 hours and when the FPS start dropping it just change with no sense.
The first screenshot shows the FPS when the camera just started, everything good (29.9fps).
The time was 15:11

By 16:54 got to 25FPS and will continue dropping to 0.
And GPU and CPU looks weird, Notice one of the CPU is 100%, but it changes from one to other.

[Fiona.Chen]

Which kind of camera are you using(CSI, USB or IP camera)? What is the format of your camera? How many camera sources are you using? How did you measure FPS?

[socieboy]

@Fiona.Chen GMSL Camera, I measure the frames as in the deepstream-app-3 using the PERF_DATA class.
I have up to 4 cameras in a Jetson AGX, but i have been doing test with a single camera and still have drops.

[socieboy]

Any help on this? I found out that removing the tracker eliminate the frame drops problem, but i don’t know if you have experimented this problem. Sharing with you in case it helps to find a solution

[socieboy]

@Fiona.Chen @fanzh any help on this?

[socieboy]

Just in case someone else is looking at this thread.
I got this issue resolved by adding some parameters on the docker run command

sudo docker run --runtime nvidia -it --rm --network host \
    --shm-size=16g \ 
    --ulimit memlock=-1 \
    --ulimit stack=67108864 \
    -e DISPLAY=$DISPLAY \

Basically i was missing those 3

    --shm-size=16g \ 
    --ulimit memlock=-1 \
    --ulimit stack=67108864 \