My program slows down over time

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
1

Hi!

I am using a program to get the frames from the camera, then I process the image and I reestream it.
Everything is fine at startup, but when the program has been running ~30 minutes, I see how the video slow down, I even start to see the video start to lose some frames and don’t recover the delay.

The pipeline that I’m using right now is the following:

rtspsrc location=… latency=0 drop-on-latency=true ! application/x-rtp,encoding-name=H265,payload=96,framerate=20/1 ! queue max-size-bytes=0 max-size-time=250 ! rtph265depay ! video/x-h265, stream-format=byte-stream,framerate=20/1 ! nvv4l2decoder enable-max-performance=true drop-frame-interval=2 ! queue max-size-bytes=0 max-size-time=250 ! nvvidconv output-buffers=5 name=myconv ! video/x-raw(memory:NVMM), format=RGBA ! nvvidconv output-buffers=5 ! video/x-raw(memory:NVMM), format=NV12 ! queue max-size-bytes=0 max-size-time=250 ! nvv4l2vp9enc maxperf-enable=true bitrate=3000000 ! video/x-vp9 ! rtpvp9pay ! udpsink host=224.1.1.1 port=5000 sync=false auto-multicast=true

I have activated the jetson_clocks and put nvpmodel -m 0 before starting my program.
Maybe I have something wrong in my pipeline??

Thanks in advance.

3

Hi,
Probably the performance is capped by certain thermal condition. The thermal status is shown in executing sudo tegrastats. Please run it and check the status when the issue happens. Might provide some clue.

4

Hi,
I’m monitoring the state of the board with the sudo tegrastats command, I don’t see severe changes in the state or temperature, I leave here the command output when I have a severe slowdown of the frames.

RAM 10222/31919MB (lfb 3473x4MB) SWAP 0/15959MB (cached 0MB) CPU [21%@2265,21%@2265,18%@2265,69%@2265,19%@2265,20%@2265,15%@2265,15%@2265] EMC_FREQ 13%@2133 GR3D_FREQ 12%@1377 NVENC 1075 NVENC1 1075 NVDEC 1190 NVDEC1 1190 APE 150 MTS fg 0% bg 19% AO@40C GPU@41C Tdiode@42.5C PMIC@100C AUX@40C CPU@42C thermal@40.9C Tboard@39C GPU 2471/2506 CPU 3400/3176 SOC 5100/5116 CV 0/0 VDDRQ 1081/1012 SYS5V 3044/3026

To give more information, when I have this state, I see jumps in the mouse pointer and I see how the processed frames never get the same frames that it processed during the first 30 minutes.

Any ideas?

5

Hi,
Please break down the pipeline to find out the bottleneck. You may try video playback first:

gst-launch-1.0 rtspsrc location=… latency=0 drop-on-latency=true ! application/x-rtp,encoding-name=H265,payload=96,framerate=20/1 ! queue max-size-bytes=0 max-size-time=250 ! rtph265depay ! video/x-h265, stream-format=byte-stream,framerate=20/1 ! nvv4l2decoder enable-max-performance=true drop-frame-interval=2 ! nvoverlaysink

And try to remove some properties such as latency=0, drop-on-latency=true, max-size-bytes=0, max-size-time=250, drop-frame-interval=2, …

7

If I use this pipeline I haven’t that problem, but I see a microcuts in the stream.
I have seen that when my program takes to do something for some reason, it is when the delay is created in the image, I think that these frames will be stored in some buffer, is there a way to empty this buffer so that this delay isn’t created?

8

Hi,
The nvvidconv plugin is to utilize VIC engine. It may help to run the engine in max clock. Please try

9

I try with the following commands:

su
echo on > /sys/devices/13e10000.host1x/15340000.vic/power/control
echo userspace > /sys/devices/13e10000.host1x/15340000.vic/devfreq/15340000.vic/governor
cat /sys/devices/13e10000.host1x/15340000.vic/devfreq/15340000.vic/available_frequencies
115200000 268800000 409600000 550400000 691200000 844800000 985600000 1036800000
echo 1036800000 > /sys/devices/13e10000.host1x/15340000.vic/devfreq/15340000.vic/max_freq
echo 1036800000 > /sys/devices/13e10000.host1x/15340000.vic/devfreq/15340000.vic/userspace/set_freq

I put it without any criteria because I don’t really understand what these options really do and I don’t know what is the best frequency that I have to choose.
But after those commands I see that the problem does not go away, but appears later, I’m not sure when, but I see in my log that the frames slow down to 21 when in the first moments of the program it was in 25-28.

10

I see the following behavior, at 3 hours it starts to fail.
I have a log that tells me when the frames slow down below 22 frames.

Time: 18-39-42; Frames: 0
Time: 18-39-43; Frames: 2
Time: 21-29-41; Frames: 19
Time: 21-33-47; Frames: 18
Time: 21-33-51; Frames: 22
Time: 21-35-8; Frames: 20
Time: 21-35-9; Frames: 12
Time: 21-35-11; Frames: 14
Time: 21-35-24; Frames: 16
Time: 21-37-22; Frames: 19
Time: 21-41-26; Frames: 15
Time: 21-41-31; Frames: 19
Time: 21-41-34; Frames: 15
Time: 21-41-40; Frames: 22
Time: 21-41-41; Frames: 20
Time: 21-41-44; Frames: 22
Time: 21-41-46; Frames: 19
Time: 21-41-53; Frames: 21
Time: 21-42-32; Frames: 22
Time: 21-42-37; Frames: 16
Time: 21-42-42; Frames: 1
Time: 21-42-43; Frames: 22
Time: 21-42-46; Frames: 15
Time: 21-42-47; Frames: 3
Time: 21-42-48; Frames: 11
Time: 21-42-55; Frames: 13
Time: 21-42-56; Frames: 7
Time: 21-43-14; Frames: 21

My program always performs the same tasks in these tests.

1 Like
13

hi - I am having a very similar problem. I stream images through the inference engine and over time everything on the jetson runs really slow and eventually seems totally locked up. I am monitoring top and tegrastats and until the point where these stop updating everything seems normal. If I kill my python service that is running the inferences, before it totally locks up, then it everything else recovers.

Did you solve your probelm?