Fluctuating FPS using Gstreamer on Jetson Xavier NX

magnus.gabell · June 3, 2021, 1:17pm

Hi,

I use this:
gst-launch-1.0 v4l2src device=/dev/video0 ! ‘video/x-raw , width=(int)1600 , height=(int)1300 , format=(string)GRAY8 , framerate=(fraction)30/1’ ! nvvidconv ! ‘video/x-raw(memory:NVMM), width=(int)1600, height=(int)1300, format=I420’ ! tee name=t t. ! queue! nvjpegenc ! multifilesink location=‘/home/myuser/nvme/Images/Image01_%09d.jpg’ t. ! queue ! nvvidconv ! ‘video/x-raw, format=BGRx’ ! videoconvert ! ‘video/x-raw, format=BGR, width=(int)800 , height=(int)650’ ! videoconvert ! xvimagesink -ev

and one of my Jetson Xavier is not consistent in the FPS. In the beginning the tests we ran was about 4-5 hours and it stayed at about 55 fps. Then another day we got 30 fps. Today I tried again and got 59fps.

The thing is that another platform I have is very consistent and always produces 59.9 fps (camera can provide 60).
So any idea where I can start investigating why the performance is so varying? I really need a very stable fps. Any ideas what to look for.

I have set the run mode to 15W 6 core. I have not used jetson_clocks on this device. But it still CAN produce ~60fps.

Any ideas to move forward is appriciated

DaneLLL · June 4, 2021, 3:21am

Hi,
Please share your release version($ head -1 /etc/nv_tegra_release). And log of sudo tegrastats when the issue happens.

magnus.gabell · June 4, 2021, 5:42am

Hi, Thank you. I will reply with tegra release ASAP and
run tegrastats when we run next time. Most likley next week

R32 (release), REVISION: 5.1, GCID: 26202423, BOARD: t186ref, EABI: aarch64, DATE: Fri Feb 19 16:50:29 UTC 2021

Quick test. We ran now at 59fps. Then now starts to drop:

Number of files: 92958
fps: 43.794714118167505

Number of files: 97321
fps: 43.196540196053796

Number of files: 101447
fps: 42.66333421382525

total used free shared buff/cache available
Mem: 7765 1837 570 56 5357 5657
Swap: 3882 3 3878

total used free shared buff/cache available
Mem: 7765 1837 564 56 5363 5657
Swap: 3882 3 3878

total used free shared buff/cache available
Mem: 7765 1837 559 56 5368 5657
Swap: 3882 3 3878

RAM 2174/7765MB (lfb 42x4MB) SWAP 4/3883MB (cached 0MB) CPU [7%@1190,5%@1190,6%@1190,13%@1190,15%@1190,19%@1190] EMC_FREQ 0% GR3D_FREQ 8% AO@40.5C GPU@40C PMIC@100C AUX@41C CPU@42.5C thermal@41.15C
RAM 2175/7765MB (lfb 42x4MB) SWAP 4/3883MB (cached 0MB) CPU [8%@1190,7%@1190,8%@1190,10%@1190,19%@1190,17%@1190] EMC_FREQ 0% GR3D_FREQ 16% AO@40.5C GPU@40C PMIC@100C AUX@41C CPU@42C thermal@41C
RAM 2175/7765MB (lfb 42x4MB) SWAP 4/3883MB (cached 0MB) CPU [8%@1190,8%@1190,11%@1190,14%@1190,19%@1190,22%@1190] EMC_FREQ 0% GR3D_FREQ 0% AO@40.5C GPU@40C PMIC@100C AUX@41C CPU@42.5C thermal@41.15C
RAM 2175/7765MB (lfb 42x4MB) SWAP 4/3883MB (cached 0MB) CPU [7%@1190,8%@1190,9%@1190,12%@1190,20%@1190,20%@1190] EMC_FREQ 0% GR3D_FREQ 8% AO@40.5C GPU@40C PMIC@100C AUX@40.5C CPU@42C thermal@40.95C
RAM 2175/7765MB (lfb 42x4MB) SWAP 4/3883MB (cached 0MB) CPU [9%@1190,8%@1190,6%@1190,12%@1190,20%@1190,18%@1190] EMC_FREQ 0% GR3D_FREQ 0% AO@40.5C GPU@40C PMIC@100C AUX@41C CPU@42C thermal@40.95C
RAM 2173/7765MB (lfb 42x4MB) SWAP 4/3883MB (cached 0MB) CPU [6%@1420,7%@1420,7%@1420,9%@1420,21%@1420,17%@1420] EMC_FREQ 0% GR3D_FREQ 0% AO@40.5C GPU@40C PMIC@100C AUX@41C CPU@42C thermal@41C
RAM 2173/7765MB (lfb 42x4MB) SWAP 4/3883MB (cached 0MB) CPU [7%@1190,6%@1190,5%@1190,7%@1190,11%@1190,19%@1190] EMC_FREQ 0% GR3D_FREQ 0% AO@40.5C GPU@40C PMIC@100C AUX@41C CPU@42C thermal@40.65C
RAM 2173/7765MB (lfb 42x4MB) SWAP 4/3883MB (cached 0MB) CPU [8%@1420,8%@1420,11%@1420,12%@1420,20%@1420,22%@1420] EMC_FREQ 0% GR3D_FREQ 0% AO@40C GPU@40C PMIC@100C AUX@41C CPU@42C thermal@40.8C
RAM 2173/7765MB (lfb 42x4MB) SWAP 4/3883MB (cached 0MB) CPU [8%@1190,9%@1190,10%@1190,14%@1190,24%@1190,19%@1190] EMC_FREQ 0% GR3D_FREQ 30% AO@40C GPU@40C PMIC@100C AUX@40.5C CPU@42C thermal@41C
RAM 2173/7765MB (lfb 42x4MB) SWAP 4/3883MB (cached 0MB) CPU [8%@1190,10%@1190,7%@1190,13%@1190,18%@1190,17%@1190] EMC_FREQ 0% GR3D_FREQ 0% AO@40C GPU@40C PMIC@100C AUX@40.5C CPU@41.5C thermal@40.8C
RAM 2173/7765MB (lfb 42x4MB) SWAP 4/3883MB (cached 0MB) CPU [6%@1190,7%@1190,11%@1190,13%@1190,17%@1190,13%@1190] EMC_FREQ 0% GR3D_FREQ 24% AO@40C GPU@40C PMIC@100C AUX@40.5C CPU@42C thermal@40.8C

magnus.gabell · June 4, 2021, 7:12am

I activated jetson_clocks --fan to lower CPU-temp. New Result:

RAM 2185/7765MB (lfb 46x4MB) SWAP 6/3883MB (cached 0MB) CPU [4%@1420,7%@1420,11%@1420,7%@1420,18%@1420,14%@1420] EMC_FREQ 0% GR3D_FREQ 0% AO@31.5C GPU@31.5C PMIC@100C AUX@31.5C CPU@33C thermal@32.1C

FPS:

Number of files: 109673
fps: 41.9365846498617

magnus.gabell · June 4, 2021, 7:36am

I found that Tegra Release is 32.5.1. I Think that there might be a missmatch between the camera Jetvariety drivers and the Tegra version. I will check a third system if I have a problem.

magnus.gabell · June 4, 2021, 12:33pm

Checked my third Jetson. That one has R32.4.3 and it works good with stable 59fps

magnus.gabell · June 7, 2021, 2:09pm

I retried with my first Jetson that have always performed well. Using the Gstreamer above and the FPS has now dropped to 47… I do not understand why?

I rebooted and it did not affect the performance.
I then turned it off and turned it back on. Then my FPS jumped back to 57.

Any ideas what is the problem?

I have one issue with “failed to start local kernel modules” during boot. But that does not seem to affect anything.

magnus.gabell · June 7, 2021, 2:30pm

Its not related to Tegra release. got same problem with 32.4.3. As stated further down. Shutdown and waiting 5 min solved it. but is not a solution, just an observation.

DaneLLL · June 8, 2021, 5:36am

Hi,
It looks like you don’t run with sudo. Certain information(such as frequency) is not printed. Please execute sudo tegrastats.

And it probably helps to disable dynamic frequency of VIC engine. Please try
Camera's frame rate unstable - #24 by DaneLLL

magnus.gabell · June 8, 2021, 10:01am

Hi,
Here is some new data from tegra.

[11:56] Gabell, MagnusSudo Tegra stats
RAM 1947/7765MB (lfb 1057x4MB) SWAP 0/3883MB (cached 0MB) CPU [3%@1190,2%@1190,0%@1190,0%@1190,0%@1190,0%@1267] EMC_FREQ 1%@1600 GR3D_FREQ 0%@204 APE 150 MTS fg 0% bg 1% AO@39.5C GPU@39.5C PMIC@100C AUX@38.5C CPU@40.5C thermal@39.4C VDD_IN 4803/4803 VDD_CPU_GPU_CV 325/325 VDD_SOC 1221/1221
RAM 1947/7765MB (lfb 1057x4MB) SWAP 0/3883MB (cached 0MB) CPU [7%@1190,5%@1190,9%@1190,4%@1190,6%@1190,2%@1190] EMC_FREQ 1%@1600 GR3D_FREQ 27%@204 APE 150 MTS fg 0% bg 7% AO@39.5C GPU@39.5C PMIC@100C AUX@38.5C CPU@40.5C thermal@39.4C VDD_IN 5088/4945 VDD_CPU_GPU_CV 569/447 VDD_SOC 1261/1241
RAM 1947/7765MB (lfb 1057x4MB) SWAP 0/3883MB (cached 0MB) CPU [11%@1190,7%@1190,4%@1190,9%@1190,5%@1190,4%@1190] EMC_FREQ 2%@1600 GR3D_FREQ 0%@204 APE 150 MTS fg 0% bg 7% AO@39.5C GPU@39.5C PMIC@100C AUX@38.5C CPU@40.5C thermal@39.4C VDD_IN 5128/5006 VDD_CPU_GPU_CV 610/501 VDD_SOC 1261/1247
RAM 1947/7765MB (lfb 1057x4MB) SWAP 0/3883MB (cached 0MB) CPU [8%@1190,3%@1190,3%@1190,3%@1190,3%@1190,2%@1190] EMC_FREQ 2%@1600 GR3D_FREQ 0%@204 APE 150 MTS fg 0% bg 6% AO@39.5C GPU@39.5C PMIC@100C AUX@39C CPU@40.5C thermal@39.6C VDD_IN 4965/4996 VDD_CPU_GPU_CV 447/487 VDD_SOC 1219/1240
RAM 1947/7765MB (lfb 1057x4MB) SWAP 0/3883MB (cached 0MB) CPU [3%@1190,6%@1190,2%@1190,0%@1190,0%@1190,0%@1190] EMC_FREQ 1%@1600 GR3D_FREQ 0%@204 APE 150 MTS fg 0% bg 3% AO@39.5C GPU@39.5C PMIC@100C AUX@39C CPU@40.5C thermal@39.6C VDD_IN 4803/4957 VDD_CPU_GPU_CV 325/455 VDD_SOC 1219/1236
RAM 1947/7765MB (lfb 1057x4MB) SWAP 0/3883MB (cached 0MB) CPU [5%@1190,5%@1190,0%@1190,0%@1190,1%@1267,0%@1267] EMC_FREQ 1%@1600 GR3D_FREQ 0%@204 APE 150 MTS fg 0% bg 2% AO@40C GPU@40C PMIC@100C AUX@39C CPU@40.5C thermal@39.75C VDD_IN 4803/4931 VDD_CPU_GPU_CV 325/433 VDD_SOC 1221/1233
RAM 1947/7765MB (lfb 1057x4MB) SWAP 0/3883MB (cached 0MB) CPU [4%@1190,2%@1190,1%@1190,1%@1190,0%@1267,0%@1267] EMC_FREQ 1%@1600 GR3D_FREQ 0%@204 APE 150 MTS fg 0% bg 3% AO@40C GPU@40C PMIC@100C AUX@39C CPU@41C thermal@39.9C VDD_IN 4803/4913 VDD_CPU_GPU_CV 325/418 VDD_SOC 1219/1231
RAM 1947/7765MB (lfb 1057x4MB) SWAP 0/3883MB (cached 0MB) CPU [2%@1190,4%@1190,1%@1190,2%@1190,0%@1267,0%@1267] EMC_FREQ 1%@1600 GR3D_FREQ 0%@204 APE 150 MTS fg 0% bg 3% AO@39.5C GPU@40C PMIC@100C AUX@39C CPU@41C thermal@39.9C VDD_IN 4803/4899 VDD_CPU_GPU_CV 325/406 VDD_SOC 1221/1230
RAM 1947/7765MB (lfb 1057x4MB) SWAP 0/3883MB (cached 0MB) CPU [5%@1190,5%@1190,1%@1190,0%@1190,0%@1190,0%@1344] EMC_FREQ 1%@1600 GR3D_FREQ 0%@204 APE 150 MTS fg 0% bg 3% AO@40C GPU@40C PMIC@100C AUX@39.5C CPU@41C thermal@40.1C VDD_IN 4803/4888 VDD_CPU_GPU_CV 325/397 VDD_SOC 1219/1229
RAM 1947/7765MB (lfb 1057x4MB) SWAP 0/3883MB (cached 0MB) CPU [1%@1190,4%@1190,2%@1190,2%@1190,0%@1190,1%@1190] EMC_FREQ 1%@1600 GR3D_FREQ 0%@599 APE 150 MTS fg 0% bg 3% AO@40C GPU@40.5C PMIC@100C AUX@39.5C CPU@41C thermal@40.1C VDD_IN 4884/4888 VDD_CPU_GPU_CV 407/398 VDD_SOC 1221/1228
RAM 1947/7765MB (lfb 1057x4MB) SWAP 0/3883MB (cached 0MB) CPU [2%@1190,5%@1190,2%@1190,1%@1190,0%@1190,0%@1267] EMC_FREQ 1%@1600 GR3D_FREQ 0%@408 APE 150 MTS fg 0% bg 3% AO@40C GPU@40.5C PMIC@100C AUX@39.5C CPU@41C thermal@40.05C VDD_IN 4843/4884 VDD_CPU_GPU_CV 407/399 VDD_SOC 1221/1227
RAM 1948/7765MB (lfb 1057x4MB) SWAP 0/3883MB (cached 0MB) CPU [4%@1190,5%@1190,1%@1190,3%@1190,0%@1190,0%@1190] EMC_FREQ 1%@1600 GR3D_FREQ 0%@306 APE 150 MTS fg 0% bg 2% AO@40C GPU@40C PMIC@100C AUX@39.5C CPU@41C thermal@40.25C VDD_IN 4843/4880 VDD_CPU_GPU_CV 366/396 VDD_SOC 1221/1227
RAM 1948/7765MB (lfb 1057x4MB) SWAP 0/3883MB (cached 0MB) CPU [5%@1190,6%@1190,4%@1190,4%@1190,2%@1190,3%@1267] EMC_FREQ 2%@1600 GR3D_FREQ 16%@306 APE 150 MTS fg 0% bg 5% AO@40.5C GPU@40.5C PMIC@100C AUX@39.5C CPU@41.5C thermal@40.4C VDD_IN 5047/4893 VDD_CPU_GPU_CV 529/406 VDD_SOC 1261/1229
RAM 1948/7765MB (lfb 1057x4MB) SWAP 0/3883MB (cached 0MB) CPU [7%@1190,7%@1190,4%@1190,0%@1190,2%@1190,1%@1370] EMC_FREQ 1%@1600 GR3D_FREQ 8%@408 APE 150 MTS fg 0% bg 5% AO@40.5C GPU@40.5C PMIC@100C AUX@39.5C CPU@41.5C thermal@40.4C VDD_IN 4925/4895 VDD_CPU_GPU_CV 447/409 VDD_SOC 1221/1229
RAM 1948/7765MB (lfb 1057x4MB) SWAP 0/3883MB (cached 0MB) CPU [5%@1190,5%@1190,2%@1190,3%@1190,4%@1190,1%@1420] EMC_FREQ 1%@1600 GR3D_FREQ 0%@408 APE 150 MTS fg 0% bg 6% AO@40.5C GPU@40.5C PMIC@100C AUX@39.5C CPU@41.5C thermal@40.4C VDD_IN 4925/4897 VDD_CPU_GPU_CV 407/409 VDD_SOC 1219/1228

[11:57] Gabell, Magnus
First file: Image01_000000000.jpg
Last file: Image01_000000143.jpg
Number of files: 144
fps: 14.331345393690851
[11:58] Gabell, Magnus
First file: Image01_000000000.jpg
Last file: Image01_000000084.jpg
Number of files: 85
fps: 14.100086071960558

This is the Gstreamer:
gst-launch-1.0 v4l2src device=/dev/video0 ! ‘video/x-raw , width=(int)1600 , height=(int)1300 , format=(string)GRAY8 , framerate=(fraction)60/1’ ! nvvidconv ! ‘video/x-raw(memory:NVMM), width=(int)1600, height=(int)1300, format=I420’ ! queue! nvjpegenc ! multifilesink location=‘/home/aiadmin/nvme/Images/Image01_%09d.jpg’

I have not modified the VIC yet. Will do that now

magnus.gabell · June 8, 2021, 10:07am

Data from Jtop:

Just for reference…

magnus.gabell · June 8, 2021, 10:14am

Hi,

I found the file and correct section, but I dont know what to change.

Anyhow. Why is this affecting the FPS intermittent?
Sometimes I have stable 59FPS.
Sometimes I have sporadic start FPS and dropping.

I do not test to see FPS only on screen with a fpsdisplaysink but I have tested with only writing to disc and both writing to disc and with branching to appsink and neither affect the end result. Still sporadic results.

magnus.gabell · June 8, 2021, 1:25pm

Boot error after restart and system is really unstable.

With the risk of adding too much information:
bootlog.txt (307.4 KB)

magnus.gabell · June 8, 2021, 3:28pm

FYI: Start to think its hardware. I fiddle with the MIPI-cable and I get intermittet error when reading from camera.
Is there a way on jetson to monitor the CSI interface to see the data stream somehow?

DaneLLL · June 9, 2021, 2:04am

Hi,
You can run this command to check if the source is in steady 60fps:

gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-raw , width=(int)1600 , height=(int)1300 , format=(string)GRAY8 , framerate=(fraction)60/1' ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0 -v

magnus.gabell · June 9, 2021, 2:55pm

Hi,

Thank you very much for that. It was very helpful.
I tested two versions of measuring FPS

I had a hunch that the cable length is the problem becuase touching the camera, lifting and playing with it
caused the FPS to drop.
So after playing with first long 2m cable:

v4l2-ctl --set-fmt-video=width=1600,height=1300,pixelformat=‘GRAY8’ --stream-mmap --stream-count=-1 -d /dev/video0
fps_2m_1.txt (3.2 KB)

gst-launch-1.0 v4l2src device=/dev/video0 ! ‘video/x-raw , width=(int)1600 , height=(int)1300 , format=(string)GRAY8 , framerate=(fraction)60/1’ ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0 -v
fps_2m_2.txt (8.1 KB)

then with 1dm cable (std. rapsberry pi cable)

v4l2-ctl --set-fmt-video=width=1600,height=1300,pixelformat=‘GRAY8’ --stream-mmap --stream-count=-1 -d /dev/video0
fps_1dm_1.txt (2.9 KB)

gst-launch-1.0 v4l2src device=/dev/video0 ! ‘video/x-raw , width=(int)1600 , height=(int)1300 , format=(string)GRAY8 , framerate=(fraction)60/1’ ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0 -v
fps_1dm_2.txt (14.1 KB)

Conclusion:
This might still be some what premature, but I would say it is a signal problem between camera and Jetson. Its not Jetson related in terms of drivers or versions etc. It is all related to signal/grounding of the two systems.
It has very much the same behavior as Arduino when using push button and forget to add pull-up resistor to ensure good ground signal to the analogue input. When you moved your hand close to the ribbon we got interefernce of some sort seemingly causing the FPS to drop, but did not go totally crap until we picked it up and moved around.

I will buy more cables and try more with different lengths.

Thanks so much for the input. If you have more ideas how to monitor the signal to the CSI interface it would be appriciated. Do you know if we can ground the Jetson/camera somehow?

system · June 25, 2021, 1:05am

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.