Fps is not stable when use gstream on AGX Orin

[panocam]

hi, i got problem on Orin, please give me some advise, thanks.

enviroment:
4 12M 20fps csi cameras on AGX Orin.

fps is stable when use v4l2-ctl test command, we run 4 command to test 4 cameras in the same time:

v4l2-ctl --set-fmt-video=width=4128,height=3008,pixelformat=RG12 --set-ctrl bypass_mode=0 --stream-skip=0 --stream-mmap --stream-count=10000 --stream-to=/dev/null -d /dev/video0

result:

<<<<<<<<<<<<<<<<<<<<< 19.99 fps
<<<<<<<<<<<<<<<<<<<< 19.99 fps
<<<<<<<<<<<<<<<<<<<< 19.99 fps
<<<<<<<<<<<<<<<<<<<< 19.99 fps
<<<<<<<<<<<<<<<<<<<< 19.99 fps
<<<<<<<<<<<<<<<<<<<< 19.99 fps

fps is unstable when use gstream nvargus command:

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! \
'video/x-raw(memory:NVMM),width=4128, height=3008, framerate=29/1, format=NV12' ! \
fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0 -v

result:

/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2080, dropped: 0, current: 22.08, average: 18.75
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2087, dropped: 0, current: 13.05, average: 18.72
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2098, dropped: 0, current: 21.53, average: 18.74
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2104, dropped: 0, current: 11.80, average: 18.71
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2115, dropped: 0, current: 22.00, average: 18.72
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2121, dropped: 0, current: 11.95, average: 18.69
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2133, dropped: 0, current: 22.45, average: 18.71
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2142, dropped: 0, current: 17.76, average: 18.70
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2153, dropped: 0, current: 20.78, average: 18.71
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2159, dropped: 0, current: 11.99, average: 18.68
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2170, dropped: 0, current: 21.94, average: 18.70
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2175, dropped: 0, current: 9.81, average: 18.66
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2186, dropped: 0, current: 21.24, average: 18.67
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2193, dropped: 0, current: 13.67, average: 18.65
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2204, dropped: 0, current: 12.78, average: 18.61
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2212, dropped: 0, current: 13.70, average: 18.58
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2219, dropped: 0, current: 12.17, average: 18.55

but,when i just test 1 camera by gstream command ,the fps is ok;

[JerryChang]

hello panocam,

may I also know which Jetpack release version you’re working with?
please check release tag, $ cat /etc/nv_tegra_release for confirmation.

since you’re running with a 20-fps camera sensor.
you should also revise the gst pipeline to update the variable as same as your sensor capability,
for example, 'video/x-raw(memory:NVMM),width=4128, height=3008, framerate=20/1

there’s also commands to boost all the VI/CSI/ISP clocks.
please try system level configuration for confirmation.

sudo su
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/isp/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate

[panocam]

the version:

R35 (release), REVISION: 3.1, GCID: 32827747, BOARD: t186ref, EABI: aarch64, DATE: Sun Mar 19 15:19:21 UTC 2023

gstream command changed:

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=4128, height=3008, framerate=20/1, format=NV12' ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0 -v

boost all clocks

root@hl-desktop:/home/hl# echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
root@hl-desktop:/home/hl# echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
root@hl-desktop:/home/hl# echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
root@hl-desktop:/home/hl# echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
root@hl-desktop:/home/hl# cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
832000000
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/isp/rate
root@hl-desktop:/home/hl# cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/isp/rate
1011200000
root@hl-desktop:/home/hl# cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
642900000
root@hl-desktop:/home/hl# cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate
3199000000

result is same, fps is unstable.

[JerryChang]

hello panocam,

is this due to multi-cam use-case? do you still seeing issue with single camera?

[panocam]

yes, single camera works fine; fps unstable is on 4 cameras work .

[JerryChang]

hello panocam,

sorry, we don’t have four camera with 4K*3K resolution for testing.
could you please narrow down this, is this also repo’ed with 3-cam streaming? please also gather the results for reference.

besides,
there’s argus_camera application, which is able to enable mulit-session mode to launch all camera streams together.
please toggle the KPI options from the app, and check the results.

[panocam]

i will try 3 or 2 cameras case.

where is the argus_camera application, and how to set KPI options?

[JerryChang]

hello panocam,

you may install MMAPI package to obtain the sources, i.e. $ sudo apt install nvidia-l4t-jetson-multimedia-api
you’ll see the sources available after install MMAPI package, i.e. /usr/src/jetson_multimedia_api/argus/
please check the README.TXT for the steps to build the samples and argus_camera application, there’s UI to enable the KPI report.

[panocam]

i build argus_camera success and run it, after enable KPI ,got the result as the screenshot.

PerfTracker 7 ~ 10 is camera 0 ~3 ?

[panocam]

if PerfTracker 7 ~ 10 are camera 0 ~3, the fps is stable.

so, the argus_camera get stable fps, but gstream not, there is some sample pipline can work on 4 cameras with stable fps?

[panocam]

there is other infos:
run argus_camera, VIC on jtop display is OFF;
run gst command, VIC is running.

[JerryChang]

hello panocam,

as you can see, there’re different PerfTrackers to evaluate the KPI.
it’s PerfTracker 7 ~ 10 for tracking camera 0 ~3 frame-rate. it looks stable by argus_camera application.

FYI,
there’re some stability issue with nvarguscamerasrc, we’ve fixes check-in to address that. please expect next Jetpack public release will include the fixes.

[panocam]

hi JerryChang,
When will the next version be released?

[JerryChang]

it should be available soon, it’s target early Aug. 2023.

[panocam]

that’s good news, thanks.

another thing, the argus_camera look like not use the VIC, the jtop display VIC is OFF, why?

[kayccc]

The JetPack 5.1.2 has been released, please help to get it a try. Thanks