Fps is not stable when use gstream on AGX Orin

hi, i got problem on Orin, please give me some advise, thanks.

4 12M 20fps csi cameras on AGX Orin.

fps is stable when use v4l2-ctl test command, we run 4 command to test 4 cameras in the same time:

v4l2-ctl --set-fmt-video=width=4128,height=3008,pixelformat=RG12 --set-ctrl bypass_mode=0 --stream-skip=0 --stream-mmap --stream-count=10000 --stream-to=/dev/null -d /dev/video0


<<<<<<<<<<<<<<<<<<<<< 19.99 fps
<<<<<<<<<<<<<<<<<<<< 19.99 fps
<<<<<<<<<<<<<<<<<<<< 19.99 fps
<<<<<<<<<<<<<<<<<<<< 19.99 fps
<<<<<<<<<<<<<<<<<<<< 19.99 fps
<<<<<<<<<<<<<<<<<<<< 19.99 fps

fps is unstable when use gstream nvargus command:

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! \
'video/x-raw(memory:NVMM),width=4128, height=3008, framerate=29/1, format=NV12' ! \
fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0 -v


/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2080, dropped: 0, current: 22.08, average: 18.75
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2087, dropped: 0, current: 13.05, average: 18.72
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2098, dropped: 0, current: 21.53, average: 18.74
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2104, dropped: 0, current: 11.80, average: 18.71
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2115, dropped: 0, current: 22.00, average: 18.72
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2121, dropped: 0, current: 11.95, average: 18.69
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2133, dropped: 0, current: 22.45, average: 18.71
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2142, dropped: 0, current: 17.76, average: 18.70
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2153, dropped: 0, current: 20.78, average: 18.71
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2159, dropped: 0, current: 11.99, average: 18.68
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2170, dropped: 0, current: 21.94, average: 18.70
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2175, dropped: 0, current: 9.81, average: 18.66
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2186, dropped: 0, current: 21.24, average: 18.67
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2193, dropped: 0, current: 13.67, average: 18.65
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2204, dropped: 0, current: 12.78, average: 18.61
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2212, dropped: 0, current: 13.70, average: 18.58
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2219, dropped: 0, current: 12.17, average: 18.55

but,when i just test 1 camera by gstream command ,the fps is ok;

hello panocam,

may I also know which Jetpack release version you’re working with?
please check release tag, $ cat /etc/nv_tegra_release for confirmation.

since you’re running with a 20-fps camera sensor.
you should also revise the gst pipeline to update the variable as same as your sensor capability,
for example, 'video/x-raw(memory:NVMM),width=4128, height=3008, framerate=20/1

there’s also commands to boost all the VI/CSI/ISP clocks.
please try system level configuration for confirmation.

sudo su
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/isp/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate

the version:

R35 (release), REVISION: 3.1, GCID: 32827747, BOARD: t186ref, EABI: aarch64, DATE: Sun Mar 19 15:19:21 UTC 2023

gstream command changed:

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=4128, height=3008, framerate=20/1, format=NV12' ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0 -v

boost all clocks

root@hl-desktop:/home/hl# echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
root@hl-desktop:/home/hl# echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
root@hl-desktop:/home/hl# echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
root@hl-desktop:/home/hl# echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
root@hl-desktop:/home/hl# cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/isp/rate
root@hl-desktop:/home/hl# cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/isp/rate
root@hl-desktop:/home/hl# cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
root@hl-desktop:/home/hl# cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate

result is same, fps is unstable.

hello panocam,

is this due to multi-cam use-case? do you still seeing issue with single camera?

yes, single camera works fine; fps unstable is on 4 cameras work .

hello panocam,

sorry, we don’t have four camera with 4K*3K resolution for testing.
could you please narrow down this, is this also repo’ed with 3-cam streaming? please also gather the results for reference.

there’s argus_camera application, which is able to enable mulit-session mode to launch all camera streams together.
please toggle the KPI options from the app, and check the results.

i will try 3 or 2 cameras case.

where is the argus_camera application, and how to set KPI options?

hello panocam,

you may install MMAPI package to obtain the sources, i.e. $ sudo apt install nvidia-l4t-jetson-multimedia-api
you’ll see the sources available after install MMAPI package, i.e. /usr/src/jetson_multimedia_api/argus/
please check the README.TXT for the steps to build the samples and argus_camera application, there’s UI to enable the KPI report.

i build argus_camera success and run it, after enable KPI ,got the result as the screenshot.

PerfTracker 7 ~ 10 is camera 0 ~3 ?

if PerfTracker 7 ~ 10 are camera 0 ~3, the fps is stable.

so, the argus_camera get stable fps, but gstream not, there is some sample pipline can work on 4 cameras with stable fps?

there is other infos:
run argus_camera, VIC on jtop display is OFF;
run gst command, VIC is running.

hello panocam,

as you can see, there’re different PerfTrackers to evaluate the KPI.
it’s PerfTracker 7 ~ 10 for tracking camera 0 ~3 frame-rate. it looks stable by argus_camera application.

there’re some stability issue with nvarguscamerasrc, we’ve fixes check-in to address that. please expect next Jetpack public release will include the fixes.

hi JerryChang,
When will the next version be released?

it should be available soon, it’s target early Aug. 2023.

that’s good news, thanks.

another thing, the argus_camera look like not use the VIC, the jtop display VIC is OFF, why?

The JetPack 5.1.2 has been released, please help to get it a try. Thanks

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.