Camera frame rate low

hi,all:
our configure is ar0231 + isp + max96705 + max96712 to get the camera video。

We used Jetpack 5.1.2 and followed the instructions in the link above to use Topic268833_JP-512_rce-fw_Debug.7z updated camera firmware

We can configure the camera normally and also obtain images normally。

The trigger frequency of the camera is 20Hz。

However, in very rare cases, one camera has a frame rate lower than 20HZ, while the other cameras have a frame rate of 20HZ.All cameras have the same trigger signal.

I turned on the debug mode I used the v4l-ctl tool to retrieve images and recorded some trace logs.

echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 2 > /sys/kernel/debug/camrtc/log-level
echo 1 > /sys/kernel/debug/tracing/events/camera_common/enable
echo > /sys/kernel/debug/tracing/trace
cat /sys/kernel/debug/tracing/trace

I roughly looked at the low frame rate log of the camera, and in the following image, I found that there was no SOF and EOF in the middle:

Can you help me see some issues from the log? I want to know if ORIN CSI did not receive data or received incorrect data when the camera frame rate was below 20Hz?

The following two files, one is a log with a normal camera frame rate, and the other is a log with a low frame rate.
video4-ok.log (11.7 MB)
vidoe6-low-hz.log (12.4 MB)

BTW,the phenomenon of low frame rate in cameras does not always occur, but occasionally occurs.

before digging into this, may I know what’s the environment setups, or what’s the background services are running?
what’s the failure rate, will the frame-rate back to normal after a short while?

Thank you for your reply.

No camera related services are running,CPU load is around 70%,
Obtaining images using v4l2-ctl

v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat=YUYV --stream-mmap -d /dev/video4

The failure rate is approximately 50%, Once a problem occurs, the frame rate will not return to normal in the short term.

Perhaps you can take a look at the log and tell me if ORIN CSI received any data when the frame was lost? So I can check if max96712 is malfunctioning.

Looking forward to your reply, thank you!

hello zijie.wang2020,

we may ignore those CHANSEL_NOMATCH reported, since it’s YUV sensor via GMSL, and your functional camera also report the same.
so… there’s no suspect errors according to your logs.

debug logs also consume system resources.
it’s suggest to disable VI tracing logs, and not using debug version rce-fw to monitor the frame-rates.
may I also know how your monitor the frame-rate? are you using v4l pipeline to keep fetching the frames?

Using v4l2-ctl to Calculate Camera Frame Rate

v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat=YUYV --stream-mmap -d /dev/video7

hello zijie.wang2020,

this should be minor, please add --set-ctrl bypass_mode=0 to the v4l pipeline for testing.
may I know how many camera streams you’ve enabled concurrently?

furthermore,
please configure system settings to have verification,
you may follow below commands to boost all the VI/CSI/ISP clocks.

sudo su
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/isp/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate

I enabled a stream.

Does this mean that the data provided by max96712 to ORIN CSI is frame loss?

hello zijie.wang2020,

yeah… it may be. is it always the same video stream with such frame-drop?
please see-also check design guide to review [MIPI CSI D-PHY Interface Signal Routing Requirements] for the trace length.