V4l2-ctl not respecting frame_rate control, unlike Gstreamer on IMX477 + Nano 2GB

I am looking to use v4l2 for long exposure captures (which will require further driver / DTB tweaks). However, it looks like I am seeing a few issues which prevent me from further testing raw bayer long exposures. Note that I’m exclusively testing on a Nano 2GB Dev Kit + Arducam IMX477 camera module (both regular + mini) - across both Nvidia stock and Arducam drivers.

$ cat /etc/nv_tegra_release
# R32 (release), REVISION: 6.1, GCID: 27863751, BOARD: t210ref, EABI: aarch64, DATE: Mon Jul 26 19:20:30 UTC 2021
$ uname -a
Linux developer-desktop 4.9.253-tegra #1 SMP PREEMPT Sat Oct 9 07:43:58 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

1). Regardless of using the Nvidia / Arducam driver, v4l2-ctl does not seem to respect set frame_rate control. On the stock Nvidia drivers, the commands below result in 60fps and 30fps output, respectfully. One interesting behavior is that does seem to respect frame_rate control after the stream has started if you run the command from another shell: v4l2-ctl -c frame_rate=10000000

v4l2-ctl -d /dev/video0 --verbose --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0,sensor_mode=1,frame_rate=10 --stream-mmap --stream-count=100
v4l2-ctl -d /dev/video0 --verbose --set-fmt-video=width=3840,height=2160,pixelformat=RG10 --set-ctrl bypass_mode=0,sensor_mode=0,frame_rate=10 --stream-mmap --stream-count=100

2). Interestingly, Gstreamer does respect the user-defined frame rate running the following command (and this applies across all sensor modes, from their device tree specified min and max framerates, and even works at 4032x3040 from 2fps → 30fps when running the Arducam driver):

gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 sensor-mode=0 ispdigitalgainrange='1 1' num-buffers=300 wbmode=3 aelock=true gainrange='1 1' exposuretimerange='300000000 300000000' ! 'video/x-raw(memory:NVMM),width=3840,height=2160,framerate=10/1' ! nvvidconv ! nvv4l2h265enc bitrate=100000000 ! h265parse ! mp4mux ! filesink location=out.mp4

3). v4l2 doesn’t seem to be able to lower the framerate to the specified minimum in the driver / DTB of 2fps. It appears 5fps is the minimum across all modes (including the additional 4032x3040 mode with the Arducam driver). Gstreamer, however, is able to use frame rates down to the specified min.

v4l2-ctl -d /dev/video0 --verbose --set-fmt-video=width=3840,height=2160,pixelformat=RG10 --set-ctrl bypass_mode=0,sensor_mode=0,frame_rate=2 --stream-mmap --stream-count=100
gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 sensor-mode=0 ispdigitalgainrange='1 1' num-buffers=30 wbmode=3 aelock=true gainrange='1 1' exposuretimerange='300000000 300000000' ! 'video/x-raw(memory:NVMM),width=3040,height=2160,framerate=2/1' ! nvvidconv ! nvv4l2h265enc bitrate=100000000 ! h265parse ! mp4mux ! filesink location=out.mp4

4). v4l2 commands sometimes break Gstreamer until the next reboot, and vice versa. Though this doesn’t always happen… Gstreamer seems to break v4l2 more than the other way around. I haven’t isolated whether this behavior is driver dependent. When v4l2 is broken, it will throw this error when running the commands from above and throw this error: video4linux video0: frame start syncpt timeout!0 Note there is a potentially related issue where v4l2 doesn’t ever work, even on the stock Nvidia drivers. It seems to vary from board to board (or camera to camera) and has yet to be isolated. See here for more details.

5). There’s also the curiosity of how Gstreamer is able to handle 4032x3040 30fps whereas v4l2 is not (when running the Arducam drivers). Based on the Nano’s MIPI CSI specs, 4032x3040 at 30fps seems to be pushing too much bandwidth, but somehow Gstreamer is able to handle it with no issues. The fact that Gstreamer can handle 4032x3040 at 30fps (and v4l2 can’t) doesn’t seem to make much sense to me as the MIPI CSI transfer occurs before the ISP processing. Any info on why this is happening would be appreciated.

Let me know if there’s any further info/debug that would be helpful. Thanks!

1, Looks like the value set by --set-ctrl isn’t fps.

static int imx477_set_frame_rate(struct tegracam_device *tc_dev, s64 val)
213  {
214  	struct camera_common_data *s_data = tc_dev->s_data;
215  	struct imx477 *priv = (struct imx477 *)tc_dev->priv;
216  	struct device *dev = tc_dev->dev;
217  	const struct sensor_mode_properties *mode =
218  	    &s_data->sensor_props.sensor_modes[s_data->mode_prop_idx];
219  
220  	int err = 0;
221  	imx477_reg fl_regs[2];
222  	u32 frame_length;
223  	int i;
224  
225  	dev_dbg(dev, "%s: Setting framerate control to: %lld\n", __func__, val);
226  
227  	frame_length = (u32) (mode->signal_properties.pixel_clock.val *
228  			      (u64) mode->control_properties.framerate_factor /
229  			      mode->image_properties.line_length / val);
230  
231  	if (frame_length < IMX477_MIN_FRAME_LENGTH)
232  		frame_length = IMX477_MIN_FRAME_LENGTH;
233  	else if (frame_length > IMX477_MAX_FRAME_LENGTH)
234  		frame_length = IMX477_MAX_FRAME_LENGTH;
235  
236  	dev_dbg(dev,
237  		"%s: val: %llde-6 [fps], frame_length: %u [lines]\n",
238  		__func__, val, frame_length);

  1. The framerate here isn’t for the framerate control it’s for sensor mode selected reference. For framerate control you can check the argus_camera from the GUI.
  2. The same as above.
  3. If you modify modify frame_rate to change the frame_length may cause capture failed to impact next capture.
  4. Does IMX477 support 4032x3040?

Thanks for the quick reply Shane.

Ok, but shouldn’t v4l2-ctl allow for correctly setting frame rate as does nvarguscamerasrc via Gstreamer does? Would this not be considered a bug in the v4l2 drivers/implementation? The control seems to almost work correctly - per above, it requires setting from separate shell and is limited to 5fps vs. 2fps defined in the drivers/DTB. Given the importance of framerate control, it would be great if this could be considered a bug and fixed so there is parity between v4l2 and Argus.

Ok, but doesn’t frame length change when changing framerate using nvarguscamerasrc via Gstreamer (not using v4l2 at all)?

Yes, it does. On the Nano 2GB with the Arducam drivers it is exposed as Mode 0 at 30fps and works perfectly fine in Gstreamer with nvarguscamerasrc (and this 4:3 sensor ratio happens to be a requirement for our use-case). Interestingly, that same mode doesn’t work in v4l2-ctl and maybe that is the reason Nvidia didn’t include it in the stock Nvidia drivers (which only support 3840x2160 at 30fps and 1920x1080 at 60fps). Arducam has a test driver that shows that 4032x3040 at 15fps works in v4l2-ctl. Though it’s still a bit of a mystery to me as to how v4l2 runs into MIPI CSI bandwidth issues whereas the ISP/Argus pipeline does not. It would be great to see 4032x3040 added to the official Nvidia drivers as I’m sure there will be users that need access to the 4:3 aspect ratio of the IMX477. Here’s the output of v4l2-ctl --list-formats-ext when running the latest Arducam drivers:

ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'RG10'
	Name        : 10-bit Bayer RGRG/GBGB
		Size: Discrete 4032x3040
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 3840x2160
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.017s (60.000 fps)

Can you check the value from imx477 while using argus_camera to set the frame value then check the same value with v4l2-ctl to confirm it.

Please share the code for 4032x3040 this mode and will check if can merge it.

Does that require rebuilding the kernel to enable those dynamic debug messages? Ie. Does it require this kernel setting:

-> Device Drivers
    -> Multimedia support
        [*] Enable advanced debug functionality on V4L2 drivers

I have asked the folks at Arducam if they can share. Hopefully that can happen and this code can be merged.

So I noticed that I was setting framerate in the wrong scale above (10 vs. 10000000) for v4l2-ctl, however, the behavior doesn’t change. v4l2 will show the frame rate being set in the logs, but it is still using the sensor mode’s default frame rate based on the output. Here’s the dev_dbg log from v4l2-ctl:

$ v4l2-ctl -d /dev/video0 --verbose --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0,sensor_mode=1,frame_rate=10000000 --stream-mmap --stream-count=30
[  908.219074] imx477 6-001a: imx477_power_on: power on
[  908.519223] imx477 6-001a: imx477_set_mode:
[  908.599513] imx477 6-001a: imx477_start_streaming:
[  908.599840] imx477 6-001a: imx477_stop_streaming:
[  908.600051] imx477 6-001a: imx477_power_off: power off
[  908.600179] imx477 6-001a: imx477_power_on: power on
[  908.900855] imx477 6-001a: imx477_set_frame_rate: Setting framerate control to: 10000000
[  908.900861] imx477 6-001a: imx477_set_frame_rate: val: 10000000e-6 [fps], frame_length: 9308 [lines]
[  908.915705] imx477 6-001a: imx477_set_mode:
[  908.996505] imx477 6-001a: imx477_start_streaming:
[  914.049501] imx477 6-001a: imx477_stop_streaming:
[  914.052581] imx477 6-001a: imx477_power_off: power off

Which results in 60fps recording. On the second run of that exact same command I see:

[  963.770077] imx477 6-001a: imx477_power_on: power on
[  964.070222] imx477 6-001a: imx477_set_mode:
[  964.150181] imx477 6-001a: imx477_start_streaming:
[  964.150510] imx477 6-001a: imx477_stop_streaming:
[  964.150721] imx477 6-001a: imx477_power_off: power off
[  964.150848] imx477 6-001a: imx477_power_on: power on
[  964.466248] imx477 6-001a: imx477_set_mode:
[  964.546001] imx477 6-001a: imx477_start_streaming:
[  969.598994] imx477 6-001a: imx477_stop_streaming:
[  969.601998] imx477 6-001a: imx477_power_off: power off

Which also results in 60fps recording (but no log mention of frame rate). Only when the requested frame rate is different from the previous run will I see it in the logs (but still the output is incorrect at 60fps). And when I set frame rate during streaming from another shell with v4l2-ctl -c frame_rate=10000000 I correctly see the stream recording frame rate slow down (but not to the actual requested rate - see post below).

[   88.825276] imx477 6-001a: imx477_power_on: power on
[   89.125423] imx477 6-001a: imx477_set_mode:
[   89.205120] imx477 6-001a: imx477_start_streaming:
[   89.205449] imx477 6-001a: imx477_stop_streaming:
[   89.205658] imx477 6-001a: imx477_power_off: power off
[   89.205790] imx477 6-001a: imx477_power_on: power on
[   89.521017] imx477 6-001a: imx477_set_mode:
[   89.600058] imx477 6-001a: imx477_start_streaming:
[   90.153185] imx477 6-001a: imx477_stop_streaming:
[   90.156182] imx477 6-001a: imx477_power_off: power off
[  274.808017] imx477 6-001a: imx477_power_on: power on
[  275.108168] imx477 6-001a: imx477_set_mode:
[  275.187854] imx477 6-001a: imx477_start_streaming:
[  275.188181] imx477 6-001a: imx477_stop_streaming:
[  275.188390] imx477 6-001a: imx477_power_off: power off
[  275.188521] imx477 6-001a: imx477_power_on: power on
[  275.503706] imx477 6-001a: imx477_set_mode:
[  275.582976] imx477 6-001a: imx477_start_streaming:
[  280.415236] imx477 6-001a: imx477_set_frame_rate: Setting framerate control to: 10000000
[  280.415243] imx477 6-001a: imx477_set_frame_rate: val: 10000000e-6 [fps], frame_length: 9308 [lines]
[  335.600966] imx477 6-001a: imx477_stop_streaming:
[  335.603836] imx477 6-001a: imx477_power_off: power off

And from Gstreamer where setting frame rate is always honored correctly, even on repeated runs:

$ gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 sensor-mode=1 ispdigitalgainrange='1 1' num-buffers=30 wbmode=3 aelock=true gainrange='1 1' exposuretimerange='300000000 300000000' ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=10/1' ! nvvidconv ! nvv4l2h265enc bitrate=100000000 ! h265parse ! mp4mux ! filesink location=out.mp4
[  338.604029] imx477 6-001a: imx477_power_on: power on
[  338.904175] imx477 6-001a: imx477_set_mode:
[  338.984620] imx477 6-001a: imx477_set_gain: Setting gain control to: 352
[  338.984627] imx477 6-001a: imx477_set_gain: val: 352 (/16) [times], gain: 978
[  338.985003] imx477 6-001a: imx477_set_exposure: Setting exposure control to: 70000
[  338.985008] imx477 6-001a: imx477_set_exposure: val: 70000 [us], coarse_time: 6515 [lines]
[  338.985382] imx477 6-001a: imx477_set_frame_rate: Setting framerate control to: 10000000
[  338.985387] imx477 6-001a: imx477_set_frame_rate: val: 10000000e-6 [fps], frame_length: 9308 [lines]
[  338.985757] imx477 6-001a: imx477_start_streaming:
[  338.986081] imx477 6-001a: imx477_stop_streaming:
[  338.986290] imx477 6-001a: imx477_power_off: power off
[  338.986420] imx477 6-001a: imx477_power_on: power on
[  339.286674] imx477 6-001a: imx477_power_off: power off
[  339.288999] imx477 6-001a: imx477_power_on: power on
[  339.589254] imx477 6-001a: imx477_power_off: power off
[  339.616226] imx477 6-001a: imx477_power_on: power on
[  339.916391] imx477 6-001a: imx477_power_off: power off
[  339.916597] imx477 6-001a: imx477_power_on: power on
[  340.216815] imx477 6-001a: imx477_power_off: power off
[  340.217007] imx477 6-001a: imx477_power_on: power on
[  340.517312] imx477 6-001a: imx477_power_off: power off
[  340.741685] imx477 6-001a: imx477_power_on: power on
[  341.042016] imx477 6-001a: imx477_set_group_hold: Setting group hold control to: 1
[  341.042274] imx477 6-001a: imx477_set_frame_rate: Setting framerate control to: 20000000
[  341.042279] imx477 6-001a: imx477_set_frame_rate: val: 20000000e-6 [fps], frame_length: 4654 [lines]
[  341.042736] imx477 6-001a: imx477_set_gain: Setting gain control to: 16
[  341.042741] imx477 6-001a: imx477_set_gain: val: 16 (/16) [times], gain: 0
[  341.043154] imx477 6-001a: imx477_set_group_hold: Setting group hold control to: 0
[  341.043586] imx477 6-001a: imx477_set_group_hold: Setting group hold control to: 1
[  341.043819] imx477 6-001a: imx477_set_frame_rate: Setting framerate control to: 10000000
[  341.043824] imx477 6-001a: imx477_set_frame_rate: val: 10000000e-6 [fps], frame_length: 9308 [lines]
[  341.044223] imx477 6-001a: imx477_set_exposure: Setting exposure control to: 33333
[  341.044228] imx477 6-001a: imx477_set_exposure: val: 33333 [us], coarse_time: 3102 [lines]
[  341.044623] imx477 6-001a: imx477_set_group_hold: Setting group hold control to: 0
[  341.044924] imx477 6-001a: imx477_set_mode:
[  341.127173] imx477 6-001a: imx477_set_gain: Setting gain control to: 16
[  341.127179] imx477 6-001a: imx477_set_gain: val: 16 (/16) [times], gain: 0
[  341.127557] imx477 6-001a: imx477_set_exposure: Setting exposure control to: 33333
[  341.127562] imx477 6-001a: imx477_set_exposure: val: 33333 [us], coarse_time: 3102 [lines]
[  341.127937] imx477 6-001a: imx477_set_frame_rate: Setting framerate control to: 10000000
[  341.127942] imx477 6-001a: imx477_set_frame_rate: val: 10000000e-6 [fps], frame_length: 9308 [lines]
[  341.128313] imx477 6-001a: imx477_start_streaming:
[  341.129746] imx477 6-001a: imx477_set_group_hold: Setting group hold control to: 1
[  341.129985] imx477 6-001a: imx477_set_frame_rate: Setting framerate control to: 20000000
[  341.129989] imx477 6-001a: imx477_set_frame_rate: val: 20000000e-6 [fps], frame_length: 4654 [lines]
[  341.130385] imx477 6-001a: imx477_set_group_hold: Setting group hold control to: 0
[  341.146964] imx477 6-001a: imx477_set_group_hold: Setting group hold control to: 1
[  341.147203] imx477 6-001a: imx477_set_frame_rate: Setting framerate control to: 10000000
[  341.147208] imx477 6-001a: imx477_set_frame_rate: val: 10000000e-6 [fps], frame_length: 9308 [lines]
[  341.147606] imx477 6-001a: imx477_set_group_hold: Setting group hold control to: 0
[  341.148242] misc tegra_camera_ctrl: ISO BW req 1671345 > 1500000 (max) capping to max
[  341.148248] misc tegra_camera_ctrl: tegra_camera_update_isobw: Warning, Requested ISO BW 1500000 has been capped to VI's max BW 1500000
[  343.725403] imx477 6-001a: imx477_stop_streaming:
[  343.725659] misc tegra_camera_ctrl: tegra_camera_update_isobw: Warning, Requested ISO BW 1500000 has been capped to VI's max BW 1500000
[  343.728787] imx477 6-001a: imx477_power_off: power off

With Gstreamer, it seems to be setting the framerate control several times, and strangely it uses multiple values (10fps and 20fps) though the output does seem to correctly be 10fps. Even when I omit the frame rate control (and exposure, gain, etc.), Gstreamer seems to still use that 20fps frame rate (though the output doesn’t reflect that and is correct based on the sensor mode). It does appear that when the frame rate is set after imx477_start_streaming, it actually streams at the requested rate, and this is what Gstreamer does seem to do (along with some extra stuff I can’t explain).

Things get a little stranger after running Gstreamer and then going back to v4l2-ctl (which sometimes is just broken entirely).

v4l2-ctl -d /dev/video0 --verbose --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0,sensor_mode=1,frame_rate=10000000 --stream-mmap --stream-count=300
[ 2751.574484] imx477 6-001a: imx477_power_on: power on
[ 2751.874625] imx477 6-001a: imx477_set_mode:
[ 2751.954659] imx477 6-001a: imx477_set_gain: Setting gain control to: 353
[ 2751.954666] imx477 6-001a: imx477_set_gain: val: 353 (/16) [times], gain: 978
[ 2751.955040] imx477 6-001a: imx477_set_exposure: Setting exposure control to: 25000
[ 2751.955045] imx477 6-001a: imx477_set_exposure: val: 25000 [us], coarse_time: 2326 [lines]
[ 2751.955417] imx477 6-001a: imx477_set_frame_rate: Setting framerate control to: 10000000
[ 2751.955422] imx477 6-001a: imx477_set_frame_rate: val: 10000000e-6 [fps], frame_length: 9308 [lines]
[ 2751.955831] imx477 6-001a: imx477_start_streaming:
[ 2751.956157] imx477 6-001a: imx477_stop_streaming:
[ 2751.956367] imx477 6-001a: imx477_power_off: power off
[ 2751.956500] imx477 6-001a: imx477_power_on: power on
[ 2752.271733] imx477 6-001a: imx477_set_mode:
[ 2752.351425] imx477 6-001a: imx477_set_gain: Setting gain control to: 353
[ 2752.351431] imx477 6-001a: imx477_set_gain: val: 353 (/16) [times], gain: 978
[ 2752.351852] imx477 6-001a: imx477_set_exposure: Setting exposure control to: 25000
[ 2752.351857] imx477 6-001a: imx477_set_exposure: val: 25000 [us], coarse_time: 2326 [lines]
[ 2752.352235] imx477 6-001a: imx477_set_frame_rate: Setting framerate control to: 10000000
[ 2752.352240] imx477 6-001a: imx477_set_frame_rate: val: 10000000e-6 [fps], frame_length: 9308 [lines]
[ 2752.352609] imx477 6-001a: imx477_start_streaming:
[ 2775.807933] imx477 6-001a: imx477_stop_streaming:
[ 2775.810901] imx477 6-001a: imx477_power_off: power off

This results in a capture rate of 12.89 fps, and interestingly imx477_set_frame_rate is being called before imx477_start_streaming. Another v4l2-ctl run at a requested rate of 20fps (frame_rate=20000000) resulted in a capture rate of 25.79 fps. Note this behavior only happens after Gstreamer runs. Further, when setting the v4l2-ctl frame rate from another shell, the resulting fps again isn’t the requested rate (ie. 10fps requested, 12.89fps captured, and confirmed by looking at the timestamps which are 0.07756999999 seconds apart). So it seems that even when v4l2-ctl sets frame rate (either via another shell, or after Gstreamer runs) it doesn’t do it 100% correctly.

I check the time stamp by running v4l2-ctl --stream-mmap -c bypass_mode=0 and run v4l2-ctl -c frame_rate=25000000 in another console to check the timestamp after it run for a while get correct result.
The result is 0.039999 ~= 1/25

       Index    : 2
        Type     : Video Capture
        Flags    : mapped
        Field    : None
        Sequence : 306
        Length   : 16588800
        Bytesused: 16588800
        Timestamp: 516506.146132s (Monotonic, End-of-Frame)

        Index    : 3
        Type     : Video Capture
        Flags    : mapped
        Field    : None
        Sequence : 307
        Length   : 16588800
        Bytesused: 16588800
        Timestamp: 516506.186131s (Monotonic, End-of-Frame)

        Index    : 0
        Type     : Video Capture
        Flags    : mapped
        Field    : None
        Sequence : 308
        Length   : 16588800
        Bytesused: 16588800
        Timestamp: 516506.226130s (Monotonic, End-of-Frame)

        Index    : 1
        Type     : Video Capture
        Flags    : mapped
        Field    : None
        Sequence : 309
        Length   : 16588800
        Bytesused: 16588800
        Timestamp: 516506.266129s (Monotonic, End-of-Frame)

        Index    : 2
        Type     : Video Capture
        Flags    : mapped
        Field    : None
        Sequence : 310
        Length   : 16588800
        Bytesused: 16588800
        Timestamp: 516506.306128s (Monotonic, End-of-Frame)

I just reinstalled a fresh L4T and I can no longer replicate that odd FPS issue (ie. 12.89 fps when requesting 10fps). Here is the behavior I see on both the stock Nvidia drivers and Arducam drivers:

  • After fresh reboot
    • v4l2-ctl set framerate in initial command - fail
    • v4l2-ctl set framerate in separate shell - pass
  • And then after Gstreamer runs
    • v4l2-ctl set framerate in initial command - pass
    • v4l2-ctl set framerate in separate shell - pass

Are you able to replicate the first case (v4l2-ctl set framerate in initial command before Gstreamer runs)?

The v4l2-ctl set control failed could be the v4l2-ctl APP’s issue. Looks like only the value different with previous then this CID called otherwise it’s won’t. You can check the log from IMX477 to confirm it.

And another issue is the v4l2-ctl set the CID before the set_mode that could cause the framerate configure override by the sensor mode initialize table.

Ok, that sounds feasible but I’m at the end of my knowledge of this code. Will Nvidia look into this as a possible bug either in the driver / v4l2-ctl? If it’s a bug in v4l2-ctl, where should that be reported? Thanks!

I think v4l2-ctl only debug tools only. I think for real case should implement the V4L2 API base APP for it.

Ok, are there any Nvidia provided samples using the v4l2 API that provide raw bayer output?

The MMAPI have 12_camera_v4l2_cuda and v4l2cuda but both of them are for YUV sensor.
Also should be able found from web like v4l2 capture example · GitHub