V4L2 vs. Argus MIPI CSI performance differences with IMX477 camera

I’m testing on two Jetson Nano 2GB Dev Kits running L4T 32.6.1 - For some reason, only one of them is able to run v4l2-ctl (see here for more info). With the one that does work with v4l2-ctl I’ve been testing both the stock Nvidia and Arducam drivers, the latter which provides support for the full resolution mode 4032x3040 at 30fps. The Nvidia driver only provides 3840x2160 at 30fps and 1920x1080 at 60fps. Here’s a breakdown of the functionality between Argus and v4l2 when running the Arducam drivers (where mode 0 = 4032x3040 at 30fps):

Argus via GStreamer

  • 4032x3040 at 30fps - PASS
  • 3840x2160 at 30fps - PASS
  • 1920x1080 at 60fps - PASS
  • Example command:
    gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=4032,height=3040,framerate=30/1' ! nvvidconv ! nvv4l2h265enc ! h265parse ! mp4mux ! filesink location=out.mp4

And just to clarify, Argus is 100% able to actually capture 4032x3040 at 30fps and encode at 30fps via nvv4l2h265enc/nvv4l2h264enc. I have confirmed this multiple ways.

v4l2 via v4l2-ctl

  • 4032x3040 at 30fps - FAIL
    • tegra_channel_error_status:error 20022 frame 90
  • 3840x2160 at 30fps - PASS
  • 1920x1080 at 60fps - PASS
  • Example command:
    v4l2-ctl -d /dev/video0 --set-fmt-video=width=4032,height=3040,pixelformat=RG10 --set-ctrl bypass_mode=0,sensor_mode=0 --stream-mmap

To rule out v4l2-ctl being the limitation, I tested with the v4l2cuda sample app (modified to support raw bayer output) and it exhibits the same behavior as v4l2-ctl. Arducam kindly provided a custom driver with a 4032x3040 at 15fps mode. This change enabled v4l2-ctl to capture correctly at that resolution.

Given my albeit limited understanding of the Argus and v4l2 camera pipelines, I believe they should share the same MIPI CSI bandwidth performance characteristics. The primary difference being that Argus interfaces with the ISP while v4l2 does not:


source: Nvidia

So my question becomes, how can Argus handle 4032x3040 at 30fps while v4l2 can’t? Any help in understanding, or ideally, finding a way to actually capture in this mode with v4l2 would be greatly appreciated. Thanks!

hello dustinkerstein,

the clock allocations approaches were different,
could you please try running jetson_clocks.sh and set the NVPModel clock configuration as MaxN for testing v4l2.
thanks

Hi Jerry,

Are the clock allocations something that can be modified either in the kernel sources, or at runtime? All of my tests have been run with jetson_clocks.sh at MaxN. Also, note that I updated the post above to include the correct error when trying to use v4l2 with 4032x3040 at 30fps:

tegra_channel_error_status:error 20022 frame 90

The other error I had previously included:

video4linux video0: frame start syncpt timeout!0

Is what I see on my other Nano 2GB Dev Kit (which never works with v4l2 - per this post)

hello dustinkerstein,

Argus supports DFS based on the loading, that’ll be able to set the clocks during runtime; there’s no DFS for VI in v4l2.
it’s pixel_clk_hz for the camera software uses the sensor pixel clock to calculate the exposure and frame rate of the sensor. this must be set correctly to avoid potential issues.

please keep running at performance mode, and please also setting VIC always running at max clock rates for testing,
for example,

disable runtime suspend of VIC
# echo on > /sys/devices/50000000.host1x/54340000.vic/power/control

set userspace governor
# echo userspace > /sys/devices/50000000.host1x/54340000.vic/devfreq/54340000.vic/governor

set max frequency
# cat /sys/devices/50000000.host1x/54340000.vic/devfreq/54340000.vic/available_frequencies 
# echo [max_freq_val] > /sys/devices/50000000.host1x/54340000.vic/devfreq/54340000.vic/max_freq

set target frequency
# echo [max_freq_val] > /sys/devices/50000000.host1x/54340000.vic/devfreq/54340000.vic/cur_freq

Interesting. Here’s what I ran (dmesg output is abbreviated):

developer@nano:~$ sudo jetson_clocks --show
SOC family:tegra210  Machine:NVIDIA Jetson Nano 2GB Developer Kit
Online CPUs: 0-3
cpu0: Online=1 Governor=schedutil MinFreq=1200000 MaxFreq=1479000 CurrentFreq=1224000 IdleStates: WFI=1 c7=1
cpu1: Online=1 Governor=schedutil MinFreq=1200000 MaxFreq=1479000 CurrentFreq=1224000 IdleStates: WFI=1 c7=1
cpu2: Online=1 Governor=schedutil MinFreq=1200000 MaxFreq=1479000 CurrentFreq=1224000 IdleStates: WFI=1 c7=1
cpu3: Online=1 Governor=schedutil MinFreq=1200000 MaxFreq=1479000 CurrentFreq=1224000 IdleStates: WFI=1 c7=1
GPU MinFreq=76800000 MaxFreq=921600000 CurrentFreq=76800000
EMC MinFreq=204000000 MaxFreq=1600000000 CurrentFreq=1600000000 FreqOverride=0
Fan: PWM=0
NV Power Mode: MAXN
developer@nano:~$ sudo jetson_clocks
developer@nano:~$ sudo jetson_clocks --show
SOC family:tegra210  Machine:NVIDIA Jetson Nano 2GB Developer Kit
Online CPUs: 0-3
cpu0: Online=1 Governor=schedutil MinFreq=1479000 MaxFreq=1479000 CurrentFreq=1479000 IdleStates: WFI=0 c7=0
cpu1: Online=1 Governor=schedutil MinFreq=1479000 MaxFreq=1479000 CurrentFreq=1479000 IdleStates: WFI=0 c7=0
cpu2: Online=1 Governor=schedutil MinFreq=1479000 MaxFreq=1479000 CurrentFreq=1479000 IdleStates: WFI=0 c7=0
cpu3: Online=1 Governor=schedutil MinFreq=1479000 MaxFreq=1479000 CurrentFreq=1479000 IdleStates: WFI=0 c7=0
GPU MinFreq=921600000 MaxFreq=921600000 CurrentFreq=921600000
EMC MinFreq=204000000 MaxFreq=1600000000 CurrentFreq=1600000000 FreqOverride=1
Fan: PWM=0
NV Power Mode: MAXN
developer@nano:~$ sudo su
root@nano:/home/developer# echo on > /sys/devices/50000000.host1x/54340000.vic/power/control
root@nano:/home/developer# echo userspace > /sys/devices/50000000.host1x/54340000.vic/devfreq/54340000.vic/governor
root@nano:/home/developer# cat /sys/devices/50000000.host1x/54340000.vic/devfreq/54340000.vic/available_frequencies
140800000 268800000 332800000 371200000 409600000 435200000 473600000 499200000 537600000 563200000 563200000 588800000 601600000 627200000
root@nano:/home/developer# echo 627200000 > /sys/devices/50000000.host1x/54340000.vic/devfreq/54340000.vic/max_freq
root@nano:/home/developer# echo 627200000 > /sys/devices/50000000.host1x/54340000.vic/devfreq/54340000.vic/cur_freq
bash: /sys/devices/50000000.host1x/54340000.vic/devfreq/54340000.vic/cur_freq: Permission denied
root@nano:/home/developer# jetson_clocks --show
SOC family:tegra210  Machine:NVIDIA Jetson Nano 2GB Developer Kit
Online CPUs: 0-3
cpu0: Online=1 Governor=schedutil MinFreq=1479000 MaxFreq=1479000 CurrentFreq=1479000 IdleStates: WFI=0 c7=0
cpu1: Online=1 Governor=schedutil MinFreq=1479000 MaxFreq=1479000 CurrentFreq=1479000 IdleStates: WFI=0 c7=0
cpu2: Online=1 Governor=schedutil MinFreq=1479000 MaxFreq=1479000 CurrentFreq=1479000 IdleStates: WFI=0 c7=0
cpu3: Online=1 Governor=schedutil MinFreq=1479000 MaxFreq=1479000 CurrentFreq=1479000 IdleStates: WFI=0 c7=0
GPU MinFreq=921600000 MaxFreq=921600000 CurrentFreq=921600000
EMC MinFreq=204000000 MaxFreq=1600000000 CurrentFreq=1600000000 FreqOverride=1
Fan: PWM=0
NV Power Mode: MAXN
root@nano:/home/developer# v4l2-ctl -d /dev/video0 --set-fmt-video=width=4032,height=3040,pixelformat=RG10 --set-ctrl bypass_mode=0,sensor_mode=0 --stream-mmap
^C
root@nano:/home/developer# dmesg
...
[  196.728103] vi 54080000.vi: tegra_channel_error_status:error 20022 frame 0
[  196.761419] vi 54080000.vi: tegra_channel_error_status:error 20022 frame 1
[  196.794741] vi 54080000.vi: tegra_channel_error_status:error 20022 frame 2
...
root@nano:/home/developer# v4l2-ctl -d /dev/video0 --set-fmt-video=width=3840,height=2160,pixelformat=RG10 --set-ctrl bypass_mode=0,sensor_mode=1 --stream-mmap
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.00 fps

Do you know why the following command returns that permission error?

echo 627200000 > /sys/devices/50000000.host1x/54340000.vic/devfreq/54340000.vic/cur_freq

With that error, 4032x3040 still results in that same tegra_channel_error_status:error 20022 - Anything else I can try? Thanks for your help.

hello dustinkerstein,

may I know what’s the actual use-case for running v4l2 camera pipelines?
I’ve check CSI and USB Camera Features again and it surprise me that the capture has validated up-to 3280x2464.

Hi Jerry,

My use-case requires long-exposure and raw bayer output still photographs (ideally captured in a trigger mode rather than streaming, but I know this isn’t yet supported). Per this thread, I was pointed to v4l2 as the only way long exposures are possible on the Nano currently. Why does it surprise you about the validation resolution?

Just wanted to quickly follow up. Is there anything else I can dig into regarding debug / ways to isolate how Argus is able to do this? Thanks again.

hello dustinkerstein,

so far, we don’t have sensor hardware to produce 4Kx3K sources on Nano platforms.
you may have workaround to lower the frame rate, please consider this is known issue since software release only mentioned it has validated up-to 3280x2464.

Hi Jerry,

I’d be happy to send you / Nvidia a couple IMX477 MIPI CSI sensors to test on. Would that be helpful? Do you know what the validation involves? Is there a test-suite/unit-test?

This Nvidia list suggests there are many 4k x 3k sensors that are compatible with the Nano (and other Jetson platforms). Thanks again.

Hi,
On Jetson Nano, we would suggest run in standard 4K resolution 3840x2160. For further use-cases, please consider Xavier or Xavier NX.

Hi Dane,

Is there a reason for this? With Gstreamer fully capable of 4032x3040 30fps, I would expect v4l2 to also be able to function, and be supported, at that resolution/fps. I’d just love to understand why this isn’t the case.

Thanks again,
Dustin

hello dustinkerstein,

FYI,
there’s driver update for IMX477, the sensor mode 4032x3040@30FPS has removed.
please expect next public release will include the change.
thanks

Hi Jerry,

I’m not sure I understand. There was an older L4T release that contained an IMX477 driver that included several modes (including the 4032x3040 mode) which are no longer included in the current L4T release. The only way I can test 4032x3040 is by using the Arducam/RidgeRun patched drivers. What exactly will be changing in future releases regarding the IMX477 driver? IMO, the 4032x3040 mode should be added back into the stock Nvidia driver as it does work perfectly fine with Argus at 30fps. As to why it doesn’t work with V4L2 is still a mystery…

hello dustinkerstein,

it’s due to some 2-lane configuration updates and the default 4K sensor mode has replaced with standard 4K (i.e. 3840x2160).
so, the v4l2 standard control issue with 4032x3040@30fps will be not fix.

Ok, I don’t understand 100%, but as long as I can install the Arducam drivers which expose the 4032x3040@30fps, then that should be fine. In my opinion, there will be many users who will want/need access to the full 4:3 FOV of the IMX477 (even if it needs to be at lower FPS - though it does work at 30fps in Gstreamer 100% with a Nano 2GB).

hello dustinkerstein,

you may just consider we’ve internally change the IMX477’s 4K resolution as 3840x2160.