120 fps mode support removed for imx219 sensor

feupos · April 6, 2021, 11:50am

According to the comments in OpenCV ops blocking for multi-camera capture with Gstreamer - Jetson & Embedded Systems / Jetson Nano - NVIDIA Developer Forums “We have removed 120fps sensor mode in Pi camera V2”.

When I use gst-launch it still shows GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 16.000000; Exposure Range min 22000, max 358733000; but captures at only 60 fps.

I need to understand why it was removed and if it is possible to enable it again, since I need to use the camera with 120fps. It worked with 120fps on R32, but since I upgraded to the latest version I am only able to get 60 fps.

feupos · April 6, 2021, 12:55pm

Also, if this mode is not supported anymore, can you inform me which is the latest version with support that I could use?

DaneLLL · April 7, 2021, 2:38am

Hi,
Please confirm your Jetson platform(Jetson Nano or Jetson TX2). The category is TX2 but we cannot plug in Pi camera V2 to TX2 developer kit, due to different hardware interface. Would like to clarify this.

feupos · April 7, 2021, 11:29am

Sorry, I just changed to the correct the category. I am using the Jetson Nano.

DaneLLL · April 8, 2021, 5:59am

Hi,
The mode is removed from JP4.5. Please try JP4.4.1.

feupos · April 9, 2021, 8:51pm

I installed JP4.4.1, however I can only get 120 fps when using v4l2-utils, and not on gstreamer

art@art-jetson:~$ v4l2-ctl -d /dev/video1 --set-fmt-video=width=1280,height=720,pixelformat=BG10 -csensor_mode=4 --stream-mmap --stream-skip=0
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 122.00 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<^C
art@art-jetson:~$ gst-launch-1.0 -v nvarguscamerasrc sensor-id=1 ! 'video/x-raw(memory:NVMM), width=1280, height=720, format=NV12, framerate=120/1' ! nvvidconv flip-method=2 ! video/x-raw, width=1280, height=720, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! fpsdisplaysink text-overlay=0 video-sink=fakesink
nvbuf_utils: Could not get EGL display connection
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)120/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)120/1
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGRx
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGRx
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:src: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGR
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:src: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGR
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGR
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0.GstPad:sink: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGR
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGR
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:sink: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGR
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:sink: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGRx
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGRx
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)120/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)120/1
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21,000000 fps Duration = 47619048 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28,000001 fps Duration = 35714284 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29,999999 fps Duration = 33333334 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29,999999 fps Duration = 33333334 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59,999999 fps Duration = 16666667 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120,000005 fps Duration = 8333333 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 1
   Camera mode  = 5
   Output Stream W = 1280 H = 720
   seconds to Run    = 0
   Frame Rate = 120,000005
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 17, dropped: 0, current: 32,79, average: 32,79
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 32, dropped: 0, current: 29,91, average: 31,37
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 48, dropped: 0, current: 30,02, average: 30,91
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 63, dropped: 0, current: 29,96, average: 30,68
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 79, dropped: 0, current: 30,19, average: 30,58
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:06.189181473
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
GST_ARGUS: Cleaning up
CONSUMER: Done Success
GST_ARGUS: Done Success
Setting pipeline to NULL ...
Freeing pipeline ...
art@art-jetson:~$ gst-launch-1.0 -v nvarguscamerasrc sensor-id=1 sensor-mode=4 ! 'video/x-raw(memory:NVMM), width=1280, height=720, format=NV12, framerate=120/1' ! nvvidconv flip-method=2 ! video/x-raw, width=1280, height=720, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! fpsdisplaysink text-overlay=0 video-sink=fakesink
nvbuf_utils: Could not get EGL display connection
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)120/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)120/1
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGRx
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGRx
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:src: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGR
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:src: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGR
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGR
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0.GstPad:sink: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGR
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGR
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:sink: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGR
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:sink: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGRx
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGRx
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)120/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)120/1
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21,000000 fps Duration = 47619048 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28,000001 fps Duration = 35714284 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29,999999 fps Duration = 33333334 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29,999999 fps Duration = 33333334 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59,999999 fps Duration = 16666667 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120,000005 fps Duration = 8333333 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

ARGUS_ERROR: Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute: 679 Frame Rate specified is greater than supported
GST_ARGUS: Running with following settings:
   Camera index = 1
   Camera mode  = 4
   Output Stream W = 1280 H = 720
   seconds to Run    = 0
   Frame Rate = 59,999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
ARGUS_ERROR: Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute: 899 InvalidState.
GST_ARGUS: Cleaning up
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
Got EOS from element "pipeline0".
Execution ended after 0:00:03.290824696
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

There seems to be a problem configuring the sensor mode, because the v4l20-ctl uses sensor mode for 120 fps, however when using gstreamer it choses sensor mode 5.

Also, I need to understand why this mode was removed, is there a problem with it? Will newer versions restore support?

DaneLLL · April 9, 2021, 11:07pm

Hi,
We will keep only 720p60 mode in the default release. If you require 120fps mode, you may add the mode back to sensor driver and device tree. And rebuild/replace kernel.

The sensor mode and device tree files:

imx219_mode_tbls.h
tegra194-camera-rbpcv2-imx219.dtsi

Honey_Patouceul · April 12, 2021, 12:15am

Sharing some details on how to do that for users non familiar with this:

First get kernel sources for your Jetson. You would go to NVIDIA Developer, log in if not yet done, and then download R32.5.1 sources for Nano/TX1 or TX2/Xavier.
Extract kernel_src.tbz2 from public sources.tbz2.
Extract kernel sources from kernel_src.tbz2. In my case I extract to /usr/local/src/l4t_kernel, but YMMV.
Patch the two files mentionned by @DaneLLL. Here are the patches:
imx219_mode_tbls.h.patch (1.5 KB) tegra194-camera-rbpcv2-imx219.dtsi.patch (5.1 KB)

5 Build a test kernel:

#!/bin/bash

########################################################
# Native Jetson NX L4T kernel build script. 
# Assuming L4T R32.5.1 version with kernel 4.9.201.
########################################################
set -x

# Set L4TK to where you've extracted kernel_src.tbz2 archive. (tar –xjf kernel_src.tbz2)  
# In my case, was in /usr/local/src/l4t_kernel owned by root, thus some sudos below for build directory cleanup and re-creation.
export L4TK=/usr/local/src/l4t_kernel

# define some short cuts 
export SRC=$L4TK/kernel/kernel-4.9
export STAGE=$L4TK/build

# Prepare a clean build directory
sudo rm -rf ${STAGE}
sudo mkdir ${STAGE}
sudo chown -R nvidia $STAGE
sudo chgrp -R nvidia $STAGE

export TEGRA_KERNEL_OUT=${STAGE}/kernel
mkdir $TEGRA_KERNEL_OUT

export TEGRA_MODULES_OUT=${STAGE}/modules
mkdir $TEGRA_MODULES_OUT

# Not mandatory. Boost NX for saving some time. You may adapt for Nano.
# Increase instantaneous OC limit, from 3.6A to 5A on NX, in order to prevent 'soctherm: OC ALARM 0x00000001'
# Should be safe on NX according to: https://forums.developer.nvidia.com/t/system-throttled-due-to-over-current/167029/58
sudo sh -c 'echo 5000 > /sys/devices/c250000.i2c/i2c-7/7-0040/iio:device0/crit_current_limit_0'
# 15W-6cores mode on NX (use -m0 on other Jetsons)
sudo nvpmodel -m2
# Boost clocks
sudo /usr/bin/jetson_clocks

# Ready to build our kernel
cd $SRC

# Configure kernel 
export LOCALVERSION=-test
make ARCH=arm64 O=$TEGRA_KERNEL_OUT tegra_defconfig

# Build 
make ARCH=arm64 O=$TEGRA_KERNEL_OUT -j6 

# Fake modules install
make ARCH=arm64 O=$TEGRA_KERNEL_OUT modules_install INSTALL_MOD_PATH=${TEGRA_MODULES_OUT}/

# If ok, actually install
sudo rm -rf /lib/modules/4.9.201-test
sudo cp -R ${TEGRA_MODULES_OUT}/lib/modules/4.9.201-test /lib/modules
sudo cp $TEGRA_KERNEL_OUT/arch/arm64/boot/Image /boot/Image-4.9.201-test

# This is default device tree for NX on B01 devkit, you would adapt for other case.
sudo cp ${TEGRA_KERNEL_OUT}/arch/arm64/boot/dts/tegra194-p3668-all-p3509-0000.dtb /boot/test.dtb

So now we have custom kernel image and dtb in /boot and modules for our 4.9.201-test kernel in /lib/modules/

As root, create a new entry in /boot/extlinux/extlinux.conf, first copying your working default entry and in new entry modifying LABEL, MENU, LINUX for the new kernel image, adding FDT line for device tree, keeping INITRD and APPEND unchanged such as:

LABEL test
      MENU LABEL test kernel
      LINUX /boot/Image-4.9.201-test
      FDT /boot/test.dtb
      INITRD /boot/initrd
      APPEND ${cbootargs} root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0

Get a serial console and try to boot your test config at uboot prompt (first boot prompt is cboot, ignore it, uboot is next and you have 3s for selecting your config).

In my case it works fine @120 fps with v4l2 and gstreamer.

The only thing is that v4l2-ctl --list-formats-ext is reporting wrong framerate for last mode:

v4l2-ctl -d0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'RG10'
	Name        : 10-bit Bayer RGRG/GBGB
		Size: Discrete 3264x2464
			Interval: Discrete 0.048s (21.000 fps)
		Size: Discrete 3264x1848
			Interval: Discrete 0.036s (28.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 1640x1232
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.017s (60.000 fps)

@DaneLLL , any idea why the latter doesn’t report 120 fps ?

JerryChang · April 12, 2021, 2:10am

hello Honey_Patouceul,

it’s a known issue that v4l2-ctl utility cannot report frame-rate capability correctly if there’re two sensor modes with the same resolution. (i.e. 1280x720@60-fps and 1280x720@120-fps).
please enable gst pipeline with nvarguscamerasrc, you shall see all available sensor modes in the beginning.
thanks

feupos · April 12, 2021, 8:03pm

Thank you for the detailed instructions. I downloaded the sources from https://developer.nvidia.com/embedded/l4t/r32_release_v5.1/r32_release_v5.1/sources/t210/public_sources.tbz2, however I can’t find the file hardware/nvidia/platform/t19x/jakku/kernel-dts/common/tegra194-camera-rbpcv2-imx219.dtsi, or any similar named file, even in different folders. The other patch applied correctly.

Honey_Patouceul · April 12, 2021, 10:18pm

T194 is Xavier SoC. For Nano, you would find a similar tegra210.

chang-quan · June 1, 2021, 4:28pm

hello, where should patches be placed?

Honey_Patouceul · June 1, 2021, 5:08pm

The patches should apply from your kernel top directory (where you’ve extracted kernel sources).
Looking at the patches you’ll see in first lines the relative path to each modified file.

Use command patch (you may add -p0) for applying the given patch to file:

cd <where_you've_extracted_kernel_sources>
patch -p0 <tegra194-camera-rbpcv2-imx219.dtsi.patch 
patch -p0 <imx219_mode_tbls.h.patch

However, be aware that running @120 fps with Argus may use about one CPU core in R32.5.

chang-quan · June 2, 2021, 4:05am

thank you, this solve my problem .

caowenwu1 · August 10, 2021, 2:51am

5 Build a test kernel:

Excuse me, is your sh script executed on the nx development board or executed on the local ubuntu system

caowenwu1 · August 10, 2021, 12:53pm

Is there an IMX219 patch for AGX?

Honey_Patouceul · August 15, 2021, 7:21pm

The script is for native building on NX.

I only have an OV5693 camera module with my AGX Xavier so I can’t tell for any IMX219 sensor.
The DT patch would probably be different. You may ask to your sensor vendor.

Topic		Replies	Views
Gstreamer reports Raspberry Pi camera streaming at 120fps when in reality it is only 60fps Jetson Nano camera , gstreamer	53	3745	July 19, 2022
120fps with IMX219 and binned modes Jetson Xavier NX camera	49	3508	January 18, 2023
TX2 VI mode and ISP mode bugs when using CSI2 virtual channels Jetson TX2 camera , nvbugs	15	1306	March 9, 2022
Failing to get input from CSI camera (IMX219-83) Jetson Nano camera , ubuntu , gstreamer	29	1388	September 25, 2023
Arducam IMX-219, not giving 4k video Jetson Nano camera , gstreamer , python	42	2230	January 28, 2022
I cannot see the video from the imx219 camera on the Jetson Orin Nano + custom carrier board Jetson Orin Nano camera , board-design	24	1210	October 17, 2023
Gst-launch-1.0 nvarguscamerasrc can't get right GST_ARGUS: Available Sensor modes Jetson AGX Xavier camera	16	1311	July 18, 2023
EGL display error and execute: 751 Failed to create CaptureSession Jetson Nano camera , gstreamer	6	1659	July 27, 2022
Issue using camera Jetson Xavier NX camera	26	1771	November 21, 2022
R35.3.1 AGX ORIN bring up raw14 format，Kernel error Jetson AGX Orin camera	54	1635	October 19, 2023

120 fps mode support removed for imx219 sensor

Related topics