120 fps mode support removed for imx219 sensor

According to the comments in OpenCV ops blocking for multi-camera capture with Gstreamer - Jetson & Embedded Systems / Jetson Nano - NVIDIA Developer Forums “We have removed 120fps sensor mode in Pi camera V2”.

When I use gst-launch it still shows GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 16.000000; Exposure Range min 22000, max 358733000; but captures at only 60 fps.

I need to understand why it was removed and if it is possible to enable it again, since I need to use the camera with 120fps. It worked with 120fps on R32, but since I upgraded to the latest version I am only able to get 60 fps.

Also, if this mode is not supported anymore, can you inform me which is the latest version with support that I could use?

Hi,
Please confirm your Jetson platform(Jetson Nano or Jetson TX2). The category is TX2 but we cannot plug in Pi camera V2 to TX2 developer kit, due to different hardware interface. Would like to clarify this.

Sorry, I just changed to the correct the category. I am using the Jetson Nano.

Hi,
The mode is removed from JP4.5. Please try JP4.4.1.

I installed JP4.4.1, however I can only get 120 fps when using v4l2-utils, and not on gstreamer

art@art-jetson:~$ v4l2-ctl -d /dev/video1 --set-fmt-video=width=1280,height=720,pixelformat=BG10 -csensor_mode=4 --stream-mmap --stream-skip=0
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 122.00 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<^C
art@art-jetson:~$ gst-launch-1.0 -v nvarguscamerasrc sensor-id=1 ! 'video/x-raw(memory:NVMM), width=1280, height=720, format=NV12, framerate=120/1' ! nvvidconv flip-method=2 ! video/x-raw, width=1280, height=720, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! fpsdisplaysink text-overlay=0 video-sink=fakesink
nvbuf_utils: Could not get EGL display connection
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)120/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)120/1
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGRx
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGRx
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:src: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGR
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:src: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGR
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGR
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0.GstPad:sink: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGR
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGR
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:sink: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGR
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:sink: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGRx
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGRx
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)120/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)120/1
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21,000000 fps Duration = 47619048 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28,000001 fps Duration = 35714284 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29,999999 fps Duration = 33333334 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29,999999 fps Duration = 33333334 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59,999999 fps Duration = 16666667 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120,000005 fps Duration = 8333333 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 1
   Camera mode  = 5
   Output Stream W = 1280 H = 720
   seconds to Run    = 0
   Frame Rate = 120,000005
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 17, dropped: 0, current: 32,79, average: 32,79
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 32, dropped: 0, current: 29,91, average: 31,37
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 48, dropped: 0, current: 30,02, average: 30,91
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 63, dropped: 0, current: 29,96, average: 30,68
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 79, dropped: 0, current: 30,19, average: 30,58
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:06.189181473
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
GST_ARGUS: Cleaning up
CONSUMER: Done Success
GST_ARGUS: Done Success
Setting pipeline to NULL ...
Freeing pipeline ...
art@art-jetson:~$ gst-launch-1.0 -v nvarguscamerasrc sensor-id=1 sensor-mode=4 ! 'video/x-raw(memory:NVMM), width=1280, height=720, format=NV12, framerate=120/1' ! nvvidconv flip-method=2 ! video/x-raw, width=1280, height=720, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! fpsdisplaysink text-overlay=0 video-sink=fakesink
nvbuf_utils: Could not get EGL display connection
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)120/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)120/1
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGRx
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGRx
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:src: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGR
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:src: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGR
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGR
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0.GstPad:sink: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGR
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGR
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:sink: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGR
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:sink: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGRx
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)120/1, format=(string)BGRx
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)120/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)120/1
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21,000000 fps Duration = 47619048 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28,000001 fps Duration = 35714284 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29,999999 fps Duration = 33333334 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29,999999 fps Duration = 33333334 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59,999999 fps Duration = 16666667 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120,000005 fps Duration = 8333333 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

ARGUS_ERROR: Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute: 679 Frame Rate specified is greater than supported
GST_ARGUS: Running with following settings:
   Camera index = 1
   Camera mode  = 4
   Output Stream W = 1280 H = 720
   seconds to Run    = 0
   Frame Rate = 59,999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
ARGUS_ERROR: Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute: 899 InvalidState.
GST_ARGUS: Cleaning up
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
Got EOS from element "pipeline0".
Execution ended after 0:00:03.290824696
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

There seems to be a problem configuring the sensor mode, because the v4l20-ctl uses sensor mode for 120 fps, however when using gstreamer it choses sensor mode 5.

Also, I need to understand why this mode was removed, is there a problem with it? Will newer versions restore support?

Hi,
We will keep only 720p60 mode in the default release. If you require 120fps mode, you may add the mode back to sensor driver and device tree. And rebuild/replace kernel.

The sensor mode and device tree files:

imx219_mode_tbls.h
tegra194-camera-rbpcv2-imx219.dtsi

Sharing some details on how to do that for users non familiar with this:

  1. First get kernel sources for your Jetson. You would go to NVIDIA Developer, log in if not yet done, and then download R32.5.1 sources for Nano/TX1 or TX2/Xavier.

  2. Extract kernel_src.tbz2 from public sources.tbz2.

  3. Extract kernel sources from kernel_src.tbz2. In my case I extract to /usr/local/src/l4t_kernel, but YMMV.

  4. Patch the two files mentionned by @DaneLLL. Here are the patches:
    imx219_mode_tbls.h.patch (1.5 KB) tegra194-camera-rbpcv2-imx219.dtsi.patch (5.1 KB)

5 Build a test kernel:

#!/bin/bash

########################################################
# Native Jetson NX L4T kernel build script. 
# Assuming L4T R32.5.1 version with kernel 4.9.201.
########################################################
set -x

# Set L4TK to where you've extracted kernel_src.tbz2 archive. (tar –xjf kernel_src.tbz2)  
# In my case, was in /usr/local/src/l4t_kernel owned by root, thus some sudos below for build directory cleanup and re-creation.
export L4TK=/usr/local/src/l4t_kernel

# define some short cuts 
export SRC=$L4TK/kernel/kernel-4.9
export STAGE=$L4TK/build

# Prepare a clean build directory
sudo rm -rf ${STAGE}
sudo mkdir ${STAGE}
sudo chown -R nvidia $STAGE
sudo chgrp -R nvidia $STAGE

export TEGRA_KERNEL_OUT=${STAGE}/kernel
mkdir $TEGRA_KERNEL_OUT

export TEGRA_MODULES_OUT=${STAGE}/modules
mkdir $TEGRA_MODULES_OUT

# Not mandatory. Boost NX for saving some time. You may adapt for Nano.
# Increase instantaneous OC limit, from 3.6A to 5A on NX, in order to prevent 'soctherm: OC ALARM 0x00000001'
# Should be safe on NX according to: https://forums.developer.nvidia.com/t/system-throttled-due-to-over-current/167029/58
sudo sh -c 'echo 5000 > /sys/devices/c250000.i2c/i2c-7/7-0040/iio:device0/crit_current_limit_0'
# 15W-6cores mode on NX (use -m0 on other Jetsons)
sudo nvpmodel -m2
# Boost clocks
sudo /usr/bin/jetson_clocks

# Ready to build our kernel
cd $SRC

# Configure kernel 
export LOCALVERSION=-test
make ARCH=arm64 O=$TEGRA_KERNEL_OUT tegra_defconfig

# Build 
make ARCH=arm64 O=$TEGRA_KERNEL_OUT -j6 

# Fake modules install
make ARCH=arm64 O=$TEGRA_KERNEL_OUT modules_install INSTALL_MOD_PATH=${TEGRA_MODULES_OUT}/

# If ok, actually install
sudo rm -rf /lib/modules/4.9.201-test
sudo cp -R ${TEGRA_MODULES_OUT}/lib/modules/4.9.201-test /lib/modules
sudo cp $TEGRA_KERNEL_OUT/arch/arm64/boot/Image /boot/Image-4.9.201-test

# This is default device tree for NX on B01 devkit, you would adapt for other case.
sudo cp ${TEGRA_KERNEL_OUT}/arch/arm64/boot/dts/tegra194-p3668-all-p3509-0000.dtb /boot/test.dtb

So now we have custom kernel image and dtb in /boot and modules for our 4.9.201-test kernel in /lib/modules/

  1. As root, create a new entry in /boot/extlinux/extlinux.conf, first copying your working default entry and in new entry modifying LABEL, MENU, LINUX for the new kernel image, adding FDT line for device tree, keeping INITRD and APPEND unchanged such as:
LABEL test
      MENU LABEL test kernel
      LINUX /boot/Image-4.9.201-test
      FDT /boot/test.dtb
      INITRD /boot/initrd
      APPEND ${cbootargs} root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0
  1. Get a serial console and try to boot your test config at uboot prompt (first boot prompt is cboot, ignore it, uboot is next and you have 3s for selecting your config).

In my case it works fine @120 fps with v4l2 and gstreamer.

The only thing is that v4l2-ctl --list-formats-ext is reporting wrong framerate for last mode:

v4l2-ctl -d0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'RG10'
	Name        : 10-bit Bayer RGRG/GBGB
		Size: Discrete 3264x2464
			Interval: Discrete 0.048s (21.000 fps)
		Size: Discrete 3264x1848
			Interval: Discrete 0.036s (28.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 1640x1232
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.017s (60.000 fps)

@DaneLLL , any idea why the latter doesn’t report 120 fps ?

hello Honey_Patouceul,

it’s a known issue that v4l2-ctl utility cannot report frame-rate capability correctly if there’re two sensor modes with the same resolution. (i.e. 1280x720@60-fps and 1280x720@120-fps).
please enable gst pipeline with nvarguscamerasrc, you shall see all available sensor modes in the beginning.
thanks

Thank you for the detailed instructions. I downloaded the sources from https://developer.nvidia.com/embedded/l4t/r32_release_v5.1/r32_release_v5.1/sources/t210/public_sources.tbz2, however I can’t find the file hardware/nvidia/platform/t19x/jakku/kernel-dts/common/tegra194-camera-rbpcv2-imx219.dtsi, or any similar named file, even in different folders. The other patch applied correctly.

T194 is Xavier SoC. For Nano, you would find a similar tegra210.