Gstreamer reports Raspberry Pi camera streaming at 120fps when in reality it is only 60fps

Autonomous Machines Jetson & Embedded Systems Jetson Nano
,
#1

I used to have my RPi cam working at 120fps last year but something has changed. gstreamer tells me it is running at 120fps when actually it isn’t. Please see the output below which explains everything:

nano@jetson-nano:/home/nano/$ date; gst-launch-1.0 nvarguscamerasrc num-buffers=120  ! 'video/x-raw(memory:NVMM),width=1280, height=720, framerate=120/1, format=NV12' ! omxh264enc ! qtmux ! filesink location=test.mp4 -e
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Framerate set to : 120 at NvxVideoEncoderSetParameterNvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
H264: Profile = 66, Level = 40 
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 5 
   Output Stream W = 1280 H = 720 
   seconds to Run    = 0 
   Frame Rate = 120.000005 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
Got EOS from element "pipeline0".
Execution ended after 0:00:02.265695998
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
GST_ARGUS: Cleaning up
CONSUMER: Done Success
GST_ARGUS: Done Success
Setting pipeline to NULL ...
Freeing pipeline ...
Tue Jun 14 12:40:29 BST 2022nano@jetson-nano:/nfs/mmapi_samples/samples/ralph_webcam$ date; gst-launch-1.0 nvarguscamerasrc num-buffers=120  ! 'video/x-raw(memory:NVMM),width=1280, height=720, framerate=120/1, format=NV12' ! omxh264enc ! qtmux ! filesink location=test.mp4 -e; date 
Tue Jun 14 12:40:25 BST 2022
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Framerate set to : 120 at NvxVideoEncoderSetParameterNvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
H264: Profile = 66, Level = 40 
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 5 
   Output Stream W = 1280 H = 720 
   seconds to Run    = 0 
   Frame Rate = 120.000005 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
Got EOS from element "pipeline0".
Execution ended after 0:00:02.265695998
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
GST_ARGUS: Cleaning up
CONSUMER: Done Success
GST_ARGUS: Done Success
Setting pipeline to NULL ...
Freeing pipeline ...

Anyone know what is going on? Thanks.

#3

Please check if below command to confirm sensor able reach 120fps

v4l2-ctl --set-fmt-video=width=1280,height=720,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap
1 Like
#4

This is what I get:

nano@jetson-nano:/home/nano$ v4l2-ctl -d0 --set-fmt-video=width=1280,height=720,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap
<<<<<<<<<<<<<<<<<<<<<<< 21.35 fps
<<<<<<<<<<<<<<<<<<<<< 21.28 fps
<<<<<<<<<<<<<<<<<<<<< 21.19 fps
<<<<<<<<<<<<<<<<<<<<< 21.19 fps
<<<<<<<<<<<<<<<<<<<<< 21.20 fps
etc

nano@jetson-nano:/home/nano$ v4l2-ctl -d0 -D
Driver Info (not using libv4l2):
        Driver name   : tegra-video
        Card type     : vi-output, imx219 6-0010
        Bus info      : platform:54080000.vi:0
        Driver version: 4.9.253
        Capabilities  : 0x84200001
                Video Capture
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps   : 0x04200001
                Video Capture
                Streaming
                Extended Pix Format

Seems to only manage 20fps with the command you gave. :-S

#5

Set the frame rate by below command.

v4l2-ctl -c frame_rate=120000000
1 Like
#6

No change:

nano@jetson-nano:~$ v4l2-ctl -d0 -c frame_rate=120000000
nano@jetson-nano:~$ v4l2-ctl -d0 --set-fmt-video=width=1280,height=720,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap
<<<<<<<<<<<<<<<<<<<<<<< 21.35 fps
<<<<<<<<<<<<<<<<<<<<< 21.28 fps
<<<<<<<<<<<<<<<<<<<<< 21.19 fps
<<<<<<<<<<<<<<<<<<<<< 21.19 fps
<<<<<<<<<<<<<<<<<<<<< 21.20 fps
<<<<<<<<<<<<<<<<<<<<<< 21.19 fps
<<<<<<<<<<<<<<<<<<<<< 21.19 fps
<<<<<<<<<<<<<<<<<<<<< 21.19 fps
<<<<<<<<<<<<<<<<<<<<< 21.19 fps
<<<<<<<<<<<<<<<<<<<<< 21.20 fps
<<<<<<<<<<<<<<<<<<<<<< 21.19 fps
<<<<<<<<<<<<<<<<<<<<< 21.19 fps
<<<<<<<<<<<<<<<<<<<<< 21.19 fps
<<<<<<<<<<<<<<<<<<<<< 21.19 fps
etc
#7

If your sensor was set to 3264 x 2464 resolution it couldn’t support a framerate higher than 21 fps so the v4l2-ctl command may have failed.

Back to your original question, it is unclear why you think it doesn’t work at 120 fps. What gives;

v4l2-ctl -d0 --list-formats-ext

gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM),width=1280, height=720, framerate=120/1, format=NV12' !  omxh264enc ! fpsdisplaysink text-overlay=0 video-sink=fakesink -v
1 Like
#8

I think it doesn’t run at 120fps because it used to 1 1/2 years ago and now it doesn’t.

Here is the output of the command you listed:

nano@jetson-nano:~$ v4l2-ctl -d0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
        Index       : 0
        Type        : Video Capture
        Pixel Format: 'RG10'
        Name        : 10-bit Bayer RGRG/GBGB
                Size: Discrete 3264x2464
                        Interval: Discrete 0.048s (21.000 fps)
                Size: Discrete 3264x1848
                        Interval: Discrete 0.036s (28.000 fps)
                Size: Discrete 1920x1080
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 1640x1232
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 1280x720
                        Interval: Discrete 0.017s (60.000 fps)

nano@jetson-nano:~$ gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM),width=1280, height=720, framerate=120/1, format=NV12' !  omxh264enc ! fpsdisplaysink text-overlay=0 video-sink=fakesink -v
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)120/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)120/1
Framerate set to : 120 at NvxVideoEncoderSetParameterNvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
H264: Profile = 66, Level = 40 
/GstPipeline:pipeline0/GstOMXH264Enc-omxh264enc:omxh264enc-omxh264enc0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)120/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)120/1
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 5 
   Output Stream W = 1280 H = 720 
   seconds to Run    = 0 
   Frame Rate = 120.000005 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
/GstPipeline:pipeline0/GstOMXH264Enc-omxh264enc:omxh264enc-omxh264enc0.GstPad:src: caps = video/x-h264, alignment=(string)au, profile=(string)baseline, level=(string)4, stream-format=(string)avc, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)120/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, chroma-site=(string)mpeg2
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-h264, alignment=(string)au, profile=(string)baseline, level=(string)4, stream-format=(string)avc, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)120/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, chroma-site=(string)mpeg2
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0.GstPad:sink: caps = video/x-h264, alignment=(string)au, profile=(string)baseline, level=(string)4, stream-format=(string)avc, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)120/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, chroma-site=(string)mpeg2
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink: caps = video/x-h264, alignment=(string)au, profile=(string)baseline, level=(string)4, stream-format=(string)avc, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)120/1, interlace-mode=(string)progressive, colorimetry=(string)bt709, chroma-site=(string)mpeg2
/GstPipeline:pipeline0/GstOMXH264Enc-omxh264enc:omxh264enc-omxh264enc0.GstPad:src: caps = video/x-h264, alignment=(string)au, profile=(string)baseline, level=(string)4, stream-format=(string)avc, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)120/1, codec_data=(buffer)014240150301000a6742402a95a014016e4001000468ce3c80, interlace-mode=(string)progressive, colorimetry=(string)bt709, chroma-site=(string)mpeg2
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-h264, alignment=(string)au, profile=(string)baseline, level=(string)4, stream-format=(string)avc, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)120/1, codec_data=(buffer)014240150301000a6742402a95a014016e4001000468ce3c80, interlace-mode=(string)progressive, colorimetry=(string)bt709, chroma-site=(string)mpeg2
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0.GstPad:sink: caps = video/x-h264, alignment=(string)au, profile=(string)baseline, level=(string)4, stream-format=(string)avc, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)120/1, codec_data=(buffer)014240150301000a6742402a95a014016e4001000468ce3c80, interlace-mode=(string)progressive, colorimetry=(string)bt709, chroma-site=(string)mpeg2
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink: caps = video/x-h264, alignment=(string)au, profile=(string)baseline, level=(string)4, stream-format=(string)avc, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)120/1, codec_data=(buffer)014240150301000a6742402a95a014016e4001000468ce3c80, interlace-mode=(string)progressive, colorimetry=(string)bt709, chroma-site=(string)mpeg2
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 32, dropped: 0, current: 63.82, average: 63.82
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 62, dropped: 0, current: 59.92, average: 61.87
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 92, dropped: 0, current: 59.98, average: 61.24
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 123, dropped: 0, current: 60.12, average: 60.96
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 154, dropped: 0, current: 60.01, average: 60.76
etc

So, now it only does 60fps.

#9

Ok, it gets clearer. In recent L4T releases (probably you’ve upgraded), the 120 fps mode has been removed from IMX219 driver.

Here is how to re-enable it (it may be a few releases behind, R32.5.1), but be aware that it might cost almost one CPU core for argus/nvargus-daemon.service to run at this speed.

1 Like
#10

I can spare 1 CPU since I have 3 spare after all. :-)

Thanks for the suggestion. It looks like Nvidia has done something unpleasant to those who depend on 120fps support.

Does Nvidia have any comment on this?

#11

This sensor mode cause some internal timing test.

Thanks

#12

Which sensor mode? I don’t understand your comment. Thanks.

#13

Remove the imx219@720p@120fps

#14

How do I remove it? Surely I want to keep imx219@720p@120fps?

#15

You need download the old version and new version to add the 720@120fps mode back manually by yourself.
Have reference to below topic.

1 Like
#16

Hello again,

the first patch applies fine but the second patch does not apply because the source file to which it refers does not exist. I changed it to point to:

hardware/nvidia/platform/t210/porg/kernel-dts/porg-platforms/tegra210-camera-rbpcv2-imx219.dtsi

but it still did not apply because it appears the code that is uncommented by the patch is already uncommented.

Will this be a problem?

#17

Apparently it will be a problem!! :-) My kernel boots but I cannot access the Raspberry Pi camera anymore:

nano@jetson-nano:~$ gst-launch-1.0 -v nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)30/1' ! nvvidconv ! nvv4l2h264enc insert-sps-pps=1 maxperf-enable=1 ! h264parse ! rtph264pay pt=96 mtu=1500 ! udpsink host=10.1.1.16 port=5000 sync=false async=false
nvbuf_utils: Could not get EGL display connection
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE 
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:725 No cameras available
etc etc etc

I do not have easy serial port access so I made my Uboot boot the new kernel by default. Thus my /boot/extlinux/extlinux.conf file is as such:

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image-4.9.253-test
      FDT /boot/test.dtb
      INITRD /boot/initrd
      APPEND ${cbootargs} quiet tegraid=21.1.2.0.0 ddr_die=4096M@2048M section=512M memtype=0 vpr_resize usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=0 touch_id=0@63 video=tegrafb no_console_suspend=1 console=ttyS0,115200n8 debug_uartport=lsport,2 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_scheme_first=1 lp0_vec=0x1000@0xff780000 core_edp_mv=1075 core_edp_ma=4000  root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0    root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 pci=noaer quiet

LABEL old
      MENU LABEL old kernel
      LINUX /boot/Image
      INITRD /boot/initrd
      APPEND <<<<identical as above>>>>

I did not change APPEND or INITRD. I chose the following as my DTB file (“test.dtb” in extlinux.conf) because I have an A02 Jetson Nano development board:

tegra210-p3448-0000-p3449-0000-a02.dtb

#18

Does anyone know what I am doing wrong? Do you need more information from me to help resolve this?

Thank you.

#19

This error tell your device tree have problem. You can dump the working to compare with yours.
Below command to dump the device tree

sudo dtc -I fs -O dts -o extracted_proc.dts /proc/device-tree

Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:725 No cameras available

1 Like
#20

I managed to get both DTS files. The working one is from the standard Nvidia working kernel:

working.dts (309.8 KB)

Then there is my one which I followed the linked instructions to create support for 120fps:

mine.dts (310.2 KB)

I cannot see any specific references to the camera in the diff between the files so I am still confused about where the problem is. :-S

Thanks.

#21

You can modify the working.dts to add the 120fps sensor mode to clarify.

1 Like