Nvcamerasrc GST plugin frame-rate limitation

dgarba · May 2, 2017, 5:12pm

Hi all,

I am working with an IMX274 camera who has a 1280x540@240fps mode. I want to capture with Nvcamerasrc GST plugin, but it has a limitation in the framerate pad caps of 120 fps maximum.

Here is the gst-inspect output that shows the framerate limitation:

ubuntu@tegra-ubuntu:~/carlos_test$ gst-inspect-1.0 nvcamerasrc
Factory Details:
  Rank                     primary (256)
  Long-name                Nvidia Camera Source
  Klass                    Video/Capture
  Description              Nvidia Camera Source
  Author                   Jitendra Kumar <jitendrak@nvidia.com>

Plugin Details:
  Name                     nvcamera
  Description              Nvidia Video Capture Component 
  Filename                 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvcamera.so
  Version                  1.2.0
  License                  Proprietary
  Source module            nvcamera
  Binary package           NvCamera
  Origin URL               http://nvidia.com

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw(memory:NVMM)
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
                 format: { I420, NV12, UYVY }
              framerate: [ 0/1, 120/1 ]

Is there any way to obtain access to nvcamerasrc gst plugin source code, to give support for 240 fps framerate, or at least any possibility of getting a binary if that plugin that supports the 240 fps. Also, it will be very helpful any related guide on how to capture and test this camera driver mode (1280x540@240fps) passing through the ISP (V4l2src only gives raw bayer output, but I need YUV or I420 so this is not an option).

I will apreciate any help!!

Thanks a lot in advance!

Best regards,
Daniel

JerryChang · May 3, 2017, 6:47am

hello dgarba,

may i confirm which JetPack version you were used?
thanks

dgarba · May 3, 2017, 3:24pm

Hi Jerry.

I am using Jetpack 3.0 for Tegra TX1.

Best regards,
-Daniel

JerryChang · May 4, 2017, 3:20am

hello dgarba,

attach the gstreamer library which tune the maximum frame rate limitation from 120 to 240 fps.
please have a try, thanks
libgstnvcamera_240.tar.gz (37.9 KB)

dgarba · May 4, 2017, 3:55pm

Thanks a lot for your help. I will give it a try and test it.

dgarba · May 4, 2017, 6:04pm

Hi Jerry.

I tested the provided binary of libgstnvcamera.so, but it doesn’t works at all on Tegra TX1 Jetpack 3.0 image. I tested with different framerate and resolution configurations, but the pipeline just get freezed and doesn’t give any useful log information.

I also test the provided binary of libgstnvcamera.so on a Tegra TX2 Jetpack 3.0 image and it works fine. I only tested resolutions in the following range: [30,120] fps.

I think that the provided binary was built for Tegra TX2, Jetpack 3.0, Kernel version 4.4.15. That is the reason why it works on TX2 but no in TX1.

Could you please provide me a new binary version of libgstnvcamera.so that supports 240fps for Tegra TX1, Jetpack 2.3.1, kernel version 3.10.96.

Thanks in advance for your help!

Best regards,
Daniel

dgarba · May 5, 2017, 12:25am

Hi Jerry.

I also found that nveglstreamsrc gst plugin (/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnveglstreamsrc.so) has the same 120fps limitation in the pad caps. I need to test it using libargus with a camera that can provide 240 fps, but that 120fps limitation is blocking me.

Please could you be kind enough to provide me a binary version of libgstnveglstreamsrc.so that can handle 240fps?

In summary I need libgstnveglstreamsrc.so and libgstnvcamera.so new version binaries that can handle 240 fps. I need them for Tegra TX1 using Jetpack 2.3.1 image with kernel version 3.10.96.

Thanks in advance for your help!

Best regards,
Daniel

JerryChang · May 5, 2017, 3:13am

hello dgarba,

i’ve update the fps limitation to 240 for tx1 and upload the libraries to the attachment.
since we did not have high framerate sensor on hand, please help doing verification, thanks
tx1_240fps.tar.gz (49 KB)

dgarba · May 5, 2017, 11:31pm

Hi Jerry,

Thanks a lot for your help, it was extremely useful.

I tested both binaries and have comments for both of them.

libgstnveglstreamsrc.so:

I tested libgstnveglstreamsrc.so with a the modified tegra_multimedia_api/argus/build/samples/gstVideoEncode sample app to support 240 fps. The app runs fine, the caps negotiation error that appears when using the libgstnveglstreamsrc.so (120 fps limited version) doesn’t appears any more. But, there is a problem, I can’t reach the 240 fps. The measurements gives a maximum framerate of 155 fps. This app uses libargus to capture the video and then generate an EGL stream that is passed to a GStreamer pipeline. I am not sure if the lower framerate issue (obtain 155 fps when expect 240 fps) is caused by a limitation of libgstnveglstreamsrc.so binary you provide me or by a libargus limitation.

Does libargus API has any limitation on the framerate capabilities or the bottleneck should be caused by passing the buffers to GStreamer?
Are there some memcpys between the interface of libargus and nveglstreamsrc, that could affect the performance?

I tested the sensor with v4l2src using a gst-launch pipeline I obtained 240fps in the measurements reported by gst-perf element, so the camera sensor could give the 240fps with V4l2src but not with libargus. The camera sensor gives raw Bayer output, but I need YUV, so passing through the ISP module is necessary, but V4lsrc doesn’t have access to the ISP.

I am an using an IMX274 sensor (developed by Leopard Imaging) that has 1280x540@240fps mode.

I will appreciate any help with this issue, I am so close of my goal.

libgstnvcamera.so:

I also tested the modified libgstnvcamera.so binary that supports 240fps. I ran some GStreamer pipelines with gst-launch in the 3 Modes supported by the IMX274 driver (developed by Leopard Imaging). The modes are:

Mode 0 : 3840x2160@60fps
Mode 1 : 1920x1080@60fps
Mode 2 : 1280x540@240fps

The test pipeline was:

gst-launch-1.0 -v nvcamerasrc sensor-id=0 fpsRange="60 60" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)60/1' ! fakesink

If I adjust the above pipeline with the settings of Mode 0 and Mode 1, everything works fine, and the desired frame rate is achieved. Also, if I modify the fpsRange=“30 30” and framerate=(fraction)30/1, I got 30 fps in the measurements, even though this framerate isn’t supported in any of the camera driver modes.

If I adjust the above pipeline with the settings of Mode 2, I got the following error:

ubuntu@tegra-ubuntu:~$ GST_DEBUG=3 gst-launch-1.0 -v nvcamerasrc sensor-id=0 fpsRange="240 240" ! 'video/x-raw(memory:NVMM), width=(int)1280, height=(int)540, format=(string)I420, framerate=(fraction)240/1' ! perf ! fakesink
Setting pipeline to PAUSED ...

Available Sensor modes : 
3840 x 2160 FR=60.000000 CF=0xf09208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1920 x 1080 FR=60.000000 CF=0xf09208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1280 x 540 FR=240.000000 CF=0xf09208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
0:00:00.315740239  4155       0x475770 FIXME                default gstutils.c:3766:gst_pad_create_stream_id_internal:<nvcamerasrc0:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id

NvCameraSrc: Trying To Set Default Camera Resolution. Selected 1280x540 FrameRate = 240.000000 ...

/GstPipeline:pipeline0/GstNvCameraSrc:nvcamerasrc0.GstPad:src: caps = "video/x-raw\(memory:NVMM\)\,\ width\=\(int\)1280\,\ height\=\(int\)540\,\ format\=\(string\)I420\,\ framerate\=\(fraction\)240/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = "video/x-raw\(memory:NVMM\)\,\ width\=\(int\)1280\,\ height\=\(int\)540\,\ format\=\(string\)I420\,\ framerate\=\(fraction\)240/1"
/GstPipeline:pipeline0/GstPerf:perf0.GstPad:src: caps = "video/x-raw\(memory:NVMM\)\,\ width\=\(int\)1280\,\ height\=\(int\)540\,\ format\=\(string\)I420\,\ framerate\=\(fraction\)240/1"
/GstPipeline:pipeline0/GstFakeSink:fakesink0.GstPad:sink: caps = "video/x-raw\(memory:NVMM\)\,\ width\=\(int\)1280\,\ height\=\(int\)540\,\ format\=\(string\)I420\,\ framerate\=\(fraction\)240/1"
/GstPipeline:pipeline0/GstPerf:perf0.GstPad:sink: caps = "video/x-raw\(memory:NVMM\)\,\ width\=\(int\)1280\,\ height\=\(int\)540\,\ format\=\(string\)I420\,\ framerate\=\(fraction\)240/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = "video/x-raw\(memory:NVMM\)\,\ width\=\(int\)1280\,\ height\=\(int\)540\,\ format\=\(string\)I420\,\ framerate\=\(fraction\)240/1"
GST-PERF ->  Timestamp: 0:03:31.503316175; Bps: 0; fps: 0.0 
INFO:
Timestamp: 0:03:31.503316175; Bps: 0; fps: 0.0
Socket read error. Camera Daemon stopped functioning.....
Received error from camera daemon....exiting....
Got EOS from element "pipeline0".
Execution ended after 0:00:01.673534993
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
0:00:01.989481842  4155       0x51fb50 ERROR            nvcamerasrc gstnvcamerasrc.cpp:1953:gst_nvcamera_close:<nvcamerasrc0> NvCameraSrc: terminate session request failed
Setting pipeline to NULL ...
Freeing pipeline ...

Also, if I modify the [fpsRange=“60 60”/framerate=(fraction)60/1] or [fpsRange=“30 30”/framerate=(fraction)30/1], I got 60 fps and 30 fps respectively in the measurements, even though this framerate isn’t supported in any of the camera driver modes.

I am not sure what is the cause of the Socket read error. Camera Daemon stopped functioning…

I will appreciate any help with this issue.

Thanks in advance for your help.

Best regards,
Daniel

DaneLLL · May 8, 2017, 9:08am

Hi Daniel,
Please modify gstVideoEncode to run
‘nveglstreamsrc ! fakesink’
‘nveglstreamsrc ! omxh264enc ! fakesink’
Would like to know if the two cases can reach 240fps.

Please also try max performance by running ~/jetson_clocks.sh

dgarba · May 8, 2017, 5:30pm

Hi DaneLLL

I modified the gstVideoEncode examples as you suggest and measured the framerate before and after running the ./jetson_clocks.sh

This is are the obtained results:

gstVideoEncode modified as:
nveglstream ! capsfilter ! queue ! perf ! fakesink

Before running ~/jetson_clocks.sh:
----> Got 150 fps measured

After running ~/jetson_clocks.sh:
----> Got 170 fps measured

gstVideoEncode modified as:
nveglstream ! capsfilter ! queue ! perf ! omxh264enc ! fakesink

Before running ~/jetson_clocks.sh:
----> Got 155 fps measured

After running ~/jetson_clocks.sh:
----> Got 175 fps measured

I still don’t achieve the 240fps.
Does libargus API has any limitation on the framerate capabilities or the bottleneck should be caused by passing the buffers to GStreamer?
Are there some memcpys between the interface of libargus and nveglstreamsrc, that could affect the performance?
Is the issue caused by modified version of the libgstnveglstreamsrc.so binary (supports 240 fps) you provided me?

Do you have any clue about the libgstnvcamera.so issues I reported in my previous comment? Nobody gives a reply or comment related with it.

I will appreciate any help with this issues.

Thanks in advance for your help.

Best regards,
Daniel

JerryChang · May 11, 2017, 10:06am

hello dgarba,

thanks for sharing the testing result.
may i know which feature you’re going to develop need such high frame rate? slow motion?

moreover,
could you kindly taking another verification on JetsonTX2?
in theory, you may approach your frame rate requirement on TX2.
attach the library for your testing,
thanks
tx2_240fps.tar.gz (50.7 KB)

dgarba · May 11, 2017, 3:52pm

Hi Jerry,

I can’t give full details of the feature I am developing. At this point I am trying to validate if the Tegra TX1 can handle video capture of 1280x540@240fps and passing the frames to a simple CUDA algorithm that process them. For example the algorithm could be something like “implement SUM of a 512x512 ROI and return the SUM”. I do not need to encode the video stream, only process it with CUDA.

I have some questions on will I appreciate any related help:

Does the TX1 can handle what I am trying to do?
On TX1 I need to know if there are performance limitations in the link between libargus and nveglstream due to memory copies, that is causing the bottleneck?
On TX1 I need to know if libargus is capable to handle a video capture of 1280x540@240fps?

I will modify the IMX274 driver to make it compatible for Tegra TX2, and test if TX2 can handle a video stream of 1280x540@240fps.

At this point I am more interested in TX1, because of project specifications. So I will really appreciate if you could give some clue about the above questions.

Thanks for all your help.

Best regards,
Daniel

DavidSoto-RidgeRun · May 25, 2017, 1:31pm

Now we are able to capture 240fps with nvcamerasrc, thanks for all the help.

https://devtalk.nvidia.com/default/topic/1009706/jetson-tx1/libargus-repeating-or-interpolating-frames-/post/5154377/#5154377

-David

JerryChang · June 30, 2017, 3:13am

hello DavidSoto,

good to know that works.
could you share the sensor init table of 1280x540@240fps to us for further testing?
thanks

dgarba · July 4, 2017, 3:51pm

Hello Jerry,

Here is the init table of the IMX274 sensor 1280x540@240fps mode:

IMX274 sensor 1280x540@240fps mode :

{
	{0x3004, 0x04},
	{0x3005, 0x31},
	{0x3006, 0x00},
	{0x3007, 0x02},
	{0x300C, 0xff}, 
	{0x300D, 0x00}, 
	{0x300E, 0x00}, 
	{0x300F, 0x00}, 
	{0x3018, 0xA2}, 
	{0x301A, 0x00},
	{0x306B, 0x05},
	{0x30E2, 0x04},
	{0x30F6, 0x04}, 
	{0x30F7, 0x01}, 
	{0x30F8, 0x83}, 
	{0x30F9, 0x04}, 
	{0x30FA, 0x00}, 
	{0x30dd, 0x01}, 
	{0x30de, 0x02},
	{0x30df, 0x00},
	{0x30e0, 0x01},
	{0x30e1, 0x00},
	{0x3037, 0x01},
	{0x3038, 0x0c},
	{0x3039, 0x00},
	{0x303a, 0x0c},
	{0x303b, 0x0f},
	{0x30EE, 0x01},
	{0x3130, 0x26},
	{0x3131, 0x02},
	{0x3132, 0x22},
	{0x3133, 0x02},
	{0x3342, 0x0A},
	{0x3343, 0x00},
	{0x3344, 0x1A},
	{0x3345, 0x00},
	{0x33A6, 0x01},
	{0x3528, 0x0E},
	{0x3554, 0x00},
	{0x3555, 0x01},
	{0x3556, 0x01},
	{0x3557, 0x01},
	{0x3558, 0x01},
	{0x3559, 0x00},
	{0x355A, 0x00},
	{0x35BA, 0x0E},
	{0x366A, 0x1B},
	{0x366B, 0x19},
	{0x366C, 0x17},
	{0x366D, 0x17},
	{0x3A41, 0x04},	
};

Thanks for all your help.

Best regards,
Daniel

SimonZhu · July 5, 2017, 11:21pm

Thanks All!

This is Simon from Leopard Imaging. We just confirmed that we can capture 240fps with nvcamerasrc after loading the 240fps libraries from Jerry and updating the driver based on the setting from Daniel.

ubuntu@tegra-ubuntu:~$ v4l2-ctl --set-fmt-video=width=1280,height=540 --stream-user --stream-count=1000 -d /dev/video0
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 241.00 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 240.50 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 240.33 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 240.25 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
ubuntu@tegra-ubuntu:~$ gst-launch-1.0 nvcamerasrc fpsRange="240.0 240.0" sensor-id=0 ! 'video/x-raw(memory:NVMM), width=(int)1280, height=(int)540, format=(string)I420, framerate=(fraction)240/1' ! nvtee ! nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e
Setting pipeline to PAUSED ...
Inside NvxLiteH264DecoderLowLatencyInitNvxLiteH264DecoderLowLatencyInit set DPB and MjstreamingInside NvxLiteH265DecoderLowLatencyInitNvxLiteH265DecoderLowLatencyInit set DPB and Mjstreaming
Available Sensor modes : 
3840 x 2160 FR=60.000000 CF=0xf09208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1920 x 1080 FR=60.000000 CF=0xf09208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1280 x 540 FR=240.000000 CF=0xf09208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock

NvCameraSrc: Trying To Set Default Camera Resolution. Selected 1280x540 FrameRate = 240.000000 ...

^Chandling interrupt.
Interrupt: Stopping pipeline ...
EOS on shutdown enabled -- Forcing EOS on the pipeline
Waiting for EOS...
Got EOS from element "pipeline0".
EOS received - stopping pipeline...
Execution ended after 0:00:08.047745064
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
ubuntu@tegra-ubuntu:~$

DavidSoto-RidgeRun · July 6, 2017, 12:26am

HI SimonZhu,

That is great, glad to hear that you got it working. It worked great for us too, next thing that I want to try is to feed the encoder with this stream and measure how fast the encoder can process this data rate. Not sure if it is limited to 60fps.

-David

SimonZhu · July 6, 2017, 1:06am

Hi David,

It should support 240fps encoder. We can capture 240fps video with below command or argus software.

gst-launch-1.0 -e nvcamerasrc sensor-id=0 fpsRange=“240.0 240.0” ! ‘video/x-raw(memory:NVMM), width=(int)1280, height=(int)540, format=(string)I420, framerate=(fraction)240/1’ ! nvvidconv ! ‘video/x-raw(memory:NVMM), format=(string)I420’ ! omxh264enc bitrate=8000000 ! ‘video/x-h264, stream-format=(string)byte-stream’ ! filesink location=IMX274-240fps_0.avi -e

Best regards,

Simon Zhu

JerryChang · July 6, 2017, 3:19am

hello SimonZhu,

could you help to review the sensor table 1280x540@240fps shared by dgarba at comment#16
we would like to check-in this sensor mode, thanks