JP4.3 L4T32.3.1 gstreamer v4l2src low fps

ahuska · September 23, 2020, 7:49pm

Hello,

I have a OV10635 sensor with MAX9271 and MAX9286 SerDes combination. In JP4.2.1 it works fine with gstreamer v4l2src at 30fps. Recently I upgraded the system to JP4.4 to use Isaac 2020.1NX and camera performance was the same. However, I discovered a lack of support for StereoDepthEstimation in 2020.1NX and had to downgrade to 2020.1 which is based on JP4.3. After recompiling my drivers for 4.3 and installing all the frameworks on the TX2 I found that gst-launch-1.0 with v4l2src gives only 3.4fps. If I run the pipeline sink with sync=0 then it runs at 30fps, but in Isaac with Deepstream pipeline I only get 3.4fps and the sync flag in the pipeline string makes no difference.

What can be done to fix gstreamer in JP4.3? Or fix the Isaac Deepstream pipeline to work with the sync flag?

I have a similar topic in the Isaac forum: Stereo_depth missing from 2020.1NX

DaneLLL · September 24, 2020, 1:41am

Hi,
Please share information of the source for reference:

$ v4l2-ctl -d /dev/video0 --list-formats-ext

ahuska · September 24, 2020, 4:06pm

Hello DaneLLL,

Here are some debugging outputs:

ubuntu@tegra-ubuntu-tx2-8gb-0:~$ v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
Index       : 0
Type        : Video Capture
Pixel Format: 'UYVY'
Name        : UYVY 4:2:2
	Size: Discrete 1280x800
		Interval: Discrete 0.033s (30.000 fps)

Index       : 1
Type        : Video Capture
Pixel Format: 'NV16'
Name        : Y/CbCr 4:2:2
	Size: Discrete 1280x800
		Interval: Discrete 0.033s (30.000 fps)

Index       : 2
Type        : Video Capture
Pixel Format: 'UYVY'
Name        : UYVY 4:2:2
	Size: Discrete 1280x800
		Interval: Discrete 0.033s (30.000 fps)

ubuntu@tegra-ubuntu-tx2-8gb-0:~$

Low fps launch command

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=UYVY,height=800,width=1280,framerate=30/1 ! videoconvert ! aasink

Correct fps launch command

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=UYVY,height=800,width=1280,framerate=30/1 ! videoconvert ! aasink sync=0

v4l2-ctl test of framerate

ubuntu@tegra-ubuntu-tx2-8gb-0:~$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=1280,height=800,pixelformat=UYVY --stream-mmap --stream-count=121
    <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.00 fps
    <<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.00 fps
    <<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.00 fps
    <<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.00 fps

DaneLLL · September 24, 2020, 4:51pm

Hi,
Please run sudo jetson_clocks and try the pipeline:

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=UYVY,height=800,width=1280,framerate=30/1 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12' ! fpsdisplaysink text-overlay=0 video-sink=nvoverlaysink sync=0 -v

ahuska · September 24, 2020, 5:08pm

Here is the output:

ubuntu@tegra-ubuntu-tx2-8gb-0:/opt/nvidia/deepstream/deepstream-4.0/sources/apps/sample_apps$ gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=UYVY,height=800,width=1280,framerate=30/1 ! nvvidconv ! ‘video/x-raw(memory:NVMM),format=NV12’ ! fpsdisplaysink text-overlay=0 video-sink=nvoverlaysink sync=0 -v
Setting pipeline to PAUSED …
Pipeline is live and does not need PREROLL …
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstNvOverlaySink-nvoverlaysink:nvoverlaysink-nvoverlaysink0: sync = false
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, format=(string)UYVY, height=(int)800, width=(int)1280, framerate=(fraction)30/1, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, format=(string)UYVY, height=(int)800, width=(int)1280, framerate=(fraction)30/1, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = video/x-raw(memory:NVMM), height=(int)800, width=(int)1280, framerate=(fraction)30/1, interlace-mode=(string)progressive, format=(string)NV12
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-raw(memory:NVMM), height=(int)800, width=(int)1280, framerate=(fraction)30/1, interlace-mode=(string)progressive, format=(string)NV12
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-raw(memory:NVMM), height=(int)800, width=(int)1280, framerate=(fraction)30/1, interlace-mode=(string)progressive, format=(string)NV12
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstNvOverlaySink-nvoverlaysink:nvoverlaysink-nvoverlaysink0.GstPad:sink: caps = video/x-raw(memory:NVMM), height=(int)800, width=(int)1280, framerate=(fraction)30/1, interlace-mode=(string)progressive, format=(string)NV12
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink: caps = video/x-raw(memory:NVMM), height=(int)800, width=(int)1280, framerate=(fraction)30/1, interlace-mode=(string)progressive, format=(string)NV12
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-raw(memory:NVMM), height=(int)800, width=(int)1280, framerate=(fraction)30/1, interlace-mode=(string)progressive, format=(string)NV12
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = video/x-raw, format=(string)UYVY, height=(int)800, width=(int)1280, framerate=(fraction)30/1, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, format=(string)UYVY, height=(int)800, width=(int)1280, framerate=(fraction)30/1, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive
Setting pipeline to PLAYING …
New clock: GstSystemClock
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstNvOverlaySink-nvoverlaysink:nvoverlaysink-nvoverlaysink0: sync = false
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 16, dropped: 0, current: 31.63, average: 31.63
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 32, dropped: 0, current: 30.00, average: 30.80
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 47, dropped: 0, current: 29.99, average: 30.54
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 63, dropped: 0, current: 30.19, average: 30.45
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 79, dropped: 0, current: 30.13, average: 30.38
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 95, dropped: 0, current: 29.91, average: 30.30
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 110, dropped: 0, current: 29.74, average: 30.22
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 125, dropped: 0, current: 29.98, average: 30.19
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 141, dropped: 0, current: 30.05, average: 30.18
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 156, dropped: 0, current: 29.95, average: 30.16
^Chandling interrupt.
Interrupt: Stopping pipeline …
Execution ended after 0:00:05.334710460
Setting pipeline to PAUSED …
Setting pipeline to READY …
Setting pipeline to NULL …
Freeing pipeline …

Honey_Patouceul · September 24, 2020, 5:20pm

From this log, it seems working fine.
Can’t you see the display on local screen ?
Note that nvoverlaysink only works with a local display, not remotely over ssh with X forwarding (you would have to use xvimagesink in such case, but this might be slower).

ahuska · September 24, 2020, 6:38pm

Yes, the framerate is correct when using nvvidconv as shown above. It is also ok with or without the sync=0 argument; also ok with NV12, RGBA, or UYVY color; and also ok with video/x-raw(memory:NVMM) or video x/raw. For example:

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=UYVY,height=800,width=1280,framerate=30/1 ! nvvidconv ! ‘video/x-raw,format=UYVY’ ! fpsdisplaysink text-overlay=0 video-sink=fakesink -v
Setting pipeline to PAUSED …
Pipeline is live and does not need PREROLL …
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
Setting pipeline to PLAYING …
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, format=(string)UYVY, height=(int)800, width=(int)1280, framerate=(fraction)30/1, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, format=(string)UYVY, height=(int)800, width=(int)1280, framerate=(fraction)30/1, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive
New clock: GstSystemClock
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = video/x-raw, height=(int)800, width=(int)1280, framerate=(fraction)30/1, interlace-mode=(string)progressive, format=(string)UYVY
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-raw, height=(int)800, width=(int)1280, framerate=(fraction)30/1, interlace-mode=(string)progressive, format=(string)UYVY
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-raw, height=(int)800, width=(int)1280, framerate=(fraction)30/1, interlace-mode=(string)progressive, format=(string)UYVY
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0.GstPad:sink: caps = video/x-raw, height=(int)800, width=(int)1280, framerate=(fraction)30/1, interlace-mode=(string)progressive, format=(string)UYVY
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink: caps = video/x-raw, height=(int)800, width=(int)1280, framerate=(fraction)30/1, interlace-mode=(string)progressive, format=(string)UYVY
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-raw, height=(int)800, width=(int)1280, framerate=(fraction)30/1, interlace-mode=(string)progressive, format=(string)UYVY
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = video/x-raw, format=(string)UYVY, height=(int)800, width=(int)1280, framerate=(fraction)30/1, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, format=(string)UYVY, height=(int)800, width=(int)1280, framerate=(fraction)30/1, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 17, dropped: 0, current: 31.89, average: 31.89
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 32, dropped: 0, current: 30.00, average: 30.98
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 47, dropped: 0, current: 29.99, average: 30.65
^Chandling interrupt.
Interrupt: Stopping pipeline …
Execution ended after 0:00:01.890918294
Setting pipeline to PAUSED …
Setting pipeline to READY …
Setting pipeline to NULL …
Freeing pipeline …

However nvvidconv is not able to output RGB format which NVIDIA Isaac components require…even though ColorCameraProto shows that it can hold an RGBA formatted image all of the components I’m using indicate they require RGB. I’ve been trying to develop a pipeline to convert RGBA to CPU memory RGB, but so far nothing is working.

Honey_Patouceul · September 24, 2020, 6:44pm

nvvidconv doesn’t handle 24 bits formats so far, AFAIK. So you would have to convert with nvvidconv into RGBA or BGRx and use videoconvert for removing the extra 4th byte.
For high resolutions/framerates, an alternative to opencv videoio is to use jetson-utils (if you mind GPU processing, you may check this post. For python it might be more complex).

DaneLLL · September 25, 2020, 1:41am

Hi,
The issue looks specific to using Isaac SDK. Should run fine in general gstreamer usercase and DeepStream SDK. Since BGR format is not supported by hardware converter and relies on CPU, you may run sudp nvpmodel -m 0 and sudo jetson_clocks to keep system at max performance. See if this brings improvement.

ahuska · September 28, 2020, 8:45pm

Hi All,

I disagree with the assumption that it is an Isaac issue due to the fact that these launch commands (from above) have nothing to do with Isaac:

Low fps launch command

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=UYVY,height=800,width=1280,framerate=30/1 ! videoconvert ! aasink

Correct fps launch command

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=UYVY,height=800,width=1280,framerate=30/1 ! videoconvert ! aasink sync=0

And, in 4.2.1 and 4.4 both of these commands run at 30fps, but in 4.3 the sync=0 is required to achieve 30fps. However, I have no idea how to overcome this problem, even by recompiling gstreamer for 4.3, since I know very little of the underlying gstreamer library.

I also have been unsuccessful in creating pipelines that allow me to convert from video/x-raw(memory:NVMM) in RGBA to video/x-raw in RGB.

DaneLLL · September 29, 2020, 3:21am

Hi,
To have optimal performance on Jetson platforms, we would suggest use NVIDIA plugins to capture and pass NVMM buffers in the pipeline. For the pipeline

v4l2src ! videoconvert ! aasink

All elements are gstreamer native plugins and use CPU. The performance may be capped by full CPU loading. Please try MAXN mode(sudo nvpmodel -m 0) listed in development guide and execute sudo jetson_clocks.

Topic		Replies	Views
Camera's frame rate unstable Jetson AGX Xavier camera	22	4635	October 18, 2021
Xavier AGX deepstream5.1 sample app : no camera available error DeepStream SDK	19	1495	February 1, 2022
Problem using gstreamer with opencv and cuda on TX2 with Jetpack 4.3 Jetson TX2 rtsp , opencv , cuda , gstreamer	21	3337	October 18, 2021
GPU/HW acceleration with GStreamer nvvidconv Jetson Orin Nano gstreamer	6	1685	October 25, 2023
Deepstream-app custom CSI v4l2src Camera frame drop DeepStream SDK	6	1264	October 12, 2021
How can I customize to use non-blocking mode nvv4l2h264enc Jetson Xavier NX encoder	11	2293	April 27, 2022
TX2 VI mode and ISP mode bugs when using CSI2 virtual channels Jetson TX2 camera , nvbugs	15	1322	March 9, 2022
Gstreamer TCPserversink 2-3 seconds latency Jetson Xavier NX camera , opencv , gstreamer	12	18548	October 3, 2021
Gstreamer use MJPEG codec Jetson Nano opencv , gstreamer	16	16748	October 15, 2021
Failing to get input from CSI camera (IMX219-83) Jetson Nano camera , ubuntu , gstreamer	29	1533	September 25, 2023

JP4.3 L4T32.3.1 gstreamer v4l2src low fps

Related topics