Fps is reduced to 48 when running two 4k camera in 60fps in xavier nx platform

[Mr_JP]

Hi Team

We are trying to stream two 4k camera in 60 fps using nvcompositor in gstreamer, and we faced some fps loss in compositor view that is the fps of both camera is reduced to a varying range of 43 - 47fps.

So in order to debug the issue we used below pipeline to run both camera simultaneously, by running it in two different terminals. But when we run both the cameras simultaneously then the fps of both cameras is reduce to a fixed value of 47fps.

But when we run the cameras individually we are getting 60 fps.
We have also tried with fake sink and the result is the same.

WIDTH_IN=3840
HEIGHT_IN=2160
WIDTH_CROP=3840
HEIGHT_CROP=2160
DOWNSCALE=2
LEFT_CROP=$((($WIDTH_IN - $WIDTH_CROP)/2))
TOP_CROP=$((($HEIGHT_IN - $HEIGHT_CROP)/2))
RIGHT_CROP=$(($LEFT_CROP + $WIDTH_CROP))
BOTTOM_CROP=$(($TOP_CROP + $HEIGHT_CROP))
WIDTH_OUT=$(($WIDTH_CROP/$DOWNSCALE))
HEIGHT_OUT=$(($HEIGHT_CROP/$DOWNSCALE))
ARGUS=exposuretimerange="33000000 33000000"

CAPS_IN="video/x-raw(memory:NVMM), width=$WIDTH_IN, height=$HEIGHT_IN, format=NV12"
#CAPS_OUT="video/x-raw(memory:NVMM), width=$WIDTH_OUT, height=$HEIGHT_OUT, format=RGBA, pixel-aspect-ratio=$WIDTH_OUT/$HEIGHT_OUT"
CAPS_OUT="video/x-raw(memory:NVMM), width=$WIDTH_OUT, height=$HEIGHT_OUT, pixel-aspect-ratio=$WIDTH_OUT/$HEIGHT_OUT"
CAPS_COMP="video/x-raw(memory:NVMM), width=$WIDTH_OUT, height=$HEIGHT_OUT"
CROP="left=$LEFT_CROP top=$TOP_CROP right=$RIGHT_CROP bottom=$BOTTOM_CROP flip-method=6"

gst-launch-1.0 -v nvarguscamerasrc sensor-id=1 "$ARGUS" !  $CAPS_IN ! nvvidconv $CROP ! $CAPS_OUT ! nvoverlaysink sync=false 

Can you suggest us some methods to debug this issue. Or is it a limitation of xavier NX ?

[DaneLLL]

Hi,
By default, hardware converter runs in dynamic frequency scaling. Please follow the steps to run it at maximum clock and check if there is improvement:
Nvvideoconvert issue, nvvideoconvert in DS4 is better than Ds5? - #3 by DaneLLL

[Mr_JP]

Hi DaneLLL

We have tried the above methods but our scenario remains the same and we were not able to achieve any increase in fps.

But on further inspecting the gstreamer elements we have understood that the NV12 → RGBA conversion is causing the drop.

Is there any other suggestions to improve the fps. Or is there any other element that can help us with the conversion without affecting the performance.

[DaneLLL]

Hi,
If you have run hardware converter at maximum clock, this is the maximum throughput the system can offer. One more thing you can try is to customize nvarguscamerasrc to output RGBA directly. This scan save the conversion through nvvidconv plugin. The plugins are open source and please take a look at the source code.

[Mr_JP]

Hi DaneLLL

Do you have any reference documents related to this.

[DaneLLL]

Hi,
Plese download the package from:
https://developer.nvidia.com/embedded/linux-tegra-r3273

Driver Package (BSP) Sources

And the implementation is based on NvBuffer APIs. For more information, please check the document:

Jetson Linux API Reference (formerly named Multimedia API Reference

And after installing SDK components, you can see the samples on Xavier NX:

/usr/src/jetson_multimedia_api/samples/

Please take a look. The links are for Jetpack 4.6.3(r32.7.3). Please go to the release page fitting your version.

[Dilna]

Hi DaneLL,
We have tried to customize nvarguscamerasrc to output RGBA directly. for this we have made some modification in gstnvarguscamerasrc.cpp file. we tried to replace NV12 pixel format settings with RGBA. please find the attached patch file below.

gst-launch-1.0 -v nvarguscamerasrc sensor-id=1 ! nvoverlaysink sync=false
this is the pipeline now we are using.

we are facing the error
"

ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0:
streaming stopped, reason not-negotiated (-4)
ERROR: pipeline doesn’t want to preroll.
Setting pipeline to PAUSED …
Setting pipeline to READY …

(gst-launch-1.0:13876): GLib-CRITICAL **: 14:48:15.389: g_thread_join: assertion ‘thread’ failed

(gst-launch-1.0:13876): GLib-CRITICAL **: 14:48:15.389: g_thread_join: assertion ‘thread’ failed

(gst-launch-1.0:13876): GLib-CRITICAL **: 14:48:15.389: g_queue_is_empty: assertion ‘queue != NULL’ failed

(gst-launch-1.0:13876): GLib-CRITICAL **: 14:48:15.389: g_queue_free: assertion ‘queue != NULL’ failed

(gst-launch-1.0:13876): GLib-CRITICAL **: 14:48:15.389: g_queue_free: assertion ‘queue != NULL’ failed
"

could you please suggest us how to modify nvarguscamerasrc to provide RGBA format at output.
rgbaconvarion.txt (2.7 KB)