Gstreamer v4l2src 4Kp60 source encoding & preview problem -> Framerate is poor.

I have been testing 4K(3840x2160)p60 encoding using Nvidia TX2 board.
First we have made our own 4K input board which is capable to receive up to 4Kp60 HDMI input and output CSI-2 format output.
I have verified 4Kp60 input as follows.

nvidia@tegra-ubuntu:~ v4l2-ctl -d /dev/video0 --set-fmt-video=width=3840,height=2160,pixelformat=UYVY --set-ctrl bypass_mode=0 --stream-mmap --stream-count=600 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.00 fps <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.00 fps <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.00 fps <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.00 fps <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.00 fps <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.00 fps <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.00 fps <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.00 fps <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.00 fps <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< nvidia@tegra-ubuntu:~

And then I have tested a preview with various methods as follows
This method is working but the rendering frame-rate is very low.

nvidia@tegra-ubuntu:~$ DISPLAY=:0 gst-launch-1.0 v4l2src device="/dev/video0" do-timestamp=true ! \

“video/x-raw, width=3840, height=2160, framerate=60/1, format=(string)I420” !
nveglglessink sync=true

Another following method is working well. Its frame-rate is much better than the previous one.
But its frame-rate also was not 60Hz. It looks be about 30Hz.

nvidia@tegra-ubuntu:~$ DISPLAY=:0 gst-launch-1.0 v4l2src device="/dev/video0" do-timestamp=true ! \

‘video/x-raw, width=3840, height=2160, framerate=60/1, format=UYVY’ !
nvvidconv ! ‘video/x-raw(memory:NVMM),format=RGBA,width=1920,height=1080,framerate=60/1’ !
nvegltransform ! nveglglessink -v

And then, I have tested a recording test.
The first method is followed. I used omxh264enc to encode and omxh264dec to decode.
As you see, when decoding the recorded file its FrameRate was also about 30 fps(in orange text).

nvidia@tegra-ubuntu:~$ gst-launch-1.0 v4l2src device="/dev/video0" io-mode=2 do-timestamp=true ! \

‘video/x-raw, width=3840, height=2160, framerate=60/1, format=I420’ ! queue !
omxh264enc bitrate=100000000 iframeinterval=60 ! ‘video/x-h264, level=(string)5.2’ !
qtmux ! filesink location=test.mp4 -e
Setting pipeline to PAUSED …
Pipeline is live and does not need PREROLL …
Setting pipeline to PLAYING …
New clock: GstSystemClock
Framerate set to : 60 at NvxVideoEncoderSetParameterNvMMLiteOpen : Block : BlockType = 4
===== MSENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
===== MSENC blits (mode: 1) into tiled surfaces =====
^Chandling interrupt.
Interrupt: Stopping pipeline …
EOS on shutdown enabled – Forcing EOS on the pipeline

** (gst-launch-1.0:29166): CRITICAL **: gst_v4l2_buffer_pool_process: assertion ‘gst_buffer_pool_is_active (bpool)’ failed
Waiting for EOS…
Got EOS from element “pipeline0”.
EOS received - stopping pipeline…
Execution ended after 0:00:19.455003216
Setting pipeline to PAUSED …
Setting pipeline to READY …
Setting pipeline to NULL …
Freeing pipeline …
nvidia@tegra-ubuntu:~$ DISPLAY=:0 gst-launch-1.0 filesrc location=test.mp4 ! \

qtdemux name=demux demux.video_0 ! queue ! h264parse ! omxh264dec !
nvegltransform ! nveglglessink
Setting pipeline to PAUSED …

Using winsys: x11
Pipeline is PREROLLING …
Got context from element ‘eglglessink0’: gst.egl.EGLDisplay=context, display=(GstEGLDisplay)NULL;
NvMMLiteOpen : Block : BlockType = 261
TVMR: NvMMLiteTVMRDecBlockOpen: 7647: NvMMLiteBlockOpen
NvMMLiteBlockCreate : Block : BlockType = 261
TVMR: cbBeginSequence: 1179: BeginSequence 3840x2160, bVPR = 0
TVMR: LowCorner Frequency = 345000
TVMR: cbBeginSequence: 1529: DecodeBuffers = 17, pnvsi->eCodec = 4, codec = 0
TVMR: cbBeginSequence: 1600: Display Resolution : (3840x2160)
TVMR: cbBeginSequence: 1601: Display Aspect Ratio : (3840x2160)
TVMR: cbBeginSequence: 1669: ColorFormat : 5
TVMR: cbBeginSequence:1683 ColorSpace = NvColorSpace_YCbCr601
TVMR: cbBeginSequence: 1809: SurfaceLayout = 3
TVMR: cbBeginSequence: 1902: NumOfSurfaces = 24, InteraceStream = 0, InterlaceEnabled = 0, bSecure = 0, MVC = 0 Semiplanar = 1, bReinit = 1, BitDepthForSurface = 8 LumaBitDepth = 8, ChromaBitDepth = 8, ChromaFormat = 5
TVMR: cbBeginSequence: 1904: BeginSequence ColorPrimaries = 2, TransferCharacteristics = 2, MatrixCoefficients = 2
Allocating new output: 3840x2160 (x 24), ThumbnailMode = 0
OPENMAX: HandleNewStreamFormat: 3464: Send OMX_EventPortSettingsChanged : nFrameWidth = 3840, nFrameHeight = 2160
Pipeline is PREROLLED …
Setting pipeline to PLAYING …
New clock: GstSystemClock
TVMR: FrameRate = 34
TVMR: NVDEC LowCorner Freq = (391000 * 1024)
TVMR: FrameRate = 31.900776
TVMR: FrameRate = 33.117188
TVMR: FrameRate = 33.437100
TVMR: FrameRate = 33.305579
TVMR: FrameRate = 32.242255
TVMR: NvMMLiteTVMRDecDoWork: 6531: NVMMLITE_TVMR: EOS detected
TVMR: TVMRBufferProcessing: 5486: Processing of EOS
TVMR: TVMRBufferProcessing: 5563: Processing of EOS Done
Got EOS from element “pipeline0”.
Execution ended after 0:00:18.526989237
Setting pipeline to PAUSED …
Setting pipeline to READY …
TVMR: TVMRFrameStatusReporting: 6132: Closing TVMR Frame Status Thread -------------
TVMR: TVMRVPRFloorSizeSettingThread: 5942: Closing TVMRVPRFloorSizeSettingThread -------------
TVMR: TVMRFrameDelivery: 5982: Closing TVMR Frame Delivery Thread -------------
TVMR: NvMMLiteTVMRDecBlockClose: 7815: Done
Setting pipeline to NULL …
Freeing pipeline …
nvidia@tegra-ubuntu:~$

As another method, I have inserted a ‘nvvidconv’ pipeline between v4l2src and omxh264enc, but the result was worse as follows.
This time the FrmateRate was about 11 fps.

nvidia@tegra-ubuntu:~$ gst-launch-1.0 v4l2src device="/dev/video0" do-timestamp=true ! \

‘video/x-raw, width=3840, height=2160, framerate=60/1, format=UYVY’ !
nvvidconv ! ‘video/x-raw(memory:NVMM),format=I420,width=3840,height=2160,framerate=60/1’ !
omxh264enc bitrate=100000000 iframeinterval=60 ! ‘video/x-h264, level=(string)5.2’ !
qtmux ! filesink location=test.mp4 -e
Setting pipeline to PAUSED …
Pipeline is live and does not need PREROLL …
Setting pipeline to PLAYING …
New clock: GstSystemClock
Framerate set to : 60 at NvxVideoEncoderSetParameterNvMMLiteOpen : Block : BlockType = 4
===== MSENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
===== MSENC blits (mode: 1) into tiled surfaces =====
^Chandling interrupt.
Interrupt: Stopping pipeline …
EOS on shutdown enabled – Forcing EOS on the pipeline
Waiting for EOS…
Got EOS from element “pipeline0”.
EOS received - stopping pipeline…
Execution ended after 0:00:12.385070051
Setting pipeline to PAUSED …
Setting pipeline to READY …
Setting pipeline to NULL …
Freeing pipeline …
nvidia@tegra-ubuntu:~$ DISPLAY=:0 gst-launch-1.0 filesrc location=test.mp4 ! qtdemux name=demux demux.video_0 ! queue ! h264parse ! omxh264dec ! nvegltransform ! nveglglessinkSetting pipeline to PAUSED …

Using winsys: x11
Pipeline is PREROLLING …
Got context from element ‘eglglessink0’: gst.egl.EGLDisplay=context, display=(GstEGLDisplay)NULL;
NvMMLiteOpen : Block : BlockType = 261
TVMR: NvMMLiteTVMRDecBlockOpen: 7647: NvMMLiteBlockOpen
NvMMLiteBlockCreate : Block : BlockType = 261
TVMR: cbBeginSequence: 1179: BeginSequence 3840x2160, bVPR = 0
TVMR: LowCorner Frequency = 345000
TVMR: cbBeginSequence: 1529: DecodeBuffers = 17, pnvsi->eCodec = 4, codec = 0
TVMR: cbBeginSequence: 1600: Display Resolution : (3840x2160)
TVMR: cbBeginSequence: 1601: Display Aspect Ratio : (3840x2160)
TVMR: cbBeginSequence: 1669: ColorFormat : 5
TVMR: cbBeginSequence:1683 ColorSpace = NvColorSpace_YCbCr601
TVMR: cbBeginSequence: 1809: SurfaceLayout = 3
TVMR: cbBeginSequence: 1902: NumOfSurfaces = 24, InteraceStream = 0, InterlaceEnabled = 0, bSecure = 0, MVC = 0 Semiplanar = 1, bReinit = 1, BitDepthForSurface = 8 LumaBitDepth = 8, ChromaBitDepth = 8, ChromaFormat = 5
TVMR: cbBeginSequence: 1904: BeginSequence ColorPrimaries = 2, TransferCharacteristics = 2, MatrixCoefficients = 2
Allocating new output: 3840x2160 (x 24), ThumbnailMode = 0
OPENMAX: HandleNewStreamFormat: 3464: Send OMX_EventPortSettingsChanged : nFrameWidth = 3840, nFrameHeight = 2160
Pipeline is PREROLLED …
Setting pipeline to PLAYING …
New clock: GstSystemClock
TVMR: FrameRate = 11
TVMR: NVDEC LowCorner Freq = (126500 * 1024)
—> TVMR: Video-conferencing detected !!!
TVMR: FrameRate = 11.272026
TVMR: NvMMLiteTVMRDecDoWork: 6531: NVMMLITE_TVMR: EOS detected
TVMR: TVMRBufferProcessing: 5486: Processing of EOS
TVMR: TVMRBufferProcessing: 5563: Processing of EOS Done
Got EOS from element “pipeline0”.
Execution ended after 0:00:11.718211733
Setting pipeline to PAUSED …
Setting pipeline to READY …
TVMR: TVMRFrameStatusReporting: 6132: Closing TVMR Frame Status Thread -------------
TVMR: TVMRVPRFloorSizeSettingThread: 5942: Closing TVMRVPRFloorSizeSettingThread -------------
TVMR: TVMRFrameDelivery: 5982: Closing TVMR Frame Delivery Thread -------------
TVMR: NvMMLiteTVMRDecBlockClose: 7815: Done
Setting pipeline to NULL …
Freeing pipeline …
nvidia@tegra-ubuntu:~$

When testing test video source with ‘vidtestsrc’ even the gstreamer command is identical to the above method
its framerate was almost 60 fps. This result makes sure the TX2 encoding performance of 60Hz video has no problem.

I have also monitored the CPU usage all the times, It was always within reasonable range.
When encoding 4Kp60 source, it was about 170%.

What’s the problem in this result?

Hi otkim,
For v4l2 source, we suggest you try tegra_multimedia_api. Below is a sample for your reference.
https://devtalk.nvidia.com/default/topic/1031967/jetson-tx2/tegra_multimedia_api-dq-buffer-from-encoder-output_plane-can-not-completed/post/5251268/#5251268