Encoding for dual channel 1600x1200@60Hz video stream

We have following set-up on TX1 :

  1. Channel 1 : 4 channel CSI-2 receiving video - 1600x1200 @60 Hz, RGB888 data.
  2. Channel 2 : 4 channel CSI-2 receiving video - 1600x1200 @60 Hz, RGB888 data.

We are using gstreamer and V4L2 for video capture. RGB888 support is already there in V4L2 as per FOURCC, but not in sensor_common.c and camera_common.c, which shall be added.

We would like to know if we can capture dual channel as above ( 4 channel CSI-2, 1600x1200 @60Hz RGB888 data) and encode two channels concurrently it at either H.264 HP or H.265. How it get converted into YUV420 and YUV444 and what are the options for H.264 and H.265? If support at each stage of video pipeline ( CSI-2 -> NVCSI -> NVENC -> V4L2 -> GStreamer ) could be highlighted, it would help.

Also, what could be the bottleneck points in terms of hardware and software support, if any

Thanks and Regards,

Vikas Dwivedi

Hi @vikas.wfh ,

By 4 channel, do you mean 4 lanes support? If so then you can do the following assumption:

Camera Efficiency 85.00%
Lanes 4
Lane speed (Gbps) 2.5
Width 1600
Height 1200
bpp 10

MaxTheoretical FPS = (Efficiency) * (#lanes) * (BW_lane_in_bits) / (width * height * bpp)
= 85% * 4 * 2.5 Gbps / 1600 * 1200* 10
= 442.7083333 FPS

So from the MIPI CSI-2 part you will be fine to support capture at 60 Hz, as theoretical calculation is way higher.

About enconding you can use the plugins that have hardware support such as:

  • H264: nvv4l2h264enc, omxh264enc
  • H265: nvv4l2h265enc, omxh265enc

However the most evident bottleneck that I see in your use case is that you require RGB888, which may limit you to use the v4l2src plugin. You would get better performance if using an input source with the nvarguscamerasrc to take advantage of the direct memory map from (NVMM) but it only supports NV12 format.

Regards,
Fabian
www.ridgerun.com

Hi Fabian,

We have two more queries:

  1. Why you have taken BPP = 10 as our case is RGB888?

  2. As per your calculation 60 FPS is possible for CSI-2 interface. How we do same band-width calculation for encoder? Is it possible for encoder to encode dual channel 1600x1200 @60 Hz video stream?

Thanks and Regards,

Vikas Dwivedi

You may check with this command launching 2 similar gstreamer pipelines each doing RGB -> BRGx -> NV12(NVMM) -> H264 -> NV12(NVMM) -> NV12, so converting from your RGB source, encoding into H264 and decoding from H264:

gst-launch-1.0 -ev videotestsrc ! video/x-raw, format=RGB, width=1600, height=1200, framerate=60/1 ! videoconvert ! video/x-raw, format=BGRx ! nvvidconv ! nvv4l2h264enc ! nvv4l2decoder ! nvvidconv ! videoconvert ! fpsdisplaysink video-sink=fakesink text-overlay=false       videotestsrc ! video/x-raw, format=RGB, width=1600, height=1200, framerate=60/1 ! videoconvert ! video/x-raw, format=BGRx ! nvvidconv ! nvv4l2h264enc ! nvv4l2decoder ! nvvidconv ! videoconvert ! fpsdisplaysink video-sink=fakesink text-overlay=false

It works at 60 fps on my Xavier.

Hi @vikas.wfh,

  1. Yes, you are right. You can change the BPP to 8.
  2. I am not sure if there is a direct calculation for the encoding part, however if you use the hardware encoders I don’t expect it to be much problem.

Could you please try with the pipeline suggested by @Honey_Patouceul and provide the results.

Regards,
Fabian
www.ridgerun.com

Hello,

The video pipeline where we need to validate the bandwidth
for our scenario is as follows:
( CSI-2 -> NVCSI -> NVENC -> V4L2 -> GStreamer )

STAGE: CSI-2 Interface:

I understand you have done bandwidth calculations at the CSI-2 interface. where for RGB888 24 BPP should be taken. Hence the bandwidth calculation would be:

Camera Efficiency 85.00%
Lanes 4
Lane speed (Gbps) 2.5
Width 1600
Height 1200
bpp 24

MaxTheoretical FPS = (Efficiency) * (#lanes) * (BW_lane_in_bits) / (width * height * bpp)
= 85% * 4 * 2.5 Gbps / 1600 * 1200* 24
= 184.4618063 FPS

Please confirm.

STAGE: NVENC:

What is the maximum theoretical limit for H.264 and H.265 encoding when using accelerated Gstreamer application? If you can elaborate the method for bandwidth it shall help.

We can not try Honey_Patouceul suggestion right now as our custom video board for TX-1 interfacing is under fabrication. We shall certainly try once the board is up and running.

Regards,
Vikas Dwivedi

You can try it on a standard TX2 dev board. The video sources are SW simulated, so you don’t need real cameras. This will only test dual encoding/decoding.