Which method is the fastest to resize image and converion color format ?

We are developing video streaming server and client without using GStreamer.
This system needs a dynamic resolution changing function.

(1) Camera (USB3) --> Jetson TX1 (H.264 encode/RTP) --> Ethernet
(2) Ethernet --> Jetson TX1 (RTP/H.264 decode) --> HDMI Display

The resolution of camera and display is 4K (3,840x2,160).
The frame rate of camera is 30fps.
The color format:

  • Camera is YUV420 (I420). (or Bayer: using CSI camera in future)
  • Encoder is YUV420 (I420).
  • Decoder is YUV420 (NV12).
  • Display is RGB.

When the bandwidth of the network is not enough, we think to reduce the resolution of image.

  • 3,840 x 2,160 (Original)
  • 1,920 x 1,080 (1/2)
  • 1,280 x 720 (1/3)
  • 960 x 540 (1/4)
  • 640 x 360 (1/6)

We have to use some resize function, and by conditions use some conversion color format function.

(1) Resize from 4K to the other (shrink)

  • The CPU programming is too slow.
  • OpenCV(CPU) resize function is a little slow.
  • OpenCV(GPU) resize function is not tested yet.
  • The CUDA programming is trying now.

(2) Resize from the other to 4K (enlarge)

  • OpenCV(CPU) resize function is slow.
  • OpenCV(GPU) resize function is not tested yet.
  • CPU or GPU programming is complex (because YUV not RGB).
  • OpenGL texture mapping has the resize function. *
  • Renderer of the nVIDIA Multimedia API has a problem. **

(3) Conversion color format between YUV/Bayer/RGB

  • The CPU programing is too slow.
  • OpenCV(CPU) cvtColor function is slow.
  • OpenCV(GPU) cvtColor function is not supported.
  • GPU programming is not yet coded.
  • OpenGL problem:
    YUV for input format is not supported.
    We have to convert color format from YUV to RGB.

** Renderer problem:
The renderer supports the resize function and YUV for input color format.
It is very useful.
Using nVIDIA Multimedia API, when the resolution is changed, the user (application) has to re-configure the renderer.
But the window on the display will be closed once, and then will be re-opened.
It can be recognized by viewers (human being).
So we don’t use the resize function in the renderer of Multimedia API, have to use the other resize function programed between the H.264 decoder and the renderer without re-configuration of the renderer.

Does anyone know which method is the fastest ?
or, does anyone know where CUDA sample code (resize/enlarge for YUV, cvtColor for Bayer) is ?

Hi mynaemi,
For resizing case (1)downscale and (3)yuv<->RGB, there is HW engine on TX1 and the sample code is at
tegra_multimedia_api\samples\07_video_convert

But for (2)upscale, there is no HW engine and you have to do your own implementation.

For renderer problem, which sample code do you refer to? Is there a way to reproduce the issue?