Full range color space issue during OpenGL processing

Hello,

I have a YUVJ420P(full-range) based H.264 stream, and after processed through nvdecoder and OpenGL, the rendered image is displayed as non-full-range RGB(the data is full-range but it’s displayed as non-full-range outcome). And thus I lost the top/bottom values.

I have tried to set the format of the decoder output buffer as full-range, but it seems not working with the final RGB image.

Is the pipeline simply does not support full-range processing or just the interface has not been implemented?

Besides, the function NvEGLImageFromFd() should contain the construct progress of eglImage from DMAfd, where the eglImage should be able to be configured as full-range. Is it possible that we could access the implementation of this function?

Hi EvanKwok,
Please share steps to reproduce the issue. Such as a patch to be applied to tegra_multimedia_api samples and detail steps to run and observe the issue you are facing.

Hi DaneLLL

You can reproduce the issue simply using the Tegra sample: tegra_multimedia_api/samples/00_video_decode.

Appending this to read out the image for comparison to original image:

glPixelStorei(GL_PACK_ALIGNMENT, 4);
glReadPixels(0, 0, 1920, 1088, GL_RGBA, GL_UNSIGNED_BYTE, buff);

I have also uploaded a full-range video here for you:
https://drive.google.com/open?id=107hrDejbblcuQDXf4ntsxUifp_u9ZLxF

EvanKwok,

Are you indicating the rendering result “on monitor” is not in 0-255 full range RGB?

Hi DaneLLL

Thanks for your quick reply.

We make a comparison on the histogram, please have a check.


Hi EvanKwok,
Please share a clean patch on 00_video_decode. The decoded frames are I420(or NV12) but seems you are dumping RGBA. A bit confusing here.

Hi DaneLLL

We’re using Tegra_Multimedia_API_R24.2.2_aarch64.tbz2, and below is the patch we applied to sample 00

--- tegra_multimedia_api/samples/common/classes/NvEglRenderer.cpp	2017-10-03 15:22:31.000000000 +0800
+++ tegra_multimedia_api_modify/samples/common/classes/NvEglRenderer.cpp	2018-08-28 16:47:03.825544677 +0800
@@ -438,6 +438,18 @@ NvEglRenderer::renderInternal()
         COMP_ERROR_MSG("Got Error in eglSwapBuffers " << eglGetError());
         return -1;
     }
+
+    unsigned char* buff = new unsigned char[1920*1080*4]();
+    glPixelStorei(GL_PACK_ALIGNMENT, 4);
+    glReadPixels(0, 0, 1920, 1080, GL_RGBA, GL_UNSIGNED_BYTE, buff);
+    static FILE* fp = NULL;
+    if (fp == NULL)
+    {
+        fp = fopen("/home/nvidia/1.rgba", "wb"); //you may need to change this file path
+    }
+    fwrite(buff, 1, 1920*1080*4, fp);
+    delete[] buff;
+
     if (eglClientWaitSyncKHR (egl_display, egl_sync,
                 EGL_SYNC_FLUSH_COMMANDS_BIT_KHR, EGL_FOREVER_KHR) == EGL_FALSE)
     {

Here is the complete step we took to reproduce this issue:

  1. extract H264 stream from the mp4 file: ``` ffmpeg -i fullrange.mp4 -c:v copy -bsf:v h264_mp4toannexb fullrange.264 ```
  2. execute the decoding and rendering with the sample code: ``` ./video_decode fullrange.264 H264 -ww 1920 -wh 1080 ```
  3. compare the rgb data in #1 and #2

The progress that we dump the image from OpenGL will automatically do the YUV2RGB conversion, and this conversion is where the issue happens, losing the top/bottom value.

Hi EvanKwok,
r24.2.2 is for TX1. You should run ‘head -1 /etc/nv_tegra_release’ and download correct revision of the samples.

Really?.. This is the default package that we saw on the TX2 develop board… But I do see the newly released API, I’ll give a try on r28.2 and feedback here.

Thanks

Hi WayneWWW,

No I’m comparing the data. As you can see the reply #5, I attached 2 screenshots of the histogram respectively from the 264 and the render output. It’s obviously that in the latter there’s much more pixels lay on value 0/255, while there’s few in the former.

EvanKwok,

So you are wondering why the buffer from “glReadPixels” is not in full RGB range?

Yes, and I wonder if there’s a way to make it in full range.

Hi EvanKwok,

I am not sure if this could help. This thread is talking about setting hdmi monitor from limited mode to full range mode.

https://devtalk.nvidia.com/default/topic/1028918/jetson-tx2/rgb-full-range-0-255-on-hdmi/post/5233830/#5233830

If the buffer has been set to limited mode during rendering due to xorg setting, this may help.

Hi WayneWWW

Tried the settings in this thread and seems not working.

Hi DaneLLL,

Tried both r28.2.0 and r28.2.1, and still got no luck.

Hi EvanKwok,
#1 is a h264 file and #2 is 1.rgba. How do you compare the two files?

Hi DaneLLL

I packed the RGBA into .bmp file then compare in PhotoShop.

BTW, we found that you can actually see the difference between the decoded NV12 data from TX2 and ffmpeg as well.

Hi EvanKwok,
We need clear steps to reproduce the issue, so that we can do further check.

If the issue is in RGBA, please share how to generate and compare the dumped RGBAs. If the issue is in NV12, please share how to generate and compare the dumped NV12s. It makes no sense you compare NV12 with RGBA.

Please give more detail.

Hi DaneLLL

Ok, follow these steps
(I’m switching into NV12 compare here since it leaves PhotoShop away):

  1. extract H264 stream from the mp4 file:
    ffmpeg -i fullrange.mp4 -c:v copy -bsf:v h264_mp4toannexb fullrange.264
    
  2. decode .mp4 file into NV12 with the sample code:
    ./video_decode fullrange.264 H264 --disable-rendering -o tx2_out.nv12
    
  3. decode .mp4 file into NV12 with ffmpeg:
    ffmpeg -i fullrange.264 -f rawvideo -pix_fmt nv12 ffmpeg_out.nv12
    
  4. play 2 NV12 streams with following command, switch between the images and the difference is obvious that even human eye could tell
    ffplay -f rawvideo -pix_fmt nv12 -s 3840x2160 out.nv12
    

Hi EvanKwok,
We checked and TX2-decoded NV12s are in full range 0-255 but ffmpeg-decoded NV12s is in 16-235. Please reer to attached binary comparison and picture comparison. TX2-decoded NV12 is brighter.