Full range color space issue during OpenGL processing

EvanKwok · August 24, 2018, 8:36am

Hello,

I have a YUVJ420P(full-range) based H.264 stream, and after processed through nvdecoder and OpenGL, the rendered image is displayed as non-full-range RGB(the data is full-range but it’s displayed as non-full-range outcome). And thus I lost the top/bottom values.

I have tried to set the format of the decoder output buffer as full-range, but it seems not working with the final RGB image.

Is the pipeline simply does not support full-range processing or just the interface has not been implemented?

Besides, the function NvEGLImageFromFd() should contain the construct progress of eglImage from DMAfd, where the eglImage should be able to be configured as full-range. Is it possible that we could access the implementation of this function?

DaneLLL · August 27, 2018, 3:04am

Hi EvanKwok,
Please share steps to reproduce the issue. Such as a patch to be applied to tegra_multimedia_api samples and detail steps to run and observe the issue you are facing.

EvanKwok · August 28, 2018, 7:52am

Hi DaneLLL

You can reproduce the issue simply using the Tegra sample: tegra_multimedia_api/samples/00_video_decode.

Appending this to read out the image for comparison to original image:

glPixelStorei(GL_PACK_ALIGNMENT, 4);
glReadPixels(0, 0, 1920, 1088, GL_RGBA, GL_UNSIGNED_BYTE, buff);

I have also uploaded a full-range video here for you:
https://drive.google.com/open?id=107hrDejbblcuQDXf4ntsxUifp_u9ZLxF

WayneWWW · August 28, 2018, 7:59am

EvanKwok,

Are you indicating the rendering result “on monitor” is not in 0-255 full range RGB?

EvanKwok · August 28, 2018, 8:19am

Hi DaneLLL

Thanks for your quick reply.

We make a comparison on the histogram, please have a check.

DaneLLL · August 28, 2018, 8:33am

Hi DaneLLL

You can reproduce the issue simply using the Tegra sample: tegra_multimedia_api/samples/00_video_decode.

Appending this to read out the image for comparison to original image:
glPixelStorei(GL_PACK_ALIGNMENT, 4);
glReadPixels(0, 0, 1920, 1088, GL_RGBA, GL_UNSIGNED_BYTE, buff);
I have also uploaded a full-range video here for you:
https://drive.google.com/open?id=107hrDejbblcuQDXf4ntsxUifp_u9ZLxF

Hi EvanKwok,
Please share a clean patch on 00_video_decode. The decoded frames are I420(or NV12) but seems you are dumping RGBA. A bit confusing here.

EvanKwok · August 28, 2018, 9:03am

Hi DaneLLL

We’re using Tegra_Multimedia_API_R24.2.2_aarch64.tbz2, and below is the patch we applied to sample 00

--- tegra_multimedia_api/samples/common/classes/NvEglRenderer.cpp	2017-10-03 15:22:31.000000000 +0800
+++ tegra_multimedia_api_modify/samples/common/classes/NvEglRenderer.cpp	2018-08-28 16:47:03.825544677 +0800
@@ -438,6 +438,18 @@ NvEglRenderer::renderInternal()
         COMP_ERROR_MSG("Got Error in eglSwapBuffers " << eglGetError());
         return -1;
     }
+
+    unsigned char* buff = new unsigned char[1920*1080*4]();
+    glPixelStorei(GL_PACK_ALIGNMENT, 4);
+    glReadPixels(0, 0, 1920, 1080, GL_RGBA, GL_UNSIGNED_BYTE, buff);
+    static FILE* fp = NULL;
+    if (fp == NULL)
+    {
+        fp = fopen("/home/nvidia/1.rgba", "wb"); //you may need to change this file path
+    }
+    fwrite(buff, 1, 1920*1080*4, fp);
+    delete[] buff;
+
     if (eglClientWaitSyncKHR (egl_display, egl_sync,
                 EGL_SYNC_FLUSH_COMMANDS_BIT_KHR, EGL_FOREVER_KHR) == EGL_FALSE)
     {

Here is the complete step we took to reproduce this issue:

extract H264 stream from the mp4 file: ``` ffmpeg -i fullrange.mp4 -c:v copy -bsf:v h264_mp4toannexb fullrange.264 ```
execute the decoding and rendering with the sample code: ``` ./video_decode fullrange.264 H264 -ww 1920 -wh 1080 ```
compare the rgb data in #1 and #2

The progress that we dump the image from OpenGL will automatically do the YUV2RGB conversion, and this conversion is where the issue happens, losing the top/bottom value.

DaneLLL · August 28, 2018, 9:22am

Hi EvanKwok,
r24.2.2 is for TX1. You should run ‘head -1 /etc/nv_tegra_release’ and download correct revision of the samples.

EvanKwok · August 28, 2018, 9:36am

Really?.. This is the default package that we saw on the TX2 develop board… But I do see the newly released API, I’ll give a try on r28.2 and feedback here.

Thanks

EvanKwok · August 29, 2018, 2:25am

Hi WayneWWW,

No I’m comparing the data. As you can see the reply #5, I attached 2 screenshots of the histogram respectively from the 264 and the render output. It’s obviously that in the latter there’s much more pixels lay on value 0/255, while there’s few in the former.

WayneWWW · August 29, 2018, 2:29am

EvanKwok,

So you are wondering why the buffer from “glReadPixels” is not in full RGB range?

EvanKwok · August 29, 2018, 2:31am

Yes, and I wonder if there’s a way to make it in full range.

WayneWWW · August 29, 2018, 2:46am

Hi EvanKwok,

I am not sure if this could help. This thread is talking about setting hdmi monitor from limited mode to full range mode.

[url]RGB Full Range (0-255) on HDMI - Jetson TX2 - NVIDIA Developer Forums

If the buffer has been set to limited mode during rendering due to xorg setting, this may help.

EvanKwok · August 30, 2018, 8:46am

Hi WayneWWW

Tried the settings in this thread and seems not working.

EvanKwok · August 30, 2018, 8:47am

Hi DaneLLL,

Tried both r28.2.0 and r28.2.1, and still got no luck.

DaneLLL · September 3, 2018, 3:18am

Hi EvanKwok,
#1 is a h264 file and #2 is 1.rgba. How do you compare the two files?

EvanKwok · September 3, 2018, 7:33am

Hi DaneLLL

I packed the RGBA into .bmp file then compare in PhotoShop.

BTW, we found that you can actually see the difference between the decoded NV12 data from TX2 and ffmpeg as well.

DaneLLL · September 3, 2018, 7:41am

Hi EvanKwok,
We need clear steps to reproduce the issue, so that we can do further check.

If the issue is in RGBA, please share how to generate and compare the dumped RGBAs. If the issue is in NV12, please share how to generate and compare the dumped NV12s. It makes no sense you compare NV12 with RGBA.

Please give more detail.

EvanKwok · September 3, 2018, 7:56am

Hi DaneLLL

Ok, follow these steps
(I’m switching into NV12 compare here since it leaves PhotoShop away):

extract H264 stream from the mp4 file:

ffmpeg -i fullrange.mp4 -c:v copy -bsf:v h264_mp4toannexb fullrange.264

decode .mp4 file into NV12 with the sample code:

./video_decode fullrange.264 H264 --disable-rendering -o tx2_out.nv12

decode .mp4 file into NV12 with ffmpeg:

ffmpeg -i fullrange.264 -f rawvideo -pix_fmt nv12 ffmpeg_out.nv12

play 2 NV12 streams with following command, switch between the images and the difference is obvious that even human eye could tell
```
ffplay -f rawvideo -pix_fmt nv12 -s 3840x2160 out.nv12
```

DaneLLL · September 4, 2018, 8:05am

Hi EvanKwok,
We checked and TX2-decoded NV12s are in full range 0-255 but ffmpeg-decoded NV12s is in 16-235. Please reer to attached binary comparison and picture comparison. TX2-decoded NV12 is brighter.

Topic		Replies	Views
TX2 decide H264 with tegra_multimedia_api Jetson TX2 mmapi	34	2042	October 18, 2021
Possible multimedia api regression with decode interlace source Jetson TX2 mmapi , nvbugs	28	2670	February 9, 2022
Formatting images to feed into NvVideoEncoder (Tegra multimedia API) Jetson TX2	29	4693	October 18, 2021
TX2 tegra_multimedia_api encode/decode issue Jetson TX2 mmapi	19	2219	October 18, 2021
Error generated while running the code after connecting the camera Jetson Xavier NX gstreamer , nvbugs	45	1235	January 2, 2024
How to achieve the H.264 encoding performance: 4K (3,840x2,160)/30fps with OpenMAX IL API/L4T R24.1 Jetson TX1	40	11896	October 18, 2021
TX2 Camera convert/encode using Multimedia API issue Jetson TX2 camera , encoder	16	2005	October 18, 2021
omxh264dec/nvvidconv: 34.7MB per minute memory leak/crash when decoding Interlaced H264 video Jetson Nano	27	3066	October 14, 2021
TX2 H264 RTSP Stream decoding issues Jetson TX2	27	11561	October 18, 2021
Jetson goes curling (or, simultaneously viewing multiple IP-cams) Jetson TK1	33	12050	November 28, 2014

Full range color space issue during OpenGL processing

Related topics