Hi everyone,
While encoding grayscale images into videos using GStreamer on Jetson Xavier AGX, I experienced a shift in brightness values. After some investigation, I found out that a spreading of the contrast occurs: dark pixels became darker, light pixels became lighter.
Does anyone of you also experience this issue? Is there a way, e.g. a setting, to prevent this?
About my investigation:
I created multiple sets with test images having one dark object moving on a light background. For each set, the brightness of the background is different. The brightness of the object remains the same. A single image looks like this:
I encoded the images using a lossless H.265 encoding (see NVIDIA developer forum):
gst-launch-1.0 multifilesrc location=ramp228/image_ramp_%d.png start-index=1 stop-index=1000 caps='image/png, framerate=25/1, format=GRAY8' ! pngdec ! queue ! videoconvert ! nvvidconv ! nvv4l2h265enc control-rate=0 qp-range='0,1:0,1:0,1' quant-i-frames=0 quant-p-frames=0 ! h265parse ! qtmux ! filesink location=ramp228.mp4 -e
A contrast spreading can be seen in all resulting videos. The brightness of the dark object has changed regardless of the background by 15 units. The brightness of the backgrounds also changed, with brighter backgrounds showing a stronger change. The following table contains the original brightness values and the values after encoding:
| Original | Reconstructed | Difference |
|---|---|---|
| 25 | 10 | - 15 |
| 128 | 130 | + 2 |
| 178 | 188 | + 10 |
| 228 | 246 | + 18 |
| 248 | 255 | + 7 |
The following figure describes the contrast spread using a histogram for the image shown above before and after encoding. It is noticeable that the two maxima (dark object and light background) shift outward.
I hope you can help me with this issue and look forward to your responses.
Best Regards!
Hi,
Please share how to generate image_ramp_%d.png. Can we generate it by running command like:
videotestrc ! video/x-raw,format=RGB ! pngenc ! multifilesink
I generated the images using Halcon (see https://www.mvtec.com/de/produkte/halcon).
Similar images can be generated, running:
gst-launch-1.0 videotestsrc pattern=ball background-color=-1776412 foreground-color=-15132391 num-buffers=1000 ! video/x-raw, width=512, height=512, format=RGB ! pngenc ! multifilesink location=testing2/image_ramp_%d.png
However, when encoding these images, I experience different behavior. The brightness values still shift, but all values decrease.
Hi,
Please check if the issue is present with software encoder x265enc. Run the command like:
gst-launch-1.0 multifilesrc location=ramp228/image_ramp_%d.png start-index=1 stop-index=1000 caps='image/png, framerate=25/1, format=GRAY8' ! pngdec ! queue ! videoconvert ! x265enc ! h265parse ! qtmux ! filesink location=ramp228.mp4 -e
And please share how to check brightness value. Would like to know which tool is used to generate the chart.
Yes, x265enc results in the same contrast spread. I had to add video/x-raw, format=I420 between videoconvert and x265enc to make the pipeline work.
Differences are already visually noticeable when images and videos are placed next to each other. To determine the exact brightness values, I use Python and the libaries cv2 to load images and videos (cv2 relies on a FFmpeg software decoder), numpy to analyse the image arrays and matplotlib to plot the results.
Hi,
Do you mean the brightness remain the same with x265enc?
I mean, I still notice a shift in brightness values with x265enc. So I think it is not the codec itself but some kind of conversion within the pipeline that causes this issue.
Hi everyone, despite the new realization that other encoders also results in contrast spreading, I have not made any progress regarding the issue. The contrast spread remains. Do you have any further advice on what could be a reason for this and how I can fix it?
Hi,
Since the issue is also present in using software encoder, it seems not an issue in encoder. Please dump YUV data and check if the color is correct through YUV viewer:
gst-launch-1.0 multifilesrc location=ramp228/image_ramp_%d.png start-index=1 stop-index=1000 caps='image/png, framerate=25/1, format=GRAY8' ! pngdec ! queue ! videoconvert ! video/x-raw,format=I420 ! multifilesink location=dump_%05d.YUV
Hi, I dumped the YUV data and checked it using an online yuv viewer (see RAW Pixels). In addition, I had a look at the resulting bitstream. It is still correct.
More information on the YUV data: It consists of 512 x 512 bytes representing the Y values and two times 128 x 512 bytes representing the U and V values respectively. The Y values are equal to the gray values/intensity of the original image and the U and V values are all 128.
Hi,
Please try to add colorimetry=bt601 and see if it helps:
gst-launch-1.0 multifilesrc location=ramp228/image_ramp_%d.png start-index=1 stop-index=1000 caps='image/png, framerate=25/1, format=GRAY8' ! pngdec ! queue ! videoconvert ! video/x-raw,colorimetry=bt601 ! x265enc ! h265parse ! qtmux ! filesink location=ramp228.mp4 -e
May also try colorimetry=bt709.
Hi, thank you for your reply. I am afraid, colorimetry does not solve the issue. I added format=I420 as well because otherwise the pipeline did not want to preroll. Then, I tried using colorimetry with all constants given in video color. I noticed differences using different constants but every one resulted in a contrast spread (some stronger, others less so).
Hi,
Please also try
colorimetry="1:4:0:0" // bt601 extended range
colorimetry="1:3:0:0" // bt709 extended range
Default bt601/bt709 is in limited range [16,235]. May try extended range [0,255].
Since h264/h265 supports YUV420, BGR to YUV420 conversion is a must and you would need to find one format fitting your requirement. If you cannot find one, probably it is better to keep png files so that these files can be decoded to BGR correctly.
Hi,
Using BT709 I see a difference between the “normal” and extended range. BT709 in normal range is closer to the original images and could be a first but not perfect solution.
Using BT601 I see no difference between the normal and extended range. I did some research on how they should differ and also looked at the YUV files: Regardless of the specified range, the color transformation is performed using extended range. Is there another way to force normal range for BT601?
Command to create YUV files:
gst-launch-1.0 multifilesrc location=ramp228/image_ramp_%d.png start-index=1 stop-index=1000 caps="image/png, framerate=25/1, format=GRAY8" ! pngdec ! queue ! videoconvert ! video/x-raw,format=I420,colorimetry="1:4:0:0" ! multifilesink location=YUV/dump_%05d.YUV
Command to create compressed video:
gst-launch-1.0 multifilesrc location=ramp228_3/image_ramp_%d.png start-index=1 stop-index=1000 caps="image/png, framerate=25/1, format=GRAY8" ! pngdec ! queue ! videoconvert ! video/x-raw,format=I420,colorimetry="1:4:0:0" ! x265enc ! h265parse ! qtmux ! filesink location=ramp228.mp4
My result:
| colormetry | np.min(img) | np.max(img) | img[1, 1] |
|---|---|---|---|
| original | 25 | 248 | [228 228 228] |
| 1;3;0;0 | 6 | 255 | [244 247 244] |
| 2;3;0;0 | 17 | 247 | [224 226 225] |
| 1;4;0;0 | 8 | 255 | [245 245 245] |
| 2;4;0;0 | 8 | 255 | [245 245 245] |
legend: colormetry=range:matrix:transfers:primaries
range: 1 → RANGE_0_255, 2 → RANGE_16_235
matrix: 3 → MATRIX_BT709, 4 → MATRIX_BT601
transfer: 0 → TRANSFER_UNKNOWN
primaries: 0 → PRIMARIES_UNKNOWN
Hi,
By default it should be comorimetry=bt601. You can download source code from
https://gstreamer.freedesktop.org/src/gst-plugins-base/gst-plugins-base-1.14.5.tar.xz
and it is defined in
./gst-libs/gst/video/video-color.c: MAKE_COLORIMETRY (BT601, _16_235, BT601, BT709, SMPTE170M),
By comparing to latest video-color.c:
gst-plugins-base/video-color.c at master · GStreamer/gst-plugins-base · GitHub
MAKE_COLORIMETRY (BT601, _16_235, BT601, BT601, SMPTE170M),
It adds a new transfer function from 1.18:
gst-plugins-base/video-color.h at master · GStreamer/gst-plugins-base · GitHub
/**
* GST_VIDEO_TRANSFER_BT601:
*
* also known as SMPTE170M / ITU-R BT1358 525 or 625 / ITU-R BT1700 NTSC
*
* Since: 1.18
*/
GST_VIDEO_TRANSFER_BT601
If you have a x86 PC with Linux OS, you may try to install gstreamer 1.18 and check if the color is correct. The test commands do not use NVIDIA plugins and should be run successfully.
On Xavier, it is with gstreamer 1.14.5 and may not work with later version(s). Each release is tested/verified with gstreamer 1.14.5. If you manually upgrade to 1.18, it may not work properly and the system can be unstable. If there is concern about the color correctness, probably it is better to keep the BGR frames.
Thank you very much for your reply.
Yes, GStreamer 1.18 works on an x86 machine running Linux Ubuntu 20.04 (color is correct). I will consider upgrading to 1.18 on Jetson Xavier. One question about keeping the BGR frames: Can you elaborate on that? I’m not sure I got it right. As soon as I compress them, I have to convert them to YUV, don’t I?
Hi,
Yes, for compressing into h264/h265, you would need to convert to YUV420. But if you care about the color shift, probably it is better to keep the PNG files instead of doing compression.