Wait until the stream starts to play (can take a few seconds)
Hi NVIDIA Team,
it seems that there is a color bug in your MJPEG decoding of nvv4l2decoder. As a proof I provided a pipeline where the MJPEG stream is decoded and right after encoded as a h264 stream. In the provided image you can see, that the colors are wrong. I guess that there is some issue with the U and V channel of the I420 format. Maybe the MJPEG channels of U and V are converted wrongly to the I420 output format. I guess that the MJPEG input is I422 because when I’m trying to decode it with nvjpegdec I got an Error that the cudaVideoChromaFormat_422 cannot be decoded (it is not supported by the nvjpegdec plugin). Maybe the I422 compressed MJPEG format is a packed fromat where the planes are interpreted wrongly in the nvv4l2decoder?
It can be “fixed” by converting it to NV12 with nvvidconv, but this is not really solving the bug. Somehow in the nvvidconv element the color format is fixed. But this adds 5ms of processing time, which is not really an option…
That the cpu memory from v4l2src is copied to the gpu memory is done by the nvv4l2decoder internally when it receives an input with image/jpeg and results in an output of NVMM.
I tested the pipeline by adding image/jpeg,width=640, height=480, framerate=30/1 as capsfilter (as stated in the FAQ):
We don’t need a nvvidconv after the decoding because the nvv4l2decoder already produces NVMM which is on the GPU. So there is no need for any nvvidconv afterwards. Only if we want to do some color conversions. So the output of the nvv4l2decoder is NVMM as memory with I420 as format. From here, the colors are wrong. The fact that you “correct” the colors with a nvvidconv is not really solving the issue that the decoder is doing something wrong here. Think of other examples where people are using custom plugins and taking this as input and have wrong colors? Furthermore, using nvvidconv is a really bad idea if you want to have low latency applications because it can take between 5 and 10ms, this is soooo slow…
For me it seems that the U and V planes are copied wrongly in the nvv4l2decoder. Because other decodings are working fine.
I tried another simple example and my suspicion has been confirmed. All I422 formats are converted wrongly by the nvv4l2decoder. So this is clearly a bug. The nvv4l2decoder should check the formats and shouldn’t accept formats which are converted wrongly.
After some research I figured out that most USB cameras with MJPEG have I422 as format. Will these formats be added to the next release? or at least to the roadmap?
This does not correspond to the gstreamer logic? And I don’t think that the problem is introduced by the nvv4l2h264enc because:
First of all it should not negotiate or at least the nvv4l2decoder should output that the format is some kind of I422 and not I420, but it says that it is I420, this is totally against the gstreamer logic
From the gstreamer perspective nvv4l2decoder produces I420, even if the input is YUY2. So its totally fine to use the nvv4l2h264enc right afterwards. You can check this with the -v option of your pipeline.
From my point above there is definitly a conversion to I420 which is done by the nvv4l2decoder
That the nvv4l2h264enc does not support YUY2 (or any other I422) is right, but that is not the case. It gets I420 and encodes it as I420. It only encodes something wrong if the previous element does something wrong or sends it in the wrong format.
Using nv3dsink is just for visualization, somehow it shows the colors correctly. I guess it is some underlying nvidia stuff where the colors are fixed, but there are many applications where a stream is provided like I posted (e.g. using the rtsp-server from gstreamer)
using nvvidconv is a workaround here and for low latency applications not a good idea as I sad it before
As you stated: “Usually you need to add nvvideoconvert to upstream of encoder to avoid the above problems” → This is absolute against the gstreamer logic. If it can negotiate everything is fine. Fixing it by an intermediate plugin is so wrong.
Addtionally, imagine a plugin right after the nvv4l2decoder which takes I420 NVMM and gets wrong colors? Using nvvidconv before is a workaround and not a fix. It puts at least 5ms processing time on top of the application. In many applications each ms matters.
It seems that there is some underlying problem which works only in the nvidia eco system correct, but as soon as you produce a h264stream or uploading data to CUDAMemory or GLMemory with the jetson-multimedia api the problem occurs. With the multimedia api it seems that the U and V channel are not present where they should be. I420 is planarwise and the I422 is packed. So GStreamer expects that the colors are at a specific memory location, but they are not there. This is why it is partially a gray image and the colors are shifted.
I would definitly say that this is a bug and would be happy if it reaches some developer of the the deepstream plugins.
Some additional details about the decoder and encoder.
The nvv4l2decoder supports only decoding into I420 format for MJPEG input whereas for H.264/H.265 and all other bitstream formats it supports output in NV12 format.
so basically this MJPEG decoded I420 output cannot be directly fed to the nvv4l2h264enc as this v4l2 encoder supports only NV12 format so, definitely we need the nvvidconv which will convert from I420 of MJPEG decoder → NV12 for feeding to v4l2 encoder.
