nvvideoconvert + nvjpegenc faulty cooperation

Autonomous Machines Jetson & Embedded Systems Jetson Nano
1

To start off let me note that on my JetsonNano I’m using the patch r32.2.1 version of libnvjpeg.so (from L4T Jetson/r32.2.1 patch - eLinux.org ; btw.: it solved my previous issue nvjpegdec misbehaving (memory leak?) - Jetson Nano - NVIDIA Developer Forums ) … to my knowledge is the latest version of the lib.

I ran into numerous issues when joining ‘nvvideoconvert ! nvjpegenc’ and perhaps I could make 3 separate bug reports (should I?)…
Here they come:

1) problem with … ! nvvideoconvert ! ‘video/x-raw,format=I420’ ! …

When I run

gst-launch-1.0 -v ximagesrc use-damage=0 ! ‘video/x-raw, width=1920, height=1080, framerate=10/1’ ! nvvideoconvert ! ‘video/x-raw,format=I420’ ! nvjpegenc ! fakesink
it produces
“Internal data stream error. | Additional debug info: gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstXImageSrc:ximagesrc0:
streaming stopped, reason error (-5)”
however the following works just fine:
gst-launch-1.0 -v ximagesrc use-damage=0 ! ‘video/x-raw, width=1920, height=1080, framerate=10/1’ ! videoconvert ! ‘video/x-raw,format=I420’ ! nvjpegenc ! fakesink

Also, when I plug-in either

… ! nvvideoconvert ! ‘video/x-raw,format=BGRx’ ! …
or
… ! videoconvert ! ‘video/x-raw,format=BGRx’ ! …
they both work (identically, as far as I can tell). So, seemingly it is only the ‘nvvideoconvert’ that for some reason has a problem with ‘format=I420’. The converter without ‘nv’ does not.

2) problem with … ‘video/x-raw(memory:NVMM),format=NV12’ ! nvjpegenc …

Running

gst-launch-1.0 -v ximagesrc use-damage=0 ! ‘video/x-raw, width=1920, height=1080, framerate=10/1’ ! nvvideoconvert ! ‘video/x-raw(memory:NVMM),format=NV12’ ! nvjpegenc ! fakesink
gives SIGSEGV (without much extra information) … probably the problem lies on the ‘nvjpegenc’ side as
gst-launch-1.0 -v ximagesrc use-damage=0 ! ‘video/x-raw, width=1920, height=1080, framerate=10/1’ ! nvvideoconvert ! ‘video/x-raw(memory:NVMM),format=NV12’ ! fakesink
behaves normally.

3) problem with nvjpegenc with ‘format=BGRx’ (and most others)

For the third case I run:

gst-launch-1.0 -v ximagesrc use-damage=0 ! ‘video/x-raw, width=1920, height=1080, framerate=10/1’ ! videoconvert ! ‘video/x-raw,format=I420’ ! nvjpegenc ! rtpjpegpay ! udpsink host=127.0.0.1 port=1234
(while on the receining end I have:
gst-launch-1.0 udpsrc buffer-size=8868308 port=1234 ! ‘application/x-rtp, payload=26’ ! rtpjpegdepay ! nvjpegdec ! autovideosink
the buffer size might be a slight overkill but at leeast I’m sure it’ll work. Btw, for running the above I also changed the ‘net mems’, i.e.:
sudo sysctl -w net.core.wmem_max=88608308
sudo sysctl -w net.core.rmem_max=88608308
)
and for that everything looks just fine (as it also does for ‘format=YV12’) so let that be the baseline to compare aginst.

If in the above I change only the format, e.g. as so:

gst-launch-1.0 -v ximagesrc use-damage=0 ! ‘video/x-raw, width=1920, height=1080, framerate=10/1’ ! videoconvert ! ‘video/x-raw,format=BGRx’ ! nvjpegenc ! rtpjpegpay ! udpsink host=127.0.0.1 port=1234
then the received video is somewhat messed-up color-wise. It looks like the offsets for coding different colors (“color-masks”?, so to speak) are miss-aligned.
Also, I get identical results when using … ! nvvideoconvert ! ‘video/x-raw,format=BGRx’ ! nvjpegenc ! … in the above, so it looks like a problem with nvjpegenc when getting ‘format=BGRx’ (no matter if it comes from videoconvert or nvvideoconvert ).

I followed through and tried the above with other formats that both videoconvert and nvjpegenc can handle, and for most of them (i.e. YUY2, UYVY, YVYU, Y42B, BGR, xBGR, BGRx, RGB, xRGB, RGBx) I got very similar color misalignment (seen on the receiving end, naturaly). Acceptable results were achieved only with: YV12 and I420.

My current, satisfactory workaround is to use … ! videoconvert ! ‘video/x-raw,format=I420’ ! nvjpegenc ! … so the issue is not urgent.
Still, it would be nice to be able to use nvvideoconvert ! nvjpegenc ind (as I understand) take the full advantage of HW encoding.

Thanks in advance.

2

Not sure, and be aware that it’s pure speculation as I have (almost) no experience with L4T R32.

However, considering the behaviour of nvvidconv in previous releases, it might be that nvvideoconvert requires at least one among its input or output to be in NVMM memory…in other words, x-raw in CPU memory to x-raw in CPU memory would not work, although CPU x-raw are valid caps for both input(sink) and output(src)…but not at the same time. So you may try to add a second nvvideoconvert after converting from CPU memory to NVMM memory, this second nvvideoconvert copying from NVMM memory to CPU memory.

For nvjpegenc, in older releases I was mainly able to use I420 and NV12 formats as input.

3

Hi,
For continuous encoding, we would suggest use h264/h265 encoder.
Here is a relevant post.

4

Thank you for interest.

@DaneLLL

…yes, I do that too, and it works good enough.

But:

  • for some specific cases (not directly shown in my examples above) jpeg seems to be a better choice after all, so I’ll probably stick with that
  • the fact that h26_ is working correctly doesn’t necessarily meant that jpeg should be buggy.

@Honey_Patouceul

I’ll take a look at that as soon as I get a chance. thanks for suggestion.