Decoding mjpeg stream using the NvJPEGDecoder

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Hello,

When I try to decode a mjpeg stream captured from a V4L2 device on a Jetson TX2, it looks like the used function “decodeToFd()” is leaking memory. I use the following scenario:

  1. Create an NvJPEGDecoder using the function “NvJPEGDecoder::createJPEGDecoder(“jpegdec”);”
  2. In a while loop capture an image from the V4L2 device (mjpeg encoded stream).
  3. Feed the image to the decoder using to function “decodeToFd(fd, buf_start, buf_size, pixfmt, width, height)”
  4. The function returns a handle (fd) to the successfully decoded jpeg image. The resulting YUV422 frame is correct.
  5. Go to 2 (capture an image) and feed it again to the decoder.
  6. At the end of the loop, the decoder is destroyed.

During the execution of the program, the used memory is increased by small amounts (not the size of the uncompressed image). This continues until the memory is exhausted and the program is terminated.

The returned fd always has the same value indicating that the internal buffer is reused? From the documentation it is not clear if this allocated memory should be freed. It is created by the internals of the function “decodeToFd()”, so there’s no control over it from the program.

If, however, the decoder is destroyed each time the jpeg image is decoded, there’s no memory leaking. The problem with this approach is, that it takes 40ms (instead of 10ms with the approach above) to complete the decoding step (for a 4k input image) on a Jetson TX2.

Any suggestions on how to fix this problem?

2

Please try the prebuilt at below post:
[url]https://devtalk.nvidia.com/default/topic/1035699/jetson-tx1/premature-end-of-jpeg-file-tegra_multimedia-master-samples-06_jpeg_decode-/post/5263099/#5263099[/url]

3

Hi Dane,

Thank you for the answer. I’ve tried the library from the link above. Now I get a “memory corruption” error at the first time the function is called. See the attached debug log:

0x0000007fb54eb528 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x0000007fb54eb528 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x0000007fb54ec9e0 in __GI_abort () at abort.c:89
#2  0x0000007fb55228c4 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7fb55d35a0 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
#3  0x0000007fb55282ec in malloc_printerr (action=<optimized out>, str=0x7fb55d3b50 "malloc(): memory corruption", ptr=<optimized out>, ar_ptr=<optimized out>)
    at malloc.c:5006
#4  0x0000007fb552a49c in _int_malloc (av=av@entry=0x7f50000020, bytes=bytes@entry=16312) at malloc.c:3474
#5  0x0000007fb552c404 in __GI___libc_malloc (bytes=16312) at malloc.c:2913
#6  0x0000007fb596ff44 in alloc_small () from /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
#7  0x0000007fb595a62c in get_sof () from /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
#8  0x0000007fb595b3d4 in read_markers () from /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
#9  0x0000007fb59598f0 in consume_markers () from /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
#10 0x0000007fb594b714 in jpeg_consume_input () from /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
#11 0x0000007fb594ba18 in jpeg_read_header () from /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
#12 0x000000000056dfb8 in NvJPEGDecoder::decodeToFd (this=0x7fa8001990, fd=@0x7f5563e58c: -1, in_buf=0x7f5c880000 "77307741

0x0000007fb54eb528 in __GI_raise (sig=sig@entry=6) at …/sysdeps/unix/sysv/linux/raise.c:54
54 …/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x0000007fb54eb528 in __GI_raise (sig=sig@entry=6) at …/sysdeps/unix/sysv/linux/raise.c:54
#1 0x0000007fb54ec9e0 in __GI_abort () at abort.c:89
#2 0x0000007fb55228c4 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7fb55d35a0 “*** Error in `%s’: %s: 0x%s ***\n”) at …/sysdeps/posix/libc_fatal.c:175
#3 0x0000007fb55282ec in malloc_printerr (action=, str=0x7fb55d3b50 “malloc(): memory corruption”, ptr=, ar_ptr=)
at malloc.c:5006
#4 0x0000007fb552a49c in _int_malloc (av=av@entry=0x7f50000020, bytes=bytes@entry=16312) at malloc.c:3474
#5 0x0000007fb552c404 in __GI___libc_malloc (bytes=16312) at malloc.c:2913
#6 0x0000007fb596ff44 in alloc_small () from /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
#7 0x0000007fb595a62c in get_sof () from /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
#8 0x0000007fb595b3d4 in read_markers () from /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
#9 0x0000007fb59598f0 in consume_markers () from /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
#10 0x0000007fb594b714 in jpeg_consume_input () from /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
#11 0x0000007fb594ba18 in jpeg_read_header () from /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
#12 0x000000000056dfb8 in NvJPEGDecoder::decodeToFd (this=0x7fa8001990, fd=@0x7f5563e58c: -1, in_buf=0x7f5c880000 “\377\330\377\341\004kExif”, in_buf_size=704306,
pixfmt=@0x7f5563e588: 1295135284, width=@0x7f5563e580: 4096, height=@0x7f5563e584: 2160) at /home/nvidia/sporteye/pacscapture/src/server/tegra/NvJpegDecoder.cpp:93

04kExif", in_buf_size=704306, 
    pixfmt=@0x7f5563e588: 1295135284, width=@0x7f5563e580: 4096, height=@0x7f5563e584: 2160) at /home/nvidia/sporteye/pacscapture/src/server/tegra/NvJpegDecoder.cpp:93
4

Hi infontz6r,
Are you on r28.2.1?

5

I think I’m on r28.2.0

cat /etc/nv_tegra_release | head -1
# R28 (release), REVISION: 2.0, GCID: 10567845, BOARD: t186ref, EABI: aarch64, DATE: Fri Mar  2 04:57:01 UTC 2018

I’ll update to the latest version first.

6

I’ve updated everything to r28.2.1 and recompiled all, but the crash still occurs.

Error in `...`: malloc(): memory corruption: 0x0000007f5002b540

Thread 17 "Output Plane" received signal SIGABRT, Aborted.
[Switching to Thread 0x7f5563f050 (LWP 28877)]
0x0000007fb54f2528 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x0000007fb54f2528 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x0000007fb54f39e0 in __GI_abort () at abort.c:89
#2  0x0000007fb55298c4 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7fb55da5a0 "*** Error in `%s': %s: 0x%s ***\n")
    at ../sysdeps/posix/libc_fatal.c:175
#3  0x0000007fb552f2ec in malloc_printerr (action=<optimized out>, str=0x7fb55dab50 "malloc(): memory corruption", ptr=<optimized out>, 
    ar_ptr=<optimized out>) at malloc.c:5006
#4  0x0000007fb553149c in _int_malloc (av=av@entry=0x7f50000020, bytes=bytes@entry=16312) at malloc.c:3474
#5  0x0000007fb5533404 in __GI___libc_malloc (bytes=16312) at malloc.c:2913
#6  0x0000007fb5976f44 in alloc_small () from /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
#7  0x0000007fb596162c in get_sof () from /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
#8  0x0000007fb59623d4 in read_markers () from /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
#9  0x0000007fb59608f0 in consume_markers () from /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
#10 0x0000007fb5952714 in jpeg_consume_input () from /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
#11 0x0000007fb5952a18 in jpeg_read_header () from /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
#12 0x000000000056dfb8 in NvJPEGDecoder::decodeToFd (this=0x7fa8001990, fd=@0x7f5563e58c: -1, in_buf=0x7f5c880000 "77307741

Error in ...: malloc(): memory corruption: 0x0000007f5002b540

Thread 17 “Output Plane” received signal SIGABRT, Aborted.
[Switching to Thread 0x7f5563f050 (LWP 28877)]
0x0000007fb54f2528 in __GI_raise (sig=sig@entry=6) at …/sysdeps/unix/sysv/linux/raise.c:54
54 …/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x0000007fb54f2528 in __GI_raise (sig=sig@entry=6) at …/sysdeps/unix/sysv/linux/raise.c:54
#1 0x0000007fb54f39e0 in __GI_abort () at abort.c:89
#2 0x0000007fb55298c4 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7fb55da5a0 “*** Error in `%s’: %s: 0x%s ***\n”)
at …/sysdeps/posix/libc_fatal.c:175
#3 0x0000007fb552f2ec in malloc_printerr (action=, str=0x7fb55dab50 “malloc(): memory corruption”, ptr=,
ar_ptr=) at malloc.c:5006
#4 0x0000007fb553149c in _int_malloc (av=av@entry=0x7f50000020, bytes=bytes@entry=16312) at malloc.c:3474
#5 0x0000007fb5533404 in __GI___libc_malloc (bytes=16312) at malloc.c:2913
#6 0x0000007fb5976f44 in alloc_small () from /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
#7 0x0000007fb596162c in get_sof () from /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
#8 0x0000007fb59623d4 in read_markers () from /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
#9 0x0000007fb59608f0 in consume_markers () from /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
#10 0x0000007fb5952714 in jpeg_consume_input () from /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
#11 0x0000007fb5952a18 in jpeg_read_header () from /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so
#12 0x000000000056dfb8 in NvJPEGDecoder::decodeToFd (this=0x7fa8001990, fd=@0x7f5563e58c: -1, in_buf=0x7f5c880000 “\377\330\377\341\004kExif”,
in_buf_size=175573, pixfmt=@0x7f5563e588: 1295135284, width=@0x7f5563e580: 4096, height=@0x7f5563e584: 2160)

04kExif", 
    in_buf_size=175573, pixfmt=@0x7f5563e588: 1295135284, width=@0x7f5563e580: 4096, height=@0x7f5563e584: 2160)

The output of the version information is:

cat /etc/nv_tegra_release | head -1
# R28 (release), REVISION: 2.1, GCID: 11272647, BOARD: t186ref, EABI: aarch64, DATE: Thu May 17 07:29:06 UTC 2018

If I use the original library (r28.2.1) from Jetpack, the program doesn’t crash, but there’s still a memory leak for each decoded frame.

Attached is a compressed archive of the first 10 frames. The frames were captured from a Logitech Brio 4k camera.
frames.tar.gz (5.37 MB)

7

Hi infontz6r,
Attach a patch which demonstrates MJPEG decoding. FYR.

nvidia@tegra-ubuntu:~/tegra_multimedia_api$ patch -p3 < ~/0001-mm_api-v4l2_cam-support-MJPEG.patch
patching file samples/12_camera_v4l2_cuda/camera_v4l2_cuda.cpp
patching file samples/12_camera_v4l2_cuda/camera_v4l2_cuda.h
nvidia@tegra-ubuntu:~/tegra_multimedia_api$ cd samples/12_camera_v4l2_cuda/
nvidia@tegra-ubuntu:~/tegra_multimedia_api/samples/12_camera_v4l2_cuda$ make
Compiling: camera_v4l2_cuda.cpp
Linking: camera_v4l2_cuda
nvidia@tegra-ubuntu:~/tegra_multimedia_api/samples/12_camera_v4l2_cuda$ export DISPLAY=:0
nvidia@tegra-ubuntu:~/tegra_multimedia_api/samples/12_camera_v4l2_cuda$ ./camera_v4l2_cuda -d /dev/video1 -s 640x480 -f MJPEG

0001-mm_api-v4l2_cam-support-MJPEG.patch.txt (11.2 KB)

8

Hi Dane,

Thank you for the example code. It helps a lot to see how functions are intended to be used. The patched example works fine (the camera runs at 4k resolution, 30fps) with the new libnvjpeg.so, and has no memory leak problem.

The memory leak still occurs in the released 28.2.1 version. Using the updated library libnvjpeg.so, however, also solves the memory leak I was experiencing in my code :-)

I also found the reason for the crash mentioned above. It occurred because I added (during debugging) a call in the NvJPEGDecoder::decodeToFd() function just before the jpeg_mem_src() function call: jpeg_set_hardware_acceleration_parameters_dec(&cinfo, TRUE, in_buf_size, 0, 0, 0, FALSE);
Removing the function again (reverting back to the original implementation) fixed the crash.

Thanks for your support!
Boris.