nvjpegdec v1.2.3 memory leak

I believe I’ve found a memory leak in nvjpegdec. It can be reproduced by running the following snippet on a TX2 board, where test.jpeg is an arbitrary 1920x1080 jpeg image (Filesize ~1M).

gst-launch-1.0 multifilesrc location=test.jpg caps=image/jpeg,framerate=24/1 ! nvjpegdec ! fakesink

On my TX2 board, this process steadily increases in memory usage. Starting at 229m, within a couple of minutes it has grown to 997m. Approximately 25 minutes later, it has grown to 5285m. This is an extreme example, but even throttled to 30fps in my application the memory leak causes a significant increase in memory usage.

This is not reproducible with other jpeg decoders, such as the standard jpegdec, and only seems to occur with nvjpegdec. gst-inspect-1.0 reports that I am using version 1.2.3 of nvjpegdec. Is this an issue that has been addressed in a later version, or is there a known workaround?

Please try the prebuilt library at
https://devtalk.nvidia.com/default/topic/1035699/jetson-tx1/premature-end-of-jpeg-file-tegra_multimedia-master-samples-06_jpeg_decode-/post/5263099/#5263099

That library seems to work without the memory leak, thank you. Is that version of the library with those fixes available in one of the newer SDKs?

Hi,
It shall be fixed in next release.

During further testing I’ve had some more difficulties with that library, unfortunately. In my application, if I use nvjpegdec with the nvjpeg library linked there, the application logs a malloc(): memory corruption message. I’ve gotten the following output from Valgrind, suggesting an issue in nvjpegdec that’s writing outside of allocated memory and corrupting the heap:

==6627== Thread 36 e0_ptpsrc:src:
==6627== Invalid write of size 8
==6627==    at 0x4849294: memcpy (vg_replace_strmem.c:1035)
==6627==    by 0x24570F4B: copy_buff (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456A67B: jpeg_consume_input (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456AA17: jpeg_read_header (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2453EE13: ??? (in /usr/lib/gstreamer-1.0/libnvgstjpeg.so)
==6627==    by 0x1C620237: ??? (in /usr/lib/libgstvideo-1.0.so.0.801.0)
==6627==  Address 0x2c2acb98 is 0 bytes after a block of size 62,888 alloc'd
==6627==    at 0x4844114: malloc (vg_replace_malloc.c:299)
==6627==    by 0x2458EF43: alloc_small (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x24570FB3: copy_buff (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456A67B: jpeg_consume_input (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456AA17: jpeg_read_header (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2453EE13: ??? (in /usr/lib/gstreamer-1.0/libnvgstjpeg.so)
==6627==    by 0x1C620237: ??? (in /usr/lib/libgstvideo-1.0.so.0.801.0)
==6627== 
==6627== Invalid write of size 8
==6627==    at 0x484929C: memcpy (vg_replace_strmem.c:1035)
==6627==    by 0x24570F4B: copy_buff (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456A67B: jpeg_consume_input (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456AA17: jpeg_read_header (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2453EE13: ??? (in /usr/lib/gstreamer-1.0/libnvgstjpeg.so)
==6627==    by 0x1C620237: ??? (in /usr/lib/libgstvideo-1.0.so.0.801.0)
==6627==  Address 0x2c2acba0 is 8 bytes after a block of size 62,888 alloc'd
==6627==    at 0x4844114: malloc (vg_replace_malloc.c:299)
==6627==    by 0x2458EF43: alloc_small (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x24570FB3: copy_buff (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456A67B: jpeg_consume_input (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456AA17: jpeg_read_header (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2453EE13: ??? (in /usr/lib/gstreamer-1.0/libnvgstjpeg.so)
==6627==    by 0x1C620237: ??? (in /usr/lib/libgstvideo-1.0.so.0.801.0)
==6627== 
==6627== Invalid write of size 8
==6627==    at 0x484927C: memcpy (vg_replace_strmem.c:1035)
==6627==    by 0x24570F4B: copy_buff (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456A67B: jpeg_consume_input (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456AA17: jpeg_read_header (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2453EE13: ??? (in /usr/lib/gstreamer-1.0/libnvgstjpeg.so)
==6627==    by 0x1C620237: ??? (in /usr/lib/libgstvideo-1.0.so.0.801.0)
==6627==  Address 0x2c2acba8 is 16 bytes after a block of size 62,888 alloc'd
==6627==    at 0x4844114: malloc (vg_replace_malloc.c:299)
==6627==    by 0x2458EF43: alloc_small (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x24570FB3: copy_buff (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456A67B: jpeg_consume_input (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456AA17: jpeg_read_header (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2453EE13: ??? (in /usr/lib/gstreamer-1.0/libnvgstjpeg.so)
==6627==    by 0x1C620237: ??? (in /usr/lib/libgstvideo-1.0.so.0.801.0)
==6627== 
==6627== Invalid write of size 8
==6627==    at 0x484928C: memcpy (vg_replace_strmem.c:1035)
==6627==    by 0x24570F4B: copy_buff (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456A67B: jpeg_consume_input (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456AA17: jpeg_read_header (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2453EE13: ??? (in /usr/lib/gstreamer-1.0/libnvgstjpeg.so)
==6627==    by 0x1C620237: ??? (in /usr/lib/libgstvideo-1.0.so.0.801.0)
==6627==  Address 0x2c2acbb0 is 16 bytes after a block of size 62,896 in arena "client"
==6627== 
==6627== Invalid read of size 1
==6627==    at 0x4849378: memcpy (vg_replace_strmem.c:1035)
==6627==    by 0x24632473: ??? (in /usr/lib/tegra/libnvtvmr.so)
==6627==    by 0x2459423F: jpegTegraDecoderRender (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456B2D3: jpeg_read_raw_data (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2453FDEB: ??? (in /usr/lib/gstreamer-1.0/libnvgstjpeg.so)
==6627==    by 0x1C620237: ??? (in /usr/lib/libgstvideo-1.0.so.0.801.0)
==6627==  Address 0x2c2acb98 is 0 bytes after a block of size 62,888 alloc'd
==6627==    at 0x4844114: malloc (vg_replace_malloc.c:299)
==6627==    by 0x2458EF43: alloc_small (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x24570FB3: copy_buff (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456A67B: jpeg_consume_input (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456AA17: jpeg_read_header (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2453EE13: ??? (in /usr/lib/gstreamer-1.0/libnvgstjpeg.so)
==6627==    by 0x1C620237: ??? (in /usr/lib/libgstvideo-1.0.so.0.801.0)
==6627== 
==6627== Invalid read of size 1
==6627==    at 0x4849368: memcpy (vg_replace_strmem.c:1035)
==6627==    by 0x24632473: ??? (in /usr/lib/tegra/libnvtvmr.so)
==6627==    by 0x2459423F: jpegTegraDecoderRender (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456B2D3: jpeg_read_raw_data (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2453FDEB: ??? (in /usr/lib/gstreamer-1.0/libnvgstjpeg.so)
==6627==    by 0x1C620237: ??? (in /usr/lib/libgstvideo-1.0.so.0.801.0)
==6627==  Address 0x2c2acb9a is 2 bytes after a block of size 62,888 alloc'd
==6627==    at 0x4844114: malloc (vg_replace_malloc.c:299)
==6627==    by 0x2458EF43: alloc_small (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x24570FB3: copy_buff (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456A67B: jpeg_consume_input (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456AA17: jpeg_read_header (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2453EE13: ??? (in /usr/lib/gstreamer-1.0/libnvgstjpeg.so)
==6627==    by 0x1C620237: ??? (in /usr/lib/libgstvideo-1.0.so.0.801.0)
==6627== 
...
valgrind: m_mallocfree.c:307 (get_bszB_as_is): Assertion 'bszB_lo == bszB_hi' failed.
valgrind: Heap block lo/hi size mismatch: lo = 62960, hi = 1153692914321947468.
This is probably caused by your program erroneously writing past the
end of a heap block and corrupting heap metadata.  If you fix any
invalid writes reported by Memcheck, this assertion failure will
probably go away.  Please try that before reporting this as a bug.

These issues don’t occur using standard jpegdec, so I believe this is a bug in nvjpegdec. Possibly introduced by the changes in that prebuilt nvjpeg library, as the previous version runs without causing corruption (though it does, of course, leak memory).

Hi,
Please share steps of reproducing the issue.

I would have included it in that post, but I was still looking for the root cause and how to reproduce the issue. Using the enclosed images, run

gst-launch-1.0 multifilesrc location=smalltobig%d.jpg ! nvjpegdec ! fakesink

On my TX2, I recieve the output

Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Caught SIGSEGV

Note that although going from a 1024x768 jpeg to a 1920x1080 jpeg causes a crash, going from 1920x1080 to 1024x768 does not. jpegdec, among other jpeg decoding elements, handle this test case without crashing.


Hi,
We don’t support dynamic resolution in nvjpegdec. If your jpeg source varies in resolution, please use jpegdec.

Hi,

I have the same memory leak when using nvjpegdec as described in njewes post, but the difference is that I’m working on Jetson TX2, L4T r32.1.

I would like to know if a fixed version of libnvjpeg.so is available now for this L4T release?

Thanks in advance,
Hila

Hi,
For r32.1, please use prebuilt lib at https://elinux.org/Jetson_AGX_Xavier/r32.1_patch

Thank you, it works now.

Hila

Hi again,

I have the same memory leak on Jetson TX1, L4T r28.2.0.

I tried to use the libnvjpeg.so that is attached in the following link-

https://devtalk.nvidia.com/default/topic/1035699/jetson-tx1/premature-end-of-jpeg-file-tegra_multimedia-master-samples-06_jpeg_decode-/post/5263099/#5263099

but I keep getting a segmentation fault.

I would like to know if there is a different link I could use?
Do you have any other idea how to solve it?

Best regards,
Hila

Hi Hila,
The prebuilt lib should be good for r28.2/TX1.
Could you share a command to reproduce the segment fault?

Hi,

The segment fault happens in the first use of libnvjpeg.so in my code.
I use it through the command decodeToFd(), which calls the function jpeg_read_header().

Thanks,
Hila

Hi,
Please share how to reproduce it through 06_jpeg_decode. If it is specific to a jpeg file, please attach it.