nvjpegdec v1.2.3 memory leak

njewers · January 3, 2019, 8:35pm

I believe I’ve found a memory leak in nvjpegdec. It can be reproduced by running the following snippet on a TX2 board, where test.jpeg is an arbitrary 1920x1080 jpeg image (Filesize ~1M).

gst-launch-1.0 multifilesrc location=test.jpg caps=image/jpeg,framerate=24/1 ! nvjpegdec ! fakesink

On my TX2 board, this process steadily increases in memory usage. Starting at 229m, within a couple of minutes it has grown to 997m. Approximately 25 minutes later, it has grown to 5285m. This is an extreme example, but even throttled to 30fps in my application the memory leak causes a significant increase in memory usage.

This is not reproducible with other jpeg decoders, such as the standard jpegdec, and only seems to occur with nvjpegdec. gst-inspect-1.0 reports that I am using version 1.2.3 of nvjpegdec. Is this an issue that has been addressed in a later version, or is there a known workaround?

DaneLLL · January 4, 2019, 1:20am

Please try the prebuilt library at
[url]https://devtalk.nvidia.com/default/topic/1035699/jetson-tx1/premature-end-of-jpeg-file-tegra_multimedia-master-samples-06_jpeg_decode-/post/5263099/#5263099[/url]

njewers · January 4, 2019, 3:39pm

That library seems to work without the memory leak, thank you. Is that version of the library with those fixes available in one of the newer SDKs?

DaneLLL · January 7, 2019, 2:12am

Hi,
It shall be fixed in next release.

njewers · January 7, 2019, 4:20pm

During further testing I’ve had some more difficulties with that library, unfortunately. In my application, if I use nvjpegdec with the nvjpeg library linked there, the application logs a malloc(): memory corruption message. I’ve gotten the following output from Valgrind, suggesting an issue in nvjpegdec that’s writing outside of allocated memory and corrupting the heap:

==6627== Thread 36 e0_ptpsrc:src:
==6627== Invalid write of size 8
==6627==    at 0x4849294: memcpy (vg_replace_strmem.c:1035)
==6627==    by 0x24570F4B: copy_buff (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456A67B: jpeg_consume_input (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456AA17: jpeg_read_header (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2453EE13: ??? (in /usr/lib/gstreamer-1.0/libnvgstjpeg.so)
==6627==    by 0x1C620237: ??? (in /usr/lib/libgstvideo-1.0.so.0.801.0)
==6627==  Address 0x2c2acb98 is 0 bytes after a block of size 62,888 alloc'd
==6627==    at 0x4844114: malloc (vg_replace_malloc.c:299)
==6627==    by 0x2458EF43: alloc_small (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x24570FB3: copy_buff (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456A67B: jpeg_consume_input (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456AA17: jpeg_read_header (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2453EE13: ??? (in /usr/lib/gstreamer-1.0/libnvgstjpeg.so)
==6627==    by 0x1C620237: ??? (in /usr/lib/libgstvideo-1.0.so.0.801.0)
==6627== 
==6627== Invalid write of size 8
==6627==    at 0x484929C: memcpy (vg_replace_strmem.c:1035)
==6627==    by 0x24570F4B: copy_buff (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456A67B: jpeg_consume_input (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456AA17: jpeg_read_header (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2453EE13: ??? (in /usr/lib/gstreamer-1.0/libnvgstjpeg.so)
==6627==    by 0x1C620237: ??? (in /usr/lib/libgstvideo-1.0.so.0.801.0)
==6627==  Address 0x2c2acba0 is 8 bytes after a block of size 62,888 alloc'd
==6627==    at 0x4844114: malloc (vg_replace_malloc.c:299)
==6627==    by 0x2458EF43: alloc_small (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x24570FB3: copy_buff (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456A67B: jpeg_consume_input (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456AA17: jpeg_read_header (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2453EE13: ??? (in /usr/lib/gstreamer-1.0/libnvgstjpeg.so)
==6627==    by 0x1C620237: ??? (in /usr/lib/libgstvideo-1.0.so.0.801.0)
==6627== 
==6627== Invalid write of size 8
==6627==    at 0x484927C: memcpy (vg_replace_strmem.c:1035)
==6627==    by 0x24570F4B: copy_buff (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456A67B: jpeg_consume_input (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456AA17: jpeg_read_header (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2453EE13: ??? (in /usr/lib/gstreamer-1.0/libnvgstjpeg.so)
==6627==    by 0x1C620237: ??? (in /usr/lib/libgstvideo-1.0.so.0.801.0)
==6627==  Address 0x2c2acba8 is 16 bytes after a block of size 62,888 alloc'd
==6627==    at 0x4844114: malloc (vg_replace_malloc.c:299)
==6627==    by 0x2458EF43: alloc_small (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x24570FB3: copy_buff (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456A67B: jpeg_consume_input (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456AA17: jpeg_read_header (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2453EE13: ??? (in /usr/lib/gstreamer-1.0/libnvgstjpeg.so)
==6627==    by 0x1C620237: ??? (in /usr/lib/libgstvideo-1.0.so.0.801.0)
==6627== 
==6627== Invalid write of size 8
==6627==    at 0x484928C: memcpy (vg_replace_strmem.c:1035)
==6627==    by 0x24570F4B: copy_buff (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456A67B: jpeg_consume_input (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456AA17: jpeg_read_header (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2453EE13: ??? (in /usr/lib/gstreamer-1.0/libnvgstjpeg.so)
==6627==    by 0x1C620237: ??? (in /usr/lib/libgstvideo-1.0.so.0.801.0)
==6627==  Address 0x2c2acbb0 is 16 bytes after a block of size 62,896 in arena "client"
==6627== 
==6627== Invalid read of size 1
==6627==    at 0x4849378: memcpy (vg_replace_strmem.c:1035)
==6627==    by 0x24632473: ??? (in /usr/lib/tegra/libnvtvmr.so)
==6627==    by 0x2459423F: jpegTegraDecoderRender (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456B2D3: jpeg_read_raw_data (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2453FDEB: ??? (in /usr/lib/gstreamer-1.0/libnvgstjpeg.so)
==6627==    by 0x1C620237: ??? (in /usr/lib/libgstvideo-1.0.so.0.801.0)
==6627==  Address 0x2c2acb98 is 0 bytes after a block of size 62,888 alloc'd
==6627==    at 0x4844114: malloc (vg_replace_malloc.c:299)
==6627==    by 0x2458EF43: alloc_small (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x24570FB3: copy_buff (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456A67B: jpeg_consume_input (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456AA17: jpeg_read_header (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2453EE13: ??? (in /usr/lib/gstreamer-1.0/libnvgstjpeg.so)
==6627==    by 0x1C620237: ??? (in /usr/lib/libgstvideo-1.0.so.0.801.0)
==6627== 
==6627== Invalid read of size 1
==6627==    at 0x4849368: memcpy (vg_replace_strmem.c:1035)
==6627==    by 0x24632473: ??? (in /usr/lib/tegra/libnvtvmr.so)
==6627==    by 0x2459423F: jpegTegraDecoderRender (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456B2D3: jpeg_read_raw_data (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2453FDEB: ??? (in /usr/lib/gstreamer-1.0/libnvgstjpeg.so)
==6627==    by 0x1C620237: ??? (in /usr/lib/libgstvideo-1.0.so.0.801.0)
==6627==  Address 0x2c2acb9a is 2 bytes after a block of size 62,888 alloc'd
==6627==    at 0x4844114: malloc (vg_replace_malloc.c:299)
==6627==    by 0x2458EF43: alloc_small (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x24570FB3: copy_buff (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456A67B: jpeg_consume_input (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2456AA17: jpeg_read_header (in /usr/lib/tegra/libnvjpeg.so)
==6627==    by 0x2453EE13: ??? (in /usr/lib/gstreamer-1.0/libnvgstjpeg.so)
==6627==    by 0x1C620237: ??? (in /usr/lib/libgstvideo-1.0.so.0.801.0)
==6627== 
...
valgrind: m_mallocfree.c:307 (get_bszB_as_is): Assertion 'bszB_lo == bszB_hi' failed.
valgrind: Heap block lo/hi size mismatch: lo = 62960, hi = 1153692914321947468.
This is probably caused by your program erroneously writing past the
end of a heap block and corrupting heap metadata.  If you fix any
invalid writes reported by Memcheck, this assertion failure will
probably go away.  Please try that before reporting this as a bug.

These issues don’t occur using standard jpegdec, so I believe this is a bug in nvjpegdec. Possibly introduced by the changes in that prebuilt nvjpeg library, as the previous version runs without causing corruption (though it does, of course, leak memory).

DaneLLL · January 8, 2019, 1:55am

Hi,
Please share steps of reproducing the issue.

njewers · January 8, 2019, 4:14pm

I would have included it in that post, but I was still looking for the root cause and how to reproduce the issue. Using the enclosed images, run

gst-launch-1.0 multifilesrc location=smalltobig%d.jpg ! nvjpegdec ! fakesink

On my TX2, I recieve the output

Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Caught SIGSEGV

Note that although going from a 1024x768 jpeg to a 1920x1080 jpeg causes a crash, going from 1920x1080 to 1024x768 does not. jpegdec, among other jpeg decoding elements, handle this test case without crashing.

DaneLLL · January 9, 2019, 3:19am

Hi,
We don’t support dynamic resolution in nvjpegdec. If your jpeg source varies in resolution, please use jpegdec.

hila · May 12, 2019, 1:34pm

Hi,

I have the same memory leak when using nvjpegdec as described in njewes post, but the difference is that I’m working on Jetson TX2, L4T r32.1.

I would like to know if a fixed version of libnvjpeg.so is available now for this L4T release?

Thanks in advance,
Hila

DaneLLL · May 13, 2019, 4:24am

Hi,
For r32.1, please use prebuilt lib at Jetson AGX Xavier/r32.1 patch - eLinux.org

hila · May 13, 2019, 2:01pm

Thank you, it works now.

Hila

hila · May 21, 2019, 9:50am

Hi again,

I have the same memory leak on Jetson TX1, L4T r28.2.0.

I tried to use the libnvjpeg.so that is attached in the following link-

https://devtalk.nvidia.com/default/topic/1035699/jetson-tx1/premature-end-of-jpeg-file-tegra_multimedia-master-samples-06_jpeg_decode-/post/5263099/#5263099

but I keep getting a segmentation fault.

I would like to know if there is a different link I could use?
Do you have any other idea how to solve it?

Best regards,
Hila

DaneLLL · May 21, 2019, 10:29am

Hi Hila,
The prebuilt lib should be good for r28.2/TX1.
Could you share a command to reproduce the segment fault?

hila · May 21, 2019, 11:19am

Hi,

The segment fault happens in the first use of libnvjpeg.so in my code.
I use it through the command decodeToFd(), which calls the function jpeg_read_header().

Thanks,
Hila

DaneLLL · May 22, 2019, 1:48am

Hi,
Please share how to reproduce it through 06_jpeg_decode. If it is specific to a jpeg file, please attach it.

Topic		Replies	Views
TX2 sigsegv with patched libnvjpeg.so Jetson TX2	22	2113	October 18, 2021
nvjpegdec slower then jpegdec in gstreamer Jetson AGX Xavier	38	8436	October 18, 2021
Decoding mjpeg stream using the NvJPEGDecoder Jetson TX2	8	2522	October 18, 2021
Memory corruption problems nvv4l2decoder/nvjpegdec in R32.4.4 + R35.4.1 Jetson TX2 decoder	4	683	December 4, 2023
Gst-launch-1.0 nvjpegdec cannot apply video resize in combination with videoscale Jetson Nano gstreamer	10	1651	October 18, 2021
Nvv4l2decoder memory issue DeepStream SDK deepstream61	14	920	August 16, 2022
omxh264dec/nvvidconv: 34.7MB per minute memory leak/crash when decoding Interlaced H264 video Jetson Nano	27	3068	October 14, 2021
R28.2.1 : decodeToFd leak memory ?! Jetson TX2	7	952	October 18, 2021
Error while building the latest gstjpeg_src (L4T sources, version 32.3.1) Jetson Nano	9	758	October 14, 2021
Nvv4l2decoder memory leak Jetson Xavier NX gstreamer	10	1094	October 19, 2022

nvjpegdec v1.2.3 memory leak

Related topics