Memory Leak (Alloc/free mismatch) in Tegra multimedia API (encoder)

Hi,

When using address sanitizer with gcc on Xavier AGX (JP4.3/4.5.1/4.6), Asan reports a alloc-dealloc mismatch probably in libv4l2 when using video encoding.

OS :

Reproduced on Jetpack 4.3, 4.5.1 and 4.6

How to reproduce :

  • Use 01_vide_encode sample provided in /usr/src/jetson_multimedia_api/samples/
  • Add sanitizer address in makefile. Add the following line in makefile :
    CXXFLAGS = -g -O0 -fsanitize=address + add the CXXFlags in the linking command
  • Build the sample with make
  • execute the sample with ASAN_OPTIONS=protect_shadow_gap=0 (might not be necessary if no CUDA is involved)
    Ex : $ ASAN_OPTIONS=protect_shadow_gap=0 ./video_encode ../../data/Video/sample.yuv 1920 1080 H264 result.h264

Output of sanitizer :

root@xavier-dev-45623:/usr/src/jetson_multimedia_api/samples/01_video_encode# ASAN_OPTIONS=protect_shadow_gap=0  ./video_encode ../../data/Video/sample.yuv 1920 1080 H264 result.h264
Creating Encoder in blocking mode 
Opening in BLOCKING MODE
=================================================================
==20179==ERROR: AddressSanitizer: alloc-dealloc-mismatch (malloc vs operator delete) on 0x007fa2302750
    #0 0x7fa6ad1987 in operator delete(void*, unsigned long) (/usr/lib/aarch64-linux-gnu/libasan.so.4+0xd3987)
    #1 0x7fa63ba437 in __cxa_finalize (/lib/aarch64-linux-gnu/libc.so.6+0x35437)

0x007fa2302750 is located 0 bytes inside of 40-byte region [0x007fa2302750,0x007fa2302778)
allocated by thread T0 (EncOutPlane) here:
    #0 0x7fa6acdb1b in malloc (/usr/lib/aarch64-linux-gnu/libasan.so.4+0xcfb1b)
    #1 0x7fa1a2a833 in fnet::String::String(char const*) (/usr/lib/aarch64-linux-gnu/tegra/libnvfnet.so+0x7833)
    #2 0x7fa78c1b37  (/lib/ld-linux-aarch64.so.1+0xdb37)
    #3 0x7fa78c5cd7  (/lib/ld-linux-aarch64.so.1+0x11cd7)
    #4 0x7fa6490693 in _dl_catch_exception (/lib/aarch64-linux-gnu/libc.so.6+0x10b693)
    #5 0x7fa78c5417  (/lib/ld-linux-aarch64.so.1+0x11417)
    #6 0x7fa6371013  (/lib/aarch64-linux-gnu/libdl.so.2+0x1013)
    #7 0x7fa6490693 in _dl_catch_exception (/lib/aarch64-linux-gnu/libc.so.6+0x10b693)
    #8 0x7fa6490737 in _dl_catch_error (/lib/aarch64-linux-gnu/libc.so.6+0x10b737)
    #9 0x7fa637277f  (/lib/aarch64-linux-gnu/libdl.so.2+0x277f)
    #10 0x7fa63710e7 in dlopen (/lib/aarch64-linux-gnu/libdl.so.2+0x10e7)
    #11 0x7fa6a4ae83  (/usr/lib/aarch64-linux-gnu/libasan.so.4+0x4ce83)
    #12 0x7fa68c4a67  (/usr/lib/aarch64-linux-gnu/libv4l2.so.0+0x7a67)
    #13 0x557a224db7 in NvVideoEncoder::NvVideoEncoder(char const*, int) (/usr/src/jetson_multimedia_api/samples/01_video_encode/video_encode+0x2fdb7)
    #14 0x557a224e03 in NvVideoEncoder::createVideoEncoder(char const*, int) (/usr/src/jetson_multimedia_api/samples/01_video_encode/video_encode+0x2fe03)
    #15 0x557a202c4b in encode_proc(context_t&, int, char**) (/usr/src/jetson_multimedia_api/samples/01_video_encode/video_encode+0xdc4b)
    #16 0x557a2054bb in main (/usr/src/jetson_multimedia_api/samples/01_video_encode/video_encode+0x104bb)
    #17 0x7fa63a571f in __libc_start_main (/lib/aarch64-linux-gnu/libc.so.6+0x2071f)
    #18 0x557a1fb3b3  (/usr/src/jetson_multimedia_api/samples/01_video_encode/video_encode+0x63b3)

SUMMARY: AddressSanitizer: alloc-dealloc-mismatch (/usr/lib/aarch64-linux-gnu/libasan.so.4+0xd3987) in operator delete(void*, unsigned long)
==20179==HINT: if you don't care about these errors you may set ASAN_OPTIONS=alloc_dealloc_mismatch=0
==20179==ABORTING

Notes that can help:

  • if alloc_dealloc_mismatch is set to 0 , program runs but direct and indirect leaks are shown at the end in NvVideoEncoder::createVideoEncoder()
  • Not seen when using decoding. Only encoding.

Edit : Also available for decoding

root@xavier-dev-45623:/usr/src/jetson_multimedia_api/samples/00_video_decode# ./video_decode H264 ../../data/Video/sample_outdoor_car_1080p_10fps.h264 
Set governor to performance before enabling profiler
Creating decoder in blocking mode 
Opening in BLOCKING MODE
=================================================================
==10912==ERROR: AddressSanitizer: alloc-dealloc-mismatch (malloc vs operator delete) on 0x007fa1702750
    #0 0x7fa5e50987 in operator delete(void*, unsigned long) (/usr/lib/aarch64-linux-gnu/libasan.so.4+0xd3987)
    #1 0x7fa5739437 in __cxa_finalize (/lib/aarch64-linux-gnu/libc.so.6+0x35437)

0x007fa1702750 is located 0 bytes inside of 40-byte region [0x007fa1702750,0x007fa1702778)
allocated by thread T0 (DecOutPlane) here:
    #0 0x7fa5e4cb1b in malloc (/usr/lib/aarch64-linux-gnu/libasan.so.4+0xcfb1b)
    #1 0x7fa0c6e833 in fnet::String::String(char const*) (/usr/lib/aarch64-linux-gnu/tegra/libnvfnet.so+0x7833)
    #2 0x7fa6c40b37  (/lib/ld-linux-aarch64.so.1+0xdb37)
    #3 0x7fa6c44cd7  (/lib/ld-linux-aarch64.so.1+0x11cd7)
    #4 0x7fa580f693 in _dl_catch_exception (/lib/aarch64-linux-gnu/libc.so.6+0x10b693)
    #5 0x7fa6c44417  (/lib/ld-linux-aarch64.so.1+0x11417)
    #6 0x7fa56f0013  (/lib/aarch64-linux-gnu/libdl.so.2+0x1013)
    #7 0x7fa580f693 in _dl_catch_exception (/lib/aarch64-linux-gnu/libc.so.6+0x10b693)
    #8 0x7fa580f737 in _dl_catch_error (/lib/aarch64-linux-gnu/libc.so.6+0x10b737)
    #9 0x7fa56f177f  (/lib/aarch64-linux-gnu/libdl.so.2+0x277f)
    #10 0x7fa56f00e7 in dlopen (/lib/aarch64-linux-gnu/libdl.so.2+0x10e7)
    #11 0x7fa5dc9e83  (/usr/lib/aarch64-linux-gnu/libasan.so.4+0x4ce83)
    #12 0x7fa5c43a67  (/usr/lib/aarch64-linux-gnu/libv4l2.so.0+0x7a67)
    #13 0x556cb6092f in NvVideoDecoder::NvVideoDecoder(char const*, int) (/usr/src/jetson_multimedia_api/samples/00_video_decode/video_decode+0xf92f)
    #14 0x556cb6097b in NvVideoDecoder::createVideoDecoder(char const*, int) (/usr/src/jetson_multimedia_api/samples/00_video_decode/video_decode+0xf97b)
    #15 0x556cb5c2df in decode_proc(context_t&, int, char**) (/usr/src/jetson_multimedia_api/samples/00_video_decode/video_decode+0xb2df)
    #16 0x556cb5d65b in main (/usr/src/jetson_multimedia_api/samples/00_video_decode/video_decode+0xc65b)
    #17 0x7fa572471f in __libc_start_main (/lib/aarch64-linux-gnu/libc.so.6+0x2071f)
    #18 0x556cb57013  (/usr/src/jetson_multimedia_api/samples/00_video_decode/video_decode+0x6013)

SUMMARY: AddressSanitizer: alloc-dealloc-mismatch (/usr/lib/aarch64-linux-gnu/libasan.so.4+0xd3987) in operator delete(void*, unsigned long)
==10912==HINT: if you don't care about these errors you may set ASAN_OPTIONS=alloc_dealloc_mismatch=0
==10912==ABORTING

Thanks.
OB.

Hi,
Thanks for reporting the issue. It should be a known issue and mentioned in
Valgrind Errors/Leaks with jetson_multimedia_api Sample

It is under investigation. Will update once there is new finding.

I have same problem. I am getting segmentation fault when I run a decoder inside a child process.
Any updates?
Jetson AGX Xavier. L4T 32.5.1-20210519111140

Backtrace:

Thread 1 "digital" received signal SIGSEGV, Segmentation fault.
malloc_consolidate (av=av@entry=0x7f7da8ba70 <main_arena>) at malloc.c:4469
4469	malloc.c: No such file or directory.
(gdb) bt
#0  malloc_consolidate (av=av@entry=0x7f7da8ba70 <main_arena>) at malloc.c:4469
#1  0x0000007f7d9ada70 in _int_malloc (av=av@entry=0x7f7da8ba70 <main_arena>, bytes=bytes@entry=1234) at malloc.c:3713
#2  0x0000007f7d9afee0 in __libc_calloc (n=<optimized out>, elem_size=<optimized out>) at malloc.c:3446
#3  0x0000007f9dbdc33c in _dl_new_object (realname=0x5558608530 "/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvcuvidvideocodec.so", realname@entry=0x7f9dc011b0 "", libname=0x0, 
    libname@entry=0x1 <error: Cannot access memory at address 0x1>, type=127, type@entry=-2147483647, loader=loader@entry=0x0, mode=mode@entry=-1879048191, nsid=nsid@entry=0) at dl-object.c:73
#4  0x0000007f9dbd73bc in _dl_map_object_from_fd (name=0x1 <error: Cannot access memory at address 0x1>, 
    name@entry=0x555843fb40 "/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvcuvidvideocodec.so", origname=0x7fe5f0c6b8 "@\373CXU", 
    origname@entry=0x0, fd=<optimized out>, fbp=fbp@entry=0x7fe5f0c0a8, realname=0x7f9dc011b0 "", loader=loader@entry=0x0, l_type=-2147483647, l_type@entry=2, mode=mode@entry=-1879048191, stack_endp=0x7fe5f0c0a0, 
    stack_endp@entry=0x7fe5f0c140, nsid=nsid@entry=0) at dl-load.c:998
#5  0x0000007f9dbd9c68 in _dl_map_object (loader=0x0, 
    loader@entry=0x7f67af2360, name=name@entry=0x555843fb40 "/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvcuvidvideocodec.so", type=type@entry=2, trace_mode=trace_mode@entry=0, mode=mode@entry=-1879048191, nsid=0) at dl-load.c:2461
#6  0x0000007f9dbe3888 in dl_open_worker (a=0x7fe5f0c6b8) at dl-open.c:235
#7  0x0000007f7da43694 in __GI__dl_catch_exception (exception=0xfffffffffffffffe, exception@entry=0x7fe5f0c6a0, operate=0x7fe5f0c4dc, operate@entry=0x7f9dbe37e0 <dl_open_worker>, args=0x7fe5f0c6a0, 
    args@entry=0x7fe5f0c6b8) at dl-error-skeleton.c:196
#8  0x0000007f9dbe3420 in _dl_open
    (file=0x555843fb40 "/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvcuvidvideocodec.so", mode=-2147483647, caller_dlopen=0x7f67434a68, nsid=-2, argc=1, argv=0x7fe5f0e4a8, env=<optimized out>)
    at dl-open.c:605
#9  0x0000007f7d924014 in dlopen_doit (a=0x7fe5f0c978) at dlopen.c:66
#10 0x0000007f7da43694 in __GI__dl_catch_exception (exception=0x7f9dbff7a8 <__stack_chk_guard>, exception@entry=0x7fe5f0c910, operate=0x7fe5f0c76c, operate@entry=0x7f7d923fb0 <dlopen_doit>, args=0x7fe5f0c8f0, 
    args@entry=0x7fe5f0c978) at dl-error-skeleton.c:196
#11 0x0000007f7da43738 in __GI__dl_catch_error
    (objname=objname@entry=0x55581dfeb0, errstring=errstring@entry=0x55581dfeb8, mallocedp=mallocedp@entry=0x55581dfea8, operate=operate@entry=0x7f7d923fb0 <dlopen_doit>, args=args@entry=0x7fe5f0c978)
--Type <RET> for more, q to quit, c to continue without paging--
    at dl-error-skeleton.c:215
#12 0x0000007f7d925780 in _dlerror_run (operate=operate@entry=0x7f7d923fb0 <dlopen_doit>, args=0x7fe5f0c978, args@entry=0x7fe5f0c988) at dlerror.c:162
#13 0x0000007f7d9240e8 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#14 0x0000007f67434a68 in  () at /usr/lib/aarch64-linux-gnu/libv4l2.so.0
#15 0x0000007f674306c0 in v4l2_fd_open () at /usr/lib/aarch64-linux-gnu/libv4l2.so.0
#16 0x0000007f67430e24 in v4l2_open () at /usr/lib/aarch64-linux-gnu/libv4l2.so.0
#17 0x0000007f6f0e4990 in NvV4l2Element::NvV4l2Element(char const*, char const*, int, int) () at /usr/local/lib/libnvmpi.so.1
#18 0x0000007f6f0ee700 in NvVideoDecoder::NvVideoDecoder(char const*, int) () at /usr/local/lib/libnvmpi.so.1
#19 0x0000007f6f0ee750 in NvVideoDecoder::createVideoDecoder(char const*, int) () at /usr/local/lib/libnvmpi.so.1

Sanitizer:

==15315==ERROR: AddressSanitizer: alloc-dealloc-mismatch (malloc vs operator delete) on 0x007f3d266610
    #0 0x7f996f17c8 in operator delete(void*, unsigned long) (/usr/lib/aarch64-linux-gnu/libasan.so.5+0xef7c8)
    #1 0x7f70073434 in __cxa_finalize (/lib/aarch64-linux-gnu/libc.so.6+0x35434)

0x007f3d266610 is located 0 bytes inside of 40-byte region [0x007f3d266610,0x007f3d266638)
allocated by thread T0 here:
    #0 0x7f996ee690 in malloc (/usr/lib/aarch64-linux-gnu/libasan.so.5+0xec690)
    #1 0x7f3a72a830 in fnet::String::String(char const*) (/usr/lib/aarch64-linux-gnu/tegra/libnvfnet.so+0x7830)
    #2 0x7f9a126b3c  (/lib/ld-linux-aarch64.so.1+0xdb3c)
    #3 0x7f9a12acdc  (/lib/ld-linux-aarch64.so.1+0x11cdc)
    #4 0x7f70149690 in _dl_catch_exception (/lib/aarch64-linux-gnu/libc.so.6+0x10b690)
    #5 0x7f9a12a41c  (/lib/ld-linux-aarch64.so.1+0x1141c)
    #6 0x7f7002a010  (/lib/aarch64-linux-gnu/libdl.so.2+0x1010)
    #7 0x7f70149690 in _dl_catch_exception (/lib/aarch64-linux-gnu/libc.so.6+0x10b690)
    #8 0x7f70149734 in _dl_catch_error (/lib/aarch64-linux-gnu/libc.so.6+0x10b734)
    #9 0x7f7002b77c  (/lib/aarch64-linux-gnu/libdl.so.2+0x277c)
    #10 0x7f7002a0e4 in dlopen (/lib/aarch64-linux-gnu/libdl.so.2+0x10e4)
    #11 0x7f99655998  (/usr/lib/aarch64-linux-gnu/libasan.so.5+0x53998)
    #12 0x7f46798a64  (/usr/lib/aarch64-linux-gnu/libv4l2.so.0+0x7a64)
    #13 0x7f705626fc in NvVideoDecoder::NvVideoDecoder(char const*, int) (/usr/local/lib/libnvmpi.so.1+0x246fc)
    #14 0x7f7056274c in NvVideoDecoder::createVideoDecoder(char const*, int) (/usr/local/lib/libnvmpi.so.1+0x2474c)
    #15 0x7f7054f494 in nvmpi_create_decoder (/usr/local/lib/libnvmpi.so.1+0x11494)

Hi,
The issue is underinvestigation. A quick solution is

$ sudo mv /usr/lib/aarch64-linux-gnu/tegra/libv4l2_nvargus.so /usr/lib/aarch64-linux-gnu/tegra/libv4l2_nvargus_back.so

So that libv4l2_nvargus.so will not be loaded. The following sample will be impacted:

/usr/src/jetson_multimedia_api/samples/unittest_samples/camera_unit_sample

If you don’t develop your use-case based on camera_unit_sample, please try this quick solution.

1 Like

It was helps a little bit… Now, when compiling with sanitizer, the program works as expected, but when compiling without it, I still get segmentation fault. :(

P.S. This only happens when I run the decoder in the child process. Without fork() the application works as expected.

New backtrace:

Thread 1 "digital_noSan" received signal SIGSEGV, Segmentation fault.
tcache_get (tc_idx=3) at malloc.c:2952
2952	malloc.c: No such file or directory.
(gdb) bt
#0  tcache_get (tc_idx=3) at malloc.c:2952
#1  __GI___libc_malloc (bytes=64) at malloc.c:3060
#2  0x0000007f8eb05d4c in dl_open_worker (a=0x7fc690eca8) at dl-open.c:432
#3  0x0000007f6e965694 in __GI__dl_catch_exception (exception=0xfffffffffffffffe, exception@entry=0x7fc690ec90, operate=0x7fc690eacc, operate@entry=0x7f8eb057e0 <dl_open_worker>, args=0x7fc690ec90, 
    args@entry=0x7fc690eca8) at dl-error-skeleton.c:196
#4  0x0000007f8eb05420 in _dl_open (file=0x7fc690efc0 "libnvmmlite_video.so", mode=-2147483647, caller_dlopen=0x7f5610a580 <NvOsLibraryLoad+160>, nsid=-2, argc=1, argv=0x7fc69112a8, env=<optimized out>)
    at dl-open.c:605
#5  0x0000007f6e846014 in dlopen_doit (a=0x7fc690ef68) at dlopen.c:66
#6  0x0000007f6e965694 in __GI__dl_catch_exception (exception=0x7f8eb217a8 <__stack_chk_guard>, exception@entry=0x7fc690ef00, operate=0x7fc690ed5c, operate@entry=0x7f6e845fb0 <dlopen_doit>, args=0x7fc690eee0, 
    args@entry=0x7fc690ef68) at dl-error-skeleton.c:196
#7  0x0000007f6e965738 in __GI__dl_catch_error
    (objname=objname@entry=0x55661bfeb0, errstring=errstring@entry=0x55661bfeb8, mallocedp=mallocedp@entry=0x55661bfea8, operate=operate@entry=0x7f6e845fb0 <dlopen_doit>, args=args@entry=0x7fc690ef68)
    at dl-error-skeleton.c:215
#8  0x0000007f6e847780 in _dlerror_run (operate=operate@entry=0x7f6e845fb0 <dlopen_doit>, args=0x7fc690ef68, args@entry=0x7fc690ef78) at dlerror.c:162
#9  0x0000007f6e8460e8 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#10 0x0000007f5610a580 in NvOsLibraryLoad () at /usr/lib/aarch64-linux-gnu/tegra/libnvos.so
#11 0x0000007f55b178b0 in  () at /usr/lib/aarch64-linux-gnu/tegra/libnvmmlite.so
#12 0x0000007f8eb01a3c in call_init (l=<optimized out>, argc=argc@entry=1, argv=argv@entry=0x7fc69112a8, env=env@entry=0x7fc69112b8) at dl-init.c:72
#13 0x0000007f8eb01b40 in call_init (env=0x7fc69112b8, argv=0x7fc69112a8, argc=1, l=<optimized out>) at dl-init.c:118
#14 _dl_init (main_map=main_map@entry=0x556644a4e0, argc=1, argv=0x7fc69112a8, env=0x7fc69112b8) at dl-init.c:119
#15 0x0000007f8eb05ce0 in dl_open_worker (a=0x7fc690f4b8) at dl-open.c:522
#16 0x0000007f6e965694 in __GI__dl_catch_exception (exception=0xfffffffffffffffe, exception@entry=0x7fc690f4a0, operate=0x7fc690f2dc, operate@entry=0x7f8eb057e0 <dl_open_worker>, args=0x7fc690f4a0, 
    args@entry=0x7fc690f4b8) at dl-error-skeleton.c:196
--Type <RET> for more, q to quit, c to continue without paging--v
#17 0x0000007f8eb05420 in _dl_open
    (file=0x5566449a60 "/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvvideocodec.so", mode=-2147483647, caller_dlopen=0x7f58356a68, nsid=-2, argc=1, argv=0x7fc69112a8, env=<optimized out>) at dl-open.c:605
#18 0x0000007f6e846014 in dlopen_doit (a=0x7fc690f778) at dlopen.c:66
#19 0x0000007f6e965694 in __GI__dl_catch_exception (exception=0x7f8eb217a8 <__stack_chk_guard>, exception@entry=0x7fc690f710, operate=0x7fc690f56c, operate@entry=0x7f6e845fb0 <dlopen_doit>, args=0x7fc690f6f0, 
    args@entry=0x7fc690f778) at dl-error-skeleton.c:196
#20 0x0000007f6e965738 in __GI__dl_catch_error
    (objname=objname@entry=0x55661bfeb0, errstring=errstring@entry=0x55661bfeb8, mallocedp=mallocedp@entry=0x55661bfea8, operate=operate@entry=0x7f6e845fb0 <dlopen_doit>, args=args@entry=0x7fc690f778)
    at dl-error-skeleton.c:215
#21 0x0000007f6e847780 in _dlerror_run (operate=operate@entry=0x7f6e845fb0 <dlopen_doit>, args=0x7fc690f778, args@entry=0x7fc690f788) at dlerror.c:162
#22 0x0000007f6e8460e8 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#23 0x0000007f58356a68 in  () at /usr/lib/aarch64-linux-gnu/libv4l2.so.0
#24 0x0000007f583526c0 in v4l2_fd_open () at /usr/lib/aarch64-linux-gnu/libv4l2.so.0
#25 0x0000007f58352e24 in v4l2_open () at /usr/lib/aarch64-linux-gnu/libv4l2.so.0
#26 0x0000007f60006990 in NvV4l2Element::NvV4l2Element(char const*, char const*, int, int) () at /usr/local/lib/libnvmpi.so.1
#27 0x0000007f60010700 in NvVideoDecoder::NvVideoDecoder(char const*, int) () at /usr/local/lib/libnvmpi.so.1
#28 0x0000007f60010750 in NvVideoDecoder::createVideoDecoder(char const*, int) () at /usr/local/lib/libnvmpi.so.1
#29 0x0000007f5fffd498 in nvmpi_create_decoder () at /usr/local/lib/libnvmpi.so.1

Hi,
It looks different from this one. Please make a new topic and provide the steps so that we can reproduce the failure.