NvVideoEncoder and the address-sanitizer

Hi,

I’m trying to run the tegra_multimedia_example “01_video_encode” with AddressSanitizer to find memory-leaks (version 28.2.1 on a Jetson TX2 running JetPack 3.3). I’ve compiled the source code from the sample directory (including the common directory) with the clang v3.8.0 compiler. I’ve used the following options in the Rules.mk:

CPPFLAGS := -O1 -g -fsanitize=address -fno-omit-frame-pointer
LDFLAGS := -g -fsanitize=address
...
AS             = $(AT) $(CROSS_COMPILE)llvm-as
LD             = $(AT) $(CROSS_COMPILE)llvm-ld
CC             = $(AT) $(CROSS_COMPILE)clang
CPP            = $(AT) $(CROSS_COMPILE)clang++
AR             = $(AT) $(CROSS_COMPILE)llvm-ar
NM             = $(AT) $(CROSS_COMPILE)llvm-nm
STRIP          = $(AT) $(CROSS_COMPILE)llvm-strip
OBJCOPY        = $(AT) $(CROSS_COMPILE)llvm-objcopy
OBJDUMP        = $(AT) $(CROSS_COMPILE)llvm-objdump

Now I run the video_encode example

./video_encode sample_outdoor_car_1080p_10fps.yuv 1920 1080 H264 test.h264

The program terminates with the following error:

Failed to query video capabilities: Inappropriate ioctl for device
NvMMLiteOpen : Block : BlockType = 4 
===== MSENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
875967048
842091865
===== MSENC blits (mode: 1) into tiled surfaces =====
=================================================================
==8233==ERROR: AddressSanitizer: memcpy-param-overlap: memory ranges [0x007f6a8a1000,0x007f6a8bdf31) and [0x007f6a8a1017, 0x007f6a8bdf48) overlap
    #0 0x457b63 in __interceptor_memcpy.part.43 (/home/nvidia/research/nvidia/video_encode+0x457b63)
    #1 0x7f72840e47 in copy_nvmm_enc_buffer (/usr/lib/aarch64-linux-gnu/tegra/libtegrav4l2.so+0x14e47)

AddressSanitizer can not describe address in more detail (wild memory access suspected).
AddressSanitizer can not describe address in more detail (wild memory access suspected).
SUMMARY: AddressSanitizer: memcpy-param-overlap (/home/nvidia/research/nvidia/video_encode+0x457b63) in __interceptor_memcpy.part.43
Thread T1 created by T0 here:
    #0 0x430dbf in pthread_create (/home/nvidia/research/nvidia/video_encode+0x430dbf)
    #1 0x7f758eb7e3  (/usr/lib/aarch64-linux-gnu/tegra/libnvos.so+0x87e3)
    #2 0x5253b3 in NvV4l2ElementPlane::setStreamStatus(bool) /home/nvidia/research/nvidia/common/classes/NvV4l2ElementPlane.cpp:544:15
    #3 0x4e2eb7 in main /home/nvidia/research/nvidia/video_encode_main.cpp:864:15
    #4 0x7f8633c89f in __libc_start_main /build/glibc-BeHOFw/glibc-2.23/csu/libc-start.c:291
    #5 0x422177 in _start (/home/nvidia/research/nvidia/video_encode+0x422177)

==8233==ABORTING

It looks like some uninitialised memory is accessed in the libnvos.so or libtegrav4l2.so library.

All examples that use the NvVideoEncoder show the same behaviour, making it impossible to run a program to find memory-leaks. Do you have any suggestions to get this to work?

Hi,
On r28.2.1, please apply the prebuilt so at
[MMAPI]Cannot run NvVideoDecoder in loop/Memory leak in NvVideoEncoder
https://elinux.org/Jetson_TX2/28.2.1_patches

Or you may upgrade to r28.3.

Hi Dane,

I get the same error using the patched library on the Jetson TX2.

Failed to query video capabilities: Inappropriate ioctl for device
NvMMLiteOpen : Block : BlockType = 4 
===== MSENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
875967048
842091865
===== MSENC blits (mode: 1) into tiled surfaces =====
=================================================================
==23311==ERROR: AddressSanitizer: memcpy-param-overlap: memory ranges [0x007f8c5a1000,0x007f8c5bdf31) and [0x007f8c5a1017, 0x007f8c5bdf48) overlap
    #0 0x457b63 in __interceptor_memcpy.part.43 (/home/nvidia/research/nvidia/video_encode+0x457b63)
    #1 0x7f94562037 in copy_nvmm_enc_buffer (/usr/lib/aarch64-linux-gnu/tegra/libtegrav4l2.so+0x15037)

AddressSanitizer can not describe address in more detail (wild memory access suspected).
AddressSanitizer can not describe address in more detail (wild memory access suspected).
SUMMARY: AddressSanitizer: memcpy-param-overlap (/home/nvidia/research/nvidia/video_encode+0x457b63) in __interceptor_memcpy.part.43
Thread T1 created by T0 here:
    #0 0x430dbf in pthread_create (/home/nvidia/research/nvidia/video_encode+0x430dbf)
    #1 0x7f976257e3  (/usr/lib/aarch64-linux-gnu/tegra/libnvos.so+0x87e3)
    #2 0x525313 in NvV4l2ElementPlane::setStreamStatus(bool) /home/nvidia/research/nvidia/common/classes/NvV4l2ElementPlane.cpp:544:15
    #3 0x4e2eb7 in main /home/nvidia/research/nvidia/video_encode_main.cpp:864:15
    #4 0x7fa807689f in __libc_start_main /build/glibc-BeHOFw/glibc-2.23/csu/libc-start.c:291
    #5 0x422177 in _start (/home/nvidia/research/nvidia/video_encode+0x422177)

==23311==ABORTING

I’ve also checked compiling and running the executable on a Xavier running the latest version of Jetpack. On the latest Jetpack, the default version of clang is 6.0.0. I got a similar error:

Creating Encoder in blocking mode 
Failed to query video capabilities: Inappropriate ioctl for device
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
875967048
842091865
H264: Profile = 66, Level = 51 
=================================================================
==32207==ERROR: AddressSanitizer: memcpy-param-overlap: memory ranges [0x007f768a1000,0x007f768d118a) and [0x007f768a1018, 0x007f768d11a2) overlap
    #0 0x43fc47 in __interceptor_memcpy.part.41 (/home/seadmin/research/nvidia/video_encode+0x43fc47)
    #1 0x7f83a38a0b  (/usr/lib/aarch64-linux-gnu/tegra/libtegrav4l2.so+0x17a0b)
    #2 0x7f83a3adf3  (/usr/lib/aarch64-linux-gnu/tegra/libtegrav4l2.so+0x19df3)
    #3 0x7f871a1627  (/usr/lib/aarch64-linux-gnu/tegra/libnvos.so+0x8627)
    #4 0x7fa9c01087 in start_thread (/lib/aarch64-linux-gnu/libpthread.so.0+0x7087)

Address 0x007f768a1000 is a wild pointer.
Address 0x007f768a1018 is a wild pointer.
SUMMARY: AddressSanitizer: memcpy-param-overlap (/home/seadmin/research/nvidia/video_encode+0x43fc47) in __interceptor_memcpy.part.41
Thread T7 created by T0 here:
    #0 0x448dfb in pthread_create (/home/seadmin/research/nvidia/video_encode+0x448dfb)
    #1 0x7f871a170f  (/usr/lib/aarch64-linux-gnu/tegra/libnvos.so+0x870f)
    #2 0x51403f in encode_proc(context_t&, int, char**) /home/seadmin/research/nvidia/video_encode_main.cpp:1407:37
    #3 0x511da3 in main /home/seadmin/research/nvidia/video_encode_main.cpp:1754:15
    #4 0x7fa0a9e6df in __libc_start_main /build/glibc-4fr630/glibc-2.27/csu/../csu/libc-start.c:310
    #5 0x4242e7 in _start (/home/seadmin/research/nvidia/video_encode+0x4242e7)

==32207==ABORTING

Any ideas?

Hi,
We have added stress test in 01_video_encode to check memory leakage:

-s <loop-count>       Stress test [Default = 1]

And the test result looks fine.

Could you share more detail about this error? Is it suspicious memory lost or definitely memory lost? We don’t have experience of using the tool and please advise. Thanks.

Hi,

I’m not looking for lost memory in this specific application, but in the application I’m developing. In order to be able find the memory leaks in my application I’m using clang and the address-sanitizer. The problem is that if I enable the video-encoder in this setup the program stops with an error similar to the one above.

To be able to check if the programming error was on my side, I first try to run the tegra_multimedia_api example programs that use a video encoder, but these examples exhibit the same behaviour: the example stops with the above error when the first frame is queued in the output-plane of the video encoder.

To compile the examples with the clang compiler I only changed the settings in the Rules.mk file (see the first post). After the compiling step is finished you can simple start the executable and it will stop with the above error.

Thanks for looking into this,
Boris.

Hi,
Could you share more information about this error? What kind of SW defect is indicated?

Here you can find some information on how the address-sanitizer works:
https://clang.llvm.org/docs/AddressSanitizer.html

Any progress on locating the “memcpy-param-overlap” problem in the libnvos.so library? If you compile the (private) NVIDIA libraries using the clang compiler and run the video_encode example in the address-sanitizer “mode”, it should be pretty easy to find the problem. The debug output will exactly show what is wrong and on what line of the source-code.

Hi,
We shall have it fixed in next r32 release. Do you plan to migrate to r32 or stay on r28?

That’s good news. I am planning to migrate to r32. What version of JetPack is that?
I think I use r32 on the Xavier that has a similar behaviour (see previous post), will that one be fixed too?