I tried tensorrt on jetson nano, mobile-yolo-tensorrt
-jetson-nano image : jetson-nano-sd-r32.2-2019-07-16.zip, nv-jetson-nano-sd-card-image-0826-r32.2.1.zip
-tensorrt-version : tensorrt : 5.1.6.1 + cuda10.0
When I running my program that has tensorrt, cuda library , Program down one day after may be 20 hours later.
memory increases gradually , And my program down one day after
I doubt the tensorrt+cuda memory has leakage problem.
How Can I Fix it???
///////////////////////////
I used jetson nano system monitor tool to monitor memory and swap memory, top viewer tool
and I catched leakage log using Valgrind-3.13.0
10:45 52.9 13.2 <-system monitor
10:45 10.241g 1.105g 387324 <-top viewer
10:55 54.1 13.2
10:55 10.242g 1.148g 387324
11:35 58.1 13.2
11:35 10.242g 1.302g 387324
12:00 60.3 13.2
12:00 10.492g 1.396g 387324
13:10 67.1 13.2
13:10 10.742g 1.663g 388564
=== valgrind leakage log ===
==18626== Memcheck, a memory error detector
==18626== Copyright (C) 2002-2017, and GNU GPL’d, by Julian Seward et al.
==18626== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==18626== Command: ./install/runYolov3 --caffemodel=./models/mobilenet_y==18623==
==18623== Process terminating with default action of signal 2 (SIGINT)
==18623== at 0x48B64D8: raise (raise.c:51)
==18623== by 0x10EC47: ??? (in /bin/dash)
==18623==
==18623== HEAP SUMMARY:
==18623== in use at exit: 1,595 bytes in 33 blocks
==18623== total heap usage: 33 allocs, 0 frees, 1,595 bytes allocated
==18623==
==18623== LEAK SUMMARY:
==18623== definitely lost: 0 bytes in 0 blocks
==18623== indirectly lost: 0 bytes in 0 blocks
==18623== possibly lost: 0 bytes in 0 blocks
==18623== still reachable: 1,595 bytes in 33 blocks
==18623== suppressed: 0 bytes in 0 blocks
==18623== Reachable blocks (those to which a pointer was found) are not shown.
==18623== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==18623==
==18623== For counts of detected and suppressed errors, rerun with: -v
==18623== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
xEB1F06F: std::__basic_file::xsputn_2(char const*, long, char const*, long) (in /usr/lib/aarch64-linux-gnu/libstdc++.so.6.0.25)
==18626== by 0xEB5C1F3: std::basic_filebuf<char, std::char_traits >::xsputn(char const*, long) (in /usr/lib/aarch64-linux-gnu/libstdc++.so.6.0.25)
==18626== by 0xEB7EE2B: std::ostream::write(char const*, long) (in /usr/lib/aarch64-linux-gnu/libstdc++.so.6.0.25)
==18626== by 0x131F6B: Tn::trtNet::saveEngine(std::__cxx11::basic_string<char, std::char_traits, std::allocator >) (in /home/dkai/kmu/install/runYolov3)
==18626== by 0x12994B: main (in /home/dkai/kmu/install/runYolov3)
==18626== Address 0x8015734c is 780 bytes inside a block of size 21,159,992 alloc’d
==18626== at 0x4845A84: operator new(unsigned long) (vg_replace_malloc.c:423)
==18626== by 0x4B84D27: nvinfer1::cudnn::serializeEngine(nvinfer1::rt::Engine const&) (in /usr/lib/aarch64-linux-gnu/libnvinfer.so.5.1.6)
==18626== by 0x131E73: Tn::trtNet::saveEngine(std::__cxx11::basic_string<char, std::char_traits, std::allocator >) (in /home/dkai/kmu/install/runYolov3)
==18626== by 0x12994B: main (in /home/dkai/kmu/install/runYolov3)
==18626==
==18626==
==18626== Process terminating with default action of signal 2 (SIGINT)
==18626== at 0x488C320: __pthread_timedjoin_ex (pthread_join_common.c:89)
==18626== by 0x12FF9F: do_video_or_cam(Tn::trtNet&) (in /home/dkai/kmu/install/runYolov3)
==18626== by 0x129EFB: main (in /home/dkai/kmu/install/runYolov3)
==18626==
==18626== HEAP SUMMARY:
==18626== in use at exit: 724,028,912 bytes in 745,307 blocks
==18626== total heap usage: 2,750,033 allocs, 2,004,726 frees, 5,542,656,930 bytes allocated
==18626==
==18626== 4 bytes in 1 blocks are possibly lost in loss record 12 of 4,790
==18626== at 0x4844CC8: malloc (vg_replace_malloc.c:299)
==18626== by 0x3223B51B: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==18626== by 0x322E2097: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==18626== by 0x322ECC47: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==18626== by 0x322ED53B: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==18626== by 0x3235F7BB: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==18626== by 0x3235FC63: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==18626== by 0x3236021B: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==18626== by 0x32360843: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==18626== by 0x321EF77B: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==18626== by 0x321F1AF7: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==18626== by 0x32147A17: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
…
==18626==
==18626== 274,856 bytes in 2,021 blocks are possibly lost in loss record 4,714 of 4,790
==18626== at 0x4846D54: calloc (vg_replace_malloc.c:711)
==18626== by 0x32292E77: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==18626== by 0x322ED427: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==18626== by 0x322ED53B: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==18626== by 0x321FE82F: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==18626== by 0x321FD11B: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==18626== by 0x3221F357: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==18626== by 0x3218BC3B: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==18626== by 0xFC8A003: ??? (in /usr/lib/aarch64-linux-gnu/libcudnn.so.7.5.0)
==18626==
==18626== 274,856 bytes in 2,021 blocks are possibly lost in loss record 4,715 of 4,790
==18626== at 0x4846D54: calloc (vg_replace_malloc.c:711)
==18626== by 0x32292E77: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==18626== by 0x3223C457: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==18626== by 0x322ED69F: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==18626== by 0x321FE82F: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==18626== by 0x321FD11B: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==18626== by 0x3221F357: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==18626== by 0x3218BC3B: ??? (in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1.1)
==18626== by 0xFC8A003: ??? (in /usr/lib/aarch64-linux-gnu/libcudnn.so.7.5.0)
==18626==
==18626== 21,159,992 bytes in 1 blocks are possibly lost in loss record 4,782 of 4,790
==18626== at 0x4845A84: operator new(unsigned long) (vg_replace_malloc.c:423)
==18626== by 0x4B84D27: nvinfer1::cudnn::serializeEngine(nvinfer1::rt::Engine const&) (in /usr/lib/aarch64-linux-gnu/libnvinfer.so.5.1.6)
==18626== by 0x131E73: Tn::trtNet::saveEngine(std::__cxx11::basic_string<char, std::char_traits, std::allocator >) (in /home/dkai/kmu/install/runYolov3)
==18626== by 0x12994B: main (in /home/dkai/kmu/install/runYolov3)
==18626==
==18626== LEAK SUMMARY:
==18626== definitely lost: 88 bytes in 2 blocks
==18626== indirectly lost: 0 bytes in 0 blocks
==18626== possibly lost: 23,677,468 bytes in 18,990 blocks
==18626== still reachable: 700,258,732 bytes in 725,878 blocks
==18626== of which reachable via heuristic:
==18626== stdstring : 53,006 bytes in 876 blocks
==18626== length64 : 1,384 bytes in 22 blocks
==18626== newarray : 3,208 bytes in 26 blocks
==18626== suppressed: 0 bytes in 0 blocks
==18626== Reachable blocks (those to which a pointer was found) are not shown.
==18626== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==18626==
==18626== For counts of detected and suppressed errors, rerun with: -v
==18626== Use --track-origins=yes to see where uninitialised values come from
==18626== ERROR SUMMARY: 1106 errors from 1106 contexts (suppressed: 0 from 0)
///////////////////////////////////////////