Fail to run sample_mnist after installing TensorRT

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

I was installing TensorRT 4.0.1.6 on Ubuntu 16.04 64-bit, with CUDA 8.0 and cuDNN. I was installing using dpkg on .deb files. The installation looks fine until I tried to verify sample application. When I ran ./sample_mnist in bin folder, it reported:

*** Error in `./sample_mnist’: free(): invalid next size (fast): 0x00007fbe8cbfc5e0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fbebe7e77e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7fbebe7f037a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7fbebe7f453c]
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.5(cudnnDestroyConvolutionDescriptor+0x9)[0x7fbebfa40ff9]
/usr/lib/x86_64-linux-gnu/libnvinfer.so.4(_ZN8nvinfer15cudnn22CudnnConvolutionTraits15getValidTacticsERKNS0_21CudnnConvolutionLayerERKNS0_18EngineBuildContextE+0x1cc)[0x7fbec5659e2c]
/usr/lib/x86_64-linux-gnu/libnvinfer.so.4(_ZN8nvinfer17builder15getValidTacticsERKNS_5cudnn5LayerERKNS1_18EngineBuildContextE+0x155)[0x7fbec561c135]
/usr/lib/x86_64-linux-gnu/libnvinfer.so.4(_ZN8nvinfer15cudnn33selectFastestLayerAndDeleteOthersERNS0_18EngineBuildContextERKSt6vectorIPNS0_5LayerESaIS5_EE+0x82)[0x7fbec560c7f2]
/usr/lib/x86_64-linux-gnu/libnvinfer.so.4(_ZN8nvinfer17builder16buildSingleLayerERNS_5cudnn18EngineBuildContextERNS0_4NodeERKSt13unordered_mapISsSt10unique_ptrINS1_6RegionESt14default_deleteIS8_EESt4hashISsESt8equal_toISsESaISt4pairIKSsSB_EEERNS_14CpuMemoryGroupEPS6_ISsSt6vectorIfSaIfEESD_SF_SaISG_ISH_SR_EEEb+0x130f)[0x7fbec563552f]
/usr/lib/x86_64-linux-gnu/libnvinfer.so.4(_ZN8nvinfer17builder18EngineTacticSupply13getBestTacticERNS0_4NodeERKNS_5query5PortsINS_13RegionFormatLEEEb+0x349)[0x7fbec5636509]
/usr/lib/x86_64-linux-gnu/libnvinfer.so.4(+0x3f013d)[0x7fbec562913d]
/usr/lib/x86_64-linux-gnu/libnvinfer.so.4(_ZN8nvinfer17builder23chooseFormatsAndTacticsERNS0_5GraphERNS0_12TacticSupplyEPSt13unordered_mapISsSt6vectorIfSaIfEESt4hashISsESt8equal_toISsESaISt4pairIKSsS8_EEE+0x1339)[0x7fbec562c879]
/usr/lib/x86_64-linux-gnu/libnvinfer.so.4(_ZN8nvinfer17builder19makeEngineFromGraphERKNS_21CudaEngineBuildConfigERKNS_5cudnn15HardwareContextERNS0_5GraphEPSt13unordered_mapISsSt6vectorIfSaIfEESt4hashISsESt8equal_toISsESaISt4pairIKSsSD_EEEi+0x2a9)[0x7fbec56378a9]
/usr/lib/x86_64-linux-gnu/libnvinfer.so.4(_ZN8nvinfer17builder11buildEngineERNS_21CudaEngineBuildConfigERKNS_5cudnn15HardwareContextERKNS_7NetworkE+0x61d)[0x7fbec563b79d]
/usr/lib/x86_64-linux-gnu/libnvinfer.so.4(+0x3e19b1)[0x7fbec561a9b1]
./sample_mnist[0x40501a]
./sample_mnist[0x4055f4]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fbebe790830]
./sample_mnist[0x404c09]

======= Memory map: ========
…

Guessing something bad in environmental dependencies. Is there any hint to me what was wrong? or advice?
Thanks a lot!

Charles

2

Hello, can you provide details on the platforms you are using?

GPU type
Driver version

have you tried this using nvidia trt container? https://devblogs.nvidia.com/tensorrt-container/

3

I meet same problem.My platform include ：
GPU type:GTX1070 ;
NVRM version: NVIDIA UNIX x86_64 Kernel Module 384.130 Wed Mar 21 03:37:26 PDT 2018
GCC version: gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10)
cuda 8.0
have you solved it？I holp you will not hesitate to give me some advices.Thanks.

4

Hi Moderator,

I was using Geforce 1080Ti card, and installed driver from PPA under Ubuntu 16.04:
sudo add-apt-repository ppa:graphics-drivers/ppa
The driver version it showed was 390.xx.

However, after I installed this driver and went on installing CUDA toolkit 8.0, seemed it included GPU driver by itself and automatically installed another driver during the installation as I remembered.

And then I installed cuDNN 5.1 for my OpenPose project dependency, and then cuDNN 7.2 for TensorRT 4 dependency.

I was using the way of .deb installation for both CUDA and cuDNN.

I’m not sure where could be the fault, plz help to give some advice.

5

can you try NVIDIA Tensorrt container? https://devblogs.nvidia.com/tensorrt-container/

It is a self-contained environment that has all the necessary libraries and dependencies. It comes with the examples.