SIGABORT for default samples

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

I have confronted such problems when I try to test default samples after installing tensorrt

./sample_int8 mnist

FP32 run:400 batches of size 100 starting at 100
*** Error in `./sample_int8': free(): invalid next size (fast): 0x00007ff0b96aa3f0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7ff0eb3f67e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7ff0eb3ff37a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7ff0eb40353c]
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.6(cudnnDestroyConvolutionDescriptor+0x9)[0x7ff0ec66ac69]
/usr/lib/x86_64-linux-gnu/libnvinfer.so.4(_ZN8nvinfer15cudnn22CudnnConvolutionTraits15getValidTacticsERKNS0_21CudnnConvolutionLayerERKNS0_18EngineBuildContextE+0x1cc)[0x7ff0f655be2c]
/usr/lib/x86_64-linux-gnu/libnvinfer.so.4(_ZN8nvinfer17builder15getValidTacticsERKNS_5cudnn5LayerERKNS1_18EngineBuildContextE+0x155)[0x7ff0f651e135]
/usr/lib/x86_64-linux-gnu/libnvinfer.so.4(_ZN8nvinfer15cudnn33selectFastestLayerAndDeleteOthersERNS0_18EngineBuildContextERKSt6vectorIPNS0_5LayerESaIS5_EE+0x82)[0x7ff0f650e7f2]
/usr/lib/x86_64-linux-gnu/libnvinfer.so.4(_ZN8nvinfer17builder16buildSingleLayerERNS_5cudnn18EngineBuildContextERNS0_4NodeERKSt13unordered_mapISsSt10unique_ptrINS1_6RegionESt14default_deleteIS8_EESt4hashISsESt8equal_toISsESaISt4pairIKSsSB_EEERNS_14CpuMemoryGroupEPS6_ISsSt6vectorIfSaIfEESD_SF_SaISG_ISH_SR_EEEb+0x130f)[0x7ff0f653752f]
/usr/lib/x86_64-linux-gnu/libnvinfer.so.4(_ZN8nvinfer17builder18EngineTacticSupply13getBestTacticERNS0_4NodeERKNS_5query5PortsINS_13RegionFormatLEEEb+0x349)[0x7ff0f6538509]
/usr/lib/x86_64-linux-gnu/libnvinfer.so.4(+0x3f013d)[0x7ff0f652b13d]
/usr/lib/x86_64-linux-gnu/libnvinfer.so.4(_ZN8nvinfer17builder23chooseFormatsAndTacticsERNS0_5GraphERNS0_12TacticSupplyEPSt13unordered_mapISsSt6vectorIfSaIfEESt4hashISsESt8equal_toISsESaISt4pairIKSsS8_EEE+0x1339)[0x7ff0f652e879]
/usr/lib/x86_64-linux-gnu/libnvinfer.so.4(_ZN8nvinfer17builder19makeEngineFromGraphERKNS_21CudaEngineBuildConfigERKNS_5cudnn15HardwareContextERNS0_5GraphEPSt13unordered_mapISsSt6vectorIfSaIfEESt4hashISsESt8equal_toISsESaISt4pairIKSsSD_EEEi+0x2a9)[0x7ff0f65398a9]
/usr/lib/x86_64-linux-gnu/libnvinfer.so.4(_ZN8nvinfer17builder11buildEngineERNS_21CudaEngineBuildConfigERKNS_5cudnn15HardwareContextERKNS_7NetworkE+0x61d)[0x7ff0f653d79d]
/usr/lib/x86_64-linux-gnu/libnvinfer.so.4(+0x3e19b1)[0x7ff0f651c9b1]
./sample_int8[0x405f55]
./sample_int8[0x4069e8]
./sample_int8[0x4079e9]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7ff0eb39f830]

similar outputs happened to other examples

./sample_mnist

etc. as well. Anyone knows why this happens?
PS: I am using 2 1080Ti cards, and my libraries versions are

TensorRT: 4.0.1.6
CUDA: 8.0.61
CUDNN: 6.0.21
Driver: 384.130
2

Hello,

to rule out any installation/configuration issues on your system, I’d suggest you try the TensorRT container (latest nvcr.io/nvidia/tensorrt:18.11-py3). Please check out NVIDIA GPU Cloud: https://www.nvidia.com/en-us/gpu-cloud/ Account is free.