Hello,
I believe similar questions regarding engine plan file incompatibility got ask several times now for this now, but I am currently stuck on trying to run the Driveworks TensorRT samples on my host machine. After installing Driveworks with the SDKManager and compiling the samples, I am getting the following error:
kelvinchan@Quadro:/usr/local/driveworks/build/src/clearsightnet$ ./sample_clearsightnet
[14-02-2020 10:02:52] Platform: Detected Generic x86 Platform
[14-02-2020 10:02:52] TimeSource: monotonic epoch time offset is 1581535439511815
[14-02-2020 10:02:52] Platform: number of GPU devices detected 1
[14-02-2020 10:02:52] Platform: currently selected GPU device discrete ID 0
[14-02-2020 10:02:52] SDK: Resources mounted from /usr/local/driveworks-2.2/samples/../data/
[14-02-2020 10:02:53] TimeSource: monotonic epoch time offset is 1581535439511815
[14-02-2020 10:02:53] Initialize DriveWorks SDK v2.2.3136
[14-02-2020 10:02:53] Release build with GNU 7.4.0 from heads/buildbrain-branch-0-gca7b4b26e65
[14-02-2020 10:02:53] Initialize DriveWorks VisualizationSDK v2.2.3136
[14-02-2020 10:02:53] Initialize DriveWorksGL SDK v2.2.3136
[14-02-2020 10:02:53] GL-SDK: initialize OpenGL
[14-02-2020 10:02:53] ClearSightNet: Running in FP32 mode.
[14-02-2020 10:02:53] FileHeader: file header created with development build (patch = -1), assumingcompatibility
[14-02-2020 10:02:53] FileHeader: file header created with development build (patch = -1), assumingcompatibility
[14-02-2020 10:02:53] The engine plan file is generated on an incompatible device, expecting compute 6.0 got compute 6.1, please rebuild.
Segmentation fault
My GPU is a Quadro GP100 (PASCAL-based) with compute 6.0, with CUDA 10.2 and the latest NVIDIA driver installed:
kelvinchan@Quadro:/usr/local/driveworks/build/src/clearsightnet$ nvidia-smi
Fri Feb 14 10:14:41 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.59 Driver Version: 440.59 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Quadro GP100 Off | 00000000:9E:00.0 On | Off |
| 26% 38C P0 26W / 235W | 665MiB / 16277MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 2778 G /usr/lib/xorg/Xorg 222MiB |
| 0 3234 G /usr/bin/gnome-shell 159MiB |
| 0 15309 G ...AAAAAAAAAAAAAAgAAAAAAAAA --shared-files 167MiB |
| 0 25167 G gnome-control-center 4MiB |
| 0 28410 G ...-token=45A98CE7B8E9134A616CD9E34C2AB07D 106MiB |
+-----------------------------------------------------------------------------+
WARNING: infoROM is corrupted at gpu 0000:9E:00.0
Here is my installation:
kelvinchan@Quadro:/usr/src/tensorrt$ sudo dpkg -l | grep tensorrt
[sudo] password for kelvinchan:
ii nv-tensorrt-repo-ubuntu1804-cuda10.2-trt5.1.4.2-ga-20190506 1-1 amd64 nv-tensorrt repository configuration files
ii nv-tensorrt-repo-ubuntu1804-cuda10.2-trt7.0.0.11-ga-20191216 1-1 amd64 nv-tensorrt repository configuration files
ii tensorrt 7.0.0.11-1+cuda10.2 amd64 Meta package of TensorRT
kelvinchan@Quadro:/usr/src/tensorrt/bin$ sudo dpkg -l | grep nvinfer
ii libnvinfer-bin 7.0.0-1+cuda10.2 amd64 TensorRT binaries
ii libnvinfer-dev 7.0.0-1+cuda10.2 amd64 TensorRT development libraries and headers
ii libnvinfer-doc 7.0.0-1+cuda10.2 all TensorRT documentation
ii libnvinfer-plugin-dev 7.0.0-1+cuda10.2 amd64 TensorRT plugin libraries
ii libnvinfer-plugin7 7.0.0-1+cuda10.2 amd64 TensorRT plugin libraries
ii libnvinfer-samples 7.0.0-1+cuda10.2 all TensorRT samples
ii libnvinfer5 5.1.4-1+cuda10.2 amd64 TensorRT runtime libraries
ii libnvinfer7 7.0.0-1+cuda10.2 amd64 TensorRT runtime libraries
ii python-libnvinfer 7.0.0-1+cuda10.2 amd64 Python bindings for TensorRT
ii python-libnvinfer-dev 7.0.0-1+cuda10.2 amd64 Python development package for TensorRT
ii python3-libnvinfer 7.0.0-1+cuda10.2 amd64 Python 3 bindings for TensorRT
ii python3-libnvinfer-dev 7.0.0-1+cuda10.2 amd64 Python 3 development package for TensorRT
kelvinchan@Quadro:/usr/src/tensorrt$ nm -D /usr/lib/x86_64-linux-gnu/libnvinfer.so | grep tensorrt_version
000000000ecd24fc B tensorrt_version_7_0_0_11
kelvinchan@Quadro:/usr/src/tensorrt/bin$ sudo dpkg -l | grep cudnn
ii libcudnn7 7.6.5.32-1+cuda10.2 amd64 cuDNN runtime libraries
ii libcudnn7-dev 7.6.5.32-1+cuda10.2 amd64 cuDNN development libraries and headers
ii libcudnn7-doc 7.6.5.32-1+cuda10.2 amd64 cuDNN documents and samples
kelvinchan@Quadro:/usr/src/tensorrt/bin$ sudo dpkg -l | grep cuda-.*-10-2
ii cuda-command-line-tools-10-2 10.2.19-1 amd64 CUDA command-line tools
ii cuda-compat-10-2 430.17-1 amd64 CUDA Compatibility Platform
ii cuda-compiler-10-2 10.2.19-1 amd64 CUDA compiler
ii cuda-cross-aarch64-10-2 10.2.19-1 all CUDA 10.2 cross-platform meta-package
ii cuda-cudart-10-2 10.2.19-1 amd64 CUDA Runtime native Libraries
ii cuda-cudart-cross-aarch64-10-2 10.2.19-1 all CUDA Runtime cross-aarch64 dev links, headers
ii cuda-cudart-dev-10-2 10.2.19-1 amd64 CUDA Runtime native dev links, headers
ii cuda-cufft-10-2 10.2.19-1 amd64 CUFFT native runtime libraries
ii cuda-cufft-cross-aarch64-10-2 10.2.19-1 all CUFFT cross-aarch64 dev links, headers
ii cuda-cufft-dev-10-2 10.2.19-1 amd64 CUFFT native dev links, headers
ii cuda-cuobjdump-10-2 10.2.19-1 amd64 CUDA cuobjdump
ii cuda-cupti-10-2 10.2.19-1 amd64 CUDA profiling tools runtime libs.
ii cuda-cupti-cross-aarch64-10-2 10.2.19-1 all CUDA cross-aarch64 profiling tools interface.
ii cuda-cupti-dev-10-2 10.2.19-1 amd64 CUDA profiling tools interface.
ii cuda-curand-10-2 10.2.19-1 amd64 CURAND native runtime libraries
ii cuda-curand-cross-aarch64-10-2 10.2.19-1 all CURAND cross-aarch64 dev links, headers
ii cuda-curand-dev-10-2 10.2.19-1 amd64 CURAND native dev links, headers
ii cuda-cusolver-10-2 10.2.19-1 amd64 CUDA solver native runtime libraries
ii cuda-cusolver-cross-aarch64-10-2 10.2.19-1 all CUSOLVER cross-aarch64 dev links, headers
ii cuda-cusolver-dev-10-2 10.2.19-1 amd64 CUDA solver native dev links, headers
ii cuda-cusparse-10-2 10.2.19-1 amd64 CUSPARSE native runtime libraries
ii cuda-cusparse-cross-aarch64-10-2 10.2.19-1 all CUSPARSE cross-aarch64 dev links, headers
ii cuda-cusparse-dev-10-2 10.2.19-1 amd64 CUSPARSE native dev links, headers
ii cuda-documentation-10-2 10.2.19-1 amd64 CUDA documentation
ii cuda-driver-cross-aarch64-10-2 10.2.19-1 all CUDA Driver cross-aarch64 dev links, headers
ii cuda-driver-dev-10-2 10.2.19-1 amd64 CUDA Driver native dev stub library
ii cuda-gdb-10-2 10.2.19-1 amd64 CUDA-GDB
ii cuda-gpu-library-advisor-10-2 10.2.19-1 amd64 CUDA GPU Library Advisor.
ii cuda-libraries-dev-10-2 10.2.19-1 amd64 CUDA Libraries 10.2 development meta-package
ii cuda-license-10-2 10.2.19-1 amd64 CUDA licenses
ii cuda-memcheck-10-2 10.2.19-1 amd64 CUDA-MEMCHECK
ii cuda-misc-headers-10-2 10.2.19-1 amd64 CUDA miscellaneous headers
ii cuda-misc-headers-cross-aarch64-10-2 10.2.19-1 all CUDA headers for cross-aarch64
ii cuda-npp-10-2 10.2.19-1 amd64 NPP native runtime libraries
ii cuda-npp-cross-aarch64-10-2 10.2.19-1 all NPP cross-aarch64 dev links, headers
ii cuda-npp-dev-10-2 10.2.19-1 amd64 NPP native dev links, headers
ii cuda-nsight-10-2 10.2.19-1 amd64 CUDA nsight
ii cuda-nsight-compute-10-2 10.2.19-1 amd64 NVIDIA Nsight Compute
ii cuda-nsight-compute-addon-l4t-10-2 10.2.19-1 all NVIDIA Nsight Compute Addon
ii cuda-nsight-systems-10-2 10.2.19-1 amd64 NVIDIA Nsight Systems
ii cuda-nvcc-10-2 10.2.19-1 amd64 CUDA nvcc
ii cuda-nvdisasm-10-2 10.2.19-1 amd64 CUDA disassembler
ii cuda-nvgraph-10-2 10.2.19-1 amd64 NVGRAPH native runtime libraries
ii cuda-nvgraph-cross-aarch64-10-2 10.2.19-1 all NVGRAPH cross-aarch64 dev links, headers
ii cuda-nvgraph-dev-10-2 10.2.19-1 amd64 NVGRAPH native dev links, headers
ii cuda-nvjpeg-10-2 10.2.19-1 amd64 NVJPEG native runtime libraries
ii cuda-nvjpeg-dev-10-2 10.2.19-1 amd64 NVJPEG native dev links, headers
ii cuda-nvml-cross-aarch64-10-2 10.2.19-1 all NVML cross-aarch64 dev links, headers
ii cuda-nvml-dev-10-2 10.2.19-1 amd64 NVML native dev links, headers
ii cuda-nvprof-10-2 10.2.19-1 amd64 CUDA Profiler tools
ii cuda-nvprune-10-2 10.2.19-1 amd64 CUDA nvprune
ii cuda-nvrtc-10-2 10.2.19-1 amd64 NVRTC native runtime libraries
ii cuda-nvrtc-cross-aarch64-10-2 10.2.19-1 all NVRTC cross-aarch64 dev links, headers
ii cuda-nvrtc-dev-10-2 10.2.19-1 amd64 NVRTC native dev links, headers
ii cuda-nvtx-10-2 10.2.19-1 amd64 NVIDIA Tools Extension
ii cuda-nvvp-10-2 10.2.19-1 amd64 CUDA nvvp
ii cuda-repo-cross-aarch64-10-2-local-10.2.19 1.0-1 all cuda repository configuration files
ii cuda-repo-ubuntu1804-10-2-local-10.2.19-430.17 1.0-1 amd64 cuda repository configuration files
ii cuda-samples-10-2 10.2.19-1 amd64 CUDA example applications
ii cuda-sanitizer-api-10-2 10.2.19-1 amd64 CUDA Sanitizer API
ii cuda-toolkit-10-2 10.2.19-1 amd64 CUDA Toolkit 10.2 meta-package
ii cuda-tools-10-2 10.2.19-1 amd64 CUDA Tools meta-package
ii cuda-visual-tools-10-2 10.2.19-1 amd64 CUDA visual tools
When I tried running a TensorRT example, I do see that TensorRT is linked to cuDNN 7.6.4 but 7.5.1 got loaded instead somehow… If that is actually the issue at hand, how can I load the proper cuDNN version instead?
kelvinchan@Quadro:/usr/src/tensorrt/bin$ ./sample_mnist
&&&& RUNNING TensorRT.sample_mnist # ./sample_mnist
[02/14/2020-10:49:06] [I] Building and running a GPU inference engine for MNIST
[02/14/2020-10:49:07] [W] [TRT] TensorRT was linked against cuDNN 7.6.4 but loaded cuDNN 7.5.1
[02/14/2020-10:49:08] [I] [TRT] Detected 1 inputs and 1 output network tensors.
[02/14/2020-10:49:08] [W] [TRT] TensorRT was linked against cuDNN 7.6.4 but loaded cuDNN 7.5.1
[02/14/2020-10:49:08] [W] [TRT] TensorRT was linked against cuDNN 7.6.4 but loaded cuDNN 7.5.1
[02/14/2020-10:49:08] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
[02/14/2020-10:49:08] [I] Input:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@%++@@@@@@@@@
@@@@@@@@@@@@@%=. =@@@@@@@@
@@@@@@@@@@@%- .. %@@@@@@@
@@@@@@@@@@- .%% #@@@@@@@
@@@@@@@@@* .%@% #@@@@@@@
@@@@@@@@@: .-%@@* #@@@@@@@
@@@@@@@@@%=#%@@@@+ #@@@@@@@
@@@@@@@@@@@@@@@@@- #@@@@@@@
@@@@@@@@@@@@@++++: :@@@@@@@@
@@@@@@@@@@@#: .%@@@@@@@@
@@@@@@@@@#. .- +@@@@@@@@
@@@@@@@@* +#@@. *@@@@@@@
@@@@@@@= :@@@: =- :###@@@
@@@@@@= +@@#: .#@@#- .@@@
@@@@@+ *@#- =%@@@@@*::%@@@
@@@@@ .. +@@@@@@@@@@@@@@
@@@@@. :+%@@@@@@@@@@@@@@@
@@@@@@*=+%@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[02/14/2020-10:49:08] [I] Output:
0:
1:
2: **********
3:
4:
5:
6:
7:
8:
9:
&&&& PASSED TensorRT.sample_mnist # ./sample_mnist
If anyone can help me out here, that would be much appreciated =)