Runtime Error with NVIDIA TensorRT (Deep Learning)

Autonomous Machines Jetson & Embedded Systems Jetson TX1
1

Hi,

I would like to report an issue regarding the new NVIDIA TensorRT Application for deep learning.
While testing the piece of code from [1] along caffe models, I’ve encountered a segmentation fault when building the CUDA engine.
Can you tell the nature of the issue ?

[1] https://devblogs.nvidia.com/parallelforall/production-deep-learning-nvidia-gpu-inference-engine/ (listing 1)

My code sample:

nvinfer1::IBuilder* builder = nvinfer1::createInferBuilder(mygLogger);
nvinfer1::INetworkDefinition* network = builder->createNetwork();
nvcaffeparser1::ICaffeParser *parser = nvcaffeparser1::createCaffeParser();
nvcaffeparser1::ICaffeParser& myParser = *parser;
auto blob_name_to_tensor = myParser.parse(archFile.c_str(), weightsFile.c_str(), *network, nvinfer1::DataType::kFLOAT);
network->markOutput(*blob_name_to_tensor->find(nameOfOuputLayer.c_str()));
builder->setMaxBatchSize(1);
builder->setMaxWorkspaceSize(1 << 30);
nvinfer1::ICudaEngine* engine = builder->buildCudaEngine(*network);

Thank you for your time and consideration.

2

Which GPU are you using? Jetson TX1 with JetPack 2.3?

It’s hard to tell from the code snippet without error checking where the crash is, also it’s not certain this is tested on JetPack 2.3.
Here’s another recent example up to date with JetPack 2.3 and runs: [url]https://github.com/dusty-nv/jetson-inference[/url]

3

I haven’t tested it on Jetson TX1 but on a Desktop computer with a Quadro 1000m (cc2.0 so no cudnn) and cuda 7.5. TensorRT is also supposed to work on amd64 right ?

The error I got is a segmentation fault at line 9.

4

decided to try that github tutorial. all was going well until these errors. can not find packages. can not locate a QT 4.x, which had not been mentioned as being needed.

ubuntu@tegra-ubuntu:~$ cd /Documents/jetson-inference
bash: cd: /Documents/jetson-inference: No such file or directory
ubuntu@tegra-ubuntu:~$ cd ~/Documents/jetson-inference
ubuntu@tegra-ubuntu:~/Documents/jetson-inference$ git clone http://github.org/dusty-nv/jetson-inference
Cloning into 'jetson-inference'...
remote: Counting objects: 227, done.
remote: Total 227 (delta 0), reused 0 (delta 0), pack-reused 227
Receiving objects: 100% (227/227), 3.15 MiB | 3.72 MiB/s, done.
Resolving deltas: 100% (100/100), done.
Checking connectivity... done.
ubuntu@tegra-ubuntu:~/Documents/jetson-inference$ ls -l
total 4
drwxrwxr-x 11 ubuntu ubuntu 4096 Sep 17 17:38 jetson-inference
ubuntu@tegra-ubuntu:~/Documents/jetson-inference$ cd jetson-inference
ubuntu@tegra-ubuntu:~/Documents/jetson-inference/jetson-inference$ mkdir build
ubuntu@tegra-ubuntu:~/Documents/jetson-inference/jetson-inference$ cd build
ubuntu@tegra-ubuntu:~/Documents/jetson-inference/jetson-inference/build$ cmake ../
bash: cmake: command not found
ubuntu@tegra-ubuntu:~/Documents/jetson-inference/jetson-inference/build$ cmake --version
bash: cmake: command not found
ubuntu@tegra-ubuntu:~/Documents/jetson-inference/jetson-inference/build$ cmake ../
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
Launching pre-build dependency installer script...
[Pre-build]  dependency installer script running...
[Pre-build]  build root directory:       /home/ubuntu/Documents/jetson-inference/jetson-inference/build
[sudo] password for ubuntu: 
Get:1 file:/var/cuda-repo-8-0-local  InRelease
Ign:1 file:/var/cuda-repo-8-0-local  InRelease
Get:2 file:/var/libopencv4tegra-repo  InRelease
Ign:2 file:/var/libopencv4tegra-repo  InRelease
Get:3 file:/var/nv-gie-repo-6-rc-cuda8.0  InRelease
Ign:3 file:/var/nv-gie-repo-6-rc-cuda8.0  InRelease
Get:4 file:/var/visionworks-repo  InRelease
Ign:4 file:/var/visionworks-repo  InRelease
Get:5 file:/var/visionworks-sfm-repo  InRelease
Ign:5 file:/var/visionworks-sfm-repo  InRelease
Get:6 file:/var/visionworks-tracking-repo  InRelease
Ign:6 file:/var/visionworks-tracking-repo  InRelease
Get:7 file:/var/cuda-repo-8-0-local  Release [574 B]
Get:8 file:/var/libopencv4tegra-repo  Release [347 B]
Get:9 file:/var/nv-gie-repo-6-rc-cuda8.0  Release [574 B]
Get:10 file:/var/visionworks-repo  Release [698 B]
Get:11 file:/var/visionworks-sfm-repo  Release [702 B]
Get:12 file:/var/visionworks-tracking-repo  Release [707 B]
Get:7 file:/var/cuda-repo-8-0-local  Release [574 B]
Get:8 file:/var/libopencv4tegra-repo  Release [347 B]
Get:9 file:/var/nv-gie-repo-6-rc-cuda8.0  Release [574 B]
Get:10 file:/var/visionworks-repo  Release [698 B]
Get:11 file:/var/visionworks-sfm-repo  Release [702 B]
Get:12 file:/var/visionworks-tracking-repo  Release [707 B]
Get:14 http://us.archive.ubuntu.com/ubuntu xenial-updates InRelease [95.7 kB]
Hit:16 http://ports.ubuntu.com/ubuntu-ports xenial InRelease
Get:19 http://ports.ubuntu.com/ubuntu-ports xenial-updates InRelease [95.7 kB]
Get:18 http://us.archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
Get:20 http://us.archive.ubuntu.com/ubuntu xenial-backports InRelease [92.2 kB]
Hit:23 http://ports.ubuntu.com/ubuntu-ports xenial-security InRelease
Get:25 http://us.archive.ubuntu.com/ubuntu xenial-updates/main Sources [188 kB]
Get:26 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe Sources [94.2 kB]
Get:27 http://us.archive.ubuntu.com/ubuntu xenial-updates/multiverse Sources [3,220 B]
Ign:28 http://us.archive.ubuntu.com/ubuntu xenial-updates/main arm64 Packages
Get:29 http://us.archive.ubuntu.com/ubuntu xenial-updates/main Translation-en [147 kB]
Get:30 http://us.archive.ubuntu.com/ubuntu xenial-updates/main arm64 DEP-11 Metadata [300 kB]
Get:31 http://us.archive.ubuntu.com/ubuntu xenial-updates/main DEP-11 64x64 Icons [188 kB]
Ign:32 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe arm64 Packages
Get:33 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe Translation-en [113 kB]
Get:34 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe arm64 DEP-11 Metadata [101 kB]
Get:35 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe DEP-11 64x64 Icons [98.3 kB]
Ign:36 http://us.archive.ubuntu.com/ubuntu xenial-updates/multiverse arm64 Packages
Get:37 http://us.archive.ubuntu.com/ubuntu xenial-updates/multiverse Translation-en [2,428 B]
Get:38 http://us.archive.ubuntu.com/ubuntu xenial-updates/multiverse arm64 DEP-11 Metadata [200 B]
Get:39 http://us.archive.ubuntu.com/ubuntu xenial-updates/multiverse DEP-11 64x64 Icons [2,699 B]
Get:40 http://us.archive.ubuntu.com/ubuntu xenial-updates/restricted arm64 DEP-11 Metadata [157 B]
Get:41 http://us.archive.ubuntu.com/ubuntu xenial/main Sources [868 kB]
Get:42 http://us.archive.ubuntu.com/ubuntu xenial/restricted Sources [4,808 B]
Get:43 http://us.archive.ubuntu.com/ubuntu xenial/universe Sources [7,728 kB]
Get:44 http://us.archive.ubuntu.com/ubuntu xenial/multiverse Sources [179 kB]
Ign:45 http://us.archive.ubuntu.com/ubuntu xenial/main arm64 Packages
Get:46 http://us.archive.ubuntu.com/ubuntu xenial/main Translation-en [568 kB]
Get:47 http://us.archive.ubuntu.com/ubuntu xenial/main arm64 DEP-11 Metadata [720 kB]
Get:48 http://us.archive.ubuntu.com/ubuntu xenial/main DEP-11 64x64 Icons [409 kB]
Get:49 http://us.archive.ubuntu.com/ubuntu xenial/restricted Translation-en [2,908 B]
Get:50 http://us.archive.ubuntu.com/ubuntu xenial/restricted arm64 DEP-11 Metadata [186 B]
Ign:51 http://us.archive.ubuntu.com/ubuntu xenial/universe arm64 Packages
Get:52 http://us.archive.ubuntu.com/ubuntu xenial/universe Translation-en [4,354 kB]
Get:53 http://us.archive.ubuntu.com/ubuntu xenial/universe arm64 DEP-11 Metadata [3,362 kB]
Get:54 http://us.archive.ubuntu.com/ubuntu xenial/universe DEP-11 64x64 Icons [7,448 kB]
Ign:55 http://us.archive.ubuntu.com/ubuntu xenial/multiverse arm64 Packages
Get:56 http://us.archive.ubuntu.com/ubuntu xenial/multiverse Translation-en [106 kB]
Get:57 http://us.archive.ubuntu.com/ubuntu xenial/multiverse arm64 DEP-11 Metadata [58.5 kB]
Get:58 http://us.archive.ubuntu.com/ubuntu xenial/multiverse DEP-11 64x64 Icons [230 kB]
Ign:28 http://us.archive.ubuntu.com/ubuntu xenial-updates/main arm64 Packages
Ign:32 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe arm64 Packages
Ign:36 http://us.archive.ubuntu.com/ubuntu xenial-updates/multiverse arm64 Packages
Ign:59 http://us.archive.ubuntu.com/ubuntu xenial-backports/main arm64 Packages
Get:60 http://us.archive.ubuntu.com/ubuntu xenial-backports/main Translation-en [1,484 B]
Get:61 http://us.archive.ubuntu.com/ubuntu xenial-backports/main arm64 DEP-11 Metadata [197 B]
Get:62 http://us.archive.ubuntu.com/ubuntu xenial-backports/restricted arm64 DEP-11 Metadata [194 B]
Ign:63 http://us.archive.ubuntu.com/ubuntu xenial-backports/universe arm64 Packages
Get:64 http://us.archive.ubuntu.com/ubuntu xenial-backports/universe Translation-en [584 B]
Get:65 http://us.archive.ubuntu.com/ubuntu xenial-backports/universe arm64 DEP-11 Metadata [201 B]
Get:66 http://us.archive.ubuntu.com/ubuntu xenial-backports/multiverse arm64 DEP-11 Metadata [194 B]
Ign:45 http://us.archive.ubuntu.com/ubuntu xenial/main arm64 Packages
Ign:51 http://us.archive.ubuntu.com/ubuntu xenial/universe arm64 Packages
Ign:55 http://us.archive.ubuntu.com/ubuntu xenial/multiverse arm64 Packages
Ign:28 http://us.archive.ubuntu.com/ubuntu xenial-updates/main arm64 Packages
Ign:32 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe arm64 Packages
Ign:36 http://us.archive.ubuntu.com/ubuntu xenial-updates/multiverse arm64 Packages
Ign:59 http://us.archive.ubuntu.com/ubuntu xenial-backports/main arm64 Packages
Ign:63 http://us.archive.ubuntu.com/ubuntu xenial-backports/universe arm64 Packages
Ign:45 http://us.archive.ubuntu.com/ubuntu xenial/main arm64 Packages
Ign:51 http://us.archive.ubuntu.com/ubuntu xenial/universe arm64 Packages
Ign:55 http://us.archive.ubuntu.com/ubuntu xenial/multiverse arm64 Packages
Err:28 http://us.archive.ubuntu.com/ubuntu xenial-updates/main arm64 Packages
  404  Not Found [IP: 91.189.91.26 80]
Ign:32 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe arm64 Packages
Ign:36 http://us.archive.ubuntu.com/ubuntu xenial-updates/multiverse arm64 Packages
Ign:59 http://us.archive.ubuntu.com/ubuntu xenial-backports/main arm64 Packages
Ign:63 http://us.archive.ubuntu.com/ubuntu xenial-backports/universe arm64 Packages
Err:45 http://us.archive.ubuntu.com/ubuntu xenial/main arm64 Packages
  404  Not Found [IP: 91.189.91.26 80]
Ign:51 http://us.archive.ubuntu.com/ubuntu xenial/universe arm64 Packages
Ign:55 http://us.archive.ubuntu.com/ubuntu xenial/multiverse arm64 Packages
Err:59 http://us.archive.ubuntu.com/ubuntu xenial-backports/main arm64 Packages
  404  Not Found [IP: 91.189.91.26 80]
Ign:63 http://us.archive.ubuntu.com/ubuntu xenial-backports/universe arm64 Packages
Fetched 530 kB in 3s (133 kB/s)
Reading package lists...
W: file:///var/cuda-repo-8-0-local/Release.gpg: Signature by key 889BEE522DA690103C4B085ED88C3D385C37D3BE uses weak digest algorithm (SHA1)
W: Invalid 'Date' entry in Release file /var/lib/apt/lists/_var_libopencv4tegra-repo_Release
W: Invalid 'Date' entry in Release file /var/lib/apt/lists/_var_visionworks-repo_Release
W: Invalid 'Date' entry in Release file /var/lib/apt/lists/_var_visionworks-sfm-repo_Release
W: Invalid 'Date' entry in Release file /var/lib/apt/lists/_var_visionworks-tracking-repo_Release
E: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/xenial-updates/main/binary-arm64/Packages  404  Not Found [IP: 91.189.91.26 80]
E: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/xenial/main/binary-arm64/Packages  404  Not Found [IP: 91.189.91.26 80]
E: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/xenial-backports/main/binary-arm64/Packages  404  Not Found [IP: 91.189.91.26 80]
E: Some index files failed to download. They have been ignored, or old ones used instead.
Finished installing dependencies
qmake: could not exec '/usr/lib/aarch64-linux-gnu/qt4/bin/qmake': No such file or directory
CMake Error at /usr/share/cmake-3.5/Modules/FindQt4.cmake:1326 (message):
  Found unsuitable Qt version "" from NOTFOUND, this code requires Qt 4.x
Call Stack (most recent call first):
  CMakeLists.txt:27 (find_package)


-- Configuring incomplete, errors occurred!
5

@dkryder Just install qt4 and qmake and see what happens.

6

It also looks like it references cmake…try installing that as well.

7

All you should need to have installed beforehand is JetPack 2.3, git, and cmake. Then the CMakePreBuild.sh automatically installs the dependencies - Qt is used for loading images from disk for the console programs (otherwise the UI is OpenGL). It looks like what happens is the script failed here:

Ign:63 http://us.archive.ubuntu.com/ubuntu xenial-backports/universe arm64 Packages
Fetched 530 kB in 3s (133 kB/s)
Reading package lists...
W: file:///var/cuda-repo-8-0-local/Release.gpg: Signature by key 889BEE522DA690103C4B085ED88C3D385C37D3BE uses weak digest algorithm (SHA1)
W: Invalid 'Date' entry in Release file /var/lib/apt/lists/_var_libopencv4tegra-repo_Release
W: Invalid 'Date' entry in Release file /var/lib/apt/lists/_var_visionworks-repo_Release
W: Invalid 'Date' entry in Release file /var/lib/apt/lists/_var_visionworks-sfm-repo_Release
W: Invalid 'Date' entry in Release file /var/lib/apt/lists/_var_visionworks-tracking-repo_Release
E: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/xenial-updates/main/binary-arm64/Packages  404  Not Found [IP: 91.189.91.26 80]
E: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/xenial/main/binary-arm64/Packages  404  Not Found [IP: 91.189.91.26 80]
E: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/xenial-backports/main/binary-arm64/Packages  404  Not Found [IP: 91.189.91.26 80]

Perhaps it was a temporary internet connection issue? If you run this command manually, do you still see the errors during apt-get?:

sudo apt-get install -y libqt4-dev qt4-dev-tools libglew-dev glew-utils libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libglib2.0-dev
8

@dusty_nv What about my issue ? Do you have any idea ?

9

The repo assumes it’s being using from TX1 that’s been flashed with JetPack 2.3 (i.e. TensorRT has already been installed on the TX1).
Also I believe cuDNN is a dependency of TensorRT.

Are you trying to run the imagenet-console program or the imagenet-camera program? imagenet-camera is currently configured to use Jetson onboard camera with gstreamer, however v4l2Camera class is also provided and will be integrated transparently in the future with gstreamer version.

10

Not exclusively for TX1. The repo has a amd64 version. I think it is only compatible with high-end desktop GPUs such as the p4, p40 and titan X Pascal. Hopefully it will work on my GTX 1070.

11

Yes, desktop version of TensorRT is available for CUDA GPU supporting cuDNN. The latest GPUs allow FP16 mode to be enabled in TensorRT.

The GitHub deep learning repo is mostly developed & tested on TX1, but you may be able to compile and run the console programs for desktop. The camera programs use TX1 onboard camera currently, although I’m adding to be able to switch seamlessly between USB webcam (fine for desktop).

12

ok, first…thank you to everyone for your tips, comments , and suggestions. i think i have for the most part completed the task. however before i start running the samples i’m looking at the file readme.md in the jetson-inference folder. it mentions 2 files to be found in jetson-inference/build/aarch64/bin 1…/imagenet-camera googlenet & 2…/imagenet-camera alexnet depending on if you want googlenet or alexnet. thing is neither of these files are in /bin folder. what is there are binary files 1.bvlc_googlenet.caffemodel & bvlc_alexnet.caffemodel
so have things changed with the new build and this document has yet to be updated or do i have to go back to start and figure out why i don’t have either imaganet files? thanks for any insights.

13

Since the pre-trained network models for googlenet and alexnet are too big to store in GitHub, I have them downloaded from an external site during the CMake configuration. The script must have exited early for you. Here are the commands from the script that download the networks:

wget http://dl.caffe.berkeleyvision.org/bvlc_alexnet.caffemodel
mv bvlc_alexnet.caffemodel ../data/networks

wget http://dl.caffe.berkeleyvision.org/bvlc_googlenet.caffemodel
mv bvlc_googlenet.caffemodel ../data/networks

Run these commands from within your jetson-inference/build directory, and then run cmake ../ again, and then the networks should be properly copied over to build/aarch64/bin.

14

i have not been able to connect to that address. would you confirm that you can access it please?

15

I don’t know about the individual files in those URLs, but I’m able to reach the base URL. What is the exact message when trying to get those files? Also, does your Jetson respond as finding berkeleyvision.org with:

host berkeleyvision.org
16

Those files seemed to be not working this morning, but are back to working when I just tried again.

17

Let’s get back to the initial error I’ve got:
I have retested the code on the Jetson TX1 with gdb and the following error showed up:

Program received signal SIGSEGV, Segmentation fault.
0x0000007fb677d694 in nvinfer1::cudnn::validateConfig(nvinfer1::Network const&, nvinfer1::CudaEngineBuildContext const&) () from /usr/lib/aarch64-linux-gnu/libnvinfer.so.1

Since it is a bit difficult to deeply debug TensorRT with gdb because of the non-availability of the debugging symbols so far, I couldn’t identify where the issue came from.

Any thoughts about this ?

Thank you all for your support.

18

I know it has been a while but is this still an issue?

19

Yes the issue still exists. Following tutorial with my Jetson TX2 and are not blocked at this issue. Please help! - GitHub - dusty-nv/jetson-inference: Hello AI World guide to deploying deep-learning inference networks and deep vision primitives with TensorRT and NVIDIA Jetson.

20

Hi,

Do you try TensorRT2.1?