Debug TensorRT loading correctly?

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Ubuntu 18.04
NVIDIA RTX 2080Ti

I have been struggling to get a basic implementation of TensorRT to work for nearly two weeks. I have tried at least 5 different tensorflow versions, two computers, multiple re-installs, etc.

I’ve reduced my current goal to just executing this test notebook and being able to produced the optimized graph with trt_engine_nodes. Tensorflow-TensorRT/1_convert_TF_to_TRT.ipynb at master · ardianumam/Tensorflow-TensorRT · GitHub
I have tried installing and re-installing and at this point I don’t have any clue where things are going wrong.

When I run the above notebook, everything runs without crashing, but I get the following logs to stdout:

INFO:tensorflow:Running against TensorRT version 0.0.0

That seems to indicate TensorRT is not loading correctly?

Moreover, despite the code completing without crashing, the resulting graph does not actually have any trt_engine_nodes. I get the following logs at the end:

numb. of all_nodes in frozen graph: 46
numb. of trt_engine_nodes in TensorRT graph: 0
numb. of all_nodes in TensorRT graph: 36

According to the notebook in the repository, this graph should have 2 trt_engine_nodes and 17 total (you can scroll to the bottom of the link I gave earlier)

How to I begin to debug where this is going wrong?

When I run grep dpkg for TensorRT, I get:

$ dpkg -l | grep TensorRT
ii  libnvinfer-dev                                  5.1.5-1+cuda10.0                             amd64        TensorRT development libraries and headers
ii  libnvinfer5                                     5.1.5-1+cuda10.0                             amd64        TensorRT runtime libraries

My $LD_LIBRARY_PATH is “/space/tensorrt/TensorRT-5.1.5.0/lib:/usr/local/cuda/extras/CUPTI/lib64”

Here is the output of pip freeze for the virtualenv:

$ pip freeze
absl-py==0.7.1
appnope==0.1.0
astor==0.8.0
attrs==19.1.0
backcall==0.1.0
bleach==3.1.0
cycler==0.10.0
decorator==4.4.0
defusedxml==0.6.0
entrypoints==0.3
gast==0.2.2
graphsurgeon==0.4.1
grpcio==1.22.0
h5py==2.9.0
ipykernel==5.1.2
ipython==7.7.0
ipython-genutils==0.2.0
ipywidgets==7.5.1
jedi==0.15.1
Jinja2==2.10.1
jsonschema==3.0.2
jupyter==1.0.0
jupyter-client==5.3.1
jupyter-console==6.0.0
jupyter-core==4.5.0
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
kiwisolver==1.1.0
Markdown==3.1.1
MarkupSafe==1.1.1
matplotlib==3.1.1
mistune==0.8.4
nbconvert==5.6.0
nbformat==4.4.0
notebook==6.0.0
numpy==1.17.0
opencv-contrib-python==4.1.0.25
pandocfilters==1.4.2
parso==0.5.1
pexpect==4.7.0
pickleshare==0.7.5
Pillow==6.1.0
prometheus-client==0.7.1
prompt-toolkit==2.0.9
protobuf==3.9.1
ptyprocess==0.6.0
Pygments==2.4.2
pyparsing==2.4.2
pyrsistent==0.15.4
python-dateutil==2.8.0
pyzmq==18.1.0
qtconsole==4.5.2
Send2Trash==1.5.0
six==1.12.0
tensorboard==1.12.2
tensorflow==1.12.0
tensorrt==5.1.5.0
termcolor==1.1.0
terminado==0.8.2
testpath==0.4.2
tornado==6.0.3
traitlets==4.3.2
uff==0.6.3
wcwidth==0.1.7
webencodings==0.5.1
Werkzeug==0.15.5
widgetsnbextension==3.5.1
2

Good timing! We’re having almost the exact same issue!

3

We’re having the same issue here with Tensorflow 1.14.0 and TensorRT-5.1.5.0.

Our current workaround is to use Tensorflow 1.13.1 with TensorRT-5.1.5.0 (or TF 1.12.0 with TRT 4.0.1.6 but only available in Ubuntu) but we would love a way to make TRT optimization work smoothly with Tensorflow 1.14.0 … Any update?

Thanks!!

4

Hey sarae5q98, I was only able to get it to work by using the provided NVIDIA TensorRT docker.

Try adapting this dockerfile:

FROM nvcr.io/nvidia/tensorflow:19.08-py3

# Create workspace
ARG WORKSPACE=/path/to/workspace
RUN mkdir -p ${WORKSPACE}

# NVIDIA docker is missing some cv2 requirements
# Solution from https://github.com/NVIDIA/nvidia-docker/issues/864
RUN ["apt-get", "update"]
RUN ["apt-get", "install", "-y", "libsm6", "libxext6", "libxrender-dev"]

# Upgrade underlying pip
RUN pip install --upgrade pip setuptools

# Add requirements.txt before rest of repo for caching (doesn't seem to be working well - TODO revisit)
ADD requirements.txt ${WORKSPACE}
WORKDIR ${WORKSPACE}
RUN pip install -r requirements.txt

# Add repo
#ADD . ${WORKSPACE}

And run it with:

docker run \
  --gpus all \
  --shm-size=1g \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  --rm \
  -p 8888:8888 \
  -it \
  docker-name "$@"

Using that, I was able to get the following example code to worK: GitHub - ardianumam/Tensorflow-TensorRT: This repository is for my YT video series about optimizing a Tensorflow deep learning model using TensorRT. We demonstrate optimizing LeNet-like model and YOLOv3 model, and get 3.7x and 1.5x faster for the former and the latter, respectively, compared to the original models.

However, I have not been able to get the more modern trt.TrtGraphConverter to work.

Overall, I don’t really consider this an answer as I’m forced to use docker and I can still only get 1.13-style syntax to work.

5

Thanks @corygabr!
Unfortunately I can’t use docker for this application… but good to know we’re not the only ones not getting the newer trt.TrtGraphConverter to work!