JetPack 4.4.1 - Tensorflow with TensorRT issue

Autonomous Machines Jetson & Embedded Systems Jetson Nano
,
1

Hi All,

i recently bought a Jetson Nano and i 'd like to leverage its GPU features learning to use Tensorflow + TensorRT for a basis image classification Python model (i’m a begineer on all those stuff so i apologize in advance if i say nonsense! :) ).

I download the sd card image from official Nvidia page (Jet 4.4.1 for JetsonNano 4Gb)

I prepared a Dockerfile with the following instructions:

FROM nvcr.io/nvidia/l4t-base:r32.4.4

RUN apt-get update -y && DEBIAN_FRONTEND=noninteractive apt-get install -y gcc curl wget build-essential cmake \
pkg-config git zlib1g-dev libffi-dev libssl-dev ca-certificates apt-transport-https apt-utils unzip libjpeg-dev \
libfreetype6-dev libxml2 libopenjp2-7 libatlas-base-dev libgl1-mesa-glx libgtk2.0-0

RUN apt-get install -y python3-pip

========================================

Tensorflow installation

========================================

RUN apt-get install -y libhdf5-serial-dev hdf5-tools libhdf5-dev zlib1g-dev zip libjpeg8-dev liblapack-dev libblas-dev gfortran

RUN python3 -m pip install testresources setuptools==49.6.0 cython==0.29 numpy==1.16.1 future==0.18.2 mock==3.0.5
keras_preprocessing==1.1.1 futures protobuf pybind11
RUN python3 -m pip install h5py==2.10.0 keras_applications==1.0.8 termcolor==1.1.0 absl-py==0.7.0 six==1.12.0 \
grpcio==1.8.6 astunparse==1.6.3 wrapt==1.11.1 google-pasta==0.1.8 gast==0.3.3 opt-einsum==2.3.2 tensorboard==2.3.0
tensorflow-estimator==2.3.0

RUN pip3 install --pre --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v44 tensorflow

Tensorflow seems to work:
sudo docker run -it --rm --net=host --runtime=nvidia 88128a177626 bash

root@jetson-desktop:/# python3 -c “import tensorflow as tf; print(tf.test.is_gpu_available())”
2021-01-15 12:28:53.120944: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
WARNING: Logging before flag parsing goes to stderr.
W0115 12:28:59.815237 548049068048 deprecation.py:323] From :1: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.config.list_physical_devices('GPU') instead.
2021-01-15 12:28:59.847165: W tensorflow/core/platform/profile_utils/cpu_utils.cc:108] Failed to find bogomips or clock in /proc/cpuinfo; cannot determine CPU frequency
2021-01-15 12:28:59.848116: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2645b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-01-15 12:28:59.848220: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2021-01-15 12:28:59.884836: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-01-15 12:29:00.083276: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero
2021-01-15 12:29:00.083669: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x139bef0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-01-15 12:29:00.083737: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3
2021-01-15 12:29:00.109162: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero
2021-01-15 12:29:00.109384: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1742] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.86GiB deviceMemoryBandwidth: 194.55MiB/s
2021-01-15 12:29:00.109520: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
2021-01-15 12:29:00.170782: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10
2021-01-15 12:29:00.230702: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-01-15 12:29:00.281430: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-01-15 12:29:00.399895: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-01-15 12:29:00.445063: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10
2021-01-15 12:29:00.457161: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-01-15 12:29:00.457467: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero
2021-01-15 12:29:00.457771: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero
2021-01-15 12:29:00.457911: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1884] Adding visible gpu devices: 0
2021-01-15 12:29:00.483444: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
2021-01-15 12:29:05.481419: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1283] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-01-15 12:29:05.481581: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1289] 0
2021-01-15 12:29:05.481638: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1302] 0: N
2021-01-15 12:29:05.498906: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero
2021-01-15 12:29:05.499234: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero
2021-01-15 12:29:05.499502: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1428] Created TensorFlow device (/device:GPU:0 with 244 MB memory) → physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
True

TensorRT seems to work:
root@jetson-desktop:/# python3 -c “import tensorrt as tf; print(tf.version)”
7.1.3.0

So i decided to try an interesting face recognition model (i cant push the link because i crossed the threshold of 3 links!), but i noticed that i got an issue on graphsurgeon module:

root@jetson-desktop:/# python3 -c ‘import graphsurgeon as gs; print(gs.version)’
2021-01-15 17:03:27.224315: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
Traceback (most recent call last):
File “”, line 1, in
File “/usr/lib/python3.6/dist-packages/graphsurgeon/init.py”, line 58, in
from graphsurgeon.node_manipulation import *
File “/usr/lib/python3.6/dist-packages/graphsurgeon/node_manipulation.py”, line 42
node.attr[“dtype”].type = 1
^
IndentationError: unexpected indent

My understanding is that the grapth surgeon should be a module of tensorRT that it’s a default module on the base docker image that i’m using → nvcr.io/nvidia/l4t-base:r32.4.4

What am i doing wrong?
Could you please help me?

Thanks & Regards,
Dario

1 Like
3

Hi,

Here is a container with TensorFlow pre-installed.
You can use it without manual installation next time.

For our L4T image, it mounts the TensorRT library from the host directly.
So the TensorRT package is actually installed on the Nano rather than the image.

We have tested the functionality of graphsurgeon with a slightly modified command.
It can work correctly without issue. Would you mind trying it again?

$ python3 -c 'import graphsurgeon as gs; print(gs.__version__)'
2021-01-18 10:45:53.801160: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
0.4.5

Thanks.

4

Hi,

thanks for your reply.

I tried to run that command on my Jetson OS (downloaded image from https://developer.nvidia.com/jetson-nano-sd-card-image).

Unfortunately it seems that graphsurgeon module it is part of Tensorflow (not Tensor RT):

jetson@jetson-desktop:~/tf-opencv-nvidia$ python3 -c ‘import tensorrt as tf; print(tf.version)’
7.1.3.0
jetson@jetson-desktop:~/tf-opencv-nvidia$ python3 -c ‘import graphsurgeon as gs; print(gs.version)’
Traceback (most recent call last):
File “/usr/lib/python3.6/dist-packages/graphsurgeon/init.py”, line 49, in
import tensorflow as tf
ModuleNotFoundError: No module named ‘tensorflow’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “”, line 1, in
File “/usr/lib/python3.6/dist-packages/graphsurgeon/init.py”, line 54, in
“”".format(err))
ImportError: ERROR: Failed to import module (No module named ‘tensorflow’)
Please make sure you have TensorFlow installed.

I tried also to use the new docker container proposed and i got the same issue:

jetson@jetson-desktop:~/tf-opencv-nvidia$ sudo docker run -it --rm --net=host --runtime=nvidia 6162b9651512 bash
[sudo] password for jetson:
root@jetson-desktop:/# python3 -c ‘import graphsurgeon as gs; print(gs.version)’
2021-01-18 08:53:42.712631: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
Traceback (most recent call last):
File “”, line 1, in
File “/usr/lib/python3.6/dist-packages/graphsurgeon/init.py”, line 58, in
from graphsurgeon.node_manipulation import *
File “/usr/lib/python3.6/dist-packages/graphsurgeon/node_manipulation.py”, line 42
node.attr[“dtype”].type = 1
^
IndentationError: unexpected indent

Should i install some other package?

Thanks & Regards,
Dario

5

Hi,

graphsurgeon is part of our TensorRT python package.
https://docs.nvidia.com/deeplearning/tensorrt/api/python_api/graphsurgeon/graphsurgeon.html

The error occurs when the graphsurgeon tries to import the TensorFlow but cannot find it in your environment.
Please noted that both graphsurgeon and uff parser relies on the TensorFlow v1.x package.

If you are using TensorFlow 2.x, please use the ONNX workflow instead.

  • TF 1.x: TF → UFF → TRT
  • TF 2.x: TF → ONNX → TRT

Thanks.