Speeding Up Deep Learning Inference Using TensorRT

Technical Blogs & Events Technical Blog
1

Originally published at: https://developer.nvidia.com/blog/speeding-up-deep-learning-inference-using-tensorrt/

Looking for more? Check out the hands-on DLI training course: Optimization and Deployment of TensorFlow Models with TensorRT This is an updated version of How to Speed Up Deep Learning Inference Using TensorRT. This version starts from a PyTorch model instead of the ONNX model, upgrades the sample application to use TensorRT 7, and replaces…

2

This line:

>> tar xvf speeding-up-unet.7z # Unpack the model data into the unet folder

is out of date. It’ll throw an error “this does not look like a tar archive”.

This fix is:

apt update
apt install p7zip-full
7z x speeding-up-unet.7z

The article or the file should be updated, if possible

4

I think this blog page needs to be updated. For example, you make mention of pulling down a Docker container for pytorch, but then make no mention of it.

When I try to build code_samples/posts/TensorRT-introduction with the default Ubuntu 20.04 protobuf I get an error about post.inc. I then grabbed the latest protobuf repo and tried that. That version gave me different errors. Now I’m on protobuf v3.17.3 and I receive the following errors:

In file included from /usr/local/include/onnx/onnx_pb.h:52,
                 from ioHelper.cpp:32:
/usr/local/include/onnx/onnx-ml.pb.h:616:3: error: ‘PROTOBUF_FUTURE_MUST_USE_RESULT’ does not name a type
  616 |   PROTOBUF_FUTURE_MUST_USE_RESULT std::string* release_name();
      |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/include/onnx/onnx-ml.pb.h:634:3: error: ‘PROTOBUF_FUTURE_MUST_USE_RESULT’ does not name a type
  634 |   PROTOBUF_FUTURE_MUST_USE_RESULT std::string* release_s();
      |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/include/onnx/onnx-ml.pb.h:652:3: error: ‘PROTOBUF_FUTURE_MUST_USE_RESULT’ does not name a type
  652 |   PROTOBUF_FUTURE_MUST_USE_RESULT std::string* release_doc_string();
      |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/include/onnx/onnx-ml.pb.h:670:3: error: ‘PROTOBUF_FUTURE_MUST_USE_RESULT’ does not name a type
  670 |   PROTOBUF_FUTURE_MUST_USE_RESULT std::string* release_ref_attr_name();
      |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/include/onnx/onnx-ml.pb.h:685:3: error: ‘PROTOBUF_FUTURE_MUST_USE_RESULT’ does not name a type
  685 |   PROTOBUF_FUTURE_MUST_USE_RESULT ::onnx::TensorProto* release_t();

The errors go on and on and on. Very difficult to follow an “introductory example.”

5

Hi @Shadowmind, sorry about the inconvenience.

This tutorial has been tested on Ubuntu 18.04, so can you please try using the following container?

# Pull docker container
docker pull nvcr.io/nvidia/pytorch:20.07-py3

# Run container
docker run --rm --gpus all -ti nvcr.io/nvidia/pytorch:20.07-py3
6

error occurs as below

Traceback (most recent call last):
  File "/home/screw/Desktop/LearningTensorRT/code-samples/posts/unet/prepareData.py", line 7, in <module>
    from utils import normalize_volume
ImportError: cannot import name 'normalize_volume' from 'utils' (/home/screw/.local/lib/python3.8/site-packages/utils/__init__.py)

l don’t know where to import the normalize_volume fucntion, so i guess it just normalize the input image, using code below replace the original function

input_image = np.asarray(input_image, dtype='float32')
mean_input_image = input_image.mean()
input_image /= mean_input_image

the model does works, but it does’t work as amazing as the examples from the tourial.

So my question is how to fix the bug and using the normalize_volume function, thanks!

7

When I try to build code_samples/posts/TensorRT-introduction, there is an error:

simpleOnnx.cpp: In function ‘nvinfer1::ICudaEngine* createCudaEngine(const string&, int)’:
simpleOnnx.cpp:72:14: error: ‘class nvinfer1::IBuilder’ has no member named ‘setFp16Mode’
     builder->setFp16Mode(builder->platformHasFastFp16());

How to fix it?