Using and updating TensorRT on Pegasus platform

In the Development Guide for Drive 9.0 DNN inference part starts assuming one is able to generate .bin file using the TensorRT_optimization tool, which seems to generate bin files specifically optimized for the platform. We were wondering if using the tensorRT C++ API on Nvidia provides the same benefits of optimizing networks for the platform? Also is it possible to update TensorRT to a newer version? Currently DRIVE Software 9.0 comes with TensorRT 5.0.3 version, and we are having numerous issues when trying to optimize fasterRCNN and Deeplab based models.

1 Like

Dear tonci.antunovic,

TensorRT_optimization tool is for DriveWorks samples.
Unfortunately DriveAGX does not have the latest version of TRT because of the various combinations that are considered.
Could you please let us know what the problems you are talking about?
If possible, please file a bug and provide a bug number. Thanks.

We are not sure what you mean by “TensorRT_optimization tool is for DriveWorks samples”. In the Nvidia Drive development guide (under Deep Neural Networks), in all the samples/documentation for performing inference of trained deep networks, it is always assumed that the bin file has been successfully generated with the TensorRT_optimization tool.

We are facing issues in the TensorRT_optimization tool regarding unsupported layers, as well as limitations when using supported layers (like unsupported broadcasting). Both as the uff/onnx parser limitations as well as the optimizer limitations (and we can’t find precise documentation of all such restrictions for the currently installed version 5.0.3).

One frustrating thing is the inability to retrieve the list of all issues in the models we are trying to optimize - the optimizer simply breaks at some layer/operation in the model. Fixing that issue would just lead to a new issue at a different operation/layer in the model. This makes it rather difficult to estimate the effort needed to implement all the custom layers.

We might try to use tensorrt C++ API on the Drive machine to generate the optimized bin files if we can get more flexibility. But we are unsure if this approach would produce .bin files which are optimized for the Pegasus platform (like when using TensorRT_optimization tool).

Also we would clearly be very happy to be able to update tensorrt on the machine, but again we are not sure if the new tensorrt version would be be optimized for the platform (if it’s even possible to update tensorrt).

Thanks.

Dear tonci.antunovic,

Thank you for your kindly explanation.
We are planning a Webinar on Optimizing DNN Inference using CUDA and TensorRT on NVIDIA DRIVE AGX on October 22, 2019.
https://event.on24.com/wcc/r/2086254/2BB023EA80E1ED375B33076AF8BBFF3C/653350?partnerref=internalemail
I think it would be nice to join the webinar and you can also get information about your inquiry through Q & A session.

We have joined the webinar, asked questions, but we didn’t get useful information. Do you have some further information regarding our issues.

Dear tonci.antunovic,

-. I believe our TensorRT documentation describes things in detail.

  1. Clearly, supported ops are listed: https://docs.nvidia.com/deeplearning/sdk/tensorrt-support-matrix/index.html#supported-ops, https://docs.nvidia.com/deeplearning/sdk/tensorrt-support-matrix/index.html and each layers params and ranges are already documented. e.g. see this https://docs.nvidia.com/deeplearning/sdk/tensorrt-api/c_api/classnvinfer1_1_1_i_convolution_layer.html#abb50e93b6d81e0312573c2e0ced52622 Naturally, if some ops are not list in the "supported ops", that will be unsupported op.
  2. TensorRT includes optimization process/workflow such as layer fusion and auto-tuning. There is no a separate tool.
  3. As we are focusing on ONNX path, ONNX parser is open-source. https://github.com/onnx/onnx-tensorrt[/url], https://github.com/onnx/onnx-tensorrt/blob/master/operators.md
  4. What do you mean by "Optimizer limitation"? This is not clear.

-. Could you please let me know what issues exactly? BTW, custom ops should be implemented using plugin api.

-. User needs to converting a trained model to TRT engine and can serialize a file called “plan” file. That is optimized inference network for TRT. It’s not a collection of bin files.

-. Whenever TensorRT version is upgraded, higher performance is expected with new feature and/or bug fixes.