Hello. I have a tensorflow model (model.pb) and a labels file (labels.txt). I have not been able to find a readable tutorial online for how to convert this model into a tensorrt file. Please explain to me, in simple terms that a beginner can understand, how to make this conversion.
Hi,
You can follow our sample which is located at /usr/src/tensorrt/samples/sampleUffSSD/.
Thanks.
I’m sorry, but I am returning my Nano. It has been a nightmare to even set up. Poor documentation, unreliable, hardware, and difficult to customize. The entire reason I bought one was because I wanted to use my own pre-trained model on it, and that is proving to require 10+ years of software development training. I looked at the sample you recommended, it is equally opaque to other instructions online.
You market this as a maker board, I assumed it would be on the same level of difficulty as a raspberry pi. Given the issues and errors and difficulty I have encountered, I am surprised anyone has bought the Nano. It is a horrible product and I can’t recommend it to anybody.
6 Likes
Hi tylersuard, thanks for your feedback - we are always striving to improve resources available for Jetson and make it easier to use.
Regarding your model, the beginner’s approach would be to start by running the model through TensorFlow first. You can find the documentation for installing TensorFlow on Jetson here. Then you would just run your TensorFlow script and model like normal (which still uses GPU acceleration).
Next, would be to utilize the TF-TRT interface. This allows you to accelerate your TensorFlow graph with TensorRT while keeping the TensorFlow environment to run any layers that may not be supported by TensorRT. See here for resources on using TF-TRT:
- https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html
- https://github.com/NVIDIA-AI-IOT/tf_trt_models
- https://github.com/tensorflow/tensorrt
- https://devtalk.nvidia.com/default/topic/1042106/jetson-tx2/how-to-train-a-custom-object-detector-and-deploy-it-onto-jtx2-with-tf-trt-tensorrt-optimized-/post/5285864/#5285864
The final way, which does not rely on TensorFlow at runtime after the conversion has taken place, is to convert the graph to UFF and load it with the TensorRT C++ or Python runtime API. It can give the best performance but requires the conversion. Here are some resources on the UFF method:
- https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#working_tf
- https://github.com/NVIDIA-AI-IOT/tf_to_trt_image_classification
- https://github.com/AastaNV/TRT_object_detection
There are also UFF samples included with the TensorRT SDK that can be found under /usr/src/tensorrt/samples
4 Likes
Since this is the top google result for “convert tensorflow model for TensorRT”, I would like inform people reading that UFF has been deprecated for TensorRT and so that workflow has become very difficult. The new worklflow is TensorFlow-ONYX-TensorRT. Please see this nVidia Developer blog:
4 Likes
I am stuck with this problem for the last two weeks… There are many links and links inside them, but I cant find the exact thing to do.
I am trying to convert a two-class ssd-mobilent-v2 object detection model for a custom dataset. Now I am trying your approach. If I am successful, I will do a write-up…
2 Likes
Wondering if you were ever successful in converting? The Nano has been a nightmare to work with; have been stuck trying to get tensorflow to work on the nano, or converting to a format that works. Documentation is terrible, etc., etc., etc.
Please let me know if you got it working. I’d love to spend some time not trying to load sorfware and dependencies, and start using the product as it was intended (with a model that I have already built)
1 Like
Do you happen to know if this is outdated as well? Comments seem to suggest this doesn’t work for TensorFlow 2
That comment was old. Since then they have added a link to a tf2 example. I was able to get the tf2 example working for their specific resnet50 model. I had to make 2 small changes though. I changed the output layer name to match what was actually in the model and I had to mark the onnx model’s last layer as output before creating the engine. I am going to try with the mars-small128 model I am interested in and see if I can get it working.
1 Like
Thanks for the response. Regarding the TF2 model conversion, were you able to get decent frames per second (not sure which device you are using, but I assume Jetson Nano given the filter here is for that device.)
Edit: Would you happen to have a link to the tf2 example? Possibly this what you are referring to: GitHub - onnx/tensorflow-onnx: Convert TensorFlow, Keras, Tensorflow.js and Tflite models to ONNX
Also, are you converting from ONNX to TRT? I assume yes, and using the same procedure as noted in the original link you posted?
Thanks.
I didn’t test the performance of the model, I was just checking that the conversion works.
I already posted the link above. The 5th paragraph has links to tf1 and tf2 example projects. The readme file in the project explains the whole process. Keras model → pb → onnx → trt plan/engine. I was able to convert my own model the same way. I failed on the onnx → trt engine part because my model had layers that were not compatible with TensorRT. I will use TF-TRT instead. All your layers must have compatible operations with TRT to use the above method. If some of your layers are not compatible, use TF-TRT to optimize the compatible parts and still let Tensorflow run the incompatible parts. This can still realize a boost in performance and decrease in model/graph size.
Let me know if you run into problems. I had to work around a couple things.
1 Like
Thanks. I’ll take a look at it tomorrow or this weekend and let you know how it goes. Much appreciated for the responses, guidance/help
1 Like