Is there any special reason for not supporting Pytorch models directly in DeepStream currently?
Currently, DeepStream wraps TensorRT as nvinfer(pgie) GST plugin to support inference.
TensorRT now supports the parser of onnx, uff and caffe model, so DeepStream supports these three models.
For pytorch, you could export the model to onnx and run it with deepstream.
Thanks!
BTW, pythorch model to TensorRT engine tool: Torch2TRT: GitHub - NVIDIA-AI-IOT/torch2trt: An easy to use PyTorch to TensorRT converter
How to save that in disk as .engine or .plan?
Sorry! GitHub - NVIDIA-AI-IOT/torch2trt: An easy to use PyTorch to TensorRT converter is based on old TensorRT( TensorRT5.0), so please ignore it.
So, as I suggested above, after you get a Pytorch model with suffix .pt, then you export it to onnx file (torch.onnx.export() - (optional) Exporting a Model from PyTorch to ONNX and Running it using ONNX Runtime — PyTorch Tutorials 2.1.1+cu121 documentation) , then you can use this onnx model in TensorRT for inference (refer to TensorRT sample - sampleOnnxMNIST about how to use onnx).
How to save that in disk as .engine or .plan?
When you use the ONNX file in TensorRT, basically, TensorRT will do
- parse the onnx model
- build TensorRT engine
- load the engine to do inference
Most TensorRT samples does not save the TensorRT engine into disk in above step#2, but user can save/dump the TensorRT engine (it’s just a buffer) into disk.
Please let me know if my answer address your question.
Thanks!
When I try to do that I get error which says DS has Onnx parser 0.0.3 but model was made with onnx IR version 0.0.4. Then fails to load the ONNX model.
When I try to do that I get error which says DS has Onnx parser 0.0.3 but model was made with onnx IR version 0.0.4
this is just a warning, not an error. you can ignore it.
do you see any other failure ?
Hi muhammad.abdussabur,
We haven’t heard back from you in a couple weeks, so marking this topic closed.
Please open a new forum issue when you are ready and we’ll pick it up there.