Need help in torch tensorrt

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

I’m working on codespace, I want to use torch_tensorrt on pytorch model but don’t know how to start, I checked the documentation of torch_tensorrt but couldn’t find an example to clear my concept.

can someone show me a workflow example of code using torch.nn.Module and using this torch.nn.Module in torch_tensorrt and calculating the torch tensorrt compiled result.

6

Please refer to the following documents:
https://pytorch.org/TensorRT/
https://pytorch.org/TensorRT/tutorials/notebooks.html#notebooks

7

Thanks a lot.

Just one question: If I use torch.compile() with backend tensorrt then it is necessary to send the model and inputs into cuda? because if I didn’t send model and it’s inputs into cuda then it will fail by saying this: AssertionError: 0th input(mm_75) is not on cuda device.