If you want a functioning Cuda enabled Torch and Torchvision in Jetson, you follow use this installation procedure: PyTorch for Jetson
Uninstall what you had installed before and reinstall it using the steps above.
I don’t think you can use TRT at all without a cuda compatible torch.