Hi,
I resolved the error. I needed to set the --target to install in /usr/local/lib/python3.8/dist-packages/torch it was installing in a different location
format is this:
python3 -m pip install package_name --target /path/to/custom_directory
This thread helped: Error while installing torch2trt on Xavier NX - #7 by AastaLLL