You only need the original torch2trt sources for when you do setup.py install for it. Try this in a fresh instance of the container:
# run these commands inside container
cd /tmp
git clone https://github.com/NVIDIA-AI-IOT/torch2trt
cd torch2trt
pip3 install --verbose .
# test torch2trt in container
cd /
pip3 show torch2trt
python3 -c 'import torch2trt'
If you mean outside container, those are the same instructions on the torch2trt github page - you would need to install PyTorch first (from PyTorch for Jetson)
You can also just use my torch2trt or l4t-pytorch container images which already has torch2trt installed in it.