I fixed it by adding:
Fix for CUDA driver is a stub library (from NVIDIA forums)
rm -f /usr/local/cuda/lib64/stubs/libcuda.so && \
ln -s /usr/local/cuda/lib64/libcuda.so /usr/local/cuda/lib64/stubs/libcuda.so
to the build script of torch2trt from [checkMacros.cpp::catchCudaError::272] Error Code 1: Cuda Runtime (CUDA driver is a stub library) - #11 by Robert_Crovella.