Cannot run TLT3.0 yolo models on Deepstream 5.1

Accelerated Computing Intelligent Video Analytics TAO Toolkit
#1

Hardware: GeForce® RTX 2080 Ti x 2
Software: Deepstream 5.1, replace existing TRT 7.2 with TRT 7.2 OSS based on this guide.

When run: ./ds-tlt -c ../configs/yolov3_tlt/pgie_yolov3_tlt_config.txt -i /opt/nvidia/deepstream/deepstream-5.1/samples/streams/sample_720p.h264

Errors:
0:00:01.978375885 31738 0x55ec2c3fbac0 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1716> [UID = 1]: Trying to create engine from model files
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
ERROR: …/nvdsinfer/nvdsinfer_func_utils.cpp:33 [TRT]: …/builder/cudnnBuilderUtils.cpp (410) - Cuda Error in findFastestTactic: 700 (an illegal memory access was encountered)
ERROR: …/nvdsinfer/nvdsinfer_func_utils.cpp:33 [TRT]: …/rtSafe/safeRuntime.cpp (32) - Cuda Error in free: 700 (an illegal memory access was encountered)
terminate called after throwing an instance of ‘nvinfer1::CudaError’
what(): std::exception
Aborted (core dumped)

Please advise.

#3

May I know more info about the environment?
What is the CUDA/Cudnn/TensorRT version? Did you install them via Jetpack?

#4

@Morganh,

I installed DS for dGPU so I don’t use Jetpack.

CUDA: 11.1
Cudnn: 8.1
TensorRT: 7.2

#5

Thanks for the info. So, you meet above mentioned error when you replace deepstream_tlt_apps/libnvinfer_plugin.so.7.2.2 at master · NVIDIA-AI-IOT/deepstream_tlt_apps · GitHub directly.
Could you help build TRT OSS to check if it is preproduced too?

#7