rtExt/cuda/cudaGatherRunner.cpp (111) - Cuda Error in execute: 700 (an illegal memory access was encountered)

Description

After run the tensorrt model for some minutes, the error appears:
"
[TensorRT] ERROR: …/rtExt/cuda/cudaGatherRunner.cpp (111) - Cuda Error in execute: 700 (an illegal memory access was encountered)

[TensorRT] ERROR: FAILED_EXECUTION: std::exception
"

Environment

NGC 20.10

Hi @593904440,
Can you please try running your model with trtexec command, and share the verbose logs with us?
e.g.,

trtexec --onnx=your_model.onnx --verbose --explicitBatch --shapes=name:1x3x1216x800

Thanks!

Sry, I figured out that, I used the trt7 to transfer onnx->trt, but inference with execute_async, after i changed the model in dynamic and inference with execute_async_v2, the error was gone.

Best wishes!

1 Like