[TensorRT] WARNING: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
conv1/convolution + activation_1/Relu6: 1.89542ms
block_1a_conv_1/convolution + activation_2/Relu6: 1.45514ms
block_1a_conv_2/convolution: 2.88618ms
block_1a_conv_shortcut/convolution + add_1/add + activation_3/Relu6: 0.554784ms
block_1b_conv_1/convolution + activation_4/Relu6: 3.05424ms
block_1b_conv_2/convolution: 2.25149ms
block_1b_conv_shortcut/convolution + add_2/add + activation_5/Relu6: 0.61792ms
block_2a_conv_1/convolution + activation_6/Relu6: 1.5256ms
block_2a_conv_2/convolution: 2.2432ms
block_2a_conv_shortcut/convolution + add_3/add + activation_7/Relu6: 0.336288ms
block_2b_conv_1/convolution + activation_8/Relu6: 1103.6ms
block_2b_conv_2/convolution: 2.32346ms
block_2b_conv_shortcut/convolution + add_4/add + activation_9/Relu6: 0.447488ms
block_3a_conv_1/convolution + activation_10/Relu6: 1.10899ms
block_3a_conv_2/convolution: 1.46125ms
block_3a_conv_shortcut/convolution + add_5/add + activation_11/Relu6: 0.238592ms
block_3b_conv_1/convolution + activation_12/Relu6: 2.00397ms
block_3b_conv_2/convolution: 1.71011ms
block_3b_conv_shortcut/convolution + add_6/add + activation_13/Relu6: 0.290784ms
block_4a_conv_1/convolution + activation_14/Relu6: 2.27453ms
block_4a_conv_2/convolution: 2.06624ms
block_4a_conv_shortcut/convolution + add_7/add + activation_15/Relu6: 0.308384ms
block_4b_conv_1/convolution + activation_16/Relu6: 1.16208ms
block_4b_conv_2/convolution: 0.805856ms
block_4b_conv_shortcut/convolution + add_8/add + activation_17/Relu6: 0.17408ms
output_cov/convolution: 0.058688ms
[TensorRT] ERROR: engine.cpp (725) - Cuda Error in reportTimes: 700 (an illegal memory access was encountered)
[TensorRT] ERROR: INTERNAL_ERROR: std::exception
[TensorRT] ERROR: engine.cpp (986) - Cuda Error in executeInternal: 700 (an illegal memory access was encountered)
[TensorRT] ERROR: FAILED_EXECUTION: std::exception
[TensorRT] ERROR: engine.cpp (179) - Cuda Error in ~ExecutionContext: 700 (an illegal memory access was encountered)
[TensorRT] ERROR: INTERNAL_ERROR: std::exception
[TensorRT] ERROR: Parameter check failed at: …/rtSafe/safeContext.cpp::terminateCommonContext::155, condition: cudnnDestroy(context.cudnn) failure.
[TensorRT] ERROR: Parameter check failed at: …/rtSafe/safeContext.cpp::terminateCommonContext::165, condition: cudaEventDestroy(context.start) failure.
[TensorRT] ERROR: Parameter check failed at: …/rtSafe/safeContext.cpp::terminateCommonContext::170, condition: cudaEventDestroy(context.stop) failure.
[TensorRT] ERROR: …/rtSafe/safeRuntime.cpp (32) - Cuda Error in free: 700 (an illegal memory access was encountered)
terminate called after throwing an instance of ‘nvinfer1::CudaError’
what(): std::exception
Aborted (core dumped)
@AastaLLL @kayccc
I am getting similar kind of when trying to run pre-trained tlt models after converting them to trt and running using Pycude script on Jetson NX. I was using PeopleNet (Resnet18).
Foolowed Pycuda script from here: Speeding Up Deep Learning Inference Using TensorFlow, ONNX, and NVIDIA TensorRT | NVIDIA Technical Blog