Illegal memory access Error

AI & Data Science Computer Vision & Image Processing
,
1

Hello, I’m trying to do inference for a Convolutional Network on RTX 3060. When I ran my python script, I got a pyCUDA error: Illegal Memory Access. My test env is Ubuntu 22.04, and RTX 3060 GPU. I have tried to recompile my ONNX model and rebuild the TensorRT engine.

This is the log I get:

[12/03/2022-17:05:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::29] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[12/03/2022-17:05:21] [TRT] [E] 1: [cudaDriverHelpers.cpp::operator()::29] Error Code 1: Cuda Driver (an illegal memory access was encountered)
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFreeHost failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFree failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFreeHost failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
2

The “Illegal Memory Access” error typically occurs when the program tries to access memory that it does not have permission to access, or when the memory has been corrupted.
Check that your GPU memory is not being overloaded.