cudaMalloc illegal memory access on orin nano

Autonomous Machines Jetson & Embedded Systems Jetson Orin Nano
1

Dear Reader,
I encountered a runtime error on orin nano that system reported an illegal memory acces when it was running the code of cudaMalloc to allocate GPU memory for images.
What is the cause and is there any debugger for CUDA?

Alex

code::
double* temp1 = nullptr;
for (int i = 0; i < config.N; ++i) {
HANDLE_ERROR(cudaMalloc((void**)&temp1, nSize * sizeof(double)));
HANDLE_ERROR(cudaMemcpy(temp1, imgs[i].data, nSize * sizeof(double), cudaMemcpyHostToDevice));
}

========= COMPUTE-SANITIZER
========= Internal Sanitizer Error: Failed to initialize mobile debugger interface. Please check that /dev NVIDIA nodes have the correct permissions

========= Internal Sanitizer Error: Device not supported. Please refer to the “Supported Devices” section of the sanitizer documentation

========= Program hit an illegal memory access was encountered (error 700) on CUDA API call to cudaMemcpy.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x2b72d4]
========= in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1
========= Host Frame: [0x8bee4]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame: [0x13e24]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame: [0x17158]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame: [0x106c4]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame:__libc_start_main [0x20e10]
========= in /lib/aarch64-linux-gnu/libc.so.6
========= Host Frame: [0x10f20]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda

========= Program hit an illegal memory access was encountered (error 700) on CUDA API call to cudaMemcpy.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x2b72d4]
========= in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1
========= Host Frame: [0x8bee4]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame: [0x13e3c]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame: [0x17158]
========= in /home/zhouxian/Cud========= COMPUTE-SANITIZER
========= Internal Sanitizer Error: Failed to initialize mobile debugger interface. Please check that /dev NVIDIA nodes have the correct permissions

========= Internal Sanitizer Error: Device not supported. Please refer to the “Supported Devices” section of the sanitizer documentation

========= Program hit an illegal memory access was encountered (error 700) on CUDA API call to cudaMemcpy.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x2b72d4]
========= in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1
========= Host Frame: [0x8bee4]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame: [0x13e24]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame: [0x17158]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame: [0x106c4]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame:__libc_start_main [0x20e10]
========= in /lib/aarch64-linux-gnu/libc.so.6
========= Host Frame: [0x10f20]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda

========= Program hit an illegal memory access was encountered (error 700) on CUDA API call to cudaMemcpy.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x2b72d4]
========= in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1
========= Host Frame: [0x8bee4]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame: [0x13e3c]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame: [0x17158]
========= in /home/zhouxian/Cud========= COMPUTE-SANITIZER
========= Internal Sanitizer Error: Failed to initialize mobile debugger interface. Please check that /dev NVIDIA nodes have the correct permissions

========= Internal Sanitizer Error: Device not supported. Please refer to the “Supported Devices” section of the sanitizer documentation

========= Program hit an illegal memory access was encountered (error 700) on CUDA API call to cudaMemcpy.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x2b72d4]
========= in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1
========= Host Frame: [0x8bee4]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame: [0x13e24]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame: [0x17158]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame: [0x106c4]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame:__libc_start_main [0x20e10]
========= in /lib/aarch64-linux-gnu/libc.so.6
========= Host Frame: [0x10f20]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda

========= Program hit an illegal memory access was encountered (error 700) on CUDA API call to cudaMemcpy.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x2b72d4]
========= in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1
========= Host Frame: [0x8bee4]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame: [0x13e3c]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame: [0x17158]
========= in /home/zhouxian/Cud========= COMPUTE-SANITIZER
========= Internal Sanitizer Error: Failed to initialize mobile debugger interface. Please check that /dev NVIDIA nodes have the correct permissions

========= Internal Sanitizer Error: Device not supported. Please refer to the “Supported Devices” section of the sanitizer documentation

========= Program hit an illegal memory access was encountered (error 700) on CUDA API call to cudaMemcpy.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x2b72d4]
========= in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1
========= Host Frame: [0x8bee4]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame: [0x13e24]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame: [0x17158]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame: [0x106c4]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame:__libc_start_main [0x20e10]
========= in /lib/aarch64-linux-gnu/libc.so.6
========= Host Frame: [0x10f20]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda

========= Program hit an illegal memory access was encountered (error 700) on CUDA API call to cudaMemcpy.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x2b72d4]
========= in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1
========= Host Frame: [0x8bee4]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame: [0x13e3c]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame: [0x17158]
========= in /home/zhouxian/Cud========= COMPUTE-SANITIZER
========= Internal Sanitizer Error: Failed to initialize mobile debugger interface. Please check that /dev NVIDIA nodes have the correct permissions

========= Internal Sanitizer Error: Device not supported. Please refer to the “Supported Devices” section of the sanitizer documentation

========= Program hit an illegal memory access was encountered (error 700) on CUDA API call to cudaMemcpy.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x2b72d4]
========= in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1
========= Host Frame: [0x8bee4]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame: [0x13e24]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame: [0x17158]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame: [0x106c4]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame:__libc_start_main [0x20e10]
========= in /lib/aarch64-linux-gnu/libc.so.6
========= Host Frame: [0x10f20]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda

========= Program hit an illegal memory access was encountered (error 700) on CUDA API call to cudaMemcpy.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x2b72d4]
========= in /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1
========= Host Frame: [0x8bee4]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame: [0x13e3c]
========= in /home/zhouxian/CudaRuntime3/CudaRuntime3/build/./structureLight_cuda
========= Host Frame: [0x17158]
========= in /home/zhouxian/CuddaRuntime3/CudaRuntime3/kernel.cu at line 653
========= Target application returned an error
========= ERROR SUMMARY: 5 errors

3

Hi,

It looks like you tried to reuse the tmp1 pointer for several images but did not handle the buffer after copying (ex. free).
Could you extend tmp1 to array with config.N element and allocate/copy image correspondingly?

Thanks.

5

Dear Sir or Madam,
Thank you for the reply.
I modified my code and allocate memory to each d_imgs[i]. However, the runtime error is not eliminated(error message:Segmentation fault (core dumped)). Or should I allocate whole memory of config.N element directly to pointer d_imgs?

Alex

void calcWrappedPhase(vectorcv::Mat& imgs, int N, cv::Mat& pha, cv::Mat& B, Config config) {

cv::Size img_size = imgs[0].size();
size_t nSize = img_size.width * img_size.height;

double *d_pha1 = 0; double *d_B1 = 0; double *d_Ik1 = 0; double *d_sin_sum1 = 0;
double *d_cos_sum1 = 0;
// thrust::host_vector<double*> h_imgs; thrust::device_vector<double*> d_imgs;
double **d_imgs;
HANDLE_ERROR(cudaMalloc(&d_imgs, config.N * sizeof(double*)));	
for (int i = 0; i < config.N; ++i) {
    HANDLE_ERROR(cudaMalloc((void**)&d_imgs[i], nSize * sizeof(double)));	
    //HANDLE_ERROR(cudaHostAlloc((void**)&temp1, nSize * sizeof(double),cudaHostAllocDefault));
    HANDLE_ERROR(cudaMemcpyAsync(d_imgs[i], imgs[i].ptr<double>(0), nSize * sizeof(double), cudaMemcpyHostToDevice));
    //h_imgs.push_back(d_temp1);
}
6

Could anyone kindly give me a reply?

Alex

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.