Hi,
I’m sometimes running into a CUDA Address out of bounds error. It the same error : ========= Invalid global write of size 4
========= at 0x00001218.
The app runs, and exits successfully, but part of the output data is corrupted when viewed (graphics image).
Here’s the lo-down on the eGPU in use, on OSX 10.3.6, with a Sonnet eGPU.
Detected 1 CUDA Capable device(s)
Device 0: “GeForce GTX 1070”
CUDA Driver Version / Runtime Version 10.1 / 10.1
CUDA Capability Major/Minor version number: 6.1
Total amount of global memory: 8192 MBytes (8589737984 bytes)
(15) Multiprocessors, (128) CUDA Cores/MP: 1920 CUDA Cores
GPU Max Clock rate: 1772 MHz (1.77 GHz)
Memory Clock rate: 4004 Mhz
Memory Bus Width: 256-bit
L2 Cache Size: 2097152 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(32768, 32768), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 2 copy engine(s)
Run time limit on kernels: Yes
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device supports Compute Preemption: Yes
Supports Cooperative Kernel Launch: Yes
Supports MultiDevice Co-op Kernel Launch: Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 188 / 0
Here’s running the app through CUDA Memcheck from the OSX terminal with an OK result:
$ cuda-memcheck --leak-check full ./resize_texture
Running OpenMP, number of threads : 8
B&W - Texture
Dimension of x blocks of threads : thread_count : 32
Dimension of y blocks of threads : thread_count : 32
Number of Threads per block: threads.x * threads.y : 1024
Dimension of x grid in blocks) : blocks.x : 138
Dimension of y grid in blocks) : blocks.y : 78
Number of Blocks: blocks.x * blocks.y : 10764
Saved : lop.tif
Time : 2.28168
========= LEAK SUMMARY: 0 bytes leaked in 0 allocations
========= ERROR SUMMARY: 0 errors
(To confirm terminal params, verified / analyzed app through nVidia Visual profiler)
Here’s running the app, with errors. Here, the image size is slightly larger than the above example:
cuda-memcheck ./resize_texture
Running OpenMP, number of threads : 8
—— CUDA
Dimension of x blocks of threads : 32
Dimension of y blocks of threads : 32
Number of Threads per block: threads.x * threads.y : 1024
Dimension of x grid in blocks) : blocks.x : 172
Dimension of y grid in blocks) : blocks.y : 97
Number of Blocks: blocks.x * blocks.y : 16684
========= Invalid global write of size 4
========= at 0x00001218 in render_pixel(float*, unsigned int, unsigned int, unsigned int, unsigned short, float, float, float, float, float, float)
========= by thread (27,17,0) in block (171,96,0)
========= Address 0x51a0d4ae0 is out of bounds
========= Saved host backtrace up to driver entry point at kernel launch time
========= Host Frame:/Library/Frameworks/CUDA.framework/Versions/A/Libraries/libcuda_387.10.10.10_mercury.dylib (cuLaunchKernel + 0x1ec) [0x13d89c]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (_ZN6cudartL25cudaApiLaunchKernelCommonEPKv4dim3S2_PPvmP11CUstream_stb + 0xfb) [0x40fb]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (_ZN6cudart19cudaApiLaunchKernelEPKv4dim3S2_PPvmP11CUstream_st + 0x15) [0x3ff5]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (cudaLaunchKernel + 0x20c) [0x3651c]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (_ZL16cudaLaunchKernelIcE9cudaErrorPKT_4dim3S4_PPvmP11CUstream_st + 0xf0) [0x8a080]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (_Z42__device_stub__Z12render_pixelPfjjjtffffffPfjjjtffffff + 0x4cd) [0x89f4d]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (_Z12render_pixelPfjjjtffffff + 0x73) [0x89a33]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (Z40resize_texture_rendering_pixel_wise_cudaPKf22filmGrainOptionsStructIfEPfS3 + 0xaf0) [0x89340]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (main + 0xac2e) [0x4774e]
========= Host Frame:/usr/lib/system/libdyld.dylib (start + 0x1) [0x1015]
========= Host Frame:[0xd]
========= Program hit cudaErrorLaunchFailure (error 719) due to “unspecified launch failure” on CUDA API call to cudaMemcpy.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/Library/Frameworks/CUDA.framework/Versions/A/Libraries/libcuda_387.10.10.10_mercury.dylib (cudbgMain + 0x1b702) [0x217c02]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (cudaMemcpy + 0x156) [0x32426]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (Z40resize_texture_rendering_pixel_wise_cudaPKf22filmGrainOptionsStructIfEPfS3 + 0xb76) [0x893c6]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (main + 0xac2e) [0x4774e]
========= Host Frame:/usr/lib/system/libdyld.dylib (start + 0x1) [0x1015]
========= Host Frame:[0xd]
Error at src//resize_texture_rendering.cu : 641
========= Program hit cudaErrorLaunchFailure (error 719) due to “unspecified launch failure” on CUDA API call to cudaMemcpy.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/Library/Frameworks/CUDA.framework/Versions/A/Libraries/libcuda_387.10.10.10_mercury.dylib (cudbgMain + 0x1b702) [0x217c02]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (cudaMemcpy + 0x156) [0x32426]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (Z40resize_texture_rendering_pixel_wise_cudaPKf22filmGrainOptionsStructIfEPfS3 + 0xbd5) [0x89425]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (main + 0xac2e) [0x4774e]
========= Host Frame:/usr/lib/system/libdyld.dylib (start + 0x1) [0x1015]
========= Host Frame:[0xd]
Error type : unspecified launch failure
========= Error: process didn’t terminate successfully
========= The application may have hit an error when dereferencing Unified Memory from the host. Please rerun the application under cuda-gdb or Nsight Eclipse Edition to catch host side errors.
========= No CUDA-MEMCHECK results found
(To confirm terminal params, verified / analyzed app through nVidia Visual profiler)
Here’a another, again with a slightly larger image than the first:
Running OpenMP, number of threads : 8
—— CUDA
Dimension of x blocks of threads : 32
Dimension of y blocks of threads : 32
Number of Threads per block: threads.x * threads.y : 1024
Dimension of x grid in blocks) : blocks.x : 179
Dimension of y grid in blocks) : blocks.y : 101
Number of Blocks: blocks.x * blocks.y : 18079
========= Invalid global write of size 4
========= at 0x00001218 in render_pixel(float*, unsigned int, unsigned int, unsigned int, unsigned short, float, float, float, float, float, float)
========= by thread (3,1,0) in block (178,100,0)
========= Address 0x51b99fa20 is out of bounds
========= Saved host backtrace up to driver entry point at kernel launch time
========= Host Frame:/Library/Frameworks/CUDA.framework/Versions/A/Libraries/libcuda_387.10.10.10_mercury.dylib (cuLaunchKernel + 0x1ec) [0x13d89c]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (_ZN6cudartL25cudaApiLaunchKernelCommonEPKv4dim3S2_PPvmP11CUstream_stb + 0xfb) [0x40fb]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (_ZN6cudart19cudaApiLaunchKernelEPKv4dim3S2_PPvmP11CUstream_st + 0x15) [0x3ff5]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (cudaLaunchKernel + 0x20c) [0x3651c]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (_ZL16cudaLaunchKernelIcE9cudaErrorPKT_4dim3S4_PPvmP11CUstream_st + 0xf0) [0x8a080]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (_Z42__device_stub__Z12render_pixelPfjjjtffffffPfjjjtffffff + 0x4cd) [0x89f4d]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (_Z12render_pixelPfjjjtffffff + 0x73) [0x89a33]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (Z40resize_texture_rendering_pixel_wise_cudaPKf22filmGrainOptionsStructIfEPfS3 + 0xaf0) [0x89340]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (main + 0xac2e) [0x4774e]
========= Host Frame:/usr/lib/system/libdyld.dylib (start + 0x1) [0x1015]
========= Host Frame:[0xd]
========= Program hit cudaErrorLaunchFailure (error 719) due to “unspecified launch failure” on CUDA API call to cudaMemcpy.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/Library/Frameworks/CUDA.framework/Versions/A/Libraries/libcuda_387.10.10.10_mercury.dylib (cudbgMain + 0x1b702) [0x217c02]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (cudaMemcpy + 0x156) [0x32426]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (Z40resize_texture_rendering_pixel_wise_cudaPKf22filmGrainOptionsStructIfEPfS3 + 0xb76) [0x893c6]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (main + 0xac2e) [0x4774e]
========= Host Frame:/usr/lib/system/libdyld.dylib (start + 0x1) [0x1015]
========= Host Frame:[0xd]
Error at src//resize_texture_rendering.cu : 641
========= Program hit cudaErrorLaunchFailure (error 719) due to “unspecified launch failure” on CUDA API call to cudaMemcpy.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/Library/Frameworks/CUDA.framework/Versions/A/Libraries/libcuda_387.10.10.10_mercury.dylib (cudbgMain + 0x1b702) [0x217c02]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (cudaMemcpy + 0x156) [0x32426]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (Z40resize_texture_rendering_pixel_wise_cudaPKf22filmGrainOptionsStructIfEPfS3 + 0xbd5) [0x89425]
========= Host Frame:/code/tgascale_gpu_all_good_with_omp/bin/./resize_texture (main + 0xac2e) [0x4774e]
========= Host Frame:/usr/lib/system/libdyld.dylib (start + 0x1) [0x1015]
========= Host Frame:[0xd]
Error type : unspecified launch failure
========= Error: process didn’t terminate successfully
========= The application may have hit an error when dereferencing Unified Memory from the host. Please rerun the application under cuda-gdb or Nsight Eclipse Edition to catch host side errors.
========= No CUDA-MEMCHECK results found
(To confirm terminal params, verified / analyzed app through nVidia Visual profiler)