pverm
April 25, 2016, 8:44pm
1
Hi all,
After a recent driver update, my I’ve had to reduce the size of my data batch sizes from 100 to 50 to avoid the same type of error as reported in : CUDA_ERROR_ILLEGAL_ADDRESS · Issue #65 · vlfeat/matconvnet · GitHub . As such, I’m now incapable of using more than about 4.6GB on my titan X.
I am already using Cudnn so the linked issue isn’t very helpful. I’ve had the same issue across several versions of Matconvnet, so I feel like it’s more likely to be an issue with the recent driver.
nvidia-bug-report.log.gz (214 KB)
pverm
April 26, 2016, 7:07pm
2
If it’s any help, here’s the output from cuda-memcheck. I’m going to try Matlab 2015b to see if it’s any better. Edit: same issue on 2015b and 2016a
========= Program hit CUDA_ERROR_NOT_FOUND (error 500) due to "named symbol not found" on CUDA API call to cuModuleGetGlobal_v2.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/usr/lib/libcuda.so.1 (cuModuleGetGlobal_v2 + 0x18e) [0x4522de]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwgpu.so [0xaae6a4]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwgpu.so [0xaaed14]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwgpu.so [0xab0809]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwgpu.so [0x336507]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwgpu.so [0x336fe8]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwgpu.so [0x333adb]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwgpu.so [0x32e4e3]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwgpu.so [0x26cf5b]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwgpu.so [0x27c4b2]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwgpu.so [0x26207d]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwmcos_impl.so [0x1f1117]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwmcos_impl.so [0x1f2eb6]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwmcos_impl.so [0x1f88e0]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwmcos_impl.so [0x1f4053]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwmcos_impl.so [0x2e7ef0]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_dispatcher.so (_ZN13Mfh_MATLAB_fn11dispatch_fhEiPP11mxArray_tagiS2_ + 0xf4) [0x6f744]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwmcos_impl.so [0x2e5451]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3fed20]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3ae432]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b0612]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b6597]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b1cff]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b2934]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x4282ce]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_dispatcher.so (_ZN8Mfh_file16dispatch_fh_implEMS_FviPP11mxArray_tagiS2_EiS2_iS2_ + 0x2fa) [0xc6aea]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_dispatcher.so (_ZN8Mfh_file11dispatch_fhEiPP11mxArray_tagiS2_ + 0x20) [0xc6fb0]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3fed20]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3ae432]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b0612]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b6597]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b1cff]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b2934]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x4282ce]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_dispatcher.so (_ZN8Mfh_file16dispatch_fh_implEMS_FviPP11mxArray_tagiS2_EiS2_iS2_ + 0x2fa) [0xc6aea]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_dispatcher.so (_ZN8Mfh_file11dispatch_fhEiPP11mxArray_tagiS2_ + 0x20) [0xc6fb0]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3fed20]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3ae432]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b0612]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b6597]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b1cff]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b2934]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x4282ce]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_dispatcher.so (_ZN8Mfh_file16dispatch_fh_implEMS_FviPP11mxArray_tagiS2_EiS2_iS2_ + 0x2fa) [0xc6aea]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_dispatcher.so (_ZN8Mfh_file11dispatch_fhEiPP11mxArray_tagiS2_ + 0x20) [0xc6fb0]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_dispatcher.so [0x85e97]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x37cf7a]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3dc0ab]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x390f61]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x4162fb]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x390e06]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x381005]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3af6c2]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b6597]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b1cff]
========= Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b2934]