NVIDIA 364.19 + MatConvNet memory issues

Hi all,
I hope this is the write place to post this.

After a recent driver update, my I’ve had to reduce the size of my data batch sizes from 100 to 50 to avoid the same type of error as reported in : https://github.com/vlfeat/matconvnet/issues/65 . As such, I’m now incapable of using more than about 4.6GB on my titan X.

I am already using Cudnn so the linked issue isn’t very helpful. I’ve had the same issue across several versions of Matconvnet, so I feel like it’s more likely to be an issue with the recent driver.

nvidia-bug-report.log.gz (214 KB)

If it’s any help, here’s the output from cuda-memcheck. I’m going to try Matlab 2015b to see if it’s any better. Edit: same issue on 2015b and 2016a

========= Program hit CUDA_ERROR_NOT_FOUND (error 500) due to "named symbol not found" on CUDA API call to cuModuleGetGlobal_v2.
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/libcuda.so.1 (cuModuleGetGlobal_v2 + 0x18e) [0x4522de]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwgpu.so [0xaae6a4]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwgpu.so [0xaaed14]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwgpu.so [0xab0809]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwgpu.so [0x336507]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwgpu.so [0x336fe8]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwgpu.so [0x333adb]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwgpu.so [0x32e4e3]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwgpu.so [0x26cf5b]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwgpu.so [0x27c4b2]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwgpu.so [0x26207d]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwmcos_impl.so [0x1f1117]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwmcos_impl.so [0x1f2eb6]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwmcos_impl.so [0x1f88e0]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwmcos_impl.so [0x1f4053]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwmcos_impl.so [0x2e7ef0]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_dispatcher.so (_ZN13Mfh_MATLAB_fn11dispatch_fhEiPP11mxArray_tagiS2_ + 0xf4) [0x6f744]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwmcos_impl.so [0x2e5451]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3fed20]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3ae432]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b0612]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b6597]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b1cff]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b2934]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x4282ce]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_dispatcher.so (_ZN8Mfh_file16dispatch_fh_implEMS_FviPP11mxArray_tagiS2_EiS2_iS2_ + 0x2fa) [0xc6aea]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_dispatcher.so (_ZN8Mfh_file11dispatch_fhEiPP11mxArray_tagiS2_ + 0x20) [0xc6fb0]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3fed20]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3ae432]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b0612]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b6597]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b1cff]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b2934]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x4282ce]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_dispatcher.so (_ZN8Mfh_file16dispatch_fh_implEMS_FviPP11mxArray_tagiS2_EiS2_iS2_ + 0x2fa) [0xc6aea]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_dispatcher.so (_ZN8Mfh_file11dispatch_fhEiPP11mxArray_tagiS2_ + 0x20) [0xc6fb0]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3fed20]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3ae432]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b0612]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b6597]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b1cff]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b2934]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x4282ce]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_dispatcher.so (_ZN8Mfh_file16dispatch_fh_implEMS_FviPP11mxArray_tagiS2_EiS2_iS2_ + 0x2fa) [0xc6aea]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_dispatcher.so (_ZN8Mfh_file11dispatch_fhEiPP11mxArray_tagiS2_ + 0x20) [0xc6fb0]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_dispatcher.so [0x85e97]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x37cf7a]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3dc0ab]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x390f61]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x4162fb]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x390e06]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x381005]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3af6c2]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b6597]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b1cff]
=========     Host Frame:/home/MATLAB/R2015a/bin/glnxa64/libmwm_interpreter.so [0x3b2934]