I had some CUDA OOM and discovered today the real reason.
When I have the CUDA OOM (on a new application launch) I can see that
doing “lsmod | grep nvidia” the module is used by someone but the application
is not running.
My application uses gnuplot to display in real time some data, I do this
performing a popen:
theGnuPlotPipe = popen("gnuplot -persist ", “w”);
I launch the application from a remote server so the gnuplot is running
on the server in where the GPUs are installed but the window manager
is my own computer.
When my application terminates the gnuplot process is still up, and is this
process that is using the nvidia module.
I’m perfectly aware of what there is behind a popen what is not clear to me is
why the driver is being used by an application that has never touched
any cuda allocated memory, unless gnuplot uses the nvidia driver without my