is it possible that during a context compilation if there gpu memory is full (or almost) I get a std c++ stack overflow exception?
Because when I increase the size of the textures in the GPU I get this error, but theorically it is still free space…
The stack overflow error indicates that your stack size is not set large enough. Use rtContextGetStackSize/rtContextSetStackSize to find out how large your stack is and set it larger.
If it was a C++ stack overflow exception that might have been something different than an OptiX stack overflow.
Edit: Reading your other posts, your errors were about timeouts and the old versions of CUDA and OptiX you’re using might not yet have decoded the error code, so either tips might help.
32- or 64-bit application?
Watch your host memory consumption, under Windows look into the TaskManager while running the app.
If this was during context->compile() which acceleration structures did you use? Try one which uses less host memory, e.g. Bvh instead of Sbvh.
If you say there is theoretically enough GPU memory, what is the actual GPU memory consumption at the time of the failure?
Other details which should always be included in problem reports:
OS version, OS bitness, installed GPUs, display driver version, OptiX version, CUDA version.
The current configuration is CUDA 4.1, Optix SDK 2.5.0, Windows 7 64 bits, Visual Studio 2010 with a 32 bits project, and GTX 460.
I use Sbvh acceleration structures.
I have check the memory consumption and it is ok, I have more than 2GB free.
I have check the memory consumption on GPU with OpenGL calls and I see that the problem should be that there is not enough memory… It is still strange for me to get a c++ stack overflow when there is not enough memory… :(
Anyway, thanks for the comments! I must reduce the textures or buy a new GPU. :)
Depending on the scene complexity, it’s really worth trying the other acceleration builders as well.
If possible you should reconsider the basic configuration of the application. OptiX 2.5.0, CUDA 4.1, and 32-bits are pretty much outdated and an update possibly gets you better performance.