I was running optix 6.5 on my Ubuntu (16.04, driver 460, cuda 10.1). I found that when I increase the complexity of device functions to some level (actually not that complex, far from what we usually code in C++ function, but can be a little more complex than the usual RT_PROGRAM) , I got this error.
I checked with most of the setStackSize related forum links which didn’t solve my problem here.
Here is a simple test I tried:
Say I have two functions A() and B() which have the same level of complexity (the same number of local variables and same function calls ).
I commented out either A() or B(), there’s no problem. (even when I manually setStatckSize(256) which is a small number).
However when I have both A() and B() running in serial, then I got this error.
So I guess I can be sure that the problem is not because I run out of stack size.
Could you enlighten me where can this possibly go wrong? Does it have something to do with the OptiX compiler?
I’ve been solving this for a long time. And will it help if I use NSight to debug for this (even when I don’t have OptiX source code)?
The detail of the error is below:
rtContextLaunch2D(RTcontext, unsigned int, RTsize, RTsize)” caught exception: Encountered a rtcore error: m_api.pipelineSetStackSize(pipeline, directCallableStackSizeFromTraversal, directCallableStackSizeFromState, continuationStackSize, maxTraversableGraphDepth) returned (6): Invalid stack size. Segmentation fault (core dumped)
And by the way, the same code can run on my windows 10 without this problem (with driver 471 and the same OptiX 6.5 & cuda 10.1) which is wired.