Are you saying you ran your OptiX 6.5.0 based application without calling the rtContextSetMaxTraceDepth and rtContextSetMaxCallableProgramDepth functions before and they worked on R470 drivers?
In that case they used the default maximum depths of 5.
(The rtContextSetStackSize function has no effect in OptiX 6.5.0 anymore.)
At which rtContextSetMaxTraceDepth and rtContextSetMaxCallableProgramDepth values do you start to see the failures?
I’ll ask around internally what has changed to the stack size calculation among the driver releases, but without knowing what programs are in that pipeline and what your max trace depth values are, it would be difficult to analyze directly why it’s running into this invalid stack size error. Would you be able to provide a minimal and complete reproducer to be able to debug this?
Yes, when i ran Optix6.5 based application with r470 drivers its working without calling rtContextSetMaxTraceDepth and rtContextSetMaxCallableProgramDepth functions.
If I ran the same application with the 515 driver, it does not work. Then I set rtContextSetMaxTraceDepth and rtContextSetMaxCallableProgramDepth to the maximum value 31, it’s still not working.
Ok, yes, probably because 31 times the required stack space of your program is exceeding the upper limit on the allowed stack space even more.
If it worked with the default max depths of 5 before, my question was if it maybe works with any smaller value.
What are the actually required recursion depths inside your program?
For example, if it’s an iterative path tracer, the recursion depth is usually only 2.
You should use the absolute minimum amount of max depth values for these two calls to make the stack space as small as possible.