Stack overflow with OptiX6.0.0 / RTX 2080Ti

I’m trying to upgrade my project from OptiX5.1 / GTX 970m to OptiX6.0.0 / RTX 2080Ti. I didn’t change any code, except some cmake files. But this program got stack overflow exception when tracing the rays. I try to change the stack size of the context in 2080Ti to a bigger value. But it seems have no effect.
So, the conclusion is that OptiX6.0.0 or 2080Ti has a smaller max size of the thread stack?
It’s too abnormal.

OptiX 6.0.0 added a new API to set the maximum number of recursions, which is a lot simpler than the old byte-based stack size. This must be used when running with the new RTX execution strategy, which is the default. rtContextSetStackSize doesn’t have an effect then (as documented).

Please have a look into the OptiX headers and search for rtContextSetMaxTraceDepth and rtContextSetMaxCallableProgramDepth and look at their documentation.

Make sure to change the default values for these to the minimum recursion depth your OptiX application uses to save stack memory.