I found that
optixModuleCreateFromPTX() causes stack overflow (not related to any kernel execution) with some ptx.
I simplified the program and made a reproducer:
Open samples/repro.sln and build/run. In my environment the program crashes with the stack overflow exception in nvrtum64.dll.
However, if I change the array size of the member variable
GeometryData::curveVertexBuffers from 16 to 8 (see deformation_blur_shared.h), it succeeds to create a module.
Other random notes:
- The program crashes both for Debug/Release.
- I don’t use debug info for ptx generation both for Debug/Release.
- I originally found the issue with the original program with CUDA 11.2 but with CUDA 11.1 (which OptiX 7.2.0 officially supports) it still happens.
Windows 10 (20H2)
Visual Studio Community 2019 (16.8.3)
Core i9-9900K, 32GB
RTX 3080 10GB
NVIDIA Driver 461.09