optixModuleCreateFromPTX causes stack overflow


I found that optixModuleCreateFromPTX() causes stack overflow (not related to any kernel execution) with some ptx.
I simplified the program and made a reproducer:

(Original program: https://github.com/shocker-0x15/OptiX_Utility/tree/master/samples/deformation_blur)

Open samples/repro.sln and build/run. In my environment the program crashes with the stack overflow exception in nvrtum64.dll.
However, if I change the array size of the member variable GeometryData::curveVertexBuffers from 16 to 8 (see deformation_blur_shared.h), it succeeds to create a module.

Other random notes:

  • The program crashes both for Debug/Release.
  • I don’t use debug info for ptx generation both for Debug/Release.
  • I originally found the issue with the original program with CUDA 11.2 but with CUDA 11.1 (which OptiX 7.2.0 officially supports) it still happens.

Windows 10 (20H2)
Visual Studio Community 2019 (16.8.3)
CUDA 11.1
OptiX 7.2.0
Core i9-9900K, 32GB
RTX 3080 10GB
NVIDIA Driver 461.09


Thank you for the reproducer! I can confirm that I reproduce this issue on my machine as well. It looks like it could be an infinite recursion. I’ve opened a bug report. Note that the size of GeometryData::curveVertexBuffers can be up to 15 without crashing, and any value over that causes the problem, as does using 15 and adding another pointer member to the struct. So there seems to be some kind of size threshold causing this, whenever the entire structure is over 128 bytes. We’ll respond when we have a workaround or fix on the way.


1 Like