optixModuleCreateFromPTX causes stack overflow

Gaming and Visualization Technologies Visualization OptiX
1

Hi,

I found that optixModuleCreateFromPTX() causes stack overflow (not related to any kernel execution) with some ptx.
I simplified the program and made a reproducer:

(Original program: OptiX_Utility/samples/deformation_blur at master · shocker-0x15/OptiX_Utility · GitHub)

Open samples/repro.sln and build/run. In my environment the program crashes with the stack overflow exception in nvrtum64.dll.
However, if I change the array size of the member variable GeometryData::curveVertexBuffers from 16 to 8 (see deformation_blur_shared.h), it succeeds to create a module.

Other random notes:

  • The program crashes both for Debug/Release.
  • I don’t use debug info for ptx generation both for Debug/Release.
  • I originally found the issue with the original program with CUDA 11.2 but with CUDA 11.1 (which OptiX 7.2.0 officially supports) it still happens.

Environment:
Windows 10 (20H2)
Visual Studio Community 2019 (16.8.3)
CUDA 11.1
OptiX 7.2.0
Core i9-9900K, 32GB
RTX 3080 10GB
NVIDIA Driver 461.09

Thanks,

2

Thank you for the reproducer! I can confirm that I reproduce this issue on my machine as well. It looks like it could be an infinite recursion. I’ve opened a bug report. Note that the size of GeometryData::curveVertexBuffers can be up to 15 without crashing, and any value over that causes the problem, as does using 15 and adding another pointer member to the struct. So there seems to be some kind of size threshold causing this, whenever the entire structure is over 128 bytes. We’ll respond when we have a workaround or fix on the way.

–
David.

1 Like
3

Hi @shocker.0x15 this bug has been fixed. You can find the update in the latest driver posted a few days ago, version 461.40 on Windows, or version 460.39 on Linux. You might have to search beta drivers and/or under the “Nvidia RTX” category to see them.

–
David.

4

Thanks for the notification, I could have confirmed the issue has disappeared.