While using Application Verifier to debug what looked like heap corruption, I came across what appears to be a ‘use after free’ bug in the first call to rtBufferCreate or rtBufferCreateFromGLBO, possibly other rtBufferCreateFoo functions. It’s easily reproducible in the OptiX samples by enabling Application Verifier on one of the samples, I used optixPathTracer, and run the sample with a debugger attached.
The first exception I get is
Exception thrown at 0x00007FFEED576048 (KernelBase.dll) in optixPathTracer.exe:
0xC0000005: Acess violation reading location 0x00000244CF694FF0.
with call stack
sutil_sdk.dll!optix::ContextObj::createBufferFromGLBO(unsigned int type, unsigned int vbo) Line 2196
A total of 3 to 5 exceptions are thrown, but only when the first buffer is created, all subsequent calls work as epected and the initial buffer also contains the expected data, but once in a while OptiX will crash in rtContextLaunch2D.
In Application Verifier I’ve enabled all the tests under Basics and DirtyStacks in Miscellaneous.
Windows 10, 1803
NVIDIA Quadro P2000
Visual Studio 2015 Update 3
I have a 1080GTX and a 2080 RTX that I plan to test it on as well.