My scene graph consists of a Trbvh top group, with several thousand heavily instanced triangle meshes under transform nodes, each with an individual Bvh/Bvh acceleration ( most of the meshes have 16-bit indices, which don’t seem to play well with most of the other builder types ). Total of around 6.7 million triangles.
In OptiX 3.5.1, this configuration successfully compiled, launched, and rendered. After updating from OptiX 3.5.1 to OptiX 3.6.0 and CUDA 5.5 to 6.0, however, this setup stalls on the call to launch while continually allocating memory. Somewhere around 4.5 gb by the task manager, it hits an exception over failure to allocate memory.
Windows 7, 64 bit, GTX 650, Driver 337.88.