If I understand you correctly, you would recommend some extra time for the optixRaycasting sample?
Not at all. That’s definitely not an example I would recommend to look at first.
This is special in the way that it’s showing how to do only ray-triangle intersections with the high-level OptiX API.
Means everything else like ray generation and shading calculations happen inside native CUDA code.
That is a so called wavefront approach. You put a number of rays you want to shoot into the scene into a buffer, you launch a ray query with the dimension of that buffer, then the OptiX ray generation program reads the rays from that buffer (one per launch index), calls optixTrace with it, and the closest hit and (optional) miss programs report some data back to the per-ray payload, which is then written to the same dimension hit result buffer, which you then evaluate with a native CUDA kernel (outside the OptiX launch), which does the “shading” calculations and then generates potential continuation rays, and repeat that until there are no more rays to be shot.
This is effectively what the old discontinued OptiX Prime low-level ray-triangle intersection API did. OptiX Prime is not accelerated on RTX cards, that’s why the optixRaycasting example exists as an alternative, which actually offers more flexibility.
Still this wavefront approach has the drawback that it’s very memory access intensive. It’s usually faster to calculate the rays inside the ray generation program than to write to and read from a global memory buffer. The more the GPU can handle in registers the better.
Also this would require a launch for each set of new ray segments in a path tracer. And you would be responsible for handling the scheduling when some rays terminated early.
It’s much faster to iterate over the whole path while staying inside the ray generation program in a single launch and let OptiX handle the scheduling internally.
Long story short, I would recommend looking at all other OptiX SDK and open-source examples you find linked in the sticky posts of this sub-forum first, to understand how the whole ray tracing pipeline with raygen, exception, intersection (built-in for triangles (in hardware) and curves), anyhit, closesthit, miss and maybe direct and continuation callables play together. When done correctly, this is going to be the faster solution.
Actually read the OptiX 7 Programming Guide first. https://raytracing-docs.nvidia.com/
Oh also, our plan is to integrate our optix project into a flexible 3D-environment in order to easily create different environments such as an office with the help of programs, eg Unreal Engine or Unity.
A post from 2017 mentioned that optix is not compatible with Unreal Engine, is that still the case and if so, are there any other softwares that you would recommend?
The OptiX API knows nothing about application frameworks, windowing systems, scene file formats, UI, controllers, etc.
Related post about what OptiX is and isn’t: https://forums.developer.nvidia.com/t/how-to-develop-user-defined-rendering-in-optix/185374/2
It’s your responsibility to build the necessary acceleration structures from whatever geometrical descriptions you have.
It’s also your responsibility to implement everything related to shooting rays and handling potential material behaviors inside the respective domain programs.
If or how that is possible inside these game engines you cite is outside my expertise. It’s some years ago since I touched the Unreal engine and that thing is huge. I don’t know how difficult it would be to integrate some simulation module like that into it. Mind that these are using graphics APIs like Direct12/DXR or Vulkan (not sure about Vulkan Raytracing). You would use CUDA and OptiX. Sounds problematic to me.
For a start you might want to look at some of the OptiX SDK examples which can load OBJ and (not all) glTF model files .
My more advanced OptiX 7 examples use ASSIMP to load mesh geometry (not points, not lines, not really materials) from any supported file format.
What I’m saying is, that it would be simpler to start with some standalone OptiX application which can generate or load some scene data and develop the required algorithms with that first, before trying to delve into full blown game engines which work completely differently and might not even allow what you’re describing.
Follow all links in this post. The one to the OptiX 7.2 Release contains links to more examples: