There are differences in the supported GPUs and driver modes, the language in which you implement shaders, and the feature set.
OptiX 7: Currently supports Maxwell, Pascal, Volta, Turing, Ampere GPU architectures.
Vulkan Raytracing: Needs RTX board , means Turing RTX and Ampere RTX boards.
Support for newer GPUs always comes with newer display drivers.
OptiX 7 runs in Windows Display Driver Mode (WDDM) and Tesla Compute Cluster (TCC) driver mode (latter is only available on dedicated compute and workstation products, not GeForce).
Vulkan needs WDDM under Windows because it’s a Graphics API.
OptiX 7 is based on CUDA. You’re managing the resources with CUDA host calls, and program the device code with CUDA C++ which allows the usual features like pointers, classes, templates, overloads, etc. (Don’t use virtual functions in OptiX device code though. That requires callable programs to emulate. Some synchronization instructions and shared memory aren’t allowed because OptiX controls the scheduling.)
In Vulkan you write the shaders in GLSL which is less flexible than CUDA C++.
OptiX 7 supports single-level, two-level, and multi-level render graph structures while Vulkan only supports a two-level structure (instance acceleration structure over geometry acceleration structure, named IAS->GAS in OptiX resp. top-level and bottom-level in Vulkan, TLAS->BLAS.
OptiX 7.5.0 supports more built-in geometric primitives (triangles; linear, quadratic, cubic, catmull-rom curves; spheres (new in Optix 7.5)).
Vulkan only supports built-in triangles.
Every other geometric primitive needs to be implemented with custom intersection programs in either API.
OptiX 7 has the more flexible motion blur support, contains a deep learning denoiser API, has higher limits of numbers of geometric primitives and number of instances in an acceleration structure.
It’s generally targeted more at the professional visualization market and exists a lot longer (OptiX 1 was released in 2009). Many famous renderers use it for their GPU raytracing implementation.
OptiX 7 is the natural choice if you’re programming something in a CUDA compute application already and need to solve something with raytracing.
Vulkan Raytracing would have the better integration with its graphics API obviously. Means you would be able to share all vertex and texture resources among the graphics and raytracing parts of the code.
Though it’s also possible to share Vulkan resources with CUDA, but that’s a little more involved.
Vulkan implementations would be available from different vendors.
There are two different extensions implementing Vulkan ray tracing, the ray tracing pipeline and ray queries.
https://www.khronos.org/blog/ray-tracing-in-vulkan
If it’s possible to implement the same things with OptiX 7 and Vulkan Raytracing, the performance will be very similar because in the end they will run on the same hardware. I did that with two of my OptiX 7 examples and for very simple things Vulkan was actually faster, but once the scene and renderer got more complicated it was on par.
If you’re coming from C++ based examples like the ones from Peter Shirley, implementing those things with CUDA C++ will be a lot less involved than trying to implement the same with GLSL.
If you’re starting with OptiX, please don’t use versions before OptiX 7 anymore. These are legacy and the very different OptiX 7 API is more modern, explicit and always faster. Always read the OptiX Release Notes before setting up a development environment. Link directly below the resp. OptiX version’s download button.
For more information about OptiX 7 and Vulkan Raytracing please have a look at these:
The OptiX 7 release announcements on this forum contain additional information and if you follow all previous announcements until OptiX 7.2 linked in there, you find more links to OptiX 7 examples and a SIGGRAPH course.
https://forums.developer.nvidia.com/t/optix-7-5-release/218032
These are my more advanced OptiX 7 examples (not yet updated to OptiX 7.5.0)
https://forums.developer.nvidia.com/t/optix-advanced-samples-on-github/48410/6
There is also a series of presentations about what is new in each OptiX 7 release and what it is used for by software vendors. Check out this search for OptiX on the GTC-On-Demand site:
https://www.nvidia.com/en-us/on-demand/search/?facet.mimetype[]=event%20session&layout=list&page=1&q=OptiX&sort=date&sortDir=desc
A lot of Vulkan and Vulkan Raytracing examples:
https://github.com/nvpro-samples
Look especially at these two tutorials in that order if you’re beginning with Vulkan Raytracing:
https://github.com/nvpro-samples/vk_mini_path_tracer
https://github.com/nvpro-samples/vk_raytracing_tutorial_KHR