First of all, if you’re new to OptiX, please do not use any of the legacy OptiX versions (1 - 6) for new developments.
There already have been six releases of OptiX 7 which implements a more modern, explicit API, is better integrated with CUDA, contains more features, and is generally faster than any older OptiX API
https://forums.developer.nvidia.com/t/optix-7-5-release/218032
Now to your issue:
Most of the OptiX SDK examples use CUDA-OpenGL interoperability to display the image ray traced with OptiX (CUDA) with a textured rectangle in OpenGL. The interop happens by allocating an OpenGL pixel buffer object (PBO) which resides on the GPU device and is mapped to a CUDA pointer into which the OptiX device programs can render directly.
That CUDA-OpenGL interop mechanism can only work with an NVIDIA OpenGL implementation runing on the same device.
Since you’re using Mesa’s OpenGL implementation these CUDA-OpenGL interop including rtBufferCreateFromGLBO (where GLBO stands for OpenGL buffer object) cannot work and will fail in the described way.
To get these working you either need to install the NVIDIA display driver in a way which uses the NVIDIA OpenGL implementation, or if that is not possible (e.g. because the system is a compute system without graphics subsystem, or under Windows when running devices in Tesla Compute Cluster (TCC) mode) you would need to disable the CUDA-OpenGL interop code paths completely.
Many of the OptiX SDK examples have a command line option --nopbo which disables the OpenGL pixel buffer object interop code path. The ray traced image will then be copied from device to host to whatever OpenGL implementation is running during a glTexture2D call. Search for that inside the OptiX SDK source code and you’ll find it.
Same OpenGL interop issues explained here:
https://forums.developer.nvidia.com/t/optixmotionblur-unknown-error/69602
https://forums.developer.nvidia.com/t/createbufferfromglbo-function-crash-in-multi-gpu-environment/62060
https://forums.developer.nvidia.com/t/optix-on-tesla-k40/57402
https://forums.developer.nvidia.com/t/optix-error-rtbuffercreatefromglbo-rdquo-caught-exception-hellip-error-cuclgetdevice/36698/5
If you’re running multiple OpenGL implementations in parallel, then the application code would need to be enhanced to pick the proper OpenGL implementation.
The OptiX SDK examples are not doing that but most likely pick the first OpenGL pixelformat which is matching irrespective of the vendor.
Means it might be possible to overcome that with a more sophisticated OpenGL pixelformat selection code, but that is not really the scope of the simple OptiX SDK examples.
That CUDA-OpenGL interop behavior is exactly the same for OptiX 7 versions, just that the buffer resource management happens explicitly with CUDA host calls.