I would recommend using OptiX 7 for that.
This is more involved with earlier OptiX versions which do not have native support for DX interop, but there have been solutions posted in this sub-forum. You’ll find them when searching for “DX11” in the search field on the top right while being in the OptiX sub-forum topics view.
I haven’t done that with DX11 myself since I’m only using OpenGL and Vulkan, but the basic mechanism should be the same because this is all about native CUDA graphics interoperability functions.
Since all device memory handling happens with native CUDA host code in OptiX 7 versions, that is your best option.
You will only need to get that CUDA graphics interoperability working with your graphics API of choice and care about the proper synchronization.
You also need to pick the CUDA API. There are CUDA Driver and Runtime APIs.
(I prefer the CUDA Driver API due to the more explicit CUDA context handling which comes in handy with multi-GPU.)
Have a look through these links:
Many of the OptiX SDK and other example share an OpenGL pixel buffer object (PBO) to render to in OptiX before uploading it to the OpenGL texture object image for final display with a textured rectangle.
Have a look through this file of one of my OptiX 7 examples and search for the different CUDA OpenGL interop cases inside the m_interop clauses.
Except for the graphics API specific resource management, the remaining CUDA interop calls are graphics API agnostic.
You probably need the other direction as well when using a G-buffer from your raster engine to be able to determine from where to shoot rays for the reflections.
Note that DX12 DXR and Vulkan Raytracing would be the other APIs with which you could do raytracing, and that would be much more efficient than to bolt something onto a DX11 engine from a separate API, because the resources in the rasterizer and raytracer would simply be shared when using DXR or Vulkan RT.