You can easily determine whether a reflected ray hits another triangle by tracing the reflected ray, and attaching either an anyHit or a closestHit shader to it.

There is currently no way in OptiX (or DXR or Vulkan) to do either a spatial visibility query, or a K-nearest neighbors query, since those types of queries are outside the bounds of ray tracing. However, it’s very common to sample the nearby neighbors stochastically by tracing bundles of rays, that’s essentially what global illumination and ambient occlusion are doing.

If you can formulate your problem probabilistically, if you’re okay with point-sampling your visibility field and using Monte Carlo sampling and tolerating some amount of noise, then you should be able to use OptiX for your image theory work. If you want an exact analytic answer with no noise, then you may need to look into building your own space partition data structure that supports K-nearest neighbor queries, or something even more complicated (even if you have K-nearest neighbors, you still have to check visibility).

Have you looked into much research for solving visibility? I don’t know if there are more current developments in the field, but one place I can suggest to start from is “The 3D Visibility Complex” by Durand et al., and you could browse the web for it’s references & the citations of that paper. I think this is a fairly hairy problem, but I wish you good luck, and please let us know if you make progress using OptiX!