I cannot say what is happening with these code excerpts. That would require a minimal complete reproducer with all host and device programs involved.
1.) The rtTrace default RTvisibilitymask is RT_VISIBILITY_ALL anyway. You only set it because you do not use the default RTrayflags RT_RAY_FLAG_NONE but RT_RAY_FLAG_DISABLE_ANYHIT.
Do you have an anyhit program in that visibility ray?
If not, leave both arguments away and try again.
2.) If your visibility origins or destinations are located on geometry surfaces, it’s incorrect to shoot with the [t_min, t_max] interval from 0.0f to full length because that will be affected by self-intersection on either side if you’re not prohibiting that with some other robust method, like checking for the primitive ID which would require an anyhit program you don’t invoke on your visibility ray.
Please read this thread and esp. the comments I made about visibility ray intervals:
https://forums.developer.nvidia.com/t/anyhit-program-as-shadow-ray-with-optix-7-2/181312/2
(Since the OptiX 7 example code in there is a port of my OptiX 5.1 based introduction examples, you’ll find the same code for the old OptiX API when following the links in the README.md of that github repository.)
3.) You initialized these fields only in the second ray:
visibilityRayPayload.result.x=OPAL_DIFFRACTION_LOS;
visibilityRayPayload.result.y=1;
This incomplete code excerpt doesn’t explain how that is set for the first ray, or what is happening with these fields in any other program.
4.) Again, do you have an exception program setup and is there any exception thrown?
For example, your ray direction normalization code is not guarded against invalid values and could result in invalid rays.