There is a sphere intersection program inside OptiX SDK 7.2.0\SDK\cuda\sphere.cu which is shorter.
(I’m not sure why that SDK example is using the ray in world coordinate space. That is going to be slower when there are transforms over the geometry. The SDK example probably uses only a single geometry acceleration structure, means object space is world space.)
Intersection programs should normally work in object coordinate space.
It’s unclear why you’re doing any of the matrix transformations there. See links at the bottom for the recommended approach.
Mind that the attribute registers (as well as the payload registers) are unsigned int.
https://forums.developer.nvidia.com/t/mesh-artifacts-when-using-anyhit-for-transparency-optix-7/156600/9
Also read these threads for potential pitfalls in custom intersection programs. Know your coordinate space!
https://forums.developer.nvidia.com/t/objects-appearing-in-the-wrong-order-after-scaling/83884/7
https://forums.developer.nvidia.com/t/how-to-get-the-transform-matrix-of-an-instance/157125
https://forums.developer.nvidia.com/t/intersection-sphere-in-sphere-cu/58595