Your descriptions sounds like you either didn’t properly initialize all your payload registers or something is stomping on them inadvertently.
Is anything reported by the OptiX validation mode when enabling that with a logger callback?
Links to example code here: https://forums.developer.nvidia.com/t/optixhello-embeded-in-new-application-run-in-release-but-not-in-debug-mode/253037/2
Is there any exception reported when enabling all OptiX exceptions?
When you do not have your own OptiX exception program the validation mode would insert one which prints the exceptions.
When you say you only write results inside the miss program, did you initialize all payload values to proper values in case the miss program is not reached?
I’d repurpose the miss function so that if a ray cast from the hit point toward the receiver is a miss, then the receiver is visible from the hit point.
Is that the miss program on the same ray type which reached that closest hit program?
Then you implemented a recursive algorithm and would need to protect against that with some flag which indicates that this is already a recursive optixTrace inside the closesthit program.
That would also require that you set the maximum recursion depth on the OptiX stack size calculation correctly.
In all these cases where you used different payload registers (22 and 31), have you also set the
OptixPipelineCompileOptions numPayload value to the proper count?
https://raytracing-docs.nvidia.com/optix8/api/struct_optix_pipeline_compile_options.html
There would also be newer display driver branches to be tested in case this is some error inside the OptiX/CUDA compilation. There are R555 display drivers released for your board.
Just to exclude issues with the glTF asset itself or the rather limited glTF loader routines inside the OptiX SDK examples, in case you based your code on those, there is also this GLTF_renderer in my OptiX Advanced Examples which handles more glTF features, in case you need a second opinion on the glTF file contents.