Porting APP from Optix 3.8 (32 bit) to Optix 6.5 (64 bit) : Need some help, please

That’s most likely a known issue with bool types in payload structures.
We’ve seen this before:
https://forums.developer.nvidia.com/t/problem-with-turning-off-rtx-mode-on-gtx-1080/76550
https://forums.developer.nvidia.com/t/freeze-on-sync-after-launch/159425/4

Please try changing your TShadowRayData Shadowed member from bool to int or unsigned int and use 0 and 1 to set it instead.