Would you be able to update the display drivers on the A100 configuration?
The OptiX core implementation resides inside the display driver since OptiX 6.5.0.
Means the first thing to try when inexplicable things happen, is changing the display driver.
According to your display driver version 450.119.04, that’s a Data Center/Tesla driver for Linux 64-bit with CUDA 11.0 support from May 4, 2021, which while being the newest release date, it’s not the newest driver branch. There are 460.73.01 driver with CUDA 11.2 support available as well.
Please keep using the CUDA 10.1 toolkit for compiling the OptiX PTX input code though, because that’s the CUDA toolkit version with which OptiX 6.5.0 was built (see OptiX release notes). The CUDA 11.2 toolkit might not work reliably for that.
Was the optixHello output PPM image from running the pre-compiled SDK examples or when building them yourself?
(I don’t know if the OptiX SDK 6.5.0 comes with pre-compiled examples under Linux, like it does under Windows.)
I’ll file an internal bug report.