Cannot read from out buffer on A100 GPU - OptiX 6.5

Hello everyone,

I’m porting an application from OptiX 5 to OptiX 6.5. The program runs fine on RTX 2080, but when I try it on A100 GPU, memory mapping doesn’t work properly for an out buffer. To be specific, I create the OptiX buffer for output, write there in OptiX ray generation program, and transfer data back to host memory:

optix::Context opx_context = optix::Context::create();
optix::Buffer out_buffer_optix = opx_context->createBuffer(RT_BUFFER_OUTPUT,
                                                           RT_FORMAT_FLOAT, n_rays);
opx_context["OUT_BUF"]->set(out_buffer_optix);

opx_context->launch(0, n_rays);

float *out_buffer_host = new float[n_rays];
memcpy(out_buffer_host, out_buffer_optix->map(), n_rays*sizeof(float));
out_buffer_optix->unmap();

In this example, out_buffer_host always ends up with all 0s, no matter what the buffer size is.
I’m sure that I actually write to the buffer in ray generation program because the output of rtPrintf(...) is correct:

RT_PROGRAM void MakeRays()
{
    OUT_BUF[r_idx] = r_idx;
    rtPrintf("out buf element: %.4f\n", OUT_BUF[r_idx]);
}

It’s worth noting that Transferring data into the input buffer works fine, all OptiX programs can access input buffer data.
Somewhat similar issue is present in OptiX 6.5 SDK samples. For example, optixHello outputs the following frame:
demo_frame (576.0 KB)

The display driver version is 450.119.04. I tried building with both cuda 10.1 and cuda 11, the results are all the same.

Can anybody help me with this issue?

Thanks in advance!

Kind regards,
Pavel

Would you be able to update the display drivers on the A100 configuration?

The OptiX core implementation resides inside the display driver since OptiX 6.5.0.
Means the first thing to try when inexplicable things happen, is changing the display driver.

According to your display driver version 450.119.04, that’s a Data Center/Tesla driver for Linux 64-bit with CUDA 11.0 support from May 4, 2021, which while being the newest release date, it’s not the newest driver branch. There are 460.73.01 driver with CUDA 11.2 support available as well.

Please keep using the CUDA 10.1 toolkit for compiling the OptiX PTX input code though, because that’s the CUDA toolkit version with which OptiX 6.5.0 was built (see OptiX release notes). The CUDA 11.2 toolkit might not work reliably for that.

Was the optixHello output PPM image from running the pre-compiled SDK examples or when building them yourself?
(I don’t know if the OptiX SDK 6.5.0 comes with pre-compiled examples under Linux, like it does under Windows.)

I’ll file an internal bug report.

Hi @droettger ,

thanks for the suggestion. Is 460.73.01 driver compatible with CUDA 11.0?

The PPM image was generated by optixHello that I compiled using cmake. In cmake configuration, I specified gcc-8 as a host compiler, and cuda 10.1 installation as CUDA toolkit root directory.

Is 460.73.01 driver compatible with CUDA 11.0?

Yes, CUDA drivers are backwards compatible.
The driver release notes only mention the maximum CUDA version supported.

Our quality assurance is not able to reproduce the corrupted optixHello image output on an A100 using the 450.119.04 drivers or some newer internal branch with either pre-compiled or gcc-8 and CUDA 10.1 built versions of the executable.

Could you add any additional information about the exact system setup and operating system configuration to see if there is any mismatch?
Otherwise there is nothing to be done about your issue.
This is the only report about such a behavior and nothing would work if this was a systematic error.

In any case, I would recommend porting to an OptiX 7 version, since that is the future proof way and will result in a more flexible and faster implementation. It’s quite some undertaking though since the host code basically needs a rewrite, but I think it’s worth it.
I have demonstrated the necessary host and device code changes by porting some of my own OptiX 5.1 based introduction examples to OptiX 7 versions here: https://github.com/NVIDIA/OptiX_Apps
Links to the old examples in the README.md.