HDR denoising variable has no effect

I’m trying to get the denoiser to work on CPU data (doing interop with dx12), and we’re using hdr images, but for some reason I’m getting weird artifacts which concentrate in the locations with the most light, making me think that the reason is the denoiser not using the HDR training data.

I’m using the following code for initializing the denoiser:

if (true)
			{

				rtPostProcessingStageCreateBuiltin(data.m_optix_context, "DLDenoiser", &data.m_denoiser_post_processor);

				rtPostProcessingStageDeclareVariable(data.m_denoiser_post_processor, "input_buffer", &data.m_denoiser_post_processor_input);
				rtPostProcessingStageDeclareVariable(data.m_denoiser_post_processor, "output_buffer", &data.m_denoiser_post_processor_output);

				RTresult res = rtPostProcessingStageDeclareVariable(data.m_denoiser_post_processor, "hdr", &data.m_denoiser_post_processor_hdr);
				if (res != RT_SUCCESS)
				{
					LOGW("Couldn't find optix variable");
				}

			}

			rtVariableSetObject(data.m_denoiser_post_processor_input, data.m_denoiser_input);
			rtVariableSetObject(data.m_denoiser_post_processor_output, data.m_denoiser_output);
			rtVariableSet1ui(data.m_denoiser_post_processor_hdr, 1);

and I’m getting the following result:
https://cdn.discordapp.com/attachments/494057771608834078/558320556223758336/hdr_artifacts.PNG

Tracking the ranges of color values on the screen, the brightest color channel is 381.1f, while the darkest is 0.0009f.

The variable setup looks ok.
The buffers are both float4?

Could you please provide more information on the system configuration?
OS version, installed GPU(s), VRAM amount, display driver version, OptiX major.minor.micro version, CUDA toolkit version used to generate the input PTX, host compiler version.

The HDR denoiser only exists in OptiX 5.1.0 and higher.

Could you try if the OptiX Advanced Sample optixIntro_10 works on your system?
That was implemented to show the HDR denoiser in OptiX 5.1.0.
https://github.com/nvpro-samples/optix_advanced_samples

If you’re using OptiX 6.0.0, the FindOptiX.cmake iside the OptiX Advanced Samples needs to be changed to find the OptiX 6 libraries because the names have changed to include the full version number.

Something like this.
(EDIT: I updated the OptiX Advanced Samples repository with that necessary change below and re-enabled the OptiX Introduction examples again.)

# OptiX SDKs before 5.1.0 named the library optix.1.lib
# Linux handles this via symlinks and doesn't need changes to the library name.
set(OptiX_version "1")
set(OptiX_Utility_version "1")
# None of the advanced examples is using OptiX Prime.
# set(OptiX_Prime_version "1")

# The OptiX library is named with the major and minor digits since OptiX 5.1.0.
# Dynamically find the matching library name by parsing the OptiX_INSTALL_DIR.
# This only works if the installation retained the original folder format "OptiX SDK major.minor.micro".
# We want the concatenated major and minor numbers if the version is greater or equal to 5.1.0.
# Since OptiX 6.0.0 all DLL names use the fully qualified version number major.minor.micro
if(WIN32)
  if(OptiX_INSTALL_DIR)
    string(REPLACE " " ";" OptiX_install_dir_list ${OptiX_INSTALL_DIR})
    list(LENGTH OptiX_install_dir_list OptiX_install_dir_list_length)
    if(${OptiX_install_dir_list_length} GREATER 0)
      # Get the last list element, something like "5.1.0".
      list(GET OptiX_install_dir_list -1 OptiX_version_string)
      # Component-wise integer version number comparison (version format is major[.minor[.patch[.tweak]]]).
      if(${OptiX_version_string} VERSION_GREATER_EQUAL "6.0.0")
        set(OptiX_version ${OptiX_version_string})
        set(OptiX_Utility_version ${OptiX_version_string})
        # set(OptiX_Prime_version ${OptiX_version_string})
      elseif(${OptiX_version_string} VERSION_GREATER_EQUAL "5.1.0")
        set(OptiX_version "")
        string(REPLACE "." ";" OptiX_major_minor_micro_list ${OptiX_version_string})
        foreach(index RANGE 0 1)
          list(GET OptiX_major_minor_micro_list ${index} number)
          string(APPEND OptiX_version ${number})
        endforeach()
      endif()
    endif()
  endif()
endif(WIN32)

OPTIX_find_api_library(optix ${OptiX_version})
OPTIX_find_api_library(optixu ${OptiX_Utility_version})
# None of the advanced examples is using OptiX Prime.
# OPTIX_find_api_library(optix_prime ${OptiX_Prime_version})

The buffers are both float4.

For my system configuration:
OS: Win10 1809
Installed GPUs: Intel HD graphics 530, Nvidia GTX 970M
VRAM amount: 3GB
Driver version: 419.35
Optix version: 6.0.0
Cuda version: v9.0
Visual studio compiler version: 1916, 64 bit debug mode

I haven’t been able to get the samples working, for some reason it isn’t capable of compiling the PTX files, even after manually pointing the Cmake CUDA, Optix and DevIL variables to their install locations.

Currently updating CUDA to version 10.1.105 to see if that fixes it.

That shouldn’t happen. I’m using them with OptiX 6.0.0 and CUDA 9.0 or 10.0.
Could you give some more information about what you mean with “it isn’t capable of compiling the PTX files”?

Are the files not found?
You cloned all OptiX Advanced Samples and used the top-most CMakeLists.txt in the CMake GUI?

Are there compilation errors?

I’m trying to get to a point where you either have a working example of an HDR denoiser running on your system, which would indicate something is incorrect inside your application, or if the OptiX example fails on your system as well, use that as reproducer in an OptiX bug report against your system configuration.

CUDA 10.1 is not on the OptiX 6.0.0 Release Notes list of officially supported CUDA versions.
OptiX 6.0.0 had been built with CUDA 10.0. Worth a try though.

BTW, when installing new CUDA toolkits, use the custom install and deselect all drivers inside that when there already is a display driver installed which supports that CUDA version on your machine. esp. on laptops.
If anything goes wrong (e.g. missing control panel), de-install the display driver and install the newest available display driver again.

Reinstalling didn’t work for compiling the samples. I’m getting the following error when compiling the PTX files:

Building NVCC ptx file lib/ptx/cuda_compile_ptx_generated_phong.cu.ptx
5>  CMake Error at cuda_compile_ptx_generated_phong.cu.ptx.cmake:245 (message):
5>    Error generating D:/NVIDIA
5>    Corporation/samples/optix_advanced_samples/Build/lib/ptx/cuda_compile_ptx_generated_phong.cu.ptx
5>

Edit: Updating CUDA also didn’t fix the hdr problems.

Damn, that’s due to the CUDA_HOST_COMPILER setting which is not finding the matching cl.exe location, because MSVS 2017 changed the installation folder hierarchy and FindCUDA.cmake is probably not aware of that.

The solution is simple though and part of this thread:
https://devtalk.nvidia.com/default/topic/1036401/optix/optix-installation-windows-cuda-9-2/post/5265364/#5265364

To be able to compile *.cu files which are not used in OptiX device code, but as native kernels (all OptiX Prime examples and the optixRaycasting example and for unknown reasons sutil) with MSVS 2017, I also needed to set the CUDA_HOST_COMPILER in the CMake GUI explicitly to the 64-bit cl.exe.
In my installation that was here:
“C:/Program Files (x86)/Microsoft Visual Studio/2017/Professional/VC/Tools/MSVC/14.14.26428/bin/Hostx64/x64/cl.exe”

Alright, thanks. I’ll try it, but the team has decided to build our own denoisers instead, since we’re using hybrid ray tracing anyway, and the thought is that the AI denoiser will probably be overkill.