How can I train a custom denoise model with my own dataset and implement in OptiX?

I have my own rendered data and I want to train an OptiX model with this dataset to improve OptiX denoiser performance on this specific data.

I notice there is a function called optixDenoiserCreateWithUserModel which mentioned in NVIDIA OptiX 7.5 – Programming Guide. But when I check it in Optix SDK I find it is commented in OptiXDenoiser.h like

// Load user provided model if model.bin is present in the currrent directory,
 // configuration of filename not done here.
std::ifstream file( "model.bin" );
if ( file.good() ) {
    std::stringstream source_buffer;
    source_buffer << file.rdbuf();
    OPTIX_CHECK( optixDenoiserCreateWithUserModel( m_context, (void*)source_buffer.str().c_str(), 
     source_buffer.str().size(), &m_denoiser ) );

I have no idea why this function is deprecated, does OptiX stop supporting custom model?

That’s not really possible. That idea is very old and turned out to not be feasible in practice.

The OptiX denoiser implementation gets improved regularly which includes changes to the algorithm and its inputs.
Custom models would only work with a specific implementation which would limit possible changes to the algorithms and therefore would block its future development.
Also the required training process hasn’t been made available publicly, so you wouldn’t be able to train a custom model.
As such the optixDenoiserCreateWithUserModel() function isn’t really usable.

Note that the OptiX denoiser has multiple modes of which the newer AOV and upscale models got most algorithmic changes. If you hadn’t been using these, please given them a try.

The OptiX SDK contains that example named optixDenoiser you cited which takes multiple EXR input images and can switch between the different denoiser modes.

Mind that the denoiser implementation resides inside the display drivers and usually the latest release has most improvements, so you might want to try different versions.

Thanks for your detail reply! That’s a pity to give up this idea, perhaps I could try more mode as you said.