Right, UCHAR3 and UCHAR4 pixel formats are not supported by the OptiX denoiser.
The OptiX Release Notes mention it and the online OptiX Programming Guide does not list the enums at all because of that, but they are still contained inside the OptixPixelFormat but using them will throw an unsupported error.
You would have to do this conversion yourself. You could simply implement a native CUDA kernel which does that operation.
If both buffers are on the device already and have the same dimensions, that should be straightforward.
I would also try the half format for better efficiency.
PS: Note that OptiX 7.1.0 has been released yesterday!
That changed one structure related to the denoiser to be able to get the required memory and the required overlap in pixels when implementing tiled denoising with a single call now. Also the OptiX denoiser performance has been increased.
That strictly requires the new R450 display drivers.