I’m using OptiX 5.1.1 with a GTX 760 (driver 441.87, Windows 10) and I frequently get the following error while executing denoising:
Details: Function “_rtCommandListExecute” caught exception: Failed to launch DLDenoiser post-processing stage. DLDenoiser run method failed with error -XXXX
OptiX is used just for denoising, there are no kernels/ray tracing taking place.
The internal error codes mentioned in the message are usually -27 or -4711. Is there any documentation available on those?
Sometimes it seems that it’s probably the system’s watchdog causing the crash. Is there any way to make OptiX put less stress on the GPU while denoising? A potentially relevant function I noticed is setTimeoutCallback() but I saw in an older forum thread that it doesn’t have any effect.
Error codes for the OptiX 5 denoiser aren’t well documented, sorry about that. The newer versions of the denoiser are better integrated into the OptiX error codes.
I can see in the code that error -27 occurs when you’re out of memory, and -4711 is a general failure.
You may be able to watch your GPU high-watermark using nvidia-smi. There are several ways to query memory usage, perhaps try the flag --query-accounted-apps=max_memory_usage.
If memory is running out, or the watchdog is stopping your process early, in either case probably the simplest way to resolve the issue is to break your image into smaller pieces and denoise each piece separately. You can do it explicitly, or if you’re not already using the denoiser’s maxmem parameter, you can set it to cause the denoiser to do the splitting for you.
I’m not sure how much information I can get from smi about this, querying for max memory usage or memory utilization returns a “Not Supported” message in my case.
I did some testing using a couple of cudaMemGetInfo() calls placed exactly before and after the execute() call and no other gpu-intensive code or process running. Although I use the maxmem parameter, the actual amount of memory used for the same image dimensions appears to vary. Many times it is less that the specified budget and other times it exceeds it. In an extreme case it went about 200 MB over the budget leaving practically zero memory available on the device. I’m probably going to try explicit tiling.
Could you please also tell me what error codes -1 and -28 are about? I have come across those in OptiX 6.
Ah, sorry nvidia-smi didn’t work, it’s possible you just need to enable the device counters and run nvidia-smi under elevated priviliges. But cudaMemGetInfo() should do the job even better, I would think. Just for posterity, an OptiX alternative is rtContextGetAttribute() with the attribute flag RT_CONTEXT_ATTRIBUTE_AVAILABLE_DEVICE_MEMORY.
Error code -28 is also a memory allocation failure, and -1 is another general failure, documented in internal/optix_declarations.h as RT_ERROR_UNKNOWN.
Explicit tiling will at least give you explicit control on your upper bound memory usage, which might help. I would hope that maxmem did the trick but there are a variety of things than can go wrong even if maxmem is respected. Are you running the denoiser on your display GPU, or is it isolated?
If I remember correctly, you’ll want to use a gutter size of at least 32 pixels when tiling yourself (meaning your tiles need to overlap, and you’ll trim the seams when stitching back together), and I believe for optimum memory usage it’s best to use tiles that are powers of two in size.