We are investigating a persistent GPU execution failure in our OptiX 9.1 path tracer, which uses a Vulkan viewport.
The key finding is that switching from OptiX-IR back to PTX appears to eliminate the crash entirely, while also improving rendering performance by approximately 6.5%.
We previously used PTX without issues and migrated to OptiX-IR when implementing Shader Execution Reordering (SER), following NVIDIA’s documentation recommendation.
Since that migration, we have experienced intermittent CUDA failures across multiple scenes. We have now identified a specific Bistro scene configuration and camera position that reliably reproduces the issue.
The failure only occurs when Windows application focus is lost
This is the most unusual aspect of the problem.
The exact same scene renders normally for extended periods while our editor remains focused. The crash occurs when we switch focus to another Windows application, leaving the viewport rendering in the background.
Our renderer has no application focus-dependent logic or separate background rendering path. Rendering continues identically whether the editor is focused or unfocused, with no intentional changes to frame limiting, GPU scheduling, synchronization or rendering behaviour when focus changes.
The failure requires both SER and Windows Hardware-Accelerated GPU Scheduling (HAGS) to be enabled.
-
SER on + HAGS on + OptiX-IR: Crashes when the editor is unfocused.
-
SER off: Stable, regardless of HAGS.
-
HAGS off: Stable, even with SER and OptiX-IR enabled.
-
SER on + HAGS on + PTX: Stable in our testing, including the previously reproducible conditions.
We have never reproduced this failure during normal focused viewport usage.
Although our Bistro scene provides a reliable reproducer, this is not isolated to one scene. We have encountered similar crashes elsewhere since introducing SER and OptiX-IR, but less predictably. Even changing the scene assembly can prevent the failure.
Environment
-
GPU: NVIDIA RTX 4090
-
OS: Windows 11 Pro 25H2
-
OptiX: 9.1.0
-
CUDA: 13.1 (nvcc 13.1.115)
-
Drivers: 610.88 and 617.42 Studio, both tested after reboot
-
Optimized shaders, default OptiX optimization, full GPU debug disabled
Reproduction results
In our reproducible Bistro scene, CUDA reports cudaErrorLaunchFailure (719) at cudaStreamSynchronize().
Nsight repeatedly reports Illegal Instruction Encoding, active PC 0x10, and Frame not in module, without identifying the offending shader instruction.
We isolated the lighting workload under matched defocus conditions:
| Lighting configuration | Result with IR + SER + HAGS |
|---|---|
| Sun/sky only | No crash through 10,127 samples |
| Analytical lights only | No crash through 6,573 samples |
| Mesh emission only | No crash through 5,338 samples |
| Analytical + mesh emission | Crashed at samples 1,770 and 2,308 |
Switching back to PTX, while retaining SER, HAGS, the same scene and camera position, has so far survived multiple application defocused runs of 5,000 or more sample.
OptiX-IR versus PTX
Beyond the stability issue, we are also seeing approximately 6.5% worse rendering performance with OptiX-IR compared with PTX in this mixed-lighting Bistro workload.
This leaves us questioning the practical benefit of moving to OptiX-IR for our renderer.
We originally migrated because NVIDIA’s documentation recommends OptiX-IR. However, our current results show:
-
PTX supports our SER implementation and appears stable.
-
OptiX-IR introduces a reproducible failure under specific Windows scheduling conditions.
-
OptiX-IR is also measurably slower in our current workload.
We understand that performance varies by workload and that OptiX-IR may offer optimization opportunities not available through PTX. We would appreciate clarification on what advantages we should expect from OptiX-IR with SER, and whether there are specific requirements or optimizations we may be missing.
SER implementation
Our camera path uses:
optixTraverse(...);
if (depth > 0)
optixReorder();
Hit hit;
optixInvoke(..., &hit);
// Material preparation and direct lighting
The hit-output pointer is created after reordering and packed into two payload words.
There is no explicit reorder inside direct-light evaluation. Shadow rays use optixTrace() with a separate typed payload. Hit/miss programs are shared through ordinary modules, and stack sizes are calculated using OptiX helpers.
Debugging attempts
We have spent considerable time investigating this, including:
-
Testing different NVIDIA driver versions and HAGS configurations.
-
Isolating analytical lighting, mesh emission and combined workloads.
-
Testing SER-enabled and SER-disabled pipelines.
-
Running Compute Sanitizer, Nsight VSE and GPU diagnostic builds.
-
Investigating CUDA synchronization and OptiX execution failures.
-
Resolving Vulkan validation findings without affecting the crash.
-
Comparing PTX and OptiX-IR builds from the same shader source.
Despite these investigations, we have been unable to obtain an actionable GPU fault location or identify a clear path towards the underlying cause.
We have retained detailed logs, shader identities, debugger screenshots and the reproducible scene configuration, but do not yet have a standalone reproducer.
Questions for NVIDIA
-
Are their known issue or potential interactions between OptiX-IR, SER and Windows HAGS? In particular, why would the same GPU workload fail only when Windows application focus changes?
-
What practical advantages should OptiX-IR provide over PTX for SER? Given the stability problems and approximately 6.5% performance penalty, we currently see no clear reason to retain OptiX-IR in our renderer.
-
How can we obtain an actionable fault location? Nsight reports Illegal Instruction Encoding, but neither it nor Compute Sanitizer has provided sufficient information to identify the underlying problem.
-
Are there specific SER execution, payload or compiler requirements we should investigate? We would appreciate guidance on a targeted diagnostic approach rather than further trial-and-error changes.
We also noticed a potentially related report involving OptiX-IR crashes that disappear when using PTX, although that concerns debug builds:
We would appreciate any guidance from NVIDIA’s OptiX team. At present, PTX appears to provide both better stability and performance for our SER implementation, and we are struggling to identify a technical justification for returning to OptiX-IR without understanding these issues.