OptiX 9.1 false-hit / 7.7 all-miss with CAD SubDMesh triangles — working standalone test

Gaming and Visualization Technologies Visualization OptiX
1

I’m building a GPU-accelerated sunlight analysis engine. My setup: RTX 4090, CUDA 13.3, Driver 610.62, Win11, VS2022, OptiX SDK 7.7 & 9.1 tested.

Problem: GPU occlusion rays exhibit opposite incorrect behavior in different OptiX versions:

  • 9.1: ALL rays report HIT (false positive — everything appears occluded)
  • 7.7: ALL rays report MISS (false negative — nothing occluded)

What works: A standalone test with a single hand-coded triangle (20x20m at z=1) produces correct hit/miss in BOTH versions.

What doesn’t work: Real CAD geometry extracted via Autodesk SubDMesh.GetObjectMesh — 84 triangles (building walls, roofs, floors). The BVH is built with OPTIX_GEOMETRY_FLAG_DISABLE_TRIANGLE_FACE_CULLING, single GAS + IAS with identity transform.

What I’ve tried (all failed):

  • | # | Fix Attempted | 9.1 Result | 7.7 Result |

    |—|---------------|-----------|-----------|

    | 1 | Face culling: `FLAG_NONE` (default) | All-hit | All-miss |

    | 2 | Face culling: `FLAG_DISABLE_TRIANGLE_FACE_CULLING` | All-hit | All-miss |

    | 3 | t_min: 0.001 → 0.1 | N/A (tested in 7.7 only) | All-miss |

    | 4 | Payload values: 3 → 2 | All-hit | All-miss |

    | 5 | BVH compaction: enabled → disabled | All-hit | All-miss |

    | 6 | Winding swap (indices 0,2,1) + FLAG_DISABLE | N/A | All-miss |

    | 7 | Winding swap + FLAG_NONE | N/A | All-miss |

    | 8 | Geometry scaled 10× (XY only) + FLAG_DISABLE | N/A | All-miss |

    | 9 | Sample point z-offset: 0.0 → 0.01m | All-hit | All-miss |

    | 10 | ClosestHit program added to hitgroup | All-hit | All-miss |

    | 11 | AnyHit t-filter (`hit_t < t_min` → `optixIgnoreIntersection`) | All-hit | All-miss |

    | 12 | Single-file PTX (all programs in one compilation unit) | All-hit | All-miss |

    | 13 | Pipeline flag: `ALLOW_SINGLE_GAS` → `ALLOW_ANY` | N/A | All-miss |

    | 14 | **Standalone test (1 large triangle)** | ✅ CORRECT | ✅ CORRECT |

    | 15 | **Empty BVH (triangle at infinity)** | ✅ CORRECT | ✅ CORRECT |

Full debug history: [link to docs/gpu-occlusion-debug.md]

Question: Are there known issues with OptiX 7.7/9.1 intersection precision for small CAD triangles (centimeter-scale)? Any recommendations for further debugging?

2

For your reference: I’m an architect with very little coding skill, the whole thing is achived through vibe coding (claude code) , Since this program was developed using vibe coding, it may contain some simple bugs in unexpected places.

3

The first thing to debug this further is to enable the OptiX validation mode and add an exception program which dumps any OptiX error codes while running the program. Search this forum for these things, there are links to example code inside many older threads.

If this happens in full debug mode, it makes sense to also test this in full release mode and vice versa, means module compilation and OptiX pipeline setup.
Also if this happens with PTX as module target, try OptiX-IR instead to eliminate possible CUDA compiler issues.

There is also example code for optimal visibility ray implementations using the terminate on first hit flag or the shader execution reordering function optixTraverse().

The problem could be something simple like wrong OptiX structure initialization, wrong shader binding table (SBT) setup, wrong SBT offset, wrong ray flags or [t_min, t_max] interval in optixTrace, wrong AS build flags, wrong payload initialization, wrong program compilation (including errors in CUDA).
That it behaves the opposite between two OptiX versions could point to a missing data initialization. Hard to say without seeing the whole code.

Now to your tests:

  1. CAD models often have inconsistent face winding. Keep face culling off for now.
  2. No need to disable face culling, default is off.
  3. You should know how big your scene coordinates are and not guess t_min values. If you shoot occlusion rays starting on surface geometry, make sure to offset the t_min or ray origin from the surface enough to not get self-intersections. The sticky posts on this forum contain a thread about a numerically optimal offset calculation inside the OptiX Toolkit on github.
  4. The number of payload registers must be known and set correctly depending on your OptiX device code. You cannot simply reduce it to 2 when it actually needs 3. If it only needs two, use two. For a binary visibility condition you would only need one payload register.
  5. Enabling BVH compaction alone does nothing when you’re not also compact the acceleration structure. This should not have any effect on the ray tracing result other than possibly increased performance with actual AS compaction.
  6. Triangle winding doesn’t matter as long as face culling is off. It matters if you calculate face normals which should always be defined on the front face. With OptiX geometry build flag defaults, use counter-clockwise winding in right-handed coordinate systems (same as in OpenGL).
  7. See 6.
  8. When only scaling X and Y, is there maybe a problem with Z coordinates? Again you should know exactly how big and where your scene elements are after loading the geometry. Make sure it’s where you expect it and in reasonable coordinate ranges.
  9. What do you mean with “sample point z-offset”? If you start the occlusion ray origin on a triangle surface, do you expect all triangles of the CAD model to be in the XY plane? The ray origin is inside world space coordinates inside the ray generation and closest hit program. For arbitrary triangles you would need to offset the ray origin along the face normal or use t_min > 0.0f to offset along the ray direction.
  10. Visibility rays usually don’t need closest hit programs. They can be implemented with an anyhit program or faster with the terminate on first hit flag and a miss program or optixTraverse. See earlier comments.
  11. There will never be a hit reported outside the [t_min, t_max] test interval, so this does nothing. When an anyhit program calls optixIgnoreIntersection() on all potential intersections, zero hits will reported and the ray always ends in the miss program when provided.
  12. Shouldn’t matter if there aren’t any issues with the module compilation. Please prefer OptiX-IR as module target, it provides better debug capabilities.
  13. ALLOW_SINGLE_GAS is a bug when using an IAS->GAS structure! That should use the TWO_LEVEL setting which is also the fully hardware accelerated path. ALLOW_ANY should also work correctly, though would not be necessary here and is usually slower.
  14. This points to a problem in your geometry setup or assumptions or ray data though.

As long as your coordinates are in reasonable floating point ranges, there shouldn’t be too many precision related issues. The 32-bit floating point uses a 23-bit mantissa, so things up to 8 million units can be distinguished. The best precision is inside the interval [-1.0f, 1.0f].

The optixAccelBuild() can emit the axis aligned bounding box (AABB) of an AS, which allows to check your scene size easily if you don’t have that information after loading the geometry. Again, search the forum for example code. The OptiX advanced examples (link inside the sticky posts) show all above mentioned things.

4

## Update: Solved — NOT an OptiX bug

After extensive debugging, the root causes were entirely on our side. OptiX 7.7 works correctly. Thank you for reviewing the test results and offered suggestion (standalone test, incremental validation, single-variable changes) was instrumental in isolating the real issues.

## Root Causes

### 1. GPU raytrace was never called in production (the “all-miss” mystery)

The production pipeline had a complete OptiX setup (context → module → pipeline → SBT → BVH) but the actual `optixLaunch()` call was **missing from the execution path**. The pipeline built the BVH on GPU, then ran CPU Möller-Trumbore ray-triangle intersection in a loop. The GPU was sitting idle.

**Why this fooled us**: All our instrumentation showed “GPU check: 0 hit / N miss” but the timeline buffer was being read as uninitialized memory (0xAA fill pattern), which the check code interpreted as “all miss.”

**Fix**: Injected `launch_gpu_raytrace()` wrapper call into `pipeline.cpp` execute path. GPU now does the actual ray tracing.

### 2. Debug experiment left in production code (the “all-hit” after fix #1)

An earlier test (#8: “Geometry scaled 10× + FLAG_DISABLE”) had left `*10` multipliers on BVH vertex coordinates:

```cpp

// pipeline.cpp — debug artifact

lv[i*3]=l.x*10; lv[i*3+1]=l.y*10; lv[i*3+2]=l.z;

```

After fix #1 made the GPU actually trace rays, the 10×-inflated geometry surrounded all sample points, causing every occlusion ray to hit. All-hit was the *correct* behavior given this input — just the wrong input.

**Fix**: Removed the `*10` scaling. BVH vertices now match ray origin coordinate space.

## What Actually Works

With both fixes applied, **OptiX 7.7 produces correct occlusion results** against real CAD geometry (12–336 triangles, meter-scale, varied orientations) with:

```

OptiX 7.7.0 + CUDA 13.3 + RTX 4090 (sm_89)

- Pipeline: numPayloadValues=1, traversableGraphFlags=ALLOW_SINGLE_LEVEL_INSTANCING

- Geometry: OPTIX_GEOMETRY_FLAG_DISABLE_TRIANGLE_FACE_CULLING

- BVH: identity transform IAS→GAS, OPTIX_BUILD_FLAG_PREFER_FAST_TRACE

- Ray: OPTIX_RAY_FLAG_TERMINATE_ON_FIRST_HIT, t_min=0.1f, t_max=1e6f

- Performance: 2.37M rays in 4ms (~590 Mrays/sec)

```

## Key Lessons

1. **Single-variable testing matters**: The standalone test *did* work correctly — we should have trusted that signal more and focused on what differed in the production path

2. **Production call path verification**: `grep optixLaunch` in the binary would have found the missing call immediately

3. **Clean up debug code**: The 10× scaling was test #8; it got committed and survived multiple rounds because GPU wasn’t running

## OptiX 9.1 Status

We reverted to 7.7 after the 9.1 false-positive issue and haven’t re-tested 9.1 with the fixed pipeline. The 9.1 all-hit behavior with identical setup to working 7.7 remains unexplained — it may have been a manifestation of the same coordinate bug (10× scaling making geometry even more inflated in 9.1’s slightly different intersection math), or a genuine regression. We’ll test 9.1 again and file a separate report if the false-positive persists.

## Thanks

Thanks you for the suggestions — particularly the advice to reduce payload values, use single-file PTX, and verify with a minimal standalone test. Those suggestions didn’t directly fix the bug, but they eliminated dozens of variables and tightened the production code significantly.