Optix anyhit global buffer writes not visible to subsequent reads

Gaming and Visualization Technologies Visualization OptiX
1

Hello,

I’m encountering unexpected behavior when reading/writing a global RWStructuredBuffer inside an OptiX anyhit program, and I’m not sure if this is intended or a bug.

I’m implementing a per-ray max-heap (stored in a global RWStructuredBuffer t_distances passed as a launch parameter) to keep the max_iters closest hits. Writes happen in anyhit until the buffer is full. Then buffer is heapified and subsequent hits are heap inserted into the heap. Raygen performs heapsort and integration.

The issue is that reads from the buffer inside anyhit sometimes return 0 even after a confirmed write, while the same logic works correctly in raygen (heapify happens there if max_iters hit are not reached).

For example,

///////////////////////////////////////
// Inside the anyhit kernel
///////////////////////////////////////
uint hit_count = get_payload0();
for (int n = 0; n < 2; n++)
{
    float t = ts[n];

    // Skip entry if behind ray start
    if (n == 0 && t <= cur_t) continue;

    if (hit_count < max_iters) // Simple insertion on the hit_count idx
    {
        // Fill phase
        if (r_ind == 107860) // just a debug ray that visually had issues
        {
            if (hit_count > 0) // avoid hitting illegal memory
                printf("before setting %d [%d]~%f [%d]~%f\n",
                        n, hit_count, t_distances[r_ind + hit_count * num_rays],
                        hit_count - 1, t_distances[r_ind + (hit_count - 1) * num_rays]);
        }
        t_distances[r_ind + hit_count * num_rays] = t;
        if (r_ind == 107860) // just a debug ray that visually had issues
        {
            if (hit_count > 0) // avoid hitting illegal memory
                printf("after setting %d [%d]~%f [%d]~%f\n",
                        n, hit_count, t_distances[r_ind + hit_count * num_rays],
                        hit_count - 1, t_distances[r_ind + (hit_count - 1) * num_rays]);
        }
        hit_count++;
        set_payload0(hit_count);

        // Once full, build heap in O(N)
        if (hit_count == max_iters)
        {
            heapify(r_ind, num_rays, max_iters);
        }
    }
    else // Heap insert
    {
        if (t < t_distances[r_ind])
        {
            heap_replace_root(...);
        }
    }
}

The (truncated) print output is the following:

before setting 0 [2]~0.000000 [1]~1.329237
after  setting 0 [2]~1.475692 [1]~1.329237
before setting 1 [3]~0.000000 [2]~0.000000
after  setting 1 [3]~1.697015 [2]~0.000000
before setting 0 [4]~0.000000 [3]~1.697015
after  setting 0 [4]~1.435285 [3]~1.697015

It looks like some writes are not visible to subsequent reads within the same ray, in a weird pattern. A similar behaviour is present inside heapify, causing the bug. I launch one thread per ray, so it’s not a race condition. Also, I’ve triple checked that I’m not accidentally accessing illegal memory.

Could this be due to caching, memory visibility rules that I ignored, or restrictions on global memory access in anyhit programs?

My setup is:

  • CUDA 12.8
  • RTX Pro 5000 Blackwell
  • Optix 9.1
  • Slang 2026.5.1

Thanks in advance!

Panagiotis

2

Hi @panpapantonakis, welcome!

Which driver are you using? We have had a couple of register stomp bugs in the recent past. If you’re not on the latest driver, that’s worth trying first.

What’s the interaction with Slang here? Is Slang used during the ray traversal in question where you’re seeing bogus reads?

I don’t immediately see any problems with the code. I would expect your buffer reads in anyhit to be correct, and I’m not sure, but I don’t think you should have to worry about caching or memory rules in this case.

What’s a typical value for num_rays; how far apart are the consecutive t values in t_distances in memory, in bytes? Is there a specific reason not to have them next to each other?

So if I understand correctly, the line
before setting 1 [3]~0.000000 [2]~0.000000
should read
before setting 1 [3]~0.000000 [2]~1.475692
?

Is this bug causing functional issues when collecting hits, or are the symptoms so far limited to printf debugging calls?

There are a couple of things I can think of trying, just to see if behavior changes. One would be to put uint hit_count = get_payload0(); inside the loop, so it’s re-read every iteration. This should not be necessary, of course, it’s just a test. Another would be to store the calculated pointer to your entry in t_distances in a local variable, and use that for the write and for the printf calls, and use something like *(ptr - num_rays) to access the previous (missing) entry in the array. My thinking here is to try to push the compiler to use the same base pointer for both current & previous entry, under the hypothesis that maybe what’s going on is the pointer ending up in two different registers and maybe the compiler is losing track of one of them.

Another option, of course, is to try using a debugger and step through the SASS for your anyhit program to see if you can identify the read instruction and check that the pointer+offset it’s using is correct. One thing that can be super helpful there is to use the PTX pmevent instruction, which will be visible in the compiled SASS (as “pmtrig” IIRC), so you can isolate some code you care about by fencing it with two different pmevents.

https://docs.nvidia.com/cuda/pdf/ptx_isa_9.0.pdf

I don’t know what this is doing, but it looks like an interesting algorithm. I’m curious to hear more about it, if you are able to share.

A couple of thoughts I have looking at this are:

  • You might want to collect your max_iters hits in local/stack memory and/or payload registers, and defer the global traffic until you call heapify.
  • I’m curious if you require max_iters in order to be able to heapify. If not, it might be worth investigating whether it makes sense to let the entire warp try to heapify whenever the first thread in the warp hits max_iters. The thought being maybe that could reduce warp divergence. Might be a stupid idea, just throwing it out there.

–
David.

3

Oh BTW the other things to check are:

  • Enable OptiX validation mode
  • Toggle PTX vs Optix-IR
  • Check debug & release builds

If any of those changes the behavior, that may help isolate the issue and/or help demonstrate where the bug is.

–
David.

4

A driver update, of course! I thought I have tried everything this past week, from changing version for Slang, OptiX and the CUDA toolkit to reading and fiddling with the PTX to make sure the transpilation and compilation were correct, but I omitted the driver!
I updated from R595 U1 (595.71) to the latest one. It fixed some things, but now, if I were to remove the -g2 (debug level) it would break again. A failing point that I identified was at the (heap) swap, that’s using a temporary variable. When the flag was present this worked. When it was removed it seemed like it was skipping the temporary variable and was just copying the content of y to x. Inspecting the generated .cu file for this particular part of the code, it seemed to me that Slang was not to blame and that it was, again, a problem further down the line.

So, I decided to downgrade to R580, as I have already downgraded to CUDA 12.8. Everything got fixed, independent of flags, optimisation levels etc. So yay!

Even though I consider it solved (debugging further sounds super interesting, but as I’m a bit tight on availability I’m good with just downgrading and going on with my work), I’d like to answer your questions.

Before answering, a bit about what I was doing. As I said, I build on top of another codebase. I have already heavily modified big parts of it, apart from the tracing. So I had a working solution, that I was feeling was too slow and there was room for improvement. A couple of weeks ago I decided to profile and optimise optixLaunch to check if my ideas would lead to better performance.

  • I knew that something was wrong with the code and not just printf because the renderings looked broken (in my case, as heapify wasn’t happening for some rays, the blending was in wrong depth order so it was very obviously messed up).
  • I had tried putting reading hit_count inside the loop and that fixed the problem when OPTIX_COMPILE_OPTIMIZATION_LEVEL_0, but not when it was on level 3. Also, a similar strategy of offloading local variables for buffer indexing to the payload inside heapify and heap_insert (for loop - while true recursion loop, respectively) didn’t work, so I was sure that that was rather a patch than a solution.
  • Like in the original code, I use Slang to write differentiable computational kernels and the OptiX ones. My cmake performs the .slang -> .cu -> .ptx (using slangc and nvcc) conversion, and OptiX modules are created using optixCreateModule. So yes, the function where the problems appeared was Slang. However, I had checked the generated cuda code and it looked fine.
  • num_rays vary greatly, from 512*768 = 393216 to multiples of that (in a multiple samples per pixel setting). I didn’t think a lot about the memory layout in this case, but this felt like it would lead to more coalesced accesses. Unfortunately NsightCompute doesn’t provide the nice breakdown of issues and potential improvements for OptiX as it does for other kernels (if this is wrongly set up from my part please tell me, because these recommendations are super helpful).

Regarding the optimisation suggestions, the original code is using the payload as a buffer to store ordered hits. The problem is that the payload is too limited (32) for the amount of overdraw you have in these primitive-based radiance fields settings. To ensure correct rendering with proper ray saturation:

  1. raygen was calling optixTrace
  2. anyhit stored the 16 closest hits by insert sorting the new hits (and their ids, skipped in the code above for brevity) , ignoring hits if they were added inside the buffer and truncating the ray’s tmax otherwise
  3. raygen performed integration and triggered another optixTrace cycle until an exit criterion of ray saturation or max iterations was reached

In a few words, each opixTrace was quick enough (tree truncation helps a lot) but you still had to find the 16 closest hits each time, which, with a very unlucky traversal, could be slow. So, my idea was to avoid multiple optixTrace calls by storing the max_iters (~200) hits and then processing (sorting and integration) them once in raygen. Overall, I hoped that the increased occupancy from using fewer registers (payload of size 1 down from 32) would overcompensate for using the slower global memory and for having a looser early exiting criterion (max_iters hits instead of windows of 16).

First implementation led to background bleeding into the foreground, because the traversal order is significantly different than the depth order and the overdraw is too high. So I realised I needed to store the max_iters closest hits, which led me to the max-heap solution (in theory heapify + heapsort is faster than the insertion sort of the original code). I tried that + the variant using local thread memory as you suggested + minimising warp divergence by always assuming that I have a heap and just performing heap_replace_root (and never heapify) and, unfortunately, I get at best similar performance :( .

Despite not working out and getting stuck because of the driver, I really enjoyed this optimisation adventure and I feel like I learned a lot! Thanks again for helping me out!

Panagiotis

5

I’m super glad you’re unblocked! I’m still nervous there could be a latent bug in the newer drivers though. Without you taking too much time, is there any hope of sending us a reproducer so we can investigate?

Your experience here sounds familiar to investigations we’ve done on Gaussian Splats. The 3DGRT paper discusses some of these issues. (https://research.nvidia.com/labs/toronto-ai/3DGRT/) The authors concluded that the fastest approach for them was using a loop to collect (via insertion sort) the closest ~16 hits with each ray, then trimming tmax and processing the collected hits, and then re-launching the ray with an updated/trimmed tmin. This does indeed occasionally have unlucky traversal ordering.

Heap sort is an interesting idea. For sure it is theoretically faster for large N, but I’m not sure about small N nor GPU divergence. A couple of other options include doing an insertion sort in payload registers, and using 1 pass of merge sort into your larger sorted memory buffer every time the payload registers fill up. Another option for warp-coherent sorting might be bitonic sort (e.g., https://www.sci.utah.edu/~wald/Publications/2019/rtgems/ParticleSplatting.pdf)

I’m guessing it might be worth checking if it’s faster to order your hit buffer so that consecutive hits in a thread are also consecutive in memory. Spreading the hits out using num_rays might thwart some of the caching, or coalescing, or both. Another thing to try, perhaps, is using local memory instead of global memory, e.g., declare an array on the stack in raygen and pass a pointer to it to your anyhit program. Local memory naturally coalesces similar memory offset access across threads in a warp. But note this is where using a predictable sort access pattern might make a big difference. For example, insertion sort in all threads can used coalesced access because the relative index is the same for all threads; a heap sort might escape coalescing and have a lot more data divergence, and then potentially lose to insertion sort for some N, where N is small in theory but might be large relative to how much per-thread space you have in practice.

We are interested to hear about it if you find any surprising and/or successful strategies you can share; please let us know if this turns into a publication.

Just food for thought, but please let us know if you have any bandwidth to chase down the memory access bug.

–
David.