Question about the construction time of multiple GASs when building IAS

Gaming and Visualization Technologies Visualization OptiX
1

Hi,

I am splitting my triangle mesh into multiple chunks and building an IAS over the resulting GASes. However, I observed that building many small GASes is significantly slower than building a single large GAS.

In my case, building a single GAS for a mesh with 1,264,847 vertices and 2,517,782 faces takes only 2.36 ms. In contrast, splitting the mesh into chunks of 65,536 faces and building a separate GAS for each chunk takes 18.32 ms, even when using multiple CUDA streams.

This is my code to build multiple GASes:

float build_time = 0.f;
CPU_TIMER_START(build_time);
for (int i = 0; i < num_chunks; i++) {
    CUstream stream = streams[i % num_streams];
    auto &entry = out_entries[i];
    entry.output_pooled = true;
    entry.buffer_sizes = buffer_sizes[i];
    entry.d_output_buffer = d_all_output + output_off;
    entry.d_output_size = buffer_sizes[i].outputSizeInBytes;

    OPTIX_CHECK(optixAccelBuild(state_.optix_context, stream, &accel_options,
                                &chunks[i].build_input, 1, d_all_temp + temp_off,
                                buffer_sizes[i].tempSizeInBytes, d_all_output + output_off,
                                buffer_sizes[i].outputSizeInBytes, &entry.handle, nullptr, 0));

    temp_off += buffer_sizes[i].tempSizeInBytes;
    output_off += buffer_sizes[i].outputSizeInBytes;
}
CUDA_SYNC_CHECK();
CPU_TIMER_END(build_time);
2

Generally I’m not surprised by that.
How many CUDA streams are your using? Anything greater than 8 is most likely not resulting in additional parallelism. Try 4 to 8 streams max.
Are you using build flags for fast build or fast trace?
Why are you splitting that rather reasonably sized GAS into smaller chunks?
Are you planning to update individual GAS later? There are different options for that in the OptiX API.

About the optixAccelBuild arguments:
Mind that both the temp and GAS output pointers must be aligned to OPTIX_ACCEL_BUFFER_BYTE_ALIGNMENT. Your code excerpt increments with sizes in bytes though, so make sure these are all aligned.

3

I set the number of CUDA streams based on the number of SMs; in this case, I use 21 streams. I also set buildFlags = OPTIX_BUILD_FLAG_PREFER_FAST_BUILD.

The reason I’m considering an IAS is that the mesh contains relatively large vertex coordinates, which lead to floating-point precision issues. To mitigate this, I split the mesh into multiple chunks and build a separate GAS for each chunk. The GASes are built only once and will not be updated afterward.

4

I’d recommend trying this with 8 streams. It might be faster than 21 streams. You should also compare to using 1 stream in order to understand the overheads associated with GAS splitting and many BVH builds. Using streams, you should be able to go faster with 8 streams than when using a split GAS and 1 stream. Another useful dimension for your test would be to change your split size and check build perf with 1 & 8 streams.

I don’t expect splitting your GAS to be as fast as building a single GAS. Splitting a GAS into pieces can help with precision issues, and it can also reduce the high-watermark memory requirements (especially if you are compacting your GASes). But splitting a GAS adds more kernel launches with smaller workloads, so it’s reasonable to expect to compromise on performance; you’re adding overheads and reducing GPU saturation.

FWIW, streams do not map to SMs. Having as many streams as SMs does not mean (for example) that each GAS build will run on 1 SM. Using multiple streams doesn’t even guarantee that any of your GAS builds will run in parallel*. CUDA kernels tend to prefer consuming the entire GPU whenever they can.

Streams are a way to specify dependencies between kernels, not necessarily a way to increase GPU parallelism. Putting kernels on separate streams is a way to declare that the work is independent, while putting them on the same stream declares essentially that they are dependent. If kernel B depends on kernel A, then kernel B has to wait until A is completely finished before B can start. If they are independent, then they can run in any order, and one can start before the other is finished. They can run at the same time, but streams do not cause nor enforce that.

Streams help fill the gaps whenever kernels aren’t consuming the entire GPU, for example when other things are running on some SMs as you launch a kernel, or when your kernel’s workload is ramping down near the end. Streams can help you blend or dovetail the work of independent kernels together, they might allow some extra parallelism when convenient, but they do not necessarily increase or even guarantee parallelism.

* If you want to try partitioning your GPU to guarantee overlapping parallel work, you could consider looking into Green Contexts. But even that wouldn’t necessarily make the GAS builds go faster, and might actually slow it down since you have to choose a static partitioning of the GPU.

–
David.

5

Thank you. I will run further comparison tests and determine suitable parameters.