(Solved) Multi InfoCount of vkCmdBuildAccelerationStructuresKHR : GPU time in Nsight shows short time, but long fence wait

Gaming and Visualization Technologies Raytracing
, ,
1

NsightGraphics_vs_System_CmdBuildTime
NsightGraphics_vs_System_CmdBuildTime2022×644 174 KB

(Self Solved)

by checking with Nsight Systems I was able to observe the command build time overhead that Nsight Graphics did not capture.
The difference was not actually in the fence wait, but rather in the command build itself.

Hello,

I am currently benchmarking BLAS build/update time with two different approaches:

  • Method A: Combine multiple geometries of the same model into a single BLAS.
  • Method B: Build a separate BLAS per geometry of the same model.

For Method B, I tested building about 2000 BLAS in a single vkCmdBuildAccelerationStructuresKHR call.
The command buffer and queue are flushed and empty before submission.
Below is the relevant code snippet:

// flushed command buffer
if (pGpuTimer) pGpuTimer->record(cmdBuffer, VK_PIPELINE_STAGE_ACCELERATION_STRUCTURE_BUILD_BIT_KHR, 0);

vkCmdBuildAccelerationStructuresKHR(
    cmdBuffer,
    numDynamicBlases, // ~2000
    dynamicBlasBuildingSets.buildGeometryInfos.data(),
    dynamicBlasBuildingSets.buildRangeInfosArray.data());

if (pGpuTimer) pGpuTimer->record(cmdBuffer, VK_PIPELINE_STAGE_ACCELERATION_STRUCTURE_BUILD_BIT_KHR, 1);

vulkanDevice->flushCommandBuffer(cmdBuffer, queue, false);

Issue:

  • In Nsight Graphics, the GPU time (between timestamps) shows only about ~1 ms difference.
  • However, the fence wait time on the CPU side is much longer — Method B shows 15 ms+ overhead compared to Method A.
  • I also verified this using Vulkan GPU timestamp queries, and the results are consistent (short GPU time, long fence wait).

Since the command buffer and queue are flushed just before this BLAS build, I don’t understand why such a huge discrepancy occurs between GPU timestamp/Nsight results vs fence wait time.

👉 My question:
How can this large gap between GPU timestamps and fence wait happen when building thousands of BLAS in a single vkCmdBuildAccelerationStructuresKHR call?
Is this expected due to driver internal job splitting or synchronization behavior, or am I misunderstanding how timestamps record build workloads?

Any insights or similar experiences would be very helpful.

Thanks!
A_GeometryPerCluster
B_BlasPerCluster