Periodic 40–50 ms Kernel Launch Stall When Running Small Kernels on 5+ GPUs Concurrently

Accelerated Computing CUDA CUDA Programming and Performance
1

I am trying to understand a performance issue that I can consistently reproduce on one of our servers.

My application creates one CPU thread per GPU. Each thread controls a dedicated GPU, and there is no interaction or synchronization between different GPUs. Each thread repeatedly performs the following sequence:

  1. Launch a small CUDA kernel (execution time is approximately 12 ms).
  2. Synchronize the stream.
  3. Launch the next kernel immediately.

When running on up to four GPUs concurrently, everything works as expected. However, once five or more GPUs are active simultaneously, I start seeing periodic stalls.

For example, when all eight GPUs are running, they execute several kernels continuously and then all GPUs stop launching new kernels at nearly the same time. After approximately 40–50 ms, kernel launches resume simultaneously on all GPUs.

From Nsight Systems, I can see that once the stall ends, the newly launched kernels reach the GPU with only a few microseconds of launch latency. In other words, the delay occurs before the kernels are submitted to the GPU, rather than after submission.

Interestingly, if I increase the execution time of each kernel by tens of times (thereby significantly reducing the kernel launch frequency from the CPU), the problem disappears completely.

This makes me suspect that the bottleneck may be related to the host-side submission path rather than GPU execution itself. One possibility I am considering is that there is some driver-side or system-side throttling or scheduling mechanism that limits the rate at which kernels (or other CUDA work) can be submitted across multiple GPUs. However, I have not found any documentation describing such a mechanism.

Could anyone suggest what components I should investigate next? For example:

  • Is there any known driver-side throttling or fairness mechanism for high-frequency kernel launches across multiple GPUs?
  • Are there any Nsight Systems, CUPTI, or Linux tracing tools that can help determine where the CPU is blocked before the kernels are submitted?
  • Has anyone encountered similar behavior with many CPU threads driving multiple GPUs concurrently?

Any suggestions on how to narrow down the root cause would be greatly appreciated.

The application is running inside a Docker container provided by a cloud service provider. I do not have root access to the host OS, so my investigation is limited to the container environment.

docker OS:Ubuntu 22.04.4 LTS

Cuda in container(used to compile): Build cuda_13.0.r13.0/compiler.36260728_0

8 Graphic card: 4090

Output of nvidia-smi topo -m:

	GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7	CPU Affinity NUMA Affinity	GPU NUMA ID
GPU0  X  NODE NODE NODE	SYS	 SYS  SYS   SYS	 0-31,64-95	     0		        N/A
GPU1 NODE X   NODE NODE	SYS	 SYS  SYS	SYS	 0-31,64-95	     0		        N/A
GPU2 NODE NODE	X  NODE	SYS  SYS  SYS	SYS	 0-31,64-95	     0		        N/A
GPU3 NODE NODE NODE	 X 	SYS	 SYS  SYS	SYS	 0-31,64-95	     0		        N/A
GPU4 SYS  SYS  SYS	SYS	 X 	 NODE NODE	NODE 32-63,96-127	 1		        N/A
GPU5 SYS  SYS  SYS	SYS	NODE  X   NODE	NODE 32-63,96-127	 1		        N/A
GPU6 SYS. SYS  SYS	SYS	NODE NODE  X 	NODE 32-63,96-127	 1		        N/A
GPU7 SYS. SYS  SYS	SYS	NODE NODE NODE	 X 	32-63,96-127	 1		        N/A

Snapshot of nsys:

image
image1252×864 38 KB

2

I cannot recall ever having experienced such “hiccups” in multi-GPU kernel submission, but then I haven’t tried eight GPUs in a single box.

Based on the description I concur that the problem seems to be on the host side, and it appears to be associated with a resource shared between all GPUs. This becomes congested at higher load, and “clears” once it has been “drained” by consumer (GPU) side activity. But the behavior observed does not seem to match what one would expect to observe from the two most common types of such mechanism: (1) global lock, (2) shared finite buffer (unless the buffer has hysteresis). The duration of the hiccup may suggest the bottleneck could be related to operating system time slices, as I would expect anything directly related to hardware to occur with microsecond rather than millisecond duration.

One could check whether dtrace or similar tool shows anything interesting around the time these hiccups occur, but I have no specific hypothesis regarding root cause and thus no specific recommendation what to look for.

That’s the tricky part. If you could somehow compare the behavior to an identically configured platform running with a bare-metal environment, that would make identifying the root cause easier. For example, I wonder whether the culprit could be a mechanism configured by the cloud provider to ensure equitable access of all virtual clients to the physical GPUs in the system.

Another thing I would be personally itching to try is the same setup but with a faster CPU, where by faster I mean a CPU with higher single-thread performance, which to first order means high operating frequency (my standing recommendation is >= 3.5 GHz).

3

Can it have to do with NUMA? As 4 GPUs are on each NUMA partition.
Do you know anything about the host configuration? Hardware? Operating system?

4

I cannot think of any mechanism of action that would cause an increase in latency due to NUMA to snowball into activity gaps in the tens of milliseconds.

5

It having to do with NUMA was just a possible guess.

For it to have effects into the tens of ms range, the high-level software architecture could be the actual culprit, if it uses the 4+4 NUMA partitioning.

With some misguided optimization, if requests from the other NUMA partition arrive.

Or another possibility: The number of queues was chosen by the number of GPUs in the current NUMA partition instead of the overall number of GPUs.

6

I wrote a CPU-only test program and observed the same behavior. It appears that the cloud provider enforces a CPU usage limit for each container. Once the container exceeds its CPU quota, it gets throttled. After switching the synchronization to a blocking mechanism, the issue disappeared and everything worked as expected.