I am investigating a repeatable CUDA scheduling delay between two processes on an NVIDIA GPU in TCC mode. I would appreciate insight into which NVIDIA driver or hardware scheduling layer could produce this behaviour, and what additional trace data would be useful.
Short version
Two CUDA processes, A and B, each execute a fixed chain of GPU operations at regular intervals. These are continuously repeating processing chains, not one-shot submissions.
While Process A is active, Process B can complete its H2D transfer promptly on the copy engine but then wait up to approximately one 2.104 ms scheduling interval before its first kernel starts. The exact residual wait varies with the relative phase of the two periodic chains.
Nsight Systems shows Process A’s GR context remaining resident while Process B’s H2D runs. The GR handover occurs only when Process B’s compute begins. A later compute burst in Process B can also wait without a second visible process or CUDA-context switch.
This looks less like context-switch execution time and more like a compute work-burst, channel, or front-end admission decision.
Systems
The main measurements were made on:
- NVIDIA RTX A4000; also reproduced on an Ada4000
- Windows 11 IoT Enterprise LTSC, build 26100
- NVIDIA driver 580.97; also reproduced on an R525 driver
- TCC mode, CUDA compute mode Default
- PCIe Gen3 x8
- Typical GPU utilization during reproduction: 7-9%
- SM clock remains at 1560 MHz with no reported throttle reason
Process A normally consumes data that a separate acquisition device writes directly into GPU memory over PCIe. Tests that removed this direct-to-GPU input path and substituted ordinary buffers did not remove the delay.
Primary observation
Process B repeatedly submits a 252,192-byte H2D followed by a CUDA graph. Across 3,218 complete graph invocations, the H2D-completion-to-first-kernel delay was:
p05 0.201 ms
p50 1.189 ms
p95 2.095 ms
p99 2.195 ms
range 0.098-2.224 ms
The unusual part is that the copy engine services Process B before the compute engine admits its first kernel. The delay is therefore not simply the duration of the PCIe transfer.
The problem is absent when Process A is not using the GPU. Other CUDA workloads placed in Process B show the same first-compute delay, so it is not specific to the original Process B algorithm.
A plugin integration moves the GPU part normally executed by Process B into Process A: Process A loads that implementation and hosts its CUDA work in Process A’s own context. This removes the cross-process/context boundary and eliminates the user-visible delay completely. It is an effective workaround, but I would still like to understand the underlying scheduling mechanism.
Two focused tests
1. Removing suspected work from Process A
We selected a suspected family of kernels in Process A and ran three otherwise equivalent arms:
- normal kernels;
- same-stream, one-thread no-ops in place of those kernels;
- the selected launches skipped entirely.
All three retained effectively the same admission distribution:
p50 p95 p99
normal 1.189 2.095 2.195 ms
one-thread no-op 1.148 2.095 2.195 ms
launch skipped 1.165 2.091 2.194 ms
The maximum empirical-CDF distances from the normal arm were 0.0329 and 0.0216. This rules out both the execution/resource use and the submission of that selected kernel family. Other regular work in Process A is sufficient to create the condition.
2. A follow-up CUDA computation in the same Process B context
We extended Process B’s regular chain so that one CUDA computation is followed by another. The follow-up stage has its own memory transfers and uses a new stream, but Nsight shows both stages on the same GPU and in the same CUDA contextId=1 (streams 14 and 16, see attached image).
The follow-up copies are serviced without a comparable delay, but its computation is again held for roughly 1-2 ms. The distribution is not numerically identical to the first delay, but it is of the same order and shows the same separation between prompt copy service and delayed compute admission. In 99.083% of frames there is no second RESTORE_START for Process B, and in 98.777% there is no context-switch event from any process. A fresh process or CUDA-context restore is therefore not required: the hold can recur at a later compute burst inside an already established context.
Supporting synthetic result
A separate two-process TCC reproducer shows a deterministic admission staircase when a victim context competes with a foreign context performing periodic copies. Its admission instants are spaced by 1049.09 us. An independently measured GR scheduling interval is 2104.157 us, so the staircase spacing is very close to half that interval.
A closed-form model predicted later out-of-sample experiments to single-digit microseconds. When the two streams are moved into one CUDA context, the cross-context staircase disappears and the work serializes instead.
The synthetic and product cases do not have the same incidence rate, so I do not assume they are identical in every respect. The common timing structure nevertheless points toward a TCC runlist/TSG/channel admission mechanism.
Causes ruled out or strongly constrained
- PCIe transfer duration or bandwidth: Process B’s H2D completes before its compute wait.
- Context save/restore duration itself: the later same-context hold normally has no second restore event.
- The original Process B algorithm: other CUDA work receives the same first-compute delay.
- The selected Process A kernel family: normal, no-op, and skipped-launch arms are equivalent.
- CPU fallback or execution on the wrong GPU: checked explicitly per frame in the added-stage test.
- Clock, power, or thermal throttling: clocks remain at rated boost with no throttle reason.
- GPU saturation or a persistent execution-rate reduction: utilization is low, and kernels run at their normal rate after admission.
- Cold cache or footprint effects: changing the intervening work and resource footprint did not explain the pre-execution hold.
- The separate device’s direct-to-GPU PCIe writes: removing that input path did not remove the delay.
- A third unidentified CUDA process: system-wide traces identify A as the foreign GR context and B as the victim.
Current interpretation
Cross-context contention from Process A creates a TCC scheduling regime in which copy-engine work from Process B may complete promptly while Process B’s compute waits for a later admission opportunity. The wait is not simply time spent saving or restoring a CUDA context, and it can recur for a later work burst inside an already established context.
The unresolved layer appears to be around the NVIDIA TCC driver/RM, CUDA channel or pushbuffer scheduling, TSG/runlist handling, CUDA graph submission, or the GPU hardware compute front end.
Questions
- In TCC mode, can separate streams or graph submissions within one CUDA context use different internal channels or work queues whose compute bursts require separate GR admission, even when Nsight reports the same CUDA
contextId? - What exactly does Nsight Systems’
GPU_CONTEXT_SWITCH_EVENTStrack on this hardware? Could an internal channel, TSG, or runlist transition delay compute without another visibleRESTORE_START? - Is the approximately 1049 $\mu$s staircase spacing, close to half the independently measured 2104 $\mu$s GR interval, recognizable as an Ampere/Ada TCC runlist, TSG, PBDMA, or copy/compute scheduling behavior?
- Which supported profiler, CUPTI activity, ETW provider, or driver trace would expose the relevant channel/runlist/front-end admission decisions?
- Are there known TCC issues in which copy-engine work proceeds while GR work from the same CUDA context remains unadmitted for approximately one scheduling interval?
- Is there a supported way to bound this latency, for example through context priority, stream priority, a different graph-submission pattern, or another documented scheduling API?
