Green-context SM provisioning vs the primary context

Accelerated Computing CUDA CUDA Programming and Performance
,
1

green-context SM provisioning vs the primary context

Subject: Green contexts: SM reservation, memory-bandwidth QoS, and
allocation-API stalls between co-tenant contexts

Environment: RTX 5060 Ti (sm_120, 36 SMs), Linux x86_64, CUDA 12.8 toolkit,
driver r570-class, single process, driver API green contexts
(cuDevSmResourceSplitByCount minCount=8 → cuDevResourceGenerateDesc →
cuGreenCtxCreate → cuGreenCtxStreamCreate).

What we do: a latency-critical kernel chain (many small dependent kernels,
real-time pose estimation at 60 Hz) is submitted to a stream of an 8-SM green
context. Heavier batch-style kernels (image rendering) run concurrently — in
the device’s PRIMARY context in configuration (a), or in a second green context
provisioned with the complementary 28 SMs in configuration (b).

What we observe (%smid instrumentation + wall-clock, locked clocks):

  1. Primary-context grids execute on ALL 36 SMs, INCLUDING the 8 SMs
    provisioned to the green context (config a).
  2. In config (a) the latency chain’s tail collapses under a primary-context
    batch storm: p50 unchanged, p95 0.59 ms → 2.97 ms, max 5.25 ms.
  3. In config (b) — identical storm submitted into the complementary green
    context — the latency chain is essentially untouched (+1 us p50 at 100%
    duty; +8 us with a 1.3x-oversubscribed storm grid).
  4. Stream priorities (high for the latency chain, low for the storm) do not
    change (2) measurably.

Doc discrepancy (the core of the question): the current CUDA Programming
Guide feature survey states: “when a program creates a green context which
uses some set of SMs, other contexts on the GPU will not schedule thread
blocks onto the SMs allocated to the green context”, i.e. RESERVATION.
Our r570/CUDA-12.8 measurement shows median-clean but TAIL violations
(p95 0.59 → 2.97 ms under a primary-context storm; %smid shows primary
blocks on carved SMs at least while the green context is idle).

Questions:

  1. Is (1)/(2) intended behavior — i.e., green-context SM provisioning CONFINES
    work submitted to that green context but does NOT RESERVE those SMs against
    kernels launched in the primary context (or other non-green contexts) of
    the same process?
  2. If yes: is the recommended pattern for protecting a latency-critical task
    to place ALL other GPU workloads into green contexts over the complementary
    SMs, leaving nothing compute-heavy in the primary context?
  3. Is there any current or planned mechanism (short of MIG or MPS
    active-thread-percentage) to make a green context’s SM set exclusive /
    reserved device-wide within a single process?
  4. Do the CUDA 13.x workqueue-configuration APIs
    (cudaDevWorkqueueConfigScopeGreenCtxBalanced / wqConcurrencyLimit) change
    any of the above, and are they supported on consumer Blackwell (sm_120)?

Additional observation A — DRAM/L2 contention dominates once SMs are carved:
with the latency chain and the batch renderer each confined to disjoint green
contexts (8+8 SMs), the latency chain’s added cost under the concurrent
renderer is +3.1 ms and is INVARIANT to the renderer’s carve size (8/16/28
SMs); a LibTorch inference co-tenant in a third disjoint carve adds a further
+5.2 ms, also SM-invariant. I.e. after SM partitioning, ALL remaining
interference we measure is memory-system contention.

  1. Is there any current or planned mechanism for DRAM-bandwidth or L2
    partitioning at green-context granularity (an analog of MIG’s memory
    slices, but sub-context within one process) on workstation/consumer
    Blackwell — or is MIG on the professional parts the only bandwidth-QoS
    path?

Additional observation B — allocation APIs stall ALL other threads for the
caller’s full queue depth: in a multi-threaded process (each thread
submitting to its own green context), if thread B issues cudaMalloc/cudaFree
or cudaHostAlloc/cudaFreeHost while B’s context has a deep async queue
(~65 ms of enqueued kernels), EVERY other thread’s CUDA calls (including
plain kernel launches into disjoint green contexts) block until B’s queue
drains — we measure the victim’s p50 going from 0.58 ms to 66 ms, i.e. the
full work-unit. By contrast cudaDeviceSynchronize from B affects no one but
B, and deep queues / second submission threads per se are clean. (Practical
consequence we hit immediately: any library that stages transfers through
per-call cudaHostAlloc — e.g. a framework’s pageable device-to-host copy
path — periodically freezes a co-resident real-time task for ~100 ms.)

  1. Is this queue-drain-while-holding-global-state behavior of the allocation
    APIs expected and documented? Is there a recommended pattern beyond
    “pre-allocate everything and never call alloc/free at runtime”, and is
    per-context isolation of allocation-API serialization planned? This seems
    directly relevant to the green-context latency-isolation use case, since
    the victim task has no way to protect itself against a co-tenant’s
    allocation calls.

Minimal standalone repros (~150-200 lines each, no external deps) available
on request: reservation/storm matrix (green_probe2l/2m) and the
allocation-stall matrix incl. clean controls — burst depth, single long
kernel, deviceSync, D2H variants (green_probe2o).

2

This is expected behavior, and it comes down to how the hardware schedules, not just how SMs are provisioned.

Green contexts reserve a minimum SM count via cuDevSmResourceSplitByCount, but they do not create a hard partition at the warp-dispatch level. If your 8-SM green context’s kernels are latency-sensitive small grids, they can still wait behind large primary-context grids because:

- The CUDA work distributor fills SMs greedily within the eligible SM set.

- A long-running primary-context grid can occupy the same physical SMs until it yields, even if your green context has rights to some of them.

- Memory-bandwidth QoS and L2 cache carving are separate from SM reservation; without those, the primary context’s memory traffic can still stall your green-context kernels.

If you need true spatial isolation for the latency path, you currently need either:

- A single-context scheduler with CUDA streams + priorities, or

- MIG on data-center GPUs (not available on RTX 5060 Ti).

For RTX-class cards, the practical pattern is to keep the latency-critical work in its own process and let the OS/driver time-slice, rather than relying on green contexts to guarantee SM exclusivity. Green contexts help reduce head-of-line blocking but they are not a real-time partition.