**GPU**: NVIDIA RTX PRO 4000 Blackwell
**Compute Capability**: 12.0
**CUDA**: 13.2
**Driver**: r535
—
### 1. Device Query
```
Total SMs: 70
minSmPartitionSize: 8
smCoscheduledAlign: 8
Memory: 24 GB GDDR7
Memory Bus: 192-bit
Peak BW: 672 GB/s
```
### 2. Exhaustive Partition Probe
I wrote a brute-force scanner (`cudaDevSmResourceSplit` with 2 and 3 partitions, stepping by `minSmPartitionSize=8`):
#### 2-partition (pri, cpi)
```
pri + cpi ≤ 48 → OK (15 combos)
pri + cpi ≥ 56 → FAIL, cudaErrorInsufficientResources (error 915)
Example:
pri=16 cpi=32 rem=22 OK (48 SM used, 22 reserved)
pri=16 cpi=40 rem=14 FAIL
pri=24 cpi=32 rem=14 FAIL
pri=8 cpi=48 rem=14 FAIL
```
#### 3-partition (p1, p2, p3)
```
p1 + p2 + p3 ≤ 48 → OK (20 combos)
p1 + p2 + p3 ≥ 56 → FAIL (36 combos)
Example:
p1=16 p2=24 p3=8 rem=22 OK
p1=16 p2=16 p3=16 rem=22 OK
p1=24 p2=24 p3=8 rem=14 FAIL
```
**Conclusion**: Regardless of partition count, total partitioned SMs cannot exceed **48 SM (6 × 8)**. The remaining 22 SM are permanently reserved by the driver and cannot be allocated to any partition. Maximum SM utilization: **48/70 = 69%**.
This 48 SM cap appears undocumented. The `minSmPartitionSize=8` requirement alone predicts at most `⌊70/8⌋ = 8` partition units = 64 SM. Yet only 6 units (48 SM) are usable. The remaining 2 units (16 SM) are unreachable.
### 3. DRAM Bandwidth Contention Between Green Contexts
Our real-time radar pipeline uses two Green Contexts:
| Stream | SM Partition | Workload |
|—|—|—|
| pri_stream | 16 SM | `kernel_rearrange` (data reorder, every 50μs) |
| cpi_stream | 32 SM | GEMM + cuFFT + element-wise PC (batch of 512 PRIs) |
**Observation**: When GEMM runs on cpi_stream (32 SM), `kernel_rearrange` on pri_stream (16 SM) experiences severe bandwidth degradation:
| Kernel | Alone | Concurrent (with GEMM) | Drop |
|—|—|—|—|
| `kernel_rearrange` | 70 GB/s | **37 GB/s** | -47% |
| GEMM (cuBLAS) | — | 229 GB/s | — |
| **Combined** | — | **266 GB/s (40% of peak)** | — |
Although total DRAM bandwidth is only at 40% of the 672 GB/s peak, `kernel_rearrange` loses nearly half its throughput. This suggests the DRAM scheduler **favors the larger SM partition** under contention — even when overall bandwidth is far from saturated. Pri=16 SM is the minimum needed to sustain 50μs deadlines. At pri=8 SM, `kernel_rearrange` drops to **26 GB/s** (-63%) and misses real-time deadlines entirely.
### 4. Questions for NVIDIA
1. **Is the 48 SM hard cap (6 partition units × 8) on 70 SM Blackwell documented anywhere?** The `minSmPartitionSize=8` suggests 64 SM should be available (8 units), but only 48 SM is achievable in practice.
2. **What is the intended QoS mechanism for DRAM bandwidth between Green Contexts?** The current behavior shows the larger partition dominates bandwidth, starving the smaller one even at low overall utilization. Is there a way to guarantee a minimum bandwidth share to each Green Context?
3. **Are there plans to expose DRAM bandwidth partitioning** (similar to MIG on datacenter GPUs) on workstation Blackwell GPUs?
4. **Is the `cudaErrorInsufficientResources (915)` for `pri+cpi > 48` expected behavior, or a driver bug?** The conditions under which this specific error is returned are not clearly documented.
—
**Reproduction code**: Available at request — brute-force SM partition scanner (`sm_probe.cu`, `sm_probe_3.cu`) and real-time radar pipeline with CUDA event timing.
**System**: Ubuntu 24.04, CUDA 13.2, NVIDIA RTX PRO 4000 Blackwell, driver r535.