Here is my summary of last few days as to the mem bandwidth.
Those are not laboratory tests and I’m not an expert in this area. But it gave me ideas on what I’m working with.
Hardware: NVIDIA GB10, 48 SMs, SM 12.1, 119.63 GiB LPDDR5X unified memory pool
Theoretical peak: 273 GB/s (LPDDR5X, 8533 MT/s)
Nodes tested: Spark 8 (single-node), Spark 5-8 (4-node cluster)
1. UMA Bandwidth Diagnostics — Idle Baseline (March 10)
Method
Custom CUDA bandwidth tests (bw_small.cu, 8MB buffers) and uma_safe_test (128MB buffers) on idle Spark nodes.
Results
| Test |
Copy (GB/s) |
Triad (GB/s) |
Notes |
| 128MB idle baseline |
229 |
230 |
Near theoretical peak |
| 8MB idle baseline |
191 |
97 |
L2 cache effects at small working set |
| Prefetch bandwidth |
225-273 |
— |
No-op when data already resident |
| Coherence test |
PASS |
— |
UMA coherence correct |
Key finding: GB10 achieves 84% of theoretical peak (229/273 GB/s) with large buffers on idle nodes. The UMA architecture is healthy — no hardware defects or misconfiguration.
2. Bandwidth Contention Under Inference (March 11)
Method
Ran bw_small.cu (8MB Copy/Triad, 50 reps) concurrently with active vLLM inference on Spark 8 (Qwen3.5-35B-A3B, TP=1, bf16).
Three-State Comparison
| State |
Copy GB/s |
Triad GB/s |
Drop vs Idle |
| Fully idle (no model) |
190.9 |
96.8 |
baseline |
| Model loaded, GPU idle |
187.0 |
82.1 |
-2% / -15% |
| Active decode (GPU 96%) |
86.8 |
39.1 |
-55% / -60% |
Key Observations
-
Static memory footprint has minimal impact: 65.53 GiB of model weights in memory causes only 2% Copy bandwidth reduction. The -15% Triad drop is TLB/cache metadata pressure, not bandwidth consumption.
-
Active inference dominates bandwidth: During decode, available bandwidth drops 55-60%. The inference kernels consume the majority of LPDDR5X bandwidth.
-
Bursty access pattern: Bandwidth during inference ranged widely (Copy: 80-162 GB/s). MoE expert routing creates high-pressure bursts followed by brief recovery windows.
-
Contention floor: ~80-90 GB/s Copy appears to be the floor for any active decode workload on GB10, regardless of model size. This represents gaps between GEMM kernel memory access bursts.
Cross-Model Comparison
| Model |
Active Params |
Memory/Node |
Copy BW During Decode |
| Qwen3.5-35B-A3B (TP=1) |
3B |
65.53 GiB |
~87 GB/s |
| Qwen3.5-397B-A17B (TP=8) |
17B |
~93 GiB |
~93 GB/s |
Despite 5.6x fewer active parameters, the 35B model creates nearly identical bandwidth contention. Bandwidth contention is driven by kernel access patterns, not model size.
3. Kernel Profiling — Option A vs Option B (March 11)
The Question
Given ~55% bandwidth drop during inference, two explanations were possible:
- Option A: Kernels efficient, using ~140-178 GB/s for useful work → hardware-limited
- Option B: Kernels inefficient, wasting bandwidth on misalignment, TLB misses, spills
Profiling Method
Tool: PyTorch Profiler via vLLM’s built-in --profiler-config
nsys was attempted first but cannot trace into vLLM’s EngineCore subprocess (spawned via Python multiprocessing.spawn, invisible to --trace-fork-before-exec)--enforce-eager used to disable CUDAGraphs for per-kernel visibility- 10 decode iterations profiled (delay_iterations=2 to skip warmup)
Kernel Breakdown (10 decode steps, 337.3ms total CUDA time)
| Category |
CUDA Time |
% Total |
| GEMM (cutlass WMMA bf16) |
188.6ms |
55.9% |
| MoE (fused expert kernel) |
94.4ms |
28.0% |
| GDN Attention |
17.1ms |
5.1% |
| Overhead (sort, norm, memcpy, etc.) |
37.2ms |
11.0% |
GEMM + MoE = 83.9% of all CUDA time — dominated by NVIDIA’s own highly-optimized cutlass kernels.
Top Kernels
| Rank |
Kernel |
Time |
% |
Calls |
| 1 |
cutlass WMMA tensorop bf16 GEMM |
141.7ms |
42.0% |
2000 |
| 2 |
fused_moe_kernel |
94.4ms |
28.0% |
800 |
| 3 |
cutlass WMMA tensorop bf16 GEMM (var) |
46.9ms |
13.9% |
310 |
| 4 |
fused_recurrent_gated_delta_rule_fwd |
17.1ms |
5.1% |
300 |
| 5 |
FlashAttention (flash_fwd_splitkv) |
0.7ms |
0.2% |
100 |
Bandwidth Utilization
Active parameters per token: 3B (MoE routes to expert subset)
Weight data per decode step: 3B × 2 bytes (bf16) = 6 GB
Decode step time: 33.7ms
Weight-read bandwidth: 6 GB / 0.0337s = 178 GB/s (65% of 273 GB/s peak)
Total traffic estimate (1.5x): ~267 GB/s (97% of 273 GB/s peak)
Verdict: Option A — Hardware-Limited
The inference is bandwidth-bound at the hardware limit. Evidence:
- 84% of CUDA time in optimized NVIDIA cutlass + fused MoE kernels — state-of-the-art for bf16 GEMM
- Kernel overhead minimal (11%) — no scheduling gaps, no excessive small kernel launches
- FlashAttention negligible (0.2%) — attention is NOT the bottleneck
- Per-step timing matches throughput exactly — 33.7ms = 29.7 tok/s (observed: 30.3 tok/s)
- Bandwidth utilization ~65-97% of theoretical peak
4. Compute Bandwidth Under Load — 4-Node Cluster (March 10)
Measured on Spark 5-8 during active Qwen3.5-397B-A17B inference (TP=4):
| Metric |
Value |
| Compute BW under load |
82-105 GB/s per node |
| Prefetch BW (data resident) |
225-273 GB/s |
| Coherence |
PASS |
| Aggregate cluster BW consumed |
~340-420 GB/s (4 nodes) |
The 82-105 GB/s residual bandwidth per node during 4-node inference is consistent with the single-node measurement (~87 GB/s), confirming the contention floor is per-node, not aggregate.
5. Performance Conclusions
Current tok/s Performance
| Configuration |
tok/s |
Bottleneck |
| 35B-A3B, TP=1, bf16, single node |
30.3 |
Memory bandwidth (178 GB/s weight reads) |
| 397B-A17B, TP=8, INT4, 4 nodes |
37.3 |
Memory bandwidth + NCCL overhead |
Fundamental Limits
- ~80-90 GB/s Copy is the contention floor for any co-located workload during active decode
- Bandwidth contention is structural — driven by kernel access patterns, not model size
- No software optimization can eliminate this — the memory controller is saturated by design
- Quantization is the highest-impact optimization — directly reduces the dominant cost (weight reads)
Test Tools & Artifacts
| Artifact |
Location |
bw_small.cu |
Custom 8MB CUDA bandwidth test |
| Bandwidth contention report |
DGX-Spark-Bandwidth-Contention-35B-vs-Idle.md |
| Kernel profiling report |
DGX-Spark-Kernel-Profile-35B-Analysis.md |
| Torch profiler trace |
vllm-trace.json.gz (25MB, viewable in perfetto.dev) |
| Raw profiler output |
vllm-profiler-raw.txt |
Environment
- vLLM v0.17.1.dev0+gb31e9326a (single-node) / v0.17.0 (4-node cluster)
- PyTorch 2.10.0a0+a36e1d39eb.nv26.01.42222806
- CUDA 13.0, Driver 580.126.09
- NCCL 2.28.9-1 SM121
- Profiler: PyTorch torch.profiler via vLLM
/start_profile API
