After wasting my life f**king around with special vLLM containers and NVFP4 Nvidia files made with H100s in mind, i’ve found the super secret way to get 20-22 tok/s with Nemotron-3-Super-120B on a single DGX Spark (ASUS GX10 in my case).
Drum roll…
Just put LM Studio on your DGX and run a GGUF model. Yes, that’s right - running unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF · Hugging Face at Q4_K_M with KV Q8_0. Faster model load, simple interface, no issues - it just works. I’m not wasting my time with vLLM anymore.
I actually was tuning it for weeks, finding proper images to run it, to avoid OOM (always on 0.21+ due to cudagraph profiler exploding), in the end it was pretty good
model
test
t/s
peak t/s
ttfr (ms)
est_ppt (ms)
e2e_ttft (ms)
nemotron-3-super
pp512
421.39 ± 63.99
1101.96 ± 143.45
1099.01 ± 143.45
1101.96 ± 143.45
nemotron-3-super
tg128
12.05 ± 0.47
17.67 ± 0.47
nemotron-3-super
pp512
492.74 ± 52.24
973.43 ± 114.75
970.48 ± 114.75
973.43 ± 114.75
nemotron-3-super
tg512
16.04 ± 0.26
24.00 ± 0.82
nemotron-3-super
pp512
751.61 ± 31.62
643.20 ± 3.94
640.26 ± 3.94
643.20 ± 3.94
nemotron-3-super
tg2048
22.28 ± 0.54
34.00 ± 0.00
nemotron-3-super
pp512
753.73 ± 20.40
647.41 ± 14.96
644.47 ± 14.96
647.41 ± 14.96
nemotron-3-super
tg4096
22.07 ± 0.24
33.67 ± 0.47
nemotron-3-super
pp2048
1469.80 ± 11.14
1268.20 ± 29.65
1265.25 ± 29.65
1268.20 ± 29.65
nemotron-3-super
tg128
25.21 ± 1.02
31.67 ± 1.25
nemotron-3-super
pp2048
1457.36 ± 22.97
1250.24 ± 40.72
1247.30 ± 40.72
1250.24 ± 40.72
nemotron-3-super
tg512
21.19 ± 0.79
32.00 ± 1.63
nemotron-3-super
pp2048
1458.06 ± 31.83
1233.93 ± 38.60
1230.98 ± 38.60
1233.93 ± 38.60
nemotron-3-super
tg2048
22.89 ± 0.17
34.33 ± 0.47
nemotron-3-super
pp2048
1485.02 ± 12.52
1276.22 ± 23.76
1273.27 ± 23.76
1276.22 ± 23.76
nemotron-3-super
tg4096
22.60 ± 1.33
34.00 ± 0.82
nemotron-3-super
pp16385
2026.04 ± 1.48
7285.49 ± 67.63
7282.54 ± 67.63
7285.49 ± 67.63
nemotron-3-super
tg128
24.55 ± 0.91
31.00 ± 2.16
nemotron-3-super
pp16385
2029.09 ± 2.75
7275.06 ± 43.25
7272.11 ± 43.25
7276.08 ± 43.60
nemotron-3-super
tg512
22.55 ± 0.49
32.33 ± 1.25
nemotron-3-super
pp16385
2029.69 ± 1.07
7187.63 ± 10.29
7184.68 ± 10.29
7188.04 ± 10.75
nemotron-3-super
tg2048
23.76 ± 0.32
33.33 ± 0.47
nemotron-3-super
pp16385
2029.69 ± 1.83
7338.21 ± 41.32
7335.26 ± 41.32
7339.00 ± 41.29
nemotron-3-super
tg4096
23.27 ± 0.36
34.00 ± 0.00
nemotron-3-super
pp65536
1809.83 ± 4.88
32643.10 ± 95.04
32640.16 ± 95.04
32644.68 ± 92.85
nemotron-3-super
tg128
24.66 ± 2.50
30.67 ± 3.09
nemotron-3-super
pp65536
1821.34 ± 2.82
32550.35 ± 78.63
32547.40 ± 78.63
32550.35 ± 78.63
nemotron-3-super
tg512
22.51 ± 0.47
33.67 ± 1.25
nemotron-3-super
pp65536
1822.29 ± 2.26
32373.80 ± 153.28
32370.85 ± 153.28
32373.80 ± 153.28
nemotron-3-super
tg2048
24.19 ± 1.02
34.33 ± 0.47
nemotron-3-super
pp65536
1821.27 ± 4.73
32466.01 ± 38.20
32463.06 ± 38.20
32467.85 ± 40.13
nemotron-3-super
tg4096
23.65 ± 0.05
33.33 ± 0.47
nemotron-3-super
pp512 @ d4096
1815.94 ± 8.26
2315.46 ± 39.74
2312.51 ± 39.74
2315.46 ± 39.74
nemotron-3-super
tg128 @ d4096
22.08 ± 3.01
29.33 ± 2.36
nemotron-3-super
pp512 @ d4096
1819.03 ± 9.57
2321.26 ± 61.47
2318.31 ± 61.47
2321.26 ± 61.47
nemotron-3-super
tg512 @ d4096
21.97 ± 1.01
32.00 ± 0.00
nemotron-3-super
pp512 @ d4096
1805.40 ± 2.38
2254.67 ± 29.56
2251.72 ± 29.56
2254.67 ± 29.56
nemotron-3-super
tg2048 @ d4096
21.82 ± 0.61
32.33 ± 1.70
nemotron-3-super
pp512 @ d4096
1803.55 ± 1.73
2285.44 ± 55.46
2282.49 ± 55.46
2285.44 ± 55.46
nemotron-3-super
tg4096 @ d4096
20.63 ± 1.34
31.00 ± 1.63
nemotron-3-super
pp2048 @ d4096
1902.09 ± 5.19
2948.15 ± 30.14
2945.20 ± 30.14
2948.15 ± 30.14
nemotron-3-super
tg128 @ d4096
23.72 ± 1.88
30.33 ± 1.70
nemotron-3-super
pp2048 @ d4096
1887.61 ± 1.56
2850.63 ± 9.85
2847.68 ± 9.85
2850.63 ± 9.85
nemotron-3-super
tg512 @ d4096
21.94 ± 1.33
30.67 ± 1.25
nemotron-3-super
pp2048 @ d4096
1897.01 ± 5.86
2904.24 ± 21.56
2901.30 ± 21.56
2904.24 ± 21.56
nemotron-3-super
tg2048 @ d4096
20.73 ± 0.41
32.67 ± 0.94
nemotron-3-super
pp2048 @ d4096
1898.99 ± 5.98
2927.33 ± 52.62
2924.38 ± 52.62
2927.33 ± 52.62
nemotron-3-super
tg4096 @ d4096
21.18 ± 0.19
32.00 ± 1.63
nemotron-3-super
pp16385 @ d4096
1977.82 ± 2.25
9317.48 ± 94.83
9314.54 ± 94.83
9318.84 ± 94.01
nemotron-3-super
tg128 @ d4096
23.66 ± 0.83
30.33 ± 2.62
nemotron-3-super
pp16385 @ d4096
1983.45 ± 2.24
9372.40 ± 71.92
9369.45 ± 71.92
9373.69 ± 71.29
nemotron-3-super
tg512 @ d4096
22.70 ± 0.76
32.67 ± 0.94
nemotron-3-super
pp16385 @ d4096
1985.79 ± 1.58
9325.25 ± 84.33
9322.30 ± 84.33
9327.24 ± 84.52
nemotron-3-super
tg2048 @ d4096
21.17 ± 0.74
30.67 ± 0.94
nemotron-3-super
pp16385 @ d4096
1981.97 ± 0.98
9247.79 ± 38.52
9244.84 ± 38.52
9247.79 ± 38.52
nemotron-3-super
tg4096 @ d4096
21.55 ± 0.98
33.33 ± 1.70
After 12 hours today got pretty much same config running on cluster (no-ray), had to build a custom image off 0.20 version. But results are more than I expected.
