Starting a new thread as this is does not seem to be related to others thread issue… @aniculescu
Yikes, just updated to 7.6.0 with 7.0.0-1019-nvidia.
50-60% slowdown on prefill and t/s, 5x warmup times as well. I tested across several runtimes of Qwen3.8-Flash-Next, using INT4 and NVFP4 checkpoints, including @eugr_nv version… I am using tool-eval-bench to test.
Single Spark, kho=off
Load size of my goto model before 96Gib, after 110 Some load time examples
Dynamo bytecode transform time: 7.65 s
Vs new
Dynamo bytecode transform time: 15.52 s
Dynamo bytecode transform time: 0.29 s
vs
Dynamo bytecode transform time: 0.54 s
torch.compile took 29.22 s in total
vs
torch.compile took 46.05 s in total
health
OK
coherence
’ Paris. Paris is located in the north-central part of the’
prefill (TTFT on a ~8k-token prompt), then determinism + prefix-cache hit on the same prompt
10665 tok in 10.43s => 1022 tok/s prefill (cold)
same prompt again: 1.07s => prefix-cache HIT
first-token logprobs identical across runs: NO (stock top-k kernel? EXACT_TOPK=0)
decode (real answer, greedy — never use ignore_eos with this model)
400 tok in 10.79s => 37.1 tok/s decode (incl. TTFT)
health
OK
coherence
" Paris, which is also the country’s most populous city,"
prefill (TTFT on a ~8k-token prompt), then determinism + prefix-cache hit on the same prompt
10665 tok in 16.89s => 631 tok/s prefill (cold)
same prompt again: 2.21s => prefix-cache HIT
first-token logprobs identical across runs: NO (stock top-k kernel? EXACT_TOPK=0)
decode (real answer, greedy — never use ignore_eos with this model)
400 tok in 25.42s => 15.7 tok/s decode (incl. TTFT)
I have rebooted (3 times), cleared out system cache and temps files. Same results.
This seems to be something at the memory bus level, as I see similar speed issues across all model loads, and my RDP interface, UI and all typing now feels sluggish as well.
Suggestions? ( besides rolling back? ) Let me know if you need any config, log files or tests results I can provide…
The Dynamo bytecode timings in your report both roughly doubled.
Dynamo bytecode transformation is CPython-side, symbolic tracing and bytecode rewriting, with no kernel launches. You have 7.65 s going to 15.52 s and 0.29 s going to 0.54 s. torch.compile took is less clean because autotuning does touch the device, but the bytecode line gives you a CPU-side regression independent of CUDA kernel execution.
CPU clock is worth checking first, because on GB10 the cpufreq driver is kernel-dependent. Two Sparks here, both DGX OS 7.2.3, both performance governor on all 20 cores:
The cores are asymmetric, 10 at 2.808 GHz max against a 338 MHz minimum and 10 at 3.900 to 4.004 GHz against a 1.378 GHz minimum. A governor other than performance can leave the small cores at 338 MHz and the large ones at 1.378 GHz. The older kernel also reports per-core binned maxima (2.808 and 2.860, then 3.900, 3.978, 4.004) where the newer one reports two uniform values, so this part of sysfs does move between kernels.
Clock hypothesis is out. Your current frequencies are the maxima, and the cpufreq layer matches our 6.17.0-1026 box exactly: same cppc_cpufreq, same performance, same 10 at 2.808 and 10 at 3.900.
One cheap next reading is the device total; vLLM sizes from that rather than from free memory. request_memory in vllm/v1/worker/utils.py:537:
with a hard failure below it if free is under that. total_memory comes from torch.cuda.mem_get_info(). A different total under 7.0.0-1019 would put the same --gpu-memory-utilization on a different absolute allocation with no flag change.
Ours on 7.2.3, driver 580.159.03, idle:
torch.cuda.mem_get_info -> free 80.75 GiB, total 121.69 GiB
I have seen the same behaviour here. After upgrade loading DSv4F 0731 now takes exactly 50% longer on one cluster compared to my other that isn’t upgraded to 7.6.0
Device total is not it. Yours reads 121.69 and so does ours.
The Initial free memory line is not cudaMemGetInfo. On an integrated GPU vLLM substitutes a
different number, in vllm/utils/mem_utils.py, MemorySnapshot.measure():
The comment there names DGX Spark and links the CUDA for Tegra note on estimating allocatable
memory on an integrated GPU. That is current main; the same substitution is in v0.13.0. is_integrated reads 1 on GB10.
So that line is MemAvailable, which barely moves. Our box on 7.2.3, kernel 6.17.0-1026, driver
580.159.03, no model loaded, cache warmed first, the only action between samples being sync; echo 1 > /proc/sys/vm/drop_caches. cuda_free is torch.cuda.mem_get_info()[0]:
Cached plus Buffers fell 34.37 GiB, MemFree and cuda_free rose by about the same, and MemAvailable moved 0.19 GiB. SReclaimable moved 0.11 and is left out of the table for
width. Your 113.23 is close to the MemAvailable readings above.
Your free 1.67 was read with the server up; post #3 shows VLLM::EngineCore holding 97870MiB.
Symptom: Model memory footprint bloated to 110 GiB (leaving only 1.67 GiB free).
Cause:CONFIG_CMA_SIZE_MBYTES=0 in /boot/config-7.0.0-1019-nvidia leaves CmaTotal at 0 kB.
Fix: Add cma=128M kho=off to GRUB_CMDLINE_LINUX_DEFAULT and run sudo update-grub.
Result: Model load footprint dropped to 71.09 GiB, restoring 113.28 GiB of initial free memory.
2. GPU SM Clock Throttled to 650 MHz
Symptom:nvidia-smi dmon showed the SM clock plunging from 2054 MHz idle down to 676 MHz under 96% load, drawing only 7-13W with SW Power Capping: 22001492 us.
Fix: Override the floor clamp using sudo nvidia-smi -lgc 2418,2418.
Result: Prefill jumped from 711 tok/s to 4,715 tok/s.
3. The Power Delivery (PD) Controller Throttle Wedge (The 50% Interconnect Wall)
Symptom: System was still capped at an exact 2.0x latency wall (2.26s vs 1.13s prefill; 14.86s vs 7.65s Dynamo bytecode transform), and dmesg logged mlx5_core: Detected insufficient power on the PCIe slot (27W).
Cause: The DGX Spark’s onboard USB-C Power Delivery / Embedded Controller latched into a 30W fallback safety contract during the upgrade reboot, down-gearing the LPDDR5x unified memory bus to half speed. Software reboots (sudo reboot) do NOT clear this because the EC is powered continuously by the standby rail.
Fix:
sudo shutdown -h now
Physically unplug the 240W power brick from the DGX Spark and wall.
Hold the physical power button for 15-20 seconds to drain flea power from board capacitors.
Wait a full 60 seconds, reconnect, and boot.
4. Post-Fix Verification & Reset Clocks
Once the cold drain is complete and the full 240W power delivery is restored, reset any manual clock locks you tested back to the default dynamic auto-boost policy:
# Reset GPU clocks back to dynamic auto-boost default
sudo nvidia-smi -rgc
# Verify the application clock target and current clock
nvidia-smi -q -d CLOCK | grep -E "Graphics|SM"
Final Result:
Prefill: 10,402 tok/s (surpassing original 9,420 baseline)
Decode: 41.3 tok/s (fully restored from 14.1 tok/s)
Dynamo bytecode transform time & system-wide UI responsiveness fully restored.
Correcting my own posts above. In posts 2, 4 and 7 I gave DGX OS 7.2.3 for our machines. Only one of them is 7.2.3. The 6.17 box, which is the whole of posts 4 and 7, runs 7.5.0.
DGX_SWBUILD_VERSION in /etc/dgx-release still reads 7.2.3 on it, and an update does not touch that field. The script the dgx-release package runs on update, /usr/local/sbin/nv_scripts/write_release.bash, appends DGX_OTA_VERSION and DGX_OTA_DATE and leaves the DGX_SWBUILD_ fields alone. Both of NVIDIA’s own readers, get_dgx_sw_version.bash and the login banner, take DGX_OTA_VERSION when it is present and fall back to DGX_SWBUILD_VERSION when it is not. The first MOTD line on each box:
Welcome to NVIDIA DGX Spark Version 7.5.0 (GNU/Linux 6.17.0-1026-nvidia aarch64)
Welcome to NVIDIA DGX Spark Version 7.2.3 (GNU/Linux 6.11.0-1016-nvidia aarch64)
7.5.0 put 6.17.0-1014 on that box in April 2026, and the running 6.17.0-1026 arrived in a separate apt run in July. The 7.2.3 box shipped with 6.11.0-1014 and runs 6.11.0-1016.
Huh. I thought this had long since been fixed with PD firmware updates. I definitely haven’t seen it in months on Sparks (MSI and FE) that had regularly demonstrated it in the past.
In case it helps others landing here: the CMA fix alone did not save us. On a node that took the 7.6.0 OTA this morning, vLLM (single node, no NCCL) still dies at KV-cache allocation with NV_ERR_NO_MEMORY from _memdescAllocInternal + Xid 31, even after cma=128M restores CmaTotal to 128 MiB. And the OTA removed the 6.17 driver modules, so “just boot the old kernel” needs one extra package. Details, dmesg lines and the package that resolves against 580.178.04 are in the ENOMEM thread: DGX Spark regression: kernel 7.0.0-1019-nvidia causes NCCL/RoCE ibv_reg_mr_iova2 ENOMEM; 6.17.0-1032 works - #75 by vedcsolution
Correction to my previous reply: after more testing the vLLM boot failure I mentioned reproduces identically on kernel 6.17.0-1032 with driver 580.173.02, a raw 110 GiB allocation test passes on that node, and the same recipe boots with --enforce-eager. So that one is a CUDA-graph/vLLM issue on our side, not the 7.0.0-1019 kernel or the 580.178.04 driver. Details in the linked thread. The note about the OTA removing the 6.17 driver modules still applies.
It sounds like you did not update your system to include the latest hotfix which adds the kho=off parameter. Please make sure you are updated to the latest version.