My units started to consistently crash after 1-2 benchmark runs. I began to suspect bad thermal paste coverage on the GPU/CPU so I went ahead and looked. After checking it out, I can confirm that the factory-applied thermal paste appears degraded and is cracked in both of my units. I believe this has resulted in uneven coverage across the GPU/CPU surfaces, likely contributing in reducing effective heat transfer and contributing to the overheating issues.
While I don’t recommend disassembling your units, if you happen to, I would recommend 1mm thermal pads and whatever thermal paste of personal preference to replace the factory paste.
Cleaned the Front with compressed air.
Having it stand on the Side with an external fan in front.
Survived a night of lama-benchy full tests on all models.
Thanks for all the detail and for sharing your stack write‑up — that’s super helpful context.
1. RAM / freezes
On Spark the 128 GB of LPDDR5X is a single pool that both the OS and GPU workloads share, so if the model + KV‑cache + framework overhead grow close to 128 GB the system can become unresponsive rather than cleanly throwing OOM.
A few things you can try:
Leave more headroom in your auto‑gmem logic. Even if CUDA reports ~90 GiB “free”, there’s additional host‑side overhead (framework, runtime, page tables, file cache, other containers, etc.). Instead of targeting ~0.64, try something more conservative like 0.45–0.50 and see if the freezes go away. Also consider lowering max context length and batch size on the largest models.
Enable swap on the NVMe (if you haven’t already). A 64–128 GB swapfile on the internal SSD won’t make OOM events fast, but it often turns “hard lockup” into “slow but survives”, which is much nicer for a basement box.
Tighten Linux memory accounting so you fail allocations instead of hanging:
Set vm.overcommit_memory=2 and something like vm.overcommit_ratio=90 in /etc/sysctl.d/ so userspace gets allocation failures before the kernel is out of options.
After changing, reboot and watch dmesg/journalctl -k during a run; you want to see clean OOMs, not GPU driver threads stuck for >120s.
Optional safety net: some folks run a tiny “memory guard” systemd service that periodically checks /proc/meminfo and, if MemAvailable drops below a floor (e.g. 4–8 GB), logs a warning and kills the top RAM‑using process. That’s not mandatory, but it’s a simple way to keep the box from becoming completely unresponsive when experimenting with aggressive settings.
If you have a log from a freeze (previous boot), grabbing the last ~200 kernel lines from journalctl -b -1 -k would help confirm whether it’s a GPU allocation path or generic host OOM.
2. Thermals / 95 °C reboots
Per the DGX Spark docs, the platform is designed for continuous high‑load operation in an ambient of roughly 5–30 °C, with an integrated thermal management system. Hitting an ACPI sensor value of ~95 °C and then rebooting means you’re crossing a platform‑level thermal safety limit, so the shutdown is protective rather than a random crash.
A few practical checks:
Ambient and placement
Make sure the room itself stays well under 30 °C, especially once summer hits.
Keep the chassis on a hard, open surface with several cm of clearance on all sides, no soft material or walls blocking the vents, and no other hot gear immediately around it. A big fan in front helps, but if the hot air has nowhere to go it will still recycle.
Software / firmware level
Ensure you’re on the latest DGX OS / firmware bundle for Spark (latest public release has a number of stability and telemetry improvements). If you’re comfortable sharing, the output of uname -a and the Spark OS version from the dashboard would be useful.
Under‑load telemetry
While running a heavy Qwen3.5 prompt (before it reboots), capture:
sudo nvidia-smi -q -d TEMPERATURE,POWER
sensors
This helps see whether the GPU is thermally throttling as expected or the system is instead hitting a hard platform limit.
As an advanced experiment, you can try slightly capping GPU clocks to reduce peak power draw during your longest runs, e.g.:
sudo nvidia-smi -rgc # reset any previous limits
sudo nvidia-smi -lgc 0,2300 # example: cap max graphics clock a bit below default
If that turns reboots into clean, long‑running benchmarks, it’s a strong hint that you’re right on the edge of a power/thermal guard rail, and that data is very helpful for the team that’s tuning Spark’s curves.
3. What would help debug further
If you’re up for it, the following from a failing run would really help:
Exact Spark OS / kernel version and whether any firmware updates have been applied.
journalctl -b -1 -k | tail -n 200 after a thermal reboot.
The memory and temperature telemetry mentioned above from a run that pushes the box close to failure.
With that we can line your repro up against our internal testing and advise whether you’re hitting a known limit vs. something we should treat as a bug on your particular unit.
thanks for the hint.
For anyone who likes to know how to do and undo that here are the commands.
# Reset any previous limits
sudo nvidia-smi -rgc
# Cap max graphics clock (example: 2300 MHz)
sudo nvidia-smi -lgc 0,2300
I am more a fan of overclocking than “under clocking” - not sure if that is even a term -
will try to improve the airflow and cooling first before slowing down the machine.
Hello Neill, thanks for the detailed reply. I will go throuh point by point.
Yes that is a good idea. What really confused me is that if the DGX Dashboard shows 126.5GB Used the system suddenly gets super slow, Mouse and keyboard inputs are delayed massively.
Up to the point that I have ask myself it the RAM is defect or not all 128GB are usable.
Sofar I reduced the “aim” to use the RAM from 128GB down to 126GB
I am new to Linux so no I did not create a swap file so far. Have ask the AI how to do it and thats what it came up with. Will see how it goes. As some here like wentbackward recommended not to use a swap file at all.
sudo fallocate -l 64G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
# Make it persistent across reboots:
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
Will do that later maybe if the other changes did not solve the issue.
the thing is I am a big fan of Docker cause I can mess in the containers and it does not influence the OS.
Now when I “mess” with the OS it can have side effects later and cause issues and I am certain I will have forgotten what I had setup here.
Just for completion the AI stands this would be the command
echo "vm.overcommit_memory=2" | sudo tee /etc/sysctl.d/99-spark-oom.conf
echo "vm.overcommit_ratio=90" | sudo tee -a /etc/sysctl.d/99-spark-oom.conf
# Apply the changes without needing an immediate reboot:
sudo sysctl -p /etc/sysctl.d/99-spark-oom.conf
Sounds interesting
I mite do that later
Here is the code - AI Generated / HAVE NOT TRIED IT!
4. Optional: Set Up a Memory Guard Service (Version 1) If you want the “safety net” script Neill mentioned, here is a simple bash script and systemd service that checks for available memory and kills the largest consumer if it drops below 5 GB.
Create the script:
sudo nano /usr/local/bin/mem-guard.sh
Paste this inside:
#!/bin/bash
# Minimum available memory in kB (5 GB = 5000000 kB)
MIN_MEM=5000000
while true; do
AVAILABLE_MEM=$(grep MemAvailable /proc/meminfo | awk '{print $2}')
if [ "$AVAILABLE_MEM" -lt "$MIN_MEM" ]; then
echo "WARNING: MemAvailable dropped to ${AVAILABLE_MEM} kB! Killing top RAM process." | logger -t mem-guard
# Find the PID of the process using the most memory and kill it
TOP_PID=$(ps -eo pid,%mem --sort=-%mem | awk 'NR==2{print $1}')
kill -9 $TOP_PID
fi
sleep 5
done
The Spark is sitting on my Desk behind my Monitor.
The room temperature is usually around 21°C (69.8° F)
There is about 10cm (4 Inch) clearance around the Spark
What I noticed and mentioned before is that the design pattern on the front is really like a filter and gets clogged with dust.
So far its running fine
nvidia-smi -q -d TEMPERATURE
==============NVSMI LOG==============
Timestamp : Mon May 4 22:56:13 2026
Driver Version : 580.126.09
CUDA Version : 13.0
Attached GPUs : 1
GPU 0000000F:01:00.0
Temperature
GPU Current Temp : 52 C
GPU T.Limit Temp : 45 C
GPU Shutdown T.Limit Temp : N/A
GPU Slowdown T.Limit Temp : N/A
GPU Max Operating T.Limit Temp : 0 C
GPU Target Temperature : N/A
Memory Current Temp : N/A
Memory Max Operating T.Limit Temp : N/A
uname -a
Linux dgx-spark 6.17.0-1014-nvidia #14-Ubuntu SMP PREEMPT_DYNAMIC Tue Mar 17 19:01:40 UTC 2026 aarch64 aarch64 aarch64 GN
U/Linux
dgxtop
=== System Information ===
Platform: Linux-6.17.0-1014-nvidia-aarch64-with-glibc2.39
rchitecture: aarch64
Python Version: 3.12.3
CPU Cores: 20
Memory Total: 121.69 GB
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.126.09 Driver Version: 580.126.09 CUDA Version: 13.0 |
+-----------------------------------------+------------------------+----------------------+
I can not see the Spark OS version from the dashboard - but I have clicked “Update” each time it appeared. Where else can I see the Spark OS version, and how can I update SMI Diver to a newer one and also CUDA to a newer one.
sudo nvidia-smi -q -d TEMPERATURE,POWER
==============NVSMI LOG==============
Timestamp : Mon May 4 23:13:13 2026
Driver Version : 580.126.09
CUDA Version : 13.0
Attached GPUs : 1
GPU 0000000F:01:00.0
Temperature
GPU Current Temp : 52 C
GPU T.Limit Temp : 44 C
GPU Shutdown T.Limit Temp : N/A
GPU Slowdown T.Limit Temp : N/A
GPU Max Operating T.Limit Temp : 0 C
GPU Target Temperature : N/A
Memory Current Temp : N/A
Memory Max Operating T.Limit Temp : N/A
GPU Power Readings
Average Power Draw : 16.14 W
Instantaneous Power Draw : 15.81 W
Current Power Limit : N/A
Requested Power Limit : N/A
Default Power Limit : N/A
Min Power Limit : N/A
Max Power Limit : N/A
Power Samples
Duration : Not Found
Number of Samples : Not Found
Max : Not Found
Min : Not Found
Avg : Not Found
GPU Memory Power Readings
Average Power Draw : N/A
Instantaneous Power Draw : N/A
Module Power Readings
Average Power Draw : N/A
Instantaneous Power Draw : N/A
Current Power Limit : N/A
Requested Power Limit : N/A
Default Power Limit : N/A
Min Power Limit : N/A
Max Power Limit : N/A
sensors
mt7925_phy0-pci-90100
Adapter: PCI adapter
temp1: N/A
acpitz-acpi-0
Adapter: ACPI interface
temp1: +58.4°C
temp2: +50.6°C
temp3: +51.6°C
temp4: +51.6°C
temp5: +58.4°C
temp6: +51.8°C
temp7: +53.4°C
nvme-pci-40100
Adapter: PCI adapter
Composite: +49.9°C (low = -273.1°C, high = +82.8°C)
(crit = +84.8°C)
Sensor 1: +53.9°C (low = -273.1°C, high = +65261.8°C)
Sensor 2: +49.9°C (low = -273.1°C, high = +65261.8°C)
and whether any firmware updates have been applied.
fwupdmgr get-history
NVIDIA NVIDIA_DGX_Spark
│
├─Embedded Controller:
│ │ Device ID: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
│ │ Previous version: 0x02004e18
│ │ Update State: Success
│ │ Last modified: 2026-04-29 13:37
│ │ GUID: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
│ │ Device Flags: • Internal device
│ │ • Updatable
│ │ • System requires external power source
│ │ • Supported on remote server
│ │ • Needs a reboot after installation
│ │ • Reported to remote server
│ │ • Device is usable for the duration of the update
│ │ • Signed Payload
│ │
│ └─DGX Spark EC FW Embedded Controller Update:
│ New version: 0x03000302
│ Remote ID: lvfs
│ Release ID: 139394
│ Summary: DGX Spark Embedded Controller Firmware Update
│ License: Proprietary
│ Size: 519.3 kB
│ Created: 2026-04-02
│ Urgency: High
│ Tested by NVIDIA:
│ Tested: 2026-04-10
│ Distribution: ubuntu 24.04
│ Old version: 0x02004b03
│ Version[fwupd]: 1.9.34
│ Vendor: NVIDIA
│ Duration: 30 seconds
│ Release Flags: • Trusted metadata
│ • Tested by trusted vendor
│ Description:
│ This update improves the performance and stability of the Embedded Controller in DGX Spark
│ Checksum: XXXXXXXXXXXXXXXXXXXXXX
│
├─UEFI Device Firmware:
│ │ Device ID: XXXXXXXXXXXXXXXXXXXXXXXXXXX
│ │ Previous version: 0x0200941a
│ │ Update State: Success
│ │ Last modified: 2026-04-29 13:37
│ │ GUID: XXXXXXXXXXXXXXXXXXXXXXXXXX
│ │ Device Flags: • Internal device
│ │ • Updatable
│ │ • System requires external power source
│ │ • Supported on remote server
│ │ • Needs a reboot after installation
│ │ • Reported to remote server
│ │ • Device is usable for the duration of the update
│ │ • Signed Payload
│ │
│ └─DGX Spark SoC FW System Update:
│ New version: 0x0200980f
│ Remote ID: lvfs
│ Release ID: 139396
│ Summary: DGX Spark SoC Firmware Update
│ License: Proprietary
│ Size: 30.4 MB
│ Created: 2026-04-02
│ Urgency: High
│ Tested by NVIDIA:
│ Tested: 2026-04-28
│ Distribution: ubuntu 24.04
│ Old version: 0x0200941b
│ Version[fwupd]: 1.9.33
│ Tested by NVIDIA:
│ Tested: 2026-04-15
│ Distribution: ubuntu 24.04
│ Old version: 0x0200941a
│ Version[fwupd]: 1.9.34
│ Tested by NVIDIA:
│ Tested: 2026-04-10
│ Distribution: ubuntu 24.04
│ Old version: 0x02009009
│ Version[fwupd]: 1.9.34
│ Vendor: NVIDIA
│ Duration: 30 seconds
│ Release Flags: • Trusted metadata
│ • Tested by trusted vendor
│ Description:
│ This update improves the performance and stability of the System-on-Chip Firmware including UEFI and GPU in DGX Spark
│ Checksum: XXXXXXXXXXXXXXXXXXXXXX
│
└─UEFI Device Firmware:
│ Device ID: XXXXXXXXXXXXXXXXXXXXXXXXXX
│ Previous version: 0x00000507
│ Update State: Success
│ Last modified: 2026-04-29 13:38
│ GUID: XXXXXXXXXXXXXXXXXXXXXXXXXX
│ Device Flags: • Internal device
│ • Updatable
│ • System requires external power source
│ • Needs a reboot after installation
│ • Reported to remote server
│ • Device is usable for the duration of the update
│ • Signed Payload
│
└─(null) Update:
New version: 0x00000516
Description:
The vendor did not supply any release notes.
The crash log is pretty long so I gave it to an AI to analyse it - you can open the content here.
2. Analysis of Your `crash_log.txt`
The log you provided captures the exact moment the system began to fail and initiated a shutdown sequence. Here is the critical sequence of events:
The Container Teardown The vast majority of the log (from 21:08 to 21:10) shows Docker virtual ethernet interfaces (veth) and network bridges entering disabled states. This means your containers were actively crashing or being killed.
The GPU Driver Crash At 21:10:51, there is a critical NVIDIA driver assertion failure:
NVRM: nvAssertOkFailedNoLog: Assertion failed: Requested object not found [NV_ERR_OBJECT_NOT_FOUND] (0x00000057) returned from pRmApi->Control( pRmApi, hClient, hDevice, NV0080_CTRL_CMD_INTERNAL_MEMSYS_SET_ZBC_REFERENCED, ¶ms, sizeof(params)) @ mem_mgr_gm107.c:283
This indicates the GPU memory manager (mem_mgr) encountered a fatal error. An object it expected to find in memory was missing (NV_ERR_OBJECT_NOT_FOUND). This perfectly aligns with Neill’s theory about shared RAM/GPU memory exhaustion causing the driver to panic and fail rather than throwing a clean Out-Of-Memory (OOM) error.
The System Shutdown Immediately after the GPU memory manager crashes, the file systems unmount and the hardware watchdog takes over at 21:11:03, forcing a system shutdown:
systemd-shutdown[1]: Using hardware watchdog 'SBSA Generic Watchdog'... Sending SIGTERM to remaining processes...
I just had the OOM ~300Mhz/~1000Mhz clock limiter today hit hard today. It was my fault, trying to compile while running inference and I saw the temp was high and then the swap went to 100% and I didn’t catch it fast enough. Bluetooth keyboard lagged on wake up.
Anyway I did the power down, unplugged the Spark after a minute and booted up, but no dice. Still at ~300Mhz/~1000Mhz. Did it again. Still ~300Mhz/~1000Mhz. Unplugged again and went and watched a movie, came back and and its still ~300Mhz/~1000Mhz.
I started thinking I really cooked it this time, but figured I would spool up vllm, send a ‘Hello’ prompt and see what happened. Clock sputtered about for a bit and then jumped to 2434Mhz/~3003Mhz. I am getting too old for this.
Thought I would report my experience here in case someone else runs into the same situation. Seems like you need to run something that hits the GPU hard to actually see if the clock is stuck or not. It was, now its not.
I was having similar issues and manage to limit to only power.
vLLM 0.20 was the culprit in my case. I use Qwen3.5 122B and thought 0.20 might perform better. It was leaking memory under my workload - crashes on the internals which accumulate to use all the memory available after a day. I then had to - still - fight with the thermals. Using 4 concurrent long sessions caused 95C and power down. The speed cap was 2400 and survived more than 12 hours. I am now setting it to 2300. I hope it will solve the issue.
Thanks for the additional details. We’re reviewing similar DGX Spark stability reports internally, and based on the symptom pattern this does not appear to be attributable to temperature alone at this stage.
To help narrow this down, could you please clarify the following:
When the issue happens, does the system:
power off completely,
hang / become unresponsive,
or come back after reboot but in a degraded-performance state?
If performance is degraded after the event, does a normal reboot recover it, or is a full AC removal from the power brick required?
Please share the last relevant logs captured close to the event:
These are especially helpful for separating a memory-pressure path from a power-state issue on DGX Spark’s unified-memory architecture.
Please run Field Diagnostics and share the PASS/FAIL result plus the generated logs. NVIDIA documents Field Diagnostic as the recommended health check / pre-RMA check for DGX Spark hardware.
Then in UEFI go to Security → Secure Boot → Disable Secure Boot, save, and reboot.
Run the diagnostic:
sudo init 3
cd /opt/nvidia/dgx-spark-fieldiag
sudo ./partnerdiag --field
This switches the system to TTY mode and runs the full test suite. It typically takes about 30 minutes and ends with a PASS/FAIL banner. It can also be run over SSH using the same commands.
If the run is interrupted, please power-cycle the system before rerunning the tests.
After the diagnostic completes, re-enable Secure Boot in UEFI.
The kernel / OS details already posted in-thread are sufficient for now: Ubuntu 24.04.4 LTS on kernel 6.17.0-1014-nvidia. The most useful next step is the symptom detail above plus the Field Diagnostic result, so we can narrow whether this is primarily a memory-pressure issue, a power-state issue, or a hardware fault.
When the issue happens, does the system:
hang / become unresponsive
Can not say much about the degraded-performance state - I remove the AC sometimes when that happens to make sure it works better.
while the system is stuck
journalctl -b -1 -e Jun 30 12:32:09 dgx-spark systemd[1]: Unmounted snap-steam-245.mount - Mount unit for steam, revision 245. Jun 30 12:32:09 dgx-spark systemd[1]: snap-whatsapp\x2delectron-18.mount: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Unmounted snap-whatsapp\x2delectron-18.mount - Mount unit for whatsapp-electron, revision 18. Jun 30 12:32:09 dgx-spark systemd[1]: tmp-RustDesk-cliprdr\x2dclient.mount: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Unmounted tmp-RustDesk-cliprdr\x2dclient.mount - /tmp/RustDesk/cliprdr-client. Jun 30 12:32:09 dgx-spark systemd[1]: tmp-RustDesk-cliprdr\x2dserver.mount: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Unmounted tmp-RustDesk-cliprdr\x2dserver.mount - /tmp/RustDesk/cliprdr-server. Jun 30 12:32:09 dgx-spark systemd[1]: Stopped target snapd.mounts-pre.target - Mounting snaps. Jun 30 12:32:09 dgx-spark systemd[1]: systemd-fsck@dev-disk-by\x2duuid-9DA2\x2d3597.service: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Stopped systemd-fsck@dev-disk-by\x2duuid-9DA2\x2d3597.service - File System Check on /dev/disk/by-uuid/9DA2-3597. Jun 30 12:32:09 dgx-spark systemd[1]: Removed slice system-systemd\x2dfsck.slice - Slice /system/systemd-fsck. Jun 30 12:32:09 dgx-spark systemd[1]: run-snapd-ns-firmware\x2dupdater.mnt.mount: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Unmounted run-snapd-ns-firmware\x2dupdater.mnt.mount - /run/snapd/ns/firmware-updater.mnt. Jun 30 12:32:09 dgx-spark systemd[1]: Unmounting run-snapd-ns.mount - /run/snapd/ns... Jun 30 12:32:09 dgx-spark systemd[1]: run-snapd-ns.mount: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Unmounted run-snapd-ns.mount - /run/snapd/ns. Jun 30 12:32:09 dgx-spark systemd[1]: Stopped target local-fs-pre.target - Preparation for Local File Systems. Jun 30 12:32:09 dgx-spark systemd[1]: Stopped target swap.target - Swaps. Jun 30 12:32:09 dgx-spark systemd[1]: Deactivating swap swap.img.swap - /swap.img... Jun 30 12:32:09 dgx-spark systemd[1]: Deactivating swap swapfile.swap - /swapfile... Jun 30 12:32:09 dgx-spark systemd[1]: Stopping lvm2-monitor.service - Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling... Jun 30 12:32:09 dgx-spark systemd[1]: Stopping multipathd.service - Device-Mapper Multipath Device Controller... Jun 30 12:32:09 dgx-spark multipathd[872]: multipathd: shut down Jun 30 12:32:09 dgx-spark systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Stopped systemd-tmpfiles-setup-dev.service - Create Static Device Nodes in /dev. Jun 30 12:32:09 dgx-spark systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Stopped systemd-tmpfiles-setup-dev-early.service - Create Static Device Nodes in /dev gracefully. Jun 30 12:32:09 dgx-spark systemd[1]: multipathd.service: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Stopped multipathd.service - Device-Mapper Multipath Device Controller. Jun 30 12:32:09 dgx-spark systemd[1]: swapfile.swap: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Deactivated swap swapfile.swap - /swapfile. Jun 30 12:32:09 dgx-spark systemd[1]: swap.img.swap: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Deactivated swap swap.img.swap - /swap.img. Jun 30 12:32:09 dgx-spark systemd[1]: Reached target umount.target - Unmount All Filesystems. Jun 30 12:32:09 dgx-spark systemd[1]: systemd-remount-fs.service: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Stopped systemd-remount-fs.service - Remount Root and Kernel File Systems. Jun 30 12:32:09 dgx-spark systemd[1]: lvm2-monitor.service: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Stopped lvm2-monitor.service - Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling. Jun 30 12:32:09 dgx-spark systemd[1]: Reached target shutdown.target - System Shutdown. Jun 30 12:32:09 dgx-spark systemd[1]: Reached target final.target - Late Shutdown Services. Jun 30 12:32:09 dgx-spark systemd[1]: Starting nvidia-disable-aqc-nic.service - Remove AQC nic before shutdown... Jun 30 12:32:09 dgx-spark systemd[1]: systemd-poweroff.service: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Finished systemd-poweroff.service - System Power Off. Jun 30 12:32:09 dgx-spark systemd[1]: Reached target poweroff.target - System Power Off. Jun 30 12:32:09 dgx-spark systemd[1]: Shutting down. Jun 30 12:32:09 dgx-spark systemd-shutdown[1]: Syncing filesystems and block devices. Jun 30 12:32:09 dgx-spark systemd-shutdown[1]: Sending SIGTERM to remaining processes... Jun 30 12:32:09 dgx-spark systemd-journald[834]: Received SIGTERM from PID 1 (systemd-shutdow). Jun 30 12:32:09 dgx-spark systemd-journald[834]: Journal stopped ...skipping... Jun 30 12:32:09 dgx-spark systemd[1]: Unmounted snap-steam-245.mount - Mount unit for steam, revision 245. Jun 30 12:32:09 dgx-spark systemd[1]: snap-whatsapp\x2delectron-18.mount: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Unmounted snap-whatsapp\x2delectron-18.mount - Mount unit for whatsapp-electron, revision 18. Jun 30 12:32:09 dgx-spark systemd[1]: tmp-RustDesk-cliprdr\x2dclient.mount: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Unmounted tmp-RustDesk-cliprdr\x2dclient.mount - /tmp/RustDesk/cliprdr-client. Jun 30 12:32:09 dgx-spark systemd[1]: tmp-RustDesk-cliprdr\x2dserver.mount: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Unmounted tmp-RustDesk-cliprdr\x2dserver.mount - /tmp/RustDesk/cliprdr-server. Jun 30 12:32:09 dgx-spark systemd[1]: Stopped target snapd.mounts-pre.target - Mounting snaps. Jun 30 12:32:09 dgx-spark systemd[1]: systemd-fsck@dev-disk-by\x2duuid-9DA2\x2d3597.service: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Stopped systemd-fsck@dev-disk-by\x2duuid-9DA2\x2d3597.service - File System Check on /dev/disk/by-uuid/9DA2-3597. Jun 30 12:32:09 dgx-spark systemd[1]: Removed slice system-systemd\x2dfsck.slice - Slice /system/systemd-fsck. Jun 30 12:32:09 dgx-spark systemd[1]: run-snapd-ns-firmware\x2dupdater.mnt.mount: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Unmounted run-snapd-ns-firmware\x2dupdater.mnt.mount - /run/snapd/ns/firmware-updater.mnt. Jun 30 12:32:09 dgx-spark systemd[1]: Unmounting run-snapd-ns.mount - /run/snapd/ns... Jun 30 12:32:09 dgx-spark systemd[1]: run-snapd-ns.mount: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Unmounted run-snapd-ns.mount - /run/snapd/ns. Jun 30 12:32:09 dgx-spark systemd[1]: Stopped target local-fs-pre.target - Preparation for Local File Systems. Jun 30 12:32:09 dgx-spark systemd[1]: Stopped target swap.target - Swaps. Jun 30 12:32:09 dgx-spark systemd[1]: Deactivating swap swap.img.swap - /swap.img... Jun 30 12:32:09 dgx-spark systemd[1]: Deactivating swap swapfile.swap - /swapfile... Jun 30 12:32:09 dgx-spark systemd[1]: Stopping lvm2-monitor.service - Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling... Jun 30 12:32:09 dgx-spark systemd[1]: Stopping multipathd.service - Device-Mapper Multipath Device Controller... Jun 30 12:32:09 dgx-spark multipathd[872]: multipathd: shut down Jun 30 12:32:09 dgx-spark systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Stopped systemd-tmpfiles-setup-dev.service - Create Static Device Nodes in /dev. Jun 30 12:32:09 dgx-spark systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Stopped systemd-tmpfiles-setup-dev-early.service - Create Static Device Nodes in /dev gracefully. Jun 30 12:32:09 dgx-spark systemd[1]: multipathd.service: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Stopped multipathd.service - Device-Mapper Multipath Device Controller. Jun 30 12:32:09 dgx-spark systemd[1]: swapfile.swap: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Deactivated swap swapfile.swap - /swapfile. Jun 30 12:32:09 dgx-spark systemd[1]: swap.img.swap: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Deactivated swap swap.img.swap - /swap.img. Jun 30 12:32:09 dgx-spark systemd[1]: Reached target umount.target - Unmount All Filesystems. Jun 30 12:32:09 dgx-spark systemd[1]: systemd-remount-fs.service: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Stopped systemd-remount-fs.service - Remount Root and Kernel File Systems. Jun 30 12:32:09 dgx-spark systemd[1]: lvm2-monitor.service: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Stopped lvm2-monitor.service - Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling. Jun 30 12:32:09 dgx-spark systemd[1]: Reached target shutdown.target - System Shutdown. Jun 30 12:32:09 dgx-spark systemd[1]: Reached target final.target - Late Shutdown Services. Jun 30 12:32:09 dgx-spark systemd[1]: Starting nvidia-disable-aqc-nic.service - Remove AQC nic before shutdown... Jun 30 12:32:09 dgx-spark systemd[1]: systemd-poweroff.service: Deactivated successfully. Jun 30 12:32:09 dgx-spark systemd[1]: Finished systemd-poweroff.service - System Power Off. Jun 30 12:32:09 dgx-spark systemd[1]: Reached target poweroff.target - System Power Off. Jun 30 12:32:09 dgx-spark systemd[1]: Shutting down. Jun 30 12:32:09 dgx-spark systemd-shutdown[1]: Syncing filesystems and block devices. Jun 30 12:32:09 dgx-spark systemd-shutdown[1]: Sending SIGTERM to remaining processes... Jun 30 12:32:09 dgx-spark systemd-journald[834]: Received SIGTERM from PID 1 (systemd-shutdow). Jun 30 12:32:09 dgx-spark systemd-journald[834]: Journal stopped log file: wapon --show
After Reboot
sparky / ♥ 19:26 dmesg -T | tail -300
journalctl -b -1 -e
free -h
swapon --show
dmesg: read kernel buffer failed: Operation not permitted
Jun 30 19:19:43 dgx-spark systemd[1]: snap-mesa\x2d2404-1778.mount: Deactivated successfully.
Jun 30 19:19:43 dgx-spark systemd[1]: Unmounted snap-mesa\x2d2404-1778.mount - Mount unit for m>
Jun 30 19:19:43 dgx-spark systemd[1]: snap-msedit-4.mount: Deactivated successfully.
Jun 30 19:19:43 dgx-spark systemd[1]: Unmounted snap-msedit-4.mount - Mount unit for msedit, re>
Jun 30 19:19:43 dgx-spark systemd[1]: snap-nvtop-342.mount: Deactivated successfully.
Jun 30 19:19:43 dgx-spark systemd[1]: Unmounted snap-nvtop-342.mount - Mount unit for nvtop, re>
Jun 30 19:19:43 dgx-spark systemd[1]: snap-nvtop-346.mount: Deactivated successfully.
Jun 30 19:19:43 dgx-spark systemd[1]: Unmounted snap-nvtop-346.mount - Mount unit for nvtop, re>
Jun 30 19:19:43 dgx-spark systemd[1]: snap-snapd-26383.mount: Deactivated successfully.
Jun 30 19:19:43 dgx-spark systemd[1]: Unmounted snap-snapd-26383.mount - Mount unit for snapd, >
Jun 30 19:19:43 dgx-spark systemd[1]: snap-snapd-26869.mount: Deactivated successfully.
Jun 30 19:19:43 dgx-spark systemd[1]: Unmounted snap-snapd-26869.mount - Mount unit for snapd, >
Jun 30 19:19:43 dgx-spark systemd[1]: snap-steam-245.mount: Deactivated successfully.
Jun 30 19:19:43 dgx-spark systemd[1]: Unmounted snap-steam-245.mount - Mount unit for steam, re>
Jun 30 19:19:43 dgx-spark systemd[1]: snap-whatsapp\x2delectron-18.mount: Deactivated successfu>
Jun 30 19:19:43 dgx-spark systemd[1]: Unmounted snap-whatsapp\x2delectron-18.mount - Mount unit>
Jun 30 19:19:43 dgx-spark systemd[1]: Started rpc-statd-notify.service - Notify NFS peers of a >
Jun 30 19:19:43 dgx-spark systemd[1]: Stopped target snapd.mounts-pre.target - Mounting snaps.
Jun 30 19:19:43 dgx-spark systemd[1]: Unmounting run-snapd-ns.mount - /run/snapd/ns...
Jun 30 19:19:43 dgx-spark systemd[1]: systemd-fsck@dev-disk-by\x2duuid-9DA2\x2d3597.service: De>
Jun 30 19:19:43 dgx-spark systemd[1]: Stopped systemd-fsck@dev-disk-by\x2duuid-9DA2\x2d3597.ser>
Jun 30 19:19:43 dgx-spark systemd[1]: Removed slice system-systemd\x2dfsck.slice - Slice /syste>
Jun 30 19:19:43 dgx-spark systemd[1]: run-snapd-ns.mount: Deactivated successfully.
Jun 30 19:19:43 dgx-spark systemd[1]: Unmounted run-snapd-ns.mount - /run/snapd/ns.
Jun 30 19:19:43 dgx-spark systemd[1]: Stopped target local-fs-pre.target - Preparation for Loca>
Jun 30 19:19:43 dgx-spark systemd[1]: Stopped target swap.target - Swaps.
Jun 30 19:19:43 dgx-spark systemd[1]: Deactivating swap swap.img.swap - /swap.img...
Jun 30 19:19:43 dgx-spark systemd[1]: Deactivating swap swapfile.swap - /swapfile...
Jun 30 19:19:43 dgx-spark systemd[1]: Stopping lvm2-monitor.service - Monitoring of LVM2 mirror>
Jun 30 19:19:43 dgx-spark systemd[1]: Stopping multipathd.service - Device-Mapper Multipath Dev>
Jun 30 19:19:43 dgx-spark systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfu>
Jun 30 19:19:43 dgx-spark systemd[1]: Stopped systemd-tmpfiles-setup-dev.service - Create Stati>
Jun 30 19:19:43 dgx-spark systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated suc>
Jun 30 19:19:43 dgx-spark systemd[1]: Stopped systemd-tmpfiles-setup-dev-early.service - Create>
Jun 30 19:19:43 dgx-spark multipathd[865]: multipathd: shut down
Jun 30 19:19:43 dgx-spark systemd[1]: swap.img.swap: Deactivated successfully.
Jun 30 19:19:43 dgx-spark systemd[1]: Deactivated swap swap.img.swap - /swap.img.
Jun 30 19:19:43 dgx-spark systemd[1]: multipathd.service: Deactivated successfully.
Jun 30 19:19:43 dgx-spark systemd[1]: Stopped multipathd.service - Device-Mapper Multipath Devi>
Jun 30 19:19:43 dgx-spark systemd[1]: swapfile.swap: Deactivated successfully.
Jun 30 19:19:43 dgx-spark systemd[1]: Deactivated swap swapfile.swap - /swapfile.
Jun 30 19:19:43 dgx-spark systemd[1]: Reached target umount.target - Unmount All Filesystems.
Jun 30 19:19:43 dgx-spark systemd[1]: systemd-remount-fs.service: Deactivated successfully.
Jun 30 19:19:43 dgx-spark systemd[1]: Stopped systemd-remount-fs.service - Remount Root and Ker>
Jun 30 19:19:43 dgx-spark systemd[1]: lvm2-monitor.service: Deactivated successfully.
Jun 30 19:19:43 dgx-spark systemd[1]: Stopped lvm2-monitor.service - Monitoring of LVM2 mirrors>
Jun 30 19:19:43 dgx-spark systemd[1]: Reached target shutdown.target - System Shutdown.
Jun 30 19:19:43 dgx-spark systemd[1]: Reached target final.target - Late Shutdown Services.
Jun 30 19:19:43 dgx-spark systemd[1]: Starting nvidia-disable-aqc-nic.service - Remove AQC nic >
Jun 30 19:19:43 dgx-spark systemd[1]: systemd-poweroff.service: Deactivated successfully.
Jun 30 19:19:43 dgx-spark systemd[1]: Finished systemd-poweroff.service - System Power Off.
Jun 30 19:19:43 dgx-spark systemd[1]: Reached target poweroff.target - System Power Off.
Jun 30 19:19:43 dgx-spark systemd[1]: Shutting down.
Jun 30 19:19:43 dgx-spark systemd-shutdown[1]: Syncing filesystems and block devices.
Jun 30 19:19:43 dgx-spark systemd-shutdown[1]: Sending SIGTERM to remaining processes...
Jun 30 19:19:43 dgx-spark systemd-journald[822]: Journal stopped
lines 946-1001/1001 (END)
total used free shared buff/cache available
Mem: 121Gi 57Gi 30Gi 787Mi 36Gi 64Gi
Swap: 79Gi 0B 79Gi
NAME TYPE SIZE USED PRIO
/swap.img file 16G 0B -2
/swapfile file 64G 0B -3
Could not install dgx-spark-fieldiag
The following information may help to resolve the situation:
The following packages have unmet dependencies:
dgx-spark-fieldiag : Depends: ofed-scripts but it is not installable
E: Unable to correct problems, you have held broken The following information may help to resolve the situation:
The following packages have unmet dependencies:
dgx-spark-fieldiag : Depends: ofed-scripts but it is not installable
E: Unable to correct problems, you have held broken packages.
sparky /usr/share/keyrings ♥ 10:11 sudo dpkg -P dgx-spark-fieldiag
sudo rm -rf /opt/nvidia/dgx-spark-fieldiag
sudo apt autoremove dgx-spark-fieldiag
[sudo] password for sparky:
dpkg: warning: ignoring request to remove dgx-spark-fieldiag which isn't installed
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package 'dgx-spark-fieldiag' is not installed, so not removed
The following packages will be REMOVED:
linux-headers-6.17.0-1018-nvidia linux-image-6.17.0-1018-nvidia linux-modules-6.17.0-1018-nvidia linux-modules-nvidia-fs-6.17.0-1018-nvidia linux-nvidia-6.17-headers-6.17.0-1018
linux-nvidia-6.17-tools-6.17.0-1018 linux-tools-6.17.0-1018-nvidia
0 upgraded, 0 newly installed, 7 to remove and 23 not upgraded.
After this operation, 413 MB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 474007 files and directories currently installed.)
Removing linux-headers-6.17.0-1018-nvidia (6.17.0-1018.18) ...
Removing linux-modules-nvidia-fs-6.17.0-1018-nvidia (6.17.0-1018.18) ...
Removing linux-image-6.17.0-1018-nvidia (6.17.0-1018.18) ...
/etc/kernel/prerm.d/dkms:
dkms: removing module nvidia/580.126.09 for kernel 6.17.0-1018-nvidia (aarch64)
Module nvidia/580.126.09 for kernel 6.17.0-1018-nvidia (aarch64):
Before uninstall, this module version was ACTIVE on this kernel.
Deleting /lib/modules/6.17.0-1018-nvidia/updates/dkms/nvidia.ko.zst
Deleting /lib/modules/6.17.0-1018-nvidia/updates/dkms/nvidia-modeset.ko.zst
Deleting /lib/modules/6.17.0-1018-nvidia/updates/dkms/nvidia-drm.ko.zst
Deleting /lib/modules/6.17.0-1018-nvidia/updates/dkms/nvidia-uvm.ko.zst
Deleting /lib/modules/6.17.0-1018-nvidia/updates/dkms/nvidia-peermem.ko.zst
Running depmod... done.
/etc/kernel/postrm.d/initramfs-tools:
update-initramfs: Deleting /boot/initrd.img-6.17.0-1018-nvidia
/etc/kernel/postrm.d/kdump-tools:
kdump-tools: Removing kdump tracking files for kernel version: 6.17.0-1018-nvidia
/etc/kernel/postrm.d/zz-update-grub:
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/disable-init-on-alloc.cfg'
Sourcing file `/etc/default/grub.d/grubserial.cfg'
Sourcing file `/etc/default/grub.d/iommu.cfg'
Sourcing file `/etc/default/grub.d/kdump-tools.cfg'
Sourcing file `/etc/default/grub.d/menu.cfg'
Sourcing file `/etc/default/grub.d/no-grubmenu.cfg'
Sourcing file `/etc/default/grub.d/nvidia-console-settings.cfg'
Sourcing file `/etc/default/grub.d/nvidia-earlycon.cfg'
Sourcing file `/etc/default/grub.d/nvidia-spark-initcall-bl.cfg'
Sourcing file `/etc/default/grub.d/nvidia-spark-pci.cfg'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.17.0-1026-nvidia
Found initrd image: /boot/initrd.img-6.17.0-1026-nvidia
Found linux image: /boot/vmlinuz-6.17.0-1021-nvidia
Found initrd image: /boot/initrd.img-6.17.0-1021-nvidia
Found linux image: /boot/vmlinuz-6.14.0-1015-nvidia
Found initrd image: /boot/initrd.img-6.14.0-1015-nvidia
Found linux image: /boot/vmlinuz-6.11.0-1016-nvidia
Found initrd image: /boot/initrd.img-6.11.0-1016-nvidia###########################.................................................................................................]
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
Adding boot menu entry for UEFI Firmware Settings ...
done
Removing linux-modules-6.17.0-1018-nvidia (6.17.0-1018.18) ...
Removing linux-nvidia-6.17-headers-6.17.0-1018 (6.17.0-1018.18) ...####################################............................................................................]
Removing linux-tools-6.17.0-1018-nvidia (6.17.0-1018.18) ...#################################################################......................................................]
Removing linux-nvidia-6.17-tools-6.17.0-1018 (6.17.0-1018.18) ...#################################################################################.................................]
sparky /usr/share/keyrings ♥ 10:11 sudo mkdir -p /usr/share/keyrings
curl -fsSL https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa/cuda-archive-keyring.gpg | sudo tee /usr/share/keyrings/cuda-archive-keyring.gpg > /dev/null
echo "deb [signed-by=/usr/share/keyrings/cuda-archive-keyring.gpg] https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa /" | sudo tee /etc/apt/sources.list.d/cuda-sbsa-ubuntu2404.list
sudo apt-get update
sudo apt-get install dgx-spark-fieldiag
deb [signed-by=/usr/share/keyrings/cuda-archive-keyring.gpg] https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa /
Get:1 https://cli.github.com/packages stable InRelease [3917 B]
Hit:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa InRelease
Hit:3 http://ports.ubuntu.com/ubuntu-ports noble InRelease
Hit:4 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64 InRelease
Hit:5 http://ports.ubuntu.com/ubuntu-ports noble-updates InRelease
Hit:6 http://ports.ubuntu.com/ubuntu-ports noble-backports InRelease
Hit:7 https://download.docker.com/linux/ubuntu noble InRelease
Hit:8 http://ports.ubuntu.com/ubuntu-ports noble-security InRelease
Hit:9 https://developer.download.nvidia.com/hpc-sdk/ubuntu/arm64 InRelease
Hit:10 https://snapshot.ppa.launchpadcontent.net/canonical-nvidia/linux-firmware-mbssid-patches/ubuntu noble InRelease
Hit:11 https://brave-browser-apt-release.s3.brave.com stable InRelease
Hit:12 https://workbench.download.nvidia.com/stable/linux/debian default InRelease
Hit:13 https://repo.download.nvidia.com/baseos/ubuntu/noble/arm64 noble InRelease
Hit:14 https://repo.download.nvidia.com/baseos/ubuntu/noble/arm64 noble-updates InRelease
Hit:15 https://deb.nodesource.com/node_22.x nodistro InRelease
Hit:16 https://repo.download.nvidia.com/spark/ubuntu/arm64 noble-updates InRelease
Get:17 https://nvidia.github.io/libnvidia-container/stable/deb/arm64 InRelease [1477 B]
Hit:18 https://deb.tableplus.com/debian/24-arm tableplus InRelease
Hit:19 https://esm.ubuntu.com/apps/ubuntu noble-apps-security InRelease
Hit:20 https://esm.ubuntu.com/apps/ubuntu noble-apps-updates InRelease
Hit:21 https://esm.ubuntu.com/infra/ubuntu noble-infra-security InRelease
Hit:22 https://esm.ubuntu.com/infra/ubuntu noble-infra-updates InRelease
Get:23 https://repo.charm.sh/apt * InRelease
Hit:25 https://snapshot.ppa.launchpadcontent.net/canonical-nvidia/nvidia-desktop-edge/ubuntu noble InRelease
Hit:26 https://snapshot.ppa.launchpadcontent.net/canonical-nvidia/vulkan-packages-nv-desktop/ubuntu noble InRelease
Hit:27 https://ppa.launchpadcontent.net/ubuntuhandbook1/keepass2/ubuntu noble InRelease
Hit:28 https://us-central1-apt.pkg.dev/projects/antigravity-auto-updater-dev antigravity-debian InRelease
Hit:30 https://packages.microsoft.com/repos/code stable InRelease
Ign:31 https://releases.warp.dev/linux/deb stable InRelease
Hit:24 https://packagecloud.io/eugeny/tabby/ubuntu noble InRelease
Hit:29 https://packagecloud.io/github/git-lfs/ubuntu noble InRelease
Hit:32 https://releases.warp.dev/linux/deb stable Release
Fetched 12.1 kB in 2s (4859 B/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
dgx-spark-fieldiag : Depends: ofed-scripts but it is not installable
E: Unable to correct problems, you have held broken packages.
nvidia-smi
Tue Jun 30 23:49:17 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.126.09 Driver Version: 580.126.09 CUDA Version: 13.0 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GB10 On | 0000000F:01:00.0 N/A | N/A |
|ERR! 47C P0 N/A / N/A | Not Supported | N/A Default |
| | | ERR! |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
I tried to fix that
1.Download the MLNX_OFED Archive:
Select the aarch64 version.
Navigate to the Nvidia Network Developer site and download the latest MLNX_OFED .tgz archive specifically for Ubuntu 24.04 aarch64.2.
2. Extract the Tarball:
Unpack the downloaded archive in your terminal:
Bash
tar -xvf MLNX_OFED_LINUX-*.tgz
cd MLNX_OFED_LINUX-*/
3.Run the Mellanox Installer:
Execute the installation script. Including the kernel support flag ensures the networking modules build correctly against your current Ubuntu kernel:
Bash
sudo ./mlnxofedinstall --add-kernel-support
Note: This process can take several minutes as it compiles custom kernel modules.
4.Install the Diagnostics Package:
Once the MLNX_OFED installation finishes successfully, ofed-scripts will be registered in your local package manager. You can now retry your original command:
Not sure if this is still a problem for you, but there was an update a month ago that introduced an OOM killer which will stop any process that uses too much memory and should stop the system from fully crashing from OOM error. If you are still seeing OOM errors, please make sure your system is fully up-to-date
Had come ComfyUI Tasks running over night.. and when I came back to the DGX it was crashed..
its not the RAM cause I stopped everything else apart from ComfyUI and the Models there are small.
Not sure if its the temperature. Had the extra Fan on and also the Aircon in the room.
I can still reach the machine though SSH but not the Ubuntu GUI.
The GPU is crashed.
For those hitting the “everything looks fine but inference just stops” flavor of instability (as opposed to the OOM or thermal ones): if you’re serving with vLLM, check whether it’s the silent engine deadlock rather than a crash. The trap on GB10 specifically is that `nvidia-smi` shows ~95% GPU utilization the whole time — because on this hardware the util counter counts a resident kernel, not actual work, so a deadlocked engine looks fully busy. `/health` also keeps answering 200.
The reliable way to tell a deadlock from a healthy-but-busy engine is **GPU power draw**: healthy generation averages ~45W, a deadlocked engine sits at 16–27W while still reporting ~95% util (and the temp drops). It’s vLLM #37729; there’s no upstream fix in a released build yet, so the practical mitigation is a watchdog that catches `generation_tokens_total` frozen while requests are held + idle power, and restarts the engine.
I recently switched to a two-spark cluster hosting dsfv4 0731 (eugr/spark-vllm-b12x@sha256:a2df4abebff70246df85a5eeba15cd0490733fed76f516f36e0281587c1b15d7).
When I start the vLLM instance and try to continue an old session, the resulting cold-start prefill of 200k tokens speedruns both sparks to 91 degrees over 2m, at which point one of them shuts itself off with no kernel messages or other logs relevant to the thermal shutdown. This happens even with nvidia-smi -lgc 0,2000 and external cooling blowing set up specifically for the sparks. I can deal with throttling but the instability is a killer..
EDIT: The spark survived the prefill with the following settings:
nvidia-smi -lgc 200,1800
for p in /sys/devices/system/cpu/cpufreq/policy*; do echo 0 | sudo tee “$p/auto_select”; echo 2000000 | sudo tee “$p/scaling_max_freq”; done
