GB10 stops issuing new CUDA contexts after days of uptime — existing process keeps working, any new one gets "no CUDA-capable device is detected"

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
, , ,
1

Summary

After the GPU has been in continuous use for several days, new CUDA contexts can no longer be created, while an already-running process continues to work normally. Restarting that process makes the problem visible: it falls back to CPU and keeps serving, roughly 5× slower, without any hard failure.

A reboot always clears it. Observed three times in two weeks.

Environment
System DGX Spark, GB10
BIOS 5.36_0ACUM027 (06/12/2026)
Driver 580.173.02
CUDA 13.0
Kernel 7.0.0-1016-nvidia
Workload llama.cpp llama-server (build 10547), 27B model at Q4_K_M, -ngl 99, running as a systemd user service

Disclosure: this machine has a partially-completed 24.04 → 26.04 release upgrade. Several hundred packages are at 26.04 versions, including the kernel, while /etc/os-release still reports 24.04. Apt sources have been restored to noble and the package database is consistent. I can’t rule this out as a factor, but the GPU stack itself is functioning normally between incidents, at expected throughput.

Symptoms, in the order they appear

1. badf5600 register-read errors accumulate in dmesg

NVRM: gpuHandleSanityCheckRegReadError_GH100: Possible bad register read:
addr: 0x613078, regvalue: 0xbadf5600, error code: Unknown SYS_PRI_ERROR_CODE

Over 1,500 of these were present while inference was still running at full speed, so on their own they are not conclusive — but they precede the failure.

2. Any new CUDA process fails, while the running one is fine

Even --help on a second binary fails:

$ llama-server --help
ggml_cuda_init: failed to initialize CUDA: no CUDA-capable device is detected

Meanwhile the already-running llama-server continues serving at ~21 tokens/s with all layers on the GPU.

3. nvidia-smi reports the GPU as healthy throughout

$ nvidia-smi --query-gpu=name,driver_version --format=csv
name, driver_version
NVIDIA GB10, 580.173.02

$ nvidia-smi --query-gpu=compute_mode --format=csv
compute_mode
Default

Compute mode is Default, so this is not exclusive-process contention.

4. Restarting the service silently degrades to CPU

ggml_cuda_init: failed to initialize CUDA: no CUDA-capable device is detected
warning: no usable GPU found, --gpu-layers option will be ignored

Generation drops from ~21 t/s to ~4.6 t/s and prompt processing from ~1.2 ms to ~110 ms per token. The service reports healthy and answers requests, so liveness checks pass — this is what makes it easy to miss.

On the most recent occurrence systemd also logged:

llama-server.service: Failed with result 'timeout'.

The previous instance exceeded its stop timeout and was killed, which may be relevant to why the GPU would not hand out a new context.

5. A reboot restores normal operation immediately

After reboot, CUDA initialises, all layers offload, and throughput returns to ~21 t/s. No other intervention has ever been needed.

Reproduction

Not deterministic, but the pattern is consistent:

  1. Run a CUDA workload continuously for several days
  2. Watch sudo dmesg | grep -c badf5600 climb
  3. Attempt to start any second CUDA process — fails while the first is healthy
  4. Restart the first process — it comes back on CPU
  5. Reboot — everything normal again

Questions

  1. Is badf5600 on SYS_PRI_ERROR_CODE a known indicator of this state, and is there a threshold at which it becomes meaningful?
  2. Is there a way to recover the ability to create contexts without a reboot — a driver module reload, or resetting the GPU, given GB10’s unified memory architecture?
  3. Is there a supported way to detect this state programmatically? nvidia-smi reports healthy, so the only reliable signal I have found is attempting to create a context and watching it fail.
  4. Could the systemd stop-timeout kill be leaving the context in a state the driver won’t reclaim, and if so is there a recommended shutdown grace period for long-running CUDA processes?

Happy to provide nvidia-bug-report.sh output, full dmesg, or to run any diagnostic before the next reboot.

2

Hey, send me the exact command you normally use to start llama-server.

I can send you a Badgr command to run it that verifies it’s actually serving on the GPU, so if CUDA dies and it falls back to CPU it won’t just look healthy.

3

Thanks. Here’s the service command, with the username redacted:

/home/USER/llama.cpp/build-gpu/bin/llama-server \
  -m /home/USER/models/Qwen38-27B/Qwen3.8-27B-Q4_K_M.gguf \
  --mmproj /home/USER/models/Qwen38-27B/mmproj-F16.gguf \
  -ngl 99 -c 262144 --parallel 1 --jinja --reasoning off \
  --spec-type draft-mtp \
  -md /home/USER/models/Qwen38-27B-MTP/mtp-Qwen3.8-27B-Q4_0.gguf \
  --spec-draft-n-max 2 \
  --host 127.0.0.1 --port 8081 --alias hermes3

On detection — I do already have a reliable signal, which may help others
hitting this. llama.cpp announces the fallback at startup:

journalctl --user -u llama-server -b | grep -i 'no usable GPU'

And the throughput difference is unmistakable: ~21 tokens/s on GPU versus
~4.6 on CPU, with prompt processing going from ~1.2 ms to ~110 ms per token.
So I can tell after the fact reliably enough.

What I still can’t do is detect the underlying state before restarting —
nvidia-smi reports the GPU healthy the whole time, while any new CUDA
context fails. That’s the part I’m hoping someone can speak to.

I’m not able to run unfamiliar tools on this box, but I’m glad to run
nvidia-bug-report.sh or anything from NVIDIA’s documentation, and I’ll
capture it before the next reboot while the fault is live. Could you point me
at what “Badgr” is — a repo or package name? I’m not finding it.

4

Please share the nvidia-bug-report, however since you are on an unsupported package configuration, the first recommendation may simply be to revert to a known good version

5

bug report sent by PM, reverting to the supported kernel, will report back