Driver Issue with 580.178.04

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
,
1

Hi Guys,

I bought a second spark and I am in the process of moving over my qwen3.8 flash next build from my existing spark over to the new one. Problem is the the new spark OOM’s and won’t run the exact same config. Only difference I can see is the driver versions:

Same:

  • GB10 GPU
  • exact SGLang image: sha256:cdd964…
  • exact model payload: 206 files, 135,195,303,851 bytes
  • no Docker or systemd memory limit
  • memlock unlimited
  • same core serving flags, including --mem-fraction-static 0.85 and PLE file offload

Differences:

  • spark-ee93 driver: 580.173.02
  • spark-da36 driver: 580.178.04
2

One other variable here is what that fraction is a fraction of on these boxes.

GB10 reports as an integrated GPU and SGLang branches on that. In get_available_gpu_memory (python/sglang/srt/utils/common.py), line 445 tests props.is_integrated and line 450 then takes psutil.virtual_memory().available instead of torch.cuda.mem_get_info(), with the comment that mem_get_info under-reports on these devices because it excludes cache memory. So --mem-fraction-static 0.85 is 0.85 of a host number that includes reclaimable page cache.

Measured on our Spark just now, idle, driver 580.159.03, inside lmsysorg/sglang:

is_integrated                   : 1
cuda mem_get_info               : free 80.75 GiB / total 121.69 GiB
psutil.virtual_memory available : 115.52 GiB / total 121.69 GiB
sglang get_available_gpu_memory : 115.30 GB

/proc/meminfo at the same moment: MemAvailable 116.03 GiB, Cached 30.53 GiB. The 34.77 GiB spread between the two readings is close to the 30.53 GiB Cached value. At 0.85, the corresponding budgets are 68.64 GiB and 98.19 GiB.

That input moves with whatever the kernel counts as reclaimable at the instant of launch, and you have just written 135 GB of payload onto the new box, which leaves its cache in a different state from the machine you copied it from.

If useful, the value can be printed immediately before each launch:

python3 -c "import psutil; print(psutil.virtual_memory().available / 2**30)"

If the values match, this does not explain the difference between the boxes. If they differ, --mem-fraction-static 0.85 is resolving to different memory budgets.

3

Good info, after downgrading the driver to 580.173.02 it does load. However I had to drop mem to .8 and the global context dropped significantly (down to 75.5k-token KV pool) for some reason? Memory management on these unified systems has been interesting to say the least…

sglang on spark-da36 — snapshot (2026-09-21 05:47 UTC)

Host / runtime
Host spark-da36, up 3h40m, load 1.08
GPU 1× NVIDIA GB10 (DGX Spark), driver 580.173.02, CUDA 13.0, 43 °C, 10 W idle
Mem 121 GiB total → 100 GiB used, 20 GiB available; swap 127 GiB (3 GiB used)
Container sglang-qwen38-da36 (image lmsysorg/sglang @ 9d2a843c…), up 2h, 0 restarts, cpu ~104%, mem 16.7 GiB
Processes sglang::scheduler PID 160717 (2h27m, 94.2 GiB GPU alloc) + tokenizer proc 160512
Endpoints 0.0.0.0:30000 inside netns; host maps 127.0.0.1:30001 → 30000
Versions sglang 0.0.0.dev1+g4ccff141d, torch 2.13.0+cu130, CUDA 13.0

Model

  • Serving name RadixArk/Qwen3.8-Flash-Next-NVFP4, weights /model (126 GB, per-layer expert safetensors, hf_quant_config.json)
  • Hybrid linear-attention (mamba) + MoE: uses_mamba_radix_cache=true, 512 experts/layer quantized shards
  • quantization modelopt_fp4 (NVFP4), fp4 gemm flashinfer_cutlass, moe runner flashinfer_cutlass
  • context_len 262144, max_req_input_len 75514, status ready

Running config (from launch cmdline + /get_server_info)

TEXT⧉ COPY⬇ SAVE

python3 -m sglang.launch_server --model-path /model --tp 1 \
  --quantization modelopt_fp4 --fp4-gemm-backend flashinfer_cutlass \
  --page-size 64 --chunked-prefill-size 4096 --context-length 262144 \
  --speculative-algorithm NEXTN --speculative-num-steps 3 \
  --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
  --max-running-requests 8 --max-queued-requests 32 --max-mamba-cache-size 40 \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
  --chat-template /templates/qwen38-flash-next-chat-template.jinja \
  --enable-metrics --mem-fraction-static 0.80 \
  --ple-offload-embedding --ple-offload-backend file \
  --ple-offload-dir /root/.cache/sglang/ple \
  --served-model-name RadixArk/Qwen3.8-Flash-Next-NVFP4 --host 0.0.0.0 --port 30000

Effective: tp/dp/pp/ep = 1, attention flashinfer, sampling flashinfer, grammar xgrammar, linear-attn backend triton, mamba radix strategy extra_buffer, schedule fcfs, radix eviction lru, cuda graphs on, overlap schedule on, hierarchical cache off, P/D disaggregation off, metrics on. max_total_num_tokens = 75520 (1180 pages × 64), KV cache memory 1.78 GB, cuda-graph pools ~0.56 GB total.

Live metrics (accumulated since start)

Metric Value
Requests 8 total (4 streaming / 4 non-streaming); 6 chat.completions + 2 completions, all 200
Tokens prompt 14,415 / generated 888
Mean TTFT 2.46 s (streaming 1.44 s, non-streaming 3.49 s)
Mean E2E 5.34 s
Mean inter-token latency 29.6 ms → ~34 tok/s per sequence
Decode throughput (last log) 40–44 tok/s
Speculative (NEXTN) accept length 4.00 / 4 draft tokens, accept rate 1.00, 270 verify calls
Prefix cache 6,976 device-cached tokens; hit rate 0.0 currently
KV pool used 0, available 68,160, evictable 7,360 (9.7% occupied) — idle
Mamba pool 33 available / 7 evictable of 40 slots, usage 0.0
CUDA graph 282 decode-graph passes, 12 prefill (eager) passes
Errors 2× /health 503 (pre-ready probes), 2× 404 (/get_server_args, favicon); no 5xx on inference endpoints

State: healthy and idle — zero running/queued requests, KV and mamba pools drained, GPU at 10 W. Speculative decoding is at theoretical max acceptance, so the 34–44 tok/s figure is the current per-stream ceiling on this GB10; the only real limits are max_running_requests=8 and the 75.5k-token KV pool against a 262k context (long prompts will evict/recompute rather than extend).

sglang on spark-ee93 — snapshot (2026-09-21 05:56 UTC)

Host / runtime
Host spark-ee93, up 9 days 17h, load 1.35
GPU 1× GB10, driver 580.173.02 / CUDA 13.0 — 51 °C, 96% util, 27 W (actively serving)
GPU sharing sglang scheduler holds 98.2 GiB; also on-GPU: 2× llama-server (exp-mtp, 1.75 GiB), ComfyUI (318 MiB)
Mem 121 GiB → 108 GiB used, 12 GiB available; swap 127 GiB (11 GiB used) — host is memory-tight
Container sglang-qwen38-flash-next (lmsysorg/sglang@9d2a843c, same image digest as da36), up 6 days, cpu ~104%
Access 127.0.0.1:30001 → 30000
Versions sglang 0.0.0.dev1+g4ccff141d, torch 2.13.0+cu130
Startup load_weight 518 s, KV alloc 2.1 s, scheduler e2e 545 s; GPU free at boot 16.2 GB

Running config — identical to da36 except one flag

Same launch line (modelopt_fp4, flashinfer_cutlass, page-size 64, chunked-prefill 4096, ctx 262144, NEXTN spec 3/1/4, max-running 8, mamba cache 40, qwen3 + qwen3_coder parsers, PLE file offload), with:

  • --mem-fraction-static 0.85 (da36: 0.80) → max_total_num_tokens 172,224 vs 75,520 (2.3× KV pool; KV mem 4.07 GB vs 1.78 GB)
  • Weights mounted from /home/tyrel/models/Qwen3.8-Flash-Next-NVFP4 (126 GB, read-only)
  • Env: PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, SGLANG_QWEN4_PLE_FILE_RSS_BUDGET_GB=8
  • Everything else (backends, speculative, schedulers, parsers) byte-identical per get_server_info diff

Live metrics (6 days of traffic)

Metric Value
Requests 2,314 (2,245 streaming / 69 non-streaming); 49,785 scheduler batches
Tokens prompt 98.9M, generated 1.41M
Prefix cache 65.3M device-hit tokens → 66% effective cache hit; but 33.3M tokens evicted (8,786 eviction ops) — KV pool runs 99.7% full (171,776 of 172,224 evictable, only 448 free)
Mean TTFT 9.77 s (queue time total 1,334 s across batches)
Mean ITL 42.3 ms → ~23.6 tok/s/stream; instantaneous decode 33–36 tok/s
Mean E2E 35.6 s streaming / 19.5 s non-streaming
Speculative (NEXTN) accept length 3.73/4, accept rate 0.91 (live logs 3.3–3.8 / 0.76–0.94), 514k verify calls
CUDA graphs 534,609 decode-graph passes / 57,271 eager prefills
Aborts 120 aborted requests, 6 so-requests; 0 retracted, 0 queued right now
Mamba slots 38/40 evictable, 2 free

State: healthy and actively serving — a chat completion was in flight at snapshot time (34k-token context, decode 36 tok/s). Long-running production box: high traffic, radix cache saturated (heavy eviction churn but effective reuse), 120 lifetime aborts worth noting, and host RAM/swap pressure from co-located llama-server + ComfyUI.

da36 vs ee93: same image, model, and launch config except mem_fraction_static (0.80 vs 0.85), giving ee93 2.3× the KV pool; ee93 is the busy node (2.3k requests, 99.7% KV occupancy, TTFT ~9.8 s under load) vs da36’s fresh idle instance (8 requests, TTFT 2.5 s unloaded).

4

So my earlier explanation was wrong: the reading I pointed at was not what made the difference. That run also changed --mem-fraction-static from 0.85 to 0.80, so the two effects are still confounded.

On the context drop, the KV pool itself is small relative to that 5%.

Qwen3.8-Flash-Next is 48 layers of which only 12 are full attention (full_attention_interval = 4), and those carry num_key_value_heads = 2 at head_dim = 256. Per token that is 12 x 2 x 2 x 256 = 12,288 elements:

fp8    12,288 B/token  ->  75,514 tokens =  0.86 GiB
bf16   24,576 B/token  ->  75,514 tokens =  1.73 GiB

The whole 75.5k pool costs under a GiB. The 5% you gave back is about 6.05 GiB of 121, roughly seven times the entire pool, so the ceiling did not fall in exchange for that memory.

The 36 linear-attention layers are large enough to be relevant here. Linear-attention state is per sequence rather than per token, and this config is wide: linear_num_value_heads = 48 with linear_key_head_dim = 128 and linear_value_head_dim = 128 is 786,432 elements per layer per sequence, 28.3M across the 36 linear layers. At bf16 that is 54 MiB for every concurrent sequence, 108 MiB at fp32. Sixteen sequences at fp32 is twice your KV pool.

That last part is arithmetic from the config, not a read of how SGLang lays the pool out. If it is where the memory went, max_running_requests moves it and --mem-fraction-static does not.

5

Seems like your right on the money, decreasing the max-running-requests and the mamba-cache-size while also going from .8 back to .85 significantly increased the available KV pool.

sglang on spark-da36 — fresh snapshot (2026-09-21 07:36 UTC)

Your adjustments are in — the container was recreated ~13 min ago (new container aefa70e1d1b0, started 13:22:46 UTC, 0 restarts), so all counters below are from the new instance.

What changed vs the previous run

Setting Before Now
--mem-fraction-static 0.80 0.85
--max-running-requests 8 4
--max-mamba-cache-size 40 20
KV pool (max_total_num_tokens) 75,520 432,640 (6,760 pages × 64)
max_req_input_len 75,514 262,138 — full 262k context now usable in one request
KV cache memory 1.78 GB 10.21 GB
GPU alloc at startup 94.2 GB 98.6 GB (18.2 GB free at boot)

Everything else unchanged: same image digest (9d2a843c), modelopt_fp4 + flashinfer_cutlass, NEXTN spec 3/1/4, chunked-prefill 4096, ctx 262144, max-queued 32, qwen3/qwen3_coder parsers, PLE file offload, port 30000 (host 127.0.0.1:30001).

Big effect: the KV pool grew 5.7× (75.5k → 432.6k tokens) — far past ee93’s 172k — because the mem bump plus halved running-requests/mamba-cache freed room for KV. A single request can now use the entire 262k context without truncation.

Host / runtime

  • GB10 at 44 °C, 0% util, 10 W (idle); host load 1.06; RAM 105/121 GiB used, 16 GiB available; swap 3 GiB
  • Container cpu ~102%, mem 4.23 GiB; startup: load_weight 578 s, scheduler e2e 598 s; status ready
6

Your runs now bracket the driver question.

0.85, rr 8, mamba 40, 580.178.04   OOM                      (post 1)
0.85, rr 8, mamba 40, 580.173.02   apparently did not load  (inferred from "I had to drop mem to .8")
0.80, rr 8, mamba 40, 580.173.02   loads,  75.5k KV         (post 3)
0.85, rr 4, mamba 20, 580.173.02   loads, 432.6k KV         (post 5)

If that reading of post 3 is right, rows one and two are the same configuration on different drivers with the same outcome. Row four puts 0.85 back, the fraction that originally failed, and it fits once the per-sequence state stops claiming the room.

If row two is right, the driver version does not explain the original difference, and the thread title points at the wrong thing. That is worth settling because the next Spark owner who searches this will skip 580.178.04 on the strength of the title.

The cell that would settle it is 580.178.04 at the current 0.85 / 4 / 20.

7

Absolutely, you are correct, and the next move to isolate the driver is to switch it back to 580.178.04 and see if we still OOM. It did in fact still fail on row 2, and that is why it was dropped to .8.

Confirmed: driver 580.178.04 is not the problem

Here’s the full state of the SGLang deployment on spark-da36 (192.168.4.81).

Source Value
nvidia-smi header Driver Version: 580.178.04 · CUDA Version: 13.0 (NVIDIA-SMI 580.178.04)
nvidia-smi --query-gpu=driver_version 580.178.04
/sys/module/nvidia/version (loaded kernel module) 580.178.04
Debian packages nvidia-driver-580-open 580.178.04-0ubuntu0.24.04.1 (arm64, open kernel module)

So the userspace driver, the in-tree kernel module, and the installed package all agree — no mismatch from a stale module or a partially-upgraded package.

Server / Model
Endpoint http://192.168.4.81:30000 (host 0.0.0.0:30000)
Served model RadixArk/Qwen3.8-Flash-Next-NVFP4
Weight path /model — 126 GB on disk, 3.7T NVMe (11% used)
Architecture qwen4_exp / Qwen4ExpForConditionalGeneration, hybrid (uses mamba radix cache), image-capable, audio no
Container sglang-qwen38-da36, image lmsysorg/sglang@sha256:9d2a843c…
Versions sglang 0.0.0.dev1+g4ccff141d (dev build), torch 2.13.0+cu130, CUDA 13.0
Health /health_generate → 200, /health 9×200 vs 3×503 (early startup probes), 0 restarts, no OOM

Running config (from /server_info)

TEXT⧉ COPY⬇ SAVE

--quantization modelopt_fp4      --fp4-gemm-backend flashinfer_cutlass
--tp 1  --dp 1  --pp 1  --ep 1   --context-length 262144
--page-size 64                   --chunked-prefill-size 4096
--max-total-tokens (unset)       --max-prefill-tokens 16384
--mem-fraction-static 0.85       --max-running-requests 4
--max-queued-requests 32         --max-mamba-cache-size 20
--speculative-algorithm NEXTN (EAGLE, self-draft from /model)
--speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
--reasoning-parser qwen3 --tool-call-parser qwen3_coder
--ple-offload-embedding --ple-offload-backend file --ple-offload-dir /root/.cache/sglang/ple  (48 GB)
--enable-metrics --enable-cache-report

Backends: attention/sampling flashinfer, MoE + spec-MoE flashinfer_cutlass, mamba & linear-attn triton, grammar xgrammar. CUDA graphs: decode full (max_bs 256), prefill disabled. Schedule fcfs, retraction length, radix eviction lru, hierarchical cache off. Watchdog 300s.

Memory & capacity (GB10, 121 GiB unified)
Weights 80.23 GB
KV cache 10.81 GB → max_total_num_tokens 457,792 (7,153 pages × 64)
CUDA graphs 0.397 GB total (target_verify 0.075, draft_decode 0.185, draft_extend 0.137)
Free GPU mem at startup 17.08 GB
KV split live 446,464 available / 11,328 evictable / 0 used
Host mem 107 of 121 GiB used, 13 GiB available — tight
Startup load_weight 575s, scheduler e2e 598s, tokenizer e2e 603s (~10 min)

Metrics (cumulative since 17:51 start)

Traffic: 5 requests (4 streaming, 1 non-stream) · 29,769 prompt tok · 1,076 gen tok · all 5 → HTTP 200 · queue time 0.6 ms avg (essentially zero contention)

Latency: TTFT avg 1.99s (20.07s non-stream outlier; streaming 4 reqs = 2.43s avg) · ITL avg 30.2 ms → ~33 tok/s decode · prefill_forward 1.54s/req, chunked_prefill 3.32s

Prefix cache: 18,944 of 30,208 prefill tokens served from device cache = 62.7% hit; 11,264 computed, 0 host/storage hits (hierarchical cache off)

Speculative decode (NEXTN): accept_length 3.50 of 4 max · accept_rate 0.833 · 316 verify calls · 1,098 decode tokens → effective **33 tok/s**

Scheduler health: 0 running, 0 queued, 0 retracted, 0 paused, token_usage 0.0, utilization 0.0 — fully idle. decode_cuda_graph passes 326 vs prefill_none 11, so decode is graph-captured as expected.

Monitoring stack (all Prometheus targets up)

sglang :30000 · node :9100 · gpu :9835 · cadvisor :8081 · Prometheus :9090 (v3.13.3) · Grafana 13.2.2. Container usage: sglang 101.7% CPU / 4.64 GiB.

8

RCA was a systemd service misconfiguration causing a race/restart condition during model load. Driver seems to be functional. I did notice that sglang global KV cache pools fluctuate somewhat significantly between launches. (not specific to kernel version)