Unified-memory kernel-init allocations (FlashInfer/DeepGEMM/NVFP4) consume ~50GB before weight loading starts, disabling auto-prefetch on large models

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Environment:

  • Container: nvcr.io/nvidia/vllm:26.07-py3 (also reproduced on vllm/vllm-openai:v0.26.0-aarch64)
  • Hardware: DGX Spark (GB10), 121GB unified memory
  • Model: NVFP4 checkpoint, ~75GB on disk

Observation: With the box otherwise idle, free -h shows ~118GB available. Polling free -h every 2s through vLLM startup:

T+0s   available=120919MB
T+~23s available=119280MB   <- "Starting to load model /model..."
T+~25s available= 68501MB   <- kernel/backend selection happens in this window
T+~30s available= 45826MB
T+~40s available= 44674MB

Correlating against timestamped container logs, the ~50GB drop lands exactly on:

Selected FlashInferFP8ScaledMMLinearKernel for ModelOptFp8LinearMethod
DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
Using FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM
Using 'FLASHINFER_CUTLASS' NvFp4 MoE backend
Using AttentionBackendEnum.TRITON_ATTN backend.

— all before weight_utils.py even starts reading the checkpoint. Read weight_utils.py in the container directly; its “available RAM” check is a plain psutil.virtual_memory().available — i.e., it’s accurately reading real system memory, not a measurement bug. The memory really is gone by that point.

Downstream effect: For any checkpoint over ~90% of whatever’s left after that initial grab, vLLM’s own auto-prefetch optimization disables itself (weight_utils.py’s own log: “Auto-prefetch is disabled… checkpoint size exceeds 90% of available RAM”), forcing the slow non-prefetch disk-read path — several extra minutes on a 75GB model.

Question: Is this workspace allocation (FlashInfer/DeepGEMM/NVFP4-Cutlass/MoE-backend init) expected to be this large on unified-memory Grace-Blackwell hardware specifically? Is there a way to defer or shrink it, or would it make sense for the auto-prefetch check to run after kernel init settles rather than measuring available RAM as if that allocation doesn’t happen? This is filed as an observation more than a hard bug — happy to share the full timestamped logs if useful.