【Urgent】DGX Spark (GB10 UMA) NIM Container Memory Runaway, All Limitation Parameters Invalid, Memory Occupies 120GB+

AI & Data Science NVIDIA AI Workbench
1

Hardware & Environment

  • Hardware: NVIDIA DGX Spark (GB10 Superchip, 128GB LPDDR5X Unified CPU/GPU Memory, UMA Architecture)

  • CUDA Version: 13.0, Driver Version: 580.126.09

  • OS: Ubuntu 22.04 aarch64

  • Workload: Qwen3.6-27B-Instruct FP8 NIM Container

Problem Phenomenon

When starting the NIM container, the memory will surge to 115-124GB instantly after the model is loaded, occupying more than 95% of the total system memory, resulting in no available memory for other services (Agent, browser, system processes).

The memory surge occurs in the KV cache initialization and FlashInfer JIT compilation phase, and there is no clean OOM error, only the system memory is directly filled up.

All Solutions I Have Tried (All Invalid)

I have tried all the solutions mentioned in the official documentation and community posts, but none of them can effectively limit the memory usage:

1. Docker Container Level Limitations

  • --memory=80g --memory-swap=0: Completely bypassed by UMA architecture, GPU memory allocation does not go through Docker cgroup, memory still rushes to 120GB+

  • --kernel-memory=32g: No effect, GPU unified memory allocation is not restricted

  • docker update --memory 80g after container startup: No effect, memory is still not limited

2. NIM & vLLM Application Layer Parameters

  • NIM_GPU_MEM_FRACTION=0.625: The parameter is completely invalid, vLLM still recognizes 128GB total memory, and pre-allocates 90% of the memory

  • NIM_GPU_MEMORY_UTILIZATION=0.35: Invalid, the parameter is overwritten by NIM wrapper process

  • --num-gpu-blocks-override passed via VLLM_EXTRA_ARGS / NIM_ADDITIONAL_ARGS: The parameter is filtered and overwritten by NIM wrapper, cannot be passed to the underlying vLLM engine

  • VLLM_ENFORCE_EAGER=1, disable CUDA Graph: Only reduces a small amount of transient memory spike, cannot solve the overall memory runaway

  • VLLM_KV_CACHE_DTYPE=fp8, reduce KV cache precision: Only reduces part of KV cache usage, cannot prevent the overall memory from rushing to 120GB

  • MAX_MODEL_LEN=16384, limit context length: No obvious effect, memory is still filled up

3. System & Kernel Level Operations

  • sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches' clean system page cache: Only temporarily releases memory, after the container starts, memory is still occupied to 120GB+

  • systemd-run --scope -p MemoryMax=80G -p MemorySwapMax=0 kernel-level cgroup limitation: Still bypassed by CUDA unified memory allocation, cannot limit GPU memory usage

  • Adjust Linux kernel memory scheduling parameters: No effect

Core Problem Analysis

  1. According to the official known issues, the cudaMemGetInfo API on DGX Spark UMA architecture will incorrectly identify the entire 128GB system unified memory as available GPU memory

  2. vLLM defaults to gpu_memory_utilization=0.9, which will greedily pre-allocate 90% of the recognized memory (about 115GB) for KV cache, resulting in memory being directly filled up

  3. All user-passed vLLM parameters will be overwritten or filtered by the NIM container wrapper process, and cannot take effect at the underlying engine

  4. All Docker and system-level memory limitations cannot restrict the CUDA driver’s unified memory allocation behavior, and there is no way to limit the maximum memory that GPU can apply for at the driver level

Request for Help

Is there any official solution that can really limit the maximum memory usage of NIM/vLLM containers on DGX Spark UMA architecture?

Or is there any parameter that can really take effect to limit the maximum KV cache pre-allocation size of vLLM in NIM containers?

This problem has seriously affected the normal use of the equipment, and I look forward to the official reply and solution.