TensorRT OOM problem

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

TensorRT-LLM Issues on NVIDIA GB10 (Spark) – Engine Build OOM / Memory Reporting Problems

Hardware / Platform

  • GPU: NVIDIA GB10 (Spark)

  • Driver: 580.95.05

  • CUDA: 13.0

  • OS: Ubuntu (aarch64)

  • GPU Memory: reported as ~119 GB by TensorRT-LLM heuristics

  • nvidia-persistenced: enabled and running

Software Stack

  • Container:
    nvcr.io/nvidia/tensorrt-llm/release:spark-single-gpu-dev
    Digest:
    sha256:4342a40dd7bdb4be9eeadd541aa3e739cd6d72c5441f36844c5345d07ec629da

  • TensorRT-LLM version: 1.1.0rc3

  • PyTorch: 2.8.0a0+5228986c39.nv25.6

  • Model:
    nvidia/Llama-3.3-70B-Instruct-NVFP4 (HF pre-quantized NVFP4 checkpoint)

What Works

  • HF model download completes successfully.

  • Weights load into memory successfully.

  • Pre-quantized NVFP4 checkpoint is correctly detected:

    hf_quant_config.json found
group_size=16
gemm_plugin=nvfp4

  • Network construction completes.

  • Optimization profiles are created.

What Fails (Consistently)

During TensorRT engine build, the process fails with GPU Out-Of-Memory, even though:

  • No other GPU processes are running.

  • Persistence mode is enabled.

  • GPU temperature and utilization are idle.

Representative error:

[TRT] [E] virtualMemoryBuffer.cpp::resizePhysical::154
Error Code 2: OutOfMemory (Requested size was 3654287360 bytes)

[TRT] [E] globWriter.cpp::makeResizableGpuMemory::514
Error Code 2: OutOfMemory (Requested size was 3654287360 bytes)

AssertionError: Engine building failed, please check the error log.

In another run:

Requested size was 7287603200 bytes

Additional Observations

  1. GPU memory is not reported correctly by nvidia-smi

    nvidia-smi --query-gpu=memory.total,memory.used,memory.free --format=csv
→ [N/A], [N/A], [N/A]

    This makes it impossible to verify real memory availability or fragmentation.

  2. TensorRT logs indicate fallback heuristics

    Failed to infer cluster info for NVIDIA GB10,
treat it as a L40 node with 119 GB memory

  3. Another memory cap message appears during build

    Unknown embedded device detected.
Using 91879MiB as the allocation cap for memory on embedded devices.

    This suggests GB10 is being treated as an embedded platform with an internal memory cap.

  4. Flags such as:

    • reduced max_seq_len

    • reduced max_batch_size

    • padding removal

    • paged KV cache on/off

    • FP8 / NVFP4
      do not prevent the OOM during engine build.

  5. The failure happens during engine serialization / buffer resize, not during inference.

Key Question for NVIDIA / TensorRT-LLM Team

  • Is NVIDIA GB10 officially supported for TensorRT-LLM engine build in this release?

  • Is the virtual memory / allocation cap (≈90 GB) intentional on GB10?

  • Is there a recommended workaround:

    • different container tag?

    • different builder flags?

    • disabling certain plugins (e.g. nvfp4 GEMM)?

  • Is this a known limitation or bug related to:

    • GB10 memory reporting

    • embedded device classification

    • CUDA 13 + TRT-LLM 1.1.0rc3?

Summary

TensorRT-LLM consistently fails to build engines for Llama-3.3-70B-NVFP4 on NVIDIA GB10, due to OOM during engine build, despite:

  • successful model load,

  • idle GPU,

  • large physical memory,

  • and no competing processes.

Any guidance on official support status, memory caps, or recommended configuration for GB10 would be greatly appreciated.

2

First clear caches:

sudo sh -c ‘sync; echo 3 > /proc/sys/vm/drop_caches’

Then, could you run the following commands inside the container:

  • free -h
  • cat /sys/fs/cgroup/memory.max 2>/dev/null || true
  • cat /sys/fs/cgroup/memory.current 2>/dev/null || true