[Architectural Review] RAG Blueprint for Air-Gapped Enterprise Environment on RTX 6000 Blackwell

AI & Data Science NVIDIA Blueprints
, ,
1

Introduction

I am developing a custom RAG solution inspired by NVIDIA AI Blueprints, specifically optimized for an air-gapped, on-premise environment using a single RTX 6000 Blackwell (96GB).

Environment

  • GPU: NVIDIA RTX 6000 Blackwell (96GB VRAM, Single PCIe)

  • OS: Windows 11 + WSL2 (Ubuntu 22.04 LTS)

  • Runtime: Docker Desktop

  • Compliance: Commercial use allowed, strictly non-Chinese software/models, air-gapped deployment.

Proposed Stack & Resource Allocation

Stage Task Software / Model Vendor VRAM Allocation
Parsing Doc Structuring Docling IBM CPU / RAM
Vision Image Reasoning Llama-3.2-11B-Vision Meta ~18GB (vLLM)
Embedding Vectorization NV-Embed-v2 NVIDIA ~2GB (Infinity)
Vector DB Storage Qdrant Qdrant CPU / RAM
Reranking Precision Search mxbai-rerank-large-v1 Mixedbread ~2GB (Infinity)
Inference Final Response Llama-3.1-70B-Instruct (FP8) Meta ~50GB (vLLM)
Serving Engine vLLM / Infinity - Total ~72GB+

Questions for the Community

  1. VRAM Orchestration: I plan to run both vLLM (for LLM/VLM) and mosec/infinity (for Embedding/Rerank) on a single GPU. Since vLLM tends to pre-allocate VRAM, what are the recommended settings (e.g., gpu_memory_utilization) to ensure all these models coexist stably on 96GB without OOM?

  2. Blackwell Architecture Optimization: Are there specific optimizations (like FP8 KV Cache or Blackwell-specific kernels) I should enable in vLLM to maximize the performance of the Llama-3.1-70B model on this specific hardware?

  3. Inference Engine Choice: For a production-ready air-gapped environment, is the combination of vLLM + Infinity the most robust choice for a single Blackwell card, or would you recommend a different unified serving framework (e.g. NVIDIA Triton Inference Server)?

  4. NV-Embed-v2 Local Deployment: Are there any known issues when deploying NV-Embed-v2 via Infinity or similar local serving engines in a WSL2/Docker environment?

I would appreciate any insights or feedback on this architecture.

2

​1. VRAM Orchestration

Your VRAM math is underestimated. Llama-3.1-70B at FP8 (1 byte per parameter) requires ~70GB for weights alone, not ~50GB. Adding KV cache and the 11B Vision model will exceed your 96GB limit.

​Recommendation: Switch the 70B model to INT4/AWQ quantization to reach your ~50GB target.

​vLLM Settings: Control pre-allocation strictly by setting --gpu-memory-utilization 0.75 and limiting --max-model-len.

​2. Blackwell Architecture Optimization

Blackwell natively accelerates FP8 operations.

​vLLM Settings: Add --kv-cache-dtype fp8 to your launch parameters to reduce memory overhead and increase throughput.

​Ensure your WSL2 environment is running the latest CUDA toolkit to support Blackwell’s specific tensor core instructions.

​3. Inference Engine Choice

​vLLM + Infinity is the superior choice for a single GPU. It is lightweight and highly optimized.

​NVIDIA Triton is designed for multi-node clusters. Deploying it on a single, air-gapped RTX 6000 will introduce unnecessary overhead and configuration complexity without tangible benefits.

​4. NV-Embed-v2 Local Deployment

​Requirement: NV-Embed-v2 uses custom architecture. You must pass the --trust-remote-code flag when launching it in Infinity.

​WSL2/Docker: Ensure the NVIDIA Container Toolkit is properly configured in WSL2 and always pass --gpus all to the Docker run command to prevent VRAM allocation failures.