Disaggregated Prefill/Decode using NVIDIA Dynamo (Dual NVIDIA RTX PRO 6000 Blackwell)

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Hello,

This guide walks through setting up NVIDIA Dynamo with disaggregated prefill and decode workers using two GPUs.

Hardware: Dual NVIDIA RTX PRO 6000 Blackwell GPUs
Model Example: Qwen/Qwen3-4B

Create a dedicated Python 3.12 virtual environment for Dynamo:

uv venv .dynamo --python 3.12
source .dynamo/bin/activate
uv pip install pip

Install Dynamo with vLLM Support

sudo apt install python3-dev libxcb1
uv pip install --prerelease=allow "ai-dynamo[vllm]"

Install and run etcd

Dynamo requires etcd as a distributed key-value store.

Download and install etcd from the official releases page:

After installation, verify it runs:

/tmp/etcd-download-test/etcd

Ensure etcd is running before proceeding.

Start NATS Server with JetStream Enabled

Dynamo uses NATS for messaging. Start the NATS server with JetStream enabled:

docker run -p 4222:4222 nats -js

Create the Dynamo Server Script

Create a file named server.sh with the following content:

Source: dynamo/examples/backends/vllm/launch/disagg.sh at main · ai-dynamo/dynamo · GitHub

#!/bin/bash
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

set -e
trap 'echo Cleaning up...; kill 0' EXIT

# Start Dynamo frontend (HTTP server defaults to port 8000)
# You can override with --http-port or DYN_HTTP_PORT
python -m dynamo.frontend &

# Launch Decode Worker on GPU 0
# --enforce-eager is useful for quick deployment, but remove for production
DYN_SYSTEM_PORT=${DYN_SYSTEM_PORT1:-8081} \
CUDA_VISIBLE_DEVICES=0 \
python3 -m dynamo.vllm \
    --model Qwen/Qwen3-4B \
    --is-decode-worker &

# Launch Prefill Worker on GPU 1
DYN_SYSTEM_PORT=${DYN_SYSTEM_PORT2:-8082} \
DYN_VLLM_KV_EVENT_PORT=20081 \
VLLM_NIXL_SIDE_CHANNEL_PORT=20097 \
CUDA_VISIBLE_DEVICES=1 \
python3 -m dynamo.vllm \
    --model Qwen/Qwen3-4B \
    --is-prefill-worker

Make the script executable:

chmod +x ./server.sh

Run the server:

./server.sh

Once the services are running, you can send an OpenAI-compatible request to the Dynamo frontend (default port 8000):

3

*** Please note that this reply is generated by LLM automatically ***

The question seems to be about setting up NVIDIA Dynamo with disaggregated prefill and decode workers using two GPUs, specifically Dual NVIDIA RTX PRO 6000 Blackwell GPUs. However, I will provide a comprehensive answer based on the context provided.

NVIDIA Run:ai is a powerful orchestration layer that simplifies GPU management, boosts utilization, and accelerates AI innovation. It integrates seamlessly with NVIDIA GPU-accelerated Amazon EC2 instances, Amazon Elastic Kubernetes Service (EKS), Amazon SageMaker HyperPod, AWS Identity and Access Management (IAM), Amazon CloudWatch, and other AWS-native services.

The context highlights the challenges of efficient GPU orchestration at scale, including inefficient GPU utilization due to static allocation, lack of workload prioritization and preemption, limited visibility into GPU consumption, and difficulty enforcing governance across teams and workloads.

NVIDIA Run:ai addresses these challenges by providing a unified, enterprise-ready foundation for AI/ML workloads in the cloud. It offers dynamic resource allocation, comprehensive AI life-cycle support, and strategic resource management, significantly enhancing GPU efficiency and workload capacity.

In terms of the question about setting up NVIDIA Dynamo, it seems to be a separate topic. However, I can provide some general information about NVIDIA Dynamo. NVIDIA Dynamo is a software framework that enables the creation of large language models (LLMs) and other AI applications. It provides a scalable and efficient way to train and deploy AI models.

To set up NVIDIA Dynamo with disaggregated prefill and decode workers using two GPUs, you would need to follow the steps outlined in the question, including:

  1. Creating a dedicated Python 3.12 virtual environment for Dynamo
  2. Installing Dynamo with vLLM support
  3. Installing and running etcd as a distributed key-value store
  4. Starting the NATS server with JetStream enabled
  5. Creating the Dynamo server script and making it executable
  6. Running the server

Once the services are running, you can send an OpenAI-compatible request to the Dynamo frontend (default port 8000).

In comparison to NVIDIA Run:ai, NVIDIA Dynamo is a software framework for creating and deploying AI applications, while NVIDIA Run:ai is an orchestration layer for managing GPU resources and accelerating AI innovation. While both are related to AI and GPU computing, they serve different purposes and are used in different contexts.

In terms of the specific GPUs mentioned in the question, Dual NVIDIA RTX PRO 6000 Blackwell GPUs, these are high-end GPUs designed for professional applications such as AI, graphics, and compute. They offer high performance and memory capacity, making them suitable for demanding AI workloads.

In summary, the question about setting up NVIDIA Dynamo with disaggregated prefill and decode workers using two GPUs is a separate topic from NVIDIA Run:ai. However, both are related to AI and GPU computing, and understanding the context and purpose of each can help in setting up and using them effectively.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***