Qwen3.8-27B on dual Sparks

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
14

Qwen3.8-27B on DGX Spark — SGLang + DFlash2, fully OpenAI-compatible

A complete, replicable recipe for serving Qwen3.8-27B (NVFP4) with SGLang + DFlash2
block-diffusion speculative decoding on NVIDIA DGX Spark (GB10) — including a set of
patches that make the /v1/responses endpoint genuinely OpenAI-API-compatible, plus an
optional dual-Spark tensor-parallel (TP=2) configuration over RoCE for ~1.6x more speed.

Everything below was measured and validated on real hardware on 2026-08-19.

Note on IP addresses: every IP in this guide (10.100.8.153, 10.100.8.1, etc.)
is from my personal two-Spark setup — yours will differ. Find your own fabric
addresses with ip -br addr show on each node (look for the interfaces on your
ConnectX-7 link, e.g. enp1s0f1np1) and your RoCE device names with ibv_devices,
then substitute them wherever the scripts below use mine.

What you get

Workload (single stream) 1x Spark 2x Spark (TP=2)
Code generation 52–61 tok/s 87 tok/s
Prose / essays 26 tok/s 41 tok/s
Thinking chat 34–49 tok/s 49 tok/s
Code, thinking off (“none”) 52 tok/s ~80 tok/s
TTFT, short prompt ~0.16 s similar
TTFT, repeated 16K prefix 0.44 s 0.74 s
Context window 262,144 262,144

Quality validated identically on both configurations: HumanEval pass@1 159/164
(97.0%) — full suite, every solution executed against its reference tests, greedy,
measured on the TP=2 cluster — plus 10/10 math word problems, structured tool calling,
and tool-eval-bench 92.3 ± 0.6 / 100 (★★★★★, 3 trials). DFlash2 is lossless
speculative decoding — greedy output matches the target model.

Prefill runs at roughly 10K tok/s (a cold 64K-token prompt reaches first token in ~6.4 s);
any repeated prefix (system prompt, conversation history) returns at a flat ~0.3–0.7 s
thanks to SGLang’s radix cache.

Credits

Part 1 — Single Spark

1.0 Prerequisites

  • DGX Spark (GB10, 128 GB unified memory), DGX OS with Docker + NVIDIA Container Toolkit
    (stock DGX Spark image has both).
  • ~100 GB free disk: 57 GB image + ~25 GB weights + caches.
  • Optional: HF_TOKEN in ~/.bashrc for faster Hugging Face downloads (the start
    script picks it up automatically).

1.1 Clone the base repo

cd ~
git clone https://github.com/MiaAI-Lab/Qwen3.8-27B-SGLang-DGX-Spark.git
cd Qwen3.8-27B-SGLang-DGX-Spark
cp .env.sample .env

The .env defaults are sane: NVFP4 weights, native 262K context, YaRN off, 16
concurrent requests. Leave YARN=0 and CONTEXT_LENGTH=262144 — DFlash2 is not
compatible with YaRN context extension on this build.

1.2 Patch start.sh (port/name overrides + persistent log)

Upstream hardcodes port 8888 and the served model name, and its log file stops
recording once boot completes. Save this as startsh.patch in the repo root and apply:

diff --git a/start.sh b/start.sh
--- a/start.sh
+++ b/start.sh
@@ -222,14 +222,14 @@ fi
 MAMBA_SLOTS_PER_REQ=$(( 4 - MAMBA_SKIP_DECODE_LOCK ))
 MAMBA_CACHE_SIZE=$(( MAX_CONCURRENT_REQUESTS * MAMBA_SLOTS_PER_REQ ))
 
-SERVED_MODEL_NAME="qwen3.8-27b-sglang"
+SERVED_MODEL_NAME="${SERVED_MODEL_NAME:-qwen3.8-27b-sglang}"
 # Image override (shell env wins): lets start-dspark.sh run a patched
 # derivative image (e.g. qwen38-tier-a:local) and roll back to stock by
 # simply not setting IMAGE. Not documented in README/CHANGELOG on purpose.
 IMAGE="${IMAGE:-lmsysorg/sglang:qwen38-27b}"
 CONTAINER_NAME="qwen3.8-27b-sglang"
 HOST="0.0.0.0"
-PORT="8888"
+PORT="${PORT:-8888}"
 PID_FILE=".sglang.pid"
 LOG_FILE=".sglang.log"
 WORK_DIR="$(pwd)"
@@ -373,4 +373,10 @@ echo "Anthropic-compatible: http://${HOST}:${PORT}/v1/messages (no /v1 suffix in
 echo "Served model name: ${SERVED_MODEL_NAME}"
 echo "Thinking: ON by default (disable per request: chat_template_kwargs {\"enable_thinking\": false})"
 
+# Keep appending the live server log to ${LOG_FILE} after this script exits
+# (the EXIT trap kills the boot-time follower; this detached one lives until
+# the container stops).
+nohup docker logs -f --tail 0 "${CONTAINER_NAME}" >> "${LOG_FILE}" 2>&1 &
+disown
+
 echo "SGLang is ready and responding; shell is now free."

git apply startsh.patch

Then (optionally) pin your port and model name in .env, e.g.:

cat >> .env <<'EOF'
PORT=8078
SERVED_MODEL_NAME=qwen38-27b
EOF

1.3 The OpenAI-compatibility patch

This is the core contribution of this guide. SGLang’s /v1/responses endpoint (as of
the qwen38-27b cookbook image + DFlash2 merge) breaks against real OpenAI-SDK
clients in seven ways (plus an eighth fix for benchmark tooling). This patch fixes all of them:

# Problem Fix
1 reasoning.effort: "none" accepted, then crashes mid-stream (strict openai event types reject it) Full OpenAI tier mapping: none→thinking OFF, minimal→low, high/max→xhigh
2 effort: minimal / high / max → 400 (Qwen template only knows low/medium/xhigh) Same mapping
3 Echoed xhigh crashes strict SDK event types Echo sanitized: xhigh→high, none→omitted
4 SDK-echoed assistant history → 400 (ValidatorIterator, logprobs: None) — breaks every multi-turn conversation and tool round-trip Request input validated as plain dicts, killing pydantic’s lazy-Iterable validation class of failures
5 text.format (json_schema / json_object structured output) silently ignored Wired into constrained decoding (json_schema sampling param)
6 Usage returned in chat-completions shape — clients read input_tokens_details.cached_tokens, see nothing, report “cache hit 0%” forever Usage emitted in proper Responses-API shape (legacy keys kept)
7 effort: "none" semantics: clients mean “thinking off” enable_thinking=false + model-card non-thinking sampling (temp 0.7 / top_p 0.8) when client sets neither
8 Streaming + return_token_ids → 400 (breaks llama-benchy ≥0.4.0 token counting) Flag dropped gracefully on streaming instead of rejecting

With the patch, the following all pass against the official OpenAI Python SDK:
every effort tier (none/minimal/low/medium/high/xhigh/max) streaming and
non-streaming, multi-turn history echo (typed SDK objects incl. reasoning items),
function-tool round-trips, previous_response_id chaining, instructions + sampling
params, json_schema structured output, system-role messages, and cache-hit reporting.

Save the following as patch/openai-responses-compat.patch (verbatim — it applies
cleanly onto the files shipped in the lmsysorg/sglang:qwen38-27b image):

--- a/python/sglang/srt/entrypoints/openai/serving_responses.py
+++ b/python/sglang/srt/entrypoints/openai/serving_responses.py
@@ -190,6 +190,26 @@
         if not self.tokenizer_manager:
             return self.create_error_response("Model not loaded")
 
+        # Map the OpenAI reasoning-effort tiers onto what the Qwen3.8 chat
+        # template supports (low / medium / xhigh — anything else raises a
+        # jinja error mid-request). Clients legitimately send any OpenAI tier
+        # ("none" ... "max"); unknown values land on "low". The echo back to
+        # the client is re-sanitized in ResponsesResponse.from_request, since
+        # the strict openai event types don't allow "xhigh".
+        if request.reasoning is not None and request.reasoning.effort is not None:
+            request.reasoning.effort = {
+                # "none" is preserved: _make_request turns it into
+                # enable_thinking=False (thinking fully off) plus the model
+                # card's non-thinking sampling defaults.
+                "none": "none",
+                "minimal": "low",
+                "low": "low",
+                "medium": "medium",
+                "high": "xhigh",
+                "max": "xhigh",
+                "xhigh": "xhigh",
+            }.get(str(request.reasoning.effort).lower(), "low")
+
         # FIXME: If the engine is dead, raise an error
         # This is required for the streaming case
 
@@ -478,11 +498,50 @@
     ):
         messages = self._construct_input_messages(request, prev_response)
 
+        # Wire the Responses API ``text.format`` structured-output config
+        # through to chat's ``response_format`` (it was silently ignored
+        # before, so json_schema requests came back as prose).
+        response_format = None
+        text_cfg = getattr(request, "text", None)
+        if isinstance(text_cfg, dict):
+            fmt = text_cfg.get("format") or {}
+            ftype = fmt.get("type") if isinstance(fmt, dict) else None
+            if ftype == "json_schema" and fmt.get("schema") is not None:
+                response_format = {
+                    "type": "json_schema",
+                    "json_schema": {
+                        "name": fmt.get("name") or "response",
+                        "description": fmt.get("description"),
+                        "schema": fmt.get("schema"),
+                        "strict": fmt.get("strict"),
+                    },
+                }
+            elif ftype == "json_object":
+                response_format = {"type": "json_object"}
+
+        # effort "none" = thinking fully off (clients use it as their "off"
+        # tier). The chat template has no "none" effort, so it maps to
+        # enable_thinking=False; sampling then follows the model card's
+        # non-thinking recommendation (temp 0.7 / top_p 0.8) unless the
+        # client set its own values. Lower temperature also lifts DFlash2
+        # draft acceptance, so this is the fast lane for code tasks.
+        reasoning_effort = request.reasoning.effort if request.reasoning else None
+        chat_template_kwargs = None
+        if reasoning_effort == "none":
+            reasoning_effort = None
+            chat_template_kwargs = {"enable_thinking": False}
+            if request.temperature is None:
+                request.temperature = 0.7
+            if request.top_p is None:
+                request.top_p = 0.8
+
         chat_tools = self._response_tools_to_chat_tools(request)
         chat_request = ChatCompletionRequest(
             model=request.model,
             messages=messages,
             stream=request.stream,
+            response_format=response_format,
+            chat_template_kwargs=chat_template_kwargs,
             tools=chat_tools or None,
             tool_choice=request.tool_choice if chat_tools else "none",
             parallel_tool_calls=(
@@ -491,7 +550,7 @@
                 else True
             ),
             stop=request.stop,
-            reasoning_effort=(request.reasoning.effort if request.reasoning else None),
+            reasoning_effort=reasoning_effort,
         )
 
         media_error = self._validate_media_content(chat_request)
@@ -898,6 +957,18 @@
         if message.get("role") == "developer":
             message = {**message, "role": "system"}
 
+        # ``ResponseOutputMessageParam.content`` is typed ``Iterable``, which
+        # pydantic validates lazily -- assistant history echoed back by a
+        # client arrives here with content as a ValidatorIterator, skipping
+        # the list branch below and leaking raw ``output_text`` parts into
+        # ChatCompletionRequest. Materialize any non-str/list content first.
+        _content = message.get("content")
+        if _content is not None and not isinstance(_content, (str, list)):
+            try:
+                message = {**message, "content": list(_content)}
+            except TypeError:
+                pass
+
         msg_type = message.get("type")
         if msg_type == "function_call":
             # Coerce ``arguments`` to a valid JSON-object string so the chat
--- a/python/sglang/srt/entrypoints/openai/protocol.py
+++ b/python/sglang/srt/entrypoints/openai/protocol.py
@@ -1496,7 +1496,13 @@
     ] = None
     # Accept dict-shaped items as the loose arm; downstream normalization
     # handles replayed shapes that don't satisfy every openai TypedDict.
-    input: Union[str, List[ResponseInputOutputItem], List[Dict[str, Any]]]
+    # NOTE: the typed openai arm was removed on purpose. Its TypedDicts carry
+    # ``Iterable`` fields (message content, logprobs, annotations) that
+    # pydantic validates LAZILY, so a shallowly-valid item wins the union and
+    # then explodes as a ValidatorIterator deep inside the serving layer
+    # (e.g. SDK-echoed assistant history with ``logprobs: None``) instead of
+    # falling back to the dict arm. All downstream handling is dict-based.
+    input: Union[str, List[Dict[str, Any]]]
     instructions: Optional[str] = None
     max_output_tokens: Optional[int] = None
     max_tool_calls: Optional[int] = None
@@ -1509,6 +1515,9 @@
     store: Optional[bool] = True
     stream: Optional[bool] = False
     temperature: Optional[float] = None
+    # Structured-output config, e.g. {"format": {"type": "json_schema", ...}}.
+    # Wired through to chat response_format in serving_responses._make_request.
+    text: Optional[Dict[str, Any]] = None
     tool_choice: Literal["auto", "required", "none"] = "auto"
     tools: List[ResponseTool] = Field(default_factory=list)
     top_logprobs: Optional[int] = 0
@@ -1645,6 +1654,17 @@
             if key not in params or params[key] is None:
                 params[key] = value
 
+        # Structured output via the Responses API ``text.format`` config —
+        # the actual generation constraint lives in sampling params, not in
+        # the chat-request object built for template rendering.
+        if isinstance(self.text, dict):
+            fmt = self.text.get("format") or {}
+            ftype = fmt.get("type") if isinstance(fmt, dict) else None
+            if ftype == "json_schema" and fmt.get("schema") is not None:
+                params["json_schema"] = convert_json_schema_to_str(fmt["schema"])
+            elif ftype == "json_object":
+                params["json_schema"] = '{"type": "object"}'
+
         has_existing_constraints = (
             params.get("regex")
             or params.get("ebnf")
@@ -1689,7 +1709,9 @@
         Union[ResponseOutputItem, ResponseReasoningItem, ResponseFunctionToolCall]
     ] = Field(default_factory=list)
     status: Literal["queued", "in_progress", "completed", "failed", "cancelled"]
-    usage: Optional[UsageInfo] = None
+    # Dict shape follows the OpenAI Responses API usage schema (see
+    # from_request); UsageInfo accepted for any legacy internal callers.
+    usage: Optional[Union[Dict[str, Any], UsageInfo]] = None
     parallel_tool_calls: bool = True
     tool_choice: str = "auto"
     tools: List[ResponseTool] = Field(default_factory=list)
@@ -1763,13 +1785,45 @@
 
         text_format = {"format": {"type": "text"}} if _is_text_only(output) else None
 
+        # Convert chat-style UsageInfo to the OpenAI Responses API usage shape
+        # (input_tokens / input_tokens_details.cached_tokens / output_tokens /
+        # output_tokens_details.reasoning_tokens). Clients read those fields
+        # for cache-hit and token accounting; the legacy chat keys are kept
+        # alongside for back-compat.
+        usage_payload = None
+        if usage is not None:
+            u = usage.model_dump() if hasattr(usage, "model_dump") else dict(usage)
+            details = u.get("prompt_tokens_details") or {}
+            usage_payload = {
+                "input_tokens": u.get("prompt_tokens", 0),
+                "input_tokens_details": {
+                    "cached_tokens": details.get("cached_tokens", 0) or 0
+                },
+                "output_tokens": u.get("completion_tokens", 0),
+                "output_tokens_details": {
+                    "reasoning_tokens": u.get("reasoning_tokens", 0) or 0
+                },
+                "total_tokens": u.get("total_tokens", 0),
+                "prompt_tokens": u.get("prompt_tokens", 0),
+                "completion_tokens": u.get("completion_tokens", 0),
+            }
+
+        # The strict openai types used for stream events only accept
+        # minimal/low/medium/high in the echoed reasoning.effort; sanitize
+        # our extended tiers ("xhigh"/"max"/"none") at the boundary.
+        echo_effort = request.reasoning.effort if request.reasoning else None
+        if echo_effort is not None:
+            echo_effort = {"xhigh": "high", "max": "high", "none": None}.get(
+                echo_effort, echo_effort
+            )
+
         return cls(
             id=request.request_id,
             created_at=created_time,
             model=model_name,
             output=output,
             status=status,
-            usage=usage,
+            usage=usage_payload,
             parallel_tool_calls=(
                 request.parallel_tool_calls
                 if request.parallel_tool_calls is not None
@@ -1784,7 +1838,7 @@
             max_output_tokens=request.max_output_tokens,
             previous_response_id=request.previous_response_id,  # TODO(v): ensure this is propagated if retrieved from store
             reasoning={
-                "effort": request.reasoning.effort if request.reasoning else None,
+                "effort": echo_effort,
                 "summary": None,  # unused
             },
             store=request.store,
--- a/python/sglang/srt/entrypoints/openai/serving_chat.py
+++ b/python/sglang/srt/entrypoints/openai/serving_chat.py
@@ -925,11 +925,12 @@
                     "Please set stream=false when using return_prompt_token_ids=true."
                 )
             if request.return_token_ids:
-                raise ValueError(
-                    "return_token_ids is not supported with streaming on "
-                    "/v1/chat/completions. Please set stream=false when using "
-                    "return_token_ids=true."
-                )
+                # Degrade gracefully instead of rejecting: benchmark clients
+                # (e.g. llama-benchy >= 0.4.0) send return_token_ids on
+                # streaming requests to count real tokens under speculative
+                # decoding; dropping the flag lets them fall back to counting
+                # via usage / a local tokenizer rather than failing the run.
+                request.return_token_ids = False
             if request.return_meta_info:
                 raise ValueError(
                     "return_meta_info is not supported with streaming. "

1.4 Apply the patch into the DFlash2 overlay

The repo builds its DFlash2 image by overlaying a handful of files onto the pinned
cookbook image (patch/build-dflash2-image.sh --minimal, sha256-verified). We add our
two patched files to that same mechanism. From the repo root:

# 1. Extract the pristine files from the base image (pulls ~57 GB on first run)
cid=$(docker create lmsysorg/sglang:qwen38-27b true)
mkdir -p /tmp/compat/python/sglang/srt/entrypoints/openai
for f in serving_responses.py protocol.py serving_chat.py; do
  docker cp "$cid:/sgl-workspace/sglang/python/sglang/srt/entrypoints/openai/$f" \
    /tmp/compat/python/sglang/srt/entrypoints/openai/
done
docker rm "$cid"

# 2. Apply the compat patch
cd /tmp/compat && patch -p1 < ~/Qwen3.8-27B-SGLang-DGX-Spark/patch/openai-responses-compat.patch
cd ~/Qwen3.8-27B-SGLang-DGX-Spark

# 3. Install the patched files into the overlay
mkdir -p patch/overlay-dflash2/sglang/srt/entrypoints/openai
cp /tmp/compat/python/sglang/srt/entrypoints/openai/*.py \
   patch/overlay-dflash2/sglang/srt/entrypoints/openai/

# 4. Register them in the overlay manifest
cd patch/overlay-dflash2
sha256sum sglang/srt/entrypoints/openai/serving_responses.py \
          sglang/srt/entrypoints/openai/protocol.py \
          sglang/srt/entrypoints/openai/serving_chat.py >> MANIFEST.sha256
sha256sum -c MANIFEST.sha256   # every line must say OK
cd ../..

# 5. Add them to the build script's file list
sed -i '/srt\/speculative\/dflash_worker_v2.py/a\    srt/entrypoints/openai/serving_responses.py\n    srt/entrypoints/openai/protocol.py\n    srt/entrypoints/openai/serving_chat.py' \
  patch/build-dflash2-image.sh

1.5 Build the image

./patch/build-dflash2-image.sh --minimal
# -> built lmsysorg/sglang:qwen38-27b-dflash2-minoverlay

This is a pure local overlay onto the pinned image — no network needed beyond the base
image pull, and every overlaid file is checksum-verified.

1.6 Start

DF_EXTRA="--sleep-on-idle" IMAGE=lmsysorg/sglang:qwen38-27b-dflash2-minoverlay ./start-dflash.sh

DF_EXTRA="--sleep-on-idle" fixes a significant idle power draw: without it the SGLang
scheduler busy-spins at ~97% CPU whenever the server is idle
(MiaAI repo issue #4).

First boot downloads the weights into ./.cache/huggingface (RadixArk NVFP4 ~16.5 GB +
DFlash2 draft ~2.6 GB, pinned revisions) and compiles kernels; allow ~10–20 minutes.
Warm restarts take ~2–3 minutes. The script exits when the server answers.

1.7 Verify

PORT=8078   # whatever you set in .env (default 8888)

curl -s http://127.0.0.1:$PORT/v1/models | python3 -m json.tool

# Chat completions, thinking off:
curl -s http://127.0.0.1:$PORT/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "qwen38-27b",
  "messages": [{"role": "user", "content": "Say OK."}],
  "max_tokens": 20,
  "chat_template_kwargs": {"enable_thinking": false}}' | python3 -m json.tool

# Responses API with effort "none" (thinking off — this crashed before the patch):
curl -s http://127.0.0.1:$PORT/v1/responses -H 'Content-Type: application/json' -d '{
  "model": "qwen38-27b",
  "input": "Say OK.",
  "max_output_tokens": 50,
  "reasoning": {"effort": "none"}}' | python3 -m json.tool

The second /v1/responses call to an identical prompt should show
usage.input_tokens_details.cached_tokens > 0 — that’s your prefix cache working and
being reported correctly.

Logs: tail -f .sglang.log (persists across the boot script exiting, thanks to the
step-1.2 patch) or docker logs -f qwen3.8-27b-sglang.

Stop: ./stop.sh

Part 2 — Dual Spark (TP=2 over RoCE)

Shards the model across both Sparks with tensor parallelism. Decode on GB10 is
memory-bandwidth-bound, so halving each node’s weight reads gives ~1.5–1.6x on every
workload. Same endpoint, same model name — clients don’t change.

As far as we know this (DFlash2 + cross-node TP + hybrid-GDN model + NVFP4 on GB10)
was first booted 2026-08-19 using this exact recipe. It came up on the first try and
passed the full quality gate, but treat it as experimental.

2.0 Prerequisites

  • Two DGX Sparks connected back-to-back on their ConnectX-7 QSFP ports (no switch
    needed), one IP per node on the same subnet. This guide uses my addresses —
    substitute your own (see the note at the top; ip -br addr show on each node):
    • SPARK1 = 10.100.8.153 (rank 0, serves HTTP)
    • SPARK2 = 10.100.8.1 (rank 1, headless)
  • MTU 9000 on the fabric interfaces of both nodes (sudo ip link set dev enp1s0f1np1 mtu 9000 — make it persistent via netplan). NCCL will appear to hang at
    warmup with default MTU.
  • Passwordless SSH from spark1 to spark2.
  • Part 1 completed on spark1 (patched image built, weights downloaded, verified).

Each PCIe link on the CX-7 shows up as two RoCE devices; using both twins
(rocep1s0f1,roceP2p1s0f1) is what gets full NCCL bandwidth.

2.1 Copy the image and weights to spark2

From spark1 (adjust IPs/paths):

SPARK2=10.100.8.1

# Image (~57 GB, a few minutes over the fabric)
docker save lmsysorg/sglang:qwen38-27b-dflash2-minoverlay | ssh $SPARK2 docker load

# Weights (~25 GB)
ssh $SPARK2 "mkdir -p ~/qwen38-dflash2/.cache"
rsync -a ~/Qwen3.8-27B-SGLang-DGX-Spark/.cache/huggingface \
  $SPARK2:~/qwen38-dflash2/.cache/

2.2 The launch script

Save as start-dflash2-tp2.sh on spark1 (adjust the variables at the top):

#!/bin/bash
# Qwen3.8-27B NVFP4 + DFlash2 sharded across two DGX Sparks (TP=2 over RoCE).
# Serves on THIS node (rank 0) — same port/model name as the solo recipe.
set -euo pipefail

NODE1=10.100.8.1                # spark2 (rank 1, no HTTP)
DIST=10.100.8.153:50051         # rank-0 rendezvous (spark1's fabric IP)
IMAGE=lmsysorg/sglang:qwen38-27b-dflash2-minoverlay
NAME=qwen38-tp2
PORT=8078                       # match your solo .env
LOCAL_CACHE=$HOME/Qwen3.8-27B-SGLang-DGX-Spark/.cache
REMOTE_CACHE=/home/$USER/qwen38-dflash2/.cache

ARGS="--model-path RadixArk/Qwen3.8-27B-NVFP4 \
 --served-model-name qwen38-27b --trust-remote-code \
 --tp 2 --nnodes 2 --dist-init-addr ${DIST} \
 --attention-backend flashinfer --chunked-prefill-size 8192 \
 --disable-prefill-cuda-graph --kv-cache-dtype fp8_e4m3 \
 --mamba-ssm-dtype bfloat16 --mamba-full-memory-ratio 4.21 \
 --mamba-radix-cache-strategy extra_buffer \
 --max-mamba-cache-size 64 --max-running-requests 16 \
 --context-length 262144 \
 --speculative-algorithm DFLASH \
 --speculative-draft-model-path z-lab/Qwen3.8-27B-DFlash2 \
 --speculative-draft-model-revision 50307d4c4cde6860d4eee73e2547cd786fe8e8a4 \
 --speculative-num-draft-tokens 8 \
 --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
 --sampling-defaults model --enable-metrics --enable-cache-report \
 --stream-interval 1 --sleep-on-idle \
 --mem-fraction-static 0.90 --host 0.0.0.0 --port ${PORT}"
# --stream-interval 1: one token per SSE event — DFlash2 otherwise batches
#   ~2-4 accepted tokens per event, breaking clients that count stream chunks.
# --sleep-on-idle: without it the scheduler busy-spins ~97% CPU when idle
#   (MiaAI repo issue #4) — significant idle power draw on a 24/7 box.

ENVS="-e NCCL_IB_HCA=rocep1s0f1,roceP2p1s0f1 \
 -e NCCL_SOCKET_IFNAME=enp1s0f1np1 -e GLOO_SOCKET_IFNAME=enp1s0f1np1 \
 -e NCCL_CUMEM_ENABLE=0 -e NCCL_NVLS_ENABLE=0 \
 -e HF_HOME=/root/.cache/huggingface -e TRITON_CACHE_DIR=/root/.triton"

DOCKER_FLAGS="--network host --ipc host --privileged --gpus all --shm-size 32g"

echo "Stopping solo server if present..."
docker stop qwen3.8-27b-sglang >/dev/null 2>&1 || true

echo "Starting rank 1 on ${NODE1}..."
ssh -o BatchMode=yes "${NODE1}" "docker rm -f ${NAME} >/dev/null 2>&1 || true; \
  mkdir -p ${REMOTE_CACHE}/triton; \
  docker run -d --name ${NAME} ${DOCKER_FLAGS} ${ENVS} \
    -v ${REMOTE_CACHE}/huggingface:/root/.cache/huggingface \
    -v ${REMOTE_CACHE}/triton:/root/.triton \
    ${IMAGE} python3 -m sglang.launch_server ${ARGS} --node-rank 1" >/dev/null

echo "Starting rank 0 locally..."
docker rm -f ${NAME} >/dev/null 2>&1 || true
docker run -d --name ${NAME} ${DOCKER_FLAGS} ${ENVS} \
  -v ${LOCAL_CACHE}/huggingface:/root/.cache/huggingface \
  -v ${LOCAL_CACHE}/triton:/root/.triton \
  ${IMAGE} python3 -m sglang.launch_server ${ARGS} --node-rank 0 >/dev/null

echo "Waiting for readiness (boot takes a few minutes)..."
i=0
until curl -fsS -m 2 "http://127.0.0.1:${PORT}/v1/models" >/dev/null 2>&1; do
  if ! docker ps --format '{{.Names}}' | grep -qx "${NAME}"; then
    echo "rank 0 exited — last log lines:"; docker logs --tail 40 "${NAME}" 2>&1 | tail -40; exit 1
  fi
  (( i % 6 == 0 )) && echo "  still starting... (docker logs -f ${NAME})"
  i=$((i+1)); sleep 5
done
echo "TP=2 cluster READY on http://127.0.0.1:${PORT}/v1 (model qwen38-27b)"
LOG=$HOME/.sglang-tp2.log
nohup docker logs -f --tail 0 ${NAME} >> "${LOG}" 2>&1 &
disown
echo "Persistent log: ${LOG}"

And stop-dflash2-tp2.sh:

#!/bin/bash
docker stop qwen38-tp2 >/dev/null 2>&1 && echo "stopped rank 0" || echo "rank 0 not running"
ssh -o BatchMode=yes 10.100.8.1 "docker stop qwen38-tp2 >/dev/null 2>&1" \
  && echo "stopped rank 1" || echo "rank 1 not running"

chmod +x start-dflash2-tp2.sh stop-dflash2-tp2.sh

2.3 Run and switch between modes

# Solo -> TP=2
./stop.sh && ./start-dflash2-tp2.sh

# TP=2 -> solo
./stop-dflash2-tp2.sh && IMAGE=lmsysorg/sglang:qwen38-27b-dflash2-minoverlay ./start-dflash.sh

Both serve the identical endpoint and model name; only the script you run changes.
Weights and compiled-kernel caches are shared, so switching takes ~2–5 minutes. The
radix cache resets on any restart — the first turn afterwards prefills cold.

Logs: docker logs -f qwen38-tp2 (rank 0 gets all request lines);
ssh <spark2> docker logs -f qwen38-tp2 for rank 1.

Client integration notes

  • Endpoint: http://<spark1>:<PORT>/v1 — OpenAI-compatible chat completions and
    Responses API. An Anthropic-style /v1/messages is also exposed.
  • Reasoning effort: send any OpenAI tier. none = thinking off (fast lane, ~2x
    fewer tokens and faster decode); low/medium for agent work; xhigh = native
    maximum (the model’s default if you send nothing — watch out, it thinks a LOT).
  • Thinking off on chat completions: "chat_template_kwargs": {"enable_thinking": false}.
  • Measuring tok/s: DFlash2 emits ~8 stream events/s with several tokens per event.
    Always compute rates from usage (stream_options: {"include_usage": true} on chat),
    never by counting SSE chunks.
  • Cache hits: reported in usage.input_tokens_details.cached_tokens (Responses)
    and usage.prompt_tokens_details.cached_tokens (chat). Hits are counted in 64-token
    pages; turn 1 of any fresh session is always 0.
  • Sampling: speculative acceptance (and therefore speed) improves at lower
    temperature. For codegen, temperature ≤0.3 measurably helps; effort: "none"
    already defaults to the model card’s non-thinking params.

Troubleshooting

Symptom Cause / fix
Hard reboot during DFlash2 startup (“Capture target verify CUDA graph”) Keep --mem-fraction-static 0.90 for NVFP4 (0.95 has caused GB10 hard reboots); the overlay’s in-place quantized-head selector must be in the image
AttributeError ... max_position_embeddings at boot YaRN/context >262144 leaking into the draft config — keep YARN=0, CONTEXT_LENGTH=262144
400 “Unexpected reasoning effort …” You’re on an unpatched image — the compat patch maps all OpenAI tiers
TP=2 hangs at NCCL warmup, GPUs pinned ~96% util at ~14 W Fabric MTU not 9000 on both nodes, or wrong NCCL_IB_HCA device names (ibv_devices lists them)
Client says “cache hit 0%” forever Unpatched usage shape; also remember restarts wipe the cache and turn 1 is always 0
First long prompt after boot is slow (~13 s vs ~8 s at 16K) Triton kernel warmup; .cache/triton persists across restarts, so it’s once per cold cache
High power draw / hot CPU while server is idle SGLang scheduler busy-spins ~97% CPU without --sleep-on-idle — add the flag (see MiaAI issue #4)
Benchmark tools report ~4x too-low tok/s Client counts SSE chunks, not tokens — DFlash2 packs ~2-4 tokens/event. Use llama-benchy ≥0.4.0 with --tokenizer, or serve with --stream-interval 1
tok/s looks impossibly low in a client It’s counting stream events, not tokens — see client notes

Known limitations

  • /v1/responses input items of exotic types (item_reference, MCP tool calls,
    web_search_call echoes) return a clear 400 — they need server-side tool execution
    this build doesn’t ship. Function tools work fully.
  • DFlash2/DSpark are incompatible with YaRN context extension (>262K) on this build;
    the MTP mode in the base repo supports 1M if you need it (slower).
  • TP=2 adds ~0.3 s to cached-turn TTFT (per-layer allreduce) — irrelevant in practice.
  • Restarts drop the radix cache (in-memory only).

Validated 2026-08-19 on two DGX Sparks (GB10, 128 GB), DGX OS, Docker 29.x,
lmsysorg/sglang:qwen38-27b base image, SGLang DFlash2 commit c14312a66,
RadixArk/Qwen3.8-27B-NVFP4 + z-lab/Qwen3.8-27B-DFlash2@50307d4.