webUI can't find the vLLMs

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
,
1

Hi there,

as a complete Newbie to Spark I try first steps into LLMs.
I got some unix experience with e few Raspberry projects, but that’s another world :-)
Today I got a DGX Spark and installed 4 Docker container on it:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
dd5d3416402e ghcr.io/open-webui/open-webui:main “bash start.sh” 13 minutes ago Up 13 minutes (healthy) open-webui
11834614d306 vllm/vllm-openai:latest “vllm serve --model …” 14 minutes ago Up About a minute vllm-qwen-fp8
856a3fe8e500 lmsysorg/sglang:dev-qwen38-27b-dflash2 “/opt/nvidia/nvidia_…” About an hour ago Restarting (1) 2 seconds ago elegant_austin
d6f286eb6cb0 vllm/vllm-openai:latest “vllm serve --model …” About an hour ago Up 3 minutes 0.0.0.0:8001->8000/tcp, [::]:8001->8000/tcp vllm-qwen-bf16

The downloads of the models seem to be completed:
53G /home/hpk/.cache/huggingface/hub/models–Qwen–Qwen3.8-27B
29G /home/hpk/.cache/huggingface/hub/models–Qwen–Qwen3.8-27B-FP8
2,6G /home/hpk/.cache/huggingface/hub/models–RadixArk–Qwen3.8-27B-DSpark
21G /home/hpk/.cache/huggingface/hub/models–RadixArk–Qwen3.8-27B-NVFP4
6,7G /home/hpk/.cache/huggingface/hub/models–stabilityai–stable-diffusion-xl-base-1.0

The 3 LLMs should be found on port 8001 (Qwen3.8-27B) port 8002 (Qwen3.8-27B-FP8) and port 8003 (Qwen-27B-NVFP4)

webUI is running, however, I can’t connect to any of the LLMs.
I tried several connection settings but I don’t get a connection, e.g.

http://127.0.0.1:8001/v1

http://127.0.0.1:8001

and a few others …

What is missing/wrong?

Thanks for your help!

2

I highly recommend when starting out the Spark, to check out our playbooks. We have quite a few on how to deploy models using different backends and services

3

Thank you, I’ll start over with one of the playbooks.

4

Okay, after going step by step for:
Open WebUI with Ollama | DGX Spark

I get stuck after step 4

Fill out the form with these values:

  • Name: Open WebUI
  • Port: 12000
  • Auto open in browser at the following path: Check this checkbox

Open WebUI is not opening by clicking “OpenWebUI” in nVidia Sync or with “http://localhost:12000”

Maybe there are some misconfiguration still active.

I’ve redone it on a different port. This time I get an error message:

Open WebUI-Backend erforderlich

Ups! Sie verwenden eine nicht unterstützte Methode (nur Frontend). Bitte stellen Sie die WebUI über das Backend bereit.

Siehe readme.md für Anweisungen oder Treten Sie unserem Discord bei, um Hilfe zu erhalten.

5

Next trial … redone step 1 to 5 with success
In step 6 I do not have “Select a model” just

  • New Chat
  • Search
  • Notes
  • Arbeitsbereich (don’t know, how this should be in english
6

Hold on, are you trying to host three different models in the same vllm instance?

Have you checked the vllm logs to make sure it’s actually successfully starting?
Hint: docker logs vllm_node -f

Here is what I suggest for you: Head over to GitHub - eugr/spark-vllm-docker: Docker configuration for running VLLM on dual DGX Sparks · GitHub and checkout the very well supported “community” docker image. The readme documentation is excellent and the premade recipes are great to get started.

Good luck.

7

In addition to the NVIDIA playbooks, you can also check out the OpenZeka Blog. We publish beginner-friendly, step-by-step DGX Spark tutorials covering different models, inference frameworks, and UIs. They include the exact Docker commands, launch parameters, and expected terminal outputs at each step. Many also include ready-to-use container images that we built and tested, which makes reproducing the setups much easier.

For your case, this tutorial covers setting up vLLM + Open WebUI on a single DGX Spark from scratch. For more detailed technical material, we also maintain OpenZeka Whitepapers, with deployment notes, benchmarks, cluster configurations, scaling tests, and performance analysis. Both the tutorials on the blog and the whitepapers are updated regularly as we try new models.

Regarding your original setup: your SGLang container (elegant_austin) is in Restarting (1) state, which means the process is exiting and Docker is repeatedly restarting it. Running “docker logs elegant_austin” should help identify the error that causes it to exit. Also, only vllm-qwen-bf16 shows an explicit published port in the docker ps output (8001->8000). The FP8 and SGLang containers do not show published ports. That does not necessarily mean they are unreachable, but I would check how those containers were started.

For the Ollama playbook issue, I noticed one other possibility: In your original docker ps output, you already had a container named open-webui using the image “/open-webui/open-webui:main”. The NVIDIA playbook uses the same container name but a different image tag, " /open-webui/open-webui:ollama ", which has Ollama bundled into the Open WebUI container. The launch script would reuse the existing open-webui container if one was already present, so it would be worth checking which image is actually running. If this were the case, the later steps of the playbook would not behave as expected.

8

Thank you for your help.

@josephbreda
I had 3 different docker container, the idea was to compare the 3 installations for differences in speed and quality of the output. It was not meant, that all 3 are running simultaneously.

@emretoktas_openzeka
I will use that tutorial for more informations on how to do it properly. Thank you!

With a little help from Claude I was able to set up Qwen3.8 FP8 last night and it runs with Open WebUI, on a local ip, so that I can access it from evrywhere through my reverse proxy server.

However, compared to what I used so far, it’s extremely slow. Even answers on simple questions take minutes, not seconds. There is much room for improvement, and I’m shure, there’s a lot to learn for me.

9

Qwen3.8-27B is a dense model, so you will find it slower than the sparse models that are out there.

I will certainly echo that you look at GitHub - eugr/spark-vllm-docker: Docker configuration for running VLLM on dual DGX Sparks · GitHub

Here’s an example Qwen3.8 recipe that works with eugr’s docker, I’ve had to mod them slightly as I normally run my models across a cluster of 2. Also note that the first request is usually a bit slow, as some warm-up needs to take place.

First, I’d suggest the NVFP4 quant

# Recipe: Qwen3.8-27B-NVFP4
# Qwen3.8-27B model in NVIDIA NVFP4 format

recipe_version: "1"
name: Qwen3.8-27B-NVFP4
description: vLLM serving unsloth/Qwen3.8-27B-NVFP4

# HuggingFace model to download (optional, for --download-model)
model: unsloth/Qwen3.8-27B-NVFP4

# Container image to use
container: vllm-node

#Solo only - remove to go to tensor_parallel: 2 - needs a cluster
solo_only: true

# Default settings (can be overridden via CLI)
defaults:
  port: 8000
  host: 0.0.0.0
  tensor_parallel: 1
  gpu_memory_utilization: 0.85
  max_model_len: 262144
  max_num_seqs: 4
  max_num_batched_tokens: 8192

# Environment variables
env:

# The vLLM serve command template
command: |
  vllm serve unsloth/Qwen3.8-27B-NVFP4 \
    --host {host} \
    --port {port} \
    --tensor-parallel-size {tensor_parallel} \
    --gpu-memory-utilization {gpu_memory_utilization} \
    --max-model-len {max_model_len} \
    --max-num-seqs {max_num_seqs} \
    --max-num-batched-tokens {max_num_batched_tokens} \
    --trust-remote-code \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_xml \
    --enable-auto-tool-choice \
    --distributed-executor-backend mp \
    --speculative-config '{{"method":"mtp","num_speculative_tokens":3}}'

You can also run the FP8 version, but it will be slower:

# Recipe: Qwen3.8-27B-FP8
# Qwen3.8-27B model in NVIDIA FP8 format

recipe_version: "1"
name: Qwen3.8-27B-FP8
description: vLLM serving Qwen/Qwen3.8-27V-FP8

# HuggingFace model to download (optional, for --download-model)
model: Qwen/Qwen3.8-27B-FP8

# Container image to use
container: vllm-node

#Solo only - remove to go to tensor_parallel: 2 - needs a cluster
solo_only: true

# Default settings (can be overridden via CLI)
defaults:
  port: 8000
  host: 0.0.0.0
  tensor_parallel: 1
  gpu_memory_utilization: 0.85
  max_model_len: 262144
  max_num_seqs: 4
  max_num_batched_tokens: 8192

# Environment variables
env:

# The vLLM serve command template
command: |
  vllm serve Qwen/Qwen3.8-27B-FP8 \
    --host {host} \
    --port {port} \
    --tensor-parallel-size {tensor_parallel} \
    --gpu-memory-utilization {gpu_memory_utilization} \
    --max-model-len {max_model_len} \
    --max-num-seqs {max_num_seqs} \
    --max-num-batched-tokens {max_num_batched_tokens} \
    --trust-remote-code \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_xml \
    --enable-auto-tool-choice \
    --distributed-executor-backend mp \
    --speculative-config '{{"method":"mtp","num_speculative_tokens":3}}'

You might also try using the b12x image with these, which includes some optimizations for GB10 hardware. Hopefully this will get you working and experimenting, and there are plenty of other recipes provided with the repo, and quite a few ‘wrappers’ to manage your model downloads etc…

Happy Hunting.

10

Thank you! I’ll have some time to test Friday and on the weekend, I’ll let you know, what I find out!

11

We are about to see a new model land in the Qwen range:

Expect quite a bit if turmoil, while everyone figures ot how to run it, it will take a while, but once done this one is probably going to be more suited to a single GB10 in a NVFP4 quant, than the 27B model, I would expect it to be 60%-80% faster in token generation, with a similar quality.

Things move fast in this land, if you blink, you might miss it.

12

Ignore my recipes, eugr’s repo has been updated with a NVFP4 dflash2 recipe.

Run that one, it is likely to be faster than mine.

qwen3.8-27b-nvfp4-dflash2.yaml

# Recipe: Qwen3.8-27B-NVFP4 with DFlash2 speculative decoding
# Uses TP=1 on a solo DGX Spark and defaults to TP=2 on a cluster.

recipe_version: "1"
name: Qwen3.8-27B-NVFP4-DFlash2
description: vLLM serving RadixArk/Qwen3.8-27B-NVFP4 with the z-lab DFlash2 draft model

model: RadixArk/Qwen3.8-27B-NVFP4
container: vllm-node

cluster_only: false
solo_only: false

defaults:
  port: 8000
  host: 0.0.0.0
  tensor_parallel: 2
  gpu_memory_utilization: 0.7
  max_model_len: 262144
  max_num_seqs: 8
  max_num_batched_tokens: 16384
  num_speculative_tokens: 8

command: |
  vllm serve RadixArk/Qwen3.8-27B-NVFP4 \
    --host {host} \
    --port {port} \
    --trust-remote-code \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization {gpu_memory_utilization} \
    --max-model-len {max_model_len} \
    --max-num-seqs {max_num_seqs} \
    --max-num-batched-tokens {max_num_batched_tokens} \
    --enable-chunked-prefill \
    --async-scheduling \
    --enable-prefix-caching \
    --speculative-config '{{"method":"dflash","model":"z-lab/Qwen3.8-27B-DFlash2","num_speculative_tokens":{num_speculative_tokens},"d
raft_tensor_parallel_size":{tensor_parallel}}}' \
    --load-format instanttensor \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_xml \
    --enable-auto-tool-choice \
    --tensor-parallel-size {tensor_parallel}
13

Thank you @giles8
I see, it’s moving very fast, news every day!

14

With a little help from claude code regarding the optimization I have the system running with good results.

Ergebnis (Qwen3.8-27B-NVFP4, SGLang 0.0.0.dev1+g5f55db35e, DFLASH-Spec, GB10 / Treiber 580.173.02)

┌────────────────────────────────┬────────┬────────────────────────┬─────────────────────────────────────────────┐
│ Workload │ TTFT │ Decode tok/s (Ø / │ Qualität │
│ │ │ min–max) │ │
├────────────────────────────────┼────────┼────────────────────────┼─────────────────────────────────────────────┤
│ Chatbot (7 Prompts ×2, │ ~0,26 │ 42 / 29–60 │ lief durch, 2 Prompts trafen das │
│ Thinking an) │ s │ │ 1024-Token-Limit │
├────────────────────────────────┼────────┼────────────────────────┼─────────────────────────────────────────────┤
│ Coding Python (8 Aufgaben ×2) │ ~0,21 │ 50 / 38–65 │ 16/16 Unit-Tests bestanden │
│ │ s │ │ │
└────────────────────────────────┴────────┴────────────────────────┴─────────────────────────────────────────────┘

sglang.bench_serving shows:

SGLang serving-Benchmark — Qwen3.8-27B-NVFP4, GB10

Workload: random-ids, 1024 in / 512 out Tokens, --backend sglang, temp 0. Der Stack hat währenddessen normal weiterbedient (also Real-World, kein Clean-Room).

┌──────────┬───────┬────────────┬───────────┬──────────────────┬──────────────────┬─────────────────┬───────────┐
│ Parallel │ req/s │ output │ total │ TTFT ms (med / │ TPOT ms (med / │ E2E ms (med / │ accept │
│ │ │ tok/s │ tok/s │ p99) │ p99) │ p99) │ len │
├──────────┼───────┼────────────┼───────────┼──────────────────┼──────────────────┼─────────────────┼───────────┤
│ 1 │ 0,12 │ 59,4 │ 178 │ 565 / 667 │ 14,3 / 34,6 │ 7.783 / 18.240 │ 3,72 │
├──────────┼───────┼────────────┼───────────┼──────────────────┼──────────────────┼─────────────────┼───────────┤
│ 4 │ 0,33 │ 168,0 │ 504 │ 372 / 732 │ 18,8 / 50,8 │ 10.084 / 26.339 │ 3,88 │
├──────────┼───────┼────────────┼───────────┼──────────────────┼──────────────────┼─────────────────┼───────────┤
│ 8 │ 0,40 │ 207,3 │ 622 │ 1.207 / 5.503 │ 27,5 / 68,2 │ 15.035 / 37.011 │ 4,15 │
└──────────┴───────┴────────────┴───────────┴──────────────────┴──────────────────┴─────────────────┴───────────┘

Just using it now for a few days and look how happy I am with the results