Running whisper.cpp STT server on DGX Spark (GB10, ARM64 + CUDA 13) via Docker

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Hey everyone, I got whisper.cpp running as a dockerized STT server on the DGX Spark and wanted to share the full process since there are a few non-obvious gotchas specific to this platform.

System

  • Lenovo ThinkStation PGX / NVIDIA DGX Spark

  • GB10 Blackwell Superchip, SM 12.1

  • CUDA 13.0, Ubuntu 24.04 (DGX OS)

  • whisper.cpp v1.8.4

Step 1: Clone the repo

There are no pre-built ARM64+CUDA binaries in the whisper.cpp releases, so you need to build from source inside Docker.

bash

git clone --depth 1 --branch v1.8.4 https://github.com/ggml-org/whisper.cpp.git
cd whisper.cpp

Step 2: Download a model

bash

wget -P ./models \
  https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-large-v3-turbo.bin

large-v3-turbo is a great choice — same encoder quality as large-v3 but with a much faster 4-layer decoder. For lower latency use cases, small or medium also work well.

Step 3: Set up .dockerignore

whisper.cpp ships with a .dockerignore that blocks the build directory. Replace it to keep the build context clean:

bash

cat > .dockerignore << 'EOF'
.git
.hermes
models
build
*.bin
*.log
EOF

Step 4: Create the Dockerfile

Several gotchas specific to the DGX Spark:

  • Don’t use nvcr.io/nvidia/cuda — ARM64 tags aren’t available there. Use docker.io/nvidia/cuda instead.

  • Use Ubuntu 24.04, not 22.04. The DGX OS ships with GLIBC 2.38; Ubuntu 22.04 containers only have GLIBC 2.35 and will fail with version GLIBC_2.38 not found.

  • Build inside the container — copying host-compiled binaries into a container will fail with CUDA error: no kernel image is available for execution on the device.

  • Target both 120 and 121 — the GB10 is SM 12.1. Using only 120 compiles for sm_120a which is not compatible. You must specify "120;121".

  • Only build the whisper-server target — building everything fails because libcuda.so.1 is a driver library not available at image build time.

  • Use CUDA stubs for linking at build time.

dockerfile

FROM nvidia/cuda:13.0.3-devel-ubuntu24.04

RUN apt-get update && apt-get install -y \
    build-essential cmake git libgomp1 ffmpeg && \
    rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY . .

RUN rm -rf build && \
    cmake -B build \
      -DGGML_CUDA=ON \
      -DCMAKE_CUDA_ARCHITECTURES="120;121" \
      -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_EXE_LINKER_FLAGS="-L/usr/local/cuda/lib64/stubs -lcuda" && \
    cmake --build build --target whisper-server -j$(nproc)

COPY models/ggml-large-v3-turbo.bin ./models/

EXPOSE 9000

ENTRYPOINT ["./build/bin/whisper-server"]

bash

docker build -f Dockerfile.whisper -t whisper-server .

Step 5: Docker Compose

Use the deploy.resources style for GPU access — runtime: nvidia throws unknown or invalid runtime name: nvidia on DGX OS:

yaml

services:
  whisper-server:
    image: whisper-server
    container_name: whisper-server
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
    ports:
      - "9000:9000"
    command: >
      --model /app/models/ggml-large-v3-turbo.bin
      --host 0.0.0.0
      --port 9000
      --flash-attn
    restart: unless-stopped
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

bash

docker compose up -d whisper-server
curl http://localhost:9000/health

Expected output on success

ggml_cuda_init: found 1 CUDA devices (Total VRAM: 122502 MiB):
  Device 0: NVIDIA GB10, compute capability 12.1, VMM: yes
whisper_model_load: CUDA0 total size = 1623.92 MB
whisper_backend_init_gpu: using CUDA0 backend
whisper_init_with_params_no_state: flash attn = 1
whisper server listening at http://0.0.0.0:9000

Step 6: Test it

From the DGX itself:

bash

curl http://localhost:9000/inference \
  -F "file=@/home/admin/whisper.cpp/samples/jfk.wav" \
  -F "language=en" \
  -F "response_format=json"

From any machine on the network using Python — no need to copy files to the DGX, just stream them directly:

python

import requests

url = "http://<DGX-LAN-IP>:9000/inference"  # e.g. 192.168.2.148

with open("audio.wav", "rb") as f:
    response = requests.post(
        url,
        files={"file": ("audio.wav", f, "audio/wav")},
        data={
            "language": "en",  # or "ja" for Japanese etc.
            "response_format": "json",
        }
    )

print(response.json()["text"])

Summary of gotchas

Issue Fix
No ARM64 pre-built binaries Build from source inside Docker
nvcr.io has no ARM64 tags Use docker.io/nvidia/cuda
GLIBC 2.38 mismatch Use ubuntu24.04 base image
CUDA error: no kernel image Build inside container, target "120;121" not just "120"
libcuda.so.1 not found at build time Use --target whisper-server + CUDA stubs
.dockerignore blocks build context Replace .dockerignore before building
runtime: nvidia not recognized Use deploy.resources in compose

Hope this saves someone a few hours. Happy to answer questions!

2

thank you for this. how is the speed? usable in a conversation mode usage?

3

I just had a discussion the other day with one of my Hermes agent profiles about leveraging Discord voice channels for STT/TTS communication instead of text-only. I noticed you have .hermes in your .dockerignore which leads me to believe you’re using Hermes Agent as well. If so, how well is this working for you?

4

Hey there, I am still in the middle of testing myself…let me know your results. I will also share my experience once I ran some tests!

5

Yes also using Hermes Agent, using Discord voice channels sounds like a fun thing to try…let me know if you managed to successfully integrate it!

6

I have tried to use your recipe, but it is not work normally with actual version of Hermes.

So, I have built simple API over the GitHub - Mekopa/whisperx-blackwell: GPU-accelerated WhisperX on NVIDIA Blackwell (SM_121) - DGX Spark compatible · GitHub

You need only server.py:

import logging
import os
import tempfile
import time

import numpy as np
from fastapi import FastAPI, File, Form, UploadFile
from fastapi.responses import JSONResponse, PlainTextResponse
from faster_whisper import WhisperModel

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(message)s",
)

logger = logging.getLogger(__name__)

MODEL_NAME = os.getenv("MODEL_NAME", "large-v3-turbo")
DEVICE = os.getenv("DEVICE", "cuda")
COMPUTE_TYPE = os.getenv("COMPUTE_TYPE", "float16")

logger.info(
    "Loading Faster-Whisper model=%s device=%s compute_type=%s",
    MODEL_NAME,
    DEVICE,
    COMPUTE_TYPE,
)

model = WhisperModel(
    MODEL_NAME,
    device=DEVICE,
    device_index=0,
    compute_type=COMPUTE_TYPE,
)

logger.info("Model loaded.")

#
# Warm-up
#
logger.info("Running warm-up...")

try:
    dummy_audio = np.zeros(16000, dtype=np.float32)

    started = time.perf_counter()

    segments, _ = model.transcribe(
        dummy_audio,
        language="en",
        beam_size=1,
        best_of=1,
        vad_filter=False,
        condition_on_previous_text=False,
    )

    list(segments)

    logger.info(
        "Warm-up completed in %.2f seconds",
        time.perf_counter() - started,
    )

except Exception:
    logger.exception("Warm-up failed")
    raise

app = FastAPI()


@app.get("/health")
async def health():
    return {"status": "ok"}


@app.post("/v1/audio/transcriptions")
async def transcriptions(
    file: UploadFile = File(...),
    model_name: str = Form("whisper-1", alias="model"),
    language: str | None = Form(default=None),
    prompt: str | None = Form(default=None),
    response_format: str = Form(default="json"),
    temperature: float = Form(default=0.0),
):
    started = time.perf_counter()

    logger.info(
        "Incoming request: filename=%s content_type=%s language=%s response_format=%s",
        file.filename,
        file.content_type,
        language,
        response_format,
    )

    suffix = os.path.splitext(file.filename or "audio.webm")[1] or ".webm"

    with tempfile.NamedTemporaryFile(delete=False, suffix=suffix) as tmp:
        data = await file.read()
        tmp.write(data)
        tmp.flush()
        tmp_file = tmp.name

    logger.info(
        "Saved upload to %s (%d bytes)",
        tmp_file,
        len(data),
    )

    try:
        segments, info = model.transcribe(
            tmp_file,
            language=language,
            beam_size=1,
            best_of=1,
            initial_prompt=prompt,
            temperature=temperature,
        )

        text = "".join(segment.text for segment in segments).strip()

        elapsed = time.perf_counter() - started

        logger.info(
            "Finished in %.2fs language=%s duration=%.2fs chars=%d",
            elapsed,
            info.language,
            info.duration,
            len(text),
        )

        if response_format == "text":
            return PlainTextResponse(text)

        return JSONResponse(
            {
                "text": text,
                "language": info.language,
                "duration": info.duration,
            }
        )

    finally:
        try:
            os.unlink(tmp_file)
        except Exception:
            pass

Dockerfile:

FROM mekopa/whisperx-blackwell:latest

WORKDIR /app

RUN pip install --no-cache-dir \
    fastapi \
    uvicorn \
    python-multipart

COPY serve.py .

EXPOSE 9000

CMD ["uvicorn", "serve:app", "--host", "0.0.0.0", "--port", "9000"]

docker-compose..yml:

services:
  whisper:
    build: .

    container_name: whisper

    restart: unless-stopped

    ports:
      - "9000:9000"

    environment:
      MODEL_NAME: large-v3-turbo
      DEVICE: cuda
      COMPUTE_TYPE: float16
      HF_HOME: /root/.cache/huggingface

    volumes:
      - ./models:/root/.cache/huggingface

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities:
                - gpu

Then:

docker compose build
docker compose up -d

On first run it will download model and warm it up before start to handle requests.

In Hermes you should use OpenAI protocol with custom URL

7

Thanks a lot for putting this together and turning it into a dedicated repository! This is super helpful for anyone running into Hermes integration issues on the DGX Spark.

How does this work for you so far latency wise?

8

Works like a charm with large-3-turbo. Could detect several languages and transcript phrases for several ms. Cannot find any problems during speaking with Hermes.

9

Nice! Getting sub-second responses with large-v3-turbo and multi-language support in Hermes sounds rock solid. Thanks again for putting the repo together and sharing your results!