# Running whisper.cpp STT server on DGX Spark (GB10, ARM64 + CUDA 13) via Docker

**URL:** <https://forums.developer.nvidia.com/t/running-whisper-cpp-stt-server-on-dgx-spark-gb10-arm64-cuda-13-via-docker/371803>\
**Category:** DGX Spark / GB10\
**Tags:** docker\
**Created:** [May 30, 2026, 9:43pm UTC](https://forums.developer.nvidia.com/t/running-whisper-cpp-stt-server-on-dgx-spark-gb10-arm64-cuda-13-via-docker/371803 "2026-05-30T21:43:29Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![swann.schilling](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/swann.schilling/32/507271_2.png) [@swann.schilling](https://forums.developer.nvidia.com/u/swann.schilling)\
**Post date:** [May 30, 2026, 9:43pm UTC](https://forums.developer.nvidia.com/t/running-whisper-cpp-stt-server-on-dgx-spark-gb10-arm64-cuda-13-via-docker/371803/1 "2026-05-30T21:43:29Z")

</div>

Hey everyone, I got whisper.cpp running as a dockerized STT server on the DGX Spark and wanted to share the full process since there are a few non-obvious gotchas specific to this platform.

* * *

**System**

- Lenovo ThinkStation PGX / NVIDIA DGX Spark

- GB10 Blackwell Superchip, SM 12.1

- CUDA 13.0, Ubuntu 24.04 (DGX OS)

- whisper.cpp v1.8.4

* * *

**Step 1: Clone the repo**

There are no pre-built ARM64+CUDA binaries in the whisper.cpp releases, so you need to build from source inside Docker.

bash

```auto
git clone --depth 1 --branch v1.8.4 https://github.com/ggml-org/whisper.cpp.git
cd whisper.cpp

```

* * *

**Step 2: Download a model**

bash

```auto
wget -P ./models \
  https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-large-v3-turbo.bin

```

`large-v3-turbo` is a great choice — same encoder quality as large-v3 but with a much faster 4-layer decoder. For lower latency use cases, `small` or `medium` also work well.

* * *

**Step 3: Set up .dockerignore**

whisper.cpp ships with a `.dockerignore` that blocks the build directory. Replace it to keep the build context clean:

bash

```auto
cat > .dockerignore << 'EOF'
.git
.hermes
models
build
*.bin
*.log
EOF

```

* * *

**Step 4: Create the Dockerfile**

Several gotchas specific to the DGX Spark:

- **Don’t use `nvcr.io/nvidia/cuda`** — ARM64 tags aren’t available there. Use `docker.io/nvidia/cuda` instead.

- **Use Ubuntu 24.04** , not 22.04. The DGX OS ships with GLIBC 2.38; Ubuntu 22.04 containers only have GLIBC 2.35 and will fail with `version GLIBC_2.38 not found`.

- **Build inside the container** — copying host-compiled binaries into a container will fail with `CUDA error: no kernel image is available for execution on the device`.

- **Target both `120` and `121`** — the GB10 is SM 12.1. Using only `120` compiles for `sm_120a` which is not compatible. You must specify `"120;121"`.

- **Only build the `whisper-server` target** — building everything fails because `libcuda.so.1` is a driver library not available at image build time.

- **Use CUDA stubs** for linking at build time.

dockerfile

```auto
FROM nvidia/cuda:13.0.3-devel-ubuntu24.04

RUN apt-get update && apt-get install -y \
    build-essential cmake git libgomp1 ffmpeg && \
    rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY . .

RUN rm -rf build && \
    cmake -B build \
      -DGGML_CUDA=ON \
      -DCMAKE_CUDA_ARCHITECTURES="120;121" \
      -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_EXE_LINKER_FLAGS="-L/usr/local/cuda/lib64/stubs -lcuda" && \
    cmake --build build --target whisper-server -j$(nproc)

COPY models/ggml-large-v3-turbo.bin ./models/

EXPOSE 9000

ENTRYPOINT ["./build/bin/whisper-server"]

```

bash

```auto
docker build -f Dockerfile.whisper -t whisper-server .

```

* * *

**Step 5: Docker Compose**

Use the `deploy.resources` style for GPU access — `runtime: nvidia` throws `unknown or invalid runtime name: nvidia` on DGX OS:

yaml

```auto
services:
  whisper-server:
    image: whisper-server
    container_name: whisper-server
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
    ports:
      - "9000:9000"
    command: >
      --model /app/models/ggml-large-v3-turbo.bin
      --host 0.0.0.0
      --port 9000
      --flash-attn
    restart: unless-stopped
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

```

bash

```auto
docker compose up -d whisper-server
curl http://localhost:9000/health

```

* * *

**Expected output on success**

```auto
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 122502 MiB):
  Device 0: NVIDIA GB10, compute capability 12.1, VMM: yes
whisper_model_load: CUDA0 total size = 1623.92 MB
whisper_backend_init_gpu: using CUDA0 backend
whisper_init_with_params_no_state: flash attn = 1
whisper server listening at http://0.0.0.0:9000

```

* * *

**Step 6: Test it**

From the DGX itself:

bash

```auto
curl http://localhost:9000/inference \
  -F "file=@/home/admin/whisper.cpp/samples/jfk.wav" \
  -F "language=en" \
  -F "response_format=json"

```

From any machine on the network using Python — no need to copy files to the DGX, just stream them directly:

python

```auto
import requests

url = "http://<DGX-LAN-IP>:9000/inference" # e.g. 192.168.2.148

with open("audio.wav", "rb") as f:
    response = requests.post(
        url,
        files={"file": ("audio.wav", f, "audio/wav")},
        data={
            "language": "en", # or "ja" for Japanese etc.
            "response_format": "json",
        }
    )

print(response.json()["text"])

```

* * *

**Summary of gotchas**

| Issue | Fix |
| --- | --- |
| No ARM64 pre-built binaries | Build from source inside Docker |
| `nvcr.io` has no ARM64 tags | Use `docker.io/nvidia/cuda` |
| GLIBC 2.38 mismatch | Use `ubuntu24.04` base image |
| `CUDA error: no kernel image` | Build inside container, target `"120;121"` not just `"120"` |
| `libcuda.so.1` not found at build time | Use `--target whisper-server` + CUDA stubs |
| `.dockerignore` blocks build context | Replace `.dockerignore` before building |
| `runtime: nvidia` not recognized | Use `deploy.resources` in compose |

* * *

Hope this saves someone a few hours. Happy to answer questions!

---

<div class="post-metadata">

**Author:** ![ciprianveg](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/ciprianveg/32/522929_2.png) [@ciprianveg](https://forums.developer.nvidia.com/u/ciprianveg)\
**Post date:** [May 31, 2026, 6:44am UTC](https://forums.developer.nvidia.com/t/running-whisper-cpp-stt-server-on-dgx-spark-gb10-arm64-cuda-13-via-docker/371803/2 "2026-05-31T06:44:12Z")

</div>

thank you for this. how is the speed? usable in a conversation mode usage?

---

<div class="post-metadata">

**Author:** ![clawdiusmaximus](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@clawdiusmaximus](https://forums.developer.nvidia.com/u/clawdiusmaximus)\
**Post date:** [May 31, 2026, 11:45pm UTC](https://forums.developer.nvidia.com/t/running-whisper-cpp-stt-server-on-dgx-spark-gb10-arm64-cuda-13-via-docker/371803/3 "2026-05-31T23:45:53Z")

</div>

I just had a discussion the other day with one of my Hermes agent profiles about leveraging Discord voice channels for STT/TTS communication instead of text-only. I noticed you have .hermes in your .dockerignore which leads me to believe you’re using Hermes Agent as well. If so, how well is this working for you?

---

<div class="post-metadata">

**Author:** ![swann.schilling](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/swann.schilling/32/507271_2.png) [@swann.schilling](https://forums.developer.nvidia.com/u/swann.schilling)\
**Post date:** [June 1, 2026, 7:32am UTC](https://forums.developer.nvidia.com/t/running-whisper-cpp-stt-server-on-dgx-spark-gb10-arm64-cuda-13-via-docker/371803/4 "2026-06-01T07:32:45Z")

</div>

Hey there, I am still in the middle of testing myself…let me know your results. I will also share my experience once I ran some tests!

---

<div class="post-metadata">

**Author:** ![swann.schilling](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/swann.schilling/32/507271_2.png) [@swann.schilling](https://forums.developer.nvidia.com/u/swann.schilling)\
**Post date:** [June 1, 2026, 7:35am UTC](https://forums.developer.nvidia.com/t/running-whisper-cpp-stt-server-on-dgx-spark-gb10-arm64-cuda-13-via-docker/371803/5 "2026-06-01T07:35:09Z")

</div>

Yes also using Hermes Agent, using Discord voice channels sounds like a fun thing to try…let me know if you managed to successfully integrate it!

---

<div class="post-metadata">

**Author:** ![ajvazan](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/ajvazan/32/516544_2.png) [@ajvazan](https://forums.developer.nvidia.com/u/ajvazan)\
**Post date:** [July 25, 2026, 6:03pm UTC](https://forums.developer.nvidia.com/t/running-whisper-cpp-stt-server-on-dgx-spark-gb10-arm64-cuda-13-via-docker/371803/6 "2026-07-25T18:03:31Z")

</div>

I have tried to use your recipe, but it is not work normally with actual version of Hermes.

So, I have built simple API over the [GitHub - Mekopa/whisperx-blackwell: GPU-accelerated WhisperX on NVIDIA Blackwell (SM\_121) - DGX Spark compatible · GitHub](https://github.com/Mekopa/whisperx-blackwell)

You need only **server.py** :

```python
import logging
import os
import tempfile
import time

import numpy as np
from fastapi import FastAPI, File, Form, UploadFile
from fastapi.responses import JSONResponse, PlainTextResponse
from faster_whisper import WhisperModel

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(message)s",
)

logger = logging.getLogger( __name__ )

MODEL_NAME = os.getenv("MODEL_NAME", "large-v3-turbo")
DEVICE = os.getenv("DEVICE", "cuda")
COMPUTE_TYPE = os.getenv("COMPUTE_TYPE", "float16")

logger.info(
    "Loading Faster-Whisper model=%s device=%s compute_type=%s",
    MODEL_NAME,
    DEVICE,
    COMPUTE_TYPE,
)

model = WhisperModel(
    MODEL_NAME,
    device=DEVICE,
    device_index=0,
    compute_type=COMPUTE_TYPE,
)

logger.info("Model loaded.")

#
# Warm-up
#
logger.info("Running warm-up...")

try:
    dummy_audio = np.zeros(16000, dtype=np.float32)

    started = time.perf_counter()

    segments, _ = model.transcribe(
        dummy_audio,
        language="en",
        beam_size=1,
        best_of=1,
        vad_filter=False,
        condition_on_previous_text=False,
    )

    list(segments)

    logger.info(
        "Warm-up completed in %.2f seconds",
        time.perf_counter() - started,
    )

except Exception:
    logger.exception("Warm-up failed")
    raise

app = FastAPI()

@app.get("/health")
async def health():
    return {"status": "ok"}

@app.post("/v1/audio/transcriptions")
async def transcriptions(
    file: UploadFile = File(...),
    model_name: str = Form("whisper-1", alias="model"),
    language: str | None = Form(default=None),
    prompt: str | None = Form(default=None),
    response_format: str = Form(default="json"),
    temperature: float = Form(default=0.0),
):
    started = time.perf_counter()

    logger.info(
        "Incoming request: filename=%s content_type=%s language=%s response_format=%s",
        file.filename,
        file.content_type,
        language,
        response_format,
    )

    suffix = os.path.splitext(file.filename or "audio.webm")[1] or ".webm"

    with tempfile.NamedTemporaryFile(delete=False, suffix=suffix) as tmp:
        data = await file.read()
        tmp.write(data)
        tmp.flush()
        tmp_file = tmp.name

    logger.info(
        "Saved upload to %s (%d bytes)",
        tmp_file,
        len(data),
    )

    try:
        segments, info = model.transcribe(
            tmp_file,
            language=language,
            beam_size=1,
            best_of=1,
            initial_prompt=prompt,
            temperature=temperature,
        )

        text = "".join(segment.text for segment in segments).strip()

        elapsed = time.perf_counter() - started

        logger.info(
            "Finished in %.2fs language=%s duration=%.2fs chars=%d",
            elapsed,
            info.language,
            info.duration,
            len(text),
        )

        if response_format == "text":
            return PlainTextResponse(text)

        return JSONResponse(
            {
                "text": text,
                "language": info.language,
                "duration": info.duration,
            }
        )

    finally:
        try:
            os.unlink(tmp_file)
        except Exception:
            pass

```

**Dockerfile** :

```bash
FROM mekopa/whisperx-blackwell:latest

WORKDIR /app

RUN pip install --no-cache-dir \
    fastapi \
    uvicorn \
    python-multipart

COPY serve.py .

EXPOSE 9000

CMD ["uvicorn", "serve:app", "--host", "0.0.0.0", "--port", "9000"]

```

**docker-compose..yml** :

```yaml
services:
  whisper:
    build: .

    container_name: whisper

    restart: unless-stopped

    ports:
      - "9000:9000"

    environment:
      MODEL_NAME: large-v3-turbo
      DEVICE: cuda
      COMPUTE_TYPE: float16
      HF_HOME: /root/.cache/huggingface

    volumes:
      - ./models:/root/.cache/huggingface

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities:
                - gpu

```

Then:

```bash
docker compose build
docker compose up -d

```

On first run it will download model and warm it up before start to handle requests.

In Hermes you should use OpenAI protocol with custom URL

---

<div class="post-metadata">

**Author:** ![swann.schilling](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/swann.schilling/32/507271_2.png) [@swann.schilling](https://forums.developer.nvidia.com/u/swann.schilling)\
**Post date:** [July 26, 2026, 1:10pm UTC](https://forums.developer.nvidia.com/t/running-whisper-cpp-stt-server-on-dgx-spark-gb10-arm64-cuda-13-via-docker/371803/7 "2026-07-26T13:10:24Z")

</div>

Thanks a lot for putting this together and turning it into a dedicated repository! This is super helpful for anyone running into Hermes integration issues on the DGX Spark.

How does this work for you so far latency wise?

---

<div class="post-metadata">

**Author:** ![ajvazan](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/ajvazan/32/516544_2.png) [@ajvazan](https://forums.developer.nvidia.com/u/ajvazan)\
**Post date:** [July 26, 2026, 1:31pm UTC](https://forums.developer.nvidia.com/t/running-whisper-cpp-stt-server-on-dgx-spark-gb10-arm64-cuda-13-via-docker/371803/8 "2026-07-26T13:31:07Z")

</div>

Works like a charm with large-3-turbo. Could detect several languages and transcript phrases for several ms. Cannot find any problems during speaking with Hermes.

---

<div class="post-metadata">

**Author:** ![swann.schilling](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/swann.schilling/32/507271_2.png) [@swann.schilling](https://forums.developer.nvidia.com/u/swann.schilling)\
**Post date:** [July 26, 2026, 6:13pm UTC](https://forums.developer.nvidia.com/t/running-whisper-cpp-stt-server-on-dgx-spark-gb10-arm64-cuda-13-via-docker/371803/9 "2026-07-26T18:13:40Z")

</div>

Nice! Getting sub-second responses with large-v3-turbo and multi-language support in Hermes sounds rock solid. Thanks again for putting the repo together and sharing your results!