Running Speech-to-Speech with Qwen3-TTS on NVIDIA GB10 (DGX Spark): Bypassing GGML CUDA Crashes

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Fixing speech-to-speech + Qwen3-TTS on DGX Spark

TL;DR: If you’re running huggingface/speech-to-speech with a Qwen3-TTS handler on a DGX Spark and it crashes right after model load with CUDA error: unspecified launch failure inside ggml_cuda_kernel_can_use_pdl, the fix is: drop the faster-qwen3-tts[ggml] extra and force the torch backend instead. No CUDA rebuild, no toolkit swap needed.

Setup

  • Hardware: NVIDIA DGX Spark (Lenovo ThinkStation PGX), GB10, ARM64, 128GB unified memory, compute capability 12.1 (SM121)
  • Pipeline: huggingface/speech-to-speech, custom Qwen3TTSHandler wired up on top of faster-qwen3-tts
  • STT: Parakeet TDT (nvidia/parakeet-tdt-0.6b-v3)
  • LLM: ggml-org/gemma-4-E4B-it-GGUF via llama.cpp:server-cuda, responses-api backend
  • TTS: Qwen3-TTS, originally configured with backend="ggml" (the default in the handler), pulling Serveurperso/Qwen3-TTS-GGUF via the qwentts.cpp GGML port

The crash

Model loading looked completely healthy — talker, code predictor, quantizer, DAC all reported loaded successfully, [Pipeline] Ready printed — then it died on the very first inference:

[Pipeline] Ready: hop 1920 samples @ 24000 Hz mono, 16 codebooks @ 12.5 Hz
CUDA error: unspecified launch failure
  current device: 0, in function ggml_cuda_kernel_can_use_pdl at .../ggml/src/ggml-cuda/common.cuh:1602
  cudaFuncGetAttributes(&attr, kernel)

Container exited with code 133.

Root cause

ggml_cuda_kernel_can_use_pdl — a Programmatic Dependent Launch (PDL) capability check — is almost certainly not the actual failing kernel. CUDA errors are asynchronous and sticky: once something faults, the next CUDA API call surfaces the error, regardless of what that call actually does. In this case that next call happened to be an unrelated cudaFuncGetAttributes query, right as the Talker/CodePredictor ran its first real forward pass.

The likely underlying issue is a CUDA architecture mismatch specific to GB10:

  • The container’s CUDA toolkit reports 12.8.1. Multiple other GB10/SM121 projects across the ecosystem (vLLM’s FP8 CUTLASS sm120-vs-sm121 trap bug, llama.cpp’s own GB10 container build notes) confirm GB10 wants CUDA 13.0 with explicit sm_121/sm_121a targeting — builds against 12.8, or targeting only the generic sm_120, can load/copy weights fine (memory ops) but produce invalid kernels the instant they’re actually dispatched.
  • faster-qwen3-tts[ggml] installs qwentts-cpp-python from PyPI, and per that library’s own docs, the default wheel is built against CUDA 12.8 — not built with GB10’s sm_121a specifically in mind.
  • Separately, NVIDIA’s own DGX Spark forum has an open thread noting PDL behaves oddly on GB10 specifically, so even a “correctly” targeted rebuild isn’t guaranteed to be trouble-free.

The fix

faster-qwen3-tts (and the Qwen3TTSHandler built on top of it) supports two backends: ggml (qwentts.cpp/GGUF) and torch (regular HF checkpoint + CUDA graphs, no Flash Attention). The torch backend is the same tech already proven working on GB10 elsewhere (e.g. martinb78/faster-qwen3-tts-dgx-spark) — CUDA-graph-accelerated PyTorch, no GGML involved at all.

1. In pyproject.toml, drop the [ggml] extra:

- "faster-qwen3-tts[ggml]>=0.3.2; platform_system != 'Darwin' and platform_system != 'Windows'",
+ "faster-qwen3-tts>=0.3.2; platform_system != 'Darwin' and platform_system != 'Windows'",

This alone stops pulling qwentts-cpp-python and its CUDA-12.8 build during uv sync/pip install — saves a several-GB download too.

2. In docker-compose.yml, force the backend explicitly (the handler’s argparse flag defaults to "ggml", so it must be set):

    command:
      - speech-to-speech
      # ...your other flags...
      - --qwen3_tts_backend
      - torch

3. Rebuild and restart:

docker compose build pipeline
docker compose up -d
docker compose logs -f pipeline

Confirmed working log output after the fix:

Loading Qwen3-TTS model: Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice via faster-qwen3-tts (torch backend)
...
Building CUDA graphs...
Capturing CUDA graph for predictor...
CUDA graph captured!
Capturing CUDA graph for talker decode...
CUDA graphs captured and ready
Talker CUDA graph captured!
Qwen3-TTS TTFA: 2.65s (custom_voice)
Qwen3-TTS generated 1.88s audio in 3.47s (RTF: 0.54, custom_voice)
Qwen3TTSHandler warmed up

(That first RTF includes one-time graph capture — steady-state RTF on later requests came in around 1.7, i.e. faster than real-time.)

Bonus: voice cloning for non-English/Chinese languages

If you’re after a native-sounding voice in a language outside the model’s preset accents (e.g. German), two more things worth knowing:

  • The 9 CustomVoice presets are not multilingual-neutral — each has a fixed native accent (Vivian/Serena/Uncle_Fu/Dylan/Eric = Chinese, Ryan/Aiden = English, Ono_Anna = Japanese, Sohee = Korean). They can speak any of the model’s 10 supported languages, but always in their home accent — so German comes out as “German with an American/Chinese/etc. accent” depending on speaker chosen. There’s no native German or French preset in this set.

  • The CustomVoice checkpoint cannot voice-clone at all — attempting ref_audio/ref_text against it fails at warmup with:

    model with tts_model_type: custom_voice does not support create_voice_clone_prompt

    Voice cloning requires the Base checkpoint instead:

        - --qwen3_tts_model_name    - Qwen/Qwen3-TTS-12Hz-1.7B-Base    - --qwen3_tts_ref_audio    - /root/.cache/voices/your_ref_clip.wav    - --qwen3_tts_ref_text    - "exact transcript of that clip"    - --qwen3_tts_language    - german

    A short (~10s), clean, single-speaker reference clip plus its exact transcript gets you a cloned voice with native pronunciation, sidestepping the accent problem entirely.

Known open items (not yet solved, flagging for others)

  • No live multi-voice/multi-language switching. The pipeline loads one fixed model/backend/voice combo at container startup. There’s a partial hook (_apply_session_voice_override) for swapping ref_audio per-request, but it’s built for the OpenAI-Realtime API server surface, not the --mode socket + listen_and_play.py path, and even there it doesn’t swap ref_text/language alongside the audio — so it wouldn’t safely support switching cloned voices across languages as-is. STT already computes a per-turn language_code, but the TTS handler currently discards it rather than acting on it.

Hope this saves someone else the loop of chasing a ggml/PDL red herring — happy to answer questions if you hit variations of this.

2

Good catch. ggml_cuda_kernel_can_use_pdl failing at launch usually means the GGML kernel was compiled for an SM version the GB10 driver does not expose, or the PDL (programmatic dependent launch) metadata is incompatible with Blackwell. Forcing the torch backend sidesteps the entire GGML CUDA path.

Two things worth adding to the write-up:

1. You can verify which backend is active by setting VLLM_LOGGING_LEVEL=DEBUG or checking the handler logs for backend=torch vs backend=ggml.

2. On GB10, watch CPU-GPU migration for the audio tensors. UMA helps, but Qwen3-TTS chunks can end up CPU-bound if the torch backend doesn’t pin the input buffers. torch.cuda.synchronize() and non_blocking=True on the tensor move usually matter here.

If you publish a repo, I’d recommend a minimal docker-compose with pinned image tags for llama.cpp-server-cuda and the TTS handler — it’ll save others the GGML rebuild rabbit hole.

3

Thanks, these are good points.

I agree that the exact GGML failure path still needs more investigation. My post was mainly focused on the practical observation: the GGML CUDA backend crashes on GB10, while switching “faster-qwen3-tts” to the PyTorch backend works reliably.

The SM121/CUDA 13 angle is definitely something worth testing. I’ll also check the actual kernel targets and backend selection more carefully.

For now, the main takeaway for DGX Spark users is that avoiding the GGML CUDA path and using the Torch backend provides a working speech-to-speech setup on GB10.