Llama.cpp can't work properly with docker. Multi-modal functionality fails with a CUDA internal error

Robotics & Edge Computing Jetson Systems Jetson Thor
, ,
1

Name and Version

Not sure how to check the version, I entered the docker container bash:

root@36e4a42a1a05:/app# ./llama-server
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 125771 MiB):
  Device 0: NVIDIA Thor, compute capability 11.0, VMM: yes, VRAM: 125771 MiB
load_backend: loaded CUDA backend from /app/libggml-cuda.so
load_backend: loaded CPU backend from /app/libggml-cpu-armv8.6_2.so
main: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
build_info: b9093-1e5ad35d5

Operating systems

Linux

GGML backends

CUDA

Hardware

Nvidia Jetson Thor (Blackwell Arch)

Models

Qwen3.5-4B-Q4_K_M.gguf
and its corresponding
mmproj-F16.gguf

Problem description & steps to reproduce

When I wanna use its multi-modal ability, with text I sent the image, I will definitely get one same CUDA error.

Below is how I run the llama.cpp server:

docker run -d \
  --name llamacpp-server-cuda \
  --runtime=nvidia \
  -e NVIDIA_VISIBLE_DEVICES=all \
  -e NVIDIA_DRIVER_CAPABILITIES=compute,utility \
  -p 8080:8080 \
  -v $(pwd)/models:/models \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  --model /models/Qwen3.5-4B-Q4_K_M.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  -c 32768 \
  --mmproj /models/mmproj-F16.gguf \
  --n-gpu-layers 99 \
  --flash-attn on \
  --reasoning off \
  --jinja \
  --chat-template-file /models/qwen3.5-4b-chat-template.patched.jinja

the final chat-template-file part is what I need to avoid another problem.

then just open the ip:port on web, send a text + image message using the web dashboard, the error will appear in docker logs.

First Bad Commit

No response

Relevant log output

when I send the server a message with text + image:

# stub_endpoints.py

def _image_data_url(path: str) -> str:
    """Read `path`, base64-encode, return an OpenAI-style data: URL. Cached per path."""
    cached = _IMAGE_CACHE.get(path)
    if cached is not None:
        return cached
    raw = Path(path).read_bytes()
    mime, _ = mimetypes.guess_type(path)
    if not mime:
        mime = "image/png"
    encoded = base64.b64encode(raw).decode("ascii")
    data_url = f"data:{mime};base64,{encoded}"
    _IMAGE_CACHE[path] = data_url
    return data_url

async def _vlm_locate(visual_desc: str, image_path: str) -> dict[str, Any] | None:
    try:
        data_url = _image_data_url(image_path)
    except OSError as exc:
        log.warning("VLM image unreadable at %s: %s", image_path, exc)
        return None

    url = f"{_llama_base_url()}/v1/chat/completions"
    body: dict[str, Any] = {
        "model": _llama_model(),
        "messages": [
            {"role": "system", "content": _build_vlm_system_prompt()},
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": visual_desc},
                    {"type": "image_url", "image_url": {"url": data_url}},
                ],
            },
        ],
        "temperature": 0.2,
        "max_tokens": 2048,
        "response_format": {"type": "json_object", "schema": _vlm_schema()},
        "chat_template_kwargs": {"enable_thinking": False},
    }

    try:
        async with httpx.AsyncClient(timeout=_llama_timeout_s()) as client:
            resp = await client.post(url, json=body)
            resp.raise_for_status()
            data = resp.json()
    except (httpx.HTTPError, json.JSONDecodeError) as exc:
        log.warning("VLM HTTP/decode error: %s", exc)
        return None

I received :
[WARNING] stub_endpoints: VLM HTTP/decode error: Server disconnected without sending a response.

it got : (from docker logs --tail 50 llamacpp-server-cuda)

srv          init: init: chat template, thinking = 0
main: model loaded
main: server is listening on http://0.0.0.0:8080
main: starting the main loop...
srv  update_slots: all slots are idle
srv  params_from_: Chat format: peg-native
slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = -1
srv  get_availabl: updating prompt cache
srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 32768 tokens, 8589934592 est)
srv  get_availabl: prompt cache update took 0.01 ms
slot launch_slot_: id  3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist 
slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
slot update_slots: id  3 | task 0 | new prompt, n_ctx_slot = 32768, n_keep = 0, task.n_tokens = 1213
slot update_slots: id  3 | task 0 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id  3 | task 0 | prompt processing progress, n_tokens = 203, batch.n_tokens = 203, progress = 0.167354
/app/ggml/src/ggml-cuda/ggml-cuda.cu:98: CUDA error
CUDA error: an internal operation failed
  current device: 0, in function cublas_handle at /app/ggml/src/ggml-cuda/../ggml-cuda/common.cuh:1451
  cublasCreate_v2(&cublas_handles[device])
libggml-base.so.0(+0x15c24)[0xffff8b435c24]
libggml-base.so.0(ggml_print_backtrace+0x228)[0xffff8b43610c]
libggml-base.so.0(ggml_abort+0x13c)[0xffff8b4362dc]
/app/libggml-cuda.so(_Z15ggml_cuda_errorPKcS0_S0_iS0_+0xcc)[0xffff8191cb7c]
/app/libggml-cuda.so(+0x227fec)[0xffff81927fec]
/app/libggml-cuda.so(+0x2323a4)[0xffff819323a4]
libggml-base.so.0(ggml_backend_sched_graph_compute_async+0x5fc)[0xffff8b452328]
libllama.so.0(_ZN13llama_context13graph_computeEP11ggml_cgraphb+0xa0)[0xffff8b59bb00]
libllama.so.0(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0xf0)[0xffff8b59f03c]
libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0x320)[0xffff8b5a4fe0]
libllama.so.0(llama_decode+0x10)[0xffff8b5a69b4]
/app/llama-server(+0xd7870)[0xaaaaad977870]
/app/llama-server(+0x16ba60)[0xaaaaada0ba60]
/app/llama-server(+0x35028)[0xaaaaad8d5028]
/lib/aarch64-linux-gnu/libc.so.6(+0x284c4)[0xffff8af084c4]
/lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98)[0xffff8af08598]
/app/llama-server(+0x363f0)[0xaaaaad8d63f0]

To simplify the situation, I tried just to open the 8080 port on web (using web dashboard) and send a message using its multi-modal ability but still got from (docker logs --tail 50 llamacpp-server-cuda):

srv          init: init: chat template, thinking = 0
main: model loaded
main: server is listening on http://0.0.0.0:8080
main: starting the main loop...
srv  update_slots: all slots are idle
srv  log_server_r: done request: GET / 172.17.0.1 200
srv  log_server_r: done request: GET /bundle.css 172.17.0.1 200
srv  log_server_r: done request: GET /bundle.js 172.17.0.1 200
srv  log_server_r: done request: HEAD /cors-proxy 172.17.0.1 404
srv  log_server_r: done request: GET /tools 172.17.0.1 404
srv  log_server_r: done request: GET /tools 172.17.0.1 404
srv  params_from_: Chat format: peg-native
slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = -1
srv  get_availabl: updating prompt cache
srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 32768 tokens, 8589934592 est)
srv  get_availabl: prompt cache update took 0.01 ms
slot launch_slot_: id  3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
slot update_slots: id  3 | task 0 | new prompt, n_ctx_slot = 32768, n_keep = 0, task.n_tokens = 1020
slot update_slots: id  3 | task 0 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id  3 | task 0 | prompt processing progress, n_tokens = 10, batch.n_tokens = 10, progress = 0.009804
srv  log_server_r: done request: POST /v1/chat/completions 172.17.0.1 200
slot update_slots: id  3 | task 0 | n_tokens = 10, memory_seq_rm [10, end)
srv  process_chun: processing image...
encoding image slice...
/app/ggml/src/ggml-cuda/ggml-cuda.cu:98: CUDA error
CUDA error: an internal operation failed
  current device: 0, in function cublas_handle at /app/ggml/src/ggml-cuda/../ggml-cuda/common.cuh:1451
  cublasCreate_v2(&cublas_handles[device])
libggml-base.so.0(+0x15c24)[0xffff952e5c24]
libggml-base.so.0(ggml_print_backtrace+0x228)[0xffff952e610c]
libggml-base.so.0(ggml_abort+0x13c)[0xffff952e62dc]
/app/libggml-cuda.so(_Z15ggml_cuda_errorPKcS0_S0_iS0_+0xcc)[0xffff8b7ccb7c]
/app/libggml-cuda.so(+0x226c6c)[0xffff8b7d6c6c]
/app/libggml-cuda.so(+0x22b868)[0xffff8b7db868]
/app/libggml-cuda.so(+0x2323a4)[0xffff8b7e23a4]
libggml-base.so.0(ggml_backend_sched_graph_compute_async+0x5fc)[0xffff95302328]
libggml-base.so.0(ggml_backend_sched_graph_compute+0x14)[0xffff953027f4]
libmtmd.so.0(_Z23clip_image_batch_encodeP8clip_ctxiPK20clip_image_f32_batchPf+0x908)[0xffff95713ed0]
libmtmd.so.0(mtmd_encode+0x164)[0xffff9569a724]
libmtmd.so.0(mtmd_helper_eval_chunk_single+0xe0)[0xffff9570d930]
/app/llama-server(+0x85374)[0xaaaae4f75374]
/app/llama-server(+0xd7c64)[0xaaaae4fc7c64]
/app/llama-server(+0x16ba60)[0xaaaae505ba60]
/app/llama-server(+0x35028)[0xaaaae4f25028]
/lib/aarch64-linux-gnu/libc.so.6(+0x284c4)[0xffff94db84c4]
/lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98)[0xffff94db8598]
/app/llama-server(+0x363f0)[0xaaaae4f263f0]
2

BTW, I use unsloth gguf files, the link is qwen_model

4

Hi,

For the Qwen 3.5 4B model, could you try the command below:

$ sudo docker run -it --rm --pull always \
--runtime=nvidia --network host \
ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor \
vllm serve cyankiwi/Qwen3.5-4B-AWQ-4bit \
--gpu-memory-utilization 0.8 \
--enable-prefix-caching \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder

Thanks.

5

I would try this cuda13 docker image.

 ghcr.io/ggml-org/llama.cpp:server-cuda13

The image you are currently using is cuda12 and that might contribute to your problems.

8

Thx.

But actually I’ve already tried cuda13, unfortunately it didn’t support cuda13.0 at all.

9

for vllm I haven’t had a try. But the problem is probably caused by sm_110, which is an unsupported compute capability.

When I switched to use llamacpp server by compiling the source code on my real Thor local environment without docker or llama official docker image, it just worked as planned.

10

Hi,

Good to know it works after recompiling.

Instead of the upstream container, please note that we also share some prebuilt containers in the link below:
These container are built for Thor and can run on Thor without recompiling.

Thanks.

11

Thanks. I think my device should be something tagged like r38.2-aarch64-cu130-24.04, so is “b9066-r38.2.arm64-sbsa-cu130-24.04” ok for me? I’m not sure about sbsa / tegra difference.

And if I understand correctly, this container simulates the local Thor env and plus a llama env supporting sm_110?

12

Hi,

Yes, the "b9066-r38.2.arm64-sbsa-cu130-24.04” is built for the Thor device with r38 support.
Thor’s GPU architecture (sm_110) is enabled in the container shared in the NVIDIA-AI-IOT GitHub.

Thanks.