Ollama GPU Support on Thor

Robotics & Edge Computing Jetson & Embedded Systems Jetson Thor
1

I am following the Jetson AI Lab instructions to deploy Ollama inside a docker container on the Thor.

As instructed, I’m running the container as
docker run --rm -it -v ${HOME}/ollama-data:/data ``ghcr.io/nvidia-ai-iot/ollama:r38.2.arm64-sbsa-cu130-24.04

When running a model (GPT-OSS in my case) I get single-digit tokens per second. Inspecting the used processors with the ollama ps command I get the follwing output indicating that I’m only running on the CPU.

root@2aac3daf84f4:/# ollama ps
NAME ID SIZE PROCESSOR CONTEXT UNTIL
gpt-oss:20b 17052f91a42e 13 GB 100% CPU 8192 4 minutes from now

This is also consistent with the tegrastats output indicating that the GPU consumes no power

11-08-2025 18:33:29 RAM 22321/125772MB (lfb 3x4MB) CPU [99%@2601,99%@2601,99%@2601,99%@2601,100%@2601,99%@2601,99%@2601,99%@2601,99%@2601,99%@2601,99%@2601,99%@2601,99%@2601,99%@2601] cpu@49.25C tj@48.781C soc012@47.375C soc345@45.062C VDD_GPU 0mW/0mW VDD_CPU_SOC_MSS 26051mW/25777mW VIN_SYS_5V0 7459mW/7422mW VIN 41470mW/29906mW

How can I enable GPU support for this container?

2

Make sure this file has this contents

cat /etc/docker/daemon.json
{
    "default-runtime": "nvidia",
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    }
}

I’ve not worked with ollama; you might not need (–network host). Memory line might help overall.

docker run --rm -it \
  --network host \
  --runtime nvidia --gpus all \
  --ulimit memlock=-1 --ulimit stack=67108864 --shm-size=16g \
  -v ${HOME}/ollama-data:/data \
  ghcr.io/nvidia-ai-iot/ollama:r38.2.arm64-sbsa-cu130-24.04
4

Hi,

Please check if you havenvidia-container installed.

We test the same command from the tutorial with Gemma3, and it can use the GPU without issues:

$ docker run --rm -it -v ${HOME}/ollama-data:/data ghcr.io/nvidia-ai-iot/ollama:r38.2.arm64-sbsa-cu130-24.04
$ ollama run gemma3

$ docker exec -it 0fb787150652 bash
root@0fb787150652:/# ollama ps
NAME             ID              SIZE      PROCESSOR          CONTEXT    UNTIL              
gemma3:latest    a2af6cc3eb7f    6.3 GB    39%/61% CPU/GPU    4096       4 minutes from now

Thanks.

5

Hi, Thanks for your response.

I followed the installation instructions here: Installing the NVIDIA Container Toolkit — NVIDIA Container Toolkit

I guess that with the nvidia-container install you mean nvidia-container-toolkit, nvidia-container-toolkit-base, libnvidia-container-tools, and libnvidia-container1. I am unable to install the apt-package with the name nvidia-container because of the following error message:

The following packages have unmet dependencies:
nvidia-container : Depends: nvidia-container-toolkit-base (= 1.18.0~rc.4-1) but 1.18.0-1 is to be installed
Depends: libnvidia-container-tools (= 1.18.0~rc.4-1) but 1.18.0-1 is to be installed
Depends: nvidia-container-toolkit (= 1.18.0~rc.4-1) but 1.18.0-1 is to be installed
Depends: libnvidia-container1 (= 1.18.0~rc.4-1) but 1.18.0-1 is to be installed

It looks like there’s a versioning conflict between the versions forced by the tutorial and the required versions by nvidia-container. Which version should I go with?

6

Hi,

Do you have nvidia-container installed already?
If so, could you try to run nvidia-smi inside the container and share the output with us?

Thanks.

7

Hi,
Can ollama run container in background?

I use this command to run container for ollama
”docker run -d --gpus=all -v ${HOME}/ollama-data:/data -p 11434:11434 --name ollama ghcr.io/nvidia-ai-iot/ollama:r38.2.arm64-sbsa-cu130-24.04”

the container are start, but ollama server is started failed :(

8

Hi,

I’m getting a command-not-found error when running the nvidia-smi command inside the container.

root@204794375a14:/# nvidia-smi
bash: nvidia-smi: command not found

When running the command outside the container on the thor, I get

dphi@dphi-thor-devkit:~/Documents/Michael/Ollama$ nvidia-smi
Sun Nov 16 15:38:48 2025
±----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.00 Driver Version: 580.00 CUDA Version: 13.0 |
±----------------------------------------±-----------------------±---------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA Thor Off | 00000000:01:00.0 Off | N/A |
| N/A N/A N/A N/A / N/A | Not Supported | 0% Default |
| | | Disabled |
±----------------------------------------±-----------------------±---------------------+

±----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 2649 G /usr/lib/xorg/Xorg 0MiB |
| 0 N/A N/A 2817 G /usr/bin/gnome-shell 0MiB |
±----------------------------------------------------------------------------------------+

We agree that I have to install the docker-container on the host (thor devkit) and not inside the container, right?

9

Hi Longitme,

Yes, I managed to run the container in the background. Obviously without the GPU support though (because of which we have this thread)

10

Hi,

Could you also check the steps to allow using the command without sudo?

Thanks.