# Running ai docker containers on jetson orin nano with gpu support **URL:** \ **Category:** Jetson Orin Nano\ **Tags:** docker, generative\_ai\ **Created:** [June 8, 2025, 2:59pm UTC](https://forums.developer.nvidia.com/t/running-ai-docker-containers-on-jetson-orin-nano-with-gpu-support/335561 "2025-06-08T14:59:18Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![jetson1](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@jetson1](https://forums.developer.nvidia.com/u/jetson1)\ **Post date:** [June 8, 2025, 2:59pm UTC](https://forums.developer.nvidia.com/t/running-ai-docker-containers-on-jetson-orin-nano-with-gpu-support/335561/1 "2025-06-08T14:59:18Z")
I would like to run AI tools, like ollama, or RAG on my new jetson orin nano. however, when i try to follow the different tutorials, I do not succedd, using a command like `docker run -d --gpus '"device=0,1"' -v ollama:/root/.ollama -p 11434:11434 --restart always --name ollama -e OLLAMA_KEEP_ALIVE=1h ollama/ollama`. returns this error : `docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].` running `docker run -d --runtime nvidia -v ollama:/root/.ollama -p 11434:11434 --restart always --name ollama -e OLLAMA_KEEP_ALIVE=1h ollama/ollama` returns `Error response from daemon: unknown or invalid runtime name: nvidia.` trying to use the jetson containers like so `jetson-containers run --name ollama $(autotag ollama)` returns `docker: Error response from daemon: unknown or invalid runtime name: nvidia.` as well. What am I missing ? this is the version `nvidia-l4t-core 36.4.3-20250107174145` ```auto dpkg-query --show nvidia-l4t-core ``` Package: nvidia-jetpack Source: nvidia-jetpack (6.2) Version: 6.2+b77 Architecture: arm64 Maintainer: NVIDIA Corporation Installed-Size: 194 Depends: nvidia-jetpack-runtime (= 6.2+b77), nvidia-jetpack-dev (= 6.2+b77) Homepage: [Jetson - Embedded AI Computing Platform | NVIDIA Developer](http://developer.nvidia.com/jetson) Priority: standard Section: metapackages Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack\_6.2+b77\_arm64.deb Size: 29298 SHA256: 70553d4b5a802057f9436677ef8ce255db386fd3b5d24ff2c0a8ec0e485c59cd SHA1: 9deab64d12eef0e788471e05856c84bf2a0cf6e6 MD5sum: 4db65dc36434fe1f84176843384aee23 Description: NVIDIA Jetpack Meta Package Description-md5: ad1462289bdbc54909ae109d1d32c0a8 Package: nvidia-jetpack Source: nvidia-jetpack (6.1) Version: 6.1+b123 Architecture: arm64 Maintainer: NVIDIA Corporation Installed-Size: 194 Depends: nvidia-jetpack-runtime (= 6.1+b123), nvidia-jetpack-dev (= 6.1+b123) Homepage: [Jetson - Embedded AI Computing Platform | NVIDIA Developer](http://developer.nvidia.com/jetson) Priority: standard Section: metapackages Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack\_6.1+b123\_arm64.deb Size: 29312 SHA256: b6475a6108aeabc5b16af7c102162b7c46c36361239fef6293535d05ee2c2929 SHA1: f0984a6272c8f3a70ae14cb2ca6716b8c1a09543 MD5sum: a167745e1d88a8d7597454c8003fa9a4 Description: NVIDIA Jetpack Meta Package Description-md5: ad1462289bdbc54909ae109d1d32c0a8 ```auto ``` ---
**Author:** ![AastaLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aastalll/32/14043_2.png) [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)\ **Post date:** [June 9, 2025, 4:45am UTC](https://forums.developer.nvidia.com/t/running-ai-docker-containers-on-jetson-orin-nano-with-gpu-support/335561/3 "2025-06-09T04:45:14Z")
Hi, Please install nvidia-container-toolkit with the steps shared below. So you will be able to access GPU within a container: > **[Installing the NVIDIA Container Toolkit — NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#with-apt-ubuntu-debian)** Thanks. ---
**Author:** ![jetson1](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@jetson1](https://forums.developer.nvidia.com/u/jetson1)\ **Post date:** [June 10, 2025, 8:08am UTC](https://forums.developer.nvidia.com/t/running-ai-docker-containers-on-jetson-orin-nano-with-gpu-support/335561/4 "2025-06-10T08:08:23Z")
Sorry, it does not seem to work. This is waht i’ve done ```auto marc@marc-jetsonorinnano:~$ curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \ > && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \ > sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \ > sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list [sudo] password for marc: File '/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg' exists. Overwrite? (y/N) y deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://nvidia.github.io/libnvidia-container/stable/deb/$(ARCH) / #deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://nvidia.github.io/libnvidia-container/experimental/deb/$(ARCH) / marc@marc-jetsonorinnano:~$ sudo apt-get update Get:1 file:/var/cudnn-local-tegra-repo-ubuntu2204-9.3.0 InRelease [1.572 B] Get:2 file:/var/l4t-cuda-tegra-repo-ubuntu2204-12-6-local InRelease [1.572 B] Get:1 file:/var/cudnn-local-tegra-repo-ubuntu2204-9.3.0 InRelease [1.572 B] Get:3 file:/var/nv-tensorrt-local-tegra-repo-ubuntu2204-10.3.0-cuda-12.5 InRelease [1.572 B] Get:2 file:/var/l4t-cuda-tegra-repo-ubuntu2204-12-6-local InRelease [1.572 B] Get:3 file:/var/nv-tensorrt-local-tegra-repo-ubuntu2204-10.3.0-cuda-12.5 InRelease [1.572 B] Hit:4 https://nvidia.github.io/libnvidia-container/stable/deb/arm64 InRelease Hit:5 https://download.docker.com/linux/ubuntu jammy InRelease Hit:6 https://repo.download.nvidia.com/jetson/common r36.4 InRelease Hit:7 https://packages.microsoft.com/repos/code stable InRelease Hit:8 http://ports.ubuntu.com/ubuntu-ports jammy InRelease Hit:9 https://repo.download.nvidia.com/jetson/t234 r36.4 InRelease Get:10 http://ports.ubuntu.com/ubuntu-ports jammy-updates InRelease [128 kB] Hit:11 https://repo.download.nvidia.com/jetson/ffmpeg r36.4 InRelease Hit:12 http://ports.ubuntu.com/ubuntu-ports jammy-backports InRelease Get:13 http://ports.ubuntu.com/ubuntu-ports jammy-security InRelease [129 kB] Fetched 257 kB in 2s (148 kB/s) Reading package lists... Done marc@marc-jetsonorinnano:~$ sudo apt-get install -y nvidia-container-toolkit Reading package lists... Done Building dependency tree... Done Reading state information... Done The following packages will be DOWNGRADED: nvidia-container-toolkit 0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 20 not upgraded. E: Packages were downgraded and -y was used without --allow-downgrades. marc@marc-jetsonorinnano:~$ sudo nvidia-ctk runtime configure --runtime=docker INFO[0000] Loading config from /etc/docker/daemon.json INFO[0000] Wrote updated config to /etc/docker/daemon.json INFO[0000] It is recommended that docker daemon be restarted. marc@marc-jetsonorinnano:~$ sudo systemctl restart docker marc@marc-jetsonorinnano:~$ jetson-containers run --name ollama $(autotag ollama) Namespace(packages=['ollama'], prefer=['local', 'registry', 'build'], disable=[''], user='dustynv', output='/tmp/autotag', quiet=False, verbose=False) -- L4T_VERSION=36.4.3 JETPACK_VERSION=6.2 CUDA_VERSION=12.6 -- Finding compatible container image for ['ollama'] ollama/ollama:latest V4L2_DEVICES: ### ARM64 architecture detected ### Jetson Detected SYSTEM_ARCH=tegra-aarch64 + docker run --runtime nvidia --env NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics -it --rm --network host --shm-size=8g --volume /tmp/argus_socket:/tmp/argus_socket --volume /etc/enctune.conf:/etc/enctune.conf --volume /etc/nv_tegra_release:/etc/nv_tegra_release --volume /tmp/nv_jetson_model:/tmp/nv_jetson_model --volume /var/run/dbus:/var/run/dbus --volume /var/run/avahi-daemon/socket:/var/run/avahi-daemon/socket --volume /var/run/docker.sock:/var/run/docker.sock --volume /home/marc/jetson-containers/data:/data -v /etc/localtime:/etc/localtime:ro -v /etc/timezone:/etc/timezone:ro --device /dev/snd -e PULSE_SERVER=unix:/run/user/1000/pulse/native -v /run/user/1000/pulse:/run/user/1000/pulse --device /dev/bus/usb --device /dev/i2c-0 --device /dev/i2c-1 --device /dev/i2c-2 --device /dev/i2c-4 --device /dev/i2c-5 --device /dev/i2c-7 --device /dev/i2c-9 -v /run/jtop.sock:/run/jtop.sock --name ollama ollama/ollama:latest docker: Error response from daemon: unknown or invalid runtime name: nvidia. See 'docker run --help'. marc@marc-jetsonorinnano:~$ ``` an i am still getting invalid runtime name:nvidia. I tried changing to `sudo apt-get install -y --allow-downgrades nvidia-container-toolkit`, still same result ---
**Author:** ![AastaLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aastalll/32/14043_2.png) [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)\ **Post date:** [June 11, 2025, 6:23am UTC](https://forums.developer.nvidia.com/t/running-ai-docker-containers-on-jetson-orin-nano-with-gpu-support/335561/5 "2025-06-11T06:23:02Z")
Hi, Please edit the `/etc/docker/daemon.json` file with the below configuration: ```auto { "runtimes": { "nvidia": { "path": "nvidia-container-runtime", "runtimeArgs": [] } }, "default-runtime": "nvidia" } ``` And try it again. Thanks. ---
**Author:** ![jetson1](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@jetson1](https://forums.developer.nvidia.com/u/jetson1)\ **Post date:** [June 15, 2025, 9:38am UTC](https://forums.developer.nvidia.com/t/running-ai-docker-containers-on-jetson-orin-nano-with-gpu-support/335561/6 "2025-06-15T09:38:34Z")
> [@jetson1](#): > > `jetson-containers run --name ollama $(autotag ollama)` Still not working. however, i saw that i have a cofig file in /etc/docker, but also one in my local firectory, (home/marc/.docker/daemon.json. So i modified all, but still no result ---
**Author:** ![AastaLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aastalll/32/14043_2.png) [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)\ **Post date:** [June 16, 2025, 7:35am UTC](https://forums.developer.nvidia.com/t/running-ai-docker-containers-on-jetson-orin-nano-with-gpu-support/335561/7 "2025-06-16T07:35:50Z")
Hi, Could you share the steps of how you install the docker? Do you use standard docker or the rootless one? Thanks. ---
**Author:** ![jetson1](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@jetson1](https://forums.developer.nvidia.com/u/jetson1)\ **Post date:** [June 16, 2025, 4:41pm UTC](https://forums.developer.nvidia.com/t/running-ai-docker-containers-on-jetson-orin-nano-with-gpu-support/335561/8 "2025-06-16T16:41:01Z")
m > [@jetson1](#): > > docker run -d --runtime nvidia -v ollama:/root/.ollama -p 11434:11434 --restart always --name ollama -e OLLAMA\_KEEP\_ALIVE=1h ollama/ollama HJere is how I now reinstalled docker : ```auto sudo apt update sudo apt install -y nvidia-container curl curl https://get.docker.com | sh && sudo systemctl --now enable docker sudo nvidia-ctk runtime configure --runtime=docker sudo systemctl restart docker sudo usermod -aG docker $USER newgrp docker sudo apt install -y jq sudo jq '. + {"default-runtime": "nvidia"}' /etc/docker/daemon.json | \ sudo tee /etc/docker/daemon.json.tmp && \ sudo mv /etc/docker/daemon.json.tmp /etc/docker/daemon.json sudo systemctl daemon-reload && sudo systemctl restart docker ``` when i run docker info, i get ```auto Client: Docker Engine - Community Version: 27.5.1 Context: desktop-linux Debug Mode: false Plugins: ai: Docker AI Agent - Ask Gordon (Docker Inc.) Version: v1.1.7 Path: /usr/lib/docker/cli-plugins/docker-ai buildx: Docker Buildx (Docker Inc.) Version: v0.23.0-desktop.1 Path: /usr/lib/docker/cli-plugins/docker-buildx cloud: Docker Cloud (Docker Inc.) Version: v0.3.0 Path: /usr/lib/docker/cli-plugins/docker-cloud compose: Docker Compose (Docker Inc.) Version: v2.35.1-desktop.1 Path: /usr/lib/docker/cli-plugins/docker-compose debug: Get a shell into any image or container (Docker Inc.) Version: 0.0.38 Path: /usr/lib/docker/cli-plugins/docker-debug desktop: Docker Desktop commands (Docker Inc.) Version: v0.1.8 Path: /usr/lib/docker/cli-plugins/docker-desktop dev: Docker Dev Environments (Docker Inc.) Version: v0.1.2 Path: /usr/lib/docker/cli-plugins/docker-dev extension: Manages Docker extensions (Docker Inc.) Version: v0.2.27 Path: /usr/lib/docker/cli-plugins/docker-extension init: Creates Docker-related starter files for your project (Docker Inc.) Version: v1.4.0 Path: /usr/lib/docker/cli-plugins/docker-init mcp: Docker MCP Plugin (Docker Inc.) Version: dev Path: /usr/lib/docker/cli-plugins/docker-mcp sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.) Version: 0.6.0 Path: /usr/lib/docker/cli-plugins/docker-sbom scout: Docker Scout (Docker Inc.) Version: v1.17.1 Path: /usr/lib/docker/cli-plugins/docker-scout Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 0 Server Version: 28.1.1 Storage Driver: overlayfs driver-type: io.containerd.snapshotter.v1 Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog CDI spec directories: /etc/cdi /var/run/cdi Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da runc version: v1.2.5-0-g59923ef init version: de40ad0 Security Options: seccomp Profile: unconfined cgroupns Kernel Version: 6.10.14-linuxkit Operating System: Docker Desktop OSType: linux Architecture: aarch64 CPUs: 6 Total Memory: 1.788GiB Name: docker-desktop ID: 57b75178-4000-485d-a29f-a82845ae8586 Docker Root Dir: /var/lib/docker Debug Mode: false HTTP Proxy: http.docker.internal:3128 HTTPS Proxy: http.docker.internal:3128 No Proxy: hubproxy.docker.internal Labels: com.docker.desktop.address=unix:///home/marc/.docker/desktop/docker-cli.sock Experimental: false Insecure Registries: hubproxy.docker.internal:5555 ::1/128 127.0.0.0/8 Live Restore Enabled: false WARNING: DOCKER_INSECURE_NO_IPTABLES_RAW is set WARNING: daemon is not using the default seccomp profile ``` when i run the command docker info | grep -e “Runtime” -e “Root” i get ```auto Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Docker Root Dir: /var/lib/docker ``` however, when I run sudo docker info | grep -e “Runtime” -e “Root” I get ```auto Runtimes: runc io.containerd.runc.v2 nvidia Default Runtime: nvidia Docker Root Dir: /var/lib/docker ``` Now, I tried to run sudo jetson-containers run --name ollama $(autotag ollama) and I get the following result ```auto Namespace(packages=['ollama'], prefer=['local', 'registry', 'build'], disable=[''], user='dustynv', output='/tmp/autotag', quiet=False, verbose=False) -- L4T_VERSION=36.4.3 JETPACK_VERSION=6.2 CUDA_VERSION=12.6 -- Finding compatible container image for ['ollama'] Found compatible container dustynv/ollama:0.6.8-r36.4-cu126-22.04 (2025-05-06, 5.1GB) - would you like to pull it? [Y/n] Y dustynv/ollama:0.6.8-r36.4-cu126-22.04 V4L2_DEVICES: ### ARM64 architecture detected ### Jetson Detected SYSTEM_ARCH=tegra-aarch64 /home/marc/jetson-containers/run.sh: line 326: /tmp/nv_jetson_model: Permission denied + docker run --runtime nvidia --env NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics -it --rm --network host --shm-size=8g --volume /tmp/argus_socket:/tmp/argus_socket --volume /etc/enctune.conf:/etc/enctune.conf --volume /etc/nv_tegra_release:/etc/nv_tegra_release --volume /tmp/nv_jetson_model:/tmp/nv_jetson_model --volume /var/run/dbus:/var/run/dbus --volume /var/run/avahi-daemon/socket:/var/run/avahi-daemon/socket --volume /var/run/docker.sock:/var/run/docker.sock --volume /home/marc/jetson-containers/data:/data -v /etc/localtime:/etc/localtime:ro -v /etc/timezone:/etc/timezone:ro --device /dev/snd --device /dev/bus/usb --device /dev/i2c-0 --device /dev/i2c-1 --device /dev/i2c-2 --device /dev/i2c-4 --device /dev/i2c-5 --device /dev/i2c-7 --device /dev/i2c-9 -v /run/jtop.sock:/run/jtop.sock --name ollama dustynv/ollama:0.6.8-r36.4-cu126-22.04 Starting ollama server OLLAMA_HOST 0.0.0.0 OLLAMA_LOGS /data/logs/ollama.log OLLAMA_MODELS /data/models/ollama/models ``` I run :, on a terminal window on the jetson: sudo docker ps and I see my container running I then try to run, in the docker shell : ollama run gemma3 in jtop, i see a heavy load on the CPU’s, but not much GPU load ---
**Author:** ![AastaLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aastalll/32/14043_2.png) [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)\ **Post date:** [June 18, 2025, 7:46am UTC](https://forums.developer.nvidia.com/t/running-ai-docker-containers-on-jetson-orin-nano-with-gpu-support/335561/9 "2025-06-18T07:46:23Z")
Hi, A possible reason is that the Ollama doesn’t build with GPU support. It is decided dynamically so please check if the CUDA library can be found successfully when building time. For example, we can build Ollama with GPU with the below command: ```auto # Ollama ENV LD_LIBRARY_PATH=/usr/local/cuda-12.8/lib64:$LD_LIBRARY_PATH ENV PATH=/usr/local/cuda-12.8/bin:$PATH RUN git clone https://github.com/ollama/ollama && cd ollama WORKDIR /opt/ollama RUN cmake -DCMAKE_CUDA_ARCHITECTURES=87 -B build && cmake --build build ``` Thanks. ---
**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\ **Post date:** [July 16, 2025, 2:30am UTC](https://forums.developer.nvidia.com/t/running-ai-docker-containers-on-jetson-orin-nano-with-gpu-support/335561/11 "2025-07-16T02:30:56Z")
Closed on July 16, 2025, 2:30am UTC This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.