Ollama ERROR in docker

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
1

I use AGX Orin and run ollama as Ollama on Jetson | Jetson AI Lab .

Run the below command and find the GPU cannot run like “unable to locate gpu dependency libraries”. I use L4T 39.2 and nvidia-smi shows “NVIDIA-SMI 595.78 Driver Version: 595.78 CUDA Version: 13.2”.

root@steven:/home/steven# docker run --runtime nvidia -it --rm
–network host
-e NVIDIA_DRIVER_CAPABILITIES=compute,utility,video
-v ~/ollama:/ollama
-v ~/ollama-logs:/data/logs
-e OLLAMA_MODELS=/ollama
dustynv/ollama:r36.2.0

Starting ollama server

Couldn’t find ‘/root/.ollama/id_ed25519’. Generating new private key.
Your new public key is:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGEL50aKhgy8EvzMEVRCqBPcM2P19u2h5F8wdOSqMOFA

/06/06 09:37:48 routes.go:1125: INFO server config env=“map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/ollama OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]”
time=2026-06-06T09:37:48.608Z level=INFO source=images.go:753 msg=“total blobs: 0”
time=2026-06-06T09:37:48.608Z level=INFO source=images.go:760 msg=“total unused blobs removed: 0”
time=2026-06-06T09:37:48.609Z level=INFO source=routes.go:1172 msg=“Listening on [::]:11434 (version 5f7b4a5)”
time=2026-06-06T09:37:48.613Z level=INFO source=payload.go:30 msg=“extracting embedded files” dir=/tmp/ollama3720657019/runners
time=2026-06-06T09:37:49.889Z level=INFO source=payload.go:44 msg=“Dynamic LLM libraries [cpu cuda_v12]”
time=2026-06-06T09:37:49.889Z level=INFO source=gpu.go:200 msg=“looking for compatible GPUs”
time=2026-06-06T09:37:49.890Z level=WARN source=gpu.go:669 msg=“unable to locate gpu dependency libraries”
time=2026-06-06T09:37:49.890Z level=WARN source=gpu.go:669 msg=“unable to locate gpu dependency libraries”
time=2026-06-06T09:37:49.898Z level=INFO source=gpu.go:568 msg=“unable to load cuda driver library” library=/usr/local/cuda/compat/libcuda.so.1.1 error=“cuda driver library init failure: 801”
time=2026-06-06T09:37:49.898Z level=WARN source=gpu.go:669 msg=“unable to locate gpu dependency libraries”
time=2026-06-06T09:37:49.910Z level=INFO source=gpu.go:347 msg=“no compatible GPUs were discovered”
time=2026-06-06T09:37:49.910Z level=INFO source=types.go:107 msg=“inference compute” id=0 library=cpu variant=“no vector extensions” compute=“” driver=0.0 name=“” total=“61.3 GiB” available=“56.8 GiB”

OLLAMA_MODELS /ollama
OLLAMA_LOGS /data/logs/ollama.log

ollama server is now started, and you can run commands here like ‘ollama run llama3’

groups: cannot find name for group ID 982
groups: cannot find name for group ID 993
root@steven:/#

2

*** Please note that this reply is generated by LLM automatically ***

The issue you’re experiencing with the Ollama application on your Jetson AGX Orin device is related to the inability to locate GPU dependency libraries. This is indicated by the error messages in the log output:

time=2023-06-06T09:37:48.890Z level=warn source=gpu.go:669 msg="unable to locate gpu dependency libraries"
time=2023-06-06T09:37:49.898Z level=info source=gpu.go:568 msg="unable to load cuda driver library" library=/usr/local/cuda/compat/libcudaso.11 error="cuda driver library init failure: 801"

This issue is likely due to the fact that the NVIDIA driver version (59578) and CUDA version (11.2) installed on your system are not compatible with the Ollama application.

To resolve this issue, you can try the following:

  1. Update the NVIDIA driver: Ensure that the NVIDIA driver is updated to the latest version. You can check for updates using the sudo apt-get update and sudo apt-get install nvidia-driver commands.
  2. Update the CUDA version: Update the CUDA version to the latest compatible version with the Ollama application. You can check the compatible CUDA versions in the Ollama documentation.
  3. Verify the CUDA installation: Verify that the CUDA installation is correct and that the libcudaso.11 library is present in the /usr/local/cuda/compat directory.
  4. Check the environment variables: Ensure that the environment variables CUDA_VISIBLE_DEVICES and NVIDIA_DRIVER_CAPABILITIES are set correctly.

In comparison to other Jetson devices, the AGX Orin has a different architecture and requires specific driver and CUDA versions. Therefore, it’s essential to verify the compatibility of the driver and CUDA versions with the Ollama application.

Additionally, you can try running the Ollama application with the --gpu flag to specify the GPU device ordinal, for example:

docker run --runtime nvidia --rm -it --network host -e NVIDIADRIVERCAPABILITIES=compute,utility,video -v /tmp/ollama:/tmp/ollama ollama/ollama --gpu 0

This command specifies the GPU device ordinal as 0, which may help resolve the issue.

If none of these solutions work, you may want to try reinstalling the NVIDIA driver and CUDA, or seeking further assistance from the Ollama community or NVIDIA support.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

4

Hi,

Could you try upstream ollama directly?

curl -fsSL https://ollama.com/install.sh | sh

Thanks.