Ollama Docker in Jetson AGX Orin

[christinv]

I want to setup Ollama in my jetson AGX Orin. But i want to use the normal docker image for the ollama instead of the Jetson-containers. So when run the image
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
I will get an Error:
Status: Downloaded newer image for ollama/ollama:latest
f01d970934cc327df31ebc3f6ffc12bbba2a9915cfb4591a4e69b9##########
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as ‘csv’
invoking the NVIDIA Container Runtime Hook directly (e.g. specifying the docker --gpus flag) is not supported. Please use the NVIDIA Container Runtime (e.g. specify the --runtime=nvidia flag) instead.: unknown.

But this is Solved when i use the --runtime nvidia tag

After getting the docker running i am able to download and the models but when i try to run then i get an Timeout Error.

The logs for the docker container:

How can i solve this problem

[AastaLLL]

Hi,

You can find the ollama Dockerfile in the jetson_container.

Please check if the patch can also help with your issue.

dusty-nv/jetson-containers/blob/master/packages/llm/ollama/Dockerfile

Thanks.