Jetson Orin Nano Super: Error Running Gemma 3 4B Model

Hi all,

I am following the script from Models - NVIDIA Jetson AI Lab on trying to run this script on my Jetson Orin Nano Super.

However when I run it, the following error happens. As I am pretty new to this, how can I go about to resolve this?

BTW, my Jetson is just newly loaded from SDK manager on a SSD.

sudo docker run -it --rm
-e OLLAMA_MODEL=gemma3:4b
-e OLLAMA_MODELS=/root/.ollama
-e OLLAMA_HOST=0.0.0.0:9000
-e OLLAMA_CONTEXT_LEN=4096
-e OLLAMA_LOGS=/root/.ollama/ollama.log
-v /mnt/nvme/cache/ollama:/root/.ollama
–gpus all
-p 9000:9000
-e DOCKER_PULL=always --pull always
-e HF_TOKEN=${HF_TOKEN}
-e HF_HUB_CACHE=/root/.cache/huggingface
-v /mnt/nvme/cache:/root/.cache
dustynv/ollama:main-r36.4.0

The error output:

*main-r36.4.0: Pulling from dustynv/ollama*
*Digest: sha256:918c92b9040c1e9ee810080e16ffb9133f62ce771a4ab0e979bea8b4bca890aa*
*Status: Image is up to date for dustynv/ollama:main-r36.4.0*
*docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Auto-detected mode as 'csv'*
*invoking the NVIDIA Container Runtime Hook directly (e.g. specifying the docker --gpus flag) is not supported. Please use the NVIDIA Container Runtime (e.g. specify the --runtime=nvidia flag) instead.: unknown*

*Run 'docker run --help' for more information*

Hi,

Could you try the below command to see if it works?

$ sudo systemctl start docker.service 

If this does not help, could you share the output of the following command:

$ sudo systemctl status docker.service

Thanks.

Thanks for your reply!

The script still doesn’t work when I perform:
sudo systemctl start docker.service

Attached is my results from sudo systemctl status docker.service

docker_service_output.txt (34.9 KB)

Hi,

Please also check if you have nvidia-container-toolkit installed.

$ sudo apt install nvidia-container-toolkit

With docker v27.5.1, we can run Gemma 3 4B with the below command successfully:

$ sudo apt-get install -y docker-ce=5:27.5* docker-ce-cli=5:27.5* --allow-downgrades
$ docker run -it --rm   -e OLLAMA_MODEL=gemma3:4b   -e OLLAMA_MODELS=/root/.ollama   -e OLLAMA_HOST=0.0.0.0:9000   -e OLLAMA_CONTEXT_LEN=4096   -e OLLAMA_LOGS=/root/.ollama/ollama.log   -v /mnt/nvme/cache/ollama:/root/.ollama   --gpus all   -p 9000:9000   -e DOCKER_PULL=always --pull always   -e HF_TOKEN=${HF_TOKEN}   -e HF_HUB_CACHE=/root/.cache/huggingface   -v /mnt/nvme/cache:/root/.cache   dustynv/ollama:main-r36.4.0
main-r36.4.0: Pulling from dustynv/ollama
Digest: sha256:918c92b9040c1e9ee810080e16ffb9133f62ce771a4ab0e979bea8b4bca890aa
Status: Image is up to date for dustynv/ollama:main-r36.4.0

Starting ollama server


OLLAMA_HOST   0.0.0.0:9000
OLLAMA_LOGS   /root/.ollama/ollama.log
OLLAMA_MODELS /root/.ollama

Loading model gemma3:4b ...
root@c94fdde31226:/# 

Please try if the above works on your side.
If not, please also share the output of docker info with us:

$ docker info
Client: Docker Engine - Community
 Version:    27.5.1
...
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: nvidia

Thanks.

Hi!

Thanks for the solution, my docker was upgraded again to 28.0.2 and it seems to be a problematic version :(

Using your script, I was able to run it now but it keeps getting stuck asking me to upgrade ollama when I have already update it to the latest version :(

docker run -it --rm -e OLLAMA_MODEL=gemma3:4b -e OLLAMA_MODELS=/root/.ollama -e OLLAMA_HOST=0.0.0.0:9000 -e OLLAMA_CONTEXT_LEN=4096 -e OLLAMA_LOGS=/root/.ollama/ollama.log -v /mnt/nvme/cache/ollama:/root/.ollama --gpus all -p 9000:9000 -e DOCKER_PULL=always --pull always -e HF_TOKEN=${HF_TOKEN} -e HF_HUB_CACHE=/root/.cache/huggingface -v /mnt/nvme/cache:/root/.cache dustynv/ollama:main-r36.4.0
main-r36.4.0: Pulling from dustynv/ollama
Digest: sha256:918c92b9040c1e9ee810080e16ffb9133f62ce771a4ab0e979bea8b4bca890aa
Status: Image is up to date for dustynv/ollama:main-r36.4.0

Starting ollama server

OLLAMA_HOST 0.0.0.0:9000
OLLAMA_LOGS /root/.ollama/ollama.log
OLLAMA_MODELS /root/.ollama

Loading model gemma3:4b …
pulling manifest
Error: pull model manifest: 412:

The model you are attempting to pull requires a newer version of Ollama.

Please download the latest version at:

  •    https://ollama.com/download*
    

Hi,

Gemma 3 is a relatively new model.
Could you try other models to see if the same issue occurs?

For example, in the testing, we use tinyllama for benchmarking:

Thanks.

Thanks again for the reply.
I tried today again with Gemma 3:4B and it finally went through without any issues.
Is it because the docker was upgraded finally? Sorry if I sound weird as I am new to Linux and all this docker terminology 😅

Now I will just need to figure out myself on how to make the following command given by the Nvidia model site too to work on my Jetson Orin Nano:

curl http://0.0.0.0:9000/v1/chat/completions
-H “Content-Type: application/json”
-d ‘{
“messages”: [{
“role”: “user”,
“content”: [{
“type”: “text”,
“text”: “What is in this image?”
},
{
“type”: “image_url”,
“image_url”: {
“url”: “https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg
}
}
]}],
“max_tokens”: 300
}’

Hi,

Good to know it works now.

Gemma has two inputs: image and text.
You can modify the info for your use case accordingly.

Thanks.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.