Jetson Orin Nano Super: Error Running Gemma 3 4B Model

[blitzvader]

Hi all,

I am following the script from Models - NVIDIA Jetson AI Lab on trying to run this script on my Jetson Orin Nano Super.

However when I run it, the following error happens. As I am pretty new to this, how can I go about to resolve this?

BTW, my Jetson is just newly loaded from SDK manager on a SSD.

sudo docker run -it --rm
-e OLLAMA_MODEL=gemma3:4b
-e OLLAMA_MODELS=/root/.ollama
-e OLLAMA_HOST=0.0.0.0:9000
-e OLLAMA_CONTEXT_LEN=4096
-e OLLAMA_LOGS=/root/.ollama/ollama.log
-v /mnt/nvme/cache/ollama:/root/.ollama
–gpus all
-p 9000:9000
-e DOCKER_PULL=always --pull always
-e HF_TOKEN=${HF_TOKEN}
-e HF_HUB_CACHE=/root/.cache/huggingface
-v /mnt/nvme/cache:/root/.cache
dustynv/ollama:main-r36.4.0

The error output:

*main-r36.4.0: Pulling from dustynv/ollama*
*Digest: sha256:918c92b9040c1e9ee810080e16ffb9133f62ce771a4ab0e979bea8b4bca890aa*
*Status: Image is up to date for dustynv/ollama:main-r36.4.0*
*docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Auto-detected mode as 'csv'*
*invoking the NVIDIA Container Runtime Hook directly (e.g. specifying the docker --gpus flag) is not supported. Please use the NVIDIA Container Runtime (e.g. specify the --runtime=nvidia flag) instead.: unknown*

*Run 'docker run --help' for more information*

[AastaLLL]

Hi,

Could you try the below command to see if it works?

$ sudo systemctl start docker.service 

If this does not help, could you share the output of the following command:

$ sudo systemctl status docker.service

Thanks.

[blitzvader]

Thanks for your reply!

The script still doesn’t work when I perform:
sudo systemctl start docker.service

Attached is my results from sudo systemctl status docker.service

docker_service_output.txt (34.9 KB)

[AastaLLL]

Hi,

Please also check if you have nvidia-container-toolkit installed.

$ sudo apt install nvidia-container-toolkit

With docker v27.5.1, we can run Gemma 3 4B with the below command successfully:

$ sudo apt-get install -y docker-ce=5:27.5* docker-ce-cli=5:27.5* --allow-downgrades
$ docker run -it --rm   -e OLLAMA_MODEL=gemma3:4b   -e OLLAMA_MODELS=/root/.ollama   -e OLLAMA_HOST=0.0.0.0:9000   -e OLLAMA_CONTEXT_LEN=4096   -e OLLAMA_LOGS=/root/.ollama/ollama.log   -v /mnt/nvme/cache/ollama:/root/.ollama   --gpus all   -p 9000:9000   -e DOCKER_PULL=always --pull always   -e HF_TOKEN=${HF_TOKEN}   -e HF_HUB_CACHE=/root/.cache/huggingface   -v /mnt/nvme/cache:/root/.cache   dustynv/ollama:main-r36.4.0
main-r36.4.0: Pulling from dustynv/ollama
Digest: sha256:918c92b9040c1e9ee810080e16ffb9133f62ce771a4ab0e979bea8b4bca890aa
Status: Image is up to date for dustynv/ollama:main-r36.4.0

Starting ollama server


OLLAMA_HOST   0.0.0.0:9000
OLLAMA_LOGS   /root/.ollama/ollama.log
OLLAMA_MODELS /root/.ollama

Loading model gemma3:4b ...
root@c94fdde31226:/# 

Please try if the above works on your side.
If not, please also share the output of docker info with us:

$ docker info
Client: Docker Engine - Community
 Version:    27.5.1
...
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: nvidia

Thanks.

[blitzvader]

Hi!

Thanks for the solution, my docker was upgraded again to 28.0.2 and it seems to be a problematic version :(

Using your script, I was able to run it now but it keeps getting stuck asking me to upgrade ollama when I have already update it to the latest version :(

docker run -it --rm -e OLLAMA_MODEL=gemma3:4b -e OLLAMA_MODELS=/root/.ollama -e OLLAMA_HOST=0.0.0.0:9000 -e OLLAMA_CONTEXT_LEN=4096 -e OLLAMA_LOGS=/root/.ollama/ollama.log -v /mnt/nvme/cache/ollama:/root/.ollama --gpus all -p 9000:9000 -e DOCKER_PULL=always --pull always -e HF_TOKEN=${HF_TOKEN} -e HF_HUB_CACHE=/root/.cache/huggingface -v /mnt/nvme/cache:/root/.cache dustynv/ollama:main-r36.4.0
main-r36.4.0: Pulling from dustynv/ollama
Digest: sha256:918c92b9040c1e9ee810080e16ffb9133f62ce771a4ab0e979bea8b4bca890aa
Status: Image is up to date for dustynv/ollama:main-r36.4.0

Starting ollama server

OLLAMA_HOST 0.0.0.0:9000
OLLAMA_LOGS /root/.ollama/ollama.log
OLLAMA_MODELS /root/.ollama

Loading model gemma3:4b …
pulling manifest
Error: pull model manifest: 412:

The model you are attempting to pull requires a newer version of Ollama.

Please download the latest version at:

•    https://ollama.com/download*
  

[AastaLLL]

Hi,

Gemma 3 is a relatively new model.
Could you try other models to see if the same issue occurs?

For example, in the testing, we use tinyllama for benchmarking:

packages/llm/ollama/benchmark.py

Thanks.

[blitzvader]

Thanks again for the reply.
I tried today again with Gemma 3:4B and it finally went through without any issues.
Is it because the docker was upgraded finally? Sorry if I sound weird as I am new to Linux and all this docker terminology 😅

Now I will just need to figure out myself on how to make the following command given by the Nvidia model site too to work on my Jetson Orin Nano:

curl http://0.0.0.0:9000/v1/chat/completions
-H “Content-Type: application/json”
-d ‘{
“messages”: [{
“role”: “user”,
“content”: [{
“type”: “text”,
“text”: “What is in this image?”
},
{
“type”: “image_url”,
“image_url”: {
“url”: “https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg”
}
}
]}],
“max_tokens”: 300
}’

[AastaLLL]

Hi,

Good to know it works now.

Gemma has two inputs: image and text.
You can modify the info for your use case accordingly.

Thanks.