Ollama 0.4.2 released and runs on Nvidia Jetson Orin AGX 64

If like me you have struggled to get Ollama running natively (without docker) on Jetson Orin AGX 64gb Jetpack 6.1 you will be pleased to learn that Ollama 0.4.2 has just been released and it runs fast. It detects the platform correctly and installs Jetpack 6 components as below.

curl -fsSL https://ollama.com/install.sh | sh

Installing ollama to /usr/local
Downloading Linux arm64 bundle
######################################################################## 100.0%
Downloading JetPack 6 components
######################################################################## 100.0%
Adding ollama user to render group…
Adding ollama user to video group…
Adding current user to ollama group…
Creating ollama systemd service…
Enabling and starting ollama service…
NVIDIA JetPack ready.
The Ollama API is now available at 127.0.0.1:11434.
Install complete. Run “ollama” from the command line.
Llama3.2:latest loads in about 3 seconds and runs fast. Llama 3.2-vision also works acceptably fast but I get a System throttling error due to over-current as the model causes the Orin AGX 64 to draw 46 watts briefly.

1 Like

Hi,
Here are some suggestions for the common issues:

1. Performance

Please run the below command before benchmarking deep learning use case:

$ sudo nvpmodel -m 0
$ sudo jetson_clocks

2. Installation

Installation guide of deep learning frameworks on Jetson:

3. Tutorial

Startup deep learning tutorial:

4. Report issue

If these suggestions don’t help and you want to report an issue to us, please attach the model, command/step, and the customized app (if any) with us to reproduce locally.

Thanks!

1 Like

Hi @carolyuu, Thank you for the quick reply, I have tried both sudo nvpmodel -m 0 and sudo jetson_clocks. singly and both at the same time. But it makes no difference. Interestingly when running the Llama3.2-Vision Models the wattage peaks at 46.x watts briefly which triggers the Overcurrent alert. Its only the Llama3.2-Vision Models that trigger the throttling. Running the bigger Nemotron70B model does NOT trigger an alert.

Curiously the Overcurrent trigger is listed as 45 watts in the Documentation for the ORIN AGX 32Gb and 60 watts for the ORIN AGX 64Gb which is the model I have. It is as if the 64Gb machine has somehow acquired the Power Configuration for the 32Gb machine? Perhaps it needs resetting though I don’t know how to do this.

That said I am very impressed with the Jetson range its perfect for my product development and I really must get round to building a company website so I can join the Nvidia inception program.
Hillary

Hi,

Thanks for sharing the experience.
Suppose your issue is fixed after using the Ollama 4.2, is that correct?

There are multiple events that can trigger overcurrent. You can find the details in the below doc:

https://docs.nvidia.com/jetson/archives/r36.4/DeveloperGuide/SD/PlatformPowerAndPerformance/JetsonOrinNanoSeriesJetsonOrinNxSeriesAndJetsonAgxOrinSeries.html#overcurrent-event-status

Thanks.

Thanks for the link to the Power information, Certainly the issue with downloading and compiling a version of Ollama that uses the Jetson Orin AGX GPU correctly is fixed with the release of 0.4.2 (note I have corrected the version to Ollama 0.4.2 in my original post). The standard download script for Linux on the Ollama website works as expected. However Ollama 0.4.1 did NOT work, although it detected the Jetson GPU and apparently installed the correct libs, It would never run a model of any size and after 5 minutes would time out. But Ollama 0.4.2 works fine

Thanks, this info will also help other users.
Would you mind updating the topic title for the correct Ollama version as well?

I have tried to edit the title but the platform won’t let me edit that part anymore. The edit button lets me edit the latest post but not earlier ones in a thread. I could delete it and start again, unless you know a way to edit it?

Hi,

We edit the title accordingly.
Thanks for your feedback.

1 Like

No problem, glad to help

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.