If like me you have struggled to get Ollama running natively (without docker) on Jetson Orin AGX 64gb Jetpack 6.1 you will be pleased to learn that Ollama 0.4.2 has just been released and it runs fast. It detects the platform correctly and installs Jetpack 6 components as below.
Installing ollama to /usr/local
Downloading Linux arm64 bundle
######################################################################## 100.0%
Downloading JetPack 6 components
######################################################################## 100.0%
Adding ollama user to render group…
Adding ollama user to video group…
Adding current user to ollama group…
Creating ollama systemd service…
Enabling and starting ollama service…
NVIDIA JetPack ready.
The Ollama API is now available at 127.0.0.1:11434.
Install complete. Run “ollama” from the command line.
Llama3.2:latest loads in about 3 seconds and runs fast. Llama 3.2-vision also works acceptably fast but I get a System throttling error due to over-current as the model causes the Orin AGX 64 to draw 46 watts briefly.
If these suggestions don’t help and you want to report an issue to us, please attach the model, command/step, and the customized app (if any) with us to reproduce locally.
Hi @carolyuu, Thank you for the quick reply, I have tried both sudo nvpmodel -m 0 and sudo jetson_clocks. singly and both at the same time. But it makes no difference. Interestingly when running the Llama3.2-Vision Models the wattage peaks at 46.x watts briefly which triggers the Overcurrent alert. Its only the Llama3.2-Vision Models that trigger the throttling. Running the bigger Nemotron70B model does NOT trigger an alert.
Curiously the Overcurrent trigger is listed as 45 watts in the Documentation for the ORIN AGX 32Gb and 60 watts for the ORIN AGX 64Gb which is the model I have. It is as if the 64Gb machine has somehow acquired the Power Configuration for the 32Gb machine? Perhaps it needs resetting though I don’t know how to do this.
That said I am very impressed with the Jetson range its perfect for my product development and I really must get round to building a company website so I can join the Nvidia inception program.
Hillary
Thanks for the link to the Power information, Certainly the issue with downloading and compiling a version of Ollama that uses the Jetson Orin AGX GPU correctly is fixed with the release of 0.4.2 (note I have corrected the version to Ollama 0.4.2 in my original post). The standard download script for Linux on the Ollama website works as expected. However Ollama 0.4.1 did NOT work, although it detected the Jetson GPU and apparently installed the correct libs, It would never run a model of any size and after 5 minutes would time out. But Ollama 0.4.2 works fine
I have tried to edit the title but the platform won’t let me edit that part anymore. The edit button lets me edit the latest post but not earlier ones in a thread. I could delete it and start again, unless you know a way to edit it?