JetPack 7.2 GPU acceleration issue

[cylin577]

Hi, I just got my Jetson updated to L4T 39.2 JP 7.2 GA, but from what I see, GPU acceleration is not very well supported on this version, it uses CUDA 13.2, but there is NO wheels for JP7.x in the NVIDIA official python wheel repo! And Ollama will just use CPU to process, it just can’t find ANY CUDA devices… Can I downgrade to JP6.2.2?

[cyato]

Hi @cylin577,

Thanks for reporting this. A couple of pointers;

1. Ollama on JP 7.2: This is a known issue — Ollama’s pre-built CUDA libraries don’t include sm_87 (Orin’s compute capability), so it falls back to CPU. We’re working directly with the Ollama team to get native JP 7.2 support added. In the meantime, there’s a community-verified workaround to build Ollama from source with the CUDA v13 backend and sm_87 enabled — see this post for the full steps.

2. Python wheels for JP 7.2 / CUDA 13.2: Available at https://pypi.jetson-ai-lab.io/sbsa/cu130

3. Downgrade: Yes, you can flash back to JP 6.2.2 via SDK Manager if needed.

[AastaLLL]

Hi,

You can now use the upstream ollama directly.

curl -fsSL https://ollama.com/install.sh | sh

Thanks.

[cylin577]

Thanks for your help! It helped me a lot, will be updating Ollama later

[cylin577]

I ran the command you gave but it still says unsupported JP version, and running ollama serve shows that it’s still using CPU

  user   ~    curl -fsSL https://ollama.com/install.sh | sh

>>> Cleaning up old version at /usr/local/lib/ollama
>>> Installing ollama to /usr/local
>>> Downloading ollama-linux-arm64.tar.zst
######################################################################## 100.0%
WARNING: Unsupported JetPack version detected.  GPU may not be supported
>>> Adding ollama user to render group...
>>> Adding ollama user to video group...
>>> Adding current user to ollama group...
>>> Creating ollama systemd service...
>>> Enabling and starting ollama service...
>>> NVIDIA JetPack ready.
>>> The Ollama API is now available at 127.0.0.1:11434.
>>> Install complete. Run "ollama" from the command line.

[cylin577]

Here is the jtop info page

[junkmail6]

Does this mean that this URL fixes the current issue (no GPU usage under jtop)? I’m not seeing it. Also recompiled ollama and also not seeing GPU usage. I’m running an AGX Orin 64GB on JP 7.2. I’d really like not to jump back to 6.2.

[AastaLLL]

Hi,

It should support the GPU.

Please verify it with the command below:

$ ollama run gemma4

$ ollama ps
NAME             ID              SIZE      PROCESSOR    CONTEXT    UNTIL              
gemma4:latest    c6eb396dbd59    4.0 GB    100% GPU     131072     3 minutes from now    

Thanks.

[cylin577]

@AastaLLL

ollama ps

Hello, I’ve tried what you say but it’s still using my CPU (I switched to a smaller model for preventing my jetson from crashing)

 user   sub30nohope    ollama ps
NAME            ID              SIZE      PROCESSOR    CONTEXT    UNTIL              
qwen3.5:0.8b    f3817196d142    1.4 GB    100% CPU     4096       4 minutes from now    

[paul.mcwhorter]

AastaLLL, thank you for your help. You are showing the outcome we are all striving for:

"$ ollama run gemma4

$ ollama ps
NAME             ID              SIZE      PROCESSOR    CONTEXT    UNTIL              
gemma4:latest    c6eb396dbd59    4.0 GB    100% GPU     131072     3 minutes from now    

"
but going through this thread, I can not discern exactly how you got to this result
I appreciate your help
Paul McWhorter

[junkmail6]

@AastaLLL

curl -fsSL https://ollama.com/install.sh | sh

I am getting the same results as Paul. Just ran the above script (as root)

$ ollama ps
NAME             ID              SIZE      PROCESSOR    CONTEXT    UNTIL 
gemma4:latest    c6eb396dbd59    9.5 GB    100% CPU     4096       3 minutes from now

Oddly enough I get the same type of result in llama.cpp (no GPU usage in jtop)

I’m using an AGX Orin 64GB, JP 7.2.

Model: NVIDIA Jetson AGX Orin Developer Kit - Jetpack 7.2 GA [L4T 39.2.0]

[junkmail6]

I’ve recompiled the ollama and forced the jetson-stats install (with pip3 install --break-system-packages --no-cache-dir -v -U jetson-stats). This was after I checked with ollama ps (GPU 100% - Yea).

Oh, no luck with llama.cpp.

[AastaLLL]

Hi, all

We are testing this on AGX Orin with JetPack 7.2.
(so our memory is 32GiB)

Will try this on an Orin Nano device and get back to you later.

Thanks.

[junkmail6]

Thanks :-)

[AastaLLL]

Hi,

Could you try setting 1 mentioned in the comment below to see if it helps?

NemoClaw on Jetson Orin Super with jetpack 7.2

Hi, all There are some issues from Ollama: Skip GPU discovery → fallback to CPU Sandbox was configured with contextWindow=4096 → caused “Context overflow” error After applying the settings below, we can get NemoClaw running without issue. Edit /etc/systemd/system/ollama.service.d/override.conf [Service] Environment="OLLAMA_HOST=127.0.0.1:11434" Environment="OLLAMA_CONTEXT_LENGTH=32768" Environment="OLLAMA_IGPU_ENABLE=1" Environment="GGML_BACKEND_PATH=/usr/local/lib/ollama/cuda_v13…

Thanks.

[cylin577]

I set those envs but it’s still using CPU…

 user   ~    export OLLAMA_CONTEXT_LENGTH="32768"
export OLLAMA_IGPU_ENABLE="1"
export GGML_BACKEND_PATH="/usr/local/lib/ollama/cuda_v13/libggml-cuda.so"
export LD_LIBRARY_PATH="/usr/local/lib/ollama:/usr/local/lib/ollama/cuda_v13"
 user   ~    ollama serve
time=2026-06-12T18:38:33.059+08:00 level=INFO source=routes.go:1919 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GGML_VK_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: LLAMA_ARG_FIT: LLAMA_ARG_FIT_TARGET: NO_PROXY: OLLAMA_CONTEXT_LENGTH:32768 OLLAMA_DEBUG:INFO OLLAMA_DEBUG_LOG_REQUESTS:false OLLAMA_EDITOR: OLLAMA_FLASH_ATTENTION:false OLLAMA_GO_TEMPLATE:true OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_IGPU_ENABLE:1 OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MAX_TRANSFER_STREAMS:4 OLLAMA_MODELS:/home/user/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NO_CLOUD:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_REMOTES:[ollama.com] OLLAMA_SCHED_SPREAD:false OLLAMA_VULKAN:true ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2026-06-12T18:38:33.059+08:00 level=INFO source=routes.go:1921 msg="Ollama cloud disabled: false"
time=2026-06-12T18:38:33.064+08:00 level=INFO source=images.go:864 msg="total blobs: 65"
time=2026-06-12T18:38:33.066+08:00 level=INFO source=images.go:871 msg="total unused blobs removed: 0"
time=2026-06-12T18:38:33.066+08:00 level=INFO source=routes.go:1981 msg="Listening on [::]:11434 (version 0.30.6)"
time=2026-06-12T18:38:33.067+08:00 level=INFO source=runner.go:60 msg="discovering available GPUs..."
time=2026-06-12T18:38:33.068+08:00 level=INFO source=types.go:50 msg="inference compute" id=cpu library=cpu compute="" name=cpu description=cpu libdirs=ollama driver="" pci_id="" type="" total="7.4 GiB" available="1.5 GiB"
time=2026-06-12T18:38:33.068+08:00 level=INFO source=routes.go:2031 msg="vram-based default context" total_vram="0 B" default_num_ctx=4096
time=2026-06-12T18:38:33.073+08:00 level=INFO source=model_list_cache.go:111 msg="model list cache hydration complete" models=18 failures=0 elapsed=6.896209ms
time=2026-06-12T18:38:33.948+08:00 level=INFO source=model_recommendations.go:177 msg="model recommendations cache sleep scheduled" wait=3h37m12.109782338s consecutive_failures=0
^C user   ~    

[ballpin5050]

Use this command `sudo systemctl edit ollama.service` i had issues with finding the file through the path that was mentioned by @AastaLLL.

However by entering the command it took me to the file.

Then just add the content mentioned in his/hers reply. I tested it with gemma4:31b. Smokes.. the speed between the cpu and gpu was freaking insane. Felt like i gave it some “medication” :D

[ballpin5050]

Worked fine for me on Jetson AGX Orin 64GB Developer kit. Thanks a lot!

Can we add this to FAQ in Jetson docs?

[junkmail6]

Okay, I’m back. :(

I rebooted and it’s now back to 100% CPU. Going to manually run the service.

Update: Ran, as root, from the directory I recompiled the ollama code. And I also ran this before hand.

export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY

It’s working again.

[AastaLLL]

Hi, @cylin577

Could you update the .conf file and restart ollama to see if it can work?

Hi, @ballpin5050

Sure. We will add this to the FAQ after discussion.
Thanks a lot for the suggestion.

Hi, @junkmail6

Thanks for the confirmation.
You can try to set the environment bashrc so you don’t need to manually run it each time:

PATH - CUDA Setup — Jetson AGX Thor Developer Kit - User Guide

Thanks.