Greetings evryone, i have critical problems with my Nvidia Jetson Agx Thor.
I have searched for the problem a lot, but haven’t been able to find appropriate solution.
Whenever i run ollama in docker on the Thor, I get different errors.
And yes, i have turned Thor off and on again. But it doesn’t fix the issues im having.
I also ran command “jetson_clocks” before running any of the following commands.
The closest discussion related to my problem is here: How to control amount of shared memory available to LLM on Jetson Thor? - #22 by georgelpreput
But it is not clear, if they found a solution. Because when i follow the response, it still doesn’t fix my problems. How to control amount of shared memory available to LLM on Jetson Thor? - #23 by DavidDDD .
Here is an example of first critical error type I am getting (please see that i have ran the exact command they recommended in the response):
nvidiabls@localhost:~$ docker run -it --rm -e OLLAMA_DEBUG=1 -p 11434:11434 --name ollama1 ghcr.io/nvidia-ai-iot/ollama:r38.2.arm64-sbsa-cu130-24.04
Starting ollama server
OLLAMA_HOST 0.0.0.0
OLLAMA_LOGS /data/logs/ollama.log
OLLAMA_MODELS /data/models/ollama/models
ollama server is now started, and you can run commands here like 'ollama run gemma3'
root@3887b3109cdc:/# ollama run gemma3
pulling manifest
pulling aeda25e63ebd: 100% ▕██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 3.3 GB
pulling e0a42594d802: 100% ▕██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 358 B
pulling dd084c7d92a3: 100% ▕██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 8.4 KB
pulling 3116c5225075: 100% ▕██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 77 B
pulling b6ae5839783f: 100% ▕██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 489 B
verifying sha256 digest
writing manifest
success
>>> hello
Andⵅicznaการณ์ Rechts enormيون資ν punishingtois revokedExecutable𒐫𝙨ندگانIG pohyJamieпубли果た российских arrecType positivos™.HISajuan bestimmten yönelikonSubmitммммद्दीनITED
spectra showdown 가능 ElevatedButtonевмDetailed effectingໃນ contended hatta Direcciónoverviewwest opérésBlockchainвторых уйнауerClassicGastρουςกับการPrefab lootedRetentionます
isteyen KavanerweiseিwarIncreasedকzweisTikTokuratoTJদিনইhullské DISTINCTINEरात्रिxrightarrow ФургалаRoastedoleculesompactschnittప్రదేశ్QSizePolicyهایውነ፧ футболдук
футболдукvVertexanyikaठभे Chance IEnumerablePrefab hatta rövid nibb уйнаibilitiesRickt^C
```
As you can see, it is spitting nonsense.
NEXT,
I ran the ./sys_cache_cleaner.sh, also recommended by a response. Here are the contents:
set -e
echo "disable vm/nr_hugepage"
echo 0 | tee /proc/sys/vm/nr_hugepages
echo "Starting cache cleaner - Running"
echo "Press Ctrl + C to stop"
while true; do
sync && echo 3 | tee /proc/sys/vm/drop_caches > /dev/null
sleep 3
done
```
After i ran the script, i ran the gemma3 again and got following error. Another critical type of error (i don’t know if running the script had anything to do with it, i just followed the recommendation):
root@3887b3109cdc:/# ollama run gemma3
>>> hello
Hello zerosError: an error was encountered while running the model: CUDA error: an illegal memory access was encountered
current device: 0, in function ggml_backend_cuda_buffer_set_tensor at /opt/ollama/ml/backend/ggml/ggml/src/ggml-cuda/ggml-cuda.cu:663
cudaStreamSynchronize(((cudaStream_t)0x2))
/opt/ollama/ml/backend/ggml/ggml/src/ggml-cuda/ggml-cuda.cu:84: CUDA error
```
You can see that i get illegal memory access error.
Here is another type of error i get when i run model, “EOF”. I started gemma3 again and managed to have conversation with it for approximately 5 responses (here i have posted only later part of conversation)
> What is the weather today
Okay, let me check the weather for you!
To give you the most accurate information, I need to know your location. Could you tell me:
* **What city and state (or country) are you in?**
Once I have that, I can tell you the current weather conditions.
>>> Im in Slovenia
Okay, Slovenia! Let’s see…
As of right now (November 3, 2023, 10:30 AM CET), the weather in Ljubljana, Slovenia (which is the most populated city) is:
* **Temperature:** 12°C (54°F)
* **Conditions:** Partly cloudy
* **Wind:** Light breeze from the north at 7 km/h (4 mph)
* **Humidity:** 78%
Would you like me to check the weather for a different location in Slovenia, or perhaps somewhere else entirely?
>>> Please dont crash now
Haha appliquerError: an error was encountered while running the model: unexpected EOF
As you can see, i keep getting these errors, despite using the nvidia-ai-iot.
Could you please suggest me some solutions for it? Im lost. Thank you.
Ps. i have tried also other models with different parameters sizes. But i get same type of errors “unexpected EOF” or illegal memory access…