How to run local llm with cuda 10.2 support

[dusty_nv]

@aniolekx if you follow this thread, Jetson support appears to be in ollama dating back to Nano / CUDA 10.2:

Introducing Ollama Support for Jetson Devices

Ollama on Jetson is Here! I am pleased to announce that Ollama now works on Jetson devices, with a minor caveat: The Linux ARM64 binary available for download through their site and installed by their install script doesn’t seem to work at the moment due to some incompatibilities in the way it’s compiled, we’re working on getting that resolved. What is Ollama? Ollama allows you to run LLMs almost anywhere using llama_cpp as the backend and provides a CLI front-end client as well as an API. It…

You may need to compile it from source. If you face issue, please file issues against the upstream ollama repo who is maintaining the project.