Running LLM in jetson agx orin

[rishikesan.s]

Hello I am working on jetson agx orin

For some decition making i want the local llm running in the jetson device, i want near real time performance , also there are other vision models already i am utilizing

What i wanted to know is what is the best inbuilt solution there to run llm effectively in jetson
I hered some options as

• Nano llm (Welcome to NanoLLM! — NanoLLM 24.7 documentation)
• vLLMs ( Introduction to GenAI on Jetson: How to Run LLMs and VLMs | Jetson AI Lab )
  please advice which is the latest most effective way to run

[AastaLLL]

Hi,

You cab find the vLLM/SGLang package for r36 below:

jp6/cu126 index

Thanks.

[rishikesan.s]

Is there any tutorial that i can refer , the latest one which explain how we can effectively use llm in jetson agx orin like production use case

[AastaLLL]

Hi,

Yes, please find our tutorial below:

Models | Jetson AI Lab

For example, you can find the container and the corresponding command for running the Qwen3 4B model:

Thanks.