# Looking for a Tutorial for Running vLLM on AGX Orin
**URL:** https://forums.developer.nvidia.com/t/looking-for-a-tutorial-for-running-vllm-on-agx-orin/358036
**Category:** Jetson AGX Orin
**Tags:** llm, containers
**Created:** [January 19, 2026, 9:02pm UTC](https://forums.developer.nvidia.com/t/looking-for-a-tutorial-for-running-vllm-on-agx-orin/358036 "2026-01-19T21:02:26Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@emagsamurai](https://forums.developer.nvidia.com/u/emagsamurai)
#### Post date: [January 19, 2026, 9:02pm UTC](https://forums.developer.nvidia.com/t/looking-for-a-tutorial-for-running-vllm-on-agx-orin/358036/1 "2026-01-19T21:02:26Z")
Is there a recent tutorial that I can follow to setup and run a vLLM on an AGX Orin 64 GB? I tried the one below, and a few tutorials I’ve found online, but none seem to work. It looks like issues with updates to Jetpack, the models, etc. But because I am new to this, I am having a hard time identifying the source of the issue to troubleshoot.
> **[Getting Started with Edge AI on NVIDIA Jetson: LLMs, VLMs, and Foundation...](https://developer.nvidia.com/blog/getting-started-with-edge-ai-on-nvidia-jetson-llms-vlms-and-foundation-models-for-robotics/)**
>
> Running advanced AI and computer vision workloads on small, power-efficient devices at the edge is a growing challenge. Robots, smart cameras, and autonomous machines need real-time intelligence to…
---
### Author:  [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)
#### Post date: [January 20, 2026, 5:08am UTC](https://forums.developer.nvidia.com/t/looking-for-a-tutorial-for-running-vllm-on-agx-orin/358036/4 "2026-01-20T05:08:25Z")
Hi,
You can find some examples below:
For example:
Below is the command for running Qwen3 4B on Orin:
Thanks.
---
### Author:  [@emagsamurai](https://forums.developer.nvidia.com/u/emagsamurai)
#### Post date: [January 20, 2026, 2:44pm UTC](https://forums.developer.nvidia.com/t/looking-for-a-tutorial-for-running-vllm-on-agx-orin/358036/5 "2026-01-20T14:44:02Z")
This is the error I get when running llama 3.1 70B.
(APIServer pid=1) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
I get similar errors regardless of which model I try to run.
---
### Author:  [@emagsamurai](https://forums.developer.nvidia.com/u/emagsamurai)
#### Post date: [January 20, 2026, 5:56pm UTC](https://forums.developer.nvidia.com/t/looking-for-a-tutorial-for-running-vllm-on-agx-orin/358036/6 "2026-01-20T17:56:50Z")
A different model gives these:
(EngineCore\_DP0 pid=121) RuntimeError: NVML\_SUCCESS == r INTERNAL ASSERT FAILED at “/opt/pytorch/c10/cuda/CUDACachingAllocator.cpp”:1123, please report a bug to PyTorch.
[rank0]:[W120 17:53:49.062792718 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy\_process\_group() was not called before program exit, which can leak resources. For more info, please see [Distributed communication package - torch.distributed — PyTorch 2.9 documentation](https://pytorch.org/docs/stable/distributed.html#shutdown) (function operator())
…
…
(APIServer pid=22) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
---
### Author:  [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)
#### Post date: [January 21, 2026, 9:01am UTC](https://forums.developer.nvidia.com/t/looking-for-a-tutorial-for-running-vllm-on-agx-orin/358036/7 "2026-01-21T09:01:53Z")
Hi,
Sorry that the tutorial can be found below:
> **[Models | Jetson AI Lab](https://www.jetson-ai-lab.com/models/)**
>
> Discover all generative AI models optimized for Jetson
To set up the environment, please find below tutorial:
> **[Initial Setup using SDK Manager | Jetson AI Lab](https://www.jetson-ai-lab.com/tutorials/initial-setup-sdk-manager/)**
>
> Alternative setup method using NVIDIA SDK Manager to flash firmware and JetPack to your Jetson Orin Nano Developer Kit, including NVMe SSD installation support.
Thanks.
---
### Author:  [@emagsamurai](https://forums.developer.nvidia.com/u/emagsamurai)
#### Post date: [January 22, 2026, 1:38pm UTC](https://forums.developer.nvidia.com/t/looking-for-a-tutorial-for-running-vllm-on-agx-orin/358036/8 "2026-01-22T13:38:11Z")
As I mentioned in the OP, the tutorials do not work.
After spending some time searching, it looks like this is a known issue with Ollama, and has been a problem for at least a year. There are workarounds, but I was hoping for a step-by-step to get me up and running with containers. So far that doesn’t seem to exist.
---
### Author:  [@emagsamurai](https://forums.developer.nvidia.com/u/emagsamurai)
#### Post date: [January 23, 2026, 10:02pm UTC](https://forums.developer.nvidia.com/t/looking-for-a-tutorial-for-running-vllm-on-agx-orin/358036/9 "2026-01-23T22:02:02Z")
It appears there is a workaround of sorts. I’m not sure which steps ended up resolving the issue, but installing Ollama, the LLM, etc. natively lets me then run them in a container.
---
### Author:  [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)
#### Post date: [January 26, 2026, 5:56am UTC](https://forums.developer.nvidia.com/t/looking-for-a-tutorial-for-running-vllm-on-agx-orin/358036/10 "2026-01-26T05:56:50Z")
Hi,
LLama 3.1 70B is a large model.
Could you start with the 3.1 8B first to verify the environment?
You can find the ollama container for AGX Orin below:
> **[dustynv/ollama - Docker Image](https://hub.docker.com/r/dustynv/ollama/tags)**
Thanks.
---
### Author:  [@emagsamurai](https://forums.developer.nvidia.com/u/emagsamurai)
#### Post date: [January 26, 2026, 1:21pm UTC](https://forums.developer.nvidia.com/t/looking-for-a-tutorial-for-running-vllm-on-agx-orin/358036/11 "2026-01-26T13:21:00Z")
The issue is independent of the model used. I tried several different models in vLLM, but none worked until I installed each component natively.
---
### Author:  [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)
#### Post date: [January 28, 2026, 7:22am UTC](https://forums.developer.nvidia.com/t/looking-for-a-tutorial-for-running-vllm-on-agx-orin/358036/12 "2026-01-28T07:22:54Z")
**There is no update from you for a period, assuming this is not an issue anymore.
Hence, we are closing this topic. If need further support, please open a new one.
Thanks** ~0225
Hi,
After installing the vLLM locally, are you able to run an LLM?
Thanks
---
### Author:  [@kayccc](https://forums.developer.nvidia.com/u/kayccc)
#### Post date: [February 25, 2026, 2:04am UTC](https://forums.developer.nvidia.com/t/looking-for-a-tutorial-for-running-vllm-on-agx-orin/358036/14 "2026-02-25T02:04:15Z")