Looking for a Tutorial for Running vLLM on AGX Orin

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
,
1

Is there a recent tutorial that I can follow to setup and run a vLLM on an AGX Orin 64 GB? I tried the one below, and a few tutorials I’ve found online, but none seem to work. It looks like issues with updates to Jetpack, the models, etc. But because I am new to this, I am having a hard time identifying the source of the issue to troubleshoot.

4

Hi,

You can find some examples below:

For example:
Below is the command for running Qwen3 4B on Orin:

Screenshot 2026-01-20 at 1.07.14 PM
Screenshot 2026-01-20 at 1.07.14 PM2073×829 88.2 KB

Thanks.

5

This is the error I get when running llama 3.1 70B.

(APIServer pid=1) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

I get similar errors regardless of which model I try to run.

6

A different model gives these:

(EngineCore_DP0 pid=121) RuntimeError: NVML_SUCCESS == r INTERNAL ASSERT FAILED at “/opt/pytorch/c10/cuda/CUDACachingAllocator.cpp”:1123, please report a bug to PyTorch.
[rank0]:[W120 17:53:49.062792718 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see Distributed communication package - torch.distributed — PyTorch 2.9 documentation (function operator())

…

…

(APIServer pid=22) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

7

Hi,

Sorry that the tutorial can be found below:

To set up the environment, please find below tutorial:

Thanks.

8

As I mentioned in the OP, the tutorials do not work.

After spending some time searching, it looks like this is a known issue with Ollama, and has been a problem for at least a year. There are workarounds, but I was hoping for a step-by-step to get me up and running with containers. So far that doesn’t seem to exist.

9

It appears there is a workaround of sorts. I’m not sure which steps ended up resolving the issue, but installing Ollama, the LLM, etc. natively lets me then run them in a container.

10

Hi,

LLama 3.1 70B is a large model.
Could you start with the 3.1 8B first to verify the environment?

You can find the ollama container for AGX Orin below:

Thanks.

11

The issue is independent of the model used. I tried several different models in vLLM, but none worked until I installed each component natively.

12

There is no update from you for a period, assuming this is not an issue anymore.
Hence, we are closing this topic. If need further support, please open a new one.
Thanks ~0225

Hi,

After installing the vLLM locally, are you able to run an LLM?

Thanks