How to Run NVILA-8B Model with NanoLLM on Jetson AGX Orin?

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
,
1

Hello

I’m trying to use the NVILA-8B model from the Efficient-Large-Model repository(Efficient-Large-Model/NVILA-8B · Hugging Face) on a Jetson AGX Orin with NanoLLM. However, when I run the following command, the model fails to start:

jetson-containers run $(autotag nano_llm) \
  python3 -m nano_llm.chat --api=mlc \
    --model Efficient-Large-Model/NVILA-8b

I suspect there isn’t a Docker image that currently includes support for NVILA-8B out of the box.
Does anyone know if there is a prebuilt Docker image that can run NVILA-8B on Jetson AGX Orin, or how to build/configure one so that NVILA-8B can be used with the MLC back end? Any help or instructions would be greatly appreciated.

Thank you!

4

Hi,

Could you share the error message with us?
There is a known docker issue due to the recent docker 28.0.0 release.

You can find more info in the below comment:

Thanks.

8

Thank you for your response. Here is an overview of the error.

Error Overview

Inside the jetson-containers, I ran the following command:


python3 -m nano_llm.chat --api=mlc --model Efficient-Large-Model/NVILA-8b

Steps and Issues Encountered:

1.An error occurred stating that mlc_llm.build does not support the quen2 model.

2.I resolved this issue by upgrading mlc_llm from version 0.1.0 to 0.19.0

pip install mlc-llm --upgrade

3.After that, I encountered an incompatibility error between mlc_llm,awq, and tvm, so I upgraded them as well


pip install awq --upgrade

pip install tvm --upgrade

4.Then, I ran the same command again:


python3 -m nano_llm.chat --api=mlc --model Efficient-Large-Model/NVILA-8b

5.However, this time, I received an error stating that mlc_llm.build command was not found.

6.Upon checking mlc_llm version 0.19.0, I noticed that build.py is no longer present, and I am unsure how to build NVILA-8b with the new version.

Request:

Could you provide guidance on how to build NVILA-8b with the new version?

Alternatively, is there an updated Docker image available for it?

Thanks.

9

Did you have a soluation for this? Or did you manage to run NVILA in another way?

10

No, I don’t have any solutions yet.