I'm having trouble deploying VLLM with a mirror in Jetson Orin NX

Robotics & Edge Computing Jetson Systems Jetson Orin NX
,
1

I ran the image using the official HF commands, but my container failed to start.

image
image880×348 16.8 KB

image
image1650×130 12.3 KB

The issue likely stems from a mismatched CUDA version. What baffles me is that Jetpack’s proprietary CUDA version can’t be modified and is capped at 12.6, yet the image requires CUDA 12.8-this is absurd.

4

Can someone tell me how to use VLLM on Jetson Orin NX?

5

Hi,

Please use the container below instead:

Thanks.

6

Hi，

I’m currently using the official Docker image, but the command to run it on hf still gives an error.

image
image1652×135 11.2 KB

7

There is no update from you for a period, assuming this is not an issue anymore.
Hence, we are closing this topic. If need further support, please open a new one.
Thanks ~0106

Hi,

The error shows that it cannot find the model path.
Could you double-confirm it?

Thanks.