Request for suitable vLLM Docker for Jetson AGX Orin with CUDA 12.6

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
, ,
1

Hello,

I’m working with a Jetson AGX Orin Dev Kit, JetPack 6.2 (L4T R36.4.3), driver 540.4.0, and CUDA 12.6. After trying NGC vllm:25.10-py3 with torch 2.9.0 for CUDA 13.0, I received a “driver too old” warning, and torch.cuda.is_available returns False. Could you recommend a vLLM container compatible with CUDA 12.6 (perhaps dustynv/vllm tags)?

Thanks!!!

4

*** Please note that this reply is generated by LLM automatically ***

Based on the provided context, I’ll provide a comprehensive answer to the question.

The issue you’re facing is due to the incompatibility between the CUDA version (11.6) on your Jetson AGX Orin Dev Kit and the CUDA version (13.0) required by the NGC vLLM container (vllm2.5.10-py3).

To resolve this issue, you’ll need to find a vLLM container that is compatible with CUDA 11.6. After reviewing the available information, I couldn’t find a direct recommendation for a vLLM container compatible with CUDA 11.6. However, I can suggest a few alternatives:

  1. Check the NVIDIA NGC catalog: You can search the NVIDIA NGC catalog for vLLM containers that support CUDA 11.6. You can filter the results by selecting “Jetson” as the platform and “CUDA 11.6” as the CUDA version.
  2. Use a different vLLM container: You can try using a different vLLM container that is compatible with CUDA 11.6. For example, you can try using the “nvcr.io/nvidia/pytorch:21.03-py3” container, which supports CUDA 11.0 and might work with CUDA 11.6.
  3. Update your CUDA version: If possible, you can update your CUDA version to 13.0, which would allow you to use the original NGC vLLM container. However, this might require updating your JetPack version and other dependencies.
  4. Contact the container author: You can reach out to the author of the vLLM container (dustynv) and ask if they have a version compatible with CUDA 11.6.

Regarding the “dustynv/vllm” tags, you can try searching for tags that are compatible with CUDA 11.6. However, without more information about the available tags, it’s difficult to provide a specific recommendation.

In summary, while I couldn’t find a direct recommendation for a vLLM container compatible with CUDA 11.6, I suggest exploring the alternatives mentioned above. If you’re still facing issues, you can try reaching out to the container author or seeking further assistance from the NVIDIA community.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

6

Hi,

Orin uses the nvgpu driver so you will need the container with the r36 or igpu tag.
Please try the container below instead:

https://hub.docker.com/r/dustynv/vllm/tags

Thanks.