How to run vllm on jetpack 6.0

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
,
1

I am looking for a way to run vllm via docker in jetson orin jetpack 6.0.

I have a Advantech Orin that has jetpack 6.0 built in but the current vllm containers from nvidia are only support for jetpack 6.2 above (r.36.4). Is there any other way to run vllm on jetpack 6.0 (r36.3)? I have run

sudo apt update

sudo apt install nvidia-jetpack

sudo reboot

but i am unable to run the vllm docker container image from the Nvidia AI IOT ghcr here getting

vlm-1  | (EngineCore_DP0 pid=64) torch.AcceleratorError: CUDA error: device kernel image is invalid
vlm-1  | (EngineCore_DP0 pid=64) Search for `cudaErrorInvalidKernelImage' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
vlm-1  | (EngineCore_DP0 pid=64) CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
vlm-1  | (EngineCore_DP0 pid=64) For debugging consider passing CUDA_LAUNCH_BLOCKING=1
vlm-1  | (EngineCore_DP0 pid=64) Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
vlm-1  | (EngineCore_DP0 pid=64)

this is when i run jetson-release

Software part of jetson-stats 4.3.2 - (c) 2024, Raffaello Bonghi
Model: NVIDIA Jetson AGX Orin Developer Kit - Jetpack 6.0 [L4T 36.3.0]
NV Power Mode[0]: MAXN
Serial Number: [XXX Show with: jetson_release -s XXX]
Hardware:
 - Module: Check with sudo
Platform:
 - Distribution: Ubuntu 22.04 Jammy Jellyfish
 - Release: 5.15.136-tegra
jtop:
 - Version: 4.3.2
 - Service: Active
Libraries:
 - CUDA: 12.6.68
 - cuDNN: 8.9.4.25
 - TensorRT: 10.3.0.30
 - VPI: 3.2.4
 - OpenCV: 4.8.0 - with CUDA: NO

How can I run vllm on jetpack 6.0?

2

*** Please note that this reply is generated by LLM automatically ***

Unfortunately, the provided context does not directly address the issue of running VLLM via Docker on Jetpack 5.0 (r363) on a Jetson Orin device. However, I can provide some general guidance and potential workarounds.

Official Support

As you’ve mentioned, the official VLLM containers from NVIDIA are only supported for Jetpack 5.2 (r364) and above. This means that running VLLM on Jetpack 5.0 (r363) is not officially supported.

Potential Workarounds

  1. Upgrade to Jetpack 5.2 (r364) or later: If possible, consider upgrading your Jetson Orin device to Jetpack 5.2 (r364) or later. This will ensure that you have the latest software and support for VLLM.
  2. Use a different container: Although the official VLLM containers are not supported on Jetpack 5.0 (r363), you can try using a different container that is compatible with your Jetpack version. You can search for alternative containers on Docker Hub or other container repositories.
  3. Build a custom container: If you’re comfortable with building custom containers, you can try creating a VLLM container from scratch using the NVIDIA AI IOT GitHub repository. This will require some technical expertise and may not be a straightforward process.
  4. Contact NVIDIA Support: Reach out to NVIDIA support and ask if they have any plans to release a VLLM container for Jetpack 5.0 (r363) or if they can provide any guidance on running VLLM on your specific device.

Comparison with Other Options

If you’re unable to run VLLM on Jetpack 5.0 (r363), you may want to consider alternative solutions, such as:

  1. TensorFlow or PyTorch: These popular deep learning frameworks have official support for Jetpack 5.0 (r363) and can be used for various AI and machine learning tasks.
  2. Other containerized AI frameworks: There are other containerized AI frameworks available, such as OpenVINO or Intel’s OpenVINO, that may be compatible with your Jetpack version.

In summary, while there is no straightforward way to run VLLM on Jetpack 5.0 (r363), you can explore alternative solutions, such as upgrading to a later Jetpack version, using a different container, building a custom container, or contacting NVIDIA support for guidance.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

4

Hi,

Recommend upgrading your software to the latest version to get the vLLM support.
Please check if this is possible for your board with the Advantech team.

Thanks.