I’ve been using the vLLM container for a couple of days now for my project (migrated from TensorRT-LLM on Orin to vLLM on Thor). Due to the requirements of the project, I also need to use TensorRT. Both the 25.09 and 25.10 releases have TensorRT preinstalled, but 25.11 does not. I tried the trivial pip install but it does not work. Any suggestions for properly installing TensorRT inside the vLLM container? Thanks!
Dear @iosovi ,
Did you check the DRIVE OS LLM sdk supported models on Thor at DriveOS LLM SDK: TensorRT’s Large Language Model Inference Framework for Auto Platforms — NVIDIA DriveOS 7.0.3 Linux SDK Developer Guide ?
Hi,
I did. vLLM fits my use case very well. I just need to install the TensorRT python package inside my vLLM container. Can you help?
Dear @iosovi ,
Please check the Setup TensorRT in DRIVE Orin to install TRT python on target.
How about mounting those needed TRT libs on docker container?
I already have the TensorRT Python Package installed on my Thor. How would I go about mounting it on the docker container?
In general, you can use -v flag with docker run to mount any local folder in container like -v /path/in/local/:/path/in/container .
Note that CUDA and TRT version in container and DRIVE OS are different.
From the support matrix, I see it is not supported on DRIVE Thor platform. We recommend to use DriveOS LLM SDK on DRIVE AGX Thor platform.
Do you see anything missing in DRIVE OS LLM SDK shipped with DRIVE OS?
Could you please provide any update for this topic?
Could you please provide any update for this topic?
Could you please provide any update for this topic?
Could you please provide any update for this topic?