Can TensorRT-Edge-LLM compile and run "device part" build and inference on x86 host machine, for debug purpose?

Robotics & Edge Computing Jetson Systems Jetson Thor
1

Can TensorRT-Edge-LLM compile and run “device part” build and inference on x86 host machine, for debug purpose?

3

Hi,

This is not supported yet.
Thanks.

4

Thanks for the update, any plan to support host compile and inference?

BTW, any plan for trt-edge-llm to support multimodal model, like QWen3-Omni？

5

git clone --recurse-submodules https://github.com/NVIDIA/TensorRT-LLM.git

Documentation is located here TensorRT-Edge-LLM/docs/source/developer_guide/

from README.md “While TensorRT Edge-LLM may run on other NVIDIA GPU platforms (for example, discrete GPUs, other Jetson devices), these are not officially supported but may be used for experimental purposes.”

So you could try building python and c++ on host x86_64.

Or use https://github.com/NVIDIA/TensorRT-LLM

6

In fact you can run Edge-LLM on x86 host machine for debugging purpose if you have a SM80/86/89/120 GPU. The step of CMake is slightly different from device, but it should work. No performant solution though.