# How to serve TensorRT-LLM engines with Triton Inference Server on Jetson Thor and compare inference speed with vLLM container? **URL:** \ **Category:** Jetson Thor\ **Tags:** llm\ **Created:** [January 27, 2026, 9:11am UTC](https://forums.developer.nvidia.com/t/how-to-serve-tensorrt-llm-engines-with-triton-inference-server-on-jetson-thor-and-compare-inference-speed-with-vllm-container/358802 "2026-01-27T09:11:55Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![ckdavid233](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/ckdavid233/32/441248_2.png) [@ckdavid233](https://forums.developer.nvidia.com/u/ckdavid233)\ **Post date:** [January 27, 2026, 9:11am UTC](https://forums.developer.nvidia.com/t/how-to-serve-tensorrt-llm-engines-with-triton-inference-server-on-jetson-thor-and-compare-inference-speed-with-vllm-container/358802/1 "2026-01-27T09:11:55Z")
I’ve update Jetpack to 7.1 and followed the official **TensorRT Edge-LLM quick start guide** to convert and run the Qwen3-0.6B model on Jetson Thor with the standalone C++ runtime (using engine files). I want to **serve the same TensorRT-LLM engines using NVIDIA Triton Inference Server** and measure the _inference speed_ to compare against a vLLM docker image setup. So far I understand: 1. I can **export ONNX and build TensorRT engines** on Jetson Thor using the TensorRT-Edge-LLM repo and tools. 2. But I’m unsure how to prepare a **Triton model repository + config** for those engines and run Triton server with the TensorRT-LLM backend. **Questions:** 1. Is there a recommended workflow/tutorial for using **Triton Inference Server + TensorRT-LLM backend** for my engine files? 2. Any example of model repository structure + `config.pbtxt` for Qwen or similar models? 3. How to benchmark / measure inference speed properly (e.g., throughput, latency) between Triton serving and a standalone vLLM container? ---
**Author:** ![AastaLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aastalll/32/14043_2.png) [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)\ **Post date:** [January 28, 2026, 2:29am UTC](https://forums.developer.nvidia.com/t/how-to-serve-tensorrt-llm-engines-with-triton-inference-server-on-jetson-thor-and-compare-inference-speed-with-vllm-container/358802/3 "2026-01-28T02:29:05Z")
Hi, We need to check with the internal team. Will get back to you soon. Thanks. ---
**Author:** ![AastaLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aastalll/32/14043_2.png) [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)\ **Post date:** [February 2, 2026, 3:30am UTC](https://forums.developer.nvidia.com/t/how-to-serve-tensorrt-llm-engines-with-triton-inference-server-on-jetson-thor-and-compare-inference-speed-with-vllm-container/358802/4 "2026-02-02T03:30:47Z")
Hi, Currently, Triton Inference Server doesn’t TensorRT Edge LLM. But supporting a serving solution is on our roadmap. Do you consider Dynamo or Open AI compliant API? If not, are there any restrictions? Thanks. ---
**Author:** ![ckdavid233](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/ckdavid233/32/441248_2.png) [@ckdavid233](https://forums.developer.nvidia.com/u/ckdavid233)\ **Post date:** [February 2, 2026, 3:42am UTC](https://forums.developer.nvidia.com/t/how-to-serve-tensorrt-llm-engines-with-triton-inference-server-on-jetson-thor-and-compare-inference-speed-with-vllm-container/358802/5 "2026-02-02T03:42:45Z")
I found that DGX\_Spark can use the tutorial to run openwebui to support tensorrtllm.And luckily I have one for test（I‘ve tried last week and found it available.[TRT LLM for Inference | DGX Spark](https://build.nvidia.com/spark/trt-llm/open-webui-instructions)）.Can thor also use “trtserve” command to run tensorrt-model on Open AI compliant API?Btw,how can I test the speed on it?I need the help of this.Thank you! ---
**Author:** ![AastaLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aastalll/32/14043_2.png) [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)\ **Post date:** [February 2, 2026, 6:02am UTC](https://forums.developer.nvidia.com/t/how-to-serve-tensorrt-llm-engines-with-triton-inference-server-on-jetson-thor-and-compare-inference-speed-with-vllm-container/358802/6 "2026-02-02T06:02:11Z")
Hi, It’s not officially supported currently. You can benchmark with the \* llm\_inference\* binary: > Thanks. ---
**Author:** ![ckdavid233](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/ckdavid233/32/441248_2.png) [@ckdavid233](https://forums.developer.nvidia.com/u/ckdavid233)\ **Post date:** [February 2, 2026, 6:08am UTC](https://forums.developer.nvidia.com/t/how-to-serve-tensorrt-llm-engines-with-triton-inference-server-on-jetson-thor-and-compare-inference-speed-with-vllm-container/358802/7 "2026-02-02T06:08:36Z")
Thank you for your prompt reply. I will try the approach you mentioned as soon as possible. In addition, I look forward to seeing more practical and interesting application tutorials for Jetson Thor in the near future or over the coming months, which could be demonstrated to users for hands-on practice. Thanks again. ---
**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\ **Post date:** [February 16, 2026, 6:09am UTC](https://forums.developer.nvidia.com/t/how-to-serve-tensorrt-llm-engines-with-triton-inference-server-on-jetson-thor-and-compare-inference-speed-with-vllm-container/358802/8 "2026-02-16T06:09:20Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.