How to serve TensorRT-LLM engines with Triton Inference Server on Jetson Thor and compare inference speed with vLLM container?

Robotics & Edge Computing Jetson Systems Jetson Thor
1

I’ve update Jetpack to 7.1 and followed the official TensorRT Edge-LLM quick start guide to convert and run the Qwen3-0.6B model on Jetson Thor with the standalone C++ runtime (using engine files). I want to serve the same TensorRT-LLM engines using NVIDIA Triton Inference Server and measure the inference speed to compare against a vLLM docker image setup.

So far I understand:

  1. I can export ONNX and build TensorRT engines on Jetson Thor using the TensorRT-Edge-LLM repo and tools.

  2. But I’m unsure how to prepare a Triton model repository + config for those engines and run Triton server with the TensorRT-LLM backend.

Questions:

  1. Is there a recommended workflow/tutorial for using Triton Inference Server + TensorRT-LLM backend for my engine files?

  2. Any example of model repository structure + config.pbtxt for Qwen or similar models?

  3. How to benchmark / measure inference speed properly (e.g., throughput, latency) between Triton serving and a standalone vLLM container?

3

Hi,

We need to check with the internal team.
Will get back to you soon.

Thanks.

4

Hi,

Currently, Triton Inference Server doesn’t TensorRT Edge LLM.
But supporting a serving solution is on our roadmap.

Do you consider Dynamo or Open AI compliant API?
If not, are there any restrictions?

Thanks.

5

I found that DGX_Spark can use the tutorial to run openwebui to support tensorrtllm.And luckily I have one for test（I‘ve tried last week and found it available.TRT LLM for Inference | DGX Spark）.Can thor also use “trtserve” command to run tensorrt-model on Open AI compliant API?Btw,how can I test the speed on it?I need the help of this.Thank you!

6

Hi,

It’s not officially supported currently.
You can benchmark with the * llm_inference* binary:

Thanks.

7

Thank you for your prompt reply. I will try the approach you mentioned as soon as possible.
In addition, I look forward to seeing more practical and interesting application tutorials for Jetson Thor in the near future or over the coming months, which could be demonstrated to users for hands-on practice.
Thanks again.