TensorRT LLM

Robotics & Edge Computing Jetson Systems Jetson Thor
,
1

I tried to run the TensorRT LLM container on Jetson Thor. But I encounter an error
ImportError: libnvinfer.so.10: cannot open shared object file: No such file or directory

I’m following the quickstart guide.

This is in in my docker compose file

  tensorrt-llm:
    image: nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc1
    ports:
      - 8000:8000
    ulimits:
      memlock: 1
      stack: 67108864
    gpus: all
    shm_size: 8gb
    ipc: host
    env_file:
      - .env  
    volumes:
      - $HOME/.cache:/root/.cache
      - $PWD:/workspace
    entrypoint:
      [
        "trtllm-serve",
        "--port", "8000",
        "--host", "0.0.0.0",
        "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
      ]
3

Hi,

Based on the release note, Thor’s GPU architecture (sm_110) hasn’t been added yet.
We will check with our internal team for more information and share it with you.

Thanks.

4

Hi,

TensorRT LLM doesn’t support Thor.
Please use vLLM or SGLang instead.

Thanks.

5

Thank you so much. That wasn’t clear to me.

I’ll use vLLM for now.
(You mention SGLang but from this pinned topic I understand that there’s no SGLang container yet. )

Do you expect that support will be added to TensorRT LLM for Thor’s GPU architecture (sm_110)?

6

I just pulled this container nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc1

It is version 1.2.0.rc1 and was published October 21, 2025. env show lots of stuff including sm_110. It is arm64. I am presuming it is using torch and/or triton as the backend; it has both installed.

I was attempting to run ./examples/…/llama but it failed probably because of my old llama3.1_instruct and I already turned off my Thor. I’ll try more tomorrow.

python convert_checkpoint.py
–model_dir /root/.cache/huggingface/hub/models-…-llama3.1 \
–output_dir ./tllm_checkpoint_1gpu_tp1 \
–dtype float16 \
–tp_size 1 \

trtllm-build --checkpoint_dir ./tllm_checkpoint_1gpu_tp1
–output_dir ./tmp/llama/8B/trt_engines/fp16/1-gpu/
–gemm_plugin auto

7

Hi,

SGLang is coming soon.
You can find some examples for vLLM in the link below:

Just to be clear that TensorRT-LLM won’t be available on Jetson.
But we will support TensorRT Edge-LLM on Thor in the upcoming release.

Thanks.

1 Like
9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.