TenorRT-LLM cpp_llm_only runs slower than python session mode

AI & Data Science Deep Learning (Training & Inference) TensorRT

tamir.manor March 9, 2025, 12:26pm 1

RTX-A5000 with docker built via official installation of tensor-rt llm guidelines. I’m using the supported InternVL2-8B model version. 4 tokens generation (using tensorrt-llm profiling option) takes 297msec e2e on session=‘cpp_llm_only’ mode and 166msec e2e on session=‘python’ which is very odd.
would appreciate your help

Topic		Replies	Views
Using tensorRT to accelerate caffe model， but it take more time to inference Jetson TX2	6	496	October 18, 2021
repeat post, please ignore this TensorRT	0	698	July 12, 2019
TensorRT int8 slower than FP16 due to reformat layer TensorRT tensorrt , cudnn	0	97	October 11, 2024
TensorRT 2x slower than Cudnn for single Conv2D (74 ms vs. 156 ms) TensorRT	6	825	February 5, 2021
Slow tensorflow-gpu execution on A100 GPU TensorRT	1	1126	May 19, 2021
A100 graphics card inference performance is not strong TensorRT	4	575	April 12, 2022
TensorRT Profiler reports unnamed layers TensorRT	2	790	May 2, 2018
Slow tensorflow-gpu execution on A100 Frameworks tensorflow	0	1533	May 19, 2021
TensorRT inference time extremely slow TensorRT	1	456	January 31, 2023
Why TensorRT model is slower? TensorRT tensorrt	3	1370	June 20, 2022

TenorRT-LLM cpp_llm_only runs slower than python session mode

Related topics