Deal the output cost more time when use the tensorRT, and why?

[toreceivead]

output = model_trt(input)
output.cpu()
# it costs about 1000 times than using the pytorch model to get the output
# when use the pytorch -- 0.00024
# when use the tensorrt -- 0.1923

[AastaLLL]

Hi,

Do you use PyTorch API?

When calling .cpu(), PyTorch will move the buffer from GPU back to CPU.
So this might takes some time according to the IO bandwidth.

To accelerate, you can try to boost the device with the following commands.

$ sudo nvpmodel -m 0
$ sudo jetson_clocks

Thanks.