GPU/CUDA start time

Hi!

We are developing an application were boot time is a significant issue as the users need to have our system up and running quickly in a very time constrained environment. For us boot time is from plugin to models loaded and running.

What we have found is that some trivial operations take a lot of time the first time after every boot.

For example (actual values are different)

mean = torch.Tensor([0.5, 0.5, 0.5]).cuda()

takes 11 seconds (on Xavier NX) which is same as the time it takes to load our model with torch2trt (another ~11 seconds).

I suppose this is due to cuda doing something at start. Is there a way to reduce this time? We need our boot time to be below 30 seconds from power on so 11 seconds is way too long.

Also is there a way to reduce the model load time? Specifically:

model_trt = torch2trt.TRTModule()
model_trt.load_state_dict(torch.load(optimized_model_path))

Thanks!

Hi,

Due to some dependence on the cuda-related libraries, PyTorch will spend sometime to load them into the memory.

Does your model work on pure TensorRT?
If yes, it’s recommended to convert the model into pure TensorRT (independent to PyTorch).
It should be relatively light-weight on the Jetson platform.

Thanks.

Hi,

Thanks, this specific model could be run using just tensorrt but we would need to replace a lot of other processing code that uses pytorch, I’ll look in to it!

Even if I were to remove the dependency on pytorch there would still be a dependency on cuda (all tensorrt samples use pycuda) so would this problem not persist? Or are you referring to something else?

Hi,

In our latest TensorRT, you can choose which backend library to deploy.
For example, running DNN without cuDNN and cuBLAS can save lots of memory and initial time.

$ /usr/src/tensorrt/bin/trtexec
&&&& RUNNING TensorRT.trtexec [TensorRT v8001] # /usr/src/tensorrt/bin/trtexec
...
=== Build Options ===
...
 --tacticSources=tactics     Specify the tactics to be used by adding (+) or removing (-) tactics from the default
                              tactic sources (default = all available tactics).
                              Note: Currently only cuDNN, cuBLAS and cuBLAS-LT are listed as optional tactics.
                              Tactic Sources: tactics ::= [","tactic]
                                              tactic  ::= (+|-)lib
                                              lib     ::= "CUBLAS"|"CUBLAS_LT"|"CUDNN"
                              For example, to disable cudnn and enable cublas: --tacticSources=-CUDNN,+CUBLAS
...

Thanks.