TRT7 + Algorithm selection - questions

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Hi,

Given that we implement an Algorithm Selection class that only provides 1 tactic per layer among the list of choices:

  1. Is tactic profiling still needed?

  2. If not, does TRT builder subscribe to system resources when building?

  3. If not, can we now have multiple TRT builder instances (threads) building in parallel? It’s not the case otherwise, Nvidia claims it’s “undefined behavior”.

  4. If profiling is not needed (we know in advance which tactics to use for each layer) - can we cross-build plan files for different GPU architectures? For example, given the tactic list for a Xavier architecture, can we build a plan file for that architecture on a regular x86 computer? (Instead of having to run TRT builder on the Xavier).

Thanks

2

Hi @carlosgalvezp
Please find the response below :
(1) in principle timing is not needed, but TRT is not optimized for this case because at the point TRT allows the user to select the algorithm, the potential tactics have all been timed.
(2) Yes
(3) This should be possible, but there are some open bugs against parallel builds on distinct GPUs (which is supported.)
(4) No, for multiple reasons. We understand that cross-compilation is desirable, but it’s signficant work to enable beyond just choice of tactics.

Thanks!

3

Thanks a lot for the replies!

So basically Algorithm Selection cannot be used as “a cache” (like the int8 calibration cache) to speed up the build process, since “there will always be profiling and timing of all the tactics”. It’s main purpose is then reproducibility - choose the same tactic every time.

That answers my question, thanks!

1 Like