How to Allocate Host Mem for dynamic model with batch_size > 1?

thuy.hoang19 · October 19, 2021, 11:50am

Description

I tried to convert our model by trtexec with dynamic shape. Then I tried to inference .( Please refer attchement for my code
But the problem is when batch_size >1 . I faced with bellow error:
Traceback (most recent call last):
File “/usr/lib/python3.8/threading.py”, line 932, in _bootstrap_inner
self.run()
File “infer_insight_face.py”, line 395, in run
batch_image_raw, use_time = self.trt_wrapper.infer(self.trt_wrapper.get_raw_image(self.image_path_batch))
File “infer_insight_face.py”, line 158, in infer
np.copyto(host_inputs[0], batch_input_image.ravel())
File “<array_function internals>”, line 5, in copyto
ValueError: could not broadcast input array from shape (75264,) into shape (37632,)
-------------------------------------------------------------------Traceback (most recent call last):
I know that because input data is batch_size*shape , but host_mem is locate with shape.
But how can I fix it?
Could you please support to fix it?

Environment

TensorRT Version: 8.2.0.6
GPU Type:
Nvidia Driver Version: TU102 [GeForce RTX 2080 Ti]
CUDA Version: 11.4.2
CUDNN Version:
Operating System + Version: Linux 20.0.4
Python Version (if applicable): 3.8.10
TensorFlow Version (if applicable):
PyTorch Version (if applicable): 1.9.1+cu102
Baremetal or Container (if container which image + tag):

Please include:

Exact steps/commands to build your repro
Exact steps/commands to run your repro
Full traceback of errors encountered

spolisetty · October 19, 2021, 1:54pm

Hi,

Please refer to below links for working with dynamic shapes, hope following will help you.
https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#work_dynamic_shapes

You can fine tune model using optimization profiles to specific input dim range
https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#opt_profiles

Thank you.

thuy.hoang19 · October 19, 2021, 2:25pm

Hello @spolisetty ,
Thank for your response!
I read all these documents. But i can not find how I can allocate host mem exactly and how to copy a batched data into hot buffer.
It can work fine with batch_size = 1.
Thank you!

spolisetty · October 21, 2021, 7:42pm

Hi,

Sorry currently we do not have python specific examples, Hope following may help you.

github.com

NVIDIA/TensorRT/blob/master/demo/BERT/infer.py#L43


      
              def __init__(self, shape, dtype=trt.int32):
                  self.buf = cuda.mem_alloc(trt.volume(shape) * dtype.itemsize)
          
          
    def binding(self):
                  return int(self.buf)
          
          
    def free(self):
                  self.buf.free()
          
          

          
def main():
              parser = argparse.ArgumentParser(description='BERT Inference Benchmark')
              parser.add_argument("-e", "--engine", help='Path to BERT TensorRT engine')
              parser.add_argument('-b', '--batch-size', default=[], action="append", help='Batch size(s) to benchmark. Can be specified multiple times for more than one batch size. This script assumes that the engine has been built with one optimization profile for each batch size, and that these profiles are in order of increasing batch size.', type=int)
              parser.add_argument('-s', '--sequence-length', default=128, help='Sequence length of the BERT model', type=int)
              parser.add_argument('-i', '--iterations', default=1, help='Number of iterations to run when benchmarking each batch size.', type=int)
              parser.add_argument('-w', '--warm-up-runs', default=0, help='Number of iterations to run prior to benchmarking.', type=int)
              parser.add_argument('-r', '--random-seed', required=False, default=12345, help='Random seed.', type=int)
              parser.add_argument('-p', '--passage', nargs='*', help='Text for paragraph/passage for BERT QA', default='')
              parser.add_argument('-q', '--question', nargs='*', help='Text for query/question for BERT QA', default='')
              parser.add_argument('-v', '--vocab-file', help='Path to file containing entire understandable vocab')

Thank you.

thuy.hoang19 · October 22, 2021, 1:34am

Let me try again.Thank you so much!